Hackers made it onto one of our production servers 😅. We’ve isolated it from the internet until we can clean the machine up. The IR team reported eight difference backdoors on the server, but didn’t say what they were and we can’t get in touch with them. We need to get this server back into prod ASAP – we’re losing money every second it’s down. Please find the eight backdoors (both remote access and privilege escalation) and remove them. Once you’re done, run /root/solveme as root to check. You have SSH access and sudo rights to the box with the connections details attached below.

PersistenceIsFutile

First check the .bashrc and a low-hanging fruit is there. Comment out it then “Issue 6 is fully remediated“

Check crontab by “crontab -l” and there is a shell to be executed by whatever dig return. “crontab -e” to get the “Issue 8 is fully remediated“

Check cron jobs

/etc>ls -al | grep "cron"
/etc>find cron.*

There are 2 files not are normally there under cron.daily folder

access-up is creating 6 digit random file name and setup 4755 permission for those files

>find / -perm 4755 2>/dev/null

Remove access-up + 3 random 6 digits filename file + “.backdoor” we got “Issue 7 is fully remediated“

Back to pyssh, delete the specific row in “/root/.ssh/authorized_keys” + remove /lib/python3/dist-packages/ssh_import_id_update + pyssh = “Issue 2 is fully remediated“

Look the SetUID Privilege Escalation and found several suspicious-looking files

>find / -type f -user root -perm -4000 -ls 2>/dev/null

Remove ppppd and got “Issue 3 is fully remediated“

Check the process

>ps auxww
>ps auxef

A reverse shell in connectivity-check and got “Issue 5 is partially remediated“

Search the trigger of connectivity-checks

>find / -type f  -name *connectivity-check* 2>/dev/null

Remove 30-connectivity-check + kill every connectivity process = “Issue 5 is fully remediated“

Regarding to alertd, check the functionality by “alertd -h”

>kill -9 [pid]

Kill all process relates to alertd, we got “Issue 1 is partially remediated“

>find / -name "alertd" 2>/dev/null
>rm /usr/bin/alertd

After that I aware of the trigger is written in root .bashrc. Remove it then “Issue 1 is fully remediated“

Check if there are malicious account created by the hacker

Check shadow the content shows gnats is registered as a logon user, but passwd shows it should be a none logon account.

After revised the passwd and shadow, we got “Issue 4 is partially remediated“. Delete the row in shadow and make gnats to /nologin makes “Issue 4 is fully remediated“