Different Way on How to Troubleshooting hacked server

[hotfuzz]

It is an unfortunate fact of life that web servers get hacked, through many different vectors including outdated scripts, compromised CMS platforms, weak passwords and many, many others. In such situation, you may receive an abuse notification via email requesting that action be taken to stop outbound abusive behaviour from being carried out on your server.

First, let's grab a list of all processes running on the server, along with all of their gory details:

[INDENT][B]/bin/ps axfwwwe -eo ppid,pid,uid,cmd --sort=ppid[/B][/INDENT]

Usually this gives enough clues to find out the cheeky culprit, allowing you to begin killing processes (kill -9 XXX, etc) and deleting or moving the hacked files. If you're struggling and know that there is abusive behaviour occurring at this very moment, grabbing a list of all open files will give some extra insight:

[INDENT][B]/usr/sbin/lsof -Pwn[/B][/INDENT]

And finally, for that extra confirmation, we can use netstat to verify all currently active network connections which will give you the concrete evidence to find which processes are behaving badly:

[INDENT][B]/bin/netstat -anpe[/B][/INDENT]

Hopefully this helps you on your sysadmin cleanup journey!

[hotfuzz]

Monitoring network connections with netstat

Netstat is a useful Linux tool to help understand who is connected to/from your server, which ports are in use, which programs are bound to specific ports, and many more things. Without further ado, here is a short but handy list of netstat commands:

List all outgoing UDP, TCP and RAW connections, in numerical form, showing the associated process:

[B]netstat -nputw[/B]

List all TCP and UDP connections to a server, sorted in order of IPs, by largest number of connections first:

[B]netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n[/B]

Show TCP total connection count by "state" (useful for checking and debunking DoS attacks):

[B]netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n[/B]

Displays a total connection count to TCP 443:

[B]netstat -anp | grep :443 | wc -l[/B]

Displays a total count of "established" (sending/receiving) connections to TCP 443:

[B]netstat -anp | grep :443 | grep ESTABLISHED | wc -l[/B]

Watch a live list of all TCP, UDP and RAW connections to/from a server:

[B]watch netstat -n -A inet[/B]

[hotfuzz]

Determining cause behind high load average

ps aux | awk '{print $11}' | sort | uniq -c | sort -nk1 | tail -n5

ps aux | awk '{print $1}' | sort | uniq -c | sort -nk1 | tail -n5

Both of those will provide information on how many threads are running of one service or user.

Check Top Processes sorted by RAM or CPU Usage in Linux

The following command will show the list of top processes ordered by RAM and CPU use in descendant form (remove the pipeline and head if you want to see the full list):

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head