Thoughts, stories and ideas.

A server that has been compromised or hacked is an unauthorized bot or person that has logged into a server to use it for malicious activities.

However, the majority of hacked servers are carried out in mass scale by automated programs by hackers who aren't too sophisticated ( script kiddies ) or silly criminals.

These kind of attackers will abuse your server and it's resources while they have access to it, and some take precautions to hide what they are doing.

Before we move on please see /disclaimer/

Signs of a compromised server

More often when a server has been hacked by an amateur or automated attack there will be a spike in resources.
Usually this can be either CPU for mining crypto-currency or spamming emails, and the most common bandwidth for launching DDoS attacks. This would mean that one of the first indications something might be amiss is that the server is responding slower e.g website serving pages are slower than usual or emails are taking longer to send or receive.

So what should you look for?

Who’s currently logged in?

w
output of w, who
last
last command

Find out if they left any shell history behind

tail -n 200 ~/.bash_history |more
cat ~/.bash_history |more

System files that have been changed recently (in the last 2 days)

sudo find /etc /var -mtime -2

Unknown Connections

More often an attacker will listen on the network for instructions. I recommend looking out for any processes in netstat (LISTEN or ESTABLISHED).
We can chain the following command to show these connections.

netstat -plunt

What’s using all the CPU?

Hackers that don't take too many precautions to hide what they are doing will run processes to consume all the CPU. This usually makes it easier to spot them.

lsof will list all networked processes.

lsof -i
output from lsof

Look at the highest proceeses in:

top
output from top

Another favorite tool of mine is htop. htop is an interactive process viewer and system monitor. It's basically top on steroids.
You can also kill a process by pressing F9 or k

htop
output from htop

My favorite command is ps auxf for providing the most information clearly. You can copy any unknown processes pid and pass it on to strace or lsof.

strace is a powerful command line tool for debugging, it captures all system calls made by a process and the signals received by the process.

If strace is not installed on your system, you may need to install it.
And let's not forget lsof has a similar option to strace.

ps auxf
strace -d -p PID
lsof -p PID

iftop works like top by showing processes that are sending and receiving network data along with their source and destination. A DDoS process or spamming will be immediately shown at the top of the list.

iftop
output from iftop

Check SSH logs

SSH logs are kept in /var/log/auth.log in a Debian distro, if you are on CentOS or RedHat distrubution they will be /var/log/secure.
Here is a example, where I'm checking the last 300 entries and grabbing only the results for sshd. This log files keeps entries for ftp,webmin, etc.

tail -n 300 /var/log/auth.log | grep sshd

Check for open ports

nmap localhost
Kernel Data Structures. 
proc file system provides an interface to kernel data structures. Read the man page for more info.
ls /proc/*/exe -la
output from proc

Common folders to look for suspicious files

/tmp
/dev/shm
/var/tmp

Authorized Keys


More often attackers use authorized keys, be sure to look if you have any new entries.

/root/.ssh/authorized_keys

Crontab jobs

It's quite common for attackers to add malicious files to be re-run in case the process gets killed.

less /etc/crontab

Systemd Timers

This is another alternative to crontab. Timers are systemd unit files, their name end in .timer and control service files or events. They have built in support for calendar time events, monotonic time events, and can be run asynchronously.

systemctl list-timers

This command will return a list of ips which are connected to your server, to see connections ranging between 1 and 50 is common for a normal web server, however if you see 100+, this is something to look into.

netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r

Tools

I will definitely recommend the following tools.

Rootkit Hunter / Wiki / index

rkhunter scans for backdoors, rootkits and local exploits. This is accomplished by comparing SHA-1 hahshes of important files with known good ones from online databases. It also searches for known directories of rootkits, misuses of permisisions, hidden files, suspicious strings in modules and many others.

Installation:

apt install rkhunter
rkhunter --check
manual check rkhunter
[ Rootkit Hunter version 1.4.2 ]

Checking system commands...

  Performing 'strings' command checks
    Checking 'strings' command                               [ OK ]

  Performing 'shared libraries' checks
    Checking for preloading variables                        [ None found ]
    Checking for preloaded libraries                         [ None found ]
    Checking LD_LIBRARY_PATH variable                        [ Not found ]

  Performing file properties checks
    Checking for prerequisites                               [ OK ]
    /usr/local/bin/rkhunter                                  [ OK ]
    /usr/sbin/adduser                                        [ OK ]
    /usr/sbin/chkconfig                                      [ OK ]
    /usr/sbin/chroot                                         [ OK ]
    /usr/sbin/depmod                                         [ OK ]
    /usr/sbin/fsck                                           [ OK ]
    /usr/sbin/fuser                                          [ OK ]
    /usr/sbin/groupadd                                       [ OK ]
    /usr/sbin/groupdel                                       [ OK ]
    /usr/sbin/groupmod                                       [ OK ]
    /usr/sbin/grpck                                          [ OK ]
    /usr/sbin/ifconfig                                       [ OK ]
    /usr/sbin/ifdown                                         [ Warning ]
    /usr/sbin/ifup                                           [ Warning ]
    /usr/sbin/init                                           [ OK ]
    /usr/sbin/insmod                                         [ OK ]
    /usr/sbin/ip                                             [ OK ]
    /usr/sbin/lsmod                                          [ OK ]
    /usr/sbin/lsof                                           [ OK ]
    /usr/sbin/modinfo                                        [ OK ]
    /usr/sbin/modprobe                                       [ OK ]
    /usr/sbin/nologin                                        [ OK ]
    /usr/sbin/pwck                                           [ OK ]
    /usr/sbin/rmmod                                          [ OK ]
    /usr/sbin/route                                          [ OK ]
    /usr/sbin/rsyslogd                                       [ OK ]
    /usr/sbin/runlevel                                       [ OK ]
    /usr/sbin/sestatus                                       [ OK ]
    /usr/sbin/sshd                                           [ OK ]
    /usr/sbin/sulogin                                        [ OK ]
    /usr/sbin/sysctl                                         [ OK ]
    /usr/sbin/tcpd                                           [ OK ]
    /usr/sbin/useradd                                        [ OK ]
    /usr/sbin/userdel                                        [ OK ]
    /usr/sbin/usermod                                        [ OK ]
....
[Press  to continue]


Checking for rootkits...

  Performing check of known rootkit files and directories
    55808 Trojan - Variant A                                 [ Not found ]
    ADM Worm                                                 [ Not found ]
    AjaKit Rootkit                                           [ Not found ]
    Adore Rootkit                                            [ Not found ]
    aPa Kit                                                  [ Not found ]
.....

[Press  to continue]


  Performing additional rootkit checks
    Suckit Rookit additional checks                          [ OK ]
    Checking for possible rootkit files and directories      [ None found ]
    Checking for possible rootkit strings                    [ None found ]

....
[Press  to continue]


Checking the network...

  Performing checks on the network ports
    Checking for backdoor ports                              [ None found ]
....
  Performing system configuration file checks
    Checking for an SSH configuration file                   [ Found ]
    Checking if SSH root access is allowed                   [ Warning ]
    Checking if SSH protocol v1 is allowed                   [ Warning ]
    Checking for a running system logging daemon             [ Found ]
    Checking for a system logging configuration file         [ Found ]
    Checking if syslog remote logging is allowed             [ Not allowed ]
...
System checks summary
=====================

File properties checks...
    Files checked: 137
    Suspect files: 6

Rootkit checks...
    Rootkits checked : 383
    Possible rootkits: 0

Applications checks...
    Applications checked: 5
    Suspect applications: 2

The system checks took: 5 minutes and 38 seconds

All results have been written to the log file: /var/log/rkhunter.log

One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)
rkhunter scan output

chkrootkit aka check rootkit is a common unix based program built to help sys admins check their system for known rootkits. It's using signatures from core system files and comparing a traversal of the /proc filesystem with the output of the process status to look for discrepancies.

Installation

apt install chkrootkit
output from chkrootkit

What should you do if you've been compromised?

The fact of the matter is that the attacker is already in, however do not panic.
You need to take control of the machine before they are aware that you know about them. If they find out you know about them they might lock you out of the server and destroy assets out of spite.

If you aren't a technical person then simply shutdown the server.

shutdown -h now
systemctl poweroff


This can also be done via your hosting provider's cPanel. Once its powered off you can think of some firewall rules and consult with your hosting provider.

However if you are more confident and you have access to a firewall in your hosting provider's cPanel, you need to add 2 rules.

  1. Allow SSH only from your IP address.
  2. Block every protocol on every port.

This operation will immediately kick them out the SSH session.

If you do not have access to a firewall you can find out their session pid from ps aux, who, and use kill to kick them out. Change passwords immediately.

As a last method, where possible, is to connect to the server via an out-of-band connection such as a serial console, and stop all the networking.

systemctl stop network.service

The most sensible move would be to copy all your data and start a fresh install.

Continue reading about How to clear your tracks, The proper way!

Thank you for reading my article, Until next time!

Your friendly neighbourhood Hacker.


You've successfully subscribed to Flaviu Popescu
Welcome back! You've successfully signed in.
Great! You've successfully signed up.
Success! Your account is fully activated, you now have access to all content.