Linux: Have I Been Hacked?
A server that has been compromised or hacked is an unauthorized bot or person that has logged into a server to use it for malicious activities.
However, the majority of hacked servers are carried out in mass scale by automated programs by hackers who aren't too sophisticated ( script kiddies ) or silly criminals.
These kind of attackers will abuse your server and it's resources while they have access to it, and some take precautions to hide what they are doing.
Before we move on please see /disclaimer/
Signs of a compromised server
More often when a server has been hacked by an amateur or automated attack there will be a spike in resources.
Usually this can be either CPU for mining crypto-currency or spamming emails, and the most common bandwidth for launching DDoS attacks. This would mean that one of the first indications something might be amiss is that the server is responding slower e.g website serving pages are slower than usual or emails are taking longer to send or receive.
So what should you look for?
Who’s currently logged in?
w
last
Find out if they left any shell history behind
tail -n 200 ~/.bash_history |morecat ~/.bash_history |moreSystem files that have been changed recently (in the last 2 days)
sudo find /etc /var -mtime -2Unknown Connections
More often an attacker will listen on the network for instructions. I recommend looking out for any processes in netstat (LISTEN or ESTABLISHED).
We can chain the following command to show these connections.
netstat -pluntWhat’s using all the CPU?
Hackers that don't take too many precautions to hide what they are doing will run processes to consume all the CPU. This usually makes it easier to spot them.
lsof will list all networked processes.
lsof -i
Look at the highest proceeses in:
top
Another favorite tool of mine is htop. htop is an interactive process viewer and system monitor. It's basically top on steroids.
You can also kill a process by pressing F9 or k
htop
My favorite command is ps auxf for providing the most information clearly. You can copy any unknown processes pid and pass it on to strace or lsof.
strace is a powerful command line tool for debugging, it captures all system calls made by a process and the signals received by the process.
If strace is not installed on your system, you may need to install it.
And let's not forget lsof has a similar option to strace.
ps auxf
strace -d -p PID
lsof -p PIDiftop works like top by showing processes that are sending and receiving network data along with their source and destination. A DDoS process or spamming will be immediately shown at the top of the list.
iftop
Check SSH logs
SSH logs are kept in /var/log/auth.log in a Debian distro, if you are on CentOS or RedHat distrubution they will be /var/log/secure.
Here is a example, where I'm checking the last 300 entries and grabbing only the results for sshd. This log files keeps entries for ftp,webmin, etc.
tail -n 300 /var/log/auth.log | grep sshdCheck for open ports
nmap localhostKernel Data Structures.
proc file system provides an interface to kernel data structures. Read the man page for more info.ls /proc/*/exe -la
Common folders to look for suspicious files
/tmp
/dev/shm
/var/tmpAuthorized Keys
More often attackers use authorized keys, be sure to look if you have any new entries.
/root/.ssh/authorized_keysCrontab jobs
It's quite common for attackers to add malicious files to be re-run in case the process gets killed.
less /etc/crontabSystemd Timers
This is another alternative to crontab. Timers are systemd unit files, their name end in .timer and control service files or events. They have built in support for calendar time events, monotonic time events, and can be run asynchronously.
systemctl list-timersThis command will return a list of ips which are connected to your server, to see connections ranging between 1 and 50 is common for a normal web server, however if you see 100+, this is something to look into.
netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -rTools
I will definitely recommend the following tools.
rkhunter scans for backdoors, rootkits and local exploits. This is accomplished by comparing SHA-1 hahshes of important files with known good ones from online databases. It also searches for known directories of rootkits, misuses of permisisions, hidden files, suspicious strings in modules and many others.
Installation:
apt install rkhunter
rkhunter --check[ Rootkit Hunter version 1.4.2 ]
Checking system commands...
Performing 'strings' command checks
Checking 'strings' command [ OK ]
Performing 'shared libraries' checks
Checking for preloading variables [ None found ]
Checking for preloaded libraries [ None found ]
Checking LD_LIBRARY_PATH variable [ Not found ]
Performing file properties checks
Checking for prerequisites [ OK ]
/usr/local/bin/rkhunter [ OK ]
/usr/sbin/adduser [ OK ]
/usr/sbin/chkconfig [ OK ]
/usr/sbin/chroot [ OK ]
/usr/sbin/depmod [ OK ]
/usr/sbin/fsck [ OK ]
/usr/sbin/fuser [ OK ]
/usr/sbin/groupadd [ OK ]
/usr/sbin/groupdel [ OK ]
/usr/sbin/groupmod [ OK ]
/usr/sbin/grpck [ OK ]
/usr/sbin/ifconfig [ OK ]
/usr/sbin/ifdown [ Warning ]
/usr/sbin/ifup [ Warning ]
/usr/sbin/init [ OK ]
/usr/sbin/insmod [ OK ]
/usr/sbin/ip [ OK ]
/usr/sbin/lsmod [ OK ]
/usr/sbin/lsof [ OK ]
/usr/sbin/modinfo [ OK ]
/usr/sbin/modprobe [ OK ]
/usr/sbin/nologin [ OK ]
/usr/sbin/pwck [ OK ]
/usr/sbin/rmmod [ OK ]
/usr/sbin/route [ OK ]
/usr/sbin/rsyslogd [ OK ]
/usr/sbin/runlevel [ OK ]
/usr/sbin/sestatus [ OK ]
/usr/sbin/sshd [ OK ]
/usr/sbin/sulogin [ OK ]
/usr/sbin/sysctl [ OK ]
/usr/sbin/tcpd [ OK ]
/usr/sbin/useradd [ OK ]
/usr/sbin/userdel [ OK ]
/usr/sbin/usermod [ OK ]
....
[Press to continue]
Checking for rootkits...
Performing check of known rootkit files and directories
55808 Trojan - Variant A [ Not found ]
ADM Worm [ Not found ]
AjaKit Rootkit [ Not found ]
Adore Rootkit [ Not found ]
aPa Kit [ Not found ]
.....
[Press to continue]
Performing additional rootkit checks
Suckit Rookit additional checks [ OK ]
Checking for possible rootkit files and directories [ None found ]
Checking for possible rootkit strings [ None found ]
....
[Press to continue]
Checking the network...
Performing checks on the network ports
Checking for backdoor ports [ None found ]
....
Performing system configuration file checks
Checking for an SSH configuration file [ Found ]
Checking if SSH root access is allowed [ Warning ]
Checking if SSH protocol v1 is allowed [ Warning ]
Checking for a running system logging daemon [ Found ]
Checking for a system logging configuration file [ Found ]
Checking if syslog remote logging is allowed [ Not allowed ]
...
System checks summary
=====================
File properties checks...
Files checked: 137
Suspect files: 6
Rootkit checks...
Rootkits checked : 383
Possible rootkits: 0
Applications checks...
Applications checked: 5
Suspect applications: 2
The system checks took: 5 minutes and 38 seconds
All results have been written to the log file: /var/log/rkhunter.log
One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)chkrootkit aka check rootkit is a common unix based program built to help sys admins check their system for known rootkits. It's using signatures from core system files and comparing a traversal of the /proc filesystem with the output of the process status to look for discrepancies.
Installation
apt install chkrootkit
What should you do if you've been compromised?
The fact of the matter is that the attacker is already in, however do not panic.
You need to take control of the machine before they are aware that you know about them. If they find out you know about them they might lock you out of the server and destroy assets out of spite.
If you aren't a technical person then simply shutdown the server.
shutdown -h now
systemctl poweroff
This can also be done via your hosting provider's cPanel. Once its powered off you can think of some firewall rules and consult with your hosting provider.
However if you are more confident and you have access to a firewall in your hosting provider's cPanel, you need to add 2 rules.
- Allow SSH only from your IP address.
- Block every protocol on every port.
This operation will immediately kick them out the SSH session.
If you do not have access to a firewall you can find out their session pid from ps aux, who, and use kill to kick them out. Change passwords immediately.
As a last method, where possible, is to connect to the server via an out-of-band connection such as a serial console, and stop all the networking.
systemctl stop network.serviceThe most sensible move would be to copy all your data and start a fresh install.
Continue reading about How to clear your tracks, The proper way!
Thank you for reading my article, Until next time!
Your friendly neighbourhood Hacker.