GoDaddy Security update

My last post GoDaddy store your passwords in clear-text and may try to SSH to your VPS without permission got a lot of traction and it reached the ears of the GoDaddy people!

I just got off the phone with Neil Warner, GoDaddy’s CSO (Chief Security Officer) and he explained the situation to me.

First, I was glad that they heard the customers, heard the complains and took the time to look at it. That was his explanation:

  1. They take security serious and spend a lot of money on intrusion/malware detection to protect their customers
  2. They have a security team 24/7 monitoring all their shared/VPS and private servers
  3. When they detect any issue, they try to fix the problem and that’s why they tried to access my box
  4. They store all the passwords encrypted (not one-way hashed which is the recommended), and they can only be retrieved and reversed after a member of the security team opens a ticket and explains the reason for using the password (like to investigate malware)

One thing that made me feel better was that they actually have a process in place to access the passwords and they hold their people accountable for that. Having them encrypted or in clear-text doesn’t make much a difference, if the process to recover them is open to anyone in their staff…

He said that most users like their free incident response and malware removal and the way they deal with security issues.

He also said that they should have contacted me before accessing the box, warning me of the possible malware, and that they will do that from now on (good to know).

I am happy they called and explained the situation. +1 for GoDaddy for being open, explaining the issue and trying to improve.

Scan your website for free:
About David Dede

Sucuri Security bot (crazy work) - Malware research updates, sucuri news and more.

  • http://hellosorld.com Andrew Benton

    wow, you're letting them off the hook pretty easily don't you think? i mean this is supposed to be a blog about security right?!?

  • http://www.blogger.com/profile/14980808976404159238 http://sucuri.net

    Andrew Benton: Not off the hook, no. The issue still stands that they store the passwords in a retrievable format. About them accessing the servers, he said that they will change their policy to communicate first with the client, which is a big PLUS for me. If anyone wants their malware removal service, it is fine for me.

    But just for calling back and talking about improvements, gives them a +1 (they are famous for ignoring their users).

  • Anonymous

    Alright, how much credit did they put on your account for you to post this?

  • http://www.blogger.com/profile/11279473912460148881 Dennis Groves

    Unfortunately, it doesn't explain nor forgive that they logg passwords in clear text. Their are both standards and devices for logging securely.

  • Anonymous

    Alright, how much credit did they put on your account for you to post this?

  • http://www.blogger.com/profile/05018134738510159518 Joel Esler

    I agree. At least they owned up to it. That's a Plus for any company to admit fault, fix it, and move on.

  • Anonymous

    so how did they actually get your passwords?

  • http://www.blogger.com/profile/14980808976404159238 http://sucuri.net

    Anonymous: How did they get the password? It is encrypted, not one-way hashed. So, they just used their "master" key to decrypt i guess…

  • Anonymous

    Ha: I did the anonymous post about encrypted != cleartext on the prior topic and was waved down for being pedantic. Here's a suggestion for a post, as a service to the community:

    What Fancy Security Terms Really Mean: How to Use Words Like "Encryption" and "One-way hash" and "Cleartext" and "Public Key" without sounding like a moron.

    Seriously, people really don't understand this stuff. The more they say they care about security, the less likely they are to understand the basics. Sometimes public-key makes sense, sometimes one-way hashing makes more sense. But throwing buzzwords around and saying everything should be one-way all the time doesn't make sense.

  • Anonymous

    I would much rather godaddy give its users the choice of whether to allow godaddy's security team access to my password or not.

    "If you want speedy responses and the burgler alarm to sound every few weeks, then check the box."

    The other issue I have with that security response is the e-mail you received from godaddy asking you for even more information. If I saw an e-mail like that immediately after I detected security issues I would probably think it was from an impostor not godaddy.com.

    Even if the system works the way its meant to, it still makes me lean farther toward hosting my own servers.

  • http://nowhere.com Joe

    I think the standard practice with other providers is to have a separate admin account that can login to the VPS and whose actions are easily audited.

    Using the VPS owner's account and password is ridiculous and could cause a legal liability for the VPS owner.

  • http://www.blogger.com/profile/17357373832895000624 Kevin

    One more example for nodaddy.com

  • http://www.blogger.com/profile/08102466843919236000 JJC

    I'll agree that it was a +1 (to a mild degree) that they called you. Unfortunately what about the average joe user that doesn't stir the pot. They only contacted you because you actually hit the radar. Had you been joe user and noticed this then called back in, my money says that it would have gone ignored.

    To further the point that it was simply "damage control" they have followed everyone that RTed or Posted information about the original unauthorized SSH access.

    Of course, I am sure that if you read the ToS that you agreed to, it basically says that they own your "rented" VPS and thusly can do whatever they want to with it. This is not the only shady thing that they do, IMHO.. but that's a topic for another day / beers!

  • http://www.blogger.com/profile/08102466843919236000 JJC

    I missed part of my second paragraph there, they followed and then tweeted "Just FYI, the author posted an update to his blog. It clarifies a lot. We hope you'll read that too. http://fwd4.me/GxT&quot; <- uh, ok

  • http://www.blogger.com/profile/11061967917509053185 CG

    i agree with the commenter who had issue with logging in with your creds to do analysis.

    how do you audit that?

    if was an intrusion now all the logs are jacked because they were in there.

    i'd personally rather them firewall off my host and email me

    additionally you should have received an email when they created an ticket about your account to get access to your creds. the fact that you didnt should be a big red flag and opens up issues of who at godaddy can arbitrarily just retrieve your passwords without your knowledge. not cool.

  • Anonymous

    You people are being drama queens. Who cares if they have a master password to decrypt your passwords? They are encrypted and it sounds like they have a good process for who can access it and when. Any true security personnel (sorry, some of you are sounding pretty amateur) know that it is a zero sum game when the people you are whining about have physical access to your box anyway. What do you want them to do instead? Shut it down?

    Worry about something real.

    (Disclaimer: I do not work for them — I found this on twitter)

  • Anonymous

    Wow. Godaddy has no business checking your servers IMO; the absolute first action should be to contact you first, draconian user agreements or no. I don't have a single VPS with them, but if Slicehost or Linode did this i'd be out of there in a heartbeat. The domains I have with Godaddy will be moved – I already knew they were bad, but this just confirms it.

  • Anonymous

    All of the providers keep the passwords available. If you are such experts and can do it yourself, just set up your own dedicated hosting somewhere..

    These are managed services. They're meant to.. manage.. them.

  • Anonymous

    I love security blogs that need javascript to post a comment…

    Anyhow, I was attracted here by Slashdot. Woohoo. I'd just like to chime in and ask for citations…also, "passwords should not be stored encrypted" is rather blanket. For instance, in some cases it may be suitable to public-encrypt a key, such that a password can be verified (I send you a phrase, you encrypt it with your private key, I can then read it with your public one…). Said facility could potentially store the private key in a somewhat secure location. (Remember: there is no absolute security, only how much trouble you're willing to put an attacker through). Said facility could then contact the said server and login, with a one-time encryption performed at the private key storage facility, where the private key itself is never exposed.

    Now, obviously GoDaddy isn't doing this. My point is only to illustrate that, it is feasible that "you should ONLY STORE PASSWORD HASHES!!11" is rather narrow-minded. It's that kind of thinking that tricks you into believing you're secure.

    Example: OS X only stores password hashes. I recently hacked a 10.4.2 laptop by booting up into terminal mode as a privileged (but not root) user, tricking it into root, and erasing a config file. I couldn't access the hash file, because if I could, I could've simply generated the hash for the password I wanted; rather, the file I deleted made the OS think it was a first-run. I chose the same username and it let me set a password after that all on its own.

    Again, the point is just to demonstrate that following one important security rule doesn't guarentee security. It makes it a little more difficult, sure, but sometimes you need a little less difficulty yourself – such as the public/private key scenario described above.

  • Anonymous

    On that post above where I wrote 'tricking it into root,' I should've written 'attained elevated privileges that were not root."

  • Anonymous

    Question about how much they paid you is still unanswered.
    You are either shilling or stupid.

  • Anonymous

    Great to know that if a bigwig calls personally, all is well with the world and their security process is *great*! "Sorry we raped your cat, we'll have our CEO call to smooth things over for ya." If you believe that they're really changing their procedures, I've got a bridge…

    I get a lot of (stupid, poorly scripted) attacks from GoDaddy servers and the first thing I would've thought had I seen your logs was not that it had anything to do with GoDaddy admins. Of course, the good passwords and follow-up email obviate that.

    Seriously, their entire way of dealing with this is screwed up. You're running a "honeypot," you probably had one of those aforementioned braindead, sad attacks emanating from it. How should they have responded?

    1. Called you immediately and worked with you to fix the issue.
    2. If no response, email.*
    3. If no response in a few hours, shut down the network connectivity to your server.

    * If exploit/attacks actually *work* and aren't the standard lame lowest common denominator attacks GoDaddy customers are famous for, skip immediately to 3, *minus* the waiting a few hours.

    My guess is they're totally overwhelmed by this stuff. Kudos to them for gauging the situation correctly and figuring out how easy it was to turn you from outraged to shill.

  • Anonymous

    If I had a VPS with godaddy (which I do not, although I did have a full dedicated server with them a while back), then I would prefer that their security teams be able to look inside all the VPS's and clearing/disabling malware.

    To the person commenting at 8:25, absolutely not. If the detected attacks emanating from it, the first they should have have done is immediately disconnect it from the internet. If you can't manage/secure your own VPS, then you deserve to get kicked off when you get hacked. Don't make the rest of the internet suffer because of your inability to secure your own stuff.

    If by some strange accident I do get malware on a VPS at godaddy, if they can clean it/disable the malware rather than taking the whole machine off the internet, then please do so, and contact me as soon as possible to let me know what happened and why.

  • Anonymous

    Haha, you so got paid in credits to say good stuff about them. LOL its so obvious!

  • Nick Berg

    As to opening a ticket and explaining the reason, how many times do you think the request is rejected? "I need the password to investigate the source of suspicious traffic coming from the IP address which has been confirmed to belong to this account owner".

    Additionally, you say they store all the passwords with reversible encryption. Is this limited to GoDaddy's logins to your server, or do they also store your passwords in a recoverable manner? If the latter, I sure hope you don't reuse the same password on multiple services otherwise GoDaddy personnel can now breach those services too.

  • http://www.protocol16.com Justin

    I'm honestly not surprised by this situation. While I use GoDaddy's reg system, I do not use their hosting, nor would I. Too many oversold hosts with domains used by a lot of spammers and such.

    I'm getting to the point where I'll self host to stop this type of problem. My own host has rooted my box when I've complained about weird things happening.

  • Anonymous

    avast is giving me malware warnings with your site. win7/chrome beta/avast 4.8

  • Anonymous

    It is NEVER permissible to store passwords of any kind for any reason. What was rule #1 when we all got our first log-ins to a network?
    The secure and prudent way to handle is is to contact the client first and ask the client for creds to access the server at which time they authenticate with you. If they store passwords in any format, the password is retrievable.
    If the possibility of impropriety is >0 then it is an eventual certainty that it will happen. Do you want that to be your server & data that it happens to? I'll host my own as its OK to keep all your eggs in one basket, just keep your eye on that basket.

  • Anonymous

    Excellent point on the liability aspect from prior poster… anyone committing shenanigans from a VPS provided by godaddy could, in court, present reasonable doubt that a godaddy staff member put all that _insert shenanigan_ on my VPS. Seems like something godaddy would want to avoid.

  • http://www.blogger.com/profile/07725726689244201897 carp

    Actually a little correction. A one way hash is the "best practice" way to store a password, for the purpose of checking it. Since the hash can only be generated reliably by the person who already knows the password (or, more strictly, the software that he gives his password to).

    If they are storing passwords to be used, a hash is useless. They may as well store nothing at all. You can't log in to a system with only a password hash.

    Frankly, they should make this a service that they offer, and not be trying to capture your password and store it without your upfront knowledge.

    However, in terms of doing it, encrypted, and NOT hashed passwords is the proper way to do it. Hopefully there is proper access control and proper logging.

    Of course, it also means that anyone who uses this service should change their root password anytime a godaddy employee leaves the company. You think they will notify you?

    Also… you should be letting them know you are running a honeypot on their network or you are asking to be shut down.

    And anonymous:

    Many security professionals disagree with you on the never part. Many sysadmins keep personal encrypted password databases. Its preferable to losing important passwords, overusing them, or keeping them too simple.

  • Anonymous

    You know everyone is so concerned about the passwords. How bout the fact that they tried to login as root by being clever. Im not sure what the agreement is but you can be sure I will never use godaddy.

  • Rob

    "One thing that made me feel better was that they actually have a process in place to access the passwords and they hold their people accountable for that."

    It's YOU that should have a process in place to access the passwords. It's YOU that should hold people accountable for that.

    Excuse the question, but did you receive anything from them to apologise? Or merely an explanation?

  • Anonymous

    I work at an MSP with over 100 clients, each client has at least 5-10 set of credentials (databases, systems, application servers, application accounts, network devices, etc.) The environment of course is mixed. What would you recommend as far as storing all of these sets of credentials in a centralized manner?

  • http://blog.jasonantman.com Jason Antman

    Wow. Some utterly amazing… insights… here (sarcastic, and referring to some of the comments, not your post).

    1) VPS – "Private" means that other users shouldn't be able to get to your data, not that admins shouldn't.

    2) Unless they give a proficiency test in admin and security before letting you rent a server, they have every right to protect their network.

    3) Who cares if they post the root passwords in plaintext on the data center wall? If someone wants root, they can just walk over to the box and pop out the drives…

  • Anonymous

    Sure, I see your point…. and -20 for trying to login as root, anyway.

  • Anonymous

    The guy saying everyone is being drama queens, you are a fool. One point here is that your credentials are decryptable, if someone hacks their system they get your password. That is a major security flaw.
    Also, another fact everyone is upset, is they tried to login without permission. This can really hose you up in a security audit (my work has had many, and we almost failed because of some logins from testing the security of a system).

    Don't act all high and mighty because you don't care about security or privacy.

  • Anonymous

    I disagree with the theory hashed passwords are necessarily better than encrypted passwords.

    Hashed passwords are often able to be compromised via offline brute force attack.

    Hashed passwords preclude use of advanced authentication algorithms with mutual authentication features.

  • B. H.

    That is absolutely ridiculous and I would not stand for any hosting provider to access a virtual or physical server without my permission. Oddly enough, the majority of virtualization software allows the administrator of the physical server to access any virtual server running on the physical server without needing to know the credentials. I work for a virtual hosting provider and we have strict rules in place about accessing customer servers. The only case in which a customer server would be accessed would be if the customer used a known stolen credit card or paypal account to purchase the system (most of the time, the virtual server is just destroyed as any files on the system), if there was an official supoena (information only disclosed if required), or if the customer requested it for work on the server. Any activities which warranted stopping activities sourced from the virtual server would involve first contacting the customer, and if no action was taken, just shutting down the virtual server.

    The activities you described in the article are a direct violation of your privacy. I'd find a new hosting provider soon.

  • Anonymous

    Did they actually explain what situation/malware prompted their attempted fixing of the server? Was your server really infected?

  • http://www.blogger.com/profile/01292170507614624819 janus zeal

    It is rather upsetting they would allow this, considering how horribly easy it is to social GoDaddy staff into giving out access. :\

  • Anonymous

    I'm confused. Was this a "managed server" or an "unmanaged server" ?

    If it was managed, then I think the owner should shut up, as godaddy was only fighting the good fight.

    If it was unmanaged, then I would stop using their servers and go somewhere else like Jaguar PC or something.

  • Anonymous

    Personally if I found out that my network network provider stored my password in a decryptable format, attempted to access one of my servers for any reason with my account username and password. Then attempted to justify it for any reason I would be looking for a new provider post haste.

    The fact of the mater is poor practices are a much bigger security issue than technical failures. Any firm that does not realize why making passwords decryptable is a very bad thing has a concerning lack of understanding of both security and privacy.

    There are better ways to manage remote access (white lists etc) if providing access is required as part of the terms and conditions. If it isn't then shouldn't such a service be opt-in?

  • Anonymous

    "I work at an MSP with over 100 clients, each client has at least 5-10 set of credentials (databases, systems, application servers, application accounts, network devices, etc.) The environment of course is mixed. What would you recommend as far as storing all of these sets of credentials in a centralized manner?"

    KeepassX works nice for passwords.
    http://www.keepassx.org/
    Cross platform also. It's eliminated quite a mess of black books and scribbled notes holding passwords for my clients.

    On the other hand, it's a LOT of data entry for you 100 * 5-10 When you add all the other fields of data to the mix that' going to be more than 10,000 lines to type in. But the good thing is that once it's in there you can change them in a hurry or look them up in a hurry as well.

    On the other hand, maybe I didn't follow your question. A lot of those passwords are on the client's shoulder to remember, unless you promised to manage passwords for them.

    Anyway I only worked for a small ISP so what do I know? nothing…

  • Anonymous

    I'm sorry you caught me, Your Honor. I'll try to improve.

  • Anonymous

    If you want your servers to be safe you wouldn't want them hosted in the US let alone godaddy, what do you expect?

  • Anonymous

    Passwords are evil, and storing foreign password in clear text (or encrypted with a key the foreigners own) is horrible.

  • Anonymous

    I like to eat passwords for breakfast

  • Anonymous

    This whole thing doesn't surprise me.
    I use their registration services, but host my domains elsewhere.
    I used to route my mail through GoDaddy, but they falsely rejected a lot of legitimate emails from developer mailing lists, and weren't willing to do anything about it, with the same, completely lame excuse: most people like "the way they deal with security issues"…
    Well, guess what, I am not "most people." I actually know what I am doing and I can filter spam myself just fine, spamassassin to the rescue. This "hand-holding" by GoDaddy is really ridiculous. My domains are hosted where I control things and the hoster just provides the hardware.

  • Anonymous

    Maybe check the EULA on your godaddy agreement. I mean atleast in my country I would've fried godaddy instantly in court on breaking in to my server without permission. A welcome addition to my bank account balance.
    But that's just me.

  • Anonymous

    @joel

    It might be good to "own" up to it. However there is a difference between a shill employee doing their best to cover the company tracks, and somebody really "owning" upto it.

    They've got a whole forum.nodaddy.com devoted to GoDaddy problems. Maybe they need a rep in there?

    What do you guys think about the $500.00 early termination fee on their "dream website" package.

    It's only $50.00 a month hosting.

  • Anonymous

    And to further my comments.

    They have to keep the password in a format which they can retrieve later.

    However, if it's dedicated hosting. You have to ask yourself how many other tenants are sharing that space with you.

    There is NO reason why someone should give a damn about your personal space and malware if your paying premium for dedicated hosting.

    What that simply means is that your premium space is subject to intrusions and breaks in or lax security practices on behalf of your neighbors.

    Maybe they should keep those kernels upgraded and provide those premiun customers with some dedicated resources.

    None of this shared masquerading as dedicated bullshit.

    Jmo…

  • Anonymous

    Your passwords are probably just "encrypted" in BASE64 for storage in their database.

  • Anonymous

    "KeepassX works nice for passwords.
    http://www.keepassx.org/
    Cross platform also. It's eliminated quite a mess of black books and scribbled notes holding passwords for my clients.

    On the other hand, it's a LOT of data entry for you 100 * 5-10 When you add all the other fields of data to the mix that' going to be more than 10,000 lines to type in. But the good thing is that once it's in there you can change them in a hurry or look them up in a hurry as well.

    On the other hand, maybe I didn't follow your question. A lot of those passwords are on the client's shoulder to remember, unless you promised to manage passwords for them.

    Anyway I only worked for a small ISP so what do I know? nothing…"

    The point is that the passwords are still decryptable. I am looking for those who say passwords should never be written down to provide a solution.

  • Anonymous

    You could always try the obvious: don't use godaddy.

  • Anonymous

    Most VPS are just virtual machines running on a host (physical server), and the third law of immutable security applies – if someone has physical access to the server, it is not your server. As much as we would like to think they are "our" servers, the fact remains, as some posters have already pointed out, that GoDaddy can (from a technical perspective) access any VPS they need to (or want to) at any time, and there is nothing anyone can do from a technical perspective to stop them. They may even be allowed to do so legally under their TOS.

    I have to agree that if one is concerned about someone else accessing his VPS, then one should either not contract for a VPS, or else not put install or store anything on there that anyone else should not see.

  • http://devcha.com Slavi

    What do you allow root to login from ssh ?
    Disable it and if needed "su root" when needed.

  • Anonymous

    Regarding the KeepassX being decryptable, yes your correct you can click on the eyeball and see the password.

    However the entire database itself is in fact encrypted, and you or your favorite BOFH employee would need a password to get into the data in the first place.

    Maybe KeepassX ain't good enough for you need. I'm sorry, I simply tried to give you a quick answer. I could just as easily left this space blank.

    With only 100 Accounts, I could see the sysadmin or the owner or both controlling the USB stick with the accounts. I've always liked the idea of having data in my HAND. What I wouldn't do however, is roll out a LAMP server for such a job.

    Anyway Good luck on whatever you use!

  • http://daniel-lange.com Daniel Lange

    OVH (French hoster, http://www.ovh.com) ask you to put their ssh key into .ssh/authorized_keys. If it's there, they can try to manage the system, if not and the system is spreading malware/spamming they'll have to take it off the net.
    I think that's the better approach, because you can decide whether you want them to access your system or not. You see their key used in your logs in case and they need not know or store any root passwords.

  • Anonymous

    Run your own server at your location and it won't be an issue. Also, don't use GoDaddy for domain registration. I wouldn't allow them to clean my toilet, let alone handle sensitive information.

  • Anonymous

    OP seems to be an attention seeker

  • Anonymous

    Holy crap. This is the security leadership of GoDaddy? Mommy, I'm afraid.

    http://twitter.com/ngwarner
    http://valleywag.gawker.com/361399/go-daddy-defrauds-customer-google-defrauds-go-daddy

    Oh. My. God.

  • Anonymous

    Wow. Big self-inflicted black-eye for the guy who writes this security blog — I can't imagine any serious employer hiring you after this incident. Hope GoDaddy paid you off darned well.

    Giving GoDaddy you root passwd is like giving your house-key to your cleaning service. This works for some people: you get your house cleaned on a regular basis, and it gets done when your not home, so it doesn't intrude into your lifestyle. If you trust your cleaning service, its great. If you are paranoid, have valuables in your house, or happened to hire a cleaning service that's been infiltrated by the mob, you are SOL. If you live in a bad neighborhood, and your cleaning lady gets mugged, you are SOL. If your cleaning service hires a juvenille delinquent, your SOL. If they're not bonded, you're SOL. There's a whole mountain of failure scenarios that open up when you allow others to have too much access to your house. Same deal with godaddy — myself, I'd be wayyyy too paranoid to put up with that, and I can't even begin to imagine the lifestyle of a self-proclaimed security expert who thinks this is acceptable.

    FWIW, My webservers have been getting SSH passwd-guessing attempts 24×7 for the last decade. Usually from China, often from eastern europe, but also California, you name it — there are script kiddies running ssh-passwd-guessers all over the planet. With that track record, ain't no way I'd let any godaddy employee anywhere near my machines. Your just setting up to get trojaned. Lots of luck w/ life…

  • Anonymous

    The farther back you read on GoDaddy, the funnier this gets. These fools actually advertise the security solutions they put in place!!

    http://www.webhosting.info/news/1/godaddy.com-boosts-web-hosting-security_0531069043.htm

    Ah tippingpoint… now we know what's looking for us.

    Great job there, Mr. SEE-ESS-OOO

  • Anonymous

    How much do you want to bet that they put all those encrypted passwords on a server connected to the Internet? They didn't brag about the server being offline, so it's probably, stupidly, online.

  • http://www.blogger.com/profile/09293084595049139112 IsCyborg

    One concern with the retrievable password storage – if you use the same password for your GoDaddy account as you do for anything else, such as your e-mail account, a snoopy CSR could potentially access it.

  • Anonymous

    Get back to work ppl. Or are all of you out of work these days and troll blogs 24/7?

  • Anonymous

    Since this is a Virtual Private Server giving the administrator who owns the real server root is not outlandish… You've given up your security to those who have physical access to the machine.

  • Pingback: JasonGiedymin.com » Blog Archive » Trust your host?

  • Pingback: Papa est là ! | Linux-backtrack.com

  • Pingback: And how do you store my password? | ashishb