This post is entirely for developers. Feel free to read, but approach it with that in mind.
There is no such thing as bug-free code. We all make mistakes and every piece of software will have issues that we did not anticipate. We ourselves find weaknesses in our code and have to work extra hard to stay ahead of the issues. The same applies to every other company out there.
However, some of these bugs may have security implications that can affect the integrity, availability or confidentiality of the users deploying it. They are called software vulnerabilities.
I think it’s fair to assume that we can all relate with this to some level. What seems to be a problem however is how we, using the collective we, handle the disclosure of these vulnerabilities when brought to our attention.
- What happens when someone identifies a vulnerability in the code we write?
- What if it can or is being misused to hack websites that employ that same code?
- How should we as developers respond and handle these situations?
I want to share some thoughts that I hope will provide some insights on a good disclosure and engagement strategy.
Accounting for Vulnerability Disclosures
If you are a builder and are shipping code the first thing you should plan for is a method to allow security researchers to contact you. This is first and foremost; you want to encourage folks to review your work and contact you if anything is found. You don’t have to set up a bug-bounty program or anything too formal, but at least some method to ensure that you get notified immediately of issues. We should all want to know if there is serious problem with our code.
Here are 6 steps we encourage every builder take into consideration when an if a vulnerability disclosure occurs:
1. Do Not Panic
Take a deep breath. Everyone makes mistakes and this is likely not the end of the world. It may seem like a big deal, and you may get worried that you will lose business and the trust from your users. However, if you deal with the disclosure properly, it may actually have the reverse effect and you may gain their trust from this experience.
I think as humans we have a predisposition for jumping straight to the negative. I’m going to lose business!! I’m going to go out of business!! We should flip this mindset to one that is more approachable, What can I learn from this? You’re in essence getting a free security review; this is a positive thing and you want to encourage an open dialog with the researchers.
As soon as you get the notification, let the reporter know that you have received the notification and are looking into it. We sometimes fail to realize how important this small step can be.
In the business of disclosures, it’s not uncommon for organizations to shun researchers away, and over time it has built a culture of distrust; forcing them to believe the organization doesn’t care. This reduces the tolerance for lack of acknowledgement and can often lead to bad outcomes for everyone involved.
A simple, Thank you so much, we’re looking into this and will get back to you within X hours will pay you huge dividends.
3. Do Your Homework.
Take all the information provided by the researcher and try to confirm the issue. If you can’t reproduce the issue, don’t jump the gun and tell them they are wrong, instead ask for something known as Proof of Concept (PoC). Remember, this researcher is taking time out of their day to report something to you to help you, attacking them or blowing them off will do little in your favour.
Once the PoC is provided, you should work to provide the researcher a timeline. In the world we live in today, with web applications, timelines that extend beyond 2 weeks really aren’t acceptable. If you must extend beyond that, be sure to communicate with the researcher on the reasons behind the delay.
The pressure to release sooner will honestly be determined by its severity, ease of exploitation and impact to the larger audience.
4. Patch It!
Please patch the problem effectively. One common mistake is to patch it where it was identified, but leave it susceptible in other parts of your code. Don’t be lazy.
Take the time to review the rest of your code for similar issues. There is nothing that will hurt you more than to have a similar vulnerability found a week later, but on another part of your code.
5. Coordinated Disclosure
Before you release the patch, involve the researcher. Send the fixed version to the researcher, ask them to review, engage them in the process and watch it pay you dividends. They will often go above and beyond to help you think through the problem, and will help build a rapport. Once this is done, you have to plan a release – there really isn’t a way around this, especially for severe vulnerabilities.
Let the researcher know when you plan to release. Work a coordinated disclosure release schedule, most researchers love to identify problems but also love to write and share about their findings.
A good email to the researcher would be:
I will be publishing a patch on DAY X at Y time (GTM). I will follow the release with a blog post of my own shortly and will email all my clients. Can you wait 24 hours for your own disclosure? I want my users to hear from me first.
This is a very acceptable email and most reasonable researchers will understand and respect your request. Please do not do a silent release – a silent release is a release where you avoid any mention of the vulnerability in the hopes that no one will ever find out. The researcher is likely monitoring all patches, and if they see you do this without any proper disclosure they are bound to call you on this act, publicly.
When writing a blog post (and in the release notes), you should explain what happened, the severity, who reported the issue to you and what you did to fix it. Provide links for the patch and a timeline if possible. Your users will appreciate and trust you more when you do this. Again, do not try to hide if it is a security issue. The idea that by not sharing you don’t have to worry about it being exploited is an antiquated way of thinking. Any attacker worth their salt can run a diff and see the issue right away; not saying anything does not solve the problem.
6. The Backlash.
This seems to be the biggest fear most builders have. What will my community think? What will clients think? Will my company survive?
These are normal feelings, and we can assure you this will pass. In many instances, buyers will gain renewed faith in your product once they see how well you responded.
The core of your message should be: Yes, an issue was found, but more importantly this is what we learned and what we’re doing moving forward to make sure it never happens again.
Example of Good Disclosure
Remember that no one is perfect and with time, any product may have a security vulnerability discovered. I want to leave you with a good example from the JetPack team. They recently handled a vulnerability disclosure and did so brilliantly in our eyes: http://jetpack.me/2014/04/10/jetpack-security-update/.
They patched the issue right away and notified everyone, including hosts and partners about the problem. Even provided detection rules that security operators could use to detect and block a possible exploit.
I welcome thought and opinions in the comments, and if something makes sense we’ll address it accordingly. As mentioned before, this was drafted to provide you some thoughts and guidance, not dictate. Yet, we hope you find it helpful.