Responsible Disclosure – Sucuri Open Letter to MailPoet and Future Disclosures

Many don’t know who I am. My name is Tony Perez, I’m the CEO of Sucuri. I have the pleasure of calling this company my family and everyday I work for every person at this company. My partner is Daniel Cid. He is one of the foremost thought leaders in the website security domain, his influence extending far beyond the communities that make up some of the most popular CMS applications today.

Together we are building one of the fastest growing Website Security companies in the domain, we have one simple mission, to create a safer web. We are a technology company built by technologists with a special, quirky, idea that we can make a difference.

Many don’t realize that the bedrock of our business is Research, all facets of research. It’s how we stay ahead of the bad guys, or attackers. It’s a responsibility we have, not just to the general public, but one that we owe to our clients – in basic terms, it’s what they pay us for. It’s how we ensure our tools and technologies stay ahead of the rest and what makes us the ideal solution for every website owner, our commitment to the Website Security domain.

This has come to head recently from the huge debacle over the past few weeks in which we reported a very serious vulnerability in the WordPress MailPoet Plugin (WHYSIJA-NEWSLETTERS). In the coming days the attackers proceeded to identify, then begin to exploit the disclosed vulnerability.

Frankly put, the entire situation was very unfortunate.

Some Background on the Recent MailPoet Issue

Here is a more accurate timeline on the order of events:

  1. 2014, Jun 16: Notified MailPoet of the vulnerability, provided patching recommendations.
  2. 2014, Jun 16: MailPoet team replied and said they were working on a fix.
  3. 2014, Jun 18: Notified Sucuri that they had fixed the bug and would released a patch soon.
  4. 2014, Jul 01: The MailPoet team updated WP.org with the new release.
  5. 2014, Jul 07: MetaSploit Module released for the Vulnerability

The total order of events from took 15 days.

Upon release of the blog post the MailPoet team did contact us to express their discontent with our actions, and this was our response in the interest of transparency:

As far as disclosing the vulnerability, this is quite a common practice and the correct way to bring awareness to a security issue. A good example of a perfect security disclosure was done by the Automattic team with JetPack:

http://jetpack.me/2014/04/10/jetpack-security-update/

As soon as they released a patch, they notified all users and contacted multiple blogs to ask them to urge everyone to upgrade.

I imagine you are worried about brand impact, but every piece of software will have bugs and security issues at some point. It rarely has any brand impact and if you respond properly, it can have the opposite effect and be very good for you plugin and team reputation. The “We had an issue, we fixed and it won’t happen again” type of message that your users would prefer to hear from you than from some external blog.

As for us, we don’t do that for publicity. It is just part of our research and work that we do every day. Even before Sucuri started, we were auditing code and disclosing security issues. Our goal is to be ahead of the bad guys to protect our clients and help the web at a whole.

I leave it for you all, unedited.

Open Letter to MailPoet

As to be expected, the MailPoet team is pretty pissed off as it would be expected. So pissed in fact that they felt compelled to question our intent and whether we shared the same goals, so let’s talk about that for a minute.

Are we sure we are all aiming for an open, safe web in the WordPress community?

In an effort to provide some peace of mind and transparency in our thought process, please read this open letter to MailPoet:

Hi Mailpoet Team

First and foremost, I am sorry for the troubles you have been experiencing as of late.

Second, I did want to take a minute to clarify a few points to avoid speculation:

1 – Let’s start with reasonable time:

MailPoet Post: It’s common practice among software security circles to disclose bugs privately with software companies, then get a reward, credit and the possibility to write about it, given a reasonable amount of time to fix it.

You see, it’s all about a reasonable amount of time.

Responsible disclosure is about time to patch. That is what we provided. We disclosed only after your organization patched and made it publicly available.

Responsible disclosure has nothing to do with providing reasonable time after the patch to wait before disclosing publicly. Especially when you look at how the issue was highlighted, or lack there of in the change log.

Sucuri - MailPoet Security Disclosure

Nothing highlighted the seriousness of the issue, so we did. That’s what we feel is our responsibility. It’s buried and lacks any emphasis, it’s why so many in the security business subscribed to Full Disclosure (i.e., https://www.schneier.com/blog/archives/2007/01/debating_full_d.html)

This was a very serious vulnerability, one that deserved attention and we did so after it was patched, as is expected and is the norm.

2 – In regards to this:

MailPoet Post: effectively giving no time to users to upgrade their MailPoet version

It’s arguable that the only reason many updated their plugins when the patch was released was because of our public release and our ability to reach 100’s of thousands of WordPress Website owners. We were also able to make contact with hosts, managed host, and development shops.

3 – In regards to this:

MailPoet Post: before posting a detailed technical disclosure

We did not post a detailed technical post. We did not share a Proof of Concept which is actually very standard, we did reference elements that we felt had a greater impact than the ecosystem in which your plugin currently operates. Here is a snippet of the technical description you are alluding too:

Sucuri Post: Because of the nature of the vulnerability, specifically it’s severity, we will not be disclosing additional technical details. The basics of the vulnerability however is something all plugin developers should be mindful of: the vulnerability resides in the fact that the developers assumed that WordPress’s “admin_init” hooks were only called when an administrator user visited a page inside /wp-admin/.

As you know, we also did not disclose any Proof of Concepts. We directed all those requests to your team to handle at your discretion.

4 – I presume this is meant to be a direct attack at us:

MailPoet Post: Are we sure we are all aiming for an open, safe web in the WordPress community?

If I misinterpreted the intent here, please do let me know. You are right though, our ambitions are much larger than the WordPress community, we’re pursuing a safer web we as a whole regardless of platform.

Again, I personally apologize and empathize with the struggles you have endured over the past week or so. Your struggles were not our intent, and not our driving force. Before this incident we had no relationship and had no interest in the space you are in.

That being said, if it ever becomes an issue in the future, for you or any other developer, we will follow the same protocol that we used with MailPoet.

All the Best,

Tony and Daniel

One small note, you mentioned:


There’s a difference between warning users and disclosing a 0 day vulnerability to the entire world on the same day of the bugfix release.

Small point of clarification, Zero Day vulnerabilities are those that are released and have no patch. Your vulnerability was patched, hence not being a Zero Day anymore.

Creating a Safer Web

Yes, in case you’re wondering, this is but the tip of the iceberg for us.

We will be proactively researching security issues across the wide spectrum that is Website Security. From CMS applications like WordPress, Joomla, osCommerce, vBulletin, etc… to web servers like Apache, NGINX, Windows IIS, and more. As stated before, it’s what makes us who we are and the responsibility we have to our clients as well as the wider audience of the web as a whole.

The time to be more proactive in our research and overall contribution to the web is now, not tomorrow or the day after. We stand fast in our convictions and will continue to push forward. Remember our responsibility is not the developers and designers, but the millions of website owners, their websites and their businesses.

All the best,

Tony / Daniel

Scan your website for free:
About Tony Perez

Tony is the Co-Founder / CEO at Sucuri. He shares a deep passion for Information Security, Business and Brazilian JiuJitsu. He approaches the business the same as he trains BJJ, one move at a time and gently. You can follow him on twitter: @perezbox.

  • willc

    Keep up the great work, Sucuri. Mail Poet needs to understand that handling this sort of situation with grace and professionalism goes a long way.

    • perezbox

      Thanks for stopping by and the kind note.

      Tony

  • http://redfishbluefishmedia.com/ Jon Schroeder

    I’m glad you were involved in this situation, and thankful for the work that you do (I recommend you to all of my clients) but I know that I got the email MailPoet sent out announcing the vulnerability (I’d updated already, actually). I can understand why they’d be upset – as they mentioned, they can’t force users to update.

    Making this issue much, much more public before most users had updated resulted, almost certainly, in more hacked sites, not less.

    If you’d simply waited a few weeks after the update was pushed before drawing so much attention to it, the bulk of users (who update regularly, not because they check your site), then I think it would have resulted in better outcomes, and everyone would be happier.

    MailPoet devs: this should have been massively highlighted in your changelog. But the email you’d sent out was perfect, and exactly what users needed (and it did highlight the severity of the issue)

    Sucuri: responsible disclosure means something specific, and you held to that. Fine. But was there a compelling reason for your timeline beyond “well, the update’s out”? Why rush to update your blog “only a few hours” after they’d released the fix? Your blog isn’t just a call to users to update. It’s a call to hackers to exploit. It looks like your scanner didn’t start seeing many blogs hacked until later in July, and that’s on users – it’s important to update. But your highlighting the issue in such a public forum so soon after the fix’s release is what they’re mad about – and I just have to wonder why it couldn’t be done a week later with the same impact? The vulnerability had already been out there for several weeks. Why the rush?

    • perezbox

      Hi Jon

      Thank you for your note, and thank you for your recommendations and support.

      In regards to this:

      But was there a compelling reason for your timeline beyond “well, the update’s out”? Why rush to update your blog “only a few hours” after they’d released the fix? Your blog isn’t just a call to users to update.

      We have a responsibility to disclose information as it becomes available to us. It’s part of our protocol. We’ve done it several times before, and will continue to do so.

      There is no answer that will satisfy an open ended question like the one you asked, so I won’t attempt at one. Just know that as security researchers we do what we feel is in accordance with our principles and best practice. We did not disclose proof of concepts, we appropriately disclosed and followed all the appropriate steps that are demanded of us as responsible stewards in the security space.

      We’ve been at this a long time. I can assure you, there is not amount wait time that would have been adequate. Just look at Rafael post:

      According to WP.org stats, MailPoet still have 75% of their users on versions older than 2.6.7, it’s a whole lot of users that haven’t upgraded yet.

      Even with the issues at hand, close to four weeks after the disclosure and people have no updated. It’s always easy from the inside looking in to see the “clear path” but I assure you there is no such thing. As a business you define your path and you execute against it. That is what we have done.

      I hope this provides you better context.

      Thanks

      • http://redfishbluefishmedia.com/ Jon Schroeder

        These are fair points, and that’s really too bad that so many users are not patched yet. In light of that – the fact that at this point it’s been a month or so, and only 25% have patched, there’s probably not a compelling reason to wait on posting your information, either, so I can get behind what you’re saying. Context provided and understood.

        There’s no way I would have predicted this stat is what it is. Then again, I don’t trust clients on my server to update their own sites – I handle it for them, and across all of my sites as of right now (about 100 or so), only a couple have even one plugin out of date.

        Thanks for all you do. If you have a recommended fix to the other major WP hack happening right now (the XMLRPC thing), I’d love to read about that as well. Or even a stopgap plugin.

    • Andrew

      Not everyone gets plugins directly from the author, so they wouldn’t get emailed by the author. The plugin may have been bundled with a theme. So it is absolutely the responsible thing to do for Sucuri to disclose this to the public as soon as a patch is released so everyone NOT on the author’s mailing list (or blog) knows about it as soon as possible. Get the word out as quickly as possible to as many places as possible. An exploit as serious as this one needs to be reported as quickly, and to as many places, as possible.

      I just don’t understand why something as serious as this is getting nitpicked in the way it was disclosed. Mail Poet is being ridiculous for being upset about this, when they should be more upset at themselves for dropping the ball so severely on the security of their plugin and should instead be thanking Sucuri for the responsible reporting. (I know I am.)

      • http://redfishbluefishmedia.com/ Jon Schroeder

        Speaking of picking nits, you replied to me to disagree – but if you read what I wrote, I agree with you. I said I thought the disclosure was too soon, read a sensible reply from Sucuri, then changed my mind based on the new information there.

        MailPoet is not being ridiculous, though; they’re in a difficult situation and (I think) making the best of it. I read their original post that started this discussion, and it’s not exactly full of fighting words. Either way, both MailPoet and Sucuri make a good product. Neither is perfect, and none of the people involved have reacted perfectly to this. But thanks to them both for playing their respective roles; the problem is solved for anyone paying attention.

  • perezbox

    Hi Rafael Ehlers

    In regards to:

    With all being said, do you guys at Sucuri really think that you have chosen the best option for the users by disclosing to the public this security flaw on the same day MailPoet version with the patch reached theWP.org repo?!

    Yes, I believe that is what I was implying in the post. Apologies that I was not clearer.

    Based on this logic:

    As you know, hackers are faster than other kind of users. You really think that a few hours is time enough for everybody to upgrade their website?!

    Followed by this response:

    According to WP.org stats, MailPoet still have 75% of their users on versions older than 2.6.7, it’s a whole lot of users that haven’t upgraded yet.

    Should I assume that your recommendation is we never say anything? Because what that tells me is that even after the recent debacle, four weeks after disclosure, no one seems to want to update. Isn’t his contradictory to the reset of the paragraph:

    If you have given us at least 1 month before disclosing it, the vast majority of our users will have had the time to do the upgrade.

    What am I missing?

    Thank you for your clear cut position:

    Security theories and terms aside, for me this whole situation is very very clear: doing a disclosure so soon like this (same day as the fix lands on the repo) will increase the chances of users getting their website hacked by a 1000%.

    We shared ours as well above.

    I presume this was your attempt at giving us a jab:

    I’m definitely sure that we are not the ones profiting with this entire situation (neither our users).

    Yes, in case it was unclear. Sucuri is a business, in other words we sell products and we are for-profit. We offer a Website Antivirus product designed to provide malware detection and remediation and a Website Firewall product, designed to keep hackers out of your website. If somewhere in the process we were being sly about this or didn’t adequately disclose that, I apologize, and I hope that clears that issue up.

    All the best

    Tony

    • http://www.marcelopedra.com.ar/blog/ Marcelo Pedra

      At the end of the day, it’s all business and marketing :)

  • Adeel Sami

    There’s no denial of how well-informed you keep your clients and the general people. I always get to know of latest vulnerabilities from your blog.

    I won’t tell you to stop but you keep growing to keep the Internet a safer place!

  • Volker

    I my users don’t upgrade regularly, it’s not a problem of disclosure.

    Any hacker can scan the release notes of all plugins for “security” fix and then bet on the fact, that there are many users who delay their update.

    Honestly, wordpress is not the best choice for non-business or hobby sites. Everyone running wordpress should be willing to either maintain the updates or to pay services to maintain it for them.

    Running several worpdress instances by ourself, we spend about 1-2 days per month to keep five multi-site instances with a total of about 50 plugins up-to date.

    I feel with the mailpoet mates, especiall the supporters. And I think the only difference the ‘early’ disclosure made is that more people upgraded. The disclosure is a plea for update. The risk of a hacker reading it shall never be a reason to delay or non-disclose it. That is when we’ll start acting like the NSA.

    That is not my way and I assume yours neither.

    • Volker

      One idea comes to mind. How about a special ‘vendor’ package by sucuri that comes along with a disclosure. It not only provides suggestions for the patch but also for the removal of the infection. This package could at least help the supporters….

  • dbarnhartAZ

    The actions you took were appropriate and I thank you.

  • Havenswift Hosting

    We would also fully support your position and the way you highlighted this issue – the only proof that is needed is the huge number of installations are still out there even so long after the disclosure and all the publicity that went with it. We host a large number of WordPress users, some we manage and update immediately, some good users who regularly update themselves and also a number that never update their plugins. Your posting with an explanation of the seriousness of the issue was plenty reason enough for us to check across all our servers to see if any user was running this plugin – luckily none were but if there had been, then we would have been able to notify / take action immediately to not only protect them but all other users on the server. keep up the good work !

  • Dave Walsh

    This is a non arguement, Sucuris methodology was 100% correct. Not to deminish Sucuris talents, but if they found this, then you can be pretty sure someone else has and is exploiting it. Not telling the public about this would be a far bigger error.

    IMHO its only a matter of courtesy to tell the developers of a vulnerability in their code, as its probably already being exploited and users should be allowed to make the decision to remove the plugin in question or not.

    Giving them time to release a fix is being generous.

    MailPoet should accept that if this vulnerability wasnt flagged by Sucuri then it would have gone on unpatched for longer allowing misuse of their code. Again, they shouldnt take it for granted that Sucuri were the only people to see this vulnerability.

    MailPeot should learn from this and move on, work with people like Tony, Daniel and their crew, we’re all on the same side afterall.

    DaveW

  • Zach

    I support Sucuri disclosing the bug and fix for Mailpoet. I think, that the more people you can let know of the vulnerability and have them fix it the better as it leads to a safer internet. I am pretty sure that hackers wait around on the Sucuri blog to find vulnerabilities, i’m pretty sure that there are other channels that they use. If I developed a plugin, you discovered a bug, and we fixed it and patched it – I would want as many people to know that it was fixed.

  • csfalcao

    As I said earlier, I love Sucuri and MailPoet, both business will stand well after this.
    I worked in security (former Windows Server Sec Expert) and it was a really daily battle, and as soon as you find something you have to act, maybe it’s a little difficult to understand from outside.
    I think the only thing to correct was that mailPoet should not take 15 days to make a update with a list of 10 features, but a security update ASAP.

  • https://www.mensmaximus.de/ Michael Weichselgartner

    I am working in the ISP business since 1997 and have seen a lot of exploits and vulnerabilities in applications, operating systems and software in general. In my current position I am dealing a lot with mediation. I try to get consent between two conflicting parties. E.g. ‘the techies’ and ‘the bosses’. What I see here is a classical clash of interests.

    Sucuri, highly respected for its research, has found a severe vulnerability and is interested in making it public. This is their business. This is how they make money. Reputation is everything. However as Mr. Perez said, they also have an obligation. The intention is not to harm the other party.

    Mailpoet, well known and highly recommended for its WordPress plugin, is interested to protect their customers and of course their market share. This is their business. They make money selling the plugin. Competitors are fast if it comes to negative press. And of course they are thankful to Sucuri for telling them about the exploit.

    Both have some things in common: they earn money from what they do, they belong to the top companies in their business and they crossed each other’s way by accident. The whole discussion is about time. Sucuri could have waited a bit longer but Mailpoet could have moved much faster.

    How long should Sucuri have wait? What if the other party would never respond? Does it mean Sucuri should not have disclosed the exploit at all? On the other hand what would be an appropriate time frame for Mailpoet to fix the bug? 14 days from getting noticed by Sucuri until releasing the fix seems long. Especially if you assume Sucuri told Mailpoet from which function the vulnerability has been born. This is kind of a chicken and egg situation. However this is something that happened. All one can do is to learn for the future. In such a case and if I would have to deal with it I would recommend the following:

    Sucuri should review there procedures and implement mile stones. Introduce a seven day sunrise period (with NDA of course) after informing the other party so they can fix the bug. Then add a 3 days grace period to get the new version released. And from the release date settle a cease period lasting 48 hours. If you discover an
    exploit tell the other party about the time frames and keep room for an extension of the sunrise period up to ten days if needed. Now the developer knows what will happen if he fails to act in time.

    Mailpoet (and any other company facing a severe issue with their product) should have a disaster plan. Team up with one or two acknowledged WordPress developers who can help in case of unforeseen problems. Introduce a 3-layer working plan making sure you invest 24 hours per day into the issue. Hire an external supervisor who is deep into the WordPress codex to review your source code to avoid problems like this.
    Or negotiate a contract with Automatic (not sure whether they offer code reviews – if not I would be happy to build such a desperately needed business unit). Talk to the people who discovered the bug. Keep them updated – daily! Don’t waste time and prepare a press release together with the researchers. Be thankful, not peeved. And always remember it is your code. If it is not a core issue than you are fully responsible for what happened and what will happen.

    And one last thing to think about. If Sucuri finds a bug you can bet others do too. And if those are from the dark side time is your biggest foe. In my humble opinion Sucuri did the correct thing. Timing might have been better communicated to Mailpoet. The biggest mistake on both sides however is the public discussion. Those things are dealt with during a telephone conference between the CEO’s or better at a dinner or a golf course.

    Just my 2 cents

  • badshark

    Hi Tony,

    I know about Daniel, I know about OSSEC and I personally think he’s a super talented security researcher we should thank for OSSEC.

    I’m at MailPoet as a software engineer and system administrator on the SaaS projects we run and I’m involved in security researching, enjoying it as you do.

    I also agree with you that this discussion can’t be completely understood if we don’t know about the history of bug disclosures and how the community arrived to the “Responsible Disclosure” concept.

    What we were really trying to say is: Are a few days of “upgrade window” really that bad?

    Let me quote the Mozilla security policy here:
    https://www.mozilla.org/en-US/about/governance/policies/security-group/bugs/

    “Before making a security bug world-readable, please provide a few days notice to the Mozilla security bug group by sending email to the private security bug group mailing list.”

    “Please try to be understanding and accommodating if a Mozilla distributor has a legitimate need to keep a bug in the security-sensitive category for some reasonable additional time period, e.g., to get a new release distributed to users.”

    The process here is that Mozilla, for example, get notified about a vulnerability, fixes a bug, sends out a new version and after a few days, when they see a good amount of users upgraded, makes the vulnerability public.

    Users are immediately notified about the upgrade, this is absolutely not about keeping vulnerabilities hidden, and it has nothing to do with brand.

    I’d also like to point out that the percentage of our users on old versions discussed in the comments (75%) is wrong. I’ve been keeping an eye on these statistics and we managed to get around 60% of our users on the fixed version in around a 10 days. This means that if you exclude abandoned installations (around 20%) there are 20% of active users still vulnerable. It looks like a reasonable number to me.

    Wouldn’t it be good if we could work together, patch immediately, warn users with security messages, and a few days after release all the proper blog posts?

    Marco

  • http://www.marcelopedra.com.ar/blog/ Marcelo Pedra

    Hello Rafael. I could probably agree with you, and not because I’m a MailPoet user. The problem with plugins are the end users: I won’t be surprised if there are lots of your premium users who still use old versions of the plugin. There are a vast majority of WordPress admins (with username “admin”!), who aren’t non-techie people, nor have the expertise, know-how or even care to keep their websites software up to date.

    In the other hand, WP cannot be automatically upgraded, neither its plugins. But it’s a new practice I’m seeing in several products (like WordFence) to add an automatic update feature. That way you are secured all the time, no matter you care or not. Maybe MailPoet could add this feature in coming versions.

    MailPoet must strenghten its security audits. It’s a fact. Anyway, you guys did a good job advising to update immediately the plugin, from the newsletter and from the plugin news itself.

    Sucuri, in the other hand, it’s commited to inform the news and vulnerabilities to those of us who maintain hundreds of sites and really care about it, the servers, the users and the security as a whole, even if that target is the 10% of sysadmins in the market. I must agree with Perez and Cid in which this wasn’t a zero day in its strict definition, but I can bet Sucuri probably didn’t waited a couple days more before publishing the vulnerability because, as Michael Weichselgartner stated, this is how they make money, and their Reputation could be at risk if the bad guys or worst, the competition, issued a review before them :)

    At the end of the day, it’s all business. Even for the bad guys. That’s why EVERYBODY must react asap. Thank you.

  • http://jb510.com Jon

    Tony & Daniel – I think you handled it perfectly and just want to thank you for the continued good work.

    Someday perhaps WP will have a mechanism where plugins can be auto-updated, maybe even selectively when a particular plugin and version has a security vulnerability. Regardless though I still expect security vulnerabilities to be announce the same day the patch is.

  • Kathy

    Anyone who has a basic understanding of the WordPress environment knows that security is an ongoing battle. I was introduced to the MailPoet plugin as a result of the post and have plans to purchase the plugin once the dust settles.

    By acting promptly to fix the problem, the MailPoet team earned my respect and admiration. Unfortunately, today’s post has knocked that down a notch.

    Tony, you’ve done a wonderful job of handling this situation. Your original post did a wonderful job of highlighting the issue in a way so other developers could check their plugins for a similar issue.

    MailPoet people – settle down and thank Tony and his people for bringing this to the public’s attention. You should be joining them in their efforts to alert the public. This is your chance to shine – don’t blow it by whining.

    Remember,you didn’t perform a coding felony – only a misdemeanor.

    You could have chosen to abandon your plugin instead of fixing it, which would have meant almost 2 million users would have a timebomb installed their site. (I’ve seen this happen before – and had to pick up the pieces for my clients when it did.)

    You fixed the problem promptly and that’s all anyone who’s reasonable would ask of your team.

    Instead of trying to protect your reputation, focus on protecting and alerting your customers. Trust me, in the long run, they’re the ones who will MAKE or BREAK your reputation.

  • Dan

    When it comes to critical security vulnerabilities there’s no such think like “reasonable amount of time”… action should take fast, not tens of days… remember Heartbleed?