Many don’t know who I am. My name is Tony Perez, I’m the CEO of Sucuri. I have the pleasure of calling this company my family and everyday I work for every person at this company. My partner is Daniel Cid. He is one of the foremost thought leaders in the website security domain, his influence extending far beyond the communities that make up some of the most popular CMS applications today.
Together we are building one of the fastest growing website security companies in the domain, we have one simple mission, to create a safer web. We are a technology company built by technologists with a special, quirky, idea that we can make a difference.
Many don’t realize that the bedrock of our business is Research, all facets of research. It’s how we stay ahead of the bad guys, or attackers. It’s a responsibility we have, not just to the general public, but one that we owe to our clients – in basic terms, it’s what they pay us for. It’s how we ensure our tools and technologies stay ahead of the rest and what makes us the ideal solution for every website owner, our commitment to the website security domain.
This has come to a head recently from the huge debacle over the past few weeks in which we reported a very serious vulnerability in the WordPress MailPoet Plugin (WHYSIJA-NEWSLETTERS). In the coming days the attackers proceeded to identify, then begin to exploit the disclosed vulnerability.
Frankly put, the entire situation was very unfortunate.
Some Background On the Recent MailPoet Issue
Here is a more accurate timeline on the order of events:
- 2014, Jun 16: Notified MailPoet of the vulnerability, provided patching recommendations.
- 2014, Jun 16: MailPoet team replied and said they were working on a fix.
- 2014, Jun 18: Notified Sucuri that they had fixed the bug and would released a patch soon.
- 2014, Jul 01: The MailPoet team updated WP.org with the new release.
- 2014, Jul 07: MetaSploit Module released for the Vulnerability
The total order of events from took 15 days.
Upon release of the blog post the MailPoet team did contact us to express their discontent with our actions, and this was our response in the interest of transparency:
As far as disclosing the vulnerability, this is quite a common practice and the correct way to bring awareness to a security issue. A good example of a perfect security disclosure was done by the Automattic team with JetPack:
http://jetpack.me/2014/04/10/jetpack-security-update/
As soon as they released a patch, they notified all users and contacted multiple blogs to ask them to urge everyone to upgrade.
I imagine you are worried about brand impact, but every piece of software will have bugs and security issues at some point. It rarely has any brand impact and if you respond properly, it can have the opposite effect and be very good for you plugin and team reputation. The “We had an issue, we fixed and it won’t happen again” type of message that your users would prefer to hear from you than from some external blog.
As for us, we don’t do that for publicity. It is just part of our research and work that we do every day. Even before Sucuri started, we were auditing code and disclosing security issues. Our goal is to be ahead of the bad guys to protect our clients and help the web at a whole.
I leave it for you all, unedited.
Open Letter to MailPoet
As to be expected, the MailPoet team is pretty pissed off. So pissed in fact that they felt compelled to question our intent and whether we shared the same goals, so let’s talk about that for a minute.
Are we sure we are all aiming for an open, safe web in the WordPress community?
In an effort to provide some peace of mind and transparency in our thought process, please read this open letter to MailPoet:
Hi Mailpoet Team
First and foremost, I am sorry for the troubles you have been experiencing as of late.
Second, I did want to take a minute to clarify a few points to avoid speculation:
1 – Let’s start with reasonable time:
MailPoet Post: It’s common practice among software security circles to disclose bugs privately with software companies, then get a reward, credit and the possibility to write about it, given a reasonable amount of time to fix it.
You see, it’s all about a reasonable amount of time.
Responsible disclosure is about time to patch. That is what we provided. We disclosed only after your organization patched and made it publicly available.
Responsible disclosure has nothing to do with providing reasonable time after the patch to wait before disclosing publicly. Especially when you look at how the issue was highlighted, or lack there of in the change log.
Nothing highlighted the seriousness of the issue, so we did. That’s what we feel is our responsibility. It’s buried and lacks any emphasis, it’s why so many in the security business subscribed to Full Disclosure (i.e., https://www.schneier.com/blog/archives/2007/01/debating_full_d.html)
This was a very serious vulnerability, one that deserved attention and we did so after it was patched, as is expected and is the norm.
2 – In regards to this:
MailPoet Post: effectively giving no time to users to upgrade their MailPoet version
It’s arguable that the only reason many updated their plugins when the patch was released was because of our public release and our ability to reach 100’s of thousands of WordPress Website owners. We were also able to make contact with hosts, managed host, and development shops.
3 – In regards to this:
MailPoet Post: before posting a detailed technical disclosure
We did not post a detailed technical post. We did not share a Proof of Concept which is actually very standard, we did reference elements that we felt had a greater impact than the ecosystem in which your plugin currently operates. Here is a snippet of the technical description you are alluding too:
Sucuri Post: Because of the nature of the vulnerability, specifically it’s severity, we will not be disclosing additional technical details. The basics of the vulnerability however is something all plugin developers should be mindful of: the vulnerability resides in the fact that the developers assumed that WordPress’s “admin_init” hooks were only called when an administrator user visited a page inside /wp-admin/.
As you know, we also did not disclose any Proof of Concepts. We directed all those requests to your team to handle at your discretion.
4 – I presume this is meant to be a direct attack at us:
MailPoet Post: Are we sure we are all aiming for an open, safe web in the WordPress community?
If I misinterpreted the intent here, please do let me know. You are right though, our ambitions are much larger than the WordPress community, we’re pursuing a safer web we as a whole regardless of platform.
Again, I personally apologize and empathize with the struggles you have endured over the past week or so. Your struggles were not our intent, and not our driving force. Before this incident we had no relationship and had no interest in the space you are in.
That being said, if it ever becomes an issue in the future, for you or any other developer, we will follow the same protocol that we used with MailPoet.
All the Best,
Tony and Daniel
One small note, you mentioned:
There’s a difference between warning users and disclosing a 0 day vulnerability to the entire world on the same day of the bugfix release.
Small point of clarification, Zero Day vulnerabilities are those that are released and have no patch. Your vulnerability was patched, hence not being a Zero Day anymore.
Creating a Safer Web
Yes, in case you’re wondering, this is but the tip of the iceberg for us.
We will be proactively researching security issues across the wide spectrum that is website security. From CMS applications like WordPress, Joomla, osCommerce, vBulletin, etc… to web servers like Apache, NGINX, Windows IIS, and more. As stated before, it’s what makes us who we are and supports the responsibility we have to our clients as well as the wider audience of the web as a whole.
The time to be more proactive, in our research and overall contribution to the web, is now – not tomorrow or the day after. We stand fast in our convictions and will continue to push forward. Remember, our responsibility is not to the developers and designers, but to the millions of website owners, their websites and their businesses.
All the best,
Tony & Daniel
Rianna MacLeod
Pilar Garcia
Ben Martin
Denis Sinegubko
Luke Leal![Malware comes in many different varieties. Analyst Krasimir Konov is on this month’s Sucuri Sit-Down to help keep them all straight. From malicious iframes to SEO spam, join host Justin Channell as he racks Krasimir’s brain on all the different types of malware. Also, Krasimir discusses his recent blog post about a malicious cURL downloader, and Justin breaks down the latest website security news, including patched plugins you should update. Podcast Transcript Justin Channell: Hello, and welcome to the Sucuri Sit Down. I'm your host, Justin Channell, and this is a monthly podcast about website security, where we get in-depth with the malware removal experts here at Sucuri. Later in the show, I'll have our analyst Krasimir Konov to chat about some different types of malware, but first, let's take a look at other topics we've published on our blog and Sucuri labs notes this month. First up, we have some new information about credit card skimming with hackers using a hybrid method to steal payment information from eCommerce websites. Our analyst Dennis Sinegubko wrote about this for the Sucuri blog back at the beginning of June. Now, most credit card stealing malware is a client side JavaScript that grabs data and sends it to a third party server. But, that approach has a drawback for bad actors because it's still possible to track the requests and catch them as being suspicious. Now, to get around that, bad actors have started harvesting information server side by modifying core PHP files. In this case, the infection would be undetectable from the outside, but it's still going to be pretty easy to find because you're rarely modifying any of those core files, so any of those changes that are going to come up are going to be suspicious. To get around both of these drawbacks, we're seeing bad actors combine the two. So client side snippets of JavaScript are sending stolen credit card data to server side scripts that they've installed on the same server as the site. Now, this allows bad actors to cover their tracks a little bit because the traffic that's being redirected is going to the same server, and that's less likely to be flagged as suspicious. It's a bit more complicated to pull this off, but our team has been seeing this hybrid approach in the wild, so it's something to be on the lookout for. Now, another month has passed, and we found more cross site scripting attacks targeting WordPress plugins. Most notably, we discovered one that affects users of the YITH WooCommerce Ajax Product Filter plugin. Now, this is a plugin that allows WooCommerce stores to be filtered by product type, and it's pretty popular. It's got about 100,000 users right now, so with it being vulnerable, it's very important that all of them update to the latest version, which is 3.11.1. Some of the other plugins we found cross site scripting vulnerabilities with included Elementor Page Builder, Careerfy, JobSearch, and Newspaper. If you're looking for a full list of vulnerabilities that have been patched this month, John Castro at the Sucuri Labs blog has you covered. Check out our show notes for the link. Also, this month I had a blog go up detailing what's called a jibberish hack. It's basically the same motivation as an SEO spam attack where bad actors use your site's good standing to redirect visitors to their own sites. But in this attack, you'll find a bunch of randomly named folders filled with a ton of HTML files with really nonsensical file names like cheap-cool-hairstyles-photos.html. It's just going to be a mishmash of keywords that clearly you didn't put there. Unfortunately, just deleting all those HTML files and folders is not going to be enough to get rid of that jibberish hack though. You're going to need to fully clean any hacked files and database tables, and then you're going to have to deal with all the damage caused to your site's standing. And just keep in mind, if you find anything about that process too daunting, we're always here to help. Now, for this month's Sit Down, we have Sucuri analyst Krasimir Konov. Earlier in June, he had written a lab's note about a malicious downloader script that used the curl function, and we chatted a bit about it, but more importantly, we went really in-depth on all the different varieties of malware that website owners need to be aware of. But, before I get started with Krasimir, I just wanted to remind you about the Sucuri Sync-Up, our sister podcast. It's a weekly website security news briefing that you can find anywhere you get your podcasts, as well as the video version on our social media feed, and now you can even get it on your Amazon Alexa smart speakers. Just search Amazon skills for Sucuri Sync-Up, add the flash briefing, and get new content delivered every Monday. Now, on with the show. Hi Krasimir, thanks for joining us on the show. I thought we could start off and maybe have you tell us a little bit about yourself and what you do here at Sucuri? Krasimir Konov: Yeah, sure. Well, I joined Sucuri originally in 2014, but I've been in the IT business for about 10 years. Nine of those I did security. And currently at Sucuri I'm one of the malware analysts. I used to work in the front lines, used to clean websites and whatnot, and then I gradually moved up, and now I'm working in the malware research department. And my day to day job is basically analyzing malware, and then once I analyze it and figure out what it is, then I will create a signature for it. And we'll add those signatures to our tools, so we can automate some of the work we do. And I also write some Labs Notes blog posts. Usually, if I find something interesting in malware or some security topic, I'll write about it. Justin Channell: Yeah. And of those topics recently that you wrote about, one was about a malicious curl downloader, and how exactly did that work? Krasimir Konov: Right, yeah. That was an interesting one, but not very unique or anything like that. We see that a lot with curl being used as a downloader. It's a very common malware. So rather than including the actual malware in the file, the attackers would use curl to download the malicious code. In this case, they'll download it from Pastebin, but it could be anything. It could be another website or anything like that, and curl would just make a call to the website, request the code. The website will respond with the code, and then later on, there is some code to either save the output somewhere on the website, or you'll just run it through eval and execute the actual code right away. Justin Channell: Right. And you said that it's commonly found in malware, but let's kind of maybe talk a bit broader about malware in general. What is everything that is classified as malware? Krasimir Konov: Well, in general it will be anything that the owner of the website didn't authorize, anything that was added by a third party. There is a lot of different malware. It could be even something like a defacement that will also be considered malware because it was something the user did not authorize. Even though it might not be doing anything malicious on the website, it's not infecting users, the visit is still something they did not authorize. So defacement would also be considered malware. And even something like ransomware where the website is technically not really damaged, it's all encrypted, but it's not infecting anybody. It's not doing anything malicious, but it's still encrypting the entire website and asking the user or the customer, the owner of the website for a ransom they need to pay in order to get the website back online. Justin Channell: Okay. Let's maybe break it down to each individual type of malware. For example, what would be a way that maybe I-frames could be maliciously used by a hacker? Krasimir Konov: Yeah. An I-frame can be used maliciously when it loads content from another location. You can look at the I-frame as a window that just opens another website. So anything that website has on it, you're pretty much loading it through the I-frame. So if that website is infected and it's serving some kind of malware, by opening an I-frame, you're loading all those elements, everything that was on this website. And sometimes the I-frame can be as small as pixel or something hidden somewhere off the screen, so you wouldn't even know that it was opening it. Justin Channell: And yeah, I feel like we've also seen a lot of them where they're used almost to mimic popups as well. Krasimir Konov: Yeah. I mean the I-frame, it could just load from another website and the other website could do anything. It could be serving just malware and it would try to infect the user that doesn't even know that they're being connected to the other website. It could just have some other JavaScript that's just trying to open up pop ups on the original website through the I-frame. Yeah, it could be a lot of things. Justin Channell: Okay. And also, let's talk a little bit about conditional redirects and how those work. What allows a script to detect which devices are coming in and where they're coming from? Krasimir Konov: Right. Yeah. That's a common one we see a lot. Basically, a conditional redirect would be something, it's a redirect on the website. It's obviously malicious, but there's certain conditions that need to be met before the redirect is actually executed or the redirect happens. For example, let's say if it's on a phishing website or a phishing page that is hidden somewhere on the website. For example, if Google visits it, obviously the attacker doesn't want Google to see the actual phishing page and record it as a phishing page. So they'll look for, for example, the IP address. They would look for the user agent. And a lot of times they can tell that it's a bot. So they'll just return a 404 response, for example, that will be like, "Oh, page not found." So Google would be like, "Oh, it looks like this page doesn't exist." But then if a regular user goes to the same page, then those conditions will be met. The actual website or the script behind the phishing will check and see, and be like, "Oh, this one is running Firefox or Chrome," and be like, okay. And then they'll look at the IP and be like, "Oh, he's in whatever, he's in United States somewhere." And he's like, "Oh, okay. That's good." And then once all of these conditions are met, then the actual script will serve them the actual phishing page. And it'll be like, "Oh, you need to fill out this to recover your account or whatever, or type in your credentials to log in here." Justin Channell: And so this is the type of thing we're really, a website owner is going to run into this more commonly when people are complaining about they're getting served bad content or whatever, and they can't seem to replicate it. It's likely probably these kind of redirects. Is that right? Krasimir Konov: Right, right. It could be something as specific as, for example, a range of IP addresses that correspond to an ISP or maybe let's say a country. It could be like, "Oh, were targeting only customers in the US," so if you're connecting from another country and you go to the same website or the same page, it would just say 404. It will give you a page not found. But then if you actually have an IP address from the United States, you're connecting from the United States, then it will actually show you the phishing page. Justin Channell: Now another type of malware I feel like we see a lot here is SEO spam. We hear people talking about that. What are some of the top SEO spam keywords that you see coming through? Krasimir Konov: Yeah. We get that a lot. We see a lot of spam on websites. A lot of times attackers will use SEO spam to gain ranking for their own website. Or they'll just try to include some kind of SEO spam in links to another website that they're currently running or something. I mean, these things change all the time. So a website might be up for a week and then it'll disappear, and then they'll start another campaign. But yeah, we see that a lot. We see all kinds of keywords they use. Most common ones will be something like Viagra. We'll have like jerseys for sale. A lot of times, they'll use name brands like Nike, Rolex, Prada. We've seen even some essay writing services for some reason. I'm not sure why, but that's common. We see, for example, pharmaceuticals a lot that will use specific medicine names. They'll use all kinds of replicas, like a replica bag of this, replica this, replica that. We'd see prescription, also payday loans. And obviously there's some adult related sites and things like that keywords. Justin Channell: So pretty much anything that people are going to be searching and clicking on are probably going to be targets for SEO spam? Krasimir Konov: Right. I think a lot of it commonly is pharma related because a lot of people are looking to buy medicine online, and a lot of times will require a prescription. So a lot of people are like, "Oh, let me see if I can find this medicine that I can buy it online somewhere." They don't need a prescription. They don't want to pay to visit a doctor and whatnot, and they'll look for it. And yeah. Justin Channell: Now, whenever somebody's website does get hacked with a SEO spam attack, what kind of effect can it have on the website beyond just being defaced? Krasimir Konov: Yeah. You can have a lot of things can happen, negative things. For example, the website can be blacklisted because of the keywords. And that usually represents a big red warning when you go on the website, depending on who blacklisted it. But if it's Google, for example, you'll see a big warning and it'll tell you this website contains malware or there's something wrong with this website. So, pretty much all the traffic on the website will be gone. And then you can also lose a lot of your reputation if there is a SEO spam on the website. For example, if you were ranked in say number five for certain keywords that represent your product on Google search engines, and then suddenly you get hit with SEO spam, then all these search engines then go and visit the website. And all of a sudden they're like, "Oh, there's all these weird key words on here, all this SEO spam that's causing a lot of mixed signals." And the search engines are like, "Oh, where do we rank this website now? Do we rank them with this product that's originally what the website is about? Or do we take into consideration all these other keywords that are mixed up that are SEO spam?" So, all of a sudden your website might go from being ranked number five on the first page to being on the 10th page. And then you rank for all these other keywords that you didn't intend to. And then people search for something completely different. They're searching for jerseys or something, or now they're searching for Prada products, and then suddenly your website pops up in there. So you're not really getting any good traffic, not targeted traffic. But, yeah. Justin Channell: Okay. So in a lot of ways, the effects of SEO spam would kind of be the same for defacements or any kind of malware with the blacklisting, but it does bring that kind of unique part to it where then it can also then bring traffic that you weren't expecting from somebody searching for jerseys, for example. I had not really ever thought about that. Krasimir Konov: Right, right. Yeah. It will definitely bring some traffic. I've seen a lot of times where websites will be connected. Let's say, there was 1,000 websites that were all infected with SEO spam, and it will kind of link each other to try to bring each other up into the rankings. And so you would see a lot of strange traffic from some random websites that were, for example, that were previously infected, even if they might not be anymore. But yeah, they'll be sending traffic to you or there'll be usually search engines sending you traffic, but for the wrong keywords. People are looking for something else, so obviously they're not going to be interested in your website. They're not going to buy anything because they're not looking for that. Justin Channell: And now, so thinking of the way websites get infected, a very common way it seems to be is through phishing campaigns. What are some recommendations you have for the best ways to avoid becoming a phishing victim? Krasimir Konov: Yeah. There is some ways. I mean, it depends really on the type of attack. Obviously, a lot of people, when they think of phishing, they think, "Oh, it's just like a PayPal phishing page and it just looks like the original," but it could be more subtle. If it's just a regular page where you're just going and you get redirected to another website, obviously the first thing to look is if you have the security padlocks, make sure that traffic is encrypted. A lot of these websites don't really have any encryption nowadays. More are starting to get that with pre SSLs being issued and whatnot. But that's the first thing to look and see, make sure. Anywhere you're typing your sensitive information, you want to make sure you have the padlock to make sure everything is encrypted. Krasimir Konov: But also you want to look at the URL of the actual website you're visiting. A lot of times they'll try to hide it. So you might have to be careful and look closely. Something that might be an I will be an L or something like that. And a capital I and L might look kind of similar into your IRL, so you might miss something like that. Say, if you're looking for PayPal and it might replace the L with an I, and if you don't look closely, it might look exactly the same. And you're like, "Oh, okay, it's paypal.com," but not really. So yeah. Just pay attention to the URL, make sure it is the actual website. There's no paypal.com dot something, dot something else, dot com. Yeah. You want it to just say paypal.com, and then it'll have forward slash and something else. But yeah, it gets more complicated when you have, for example, a phishing page that's injected into a regular page. For example, you have a checkout page on a website that you're buying things from and you go through the checkout page and you're looking at where you type in your credit card information and whatnot. And you might have a phishing page that actually looks exactly like a little box that gives you where you put in your credit card number, or your name, your address, and all that. So that will be more subtle. For example, that could be also an I-frame that's just coming from another page. And it will look exactly like it's part of the website. You're on the legitimate website, but only that portion of the website is actually the phishing page. And you look at it and you're like, "Oh, okay. It looks fine. I'm just putting my credentials." So that one could be a lot harder to figure it out. Usually, if it's something like that, I look for something that looks kind of out of place. Maybe they didn't get the right font. It might not be the same as the original website or there might be something out of place, some fields that are missing or some fields that are squished into the left or the right. It looks kind of awkward. It's like, why would this be like this? The whole website looks professional. There's a pink background or something, for example, and then suddenly there's this white box in the middle. It's like, ah, it looks kind of weird, out of place. Justin Channell: So pretty much if anything looks slightly out of place, you really should double check everything at that point. Krasimir Konov: Right. Right. Yeah. Obviously there's more ways that you can check, but I wouldn't get into more technical, like inspecting elements and looking at stuff, but yeah. Justin Channell: And now another type of malware that's kind of, and it kind of plays in with whatever the other infection is, is backdoors. Can you give us some examples of what backdoors can be? It's mainly just when a hacker can get back into the site to reinfect it, but I know there are a ton of different methods. And what are some of the more common ones and then maybe some that really interesting that you've seen? Krasimir Konov: Yeah, there is a lot. They'll probably be one of the first things the hacker would do is if they compromise a website, obviously they'll try to spread backdoors and just inject code everywhere so they can get back in, even if the owner of the website or webmaster cleans it. They want to try and hide some malicious code somewhere so they can always get back in. There's many variations. A backdoor could be something as simple as a single line of code to just [inaudible 00:20:19] argument, some kind of string or something via get or post. Krasimir Konov: And then it runs into an eval, so it evaluates the code and executes it. And some backdoors are very complex and they can be included in, let's say you have a WordPress site and you have a specific login page where all the login credentials are being processed and everything else. They could even inject code into that to basically bypass the whole login mechanism so that they can just bypass everything. They don't even have to know any user. They don't have to know the password, nothing. They'll just include some lines in there, and every time they'll be just able to log in. Yeah. It gets pretty crazy. Yeah. I mean, there's all kinds of malware. There is always a malware, for example, that just targets credit cards and will just target the eCommerce websites. And they'll just try to steal the login credentials, I mean, the credit cards. They'll try to get your address, your credit card information, any kind of CVV code or whatever you typed into the billing address, everything. And then there's also malware like the backdoors that are just trying to keep the attacker in control and trying to get them back into the website. There's just so many variations of what a malicious user might want to do on a website. Some can be something as simple as just reinfecting the website. They don't want to keep control. They just want to keep reinfecting it with some kind of malware. So even if you clean it, it would just get reinfected. Some of them in the database, otherwise might be in the files. We've seen some added into a [inaudible 00:22:14] job that just keeps running on the server. There could be malware that is just a giant to, for example, attack out of websites. Like for example, a distributed denial service where they put the same malware on thousands of websites. And then they try to send traffic to one website to try to bring it down. Yeah. People try to do all kinds of stuff with websites. We've seen even some cryptocurrency mining malware that you go onto a website and suddenly your PC starts running like crazy. And you're like, what the hell is going on? Your fans turn on and the PC is 100% CPU. And it turns out that the website has some malware that's just by mining Bitcoins with your CPU and it's using all of it. Justin Channell: Wow. Okay. So one question now, the last question I have is of all the malware that you've seen, what do you think is the coolest piece of malware that you've ever seen? Krasimir Konov: I think the coolest would be the ones that are so subtle that you don't even know that it's there. For example, we've seen some that were pretty innovative. It will be just a one liner code that's just one line. And for example, it will be let's say 40-50 characters, something like that. And that's all it is. And they'll hide it somewhere in between the legitimate code. And if you don't know what you're looking for, you would never see it. It doesn't look suspicious. There is no links to some other website. There's no some kind of encrypted code or anything like that. It's just a simple one line. And then if you're just scrolling through the file looking for something, you would never see it. It just looks like all the other code. And then if you look closely, you're like, "Oh, there's this..." Look closely, and you're like, "Oh wow, this is not supposed to be there." And then you keep looking at it and you're like, "This looks really weird." And then you see that it's actually doing some malicious things and trying to evaluate some code or taking output from the outside, I mean, some input from outside, you can call it and give it code to run. Justin Channell: Well, Krasimir, thanks for coming on and talking to us for today. Krasimir Konov: Yeah. Thank you. Thank you. I'm so happy. I'm glad I was able to do this podcast and I can't wait to do another one. Justin Channell: Yeah, we'll have you on again. Thanks. Krasimir Konov: Thank you. Justin Channell: Thanks again to Krasimir for joining us here on the Sit Down. We'll be back with another episode next month. So be sure to subscribe on Apple podcasts, Spotify, Stitcher, or any podcasting platform. Also, be sure to follow us on social media at Sucuri Security and check us out at sucuri.net. That's S-U-C-U-R-I.net. I'm Justin Channell, And this has been the Sucuri Sit Down. Stay safe out there.](https://blog.sucuri.net/wp-content/uploads/2020/05/20-sucuri-podcast-blog-post_blog_image-390x183.jpg)
Justin Channell
Val Vesa
Allison Bondi
Cesar Anjos
28 comments
With all being said, do you guys at Sucuri really think that you have chosen the best option for the users by disclosing to the public this security flaw on the same day MailPoet version with the patch reached the WP.org repo?!
As you know, hackers are faster than other kind of users. You really think that a few hours is time enough for everybody to upgrade their website?! Even when we blog posted about, tweeted about it and you guys did the same, and all the twitter did the same, it wasn’t enough time. According to WP.org stats, MailPoet still have 75% of their users on versions older than 2.6.7, it’s a whole lot of users that haven’t upgraded yet. So the question remains, what’s best in the end?! If you have given us at least 1 month before disclosing it, the vast majority of our users will have had the time to do the upgrade.
For me this is simple, if one hacker have discovered it, he will use the exploit where he wants when he wants, and of course maybe share this information with some other hackers (or better yet, crackers), but that’s it. Now instead, we have dozens of script kiddies using Metasploit searching all the web for older versions of MailPoet to install their malware. Security theories and terms aside, for me this whole situation is very very clear: doing a disclosure so soon like this (same day as the fix lands on the repo) will increase the chances of users getting their website hacked by a 1000%.
I work on the Support there, and I bet that I’m one of the most affected by all of this. I’m the one that is reading all the angry our hacked users on our support channel all day long. We are only 2 guys on support, spending at least 1 hour to clean one user infected website at a time. I’m definitely sure that we are not the ones profiting with this entire situation (neither our users). As one of them previously wrote to me last week:
————
Hi Rafael
I like your plugin and figure that you went through difficult times.
Unfortunately, when I look at the logs of modifications with Sucuri, the first file which has been modified during the hack is a mailpoet one. But you had sent me big warnings and since I was in vacation I did not pay enough attention.
Meanwhile I paid for a restore by Sucuri, so that should be OK.
Regards
(name removed for privacy reasons)
————
FYI: This is my own personal opinion, and not the opinion of my employer.
Keep up the great work, Sucuri. Mail Poet needs to understand that handling this sort of situation with grace and professionalism goes a long way.
Thanks for stopping by and the kind note.
Tony
I’m glad you were involved in this situation, and thankful for the work that you do (I recommend you to all of my clients) but I know that I got the email MailPoet sent out announcing the vulnerability (I’d updated already, actually). I can understand why they’d be upset – as they mentioned, they can’t force users to update.
Making this issue much, much more public before most users had updated resulted, almost certainly, in more hacked sites, not less.
If you’d simply waited a few weeks after the update was pushed before drawing so much attention to it, the bulk of users (who update regularly, not because they check your site), then I think it would have resulted in better outcomes, and everyone would be happier.
MailPoet devs: this should have been massively highlighted in your changelog. But the email you’d sent out was perfect, and exactly what users needed (and it did highlight the severity of the issue)
Sucuri: responsible disclosure means something specific, and you held to that. Fine. But was there a compelling reason for your timeline beyond “well, the update’s out”? Why rush to update your blog “only a few hours” after they’d released the fix? Your blog isn’t just a call to users to update. It’s a call to hackers to exploit. It looks like your scanner didn’t start seeing many blogs hacked until later in July, and that’s on users – it’s important to update. But your highlighting the issue in such a public forum so soon after the fix’s release is what they’re mad about – and I just have to wonder why it couldn’t be done a week later with the same impact? The vulnerability had already been out there for several weeks. Why the rush?
Hi Jon
Thank you for your note, and thank you for your recommendations and support.
In regards to this:
We have a responsibility to disclose information as it becomes available to us. It’s part of our protocol. We’ve done it several times before, and will continue to do so.
There is no answer that will satisfy an open ended question like the one you asked, so I won’t attempt at one. Just know that as security researchers we do what we feel is in accordance with our principles and best practice. We did not disclose proof of concepts, we appropriately disclosed and followed all the appropriate steps that are demanded of us as responsible stewards in the security space.
We’ve been at this a long time. I can assure you, there is not amount wait time that would have been adequate. Just look at Rafael post:
Even with the issues at hand, close to four weeks after the disclosure and people have no updated. It’s always easy from the inside looking in to see the “clear path” but I assure you there is no such thing. As a business you define your path and you execute against it. That is what we have done.
I hope this provides you better context.
Thanks
These are fair points, and that’s really too bad that so many users are not patched yet. In light of that – the fact that at this point it’s been a month or so, and only 25% have patched, there’s probably not a compelling reason to wait on posting your information, either, so I can get behind what you’re saying. Context provided and understood.
There’s no way I would have predicted this stat is what it is. Then again, I don’t trust clients on my server to update their own sites – I handle it for them, and across all of my sites as of right now (about 100 or so), only a couple have even one plugin out of date.
Thanks for all you do. If you have a recommended fix to the other major WP hack happening right now (the XMLRPC thing), I’d love to read about that as well. Or even a stopgap plugin.
I just think this “disclose soon” protocol really don’t work for our case scenario – WordPress plugins (or even themes) that are hosted by the users. If it was a bug in WordPress.com or other software that is hosted on the companies’s server, for example Facebook, then this principle is okay. You update the code and Boom! everyone is safe. But this ‘protocol’ clearly does not work for our scenario, we have to revisit these rules IMHO.
Not everyone gets plugins directly from the author, so they wouldn’t get emailed by the author. The plugin may have been bundled with a theme. So it is absolutely the responsible thing to do for Sucuri to disclose this to the public as soon as a patch is released so everyone NOT on the author’s mailing list (or blog) knows about it as soon as possible. Get the word out as quickly as possible to as many places as possible. An exploit as serious as this one needs to be reported as quickly, and to as many places, as possible.
I just don’t understand why something as serious as this is getting nitpicked in the way it was disclosed. Mail Poet is being ridiculous for being upset about this, when they should be more upset at themselves for dropping the ball so severely on the security of their plugin and should instead be thanking Sucuri for the responsible reporting. (I know I am.)
Speaking of picking nits, you replied to me to disagree – but if you read what I wrote, I agree with you. I said I thought the disclosure was too soon, read a sensible reply from Sucuri, then changed my mind based on the new information there.
MailPoet is not being ridiculous, though; they’re in a difficult situation and (I think) making the best of it. I read their original post that started this discussion, and it’s not exactly full of fighting words. Either way, both MailPoet and Sucuri make a good product. Neither is perfect, and none of the people involved have reacted perfectly to this. But thanks to them both for playing their respective roles; the problem is solved for anyone paying attention.
Hi Rafael Ehlers
In regards to:
Yes, I believe that is what I was implying in the post. Apologies that I was not clearer.
Based on this logic:
Followed by this response:
Should I assume that your recommendation is we never say anything? Because what that tells me is that even after the recent debacle, four weeks after disclosure, no one seems to want to update. Isn’t his contradictory to the reset of the paragraph:
What am I missing?
Thank you for your clear cut position:
We shared ours as well above.
I presume this was your attempt at giving us a jab:
Yes, in case it was unclear. Sucuri is a business, in other words we sell products and we are for-profit. We offer a Website Antivirus product designed to provide malware detection and remediation and a Website Firewall product, designed to keep hackers out of your website. If somewhere in the process we were being sly about this or didn’t adequately disclose that, I apologize, and I hope that clears that issue up.
All the best
Tony
At the end of the day, it’s all business and marketing 🙂
There’s no denial of how well-informed you keep your clients and the general people. I always get to know of latest vulnerabilities from your blog.
I won’t tell you to stop but you keep growing to keep the Internet a safer place!
I my users don’t upgrade regularly, it’s not a problem of disclosure.
Any hacker can scan the release notes of all plugins for “security” fix and then bet on the fact, that there are many users who delay their update.
Honestly, wordpress is not the best choice for non-business or hobby sites. Everyone running wordpress should be willing to either maintain the updates or to pay services to maintain it for them.
Running several worpdress instances by ourself, we spend about 1-2 days per month to keep five multi-site instances with a total of about 50 plugins up-to date.
I feel with the mailpoet mates, especiall the supporters. And I think the only difference the ‘early’ disclosure made is that more people upgraded. The disclosure is a plea for update. The risk of a hacker reading it shall never be a reason to delay or non-disclose it. That is when we’ll start acting like the NSA.
That is not my way and I assume yours neither.
One idea comes to mind. How about a special ‘vendor’ package by sucuri that comes along with a disclosure. It not only provides suggestions for the patch but also for the removal of the infection. This package could at least help the supporters….
“Honestly, wordpress is not the best choice for non-business or hobby sites.” I’d agree for WordPress.org.
The actions you took were appropriate and I thank you.
We would also fully support your position and the way you highlighted this issue – the only proof that is needed is the huge number of installations are still out there even so long after the disclosure and all the publicity that went with it. We host a large number of WordPress users, some we manage and update immediately, some good users who regularly update themselves and also a number that never update their plugins. Your posting with an explanation of the seriousness of the issue was plenty reason enough for us to check across all our servers to see if any user was running this plugin – luckily none were but if there had been, then we would have been able to notify / take action immediately to not only protect them but all other users on the server. keep up the good work !
This is a non arguement, Sucuris methodology was 100% correct. Not to deminish Sucuris talents, but if they found this, then you can be pretty sure someone else has and is exploiting it. Not telling the public about this would be a far bigger error.
IMHO its only a matter of courtesy to tell the developers of a vulnerability in their code, as its probably already being exploited and users should be allowed to make the decision to remove the plugin in question or not.
Giving them time to release a fix is being generous.
MailPoet should accept that if this vulnerability wasnt flagged by Sucuri then it would have gone on unpatched for longer allowing misuse of their code. Again, they shouldnt take it for granted that Sucuri were the only people to see this vulnerability.
MailPeot should learn from this and move on, work with people like Tony, Daniel and their crew, we’re all on the same side afterall.
DaveW
What would be optimum for the customer base is to initially narrow the release to the vendor only so they can contact the customer affected. Then have a “blast” release after customers are notified. Mailpoet here seemed to not notify its customers initially. I found out about the hack just via dumb luck browsing the internet. The email from mailpoet needed a subject line like “Upgrade your blog now!!!” And sent daily for a week. I think this situation reveals where both sucuri and mailpoet could improve their processes to make the service better for the internet users. And wordpress.org should be able to have the plugin highlight itself when it needs a security upgrade. The whole situation was very passive while the hackers are very active.
Hi Tony,
I know about Daniel, I know about OSSEC and I personally think he’s a super talented security researcher we should thank for OSSEC.
I’m at MailPoet as a software engineer and system administrator on the SaaS projects we run and I’m involved in security researching, enjoying it as you do.
I also agree with you that this discussion can’t be completely understood if we don’t know about the history of bug disclosures and how the community arrived to the “Responsible Disclosure” concept.
What we were really trying to say is: Are a few days of “upgrade window” really that bad?
Let me quote the Mozilla security policy here:
https://www.mozilla.org/en-US/about/governance/policies/security-group/bugs/
“Before making a security bug world-readable, please provide a few days notice to the Mozilla security bug group by sending email to the private security bug group mailing list.”
“Please try to be understanding and accommodating if a Mozilla distributor has a legitimate need to keep a bug in the security-sensitive category for some reasonable additional time period, e.g., to get a new release distributed to users.”
The process is that Mozilla, for example, get notified about a vulnerability, fixes a bug, sends out a new version and after a few days, when they see a good amount of users upgraded, makes the vulnerability public.
Users are immediately notified about the upgrade, this is absolutely not about keeping vulnerabilities hidden, and it has nothing to do with brand.
I’d also like to point out that the percentage of our users on old versions discussed in the comments (75%) is wrong. I’ve been keeping an eye on these statistics and we managed to get around 60% of our users on the fixed version in around a 10 days. This means that if you exclude abandoned installations (around 20%) there are 20% of active users still vulnerable. It looks like a reasonable number to me.
Wouldn’t it be good if we could all work together, patch immediately, warn users with security messages, and a few days after release all the proper blog posts?
Marco
I support Sucuri disclosing the bug and fix for Mailpoet. I think, that the more people you can let know of the vulnerability and have them fix it the better as it leads to a safer internet. I am pretty sure that hackers wait around on the Sucuri blog to find vulnerabilities, i’m pretty sure that there are other channels that they use. If I developed a plugin, you discovered a bug, and we fixed it and patched it – I would want as many people to know that it was fixed.
As I said earlier, I love Sucuri and MailPoet, both business will stand well after this.
I worked in security (former Windows Server Sec Expert) and it was a really daily battle, and as soon as you find something you have to act, maybe it’s a little difficult to understand from outside.
I think the only thing to correct was that mailPoet should not take 15 days to make a update with a list of 10 features, but a security update ASAP.
I am working in the ISP business since 1997 and have seen a lot of exploits and vulnerabilities in applications, operating systems and software in general. In my current position I am dealing a lot with mediation. I try to get consent between two conflicting parties. E.g. ‘the techies’ and ‘the bosses’. What I see here is a classical clash of interests.
Sucuri, highly respected for its research, has found a severe vulnerability and is interested in making it public. This is their business. This is how they make money. Reputation is everything. However as Mr. Perez said, they also have an obligation. The intention is not to harm the other party.
Mailpoet, well known and highly recommended for its WordPress plugin, is interested to protect their customers and of course their market share. This is their business. They make money selling the plugin. Competitors are fast if it comes to negative press. And of course they are thankful to Sucuri for telling them about the exploit.
Both have some things in common: they earn money from what they do, they belong to the top companies in their business and they crossed each other’s way by accident. The whole discussion is about time. Sucuri could have waited a bit longer but Mailpoet could have moved much faster.
How long should Sucuri have wait? What if the other party would never respond? Does it mean Sucuri should not have disclosed the exploit at all? On the other hand what would be an appropriate time frame for Mailpoet to fix the bug? 14 days from getting noticed by Sucuri until releasing the fix seems long. Especially if you assume Sucuri told Mailpoet from which function the vulnerability has been born. This is kind of a chicken and egg situation. However this is something that happened. All one can do is to learn for the future. In such a case and if I would have to deal with it I would recommend the following:
Sucuri should review there procedures and implement mile stones. Introduce a seven day sunrise period (with NDA of course) after informing the other party so they can fix the bug. Then add a 3 days grace period to get the new version released. And from the release date settle a cease period lasting 48 hours. If you discover an
exploit tell the other party about the time frames and keep room for an extension of the sunrise period up to ten days if needed. Now the developer knows what will happen if he fails to act in time.
Mailpoet (and any other company facing a severe issue with their product) should have a disaster plan. Team up with one or two acknowledged WordPress developers who can help in case of unforeseen problems. Introduce a 3-layer working plan making sure you invest 24 hours per day into the issue. Hire an external supervisor who is deep into the WordPress codex to review your source code to avoid problems like this.
Or negotiate a contract with Automatic (not sure whether they offer code reviews – if not I would be happy to build such a desperately needed business unit). Talk to the people who discovered the bug. Keep them updated – daily! Don’t waste time and prepare a press release together with the researchers. Be thankful, not peeved. And always remember it is your code. If it is not a core issue than you are fully responsible for what happened and what will happen.
And one last thing to think about. If Sucuri finds a bug you can bet others do too. And if those are from the dark side time is your biggest foe. In my humble opinion Sucuri did the correct thing. Timing might have been better communicated to Mailpoet. The biggest mistake on both sides however is the public discussion. Those things are dealt with during a telephone conference between the CEO’s or better at a dinner or a golf course.
Just my 2 cents
Hi Tony,
I know about Daniel, I know about OSSEC and I personally think he’s a super talented security researcher we should thank for OSSEC.
I’m at MailPoet as a software engineer and system administrator on the SaaS projects we run and I’m involved in security researching, enjoying it as you do.
I also agree with you that this discussion can’t be completely understood if we don’t know about the history of bug disclosures and how the community arrived to the “Responsible Disclosure” concept.
What we were really trying to say is: Are a few days of “upgrade window” really that bad?
Let me quote the Mozilla security policy here:
https://www.mozilla.org/en-US/about/governance/policies/security-group/bugs/
“Before making a security bug world-readable, please provide a few days notice to the Mozilla security bug group by sending email to the private security bug group mailing list.”
“Please try to be understanding and accommodating if a Mozilla distributor has a legitimate need to keep a bug in the security-sensitive category for some reasonable additional time period, e.g., to get a new release distributed to users.”
The process here is that Mozilla, for example, get notified about a vulnerability, fixes a bug, sends out a new version and after a few days, when they see a good amount of users upgraded, makes the vulnerability public.
Users are immediately notified about the upgrade, this is absolutely not about keeping vulnerabilities hidden, and it has nothing to do with brand.
I’d also like to point out that the percentage of our users on old versions discussed in the comments (75%) is wrong. I’ve been keeping an eye on these statistics and we managed to get around 60% of our users on the fixed version in around a 10 days. This means that if you exclude abandoned installations (around 20%) there are 20% of active users still vulnerable. It looks like a reasonable number to me.
Wouldn’t it be good if we could work together, patch immediately, warn users with security messages, and a few days after release all the proper blog posts?
Marco
Hello Rafael. I could probably agree with you, and not because I’m a MailPoet user. The problem with plugins are the end users: I won’t be surprised if there are lots of your premium users who still use old versions of the plugin. There are a vast majority of WordPress admins (with username “admin”!), who aren’t non-techie people, nor have the expertise, know-how or even care to keep their websites software up to date.
In the other hand, WP cannot be automatically upgraded, neither its plugins. But it’s a new practice I’m seeing in several products (like WordFence) to add an automatic update feature. That way you are secured all the time, no matter you care or not. Maybe MailPoet could add this feature in coming versions.
MailPoet must strenghten its security audits. It’s a fact. Anyway, you guys did a good job advising to update immediately the plugin, from the newsletter and from the plugin news itself.
Sucuri, in the other hand, it’s commited to inform the news and vulnerabilities to those of us who maintain hundreds of sites and really care about it, the servers, the users and the security as a whole, even if that target is the 10% of sysadmins in the market. I must agree with Perez and Cid in which this wasn’t a zero day in its strict definition, but I can bet Sucuri probably didn’t waited a couple days more before publishing the vulnerability because, as Michael Weichselgartner stated, this is how they make money, and their Reputation could be at risk if the bad guys or worst, the competition, issued a review before them 🙂
At the end of the day, it’s all business. Even for the bad guys. That’s why EVERYBODY must react asap. Thank you.
Tony & Daniel – I think you handled it perfectly and just want to thank you for the continued good work.
Someday perhaps WP will have a mechanism where plugins can be auto-updated, maybe even selectively when a particular plugin and version has a security vulnerability. Regardless though I still expect security vulnerabilities to be announce the same day the patch is.
Anyone who has a basic understanding of the WordPress environment knows that security is an ongoing battle. I was introduced to the MailPoet plugin as a result of the post and have plans to purchase the plugin once the dust settles.
By acting promptly to fix the problem, the MailPoet team earned my respect and admiration. Unfortunately, today’s post has knocked that down a notch.
Tony, you’ve done a wonderful job of handling this situation. Your original post did a wonderful job of highlighting the issue in a way so other developers could check their plugins for a similar issue.
MailPoet people – settle down and thank Tony and his people for bringing this to the public’s attention. You should be joining them in their efforts to alert the public. This is your chance to shine – don’t blow it by whining.
Remember,you didn’t perform a coding felony – only a misdemeanor.
You could have chosen to abandon your plugin instead of fixing it, which would have meant almost 2 million users would have a timebomb installed their site. (I’ve seen this happen before – and had to pick up the pieces for my clients when it did.)
You fixed the problem promptly and that’s all anyone who’s reasonable would ask of your team.
Instead of trying to protect your reputation, focus on protecting and alerting your customers. Trust me, in the long run, they’re the ones who will MAKE or BREAK your reputation.
When it comes to critical security vulnerabilities there’s no such think like “reasonable amount of time”… action should take fast, not tens of days… remember Heartbleed?
Comments are closed.