Distributed Denial of Service (DDoS) attacks are becoming a common trend on our blog lately, and that’s okay because it’s a very serious issue for every website owner. Today I want to talk about a large DDoS attack that leveraged thousands of unsuspecting WordPress websites as indirect source amplification vectors.
Any WordPress site with pingback enabled (which is on by default) can be used in DDOS attacks against other sites. Note that XMLRPC is used for pingbacks, trackbacks, remote access via mobile devices and many other features you’re likely very fond of. But, it can also be heavily misused like what we are seeing.
The Story
It all happened against a popular WordPress site that had gone down for many hours due to a DDoS. As the attack increased in size, their host shut them down and then they decided to ask for help. They subscribed to our Website Firewall.
Once the DNS was ported over, we were able to see what was going on. It was a large HTTP-based (layer 7) distributed flood attack, sending hundreds of requests per second to their server. The requests looked like this:
74.86.132.186 - - [09/Mar/2014:11:05:27 -0400] "GET /?4137049=6431829 HTTP/1.0" 403 0 "-" "WordPress/3.8; http://www.mtbgearreview.com" 121.127.254.2 - - [09/Mar/2014:11:05:27 -0400] "GET /?4758117=5073922 HTTP/1.0" 403 0 "-" "WordPress/3.4.2; http://www.kschunvmo.com" 217.160.253.21 - - [09/Mar/2014:11:05:27 -0400] "GET /?7190851=6824134 HTTP/1.0" 403 0 "-" "WordPress/3.8.1; http://www.intoxzone.fr" 193.197.34.216 - - [09/Mar/2014:11:05:27 -0400] "GET /?3162504=9747583 HTTP/1.0" 403 0 "-" "WordPress/2.9.2; http://www.verwaltungmodern.de" ..
If you notice, all queries had a random value (like “?4137049=643182“) that bypassed their cache and force a full page reload every single time. It was killing their server pretty quickly.
But the most interesting part is that all the requests were coming from valid and legitimate WordPress sites. Yes, other WordPress sites were sending that random requests at a very large scale and bringing the site down.
WordPress Insecure Default Option = Very Large Botnet
Just in the course of a few hours, over 162,000 different and legitimate WordPress sites tried to attack his site. We would likely have detected a lot more sites, but we decided we had seen enough and blocked the requests at the edge firewall, mostly to avoid filling the logs with junk.
Can you see how powerful this can be? One attacker can use thousands of popular and clean WordPress sites to perform their DDOS attack while being hidden in the shadows. That all happens with a simple pingback request to the XML-RPC file:
$ curl -D - "www.anywordpresssite.com/xmlrpc.php" -d '<methodCall><methodName>pingback.ping</methodName><params><param><value><string>http://victim.com</string></value></param><param><value><string>www.anywordpresssite.com/postchosen</string></value></param></params></methodCall>'
Yes, that simple command on Linux can start it all.
Is Your Site Attacking Others?
It might be, and you could have no idea. To verify, look through your logs for any POST requests to the XML-RPC file, similar to the one below. If you see a pingback to a random URL, you know your site is being abused.
93.174.93.72 - - [09/Mar/2014:20:11:34 -0400] "POST /xmlrpc.php HTTP/1.0" 403 4034 "-" "-" "POSTREQUEST:<?xml version=x221.0x22 encoding=x22iso-8859-1x22?>x0A<methodCall>x0A<methodName>pingback.ping</methodName>x0A<params>x0A <param>x0A <value>x0A <string>http://fastbet99.com/?1698491=8940641</string>x0A </value>x0A </param>x0A <param>x0A <value>x0A <string>yoursite.com</string>x0A </value>x0A </param>x0A</params>x0A</methodCall>x0A" 94.102.63.238 – - [09/Mar/2014:23:21:01 -0400] "POST /xmlrpc.php HTTP/1.0" 403 4034 "-" "-" "POSTREQUEST:x0Ax0Apingback.pingx0Ax0Ax0A x0A http://www.guttercleanerlondon.co.uk/?7964015=3863899x0A x0A x0Ax0A x0A yoursite.comx0A x0A x0Ax0Ax0A"
In these cases someone tried to use one of our honeypots to DDoS fastbet99.com and guttercleanerlondon.co.uk (we don’t know or service these websites).
Prevent XML-RPC Abuse
To stop your WordPress website from being misused, you will need to disable the XML-RPC (pingback) functionality on your site.
You can do this by removing the file, xmlrpc.php, or you can disable notifications in your settings. The biggest challenge you’ll find with removing the file is that on an update it’ll come right back, annoying, I know. Some preliminary tests are showing that we’re able to bypass the disable notification setting but we are still investigating.
Update: A better way to block it is by creating a plugin that adds the following filter:
add_filter( ‘xmlrpc_methods’, function( $methods ) {
unset( $methods[‘pingback.ping’] );
return $methods;
} );
This is a well known issue within WordPress and the core team is aware of it. It’s not something that will be patched though. In many cases this same issue is categorized as a feature, one that many plugins use, so in therein lies the dilemma.
Online Tool to Check if Your Site Was Misused
Because of how prevalent an issue this is becoming, we’ve put together a little scanner that will check if your website has shown up in our logs. This scanner is only looking to see if your site has used to attack anyone within our network.
Use our WordPress DDOS Scanner to check if your site is DDOS’ing other websites.
Try it out and do your part to make the Internet a safer place for everyone.
Very interesting! Thank you for this.
How will this affect (or not) the JetPack plugin ?
It will prevent Jetpack from functioning, as you’re hacking core — which is always a bad idea.
What you should do, is `add_filter(‘xmlrpc_enabled’,’__return_false’);` or install http://wordpress.org/plugins/disable-xml-rpc/ instead.
Hello. I see a lot of executions in one of our sites. How can I tell which ones are from JetPack servers?
Can anyone confirm if the plugin Disable XML-RPC may negatively effect JetPack functions?
If you use stats module, you need XML-RPC enabled.
But you don’t need to turn off entire XML-RPC, you can turn off only pingbacks. If you don’t want to mess with code, simply install and activate this plugin http://wordpress.org/plugins/disable-xml-rpc-pingback/
Also, to check if the plugin is working, you can test XML-RPC remotely from http://xmlrpc.eritreo.it/
You could do something very simple that just disables Pingbacks and not all of XML-RPC. Something like this should work ( not tested ):
add_action( ‘xmlrpc_call’, function( $method ) {
if ( $method === ‘pingback.ping’ ) {
wp_die( ‘No pingbacks’, ‘Pingback is disabled’, array( ‘response’ => 403 ) );
}
} );
There should be no need to hack core WordPress or disable all of XML-RPC.
It looks to me like add_filter(‘xmlrpc_enabled’,’__return_false’) does NOT prevent pingbacks in wp 3.8.1, it’s only checked in wp_xmlrpc_server::login, which isn’t used for pingbacks.
‘xmlrpc_call’ is checked for pingbacks.
I agree with George in his reply, you should not hack core and that wasn’t the intention of mentioning that. Our solution to disabling XML-RPC was adding the option in CloudProxy, our website firewall.
XML-RPC is required in JetPack to auth with WordPress.com so if you disable XML-RPC, you will get a connection error when attempting to authenticate.
We actually use it after the initial authentication as well, for the ongoing two-way communication between WPCOM servers and the Jetpack site — for example, asynchronously pulling posts up into our elasticsearch index if the site has Related Posts turned on, for the WPCOM REST API module, and others.
(just to specify that temporarily unblocking it for authorization and then reblocking can still have bad side effects)
(if you’d like details on IP ranges that Jetpack requests may come in from to whitelist, just drop me a line on Skype — same as my Twitter handle)
We’ve identified the source of the pingbacks, and we’re looking to see if the Akismet plugin can help to prevent it.
That would be awesome. Any luck there?
As far as I am concerned you fail to explain the amplification. If it’s not amplified, I see no reason why I should care.
Just wondering if unchecking the “Attempt to notify any blogs linked to from the article” &
“Allow link notifications from other blogs (pingbacks and trackbacks)” options (on the SETTINGS>DISCUSSION page) would perform the same functionality as unsetting the Pingback with the Filter you mentioned above? Or would it still actually get around this?
Now that’s a very good question. I’m interested in reading an answer to this too.
unchecking “Allow link notifications from other blogs” will certainly turn off pingbacks for future posts, though you would still need to go in and edit existing posts to disable it for them (bulk edit is your friend here). Filtering ‘xmlrpc_methods’ is just a much faster way to disable pingbacks site-wide.
The “Attempt to notify any blogs linked to from the article” controls whether you ping other sites that you’ve linked to in your posts. It still generates network traffic to those sites, but cannot be triggered remotely as this article is discussing. Leaving this enabled is fine.
ok, and if the site dont need pingbacks, would recommend the “add_filter(‘xmlrpc_enabled’,’__return_false’);” method?
if you don’t need XML-RPC at all, then sure. But it’s used for more than just pingback… notably Jetpack and any other service that you may have that posts to WordPress for you.
Yesterday I implemented Disable XML-RPC plugin in a couple sites and can confirm that at least JetPack stats are working. Also JetPack Post Views plugin works well too.
I don’t post items remotely so it looks like it’s safe to disable XML RPC in this particular type of cases.
Also, in websites where WP is used as a simple CMS, with no posts, just pages, it also should be safe to deactivate.
JetPack is still able to publish to Linkedin, Google+, Facebook and Twitter.
And services like dlvr.it are able to retrieve information to send to social networks.
Everything seems to work ok, yet the XML RPC is disabled as I can test it from http://xmlrpc.eritreo.it/
Hope this help others.
I know Will, but AFAIK, XML RPC is ONLY used when you publish from third parties TO WordPress, not in the opposite way. I deactivated XML RPC and didnt notice any malfunction in JetPack; I use Stats, Comments and Publish to social networks. What JetPack features would require XML RPC enabled? Post by Email? Monitor? JSON API?
Maybe I’m blind, but I don’t think XML RPC is a feature every site cannot live without. Please clarify if I’m wrong.
Thanks for the clarification Will 🙂
great article. very informative. thanks. the wp filters around ping backs, etc gives me the idea to create a plugin – that for example allows admin to determine blacklist & white list for ping-backers, track-backers, etc. might be neatly implemented as simple extension to existing wordpress firewall type plugins – eg: wp-anti-spammer-anti-splogger plugin. will look into it.
The plugin for disabling pingback functionality for your entire site:
https://wordpress.org/plugins/disable-pingbacks/
Some simple PHP lines in the beginning of the code on the attacked site might would have changed this whole story. Something like this maybe:
[code][/code]
I’ve tried using the suggested code snippet to disable trackbacks and I get the following:
PHP Notice: Use of undefined constant xe2x80x98xmlrpc_methodsxe2x80x99 – assumed ‘xe2x80x98xmlrpc_methodsxe2x80x99’ in /wp-content/themes/child-theme/functions.php on line 39, referer: /wp-admin/options-writing.php
I oringally want it to be a MU plugin, but gave the same error, so I moved it to be a normal plugin, same error and finally moved to the theme folder. I’m using WP 3.8.1 – wonder if anyone else has had the same problem, might know the answer?
Thanks for this. Unfortunately the code in your Update isn’t a complete solution: it contains erroneous “smartquotes,” it doesn’t work in PHP 5.2, and it’s not packaged as a plugin (although I guess an alternative would be to put it in a theme’s functions.php file, right?).
Has anybody put together a proper plugin, preferably compatible with PHP 5.2?
Try “Disable XML RPC” 😉 Worked for me under PHP 5.4 and 5.2 🙂
Sweet! Thanks. Link: https://wordpress.org/plugins/disable-xml-rpc/
The guts of the plugin is one line:
add_filter( ‘xmlrpc_enabled’, ‘__return_false’ );
I take it that will disable pingbacks, and possibly other applications of XML-RPC as well? Fine by me, just wanting to understand all the implications.
It blows my mind how sometimes something so simple can cause so much trouble. One little line of code in the wrong place at the wrong time can wreck havoc for thousands of people!
There’s a plugin here with the patch applied http://wpshed.com/wordpress-distributed-denial-service-attack-ddos-security-plugin/
Thanks so much for sharing this information, Daniel. I checked my site and breathed a sigh of relief. Cheers!
I’ve found one of my sites is being used in DDOS. I have installed “Disable XML-RPC Pingback” hopefully that’ll do the trick. I don’t use JetPack and I don’t think I’ll need the pingback.
How long were companies impacted (unable to access the internet) during this attack? How do these attacks affect productivity and corporate image? I have found a service from TW Telecom which sits at the peering points will remove packets during the attack but NOT interfere with the internet traffic flow so the company continues to function as if it were not happening…thoughts?
website designing company Noida, web development company Noida, Web designing company in Noida, Noida, Delhi, software company in Noida, SEO , logo design and more
Ok to sum it up will disabling “Allow link notifications from other blogs”, but leaving enabled “Attempt to notify any blogs linked to from the article” still present security risk to my site or do I need to disable both and maybe install a 3rd party plugin?
This is very interesting 🙂 Essentially with this you don’t even need zombies for DDOS, and its so easy to find WordPress websites these days, its almost all over the web.
maybe the site should do a dns check to see if the ip in of the pingback request matches the domain in the included url *before* doing a request to that url to check for links.(most likely it wouldn’t match in the case of such an attack if the attacker isn’t also doing some kind of dns hack (far less likely) or using the same same host (also far less likely) – stopping it right there
I think its a small thing that could make such misuse harder to do
also the way that caching is done obviously needs to be looked at too.
(as that in itself sounds like it presents a DOS risk for the site (not other sites though) – given the ram each process handling a php request needs on 64 bit platforms these days it wouldn’t need many concurrent requests to send most sites into hardcore swap if too many of them can’t be handled very quickly from cache.
Over the last 3 days, I have noticed a massive amount of traffic hitting my VPS client sites xmlrpc file. At one stage there were more than 100 bots per second. This flood successfully brought down two of our busy ecommerce sites, leading to client calls (clients don’t really know what a DDOS is and don’t care, they just want their site to work. Though they were patient, this kind of issue can cast a shadow on your service).
We tried a whole range of things to try to mitigate the issue, SYNFLOOD in our CSF firewall (no effect), Wordfence plugin (no effect), BRUTE protect plugin (no use).
Finally we blocked the file using
Order deny,allow Deny from allbut the bots simply turned to the theme index file and continued their DOS attack.Then we came up with a mod_sec rule which had more success:
SecRule REQUEST_LINE "POST .*wp-login.*" "pass,initcol:ip=%{REMOTE_ADDR},setvar:ip.maxlimit=+1,deprecatevar:ip.maxlimit=1/600,nolog,id:35011"
SecRule IP:MAXLIMIT "@gt 10" "log,deny,id:350111,msg:'wp-bruteforce: denying %{REMOTE_ADDR} (%{ip.maxlimit} connection attempts)'"
SecRule REQUEST_LINE "POST .*xmlrpc.*" "pass,initcol:ip=%{REMOTE_ADDR},setvar:ip.maxlimit=+1,deprecatevar:ip.maxlimit=1/600,nolog,id:35012"
SecRule IP:MAXLIMIT "@gt 2" "log,deny,id:350112,msg:'wp-bruteforce-xmlrpc: denying %{REMOTE_ADDR} (%{ip.maxlimit} connection attempts)'"
The problem with disabling xmlrpc is that a lot of plugins and sites use it legitimately. Removing it will break wordpress. I haven’t tried your suggested plugin / function, will give that a go.
Interesting approach, limiting the amount of connections by IP could work, but what about and attack with 500+ source IP. It could overflow the iptables list if your server has not enough RAM memory available.
Are those websites using some kind of cache? CDN? Using it may help to mitigate the cpu load and disk stress. But obviously, it must be correctly configured.
Yes, IP table list is limited to 500 entries, we also use cloudlinux to manage consecutive connections so that any one account cannot chew up the RAM.
We’ve also installed bruteprotect plugin and that is definitely helping.
We are now looking at cloudflare and cloudproxy and considering which to try. Cloudflare seems more comprehensive and cheaper than cloudproxy and I’m not sure why? What do you use?
Glad you asked. I do choose a variety of things: MySQL cache in the server + W3 Total Cache to generate html versions + CloudFlare to allocate those html versions + Amazon Cloudfront to allocate all the other files (images/js/css). This combination is a relief for Apache. Plugins: CSF in the server + WordFence + iThemes Security. All these elements provide a resilient configuration and almost 0% downtime. Obviously counting on a very reliable hosting service is the cornerstone for your online buildings.
If you need to solve the configuration with a professional approach don’t hesitate in contacting me.
All the best
Simply disabled xmlrpc and black listed ips logged with malicious requests
( verifying pingback from 195.154.136.19 ) I get attacked recently on and on from that source…
Use captcha on login form
This shows the importance of keeping up with your WordPress security. Thanks for the free scanner. I just checked and thankfully my site is clean.
is this still an issue w/newest WP? 4.3.1 as of today, i think.
Wow, this is spooky. I had no idea you could weaponize other people’s websites against another one.
Nice post Daniel, I basically delete the xmlrpc.php file from WordPress and it is the best way to get rid of these DDoS Attacks. This is useful method if you are not relying on Jetpack like plugins which uses xmlrpc file.