Ask Sucuri: How Does Sucuri Clean a Website?

Question: How does Sucuri clean hacked websites? What is the process?

We clean a lot of websites, ~ 400 / 500, daily during our normal load. To understand how we do it, you have to understand where it all comes from.

The biggest challenge with providing incident response services (remediation) on compromised websites is that a majority of website owners (webmasters) are not prepared. Most website owners lack security knowledge and fail to invest the time necessary to become familiarized with its concepts and how it’s applicable to their environment. They fail to get their websites ready for when, not if, an attack or disaster happens. In many instances, if the webmaster had been prepared, the entire remediation process would’ve been streamlined.

Being prepared can be as simple as having visibility (logging enabled), monitoring the logging activity and having a backup. While it would not stop the attacks, it would provide a solid foundation from which you can work from in the event of a compromise. These basics would do a lot for most website owners during the initial post-compromise phase. The phase in which we’re often engaged.

Because of this lack of data, it’s impossible to offer formal forensic services in addition to remediation services at scale. The majority of websites we work with are on shared hosting platforms and Virtual Private Servers (VPS) that rarely have backups, logging enabled and very little knowledge as to how or when the compromise occurred. This is the world in which we live in, daily cleaning websites at scale.

Blackbox Incident Response

Our incident response process is what I categorize as Blackbox Incident Response. We have some access, but generally lack the visibility required to understand what truly happened. We divide our black box incident response into five (5) steps:

1 – Establish a Baseline of the Environment

This is an important step, and it is the very first one we take in every case. It doesn’t matter what the site is, we work to establish a baseline. The baseline focuses on identifying what websites exist in the environment, specifically if there are any out of date software instances in the environment, worse yet, if there are any known vulnerable components (i.e., themes, plugins, etc.).

This may sound like a simple step, but it’s shocking at times to inform a webmaster of the things found in an account or server. Often we find old installations and configurations that have been forgotten with the hustle of the day to day operations. We take this information and share it with the website owner, and highly encourage them to take action if they want to avoid future issues.

2- Identify Known Compromised Files

We then move into the next step where we look for those things we know to be bad. We accomplish this by using a number of tools we’ve developed over the years that have helped us automate the detection and infection removal process. This process is designed to identify known issues. Many will attribute this to signature based remediation, but it’s a bit more involved than that. This is part of our artificial intelligence engine.

The past 6 years have given us a lot of insight in how to quickly locate malware, spam, defacements, backdoors, phishing file and a wide range of malicious files. We’ve taken those insights and built tools that help streamline the process. The process includes creating a backup of every file and saving it on the host in a safe location for reference if needed.

The key to this process is that it only removes things we know are bad with a 99.9999% confidence level. This phase is automated, but extra conservative in an effort to avoid breaking websites.

This process is very different than what we might come to expect with endpoint antivirus software. These technologies have the luxury of being able to quarantine infections.  When working with website malware however, it’s very difficult to use a similar strategy. Often the infections are embedded within good files, making the use of quarantines next to impossible. These are the biggest contributors to broken websites. This  is where our malware research team spends the majority of their time: refining this system and process.

3- Identify Anomalies and Signs of Compromise

We then move into a step we refer to as the human intelligence phase. One of the biggest decisions, realizations, we made early in our inception was that unlike traditional antivirus systems, when working with websites, it would be next to impossible to provide a 100% automated solution. The process would require some level of engagement from a trained professional to ensure that we not only identify and remove known issues, but spend time analyzing the anomalies and signs of compromise.

What wasn’t described in Step 2 is that the system not only identifies and removes known issues, but it flags and categorizes a number of other issues that require human interaction and verification. The analysts are critical to this process. It’s also critical to the research we perform. It ensures that there is a constant feedback loop to our remediation engine. It’s how we continuously adapt our tools  and remediation techniques to the emerging threats. The threats evolve and new techniques are employed to help evade detection on a daily basis. This helps combat that trend.

The analysts spend time reviewing all warnings, errors and performing an audit of the website to find potentially new malware variants or other security-related issues. Human intelligence beautifully complements the artificial intelligence described earlier.

Lastly, one of the key functions our analysts have is to verify that the website is operational post-compromise.

4- Check for Integrity Issues with Known Goods

The final phase of the remediation process includes identifying any potential integrity issues with known goods.

Any website that leverages open-source CMS applications like WordPress, Joomla!, Magento, Drupal and others should always perform an integrity check post-compromise. It not only looks for anomalies in files, but also any files that might have been added in directories that don’t belong. Our process extends beyond the core of the applications, including extensible components like plugins, themes, extensions, and modules when possible.

5- Remove from Blacklists

This step is not necessary in every case, but if a website is blacklisted, we do engage with the blacklisting authority on the website owners behalf to get the website removed.

The Importance of Being Prepared

Once the cleanup is complete, the real work begins. We follow the post-compromise steps with the customer in an effort to avoid future issues.  We also work to configure and deploy our protection platform, via our WAF / IPS, to avoid reinfections.

Some of the steps we recommend include:

1. Enable a WAF. A good WAF will block known attacks and exploits and also offer protection against zero days (0-days) through virtual hardening and similar technologies.
2. Update all CMS’, modules, plugins, extensions; more importantly, create a strategy to keep them updated. Make a habit of checking them and updating routinely.
3. Change all passwords. Use a password manager. It will enforce good passwords and not allow you to re-use them. Ever.
4. If you own a dedicated server or VPS, restrict SSH/FTP and cPanel access to only known whitelisted IP addresses.
5. If you have a dedicated server or VPS, monitor your logs. OSSEC, is a great open-source solution to consider.
6. If you are using WordPress, install our free open-source plugin to increase your visibility into application activities.
7. Restrict access to every admin and login page whenever possible. If you are using our WAF, you can do that very easily with the “Protected Page” option (set a captcha, IP white listing or additional password), can also be done via .htaccess

The cleanup process is important because it addresses the immediate problem. It does little however to keeping it secure and infection free. This is accomplished through combined proactive steps, like those described above, between Sucuri and our customers. Keeping a website secure is a process that every webmaster must embrace and realize they need to be a part of.

Here at Sucuri we provide a wide variety of tools and technologies that help facilitate the security process, but they have to be utilized and employed by website owners.

Be prepared.

Please direct any questions about malware, blacklisting, or security in general to: support@sucuri.net.

We’ve been compiling a number of #asksucuri articles. We encourage you to explore them and let us know what you think.