• Skip to primary navigation
  • Skip to content
  • Skip to primary sidebar
  • Skip to footer

Sucuri Blog

Website Security News

  • Products
    • Website Security Platform
    • Website Firewall (WAF)
    • Enterprise Website Security
    • Multisite Solutions
  • Features
    • Detection
    • Protection
    • Performance
    • Response
    • Backups
  • Partners
    • Agency Solutions
    • Partners
    • Referral Program
    • Ecommerce
  • Resources
    • Guides
    • Webinars
    • Infographics
    • SiteCheck
    • Reports
    • Email Courses
  • Immediate Help
  • Login
From .tk Redirects to PushKa Browser Notification Scam

Web Crawler & User Agent Blocking Techniques

August 14, 2020Luke Leal

11
SHARES
FacebookTwitterSubscribe

This is a simple script that allows hackers to block specific crawlers based upon website requests from specific user-agents. This is useful when you don’t want certain traffic from being able to load certain content – usually a phishing page or a malicious download.

if(preg_match('/bot|crawler|spider|facebook|alexa|twitter|curl/i', $_SERVER['HTTP_USER_AGENT'])) {
    logger("[BOT] {$_SERVER['REQUEST_URI']} - 500");

    header('HTTP/1.1 500 Internal Server Error');
    exit();
}

Using preg_match, the script looks for certain known crawler strings in the user-agent. If it finds a match, then instead of serving any website page, it instead reports a 500 Internal Server Error to the detected crawler. It accomplishes this through the header function, which can modify HTTP headers for incoming requests.

This can be verified by checking out the HTTP access logs for the website. Here is an example of a request sent from Googlebot, which receives a 500 Internal Server Error, and a request sent from an iPhone that goes through successfully (200 response code instead of 500):

127.0.0.1 - - [09/Jul/2020:11:36:52 -0500] "GET /test.php HTTP/1.1" 500 185 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

127.0.0.1 - - [09/Jul/2020:11:37:10 -0500] "GET /test.php HTTP/1.1" 200 147 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/47.0.2526.70 Mobile/13C71 Safari/601.1.46"

These types of scripts can also be used to trick bots or users from being able to determine if a phishing page or malicious download still exists.

To detect and prevent these issues, we highly recommend having file integrity monitoring in place and clean backups of your files/database. If your website becomes compromised, you’d be able to identify indicators of compromise and malicious behavior within your environment.

11
SHARES
FacebookTwitterSubscribe

Categories: Security Education, Sucuri Labs, Website SecurityTags: Black Hat Tactics, Labs Note, Obfuscation, Redirects

About Luke Leal

Luke Leal is a member of the Malware Research team and joined the company in 2015. Luke's main responsibilities include threat research and malware analysis, which is used to improve our tools. His professional experience covers over eight years of deobfuscating malware code and using unique data from it to help in correlating patterns. When he’s not researching infosec issues or working on websites, you might find Luke traveling and learning about new things. Connect with him on Twitter.

Reader Interactions

Primary Sidebar

Socialize With Sucuri

We're actively engaged across multiple platforms. Follow us and let's connect!

  • Facebook
  • Twitter
  • LinkedIn
  • YouTube
  • Instagram
  • RSS Feed

2019 Threat Report

Join Over 20,000 Subscribers!

Footer

Products

  • Website Firewall
  • Website AntiVirus
  • Website Backups
  • WordPress Security
  • Enterprise Services

Solutions

  • DDos Protection
  • Malware Detection
  • Malware Removal
  • Malware Prevention
  • Blacklist Removal

Support

  • Blog
  • Knowledge Base
  • SiteCheck
  • Research Labs
  • FAQ

Company

  • About
  • Media
  • Events
  • Employment
  • Contact
  • Testimonials
  • Facebook
  • Twitter
  • LinkedIn
  • Instagram

Customer Login

Sucuri Home

  • Terms of Use
  • Privacy Policy
  • Frequently Asked Questions

© 2021 Sucuri Inc. All rights reserved

Sucuri Cookie Policy
See our policy>>

Our website uses cookies, which help us to improve our site and enables us to deliver the best possible service and customer experience.