Meta’s AI crawlers are overloading small business websites: how to protect yours

Fast Company and Business Insider report on small businesses hit by Meta’s crawlers, the bots that move through the web automatically. US artist Chelsea Paulus says a bot called Meta-ExternalAgent got stuck in a loop on her site’s remove-from-cart function, reaching about 550,000 hits within days. After she blocked it, another bot, Meta-WebIndexer, pushed CPU use to 100% and crashed the site.

In Australia, Robyn Gough, who runs a small digital design business, says her servers delivered 2.45 TB of data between September 30 and October 28, with 97.6% going to Meta’s crawlers, and that customers hit errors when downloading their purchases. Meta’s documentation says the first bot serves uses such as training AI models, and the second improves Meta AI search results.

Meta responds that website managers can control its crawlers through robots.txt and that it respects their preferences. Paulus says, however, that robots.txt and a firewall can take 24 hours to work, and that the immediate fix came from her host, through code added to an .htaccess file. According to Cloudflare, bots made up 55.1% of US web traffic over the last four weeks.

What it means for you

Even a small site can go down because of visitors who aren’t human, and that costs you real customers. Three simple steps to protect yourself:

• Watch the numbers: open your hosting dashboard and check resource use and the names of automated visitors, especially if your site suddenly slows down.
• Decide what you allow: add rules to your robots.txt file for the bots you don’t want, keeping in mind that each bot has to be blocked separately.
• Prepare an emergency plan: ask your host now how it responds to bot overload, and look at the Cloudflare human verification tool that Paulus is exploring.