Skip to main content
All articles
AI Strategy

Meta-ExternalAgent Took More Bandwidth Than 5 Crawlers Combined

Meta-ExternalAgent pulled 955 MB from cintra.run in 30 days, more than 5 AI and search crawlers combined. What it is, why Meta crawls and what to block.

T
Tanush Yadav

October 7, 202612 min read

Meta-ExternalAgent Took More Bandwidth Than 5 Crawlers Combined

Meta-ExternalAgent pulled 955 MB from cintra.run in 30 days. OpenAI, Google, Perplexity and Apple's crawlers pulled 848 MB combined. Meta AI sent zero visitors back.

TL;DR

  • The data: Cloudflare logged about 51,300 requests and 955 MB from Meta-ExternalAgent on cintra.run over the last 30 days. OAI-SearchBot, Googlebot, ChatGPT-User, Applebot and PerplexityBot together made about 23,900 requests and pulled 848 MB.
  • What it is: Meta-ExternalAgent is Meta's crawler for AI training and product indexing. It is not the crawler that gets a site cited in Meta AI. That job belongs to Meta-WebIndexer.
  • It's not just one site: DataDome counted 9.1 billion requests from Meta's two crawlers in Q2 2026, more than half of all AI agent traffic on its network, with almost no visitors sent back.
  • Cintra's call: block or rate-limit Meta-ExternalAgent. Keep Meta-WebIndexer, Meta-ExternalFetcher and facebookexternalhit open. Never block the AI search crawlers that drive citations.

What is Meta-ExternalAgent?

Meta-ExternalAgent is Meta's web crawler for training foundation AI models and indexing content for Meta products. It identifies itself as meta-externalagent/1.1 and, per Meta, respects robots.txt.

Meta's crawler documentation says it "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." In plain terms, it collects the open web so Meta's models and AI features have material to learn from.

It is one of five crawlers Meta documents, and they do very different jobs. Link previews, the card that appears when someone shares a URL on Facebook, come from facebookexternalhit, a separate bot. Citations in Meta AI come from Meta-WebIndexer. That split matters, because most "block Meta" advice treats all of them as one thing.

How much did Meta-ExternalAgent crawl cintra.run?

See where you rank across all AI answer engines.

Enter your domain and we'll scan your citation rate across ChatGPT, Perplexity, and Google AI.

Prefer to talk? Book a free 30-min call

In 30 days, Meta-ExternalAgent made about 51,300 requests to cintra.run and transferred 955 MB, more than the next five crawlers in Cloudflare's dashboard combined.

The numbers come from Cloudflare's AI Crawl Control dashboard for cintra.run, last 30 days, captured on 6 October 2026:

Cloudflare AI Crawl Control on cintra.run: Meta-ExternalAgent at 955.24 MB with 39.59k allowed and 11.75k unsuccessful requests in 30 days

Crawler Operator Cloudflare category Data transferred Requests (allowed + unsuccessful)
Meta-ExternalAgent Meta AI Crawler 955.24 MB 51,340
OAI-SearchBot OpenAI AI Search 401.68 MB 7,605
Googlebot Google Search Engine Crawler 315.01 MB 6,850
ChatGPT-User OpenAI AI Assistant 68.72 MB 3,958
Applebot Apple AI Search 46.52 MB 2,692
PerplexityBot Perplexity AI Search 15.62 MB 2,822
The other five combined 847.55 MB 23,927

Request totals add Cloudflare's rounded "allowed" and "unsuccessful" figures.

What the numbers show

  • Meta-ExternalAgent made 68% of the requests among these six crawlers, and 7.5 times as many as Googlebot.
  • It used 53% of the bandwidth, three times what Googlebot used.
  • It came in bursts. Its request chart spikes early in the 30-day window, then flattens. Cintra has since switched on the block for Meta-ExternalAgent in AI Crawl Control.
  • It sent nothing back. Google Analytics shows zero sessions from Meta AI (meta.ai) on cintra.run in the last 90 days.

Two cautions keep these numbers honest. First, Cloudflare counts any failed or blocked request as "unsuccessful", not only requests stopped by the AI Crawl Control block, according to Cloudflare's AI Crawl Control documentation. Second, a click from Meta AI inside Facebook, Instagram or WhatsApp can be recorded as a social or direct visit, so analytics can undercount Meta AI. Even allowing for that, there is no sign of meaningful referral traffic.

Why is Meta crawling the web so aggressively?

Meta is doing three jobs at once: gathering training data for its models, building its own search index for Meta AI, and fetching pages live when users ask questions.

1. Training data at scale

Training drives most AI crawling. Cloudflare's crawl data shows training made up 79% of AI bot activity by July 2025, up from 72% a year earlier. Meta leads by volume: Fastly's Q2 2025 research put Meta at 52% of AI crawler traffic, against 23% for Google and 20% for OpenAI.

2. A search index of its own

Meta has been working to stop depending on Google and Bing for live answers. The Information reported in October 2024 that Meta was building an AI search engine for Meta AI. Meta now documents Meta-WebIndexer, a crawler that "navigates the web to improve Meta AI search result quality" and helps Meta "cite and link to your content in Meta AI's responses."

3. The 2026 surge

The volume is still climbing. DataDome's AI Traffic Report for Q2 2026 counted 9.1 billion requests from Meta's two crawlers between April and June 2026, more than half of all AI agent traffic on its network.

  • Meta-ExternalAgent grew 74% quarter on quarter, from 3.1 billion to 5.3 billion requests.
  • Meta-WebIndexer grew 163%, from 1.4 billion to 3.75 billion. In June 2026 it passed Meta-ExternalAgent in monthly volume for the first time.
  • DataDome found those requests "sent almost no real visitors back." ChatGPT, by contrast, accounted for 80% to 88% of AI referral traffic every month.

cintra.run's dashboard shows the same gap on a single site: heavy crawling and no referrals.

Meta-ExternalAgent vs Meta-WebIndexer: what is the difference?

Meta-ExternalAgent gathers content for AI training and product indexing. Meta-WebIndexer builds the search index Meta AI uses to cite and link to websites in its answers.

Blocking one does not block the other. Each crawler has its own robots.txt token, so a site can opt out of training and stay eligible for Meta AI citations. Here is how Meta's five documented crawlers compare:

Crawler (robots.txt token) What it does Follows robots.txt? Cintra's call
meta-externalagent Trains foundation AI models and indexes content for Meta products Yes Block or rate-limit
meta-webindexer Builds the Meta AI search index and helps Meta cite and link to your pages Yes Allow; rate-limit only if it gets heavy
meta-externalfetcher Fetches a single link when a user asks Meta AI about it May skip it for user requests Allow
facebookexternalhit Builds the preview card when a URL is shared in Meta's apps May skip it for security checks Allow
meta-externalads Crawls for Meta's advertising and business products Yes Allow if you run Meta ads

Source: Meta's crawler documentation, checked 6 October 2026.

Keeping Meta-WebIndexer open is a bet, not a sure thing. DataDome's numbers show it crawling heavily too, and Meta AI sends little traffic today. But it is the only Meta crawler whose stated purpose includes citing and linking to your site, which makes it the one worth keeping.

Should you block Meta-ExternalAgent?

Block or rate-limit Meta-ExternalAgent if it costs you real bandwidth. Keep Meta-WebIndexer, Meta-ExternalFetcher and facebookexternalhit open so you keep Meta AI citations and link previews.

The rule Cintra applies is simple: decide by what a crawler is for, not by which company runs it.

Block or rate-limit: crawlers that cost more than they return

Meta-ExternalAgent is the clearest case. On cintra.run it used more bandwidth than Googlebot and every AI search crawler combined, and its job gives it no way to send a visitor back. On hosting billed by usage, that is real money.

There is a trade-off. Pages Meta-ExternalAgent can't fetch won't feed Meta's future training data. If the crawling is cheap for your site, letting it through costs little. If it is driving up your bill or slowing the site, a rate limit sits between a full block and an open door.

Always allow: crawlers that lead to citations and visitors

  • AI search crawlers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot and Meta-WebIndexer.
  • User-triggered fetchers: ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher. These fire when a real person asks an assistant about your page.
  • Search engines: Googlebot and Bingbot. Google's AI Overviews and AI Mode draw on Google's index, and Bing's index feeds Microsoft Copilot.
  • Link previews: facebookexternalhit. Block it and your shared links lose their preview cards.

On cintra.run, OAI-SearchBot and ChatGPT-User made about 11,600 requests in the same 30 days. That is the cost of being findable in ChatGPT, and it is worth paying.

What a blanket block costs

A "block all AI bots" switch stops the crawlers that put a brand into AI answers along with the ones that only take. Google's own AI Overview for "should I block AI crawlers" frames it as a choice between protecting content and staying visible in AI search. For a brand that wants AI to recommend it, the better answer is to block by purpose.

How do you block Meta-ExternalAgent?

Add a robots.txt rule for meta-externalagent, then enforce it at the edge with a Cloudflare block or a server rate limit, because robots.txt is a request, not a lock.

Step 1: add a robots.txt rule

# Opt out of Meta's AI training crawler
User-agent: meta-externalagent
Disallow: /

# Stay eligible for Meta AI citations
User-agent: meta-webindexer
Allow: /

Don't add disallow rules for facebookexternalhit or meta-externalfetcher. And avoid a blanket User-agent: * disallow, which would also catch the crawlers you want to keep. Name each crawler instead.

Step 2: block it in Cloudflare AI Crawl Control

Meta says Meta-ExternalAgent follows robots.txt. Even so, several of Google's top results for "meta-externalagent" are site owners reporting heavy load, on Stack Overflow, Reddit and Meta's own developer forum. If the cost matters, enforce the rule at the edge:

  1. Open AI Crawl Control for your domain in the Cloudflare dashboard. It is available on all plans.
  2. Find Meta-ExternalAgent in the crawler list.
  3. Switch Block Crawler on. Cloudflare creates a WAF rule and returns a 403, or a 402 if you want to signal that access is paid.
  4. Watch the request chart for a few days. Blocked requests move into "Unsuccessful".

Step 3 (optional): rate-limit instead of blocking

If you would rather slow Meta-ExternalAgent than shut it out, limit it by user agent. On nginx:

# In the http block
map $http_user_agent $meta_training_bot {
    default               "";
    ~*meta-externalagent  "meta-externalagent";
}
limit_req_zone $meta_training_bot zone=metabot:10m rate=1r/s;

# In the server or location block
limit_req zone=metabot burst=10 nodelay;
limit_req_status 429;

Every other visitor gets an empty key, which nginx doesn't limit. Because all Meta-ExternalAgent requests share one key, the cap holds even when its IP addresses change. In Cloudflare, a rate-limiting rule that matches the user agent does the same job.

Step 4: check it is really Meta

User-agent strings can be faked. DataDome's report advises checking crawler IPs against published ranges rather than trusting the name alone. Cloudflare's verified-bot labels do this for you. On your own server, check the requesting IP before you write rules around it.

What does Meta's crawling mean for your AI visibility?

Crawl volume is not visibility. Being recommended by AI depends on search crawlers and user fetchers reaching your best pages, and on what other sites say about you.

Still, a spike like this is a good prompt to check crawler access as a whole. Bot protection set to stop "all bots" can block AI search crawlers too, without anyone deciding it should. How AI models read websites covers what those crawlers can and can't see once they get in. How AI engines choose sources covers why they cite one page over another.

A quick check any brand can run this week:

  1. Open your CDN's bot or AI crawler report and list the top crawlers by bandwidth.
  2. Confirm OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Applebot, Googlebot and Bingbot are allowed.
  3. Block or rate-limit training crawlers that cost more than they return.
  4. Track whether AI engines actually cite and recommend you. Crawler logs show who reads your site; citation tracking shows who recommends it. How to measure AI visibility explains the metrics.

Frequently asked questions about Meta-ExternalAgent

These questions come up again and again in Google's People Also Ask boxes and related searches for Meta's crawlers.

Does Meta-ExternalAgent respect robots.txt?

Meta says it does. Its documentation lists Meta-ExternalAgent among the crawlers that follow robots.txt. Site owners still report heavy crawling, so enforce the rule at your CDN or server if the load matters.

No. Link previews come from facebookexternalhit, a separate crawler. Blocking meta-externalagent leaves previews on Facebook, Instagram and Messenger untouched.

Will blocking Meta-ExternalAgent remove my site from Meta AI answers?

Not on its own. Meta names Meta-WebIndexer as the crawler that helps Meta AI cite and link to your content, and it has its own robots.txt token. Block only Meta-ExternalAgent and Meta-WebIndexer can still index your pages.

Does Meta AI send traffic to websites?

Very little so far. DataDome found Meta's crawlers sent almost no visitors back in Q2 2026, while ChatGPT drove 80% to 88% of AI referrals. cintra.run recorded zero Meta AI sessions in 90 days.

How can I tell if Meta-ExternalAgent traffic is really from Meta?

Don't rely on the user agent; it can be faked. Use your CDN's verified-bot label, or check the requesting IP against the operator's published ranges.

Does llms.txt control AI crawlers?

No. llms.txt is a proposed file that summarises a site for language models. It doesn't grant or deny access; robots.txt and your CDN rules do. Does llms.txt work? covers what it can and can't do.

See where your brand appears in AI search

Free tool. Scan ChatGPT, Perplexity, and Google AI across buyer-intent queries, instantly, no sign-up.

Run free scan

Find out if AI is sending buyers to your competitors

We audit your AI visibility across ChatGPT, Perplexity, and Google AI, and show you exactly where you rank and what to fix.

Get my free AI audit

“We went from 200 visitors/day to 1,900 visitors/day and 40% of demos are from AI search.”

Sumanyu SharmaCEO, Hamming.ai

All articles

Related articles