If your business lives on Instagram, you agreed to something new this month. It showed up in Meta Business Suite as one more “we’ve updated our terms” notice, and you may have clicked past it on your way to answer a DM, like every reasonable person with forty tabs open.
Starting November 20, if you keep using Meta’s business tools, you’re agreeing to let Meta’s bots crawl your website. The first version of the terms listed training AI as one of the reasons. After publishers and lawyers pushed back, Meta removed the word “train,” pushed the start date back from October 30, and says its crawling hasn’t changed. The only way to reject the new terms is to stop using Meta for your business, which for most small businesses means walking away from Instagram.
If you’re in the EU, you may remember Meta’s form for objecting to AI training on your posts. That form covers what you post on Facebook and Instagram, and the new terms are about your website, which only your robots.txt file can speak for.
So the blog posts you finally finished, your service pages and the About page you’ve rewritten more times than you’d like to admit all sit on a website Meta now has permission to take a good long look at.
When I read this, I went straight to my own site to lock things down. Squarespace has a checkbox for blocking AI crawlers, so I ticked it and opened my robots.txt file to make sure it worked.
Squarespace had listed 26 AI bots by name, including the ones from ChatGPT, Claude, Google and Meta, and then left out the one line that tells them to stay away. I checked in a private window, unticked and reticked the box, and got the same file every time. My website had a well organized guest list with no way to turn any creepy uninvited attendees away.
The setting seems bugged right now. Even when it works, though, Squarespace decides which bots make the list, and I can’t add or remove a single one. That lack of control is a big reason I’m in the process of moving my site to WordPress, where I get to write the guest list myself, and exclude that creepy neighbor 2-doors down who always ends up with your opened mail.
The two reasons AI visits your website
The first reason is training. The bot copies your words into the giant pile of writing an AI company uses to teach its tools how to answer questions and write like a person. Your posts become study material, and you get no credit, no link and no client out of it.
The second reason is search. Someone asks ChatGPT for a brand designer who works with therapists, and the AI reads websites to build its answer. This is the visit that can cite or link to your business and send someone to your booking page.
You can turn off training and stay in search. Being cited and staying in search is still important, since more and more people ask AI for recommendations the way they used to ask Google.
Meet your robots.txt file
Every website has a small text file called robots.txt that tells bots where they’re welcome. Type yoursite.com/robots.txt into your browser and you’ll see yours.
Each rule in the file only applies to the exact bot name it mentions. Any bot left off the list is welcome by default, so you only need to list the ones you don’t want accessing your site or content.
Who’s who in your robots.txt file
| Company | Bot name | What it does | Block or keep? |
|---|---|---|---|
| OpenAI | GPTBot |
Collects content to train ChatGPT’s future models | Block to opt out of training |
| OpenAI | OAI-SearchBot |
Finds pages for ChatGPT search | Keep |
| OpenAI | ChatGPT-User |
Fetches a page when someone asks ChatGPT to look at it | Keep |
| Anthropic | ClaudeBot |
Collects content to train Claude | Block to opt out of training |
| Anthropic | Claude-SearchBot |
Finds pages for Claude’s search results | Keep |
| Anthropic | Claude-User |
Fetches a page when someone asks Claude a question | Keep |
| Perplexity | PerplexityBot |
Finds pages for Perplexity search, with no training | Keep |
| Perplexity | Perplexity-User |
Fetches a page when someone asks Perplexity | Keep |
Google-Extended |
Controls whether Google trains Gemini on your content | Block to opt out of training | |
Googlebot |
Runs Google Search, including AI Overviews | Keep | |
| Microsoft | bingbot |
Runs Bing search, which also feeds Copilot | Keep |
| Meta | meta-externalagent |
Crawls the web for Meta’s AI | Block to opt out of training |
| Meta | meta-externalfetcher |
Fetches a page when someone asks Meta AI | Keep |
| Amazon | Amazonbot |
Collects content to train Amazon’s AI | Block to opt out of training |
| Amazon | Amzn-SearchBot |
Powers Amazon search and Alexa, with no training | Keep |
| Amazon | Amzn-User |
Fetches pages for live Alexa questions | Keep |
| Apple | Applebot-Extended |
Controls whether Apple trains its AI on your content | Block to opt out of training |
| Common Crawl | CCBot |
Builds a giant public dataset many AI companies train on | Block to opt out of training |
| ByteDance | Bytespider |
Crawls for TikTok’s parent company, with no official documentation | Block, and ask your host to block it too if it floods your site |
The bots that fetch a page when someone asks (the ones ending in -User or -fetcher) may skip your robots.txt anyway, since a real person requested the page.
If you’re on Squarespace
Squarespace keeps the file locked. You get one checkbox for AI crawlers, and Squarespace decides which bots go on that list. You can’t add a bot it missed, and you can’t take off one you’d rather keep.
That matters more than the box makes it look. Right now the list includes DuckAssistBot and YouBot, which help DuckDuckGo and You.com answer people’s questions, so ticking it can cost you visibility in those AI answers. And any new training bot that launches gets a free pass until Squarespace gets around to adding it.
So before you tick anything, open your robots.txt, read the names on the list, and decide whether that trade-off works for your business. Leave the search engine box alone no matter what, because that one takes you out of Google. If you want to choose bot by bot, you need a platform that lets you edit the file yourself, like WordPress.
Grab my starter file
If you’re on WordPress, Wix (I hope not), Shopify or anything else that lets you edit robots.txt, paste this at the top of your file. Keep whatever’s already there underneath, especially your sitemap line, and hand this part to whoever built your site if you’d rather not touch it.
Make sure to keep any existing rows in disallow as these may be there by default and important
# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# Stay visible in AI search and answers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: meta-externalfetcher
Allow: /The second group is technically optional, since any bot you don’t name is allowed by default. I include it anyway so anyone reading the file can see exactly what I meant. Google Search and Bing aren’t named at all, so they keep crawling as normal, and blocking Google-Extended leaves you in Google Search and AI Overviews.
Keep your paid work behind a password
Robots.txt works on the honor system. The big companies say they follow it, Meta included, but nothing forces them to, and a promise from a tech company only lasts until its next terms update. Some smaller bots ignore it completely and help themselves anyway. Anything you’d be furious to see copied, like a paid workbook or the client templates you spent months building, belongs behind a password or inside a members area.