You’ve possibly never ever utilized Brave Search. Well, not directly. Yet the AI you ask concerns could have, and it might have located your site there. On Nov. 2, the New york city Times sued Brave , the firm behind the Brave internet browser and Brave Search. The Times states Brave copied more than 30 billion webpages “without determining itself or providing web site proprietors with any device to obstruct its crawler,” after that offered that web content to AI companies. Brave hasn’t answered yet, so for now those are cases. However why can’t you keep its spider out without keeping Google out also? Brave describes that itself, right there on its assistance page.

Brave Sells Its Search Index To Cohere, Mistral And Perplexity

Brave sells accessibility to its search index to AI companies, through a product called the Brave Search API. When an AI assistant needs something current from the internet, it doesn’t always go and read the web itself. It can pay to browse another person’s index instead. Brave’s Regarding web page states the API is “Utilized by several of the most significant names in AI, consisting of Cohere, Mistral, Perplexity, and You.com.” Its AI page states the index holds “over 30 billion pages, with countless new pages indexed daily.”

The Times’s grievance opens up by naming five AI business it says Brave materials: Anthropic, Perplexity, Mistral, xAI and Cohere. None of them is being filed a claim against. Claude is the fuzzy one, due to the fact that the proof is old. The complaint indicate a 2023 Brave message about Anthropic training Claude 2 In March 2025 Anthropic added Brave Browse to its listing of subprocessors, the outdoors companies it uses, which TechCrunch reported The exact same week, Simon Willison, a developer that composes a commonly read blog concerning AI tools, asked Claude to search for pelican truths , and the 10 results it pointed out matched the very same search on Brave precisely. Whether Claude still makes use of Brave today, I couldn’t confirm.

Brave’s Spider Does not Have A Name, So You Can not Block It Without Blocking Google

Websites choose which crawlers may read them in a tiny data called robots.txt, and that just functions because crawlers have names OpenAI and Anthropic each release the names of theirs , so you can let one business’s spiders in and transform the various other’s away.

Brave’s help page , the one at the top of this write-up, claims its spider “does not advertise a separated user representative.” In ordinary words, Brave’s spider doesn’t wear a name tag, and Brave offers a reason: Some internet sites allow Google in and no one else , so a crawler with Brave’s name on it would get turned away. Brave says it adheres to Google’s guidelines rather: whatever Googlebot isn’t permitted to check out, Brave won’t read either.

So you can not allow Google in and maintain Brave out. You obtain both, or neither. To get out of Brave’s index, the very same web page factors you to the noindex tag , which informs Google to drop the web page as well. The Times calls this Brave’s “refusal to launch the name of its internet spider robot.”

My Web site Is In Brave’s Index, And My Robots.txt Can Not Also State It

A Brave look for site: nohacks.co on Nov. 3 revealed 16 of my web pages on the initial page of outcomes, 13 of them posts.

Brave’s outcome also reveals the homepage summary I replaced previously that exact same day, so what Brave serves is a stored copy, not a live consider my internet site. My robots.txt names 13 AI crawlers , from GPTBot to Bytespider, and obstructs 2 of them, Meta’s and ByteDance’s. There’s no line I might add for Brave. It’s tough to obstruct a name that does not exist.

Inspecting yours takes about 30 secs: go to search.brave.com and type website: and your domain. Whatever that appears is in the index Brave sells.

A Search Engine Sends Out Individuals Back, And An Index Marketed To AI Does Not Have To

Being in an internet search engine is a deal every internet site understands. A person searches, sees your web page, clicks, and arrive at your website. Brave Browse’s normal outcomes work like that, though Brave has actually included AI solutions of its very own, called Ask Brave , which The Times is additionally filing a claim against over.

Marketing the index is a various deal. The complaint prices quote Brave defining one component of it,”LLM Context,” as handing over”removed web page pieces and source metadata in a form and size that satisfies the needs of an LLM without need for further scratching. “So the AI company obtains your web page, currently chopped up, and composes a response from it. Whether that solution sends out any person back to you is the AI company’s phone call. It isn’t your own, and it isn’t something Brave is marketing. That’s the component that troubles me.

A Court Will Certainly Choose Whether Brave Can Market Its Index To AI Business

The New York City Times is declaring copyright violation and getting around its paywall, to name a few things, and the Daily Information, the Chicago Tribune and the Orlando Guard submitted a different situation the exact same day. Brave hadn’t reacted to either when I wrote this. It has argued its side prior to, though: when Information Corp sent it a cease-and-desist, Brave sued first, claiming it was only indexing the web, which “all internet search engine operators should do to exist,” and in July 2026 Information Corp sued back, saying Brave conceals its crawlers so publishers can not detect or obstruct them, Semafor reported A ruling on The Times’s pages will not immediately cover any person else’s, but it will state a whole lot concerning every site because index, which’s possibly yours as well. So I’ll be watching this.

Extra Resources:

This post was initially published on No Hacks

Included Image: FotoField/Shutterstock