← marketing-mindset

AI crawler robots.txt generator

One file, two independent decisions: which answer engines may read and cite your pages, and which crawlers may take your content for training. Blocking the first makes you invisible to citations; blocking the second changes nothing about how you appear in answers. This page writes both groups explicitly and adds the private paths and the Sitemap line. Everything happens in your browser; nothing is sent anywhere.

1. Your site and the paths that stay private

2. The two decisions

Your robots.txt



 



Checks





A robots.txt decision only counts if the page it protects can be read once it is
allowed: is the page
quotable at all, the
llms.txt that carries the same facts in one file,
the Q&A markup that
makes an answer quotable.


Agent-readable version of the same rules:
llms.txt ·
robots.txt in the wild.


Want the AI-readable layer built instead? Write to me on Telegram →








Also: A/B verdict calculator — was the difference separated from zero, and was the volume ever above the floor the observed lift needs?


Also: honest test queue planner — turn volume into a queue: the floor per arm, the days each test needs and the day each verdict lands.


Also: what one honest test costs to finish — the contacts a verdict needs, the days it takes and how many funded tests a budget carries.


Also: which step of your funnel can fund a verdict — the floor, the events each arm sees in a month, and the months a readable unit takes at your volume.


Also: which of your channels can actually fund a verdict — floor per arm, months to verdict, the cost of one full two-arm test and the channel to run first at your volume.