Every run, Hoppla fetches your robots.txt, parses the directives, and checks the syntax. It captures your User-agent rules and Sitemap: lines and flags SEO mistakes. And it does the part most checkers skip: it tells you which AI crawlers (GPTBot, ClaudeBot and the rest) you're letting in or shutting out. Then it hashes the file and warns on the next run when a rule changes, so an accidental edit doesn't go unseen.
GET /robots.txt → 200 · 412 B · Disallow: / · GPTBot blocked · ClaudeBot blocked · hash changed · WARNING
Fictional values. Not a live measurement.
On every run, Hoppla fetches /robots.txt, parses the directives, and validates the syntax, pulling out each User-agent block, every Sitemap: line, and any SEO issue it trips over on the way. It singles out AI-crawler access and reports, per crawler, whether GPTBot, ClaudeBot and the others are allowed or blocked. Then it hashes the contents and warns when a rule in them changes, so the file you reviewed last month is still the file you're serving today.
Hoppla requests /robots.txt and reads the response, capped at 1 MB so a misconfigured route serving a giant body can't run away with the check. A 404 or a 5xx is a finding on its own. When robots.txt isn't there, crawlers fall back to crawling everything, rarely what you meant.
Every User-agent block, Allow, Disallow, and Sitemap: line gets parsed and syntax-checked. A malformed line that silently does nothing shows up as a finding, and a blanket Disallow: / or a blocked CSS/JS path as an SEO issue, instead of quietly mis-routing crawlers for weeks.
Hoppla reads the rules against the known AI crawlers and reports allow/block status for each: GPTBot, ClaudeBot and the rest. You see at a glance whether you're feeding the answer engines or locking them out. Either way it becomes a call you made on purpose, not a default you inherited.
Hoppla stores a content hash of the file. On the next run it compares. Same hash, nothing to say. Different hash, it diffs the rules, and a changed rule is a WARNING. A deploy that rewrote robots.txt, an SEO plugin that edited it, a staging config that shipped to prod: all of it shows up the run after it happens, not three weeks later.
The most expensive robots.txt mistake is a single line: a staging Disallow: / that rides a deploy into production and tells every search engine to stop crawling you. Hoppla flags it on the next run, while it's still a warning and not a traffic cliff.
Your robots.txt is quietly deciding whether ChatGPT and Claude can read your pages, usually via a default you never set. Hoppla names which AI crawlers you allow or block, so the answer-engine decision is yours to make instead of your CMS's to guess.
robots.txt rarely changes on purpose. So when it does, that's the news. Hoppla hashes the file and warns when a rule changes, caught the run after a deploy or a plugin touches it, not three weeks later once traffic has already slid.
You test crawler rules on staging with a blanket Disallow: /, then ship. Three weeks later organic signups are flat, you finally open robots.txt, and there it is: the staging block came along for the ride, and Google's been quietly de-indexing you the whole time. Hoppla would have warned you the morning after that deploy, when the hash changed and the Disallow: / showed up in the finding. You fix a one-line file instead of clawing back a month of rankings.
A client launches a redesign and asks the obvious question: is the new site crawlable, by Google and by AI? Instead of opening DevTools and eyeballing the file, you point at Hoppla: robots.txt fetched, syntax valid, sitemap present, GPTBot and ClaudeBot allowed. A five-second answer backed by a real check, on the same monitor already watching their uptime and SSL.
Across hundreds of client sites, an accidental block isn't an if, it's a when, and it lands on the one site nobody's looking at. With robots.txt on every monitor, a 404'd file or a fresh Disallow: / surfaces as a warning on that client's board the run after it ships. You catch it as a finding instead of reading about it in an angry SEO report a month later.
Crawlability and AI-readiness are the kind of check customers don't think to ask for until it bites them. Add robots.txt monitoring to the tiers you resell: AI-crawler status, change alerts, syntax validation, all under your brand and your domain. Your customers see a host that watches whether their site can even be found, and Hoppla's name appears nowhere.
Import the monitors you already run in another tool. What comes across depends on the provider. Run both side by side for 24 hours or 7 days before you switch.
The importer opens in your workspace once you have access.
Get early accessYou can, but you only look once you already suspect something's wrong, which is after the damage. A manual check can't tell you the file changed on last Tuesday's deploy, and it won't parse the rules to show whether GPTBot is shut out site-wide, let into only a few paths, or allowed. Hoppla checks it on every run and warns you on the next run after it breaks, so you're not relying on remembering to look.
No. This is an advisory check: it raises a WARNING, never a downtime incident. A missing file, a 5xx, an SEO issue, or a changed rule flags the monitor without opening an incident on your status page. It tells you something needs attention without pretending your site is down.
It reads your directives against the known AI crawlers and reports, per crawler, whether you're allowing or blocking them: GPTBot, ClaudeBot and others. Most sites never made this decision on purpose; the rule is whatever a default or a plugin left behind. Hoppla makes it explicit, so you can choose to feed the answer engines or keep them out instead of finding out by accident.
A few things: the file is missing (a 404) or can't be fetched (a 5xx, another error status, a timeout), the parser finds an SEO issue such as a blanket Disallow: / or no Sitemap: line, or a rule changed since the last run. If the file is present, clean, and unchanged, the check is OK and stays quiet.
Checks, alerts and status pages in one workspace. Now in early access: join the list for an invite.