It stays on the site
The crawl starts at the address the customer gave and follows links on that host only. It never requests other domains.
HopplaBotFound HopplaBot
Found HopplaBot
in your logs?
Hoppla is a website monitoring service from Germany. Our customers add the websites they look after, and we check them: is the site up, is the certificate valid, do the links work. This page says which requests are ours, how to keep them out, and how to reach us.
- User agent
- HopplaBot/1.0link crawl and site checks
- robots.txt
- Respectedby the link crawl, by default
- Contact
- [email protected]questions and requests
Our requests
How each check identifies itself
Most of what reaches a website carries HopplaBot in its User-Agent header. Some checks do not yet; we are moving them over.
| Check | User agent | Reads robots.txt |
|---|---|---|
| Broken-link crawl | HopplaBot/1.0 (+https://hoppla.dev/bot) | Yes, unless the customer switched it off |
| Sitemap, robots.txt and GEO checks | HopplaBot/1.0 (+https://hoppla.dev/bot) | No. They fetch robots.txt, the sitemap and single pages. |
| Uptime, security headers, redirects, mixed content, health checks | GuzzleHttp/7 | No. Each fetches the one address the customer chose. |
| Performance audits (Lighthouse) | A regular Chrome browser, mobile and desktop | No. They load one page like a visitor does. |
| Certificate, port, ping and DNS checks | None | No HTTP request is made. |
When Cloudflare blocks the crawl
If Cloudflare blocks the link crawl (a 403 or 429 from Cloudflare, or an error from 520 to 530), the crawl retries with a regular Chrome user agent and keeps it for the rest of that run:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36To have your site crawled under the HopplaBot name, allow HopplaBot in Cloudflare.
Customers can give the link crawl a user agent of their own, so a crawl can also arrive under another name.
The crawl
One site, a few requests a second
It goes slowly
Two requests at a time, two per second, up to 500 pages, once a day. Customers can change this within limits: five at a time, ten per second, 5,000 pages, every six hours at most. A crawl stops after 30 minutes.
It leaves files alone
For PDFs, images, videos and other files the crawler reads the response headers and hangs up. Only pages are downloaded, to find their links.
Uptime checks request one address, every minute by default and every 30 seconds at the fastest. When one fails, a second location checks again before anyone is alerted. A link that times out or answers with a server error during a crawl is requested once more afterwards.
robots.txt
Keep the crawler out
The link crawl follows your robots.txt unless the customer switched that off for their monitor. It uses the group for HopplaBot and otherwise the * group. This keeps it off your whole site:
User-agent: HopplaBot
Disallow: /- The rules apply to every page, including the ones listed in your sitemap.
- A page marked noindex is not recorded. The crawler does not follow links on a page marked nofollow (meta robots or X-Robots-Tag), nor links marked rel="nofollow".
- Crawl-delay is not supported. The crawl’s own pace applies.
- An address a customer lists explicitly is checked even where robots.txt disallows it, and single-address checks such as uptime do not read robots.txt. Write to us about those.
Allow-listing
Let our checks through your firewall
Allow the user agent HopplaBot for the crawl and the site checks. Uptime and the other single-address checks do not carry it yet, so allow those by IP address.
- Our checks run from Falkenstein, Germany. When an uptime check fails, Helsinki, Finland checks again. All requests use IPv4.
- For the current IP addresses, write to [email protected]. We are preparing a public list.
Contact
Questions about our requests?
Write to us with your domain and roughly when the requests came. We look into it and get back to you.