This page holds aggregated numbers only: no raw log lines, no individual IP addresses and no visitor networks. Try other policies or actions below; your changes stay in this tab unless you own the report.
Crawl Ledger reads your web server's access log, groups every request by the bot or client that made it, and shows how much bandwidth each one took. Then it writes the robots.txt, nginx, Apache, Caddy or Cloudflare rules to block or slow down the ones you never agreed to feed.
🔒 Analysis runs in this browser tab and your log never leaves your device. Saving a shareable report stores only aggregated numbers.
1. Load a log
Apache or nginx combined format, or JSON lines (Caddy, Traefik, Cloudflare). Plain or .gz.
Two weeks of logs from Polyshelf, a fictional free 3D-model library on a small VPS. Model downloads are 3 to 200 MB each, so a single greedy scraper can cost more than every human visitor combined.
Invented data: IPs come from reserved and documentation ranges, and the crawler behaviour is staged to show each feature. It does not describe how any real company's crawler behaves.
Your robots.txt (optional, used to spot crawlers that ignore it)
Reports you saved in this browser
Bandwidth served
Went to bots and scripts
Biggest single consumer
Requests ignoring robots.txt
2. Traffic per day, by who asked
Hover or tab through the days for exact numbers.
Show as table
3. Every client, ranked by bandwidth
Browsers are grouped as human visitors unless a network behaves like a scraper. Select a row for paths, networks and user agents.
Client
Type
Requests
Bandwidth
Peak / min
robots.txt
Action
4. Rules to apply
Pick a policy, adjust any client in the table above, and copy the config for your stack.
5. Save and share this audit
Get a private link for your host, your team or your future self. Only the aggregated numbers and your rule choices are stored, never the log itself, individual IPs, query strings or visitor networks. Links expire after 180 days.
How it decides
Identifying clients
Each request's user agent is matched against about 80 known crawlers, AI agents, SEO tools, link-preview fetchers and HTTP libraries. Anything else that calls itself a bot is listed by name. Browsers are grouped by network (/24 for IPv4, /48 for IPv6).
Spotting scrapers in disguise
A network that uses a normal browser user agent is flagged when it pulls far more data than typical visitors, skips the CSS, scripts and images a real browser loads, or requests pages faster than people click.
Checking claims
Googlebot, bingbot and GPTBot requests are checked against a built-in snapshot of their published IP ranges. Requests from outside those ranges are listed separately as unverified. robots.txt rules are applied the way crawlers should: longest match wins.
What is simulated
Analysis runs in your browser; the server only stores reports you choose to save. The IP-range check uses a static snapshot instead of live reverse-DNS lookups, savings assume the rules applied over the whole period, and the rate-limit estimate replays each client's per-minute request counts.