Skip to main content

​
Bot Analytics

Bot Analytics shows which AI crawlers visit your site, which pages they request, and whether those requests succeed. Use it to answer practical questions:
  • Can ChatGPT, Claude, Perplexity, Google, and other AI crawlers reach your important pages?
  • Which URLs are crawled most often?
  • Are bots hitting redirects, 404s, or server errors?
  • Did an upload add new data, or was it already imported before?

​
What you need

You need a server access log from the website you want to analyze. Finseo accepts .log and .txt files and auto-detects the format from the first lines.
The log must include the request User-Agent. Without the User-Agent, Finseo cannot reliably tell whether a request came from ChatGPT, Claude, Perplexity, Googlebot, or a normal browser.

​
Upload a log file

  1. Open Bot Analytics in the dashboard.
  2. Select the project you want to analyze.
  3. Click Upload Logs.
  4. Drop a .log or .txt file into the upload dialog.
  5. Wait for processing to finish.
Finseo stores only detected bot visits. Duplicate bot visits are skipped automatically, so you can re-upload overlapping files without double-counting the same requests. Large files are supported up to 1GB. Files above 50MB are queued for background processing.

​
Supported formats

Finseo currently detects Nginx, Apache combined access logs, Cloudflare-style logs, and a custom fallback format.

​
Nginx combined access log

Expected fields:
  • IP address
  • Timestamp in [DD/Mon/YYYY:HH:mm:ss +0000]
  • HTTP method and path
  • HTTP status code
  • Response size in bytes
  • Referrer
  • User-Agent

​
Apache combined access log

Apache “common” logs without a User-Agent are not enough for bot classification. Configure Apache to use the combined log format before exporting.

​
Cloudflare-style log

Expected order:

​
Custom fallback

Finseo can also parse lines where the IP appears first, followed by a quoted request, status, size, and a quoted User-Agent.
If your format is different, export or transform it into one of the examples above before uploading.
Custom Nginx formats with extra fields, such as request timing fields between the byte count and referrer, may not parse. Export a standard combined access log when possible.

​
How to get logs

For managed hosts, look for Access logs, Raw logs, HTTP logs, or Web server logs in your hosting dashboard. Download the unfiltered access log, not only error logs. Upload the last 30 to 90 days first. That is usually enough to see crawler patterns and error rates. After the first upload, you can add new logs weekly or monthly.

​
What Finseo extracts

For each detected bot request, Finseo stores:
  • Bot platform and bot name
  • IP address and User-Agent
  • Request path and query parameters
  • HTTP status code
  • Response size
  • Referrer
  • Visit timestamp
The dashboard then groups this into:
  • Bot visits over time: trend by bot and date.
  • Crawled pages: URLs AI bots reached most often.
  • Performance: response status and response-size patterns.
  • Status codes: successful 2xx, redirects 3xx, client errors 4xx, and server errors 5xx.
The current log-upload parser focuses on crawl volume, URLs, response sizes, and HTTP status codes. It does not currently extract response-time fields from custom logs.

​
Sync integrations

The Sync tab may show providers such as Cloudflare, Fastly, AWS CloudFront, or API sync as future ingestion options. Manual .log and .txt upload is the current production path.

​
Troubleshooting

Check that the file includes User-Agent values and that the time window contains requests from known crawlers such as ChatGPT-User, GPTBot, ClaudeBot, Claude-Web, PerplexityBot, Googlebot, Bingbot, Applebot, AhrefsBot, or SemrushBot.
Make sure you exported the combined access log. A short Apache common log line usually lacks referrer and User-Agent fields, which are required for bot detection.
Finseo deduplicates visits by bot, timestamp, IP, and page path. Re-uploading overlapping files should not inflate your numbers.
Files above 50MB can be processed in the background. Check back after a few minutes. If the status fails, split the log into smaller date ranges and upload again.