> ## Documentation Index
> Fetch the complete documentation index at: https://docs.finseo.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Bot Analytics
> Upload server logs to see how AI crawlers access your website.
# Bot Analytics
Bot Analytics shows which AI crawlers visit your site, which pages they request, and whether those requests succeed. Use it to answer practical questions:
* Can ChatGPT, Claude, Perplexity, Google, and other AI crawlers reach your important pages?
* Which URLs are crawled most often?
* Are bots hitting redirects, 404s, or server errors?
* Did an upload add new data, or was it already imported before?
## What you need
You need a server access log from the website you want to analyze. Finseo accepts `.log` and `.txt` files and auto-detects the format from the first lines.
The log must include the request User-Agent. Without the User-Agent, Finseo cannot reliably tell whether a request came from ChatGPT, Claude, Perplexity, Googlebot, or a normal browser.
## Upload a log file
1. Open **Bot Analytics** in the dashboard.
2. Select the project you want to analyze.
3. Click **Upload Logs**.
4. Drop a `.log` or `.txt` file into the upload dialog.
5. Wait for processing to finish.
Finseo stores only detected bot visits. Duplicate bot visits are skipped automatically, so you can re-upload overlapping files without double-counting the same requests.
Large files are supported up to `1GB`. Files above `50MB` are queued for background processing.
## Supported formats
Finseo currently detects Nginx, Apache combined access logs, Cloudflare-style logs, and a custom fallback format.
### Nginx combined access log
```text theme={"system"}
23.98.142.45 - - [03/Oct/2025:14:23:11 +0000] "GET /blog/ai-seo-guide HTTP/1.1" 200 15234 "https://www.google.com/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
```
Expected fields:
* IP address
* Timestamp in `[DD/Mon/YYYY:HH:mm:ss +0000]`
* HTTP method and path
* HTTP status code
* Response size in bytes
* Referrer
* User-Agent
### Apache combined access log
```text theme={"system"}
52.70.240.171 - - [03/Oct/2025:14:28:10 +0000] "GET /pricing HTTP/1.1" 200 18422 "-" "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com)"
```
Apache "common" logs without a User-Agent are not enough for bot classification. Configure Apache to use the combined log format before exporting.
### Cloudflare-style log
```text theme={"system"}
2025-10-03T14:31:22Z 44.220.185.12 GET /docs 200 9310 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/bot"
```
Expected order:
```text theme={"system"}
timestamp ip method path status bytes "referrer" "user-agent"
```
### Custom fallback
Finseo can also parse lines where the IP appears first, followed by a quoted request, status, size, and a quoted User-Agent.
```text theme={"system"}
13.65.138.112 app "GET /features HTTP/1.1" 200 12482 "ChatGPT-User/1.0"
```
If your format is different, export or transform it into one of the examples above before uploading.
Custom Nginx formats with extra fields, such as request timing fields between the byte count and referrer, may not parse. Export a standard combined access log when possible.
## How to get logs
```bash Nginx theme={"system"}
# Common locations
ls /var/log/nginx/
# Copy the current access log
cp /var/log/nginx/access.log ./finseo-access.log
# Optional: compress first, then extract locally before upload
gzip -k ./finseo-access.log
```
```bash Apache theme={"system"}
# Debian/Ubuntu
cp /var/log/apache2/access.log ./finseo-access.log
# RHEL/CentOS/Amazon Linux
cp /var/log/httpd/access_log ./finseo-access.log
```
```bash Docker theme={"system"}
# Replace container_name with your web server container
docker logs container_name > finseo-access.log
```
For managed hosts, look for **Access logs**, **Raw logs**, **HTTP logs**, or **Web server logs** in your hosting dashboard. Download the unfiltered access log, not only error logs.
## Recommended time window
Upload the last `30` to `90` days first. That is usually enough to see crawler patterns and error rates. After the first upload, you can add new logs weekly or monthly.
## What Finseo extracts
For each detected bot request, Finseo stores:
* Bot platform and bot name
* IP address and User-Agent
* Request path and query parameters
* HTTP status code
* Response size
* Referrer
* Visit timestamp
The dashboard then groups this into:
* **Bot visits over time**: trend by bot and date.
* **Crawled pages**: URLs AI bots reached most often.
* **Performance**: response status and response-size patterns.
* **Status codes**: successful `2xx`, redirects `3xx`, client errors `4xx`, and server errors `5xx`.
The current log-upload parser focuses on crawl volume, URLs, response sizes, and HTTP status codes. It does not currently extract response-time fields from custom logs.
## Sync integrations
The **Sync** tab may show providers such as Cloudflare, Fastly, AWS CloudFront, or API sync as future ingestion options. Manual `.log` and `.txt` upload is the current production path.
## Troubleshooting
Check that the file includes User-Agent values and that the time window contains requests from known crawlers such as `ChatGPT-User`, `GPTBot`, `ClaudeBot`, `Claude-Web`, `PerplexityBot`, `Googlebot`, `Bingbot`, `Applebot`, `AhrefsBot`, or `SemrushBot`.
Make sure you exported the combined access log. A short Apache common log line usually lacks referrer and User-Agent fields, which are required for bot detection.
Finseo deduplicates visits by bot, timestamp, IP, and page path. Re-uploading overlapping files should not inflate your numbers.
Files above `50MB` can be processed in the background. Check back after a few minutes. If the status fails, split the log into smaller date ranges and upload again.