Finseo

Fastly CDN Log Access

How to set up and access real-time logs from Fastly CDN for AI bot analytics and visibility tracking.

About Fastly

Fastly is an edge cloud platform that provides content delivery network (CDN), edge computing, and security services. It offers powerful real-time log streaming capabilities, making it ideal for capturing detailed visitor data including AI bot traffic.

Enterprise CDN
Real-Time Logging
Edge Computing

Overview

Fastly provides comprehensive real-time log streaming that captures all visitor data, including AI bot traffic. Unlike traditional CDNs, Fastly allows instant configuration changes and provides immediate visibility into all requests, making it perfect for bot analytics.

Key Advantage

Fastly's real-time logging streams data as it happens, allowing you to monitor AI bot visits live and export historical data on demand. This immediate visibility is crucial for tracking AI search engine behavior.

Setting Up Real-Time Log Streaming

Follow these steps to configure Fastly to stream logs for your website:

1

Create a Fastly Service

Sign up for Fastly and create a new service for your domain:

  1. Log in to the Fastly control panel
  2. Click "Create Service"
  3. Add your domain name
  4. Configure your origin server (your website's hosting)
2

Configure Logging

In your Fastly service configuration:

  1. Navigate to the "Logging" section
  2. Click "Create Endpoint"
  3. Choose your preferred logging destination
  4. Configure the log format (ensure User-Agent and IP are included)
3

Update DNS Settings

Point your domain to Fastly:

Update your DNS records:

  • CNAME: www → global.prod.fastly.net
  • A: @ → Fastly Anycast IPs (provided in dashboard)
4

Activate Configuration

Click "Activate" to deploy your configuration. Fastly will immediately start routing traffic and logging all requests, including AI bot visits.

Configure Logging Endpoints

Fastly supports multiple logging endpoints. Choose the one that best fits your workflow:

Amazon S3

Most popular for long-term storage and batch processing:

  1. Create an S3 bucket for log storage
  2. Configure IAM permissions for Fastly
  3. Set up hourly or daily log rotation
  4. Download logs via AWS CLI or S3 console

Google Cloud Storage

Similar to S3 with Google Cloud integration:

  1. Create a GCS bucket
  2. Generate service account credentials
  3. Configure in Fastly logging settings
  4. Access via gsutil or Cloud Console

HTTPS Endpoint

Stream logs to your own server or service:

  1. Set up an HTTPS endpoint to receive logs
  2. Configure authentication (optional)
  3. Process logs in real-time
  4. Store or forward as needed

Syslog

Traditional logging for existing infrastructure:

  1. Configure your syslog server details
  2. Set up TLS for secure transmission
  3. Parse logs with your preferred tools
  4. Export filtered logs as needed

Recommended Log Format

Use this log format in Fastly to capture all necessary data for bot analytics:

{
  "timestamp": "%{begin:%Y-%m-%dT%H:%M:%S}t",
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "method": "%{req.method}V",
  "url": "%{req.url}V",
  "user_agent": "%{req.http.User-Agent}V",
  "status": "%{resp.status}V",
  "bytes": "%{resp.body_bytes_written}V",
  "time_elapsed": "%{time.elapsed.usec}V",
  "host": "%{req.http.host}V",
  "referer": "%{req.http.Referer}V"
}

Export Logs for Analysis

Once your logs are being collected, export them for Bot Analytics:

From Cloud Storage (S3/GCS)

  1. Access your storage bucket
  2. Download log files for your desired date range
  3. Combine multiple files if needed (they're usually hourly)
  4. Convert from JSON to standard log format if required
aws s3 sync s3://your-bucket/logs/ ./local-logs/

From HTTPS/Syslog Endpoints

Export from your log aggregation system:

  • Query logs for the required time period
  • Filter by your Fastly service if needed
  • Export as CSV or TXT format
  • Ensure User-Agent and IP fields are included
Upload to Bot Analytics

Best Practices

Log Retention: Configure appropriate retention periods in your storage endpoint. 30-90 days is typically sufficient for bot analytics.

Compression: Enable gzip compression for log files to reduce storage costs and transfer times.

Monitoring: Set up alerts for logging failures to ensure continuous data collection.

Testing: Use Fastly's real-time log tailing feature to verify logs are being captured correctly before relying on them for analytics.

Cost Management: Monitor your Fastly bandwidth usage and optimize caching rules to control costs while maintaining log accuracy.

Advanced Features

Edge Computing with Compute@Edge

For advanced users, Fastly's Compute@Edge allows custom log processing:

  • Filter and enrich logs at the edge before streaming
  • Add custom bot detection logic
  • Aggregate data in real-time
  • Route different log types to different endpoints

Next Steps

After setting up Fastly logging and uploading to Finseo Bot Analytics:

  • Monitor real-time AI bot traffic to your site
  • Analyze which content attracts the most AI crawler attention
  • Track bot behavior patterns and visit frequency
  • Optimize your content strategy for AI search visibility
  • Use Fastly's edge caching to improve bot crawl efficiency
Back to Bot Analytics OverviewFastly Documentation