Log file analysis
Your server logs record every request search and AI bots make. Upload them and see how Google really spends its time on your site, and which of your pages it is ignoring.
What's included
- Bot breakdown for search engines and AI crawlers
- Most-crawled URLs and crawl activity wasted on errors and redirects
- Pages bots never visit, compared against your crawl
Download an access log from your hosting control panel and upload it (plain text or compressed, up to 50 MB at a time). Choose a date range and see how often each bot visits, which URLs it requests most, which responses it received and how much of its effort went on errors and redirects.
Run a crawl and the platform compares bot visits with the pages it knows about, so you can list the important pages that bots never reach. It also checks heavily crawled URLs, such as filters and search results, against your robots.txt.
How it works
The analysis reads Apache and Nginx access logs in the common and combined formats. Bots are identified by the name they announce, then summarised by day, bot, URL and response. Split very large logs into daily files to keep uploads quick.
Why look at logs
- See real crawl behaviour rather than guessing from a sitemap.
- Recover wasted crawl budget by fixing the errors and redirects bots keep hitting.
- Find ignored pages and give them the internal links they need.
What it does not do
Bots are identified by their announced name, which can be faked, so treat odd patterns with care. The analysis only covers the logs you upload; it does not connect to your server.
Works with the rest of the platform
Pair it with the full-site crawler, which supplies the list of known pages, and the technical checks for the fixes.
Frequently asked questions
Which log formats work?
Apache and Nginx combined and common formats. Split very large logs into daily files.
Can someone fake being Googlebot?
Yes. Bots are identified by their announced name, which can be spoofed, so treat odd patterns with care.
Do I need to give access to my server?
No. You download a log from your hosting panel and upload it.
Is there a size limit?
Up to 50 MB per upload, plain text or compressed.