Technical SEO
Log File
A server log file is a text record generated by web servers that captures every HTTP request made to the server, including the requesting user agent (browser or bot), IP address, requested URL, response status code, timestamp, and bytes transferred. Log file analysis is one of the most reliable methods for understanding bot crawl behavior.
Log files are analyzed using specialized tools like Screaming Frog Log File Analyser, Botify, OnCrawl, and Excel/Python scripts. Analyzing Googlebot's activity in log files reveals crawl frequency, crawl depth, response codes served to bots, and pages being crawled but not targeted — information not fully available in Google Search Console.
Authority Links
Related Terms
Technical SEO
Crawl Budget
The number of URLs a crawler will fetch on a site within a given timeframe — a function of server capacity (crawl rate limit) and content popularity/freshness (crawl demand).
Technical SEO
Googlebot
Refers to the software developed by Google to create a searchable index and crawl the web.
Technical SEO
Robots.txt File
A plain-text file at the root of a domain that uses the Robots Exclusion Protocol to tell compliant crawlers which paths they may or may not request.
Technical SEO
Server Status Codes
Refers to the three-digit responses that a server gives to a browser's request.