Technical SEO
Googlebot
Googlebot is Google's web crawling bot (spider) responsible for discovering and fetching web pages to build Google's searchable index. It includes Googlebot Smartphone (simulates a mobile browser) and Googlebot Desktop, and uses a headless Chromium browser to render JavaScript-based pages. Googlebot identifies itself via a user-agent string in HTTP requests.
Googlebot activity is monitored through server log analysis using tools like Screaming Frog Log File Analyser, Botify, and OnCrawl. Google Search Console's Crawl Stats report shows how many requests Googlebot made, average response times, and any crawl anomalies over 90 days.
Authority Links
Related Terms
Technical SEO
Crawl Budget
The number of URLs a crawler will fetch on a site within a given timeframe — a function of server capacity (crawl rate limit) and content popularity/freshness (crawl demand).
Technical SEO
Crawling
Crawling refers to the process whereby bots systematically browse through a website.
Technical SEO
Robots.txt File
A plain-text file at the root of a domain that uses the Robots Exclusion Protocol to tell compliant crawlers which paths they may or may not request.
Technical SEO
User Agent
Refers to the robots.txt command for giving permissions to bots visiting a website.