Technical SEO
User Agent
In the context of robots.txt, the User-agent directive specifies which crawler or bot a set of rules applies to. For example, "User-agent: Googlebot" targets only Google's crawlers, while "User-agent: *" applies rules universally to all bots. In HTTP requests, the user-agent is a string identifying the browser or bot making the request.
User-agent configurations in robots.txt are tested using Google Search Console's robots.txt Tester. Screaming Frog can crawl a site while mimicking different user agents to identify content discrepancies served to bots versus users — a key test for detecting cloaking.
Authority Links
Related Terms
Technical SEO
Allow Command
Refers to the code for web pages that are to be crawled and indexed.
Technical SEO
Disallow Command
A command used for web pages that are not to be crawled and indexed by bots.
Technical SEO
Googlebot
Refers to the software developed by Google to create a searchable index and crawl the web.
Technical SEO
Robots.txt File
A plain-text file at the root of a domain that uses the Robots Exclusion Protocol to tell compliant crawlers which paths they may or may not request.