Technical SEO
Allow Command
The Allow command is a directive used in a robots.txt file to explicitly permit search engine bots to crawl specific URLs or directories, even within a section that has been broadly disallowed. It overrides the Disallow directive for the specified paths and is supported by major bots including Googlebot.
Webmasters configure Allow rules in their robots.txt file to fine-tune crawl access. Tools such as Google Search Console's robots.txt tester and Screaming Frog can validate that Allow directives are correctly interpreted by crawlers.
Authority Links
Related Terms
Technical SEO
Disallow Command
A command used for web pages that are not to be crawled and indexed by bots.
Technical SEO
Robots.txt File
A plain-text file at the root of a domain that uses the Robots Exclusion Protocol to tell compliant crawlers which paths they may or may not request.
Technical SEO
Crawling
Crawling refers to the process whereby bots systematically browse through a website.
Technical SEO
Crawl Budget
The number of URLs a crawler will fetch on a site within a given timeframe — a function of server capacity (crawl rate limit) and content popularity/freshness (crawl demand).