Search crawl handoff / rule decisions

Decide robots.txt allow and disallow rules

Separate the target bot, target path and directive. robots.txt is not access control and does not guarantee removal from search results.

View robots.txt Save the decision sheet

Separate four decisions

The same Disallow can mean different things when the bot or path changes. Keep robots decisions separate from work another owner must do.

State	Meaning	Decision	Next action
allow	Allow matches the declared bot/path	Record crawl rule	Share with owner
disallow	Disallow matches the declared bot/path	Record crawl rule	Recheck public scope
not-controlled	robots is not the mechanism for the goal	Hand off elsewhere	Ask SEO/app owner
unknown	Bot, path or directive is unreadable	Do not finalize	Recheck the source

Review four synthetic cases

Examples cover a Googlebot disallow for /private/, a wildcard temporary path, an allowed public asset path and a search query that robots alone does not control. The fixture is not a live-site audit.

User-agent: Googlebot
Disallow: /private/
→ disallow

Bot and path match the rule.

User-agent: Bingbot
/search?query=
→ not controlled by robots alone

Another control or owner is needed.

Handoff sequence

Fix the target bot, target path and desired state.
Compare the robots directive with the declared path.
Record allow/disallow as a crawl rule, separate from access control.
Hand off removal, authentication and indexing goals to the SEO/app owner.
Do not call an unknown row resolved or configured.

This guide and its fixtures are static files. They do not fetch a site or change configuration.

What robots.txt does not guarantee

robots.txt is not authentication or secrecy and does not guarantee search-result removal, current crawler activity or index state. Confirm the page-level setting or owner process for the actual goal.