GEO glossary

Robots.txt

Robots.txt is a site file declaring crawling permissions for compliant automated user agents.

Why this matters for GEO

Check that important public pages are not unintentionally blocked.

Related terms

• Canonical URL
• AI Crawler
• JSON-LD
• Web Indexing
• Answer-First Content
• Extractable Passage

Continue reading

• Canonical URL
• AI Crawler

All 150 GEO and AI visibility terms

Updated November 2026.