Robots.txt runs on three directives: User-agent picks the crawler, Disallow blocks paths, Sitemap points to your XML sitemap. Block carts, checkouts, and internal search — nothing else. And remember the distinction that trips everyone: Disallow stops crawling, not indexing. A blocked URL can still sit in Google's results.
Three Directives Do Everything
The file lives at yourdomain.com/robots.txt and crawlers fetch it before anything else. User-agent says who a group of rules applies to, and an asterisk means everyone. Disallow lists path prefixes that group may not crawl; Allow carves exceptions back out. Sitemap, on its own line, hands crawlers the location of your XML sitemap. That is the entire working vocabulary. Anything fancier you have seen, crawl-delay or wildcards mid-path, is either ignored by Google or a sign the site structure needs fixing instead.
Blocked Is Not Removed
The classic mistake deserves its own section. Disallow prevents crawling. It does not prevent indexing. If other pages link to a blocked URL, Google can index the bare address without ever fetching it, which is how you end up with listings that have no title or description — and how Search Console ends up warning "Indexed, though blocked by robots.txt." Worse, blocking a page you noindexed means Google can never crawl it to see the noindex. If you want a page out of search, let it be crawled and serve a noindex, or return a 404. Robots.txt is a crawl budget tool, not a removal tool.
A Sane File for a Dispensary Site
Worked example, four rules and a pointer: User-agent: * / Disallow: /cart/ / Disallow: /checkout/ / Disallow: /?s= / Sitemap: https://yourstore.com/sitemap.xml. Carts and checkouts have no business in search, and internal search results (/?s= on WordPress) generate infinite thin URLs. Note the trailing slashes, because Disallow matches by prefix. A careless Disallow: /c would block /cart/ — along with /cbd-gummies/, /concentrates/, and /contact. One character, three sections gone. Product and category pages never get blocked, and neither do CSS or JavaScript files, which Google needs to render your pages.
Test Before It Costs You
Search Console has a robots.txt report showing the file Google last fetched and whether it parsed. For any single URL you are unsure about, URL Inspection tells you flat out whether the page is crawlable and indexed. Check both after every edit, and any time a redesign or replatform ships — an astonishing number of sites go live with the developer's Disallow: / still in place, blocking everything. That one line has erased more search traffic than any algorithm update, and it takes about thirty seconds to catch if anyone thinks to look.
Key takeaways
- Three directives (User-agent, Disallow, Sitemap) cover everything a cannabis site needs.
- Disallow stops crawling, not indexing; blocked URLs can still appear in results.
- Never block a page that carries noindex, or Google cannot see the noindex.
- Disallow matches prefixes: /c blocks /cbd-gummies/ and /contact too.
- Verify with Search Console's robots.txt report and URL Inspection after every change and every relaunch.
Frequently asked questions
Can robots.txt remove a page from Google?
No. Disallow stops crawling, not indexing, and a blocked URL can stay in results indefinitely. To remove a page, let Google crawl it and see a noindex tag, return a 404 or 410, or use the Removals tool in Search Console for urgent cases.
Does my site even need a robots.txt file?
A small dispensary site with nothing to hide runs fine without one; no file means everything may be crawled. It is still worth creating for the Sitemap line alone, and to keep carts and internal search results out of the crawl.
Want this done for your brand?
We build cannabis & CBD search visibility with full transparency, for dispensaries, brands, and MSOs.
Get a free cannabis SEO audit ›