Technical SEO
How to Edit Shopify Robots.txt Safely (With Examples)
By Shayan Ahmad · · 6 min read

For years, Shopify store owners could not change their robots.txt file at all. Now you can, through a theme template called robots.txt.liquid. That flexibility is useful, but it also means a single wrong line can hide large parts of your store from Google.
This guide explains what robots.txt does, when it is actually worth customising on Shopify, what you should never block, and how to test your changes before they cause problems. It is written for store owners and the developers who work with them.
What Robots.txt Does (and Does Not Do)
Robots.txt is a plain text file at yourstore.com/robots.txt. It gives instructions to search engine crawlers about which parts of your site they may or may not fetch.
Two things trip people up:
- Robots.txt controls crawling, not indexing. Blocking a URL stops Google fetching it, but Google can still list that URL in search results if other pages link to it, usually without a description.
- Blocking a page hides its noindex tag. If you want a page removed from Google's index, you usually need a noindex tag, and Google has to be able to crawl the page to see it. Blocking it in robots.txt at the same time can stop the noindex from working.
In short, robots.txt is a tool for managing where crawlers spend their time, not for hiding pages.
Shopify's Default Robots.txt Is Already Sensible
Before you edit anything, open your own store's robots.txt in a browser and read it. Shopify's default file already blocks many areas that should not be crawled, including:
- The admin area, cart and checkout.
- Customer account pages.
- Internal search results pages.
- Many sorting and filtering URL patterns on collections.
- Combinations of multiple tags on collection and blog pages.
It also points crawlers to your sitemap. For most stores, the default file is perfectly adequate. Shopify itself recommends leaving it alone unless you have a specific reason. I agree: the majority of stores I audit do not need a custom robots.txt at all.
When Customising Makes Sense
There are a few situations where editing it is worth considering:
- A filtering app creates lots of crawlable URLs. Some third-party filter apps generate URL patterns that the default rules do not cover. If Search Console shows Google crawling thousands of filtered variations, a targeted rule can help.
- An app creates low-value pages. Some wishlist, comparison or review apps create pages that add nothing for searchers.
- You want to manage specific crawlers. For example, some store owners want to decide whether particular AI crawlers can access their content. That is a business decision with trade-offs, which our guide on getting your business cited by AI search covers in more detail.
- You need to unblock something. Rarely, a default rule blocks a URL pattern you genuinely want crawled.
If your reason is not on this list, think hard before editing. A proper SEO audit will usually show whether crawling is a real problem on your store before you change anything.
What You Should Never Block
I have seen each of these mistakes on real stores. Any of them can quietly damage your rankings:
- /products/ or /collections/. These are your money pages. Blocking them, even partially, can remove them from search results over time.
- CSS and JavaScript files. Google needs these to render your pages and understand how they look on mobile.
- Images on the Shopify CDN. Blocking them hurts image search and can affect how Google understands product pages.
- Your sitemap. Keep the sitemap line in place.
- Pages you are trying to noindex. As explained above, blocking them prevents Google from seeing the noindex tag.
- Everything, with a single slash. A line reading Disallow: / under the general crawler group blocks your entire store. It sounds obvious, but it happens, often left over from a development site.
How to Edit Robots.txt on Shopify
Shopify does not let you upload a robots.txt file directly. Instead you create a template in your theme.
- Duplicate your live theme first, so you have a backup.
- In your admin, go to Online Store > Themes, open the menu on your live theme and choose Edit code.
- Under Templates, choose to add a new template and select robots.txt. This creates robots.txt.liquid, pre-filled with code that outputs Shopify's default rules.
- Add your custom rules within that code rather than deleting it. Keeping the default rules in place means you still benefit from future updates Shopify makes to them.
- Save, then visit yourstore.com/robots.txt to see the result.
Because the template is part of your theme, remember that switching to a different theme later means recreating the customisation in the new theme. Keep a note of any changes you make.
Examples of Safe Customisations
The exact code depends on your theme and the URL patterns involved, so treat these as illustrations of the approach rather than code to paste blindly.
Example 1: Blocking an app's filter parameter
Suppose a filter app adds a parameter such as ?pf_ to collection URLs, creating thousands of variations. The approach is to add one extra Disallow rule for that specific pattern to the general crawler group, inside the default loop. You are blocking one precise pattern, not whole sections of the site.
Example 2: Blocking a low-value app page
If an app creates pages under a path like /apps/wishlist/ that you do not want crawled, you can add a rule for that path. Again, the key is that it is narrow and you know exactly what it matches.
Example 3: Rules for a specific crawler
To give instructions to one named crawler, you add a separate group after the default rules, starting with that crawler's user-agent name and followed by its rules. This leaves the rules for Google and other crawlers untouched.
How to Test Changes Before They Hurt You
Testing is the part most people skip. Do not.
- Read the live file. After saving, open yourstore.com/robots.txt and read every line. Check that the default rules and sitemap line are still there.
- Test specific URLs. Pick a product URL, a collection URL, and an example of the URL you meant to block. Work through each against your rules and confirm only the intended one is blocked.
- Use Search Console's robots.txt report. Found in Settings, it shows the version Google last fetched and flags any errors.
- Inspect a product page. Use URL Inspection on a key product to confirm it is still crawlable.
- Watch the Pages report for a few weeks. A sudden rise in "Blocked by robots.txt" for URLs you care about is your early warning.
Making robots.txt changes as part of a wider review is safer than editing in isolation. That is why it sits within our audit-first process rather than being a standalone quick fix.
FAQs
Can I delete Shopify's default robots.txt rules?
Technically yes, but it is rarely a good idea. The default rules block areas like checkout and internal search that should not be crawled. It is safer to add to them than to remove them.
How quickly does Google notice robots.txt changes?
Google typically fetches robots.txt frequently and usually uses a cached copy for up to about a day. The effect on crawling of individual pages can take longer to show in Search Console.
Will robots.txt remove pages from Google?
Not reliably. To remove a page from search results, use a noindex tag and keep the page crawlable, or delete it and let it return a 404. Robots.txt alone only stops crawling.
Does robots.txt matter for AI search tools?
Some AI crawlers say they respect robots.txt rules. Whether to allow them depends on whether you want your content to be used and cited by those tools. It is worth deciding deliberately rather than by accident.
Get Your Crawl Setup Checked
Robots.txt mistakes are easy to make and slow to notice. If you have edited yours, or inherited a store where someone else did, our technical SEO team can check it alongside the rest of your setup. Request a free SEO audit and we will flag anything that is blocking pages you need.
- shopify robots.txt
- crawling
- technical seo
- search console
- ai crawlers
Have a question about this for your own site?
A 15-minute call is usually enough to tell whether it applies to you and what to do first.


