Robots.txt

AI crawlers like PerplexityBot, ChatGPTBot, and Googlebot-Extended visit your store every day to understand your products and content. If your robots.txt blocks them or your server rules reject them, your store is invisible to AI shopping agents. Opening the door is a five-minute configuration change that determines whether AI agents can recommend your products. What Are AI Crawlers? AI crawlers are automated programs sent by AI platforms to read and understand your website content. They serve the same purpose as traditional search engine crawlers like Googlebot but for a different audience: large language models and AI agents. ...

llms.txt is a plain text file you place at the root of your website that gives AI models a structured summary of what your site is about, what content exists, and how to navigate it. For ecommerce stores, it’s becoming the difference between AI agents understanding your catalog and guessing at it. What Is llms.txt? llms.txt is a proposed web standard, similar in concept to robots.txt but serving a different purpose. While robots.txt tells crawlers what not to access, llms.txt tells AI models what your site is and where the important content lives. ...