In the world of e-commerce, visibility is everything. But what happens when search engines index the wrong version of our site? Imagine a potential customer clicking a Google link, only to land on an unstable sandbox with test-mode payment gateways and an incomplete product catalog. Not only does this create a poor user experience, but it can also destroy trust and cost you real revenue.<\/p>\n\n\n\n
The gatekeeper preventing this chaos is a simple text file: Before diving into environment management, let\u2019s review the syntax. The In Salesforce B2C Commerce, we typically have a primary group containing Development, Staging, and Production instances, plus on-demand Sandboxes. The transcript highlights a shocking reality: a simple Google search for Here is how to configure Go to Merchant Tools > SEO > Robots<\/strong>. Note that After applying changes, visit If our site is password-protected\u2014set via Administration > Sites > Manage Sites > [Site] > Site Status<\/strong>\u2014Salesforce automatically generates a The Warning:<\/strong> If we ever disable password protection for testing or an integration, Salesforce stops generating that file. Our previous SEO Robots settings will instantly become active. If those settings were “Allow all” on a staging site, they will be indexed immediately. Always keep a manual “Block all” configuration as a backup.<\/strong><\/p>\n\n\n\n A common point of confusion is where<\/em> to configure the production settings. We do not log directly into Production to edit the If we prefer to store the file in our code base (cartridge), place it under <\/p>\n\n\n\n Managing Our action plan:<\/strong><\/p>\n\n\n\n By treating In the world of e-commerce, visibility is everything. But what happens when search engines index the wrong version of our site? Imagine a potential customer clicking a Google link, only to land on an unstable sandbox with test-mode payment gateways and an incomplete product catalog. Not only does this create a poor user experience, but it can also destroy trust and cost you real revenue.<\/p>\n","protected":false},"author":1,"featured_media":2446,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20,13,8],"tags":[],"class_list":["post-2388","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-backend","category-ecommerce","category-technology"],"yoast_head":"\nrobots.txt<\/code>. While it looks basic, managing this file across Salesforce B2C Commerce<\/strong> environments (Development, Staging, Sandboxes, and Production) is a critical discipline that many teams overlook to ensure only our production site appears in search results.<\/p>\n\n\n\nThe Anatomy of a Robots.txt File<\/h2>\n\n\n\n
robots.txt<\/code> file lives at the root of your domain (e.g., yourdomain.com\/robots.txt<\/code>). It tells web crawlers (like Googlebot, Bingbot, or Yandex) which parts of our site they can or cannot access. Here is a standard example:<\/p>\n\n\n\nUser-agent: Googlebot\nDisallow: \/nogooglebot\/\n\nUser-agent: *\nAllow: \/\n\nSitemap: https:\/\/yourdomain.com\/sitemap.xml<\/pre>\n\n\n\n
\n
*<\/code> means all crawlers).<\/li>\n\n\n\nDisallow<\/code>).<\/li>\n\n\n\nThe Impact: Leaking Sandbox Environments<\/h2>\n\n\n\n
dx.commercecloud.salesforce.com<\/code> reveals dozens of live yet index-able sandboxes. Why this is dangerous?<\/p>\n\n\n\n\n
Step-by-Step Configuration in Business Manager<\/h2>\n\n\n\n
robots.txt<\/code> directly in Salesforce B2C Commerce without requiring a full code deployment.<\/p>\n\n\n\nStep 1: Navigate to the SEO Settings<\/h3>\n\n\n\n
robots.txt<\/code> is defined per site<\/strong> (e.g., RefArch).<\/p>\n\n\n\nStep 2: Choose Our Strategy<\/h3>\n\n\n\n
\n
User-agent: * Disallow:<\/code>) \u2013 Best for Production<\/strong>.<\/li>\n\n\n\nUser-agent: * Disallow: \/<\/code>) \u2013 Best for Staging, Dev, and Sandboxes<\/strong>.<\/li>\n\n\n\n
<\/li>\n<\/ol>\n\n\n\nStep 3: Implement the Safe Defaults<\/h3>\n\n\n\n
\n
Step 4: Flush the Static Cache<\/h3>\n\n\n\n
https:\/\/yourdomain.com\/robots.txt<\/code>. If we don’t see our updates immediately, we must flush the static cache<\/strong> in Business Manager. Without this, the old cached file will persist.<\/p>\n\n\n\nThe “Online Protected” Caveat<\/h2>\n\n\n\n
robots.txt<\/code> that blocks all bots. This is convenient but risky.<\/p>\n\n\n\nDeployment: From Staging to Production<\/h2>\n\n\n\n
robots.txt<\/code>. Instead, follow this pattern:<\/p>\n\n\n\n\n
static\/default\/robots.txt<\/code>. However, this requires a full deployment (code review, tests, replication) for every change. For quick fixes, editing directly in Business Manager is far more efficient.<\/p>\n\n\n\nKey Takeaways<\/h2>\n\n\n\n
\n
robots.txt<\/code> relies on three core directives: User-agent<\/code> (which bot), Disallow<\/code> (block), and Allow<\/code> (permit), plus Sitemap<\/code> (location of your XML sitemap).<\/li>\n\n\n\nrobots.txt<\/code> for granular control. All non-production environments (Staging, Dev, Sandboxes) should be set to block all robots<\/strong>.<\/li>\n\n\n\nrobots.txt<\/code>. However, if we remove that password, our manual SEO settings immediately take over.<\/li>\n\n\n\nrobots.txt<\/code> settings are configured in the staging instance but pushed to production via system preferences and data replication\u2014not just code deployment.<\/li>\n<\/ul>\n\n\n\nConclusion<\/h2>\n\n\n\n
robots.txt<\/code> across multiple Salesforce B2C environments is not just an SEO “nice-to-have”\u2014it is a technical necessity. A single miss-configured sandbox can lead to lost sales, confused customers, and damaged brand reputation.<\/p>\n\n\n\n\n
robots.txt<\/code> live after flushing static cache.<\/li>\n<\/ol>\n\n\n\nrobots.txt<\/code> as a critical environment control\u2014not an afterthought\u2014we ensure that when customers search for our site, they always land on the real, production-ready store, ready to buy.<\/p>\n","protected":false},"excerpt":{"rendered":"