managing robots txt file for sfcc b2c commerce

Managing Robots.txt Across Salesforce B2C Commerce Environments

by Ichwan Chandra

Introduction

In the world of e-commerce, visibility is everything. But what happens when search engines index the wrong version of our site? Imagine a potential customer clicking a Google link, only to land on an unstable sandbox with test-mode payment gateways and an incomplete product catalog. Not only does this create a poor user experience, but it can also destroy trust and cost you real revenue.

The gatekeeper preventing this chaos is a simple text file: robots.txt. While it looks basic, managing this file across Salesforce B2C Commerce environments (Development, Staging, Sandboxes, and Production) is a critical discipline that many teams overlook to ensure only our production site appears in search results.

The Anatomy of a Robots.txt File

Before diving into environment management, let’s review the syntax. The robots.txt file lives at the root of your domain (e.g., yourdomain.com/robots.txt). It tells web crawlers (like Googlebot, Bingbot, or Yandex) which parts of our site they can or cannot access. Here is a standard example:

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
  • User-agent: Specifies which crawler the rule applies to (* means all crawlers).
  • Disallow: Tells the bot not to crawl a specific path.
  • Allow: Grants access to a path (often used to override a broader Disallow).
  • Sitemap: The location of your XML sitemap, helping bots find important pages.

The Impact: Leaking Sandbox Environments

In Salesforce B2C Commerce, we typically have a primary group containing Development, Staging, and Production instances, plus on-demand Sandboxes. The transcript highlights a shocking reality: a simple Google search for dx.commercecloud.salesforce.com reveals dozens of live yet index-able sandboxes. Why this is dangerous?

  1. Test Mode Exposure: Sandboxes often have payment gateways in test mode. A customer trying to buy will hit an error.
  2. Content Leaks: Unreleased campaigns, internal navigation, or half-finished copy become publicly visible.
  3. Duplicate Content: Search engines penalize duplicate content. If our staging site mirrors production, our SEO ranking suffers.

Step-by-Step Configuration in Business Manager

Here is how to configure robots.txt directly in Salesforce B2C Commerce without requiring a full code deployment.

Step 1: Navigate to the SEO Settings

Go to Merchant Tools > SEO > Robots. Note that robots.txt is defined per site (e.g., RefArch).

Step 2: Choose Our Strategy

  1. Allow all robots: (User-agent: * Disallow:) – Best for Production.
  2. Block all robots: (User-agent: * Disallow: /) – Best for Staging, Dev, and Sandboxes.
  3. Custom directives: Paste your own rules (up to 50,000 characters). Recommended for Production when we need granular control.

Step 3: Implement the Safe Defaults

  • For Production: Select Custom and paste our official rules (e.g., blocking admin paths, allowing product pages).
  • For all other environments: Select Block all robots. This ensures no search engine indexes our test systems.

Step 4: Flush the Static Cache

After applying changes, visit https://yourdomain.com/robots.txt. If we don’t see our updates immediately, we must flush the static cache in Business Manager. Without this, the old cached file will persist.

The “Online Protected” Caveat

If our site is password-protected—set via Administration > Sites > Manage Sites > [Site] > Site Status—Salesforce automatically generates a robots.txt that blocks all bots. This is convenient but risky.

The Warning: If we ever disable password protection for testing or an integration, Salesforce stops generating that file. Our previous SEO Robots settings will instantly become active. If those settings were “Allow all” on a staging site, they will be indexed immediately. Always keep a manual “Block all” configuration as a backup.

Deployment: From Staging to Production

A common point of confusion is where to configure the production settings. We do not log directly into Production to edit the robots.txt. Instead, follow this pattern:

  1. Log into our Staging instance.
  2. Navigate to the SEO Robots settings.
  3. Define the rules for both Staging (Block all) and Production (Custom).
  4. Rely on data replication. The system preferences for each site replicate to Production automatically.

If we prefer to store the file in our code base (cartridge), place it under static/default/robots.txt. However, this requires a full deployment (code review, tests, replication) for every change. For quick fixes, editing directly in Business Manager is far more efficient.

Key Takeaways

  • The Risk Is Real: Unprotected staging or sandbox environments are frequently indexed by Google, leading to test checkouts and broken campaigns appearing in public search results.
  • Syntax Simplicity: The robots.txt relies on three core directives: User-agent (which bot), Disallow (block), and Allow (permit), plus Sitemap (location of your XML sitemap).
  • Environment Strategy: Production should use a custom robots.txt for granular control. All non-production environments (Staging, Dev, Sandboxes) should be set to block all robots.
  • Password Protection Override: If a site is set to “Online Protected” in Business Manager, Salesforce auto-generates a blocking robots.txt. However, if we remove that password, our manual SEO settings immediately take over.
  • Deployment via Data Replication: robots.txt settings are configured in the staging instance but pushed to production via system preferences and data replication—not just code deployment.

Conclusion

Managing robots.txt across multiple Salesforce B2C environments is not just an SEO “nice-to-have”—it is a technical necessity. A single miss-configured sandbox can lead to lost sales, confused customers, and damaged brand reputation.

Our action plan:

  1. Audit immediately: Google search our Salesforce subdomains to see if any sandboxes are exposed.
  2. Set strict defaults: Block all crawlers on every non-production environment.
  3. Document overrides: If we use “Online Protected” mode, document what happens when the password is removed.
  4. Test the cache: Always verify robots.txt live after flushing static cache.

By treating robots.txt as a critical environment control—not an afterthought—we ensure that when customers search for our site, they always land on the real, production-ready store, ready to buy.