Bots (SEO & Performance tools)

In most cases, bots that visit your website for indexing (such as Google, Bing, etc.) or performance evaluation (like Google PageSpeed Insights, GTmetrix, etc.) do not need to see the consent collection UI. In fact, displaying a consent notice to bots can negatively impact your SEO ranking or performance score, as bots cannot interact with consent banners and might index them as part of your site content.

The Didomi Web SDK includes built-in options to control how it behaves when a bot visits your website. You can configure it to bypass consent collection entirely and act “as if” consent had been given — for example, by loading all third-party vendors. This is recommended since bots are not human visitors and do not provide consent.

Default behavior: robots.txt blocks all bots

By default, Didomi serves a robots.txt file on sdk.privacy-center.org that completely prevents bots from crawling or loading any SDK files:

User-agent: *
Disallow: /

This configuration tells all compliant crawlers (for example: Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider, Twitterbot, LinkedInBot, Slackbot, FacebookExternalHit, AhrefsBot, SemrushBot, MJ12bot, DotBot, CensysInspect, PetalBot, Archive.org_bot, CommonCrawlBot) not to crawl or load any resources from the SDK domain.

As a result, the Didomi SDK is not loaded at all for bots that respect this file.

The configuration described below only applies to non-compliant or synthetic clients, such as headless browsers (Puppeteer, Playwright) or monitoring tools that do not follow robots.txt.

Allowing bots to load the SDK under a custom domain

If you want the configuration below to apply to all bots, including compliant ones, you can set up a custom domain for serving the Didomi SDK and configure that domain to allow bots in its own robots.txt. This can be done through the Didomi Console or via our API.

To indicate that consent should not be collected for bots, set the user.bots.consentRequired property to false in your SDK configuration:

<script type="text/javascript">
window.didomiConfig = {
  user: {
    bots: {
      /**
       * Indicate whether consent is required for bots.
       * Defaults to false when configuring the consent notice from
       * the Didomi console. Defaults to true otherwise.
       */
      consentRequired: false,
      
      /**
       * Predefined types of bots to identify
       * Defaults to all types supported by the SDK
       * (Optional)
       */
      types: ['crawlers', 'performance'],
    }
  }
};
</script>

The user.bots.types and user.bots.extraUserAgents properties give you extra control on what user agents are identified as bots. The Didomi SDK can automatically identify all the most common search engine bots and performance tools.

When configuring the consent notice from the Didomi console, consent is not collected for bots by default (consentRequired is set to false). This is usually the expected behavior, and it is thus unnecessary to use the above code.

Example using extraUserAgents

The SDK internally uses the RegExp object constructor to generate the needed regular expressions out of the provided extraUserAgents array list. Normal string escape rules apply, i.e, preceding special characters with the \ character, reference.

The following is a valid example using the extraUserAgents property:

<script type="text/javascript">
window.didomiConfig = {
  user: {
    bots: {
      /**
       * List of additional regular expressions to match
       * against the User Agent of the visitor.
       * When one of the regular expressions matches, the user
       * is considered to be a bot.
       * This allows you to identify extra bots that the SDK does not
       * know about.
       * Regular expressions must be specified as strings and correctly
       * escaped.
       * (Optional)
       */
      extraUserAgents: [
        "Mozilla\\/5.0 \\(Windows NT 10.0; Win64; x64; rv:88.0\\) Gecko\\/20100101 Firefox\\/88.0 \\(compatible; MonetoringBot\\/2.1\\)",
        "Mozilla\\/5.0 \\(compatible; SiteAnalyzerBot\\/5.0; \\+https:\\/\\/www.site-analyzer.com\\/\\)"
      ],
    }
  }
};
</script>

{
  "user": {
    "bots": {
      "extraUserAgents": [
        "Mozilla\\/5.0 \\(Windows NT 10.0; Win64; x64; rv:88.0\\) Gecko\\/20100101 Firefox\\/88.0 \\(compatible; MonetoringBot\\/2.1\\)",
        "Mozilla\\/5.0 \\(compatible; SiteAnalyzerBot\\/5.0; \\+https:\\/\\/www.site-analyzer.com\\/\\)"
      ]
    }
  }
}

Bots categories

By default and if you configure the SDK to not collect consent from bots, all bots are impacted. If you want to control what categories of bots require consent more specifically, the Didomi SDK categorizes bots in the following categories and allows you to enable only some of them: