Analytics export

The Analytics Data Export provides a way to access analytics data (aggregated data) collected through the Didomi platform at once, for CMP. Analytics Data Export is a premium feature.

Destinations

• AWS S3 Bucket

• GCP Storage Bucket

Logging and monitoring

Didomi offers the ability to retrieve information and monitor the status of your Analytics export. Read more on logging here.

Format

Data exported at the destination follows the Apache Hive naming conventions for partitioning and exposes 2 partitions:

Partition name	Description
export-id	Unique ID for the configured batch export. This allows you to distinguish files exported between multiple configured exports.
date	The date/hour when the export happened (YYYY-MM-DD format).

Example of a path to the exported data: /data/export-id={batch export ID}/date={YYYY-MM-DD}/

CMP Hourly Metrics

Structure

Didomi exports CMP hourly metrics data in a /cmp_hourly_metrics folder and in files formatted as newline-delimited JSON (one line = one JSON object) and compressed as GZIP. We use the file extension json.gz.

Example: /data/export-id={export-id}/date={YYYY-MM-DD}/cmp_hourly_metrics

For every partition, the exported data can be split up into multiple files. Make sure that you read every .json.gz file in the partition folder to get all the records belonging to that export.

File names are subject to change over time so make sure that your read files from their extension (.json.gz) and not from a specific filename or format.

Additionally, a file named _SUCCESS is created once the export is complete. Do not read files from a partition until the _SUCCESS file is created as you might read partial records otherwise.

Content

The files exported by Didomi contain CMP hourly metrics for the period covered by the export. Every line of the files contains a self-sufficient JSON record (newline-delimited JSON) that includes CMP hourly metrics.

You should first split the files on new lines and then parse every line as an individual JSON record. It is a standard encoding for data processing tools like Hive, Spark, Presto, etc. and should be straightforward to import in your existing tools.

The schema of the objects exported is as follows:

Column Name	Example	Data Type	Description
window_start	06/07/21 21:00	datetime	Day and hour of the data collected (YYYY-MM-DD-HH)
apikey	e23a01f6-a508-4e71-8f50-c1a9cae7c0d0	Text	API key of the organization
device_os	iOS	Text	OS info are regrouped under the following families: Windows, GNU/Linux, Mac, Android, iOS, Android TV, tvOS, Fire TV and other.
device_type	desktop	Text	Type of device: desktop, smartphone, phablet, tablet, television and other.
deployment_id		Text	Version ID of the PUBLISHED notice. Named production_config_id of deployment_id in the API.
domain	www.journaldugolf.fr	Text	subdomain or app ID where the consent is collected
experiment_group	control or test	Text	Only filled when a AB test is running. One test is the test group the other one is the control group
experiment_id		Text	Only filled when a AB test is running ID of the test. Defined by the client
sdk_type	sdk-web	Text	Type of SDK: sdk-mobile, sdk-app, sdk-ctv or sdk-amp
consents_asked	21	integer	Number of consent asked (from consent.asked event - sampled)
consents_given	21	integer	Number of consent given (from consent.given event - not sampled)
consents_given_denied_click	21	integer	Number of denied consent given (from consent.given event - not sampled) - click
consents_given_denied_external	21	integer	Number of denied consent given (from consent.given event - not sampled) - external
consents_given_denied_other	21	integer	Number of denied consent given (from consent.given event - not sampled) - do not fall into the other categories
consents_given_denied_webview	21	integer	Number of denied consent given (from consent.given event - not sampled) - webview
consents_given_empty	21	integer	Number of consent given (from consent.given event - not sampled)
consents_given_invalid	21	integer	Number of invalid consent given (from consent.given event - not sampled)
consents_given_positive_click	21	integer	Number of positive consent given (from consent.given event - not sampled) - click
consents_given_positive_external	21	integer	Number of positive consent given (from consent.given event - not sampled) - external
consents_given_positive_navigate	21	integer	Number of positive consent given (from consent.given event - not sampled) - navigate
consents_given_positive_other	21	integer	Number of positive consent given (from consent.given event - not sampled) - do not fall into the other categories
consents_given_positive_scroll	21	integer	Number of positive consent given (from consent.given event - not sampled) - scroll
consents_given_positive_webview	21	integer	Number of positive consent given (from consent.given event - not sampled) - webview
pageviews	21	integer	Number of pageviews/app sessions (from pageview events - sampled)
pageviews_with_denied_consent	21	integer	Number of denied consent pageviews/app sessions (from pageview events - sampled)
pageviews_with_empty_consent	21	integer	Number of empty consent pageviews/app sessions (from pageview events - sampled)
pageviews_with_invalid_consent	21	integer	Number of invalid consent pageviews/app sessions (from pageview events - sampled)
pageviews_with_positive_consent	21	integer	Number of positive consent pageviews/app sessions (from pageview events - sampled)
median_lead_time	not supported anymore

Example of an exported file:

{"window_start":"2022-01-24T08:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"io.didomi.sampleuitest","sdk_type":"sdk-ctv","device_type":"television","device_os":"Android TV","experiment_id":"","experiment_group":"","deployment_id":"6ExpWCHh","consents_asked":5.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":0.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":0.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T14:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"sandbox.app.domain","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"RDkUQWNK","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":1.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":1.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T15:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"localhost:8086","sdk_type":"sdk-web","device_type":"desktop","device_os":"Windows","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":20.0,"pageviews_with_positive_consent":20.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":0.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T08:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-fix-pcp-12-df4ht3--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":1.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":1.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T09:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-fix-pcp-12-df4ht3--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":16.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":16.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T10:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-fix-pcp-12-df4ht3--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":1.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":1.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T11:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-feat-updat-lh42zm--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":2.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":2.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T13:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-feat-updat-lh42zm--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":3.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":3.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T14:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-fix-pcp-12-df4ht3--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":1.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":1.0,"pageviews_with_invalid_consent":0.0}
{"window_start":"2022-01-24T17:00:00.000Z","apikey":"7685b6f7-3062-491b-ba50-207f440951dc","domain":"review-feat-updat-lh42zm--pmp-didomi.netlify.app","sdk_type":"sdk-web","device_type":"desktop","device_os":"Mac","experiment_id":"","experiment_group":"","deployment_id":"","consents_asked":0.0,"consents_given":0.0,"consents_given_positive_other":0.0,"consents_given_denied_other":0.0,"consents_given_positive_scroll":0.0,"consents_given_positive_navigate":0.0,"consents_given_positive_click":0.0,"consents_given_denied_click":0.0,"consents_given_positive_external":0.0,"consents_given_denied_external":0.0,"consents_given_positive_webview":0.0,"consents_given_denied_webview":0.0,"consents_given_empty":0.0,"consents_given_invalid":0.0,"pageviews":1.0,"pageviews_with_positive_consent":0.0,"pageviews_with_denied_consent":0.0,"pageviews_with_empty_consent":1.0,"pageviews_with_invalid_consent":0.0}

Configuration

Please read our dedicated tutorial on configuring an analytics batch export