Batch export

Didomi can export daily batches of user consent data to provide you with a dump of the users, events, and proofs stored on the platform.

The Batch export is not yet compatible with a multi-regulation approach. At this moment in time, the user consent will be associated with GDPR in the historical data to which you have access.

Destinations

Didomi can send exported data to various locations:

• AWS S3 Bucket (owned by Didomi)

• GCP Storage Bucket

Types of data

Didomi can export different types of consents data depending on the product and business requirements:

• Types of data

• Exporting CMP consents

Logging and monitoring

Didomi offers the ability to retrieve information and monitor the status of your batch exports. Read more on logging.

Format

Data exported at the destination follows the Apache Hive naming conventions for partitioning and exposes 2 partitions:

Partition name	Description
export-id	Unique ID for the configured batch export. This allows you to distinguish files exported between multiple configured exports.
date	The date/hour when the export happened (YYYY-MM-DD format).

Example of a path to the exported data: /data/export-id={batch export ID}/date={YYYY-MM-DD}/

Users & Events

Structure

Didomi exports user-level data in a users/ folder and in files formatted as newline-delimited JSON (one JSON object per line), and compressed as GZIP. We use the file extension .json.gz.

Example: /data/export-id={batch export ID}/date={YYYY-MM-DD}/users/

For every partition, the exported data can be split up into multiple files. Make sure that you read every .json.gz file in the partition folder to get all the records belonging to that export. File names are subject to change over time so make sure that your read files from their extension (.json.gz) and not from a specific filename or format. Additionally, a file named _SUCCESS is created once the export is complete. Do not read files from a partition until the _SUCCESS file is created as you might read partial records otherwise.

Content

The files exported by Didomi contain users and their events for the period covered by the export. Every line of the files contains a self-sufficient JSON record (newline-delimited JSON) that includes both the user and its associated events.

You should first split the files on new lines and then parse every line as an individual JSON record. It is a pretty standard encoding for data processing tools like Hive, Spark, Presto, etc. and should be straightforward to import in your existing tools.

The schema of the objects exported is as follows:

{
  /**
   * User object following the User schema
   * https://developers.didomi.io/api/consents/users#user-schema
   */
  "user": {
    ...
  },
  
  /**
   * Array of event objects following the Event schema
   * https://developers.didomi.io/api/consents/events#event-schema
   */
  "events": [
    { ... },
    { ... },
    ...
  ]
}

Example of an exported file:

{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[e-mail]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[e-mail]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}

Proofs

Structure

Didomi exports proofs in a proofs/ folder as a proofs.tar.gz archive.

Example: /data/export-id={batch export ID}/date={YYYY-MM-DD}/proofs/

For every export, the proofs file is split up into files of up to 4GB with the naming convention proofs.tar.gz.XY where XY is a number indicating the split number. For instance, if the total size of the proofs is 12GB, three files will be generated (proofs.tar.gz.00, proofs.tar.gz.01, proofs.tar.gz.02).

To be able to extract the proofs, you need to join the different files together to generate the original proofs.tar.gz archive:

# Join all parts of the proofs archive together
cat proofs.tar.gz.* > proofs.tar.gz

# Extract the proofs
tar -xvf proofs.tar.gz

Additionally, a file named _SUCCESS is created once the export is complete. Do not read files from a partition until that file is created as you might get partial records.

Content

The files exported by Didomi contain proofs for the period covered by the export. Every file is named after the proof ID that is present in the consent events.

Configuration

Read our dedicated tutorial on configuring a batch export.