Batch export
Didomi can export batches of user consent data to provide you with a full dump of the users, events, and proofs stored on the platform.
The Batch export is not yet compatible with a multi-regulation approach. In the history you have access, the user consent will be considered as GDPR consent.
Didomi can send exported data to various locations:
Didomi can export different types of consents data depending on the product and business requirements:
Didomi offers the ability to retrieve information and monitor the status of your batch exports. Read more on logging.
Data exported at the destination follows the Apache Hive naming conventions for partitioning and exposes 2 partitions:
Partition name | Description |
export-id | Unique ID for the configured batch export. This allows you to distinguish files exported between multiple configured exports. |
date | The date/hour when the export happened (YYYY-MM-DD format). |
Example of a path to the exported data:
/data/export-id={batch export ID}/date={YYYY-MM-DD}/Didomi exports user-level data in a
users/ folder and in files formatted as newline-delimited JSON (one JSON object per line), and compressed as GZIP. We use the file extension .json.gz.Example:
/data/export-id={batch export ID}/date={YYYY-MM-DD}/users/For every partition, the exported data can be split up into multiple files. Make sure that you read every
.json.gz file in the partition folder to get all the records belonging to that export. File names are subject to change over time so make sure that your read files from their extension (.json.gz) and not from a specific filename or format.
Additionally, a file named _SUCCESS is created once the export is complete. Do not read files from a partition until the _SUCCESS file is created as you might read partial records otherwise.The files exported by Didomi contain users and their events for the period covered by the export. Every line of the files contains a self-sufficient JSON record (newline-delimited JSON) that includes both the user and its associated events.
You should first split the files on new lines and then parse every line as an individual JSON record. It is a pretty standard encoding for data processing tools like Hive, Spark, Presto, etc. and should be straightforward to import in your existing tools.
The schema of the objects exported is as follows:
{
/**
* User object following the User schema
* https://developers.didomi.io/api/consents/users#user-schema
*/
"user": {
...
},
/**
* Array of event objects following the Event schema
* https://developers.didomi.io/api/consents/events#event-schema
*/
"events": [
{ ... },
{ ... },
...
]
}
Example of an exported file:
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
{"user":{"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}},"metadata":{},"organization_user_id":"[email protected]","organization_user_id_type":null,"version":1,"organization_id":"organization_id","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa","created_at":"2019-11-05T16:24:34.450Z","updated_at":null},"events":[{"id":"747f7907-facb-4ddf-811f-e3eaa1df1d21","organization_id":"organization_id","created_at":"2019-11-05T16:24:34.450Z","user":{"organization_user_id":"[email protected]","id":"00019ba4-1561-4d55-a8ae-b8b1d203aeaa"},"source":{},"consents":{"purposes":[{"id":"OeWK1234","enabled":true,"preferences":[{"id":"commercial_communication","enabled":true,"channels":[{"id":"email","enabled":true}]}]}],"vendors":{"enabled":[],"disabled":[]}}}]}
Didomi exports proofs in a
proofs/ folder as a proofs.tar.gz archive.Example:
/data/export-id={batch export ID}/date={YYYY-MM-DD}/proofs/For every export, the proofs file is split up into files of up to 4GB with the naming convention
proofs.tar.gz.XY where XY is a number indicating the split number.
For instance, if the total size of the proofs is 12GB, three files will be generated (proofs.tar.gz.00, proofs.tar.gz.01, proofs.tar.gz.02).To be able to extract the proofs, you need to join the different files together to generate the original
proofs.tar.gz archive:# Join all parts of the proofs archive together
cat proofs.tar.gz.* > proofs.tar.gz
# Extract the proofs
tar -xvf proofs.tar.gz
Additionally, a file named
_SUCCESS is created once the export is complete. Do not read files from a partition until that file is created as you might get partial records.The files exported by Didomi contain proofs for the period covered by the export. Every file is named after the proof ID that is present in the consent events.
Last modified 24d ago