Configure a batch export

Tutorial for configuring a batch export integration.

Batch export is a premium feature that is not enabled for organizations by default. Make sure you reach out to [email protected] to enable this integration before configuring one for your organization.

Introduction

Batch exports consist of two main components:

• Properties define the type of data to export as well as the frequency for delivering the data.

• Destinations define the cloud storage (bucket) that will receive the data and the credentials to connect to them.

Configure a batch export

To configure a batch export, send a POST /marketplace/integrations request with the type of destination to use, the credentials, data type, and the ID of the organization that will host it.

AWS S3

For AWS S3 Bucket (owned by Didomi) destinations, you need to specify your AWS account ID in the destination config:

POST https://api.didomi.io/v1/marketplace/integrations?organization_id=YOUR_ORG_ID

BODY
{
    "type_id": "batch-exports-s3",
    "authentication": {
        "account_id": "0000000000"
    },
    "properties": {
        "frequency": 1,
        "data": ["users"]
    },
    "mappings": []
}

The API will return something similar to:

{
    "id": "58g9gnu3b5",
    "organization_id": "didomi",
    "type_id":  "batch-exports-s3",
    "authentication": {
        "account_id": "0000000000"
    },
    "properties": {
        "frequency": 1,
        "data": ["users"],
        "bucket_name": "my-didomi-bucket", // <-------- S3 Bucket name
        "bucket_access_role_arn": "arn:aws:0000000000:iam:role/my-didomi-bucket-role" // <---- IAM Role
    },
    "mappings": []
}

The destination's bucket_name is the name of the created AWS S3 bucket where your Didomi data will be delivered. Assuming the AWS IAM role returned as bucket_access_role_arn is required to get the correct credentials to read the created AWS S3 bucket.

GCP storage

If you are configuring a GCP Storage Bucket destination instead, you must set the type to batch-exports-gcp and specify the GCP bucket name, your project ID, and the JSON key in the config payload:

POST https://api.didomi.io/v1/marketplace/integrations?organization_id=YOUR_ORG_ID

BODY
{
    "type_id": "batch-exports-gcp",
    "authentication": {
        "project_id": "didomi-cmp",
        "json_key": "{\"type\":\"service_account\",\"project_id\":\"didomi-cmp\"..."
    },
    "properties": {
        "frequency": 1,
        "data": ["users"],
        "bucket_name": "didomi-cmp-bucket"
    },
    "mappings": []
}

The current version of the Didomi export configurations only support daily incremental exports (1 day frequency).

This will schedule a daily batch export that will deliver the delta updates of the organization's consent data to the configured destination. The first batch will be delivered 24 hours after the creation date of the export configuration.

For more information on batch exports, please visit the Batch export documentation.