Data Locker for marketers

Reports available via Data Locker

The structure of your data in storage depends if the data is sent to cloud storage or a data warehouse. The folder structure described here applies to storage (buckets). In the case of data warehouse storage,  consider that the reference to folders applies to views. See Snowflake. 

Data is written to your selected storage option. In the case of cloud storage, the storage is owned by AppsFlyer on AWS or owned by you on AWS, GCS, or Yandex. You can switch storage options at any time or send some or all of your data to multiple storage options. 

Data in the cloud bucket storage is organized in a hierarchical folder structure, according to report type, date, and time. The following figure contains an example of this structure:

Data of a given report is contained in the hour (h) folders associated with that report:

• The number of hour folders depends on the report data freshness (hourly, daily or versioned).
• Data is provided in Snappy or GZIP compressed files, or uncompressed files, having Parquet or CSV format.
• Data files consist of columns (fields). 
• The schema (field) structure of the user journey reports is identical to each other and depends on the fields selected by you. Other reports each have their own explicit fields, AKA schemaless reports. See Data Locker marketer reports for the reports available and links to the report specifications. 

Data files depend on segregation type.

• Data is written to a storage owner of your choice as follows:
  • AppsFlyer storage
  • Customer storage—AWS, GCS, Azure, Yandex, BigQuery, and Snowflake
• You can change the storage selection at any time.
• If you change the storage, the following happens:
  • We start writing to the newly selected storage within one hour.
  • We continue writing to the existing storage during a transition period of 7 days. The transition period expiry time displays in the user interface. Use the transition period to update your data loading processes. You can restart the transition period or revert to the AppsFlyer bucket if needed. 
  • Changing storage: You can migrate from one storage option to another by using the multi-storage option and sending data to multiple destinations simultaneously. Once you have completed the migration and testing, delete the storage option you no longer need. 

Notice to security officers in the case of customer-controlled storage

Consider:

• The bucket or destination is for the sole use of AppsFlyer. There should be no other entity writing to a given destination.
• You can delete data in the destination 25 hours after we write the data.
• Data written to the destination is a copy of data already in our servers. The data continues to be in our servers in accordance with our retention policy. 
• For technical reasons, we sometimes delete and rewrite the data. For this reason, we need delete and list permissions. Neither permissions are a security risk for you. In the case of list, we are the sole entity writing to the bucket. In the case of delete, we are able to regenerate the data.
• For additional information, you can contact our security team via hello@appsflyer.com or your CSM.  

In Data Locker you can send some or all of your data to 2 destinations (defined in the connection settings). For example, you can send App A data to AWS, and App B data to GCS.

Each connection consists of a complete set of Data Locker settings, including a destination. Connection settings are independent of one another.

In managing your connections, consider:

• In Data Locker settings, connections are shown in tabs. Each connection has its own settings tab from which you can manage the connection. The icon of each tab represents the storage type.
• To see connection details, duplicate a connection, or delete a connection, click ⋮ (options).

Both admins and team members, with the correct permissions, can access Data Locker.

Admins

Admins can access the Data Locker page, create and manage all connections, add editors, and assign owners to existing connections.

Team members

Team members can access the Data Locker page, edit existing connections that they own, or create new connections.

Providing permissions

• To provide a team member permission to access Data Locker, assign them a role with Data Locker set to 'Manage'.
• To transfer ownership or add a team member as an editor on an existing connection, click on the three-dot options menu within an existing connection, and then Manage ownership to either change the connection owner or add editors.

Use this procedure to set up Data Locker. Any changes to Data Locker settings take up to 3 hours to take effect. 

Prerequisites

To set up marketer-owned storage:

If you are setting up Data Locker using a marketer-owned cloud storage service, complete one or more of the following procedures now. 

• Set up:
  • AWS bucket
  • GCS bucket
  • [Beta] Azure Blob
  • [Beta] Yandex bucket
  • BigQuery data warehouse
  • Snowflake warehouse

Note! If you don't have a Data Locker subscription and you access Cohorts analytics or SKAN data, you must still complete a marketer-owned cloud storage service procedure. 

To set up Data Locker:

1. An admin needs to perform the setup. 
2. In AppsFlyer, go to Integration > Data Locker. 
3. [Optional] If you already have an active Data Locker destination and want to add a destination, click Add connection. Name your connection.
4. Select a cloud service data destination. Do one of the following:
   
   • Select AppsFlyer AWS bucket (option available to Data Locker subscribers only). Click Save and continue to step 5. 
   • Select S3.
     1. Enter your AWS S3 bucket name. af- prefix is mandatory, and should be entered manually.
     2. Click Test connection.
     3. Verify that an error message indicating that the bucket path is invalid isn't displayed.
     4. Select whether to Make this connection compatible with Adobe Experience Platform. If selected, click Save and continue to Step 6.
     5. Click Save.
   • Select GCS.
     1. Enter your GCS bucket name.
     2. Click Test connection.
     3. Verify that an error message indicating that the bucket path is invalid isn't displayed.
     4. Select whether to Make this connection compatible with Adobe Experience Platform. If selected, click Save and continue to Step 6.
     5. Click Save.
   • [Beta] Select Azure.
     1. Enter your Connection name, Storage account name, and Key.
     2. Verify that an error message indicating that the bucket path is invalid isn't displayed.
     3. Select whether to Make this connection compatible with Adobe Experience Platform. If selected, click Save and continue to Step 6.
     4. Click Save.
   • [Beta] Select Yandex.
     1. Enter your Bucket name, Access key, and Secret key.
     2. Verify that an error message indicating that the bucket path is invalid isn't displayed.
     3. Select whether to Make this connection compatible with Adobe Experience Platform. If selected, click Save and continue to Step 6.
     4. Click Save.
   • Select BigQuery.
     1. Enter your BigQuery project ID and dataset name.
     2. Click Test connection.
     3. Verify that an error message indicating that the bucket path is invalid isn't displayed.
     4. Click Save and continue to Step 6.
   • Select Snowflake.
     1. Enter your Snowflake region and account ID.
     2. Click Test connection.
     3. Verify that an error message indicating that the bucket path is invalid isn't displayed.
     4. Click Save and continue to Step 6.
5. Complete the Data settings section:
   1. Select the file format you want:
      • [Default] Parquet
      • CSV
   2. Select the file compression type you want:
      • Snappy (only available for Parquet files)
      • GZIP
      • Uncompressed
   3. Select the max rows you want in your file: Either 10k, 25k, 50, 100k, 200k, or 500k. More rows in the file means fewer files but a larger file size.
   4. Select folder structure (data segregation):
      • [Default] Unified
      • Segregated by app
6. Complete the Data Locker content section:
7. Select one or more Apps to include in the reports. Select all to automatically include apps added in the future. 
8. Click Apply. 
9. [Optional] Select one or more Media Sources to include in reports.
   • Default=All. This means that media sources added in the future are automatically added.
10. Click Apply. 
11. [Optional] Select the Fields to include in reports. Note: Sometimes we make additional fields available. Take this into account in your data import process.
12. Click Apply. 
13. Select the report types. You must select at least 1. 
14. [Optional] For the In-app events report, select the in-app events to include. If you have more than 100 in-app event types, you can't search for them. Enter their names exactly to select them. 
    • Default=All. This means that in-app events added in the future are automatically added.
15. Click Apply. 
16. Click Save connection. One of the following occurs:
    
    • If you selected AppsFlyer AWS bucket:
      • A dedicated AWS bucket is created. The bucket credentials display.
      • The bucket is accessible using the credentials. The credentials provide you with read-only access to the bucket.
    • If you selected one of your cloud storage services: Data will be written to your service within 3 hours. 

An admin can reset the AppsFlyer bucket credentials at any time. Note! If you reset the credentials, you must update your data import scripts with the updated credentials.

To reset the credentials of AppsFlyer owned storage:

1. In AppsFlyer, go to Integration > Data Locker. 
2. Select the AppsFlyer-owned destination.
3. In the Credentials section, click Reset credentials.
   A confirmation window displays.
4. Click Reset.
5. Wait (about 20 seconds) until the Credentials successfully reset message displays.
   The updated credentials are available.

• Symptom: Unable to retrieve data using AWS CLI
• Error message: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
• Cause: The AWS credentials being used are not the correct credentials for the AppsFlyer bucket. This can be caused by having multiple or invalid credentials on your machine. 
• Solution:
  1. Use a different method, like Cyberduck to access the bucket, meaning not the CLI. Do this to verify that the credentials you are using are working. If you are able to connect using Cyberduck, this indicates an issue with the credentials cache. 
  2. Refresh the AWS credentials cache.
     Screenshot from AWS

     

Before you begin:

• Install the AWS CLI on your computer.
• In AppsFlyer, go to Data Locker, and retrieve the information contained in the credentials panel.

To use AWS CLI:

1. Open the terminal. To do so in Windows, <Windows>+<R>, click OK.
   The command line window opens.
2. Enter aws configure.
3. Enter the AWS Access Key as it appears in the credentials panel.
4. Enter your AWS Secret Key as it appears in the credentials panel.
5. Enter eu-west-1.
6. Press Enter (None).

Use the CLI commands that follow as needed.

In the following commands, the value of {home-folder} can be found

To list folders in your bucket:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/

Listing files and folders

There are three types of folders in your Data Locker bucket:

• Report Type t=
• Date dt=
• Hour h=

To list all the reports of a specific report type:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/

To list all the reports of a specific report type for a specific day:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17

To list all the reports of a specific report, in a specific hour of a specific day:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17/h=23

To download files for a specific date:

aws s3 cp s3://af-ext-reports/<home-folder>/data-locker-hourly/t=installs/dt=2020-08-01/h=9/part-00000.gz ~/Downloads/

Before you begin:

• Install the Cyberduck client.
• In AppsFlyer, go to Data Locker and retrieve the information contained in the credentials panel.

To configure Cyberduck:

1. In Cyberduck, click Action.
2. Select New Bookmark. The window opens.
3. In the first field (marked [1] in the screenshot below) select Amazon S3.

   

4. Complete the fields as follows:
   • Nickname: Free text
   • Server: s3.amazonaws.com
   • Access Key ID: Copy the AWS Access Key as it appears in the credentials panel in AppsFlyer
   • Secret Access Key: Copy the Bucket Secret key as it appears in the credentials panel in AppsFlyer.
   • Path: {Bucket Name}/{Home Folder} For example: af-ext-reports/1234-abc-ffffffff
5. Close the window. To do so, click the X in the upper-right corner of the window.
6. Select the connection.
   The data directories are displayed.

Before you begin:

• Install the Amazon S3 Browser.
• In AppsFlyer, go to Data Locker and retrieve the information contained in the credentials panel.

To configure the Amazon S3 Browser:

1. In the S3 browser, Click Accounts > Add New Account.
   The Add New Account window opens.

   

2. Complete the fields as follows:
   • Account Name: free text. 
   • Access Key ID: copy the AWS Access Key as it appears in the credentials panel. 
   • Secret Access Key: copy the Bucket Secret key as it appears in the credentials panel.
   • Select Encrypt Access Keys with a password and enter a password. Make a note of this password.
   • Select Use secure transfer. 
3.  Click Save changes.
4. Click Buckets > Add External Bucket.
   The Add External Bucket window opens.
   

   

5. Enter the Bucket name. The Bucket name has the following format: {Bucket Name}/{Home Folder}. The values needed for bucket name and home folder appear in the credentials window. 
6. Click Add External bucket.
   The bucket is created and displays in the left panel of the window.
   You can now access the Data Locker files.