Data Locker—raw data in a bucket

Data partition

AppsFlyer creates an AWS principal (ARN in Amazon terms) and generates credentials for that principal. A policy is then set allowing the principal to browse and retrieve files from their bucket.

In the bucket, data is organized by report type. The data for a given report is stored in its folder. 

Folder and file structure

• Folder structure is: af-ext-reports/<Home Folder>/data-locker-hourly/t=<event-type>/dt=<date YYYY-MM-dd>/h=<Hour h>
  • The Home Folder is the Home Folder that appears in the Credentials window (see the setup instruction in the prior section) 
  • For example, for the date 2016-08-12 the report appears in: s3://af-ext-reports/12345678911-acc-1abc234/data-locker-hourly/t=installs/dt=2016-08-12/
• The day folder dt=yyyy-mm-dd is split into 25 hourly folders. These folders represent the arrival hour of the event, not the event hour of the event itself. The folders are named h=0, h=1, h=2, and so on, up to h=23, and h=late. For example, the folder h=0 contains the events that arrive between 00:00 UTC and 01:00 UTC similarly, the folder h=20 contains the events that arrive between 20:00 and 21:00.

• In each folder:

  • Data is split into multiple files to avoid large files. File names are: part-00000, part-00001, part-00002, and so on. There can be up to 1000 files. We may increase this maximum number in the future without advance notice.

  • The last file to be written is an empty file named _SUCCESS. This file is a flag indicating that no further data will be written to the folder. As such, do not read data in a folder before verifying that the _SUCCESS file exists. Note: The _SUCCESS flag is also written in cases where there is no data to be written to the folder.

• Late folder

  • The late folder contains events of the preceding day that arrived after midnight. Meaning arrives between 00:00–02:00 UTC of the following day. For example, a user installs an app on Monday 08:00, the event arrives on Tuesday at 01:00. The event is recorded in Monday's late folder. 

  • The folder also contains the _SUCCESS flag as described in the previous section.

  • Data in the late folder are not recorded in any other folder.

  • Automated processes should look for data in the late folder similarly to that of other folder. 

File structure and format

• Data Locker files are based on Raw Data Reports V5 (see: Raw Data Reports V5).
• The actual data file is in CSV format but it has no file extension.
• The report files are zipped in .gz format.
• Each file has a header row.
• Values that have a coma in them are contained between double quotes `"`, for example `"iPhone6,1"`.

Using reports as data sources

You can use the data from the reports and add it to your own databases. To extract the data and add it to your databases you need to know the report format. Data Locker reports are based on Raw Data Reports. However, the final report format depends on the fields that you choose to include.

App-specific timezone and currency settings don't have an effect on data in Data Locker.

• Timezone: Data Locker reports use the UTC timezone
• Currency: The field event_revenue_usd is in USD. 

To setup Data Locker:

1. The admin needs to perform the setup. 
2. In Appsflyer, go to Integration > Data Locker. 
3. Select one or more or all apps.  Select all to automatically include apps you add in the future.
4. Click Apply. 
5. [optional] Media Sources (default=All. Means that media sources added in the future are automatically added.): Select one or more Media Sources to include in reports. 
6. Click Apply. 
7. Select one or more report types.
8. [optional] In-app events (default=All. Means that in-app events added in the future are automatically added.): Select the in-app events to include. If you have more than 100 in-app event types, you can't search for them. Enter their names exactly to select them. 
9. Click Apply. 
10. [optional] Fields (default=All): Select the fields to include in the reports. Note: We add fields from time to time take this into account in your data import process..
11. Click Create Bucket.
12. Click Save Configuration. 
    
    • If you selected AppsFLyer AWS bucket:
      • A dedicated AWS bucket is created. The bucket credentials display.
      • The bucket is accessible using the credentials. The credentials provide you with read-only access to the bucket.

Before you begin:

• Install the AWS CLI on your computer.
• In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel as it is needed to perform this procedure. 

To use AWS CLI:

1. Open the terminal. To do so in Windows, <Windows>+<R>, click OK.
   The command line window opens.
2. Enter aws configure
3. Enter the AWS Access Key as it appears in the credentials panel.
4. Enter your AWS Secret Key as it appears in the credentials panel.
5. Enter eu-west-1
6. Press Enter (None)

Use the CLI commands that follow as needed.

In the following commands, the value of {home-folder} can be found

To list folders in your bucket

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/

Listing files and folders

There are three types of folders in your Data Locker bucket:

• Report Type t=
• Date dt=
• Hour h=

To list all the reports of a specific report type:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/

To list all the reports of a specific report type for a specific day:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17

To list all the reports of a specific report, in a specific hour of a specific day:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17/h=23

To download files for a specific date:

aws s3 cp s3://af-ext-reports/<home-folder>/data-locker-hourly/t=installs/dt=2020-08-01/h=9/part-00000.gz ~/Downloads/

Before you begin:

• Install the Cyber Duck client.
• In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel. You will need this information when you configure Cyber Duck. 

To configure Cyber Duck:

1. In Cyber Duck, click Action.
2. Select New Bookmark. The window opens.
3. In the first field, (marked [1] in the screenshot that follows,) select Amazon S3.

   

4. Complete the fields as follows:
   • Nickname: free text
   • Server: s3.amazonaws.com
   • Access Key ID: copy the AWS Access Key as it appears in the credentials panel in AppFlyer
   • Secret Access Key: copy the Bucket Secret key as it appears in the credentials panel in AppsFlyer.
   • Path: {Bucket Name}/{Home Folder} For example: af-ext-reports/1234-abc-ffffffff
5. Close the window, to do so, use the X in the upper-right corner of the window.
6. Select the connection.
   The data directories are displayed.

Before you begin:

• Install the Amazon S3 Browser.
• In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel as it is needed to perform this procedure. 

To configure the Amazon S3 Browser:

1. In the S3 browser, Click Accounts > Add New Account.
   The Add New Account window opens.

   

2. Complete the fields as follows:
   • Account Name: free text. 
   • Access Key ID: copy the AWS Access Key as it appears in the credentials panel. 
   • Secret Access Key: copy the Bucket Secret key as it appears in the credentials panel.
   • Select Encrypt Access Keys with a password and enter a password. Make a note of this password.
   • Select Use secure transfer. 
3.  Click Save changes.
4. Click Buckets > Add External Bucket.
   The Add External Bucket window opens.
   

   

5. Enter the Bucket name. The Bucket name has the following format: {Bucket Name}/{Home Folder}. The values needed for bucket name and home folder appear in the credentials window. 
6. Click Add External bucket.
   The bucket is created and displays in the left panel of the window.
   You can now access the Data Locker files. 

• Symptom: Unable to retrieve data using AWS CLI
• Error message: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
• Cause: The AWS credentials being used not the correct credentials for the AppsFlyer bucket. This can be caused by having multiple or invalid credentials on your machine. 
• Solution:
  1. Use a different method, like Cyber Duck to access the bucket, meaning not the CLI. Do this to verify that the credentials you are using are working. If you are able to connect using Cyber Duck, this indicates an issue with the credentials cache. 
  2. Refresh the AWS credentials cache.
     Screenshot from AWS`