1. Help Center
  2. Reporting
  3. Raw data reporting/APIs

Using Data Locker—raw-data deposited into an S3 bucket

Avatar
Gray Goldstein
Last update:
  • Advertisers
  • Developers

At a glance: Data Locker deposits raw-data into an AWS S3 bucket in near-real-time (lag 6 hours.) The volume of data is unlimited. Data retention is 30 days. 

Data Locker

Data locker main features

  • Apps: supports multiple apps which can be automatically added as you add apps
  • Simplicity: data is deposited into an Amazon S3 bucket which manages the storage requirements
  • Reliability: data is stored in AWS which ensures data persistence
  • Flexibility: choose what data you want to include in the reports by field and in-app event
  • Granularity: data is segmented into report types, days and hours
  • Accessibility: pull data when required
  • Data freshness: 6-hour lag using or daily depending on the report type.  The lag-time is the same (6 hours) irrespective of the app-specific time zone. 
Category Report type (topic) Data freshness* Organic* Non-organic*
User acquisition clicks 6-hour lag - ✓+
Retargeting clicks_retargeting 6-hour lag - ✓+
User acquisition impressions 6-hour lag - ✓+
Retargeting impressions_retargeting 6-hour lag - ✓+
User acquisition installs 6-hour lag ✓ ✓
User acquisition inapps 6-hour lag ✓ ✓
User acquisition attributed_ad_revenue Daily* - ✓
User acquisition organic_ad_revenue Daily* ✓ -
Retargeting retargeting_ad_revenue Daily* - ✓
Retargeting conversions_retargeting 6-hour lag - ✓
Retargeting inapps_retargeting 6-hour lag - ✓
Retargeting retargeting_sessions 6-hour lag - ✓+
User acquisition sessions 6-hour lag ✓+ ✓+
User acquisition uninstalls 6-hour lag - ✓
User acquisition organic_uninstalls Daily ✓+ -
Protect360 blocked_installs 6-hour lag - ✓
Protect360 blocked_inapps 6-hour lag - ✓
Protect360 blocked_clicks 6-hour lag - ✓
Protect360 [FF*] [AG*] post_attribution_installs Daily - ✓
People-Based Attribution web_events Daily ✓+ ✓+
People-Based Attribution web_touch_points  Daily ✓+ ✓+
People-Based Attribution [FF*] web_to_app Daily ✓+ ✓+
People-Based Attribution [FF*] conversion-paths Daily ✓+ ✓+

* Key to abbreviations

✓+ reports unique to Data Locker

[FF] Report fields are fixed by Appsflyer. They are not related to the fields selected for inclusion in reports.

[AG] Agency transparency not supported.

6-hour lag: Data is separated into arrival hour folders. That is the hour that the event was deposited to Data Locker. Some data Locker folders are written about six hours after the actual event time for real-time events. There are 24 folders for each hour of the day, from 0 to 23, and an additional folder for data that arriving late. The lag time is the same irrespective of the app-specific timezone.

Daily: Reports having a data freshness rate of daily are written to the h=23 folder. These reports are typically available at 06:00-12:00 UTC in the h=23 folder of the preceding day. For example, the report for data generated during Monday is in the Monday H=23 folder. The data is available after 06:00 UTC on Tuesday.

Daily*: Ad revenue data is available after 2 days, meaning that data generated during Monday, becomes available in the Monday H=23 folder after 06:00 UTC on Wednesday.
Report types available in Data Locker

Data segmentation

DataLockerFolders_us-en.png

Data in the bucket is segmented into folders as follows:

  • t=topic
  • dt=date
  • h=hour

This means that for each report type, on a given day, the data is separated into separate folders by arrival hour and not by the data event time. For example, ../t=installs/dt=2019-01-17/ contains 25 folders. 

To understand the folder structure and how hourly segmentation works, see folder structure and format.

Implementing Data Locker

Configuring Data Locker

Prerequisite: The admin needs to configure Data Locker. Team members can view the settings. 

AppsFlyerAdmin_us-en.png To configure Data Locker the

  1. In Appsflyer, go to Integration > Data Locker. 
  2. Select one or more or all apps.  Select all to automatically include apps you add in the future.
  3. Click Apply. 
  4. [optional] Media Sources (default=All. Means that media sources added in the future are automatically added.): Select one or more Media Sources to include in reports. 
  5. Click Apply. 
  6. Select one or more report types.
  7. [optional] In-app events (default=All. Means that in-app events added in the future are automatically added.): Select the in-app events to include. If you have more than 100 in-app event types,  you can't search for them. Enter their names exactly to select them.  
  8. Click Apply. 
  9. [optional] Fields (default=All): Select the fields to include in the reports. Note: We add fields from time to time take this into account in your data import process..
  10. Click Create Bucket.

Bucket credentials

Once the configuration is saved, a dedicated AWS bucket is created. The bucket details appear at the top right-hand corner of the screen. They include the Bucket Name, Home Folder, and credentials for accessing data.

data-credentials.png

  • The bucket is accessible using the credentials.
  • Access to the bucket is audited. 
  • The bucket owner is AppsFlyer. We have read/write permission.
  • The the app owner has read permission. 

Data availability

  • Data is updated hourly after a six-hour delay
  • Each file includes the apps selected 
  • Retention: Files and folders are available for 30 days. After 30 days the data is deleted

Folder structure and format

  • Folder structure is: af-ext-reports/<Home Folder>/data-locker-hourly/t=<event-type>/dt=<date YYYY-MM-dd>/h=<Hour h>
  • The Home Folder is the Home Folder that appears in the Credentials window (see the setup instruction in the prior section) 
  • For example, for the date 2016-08-12 the relevant report appears under: s3://af-ext-reports/12345678911-acc-1abc234/data-locker-hourly/t=installs/dt=2016-08-12/
  • The folder dt=yyyy-mm-dd is split into 25 hourly folders. These folders represent the arrival hour of the event, not the event hour of the event itself. The folders are named h=0, h=1, h=2, and so on, up to h=23, and h=late. For example, the folder h=0 contains the events that arrive between 00:00 UTC and 01:00 UTC similarly, the folder h=20 contains the events that arrive between 20:00 and 21:00.

  • In each folder:

    • Data may is split into multiple files to avoid large files.  File names are: part-00000, part-00001, part-00002, and so on. There can be up to 1000 files. We may increase this maximum number in the future without advance notice.

    • The last file to be written is an empty file named _SUCCESS. This file is a flag indicating that no further data will be written to the folder. As such, do not read data in a folder before verifying that the _SUCCESS file exists. Note: The _SUCCESS flag is also written in cases where there is no data to be written to the folder. 

Late folder

The Late folder contains events of the preceding day that arrived after 0000 UTC +0 (midnight) midnight and up to 02:00 UTC +0. It also contains the _SUCCESS flag as described in the previous section. Automated processes should look for data in the late folder as is done for all the other folders of the day. 

 Example

An event is received by AppsFlyer on January 21st at 1:15 AM. The event has a timestamp of January 20th at 18:45. Because this event arrived late, it will be placed in the late folder inside the /dt=2019-20-01/ h=late folder.

File structure and format

  • Data Locker files are based on Raw Data Reports V5 (see: Raw Data Reports V5).
  • The actual data file is in CSV format but it has no file extension.
  • The report files are zipped in .gz format (to make the download process efficient).
  • Each file has a header row.
  • Values that have a coma in them are contained between double quotes `"`, for example `"iPhone6,1"`.

Retrieving data from Data Locker

AppsFlyer creates an AWS principle (ARN in Amazon terms) and generates credentials for that principle. A policy is then set allowing the principle to browse and retrieve files from the bucket.

You can access the bucket using AWS command-line tools and most FTP clients. To use these tools, retrieve the credentials, AWS Access Key and AWS Secret, from the Credentials section.

Data can be accessed using the following tools, amongst others:

AWS CLI

Before you begin:

  • Install the AWS CLI on your computer.
  • In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel as it is needed to perform this procedure. 

To use AWS CLI:

  1. Open the terminal. To do so in Windows, <Windows>+<R>, click OK.
    The command line window opens.
  2. Enter aws configure
  3. Enter the AWS Access Key as it appears in the credentials panel.
  4. Enter your AWS Secret Key as it appears in the credentials panel.
  5. Enter eu-west-1
  6. Press Enter (None)

Use the CLI commands that follow as needed.

In the following commands, the value of {home-folder} can be found

To list folders in your bucket

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/

Listing files and folders

There are three types of folders in your Data Locker bucket:

  • Report Type t=
  • Date dt=
  • Hour h=

To list all the reports of a specific report type:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/

To list all the reports of a specific report type for a specific day:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17

To list all the reports of a specific report, in a specific hour of a specific day:

aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17/h=23

To download files for a specific date:

aws s3 cp s3://af-ext-reports/<home-folder>/data-locker-hourly/t=installs/dt=2020-08-01/h=9/part-00000.gz ~/Downloads/

Cyber Duck

Before you begin:

  • Install the Cyber Duck client.
  • In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel. You will need this information when you configure Cyber Duck. 

To configure Cyber Duck:

  1. In Cyber Duck, click Action.
  2. Select New Bookmark. The window opens.
  3. In the first field, (marked [1] in the screenshot that follows,) select Amazon S3.

    DataDuckSmall2.png

  4. Complete the fields as follows:
    • Nickname: free text
    • Server: s3.amazonaws.com
    • Access Key ID: copy the AWS Access Key as it appears in the credentials panel in AppFlyer
    • Secret Access Key: copy the Bucket Secret key as it appears in the credentials panel in AppsFlyer.
    • Path: {Bucket Name}/{Home Folder} For example: af-ext-reports/1234-abc-ffffffff
  5. Close the window, to do so, use the X in the upper-right corner of the window.
  6. Select the connection.
    The data directories are displayed.

Amazon S3 browser

Before you begin:

  • Install the Amazon S3 Browser.
  • In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel as it is needed to perform this procedure. 

To configure the Amazon S3 Browser:

  1. In the S3 browser, Click Accounts > Add New Account.
    The Add New Account window opens.

    mceclip0.png

  2. Complete the fields as follows:
    • Account Name: free text. 
    • Access Key ID: copy the AWS Access Key as it appears in the credentials panel. 
    • Secret Access Key: copy the Bucket Secret key as it appears in the credentials panel.
    • Select Encrypt Access Keys with a password and enter a password. Make a note of this password.
    • Select Use secure transfer. 
  3.  Click Save changes.
  4. Click Buckets > Add External Bucket.
    The Add External Bucket window opens.

    mceclip2.png

  5. Enter the Bucket name. The Bucket name has the following format: {Bucket Name}/{Home Folder}. The values needed for bucket name and home folder appear in the credentials window. 
  6. Click Add External bucket.
    The bucket is created and displays in the left panel of the window.
    You can now access the Data Locker files. 

Report format and folders

Using reports as data sources

You can use the data from the reports and add it to your own databases. To extract the data and add it to your databases you need to know the report format. Data Locker reports are based on Raw Data Reports. However, the final report format depends on the fields that you choose to include.

Report format

The fields available in Data Locker are listed in the data field dictionary V5.0

 Tip

The reports contain data that you can use for campaign optimization and retargeting.

Hourly reports

Data Locker separates data into hourly folders. The hourly folder represents the processing hour and not the hour that the event occurred. The data is written to Data Locker within six hours of processing. 

 Example

AppsFlyer receives data for activity between 14:00 and 15:00 on January 17, 2019. At some time after 15:00 AppsFlyer begins processing the data. Due to processing, the data is not written to Data Locker immediately. Thus the data in folder /t=installs/dt=2019-17-01/h=14 is not available on January 17th, 2019 at 15:00 but rather six hours later.

Creating hourly folders

In cases where there is no data for a specific hour, Data Locker creates a folder for that hour. This is to indicate to you that there was no data in that hour. The folder will contain a `_SUCCESS` file which indicates that AppsFlyer has completed writing to this folder. When designing automated processes this should be taken into account, meaning design your data retrieval processes so that they can handle empty hourly folders.

Timezone and currency

App-specific timezone and currency settings don't have an effect on data in Data Locker.

TImezone: Data Locker reports use the UTC timezone

Currency: The field event_revenue_usd is in USD. 

Traits and Limitations

Traits
Trait Remarks 
Ad networks Not for use by ad networks. 
Agencies Not for use by agencies
App-specific time zone Not Applicable. Data locker folders are divided into hours using UTC. The actual events contain times in UTC. Convert the times to any other time zone as needed. Irrespective of your app time-zone the lag from event occurrence until it is recorded in Data Locker remains the same; that is 6 hours. 
App-specific currency  Not supported
Size limitations Not applicable
Data freshness Files are updated hourly with a lag of six hours from the event time.
Historical data Not supported. Event data is sent after configuring Data Locker. If you need historical data use Pull API. 
Team member access Team members cannot configure Data Locker. 
Single app/multiple app Multi app support. Data locker is at the account level

Troubleshooting

  • Symptom: Unable to retrieve data using AWS CLI
  • Error message: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
  • Cause: The AWS credentials being used not the correct credentials for the AppsFlyer bucket. This can be caused by having multiple or invalid credentials on your machine. 
  • Solution:
    1. Use a different method, like Cyber Duck to access the bucket, meaning not the CLI. Do this to verify that the credentials you are using are working. If you are able to connect using Cyber Duck, this indicates an issue with the credentials cache. 
    2. Refresh the AWS credentials cache.
      Screenshot from AWS`mceclip0.png 
Was this article helpful?

Articles in this section

Related articles