Data Locker is a robust solution for storing and extracting raw data. For customers with large amounts of data, Data Locker is an alternative to exporting data by using Pull API. Customers can choose to view and extract raw data by report type, days, and hours. You can use automated scripts to pull and process the data, import it into your BI systems or make it available on demand.
Data Locker, an AppsFlyer premium solution
For information about selecting the best data APIs for you
Benefits of using Data Locker
- Coverage: the reports can contain data about more than one application
- Simplicity: we manage data storage requirements through AWS
- Reliability: data is stored in AWS which ensures data persistence
- Flexibility: choose what data you want to include in the reports
- Granularity: data is segmented into report types, days and hours
- Unique data: get more data such as Organic Installs, In-App Events, Sessions, Clicks, and Impressions
- Accessibility: pull data when required
Data segmentation
Data in Data Locker is segmented into report types, days and hours. Meaning that for each report type, on a given day, the data is separated into separate folders by arrival hour and not by the data event time.
The structure of the folder relating to a specific day, for example, the folder path /t=installs/dt=2019-01-17/ contains 25 folders. There are 24 folders for each hour of the day, from 0 to 23 and an additional folder for data that arrives late.
- To understand the folder structure and how hourly segmentation works, see listing files and folders.
- For an explanation on the Late folder, click here.
Data freshness: In Data Locker data is separated into hourly folders. The data is not updated in real-time but typically with a delay of six hours. To learn more, click here.
Configuring Data Locker
Prerequisite: The account admin shall configure data locker.
To configure Data Locker:
- In Appsflyer, go to Integration > Data Locker
- Select one or more apps.
- Click Apply.
- (optional) Media Sources (default=All): Select one or more Media Sources to include in the reports.
- Click Apply.
- Select the report type: you can select Acquisition, Retargeting and Protect 360 data.
- Acquisition: Clicks, Impressions, Installs, In-App Events, Sessions, Uninstalls
- Retargeting: Retargeting Clicks, Retargeting Conversions, Retargeting In-App Events
- Protect 360: Blocked Installs, Blocked In-App Events, Blocked Clicks
- People Based Attribution: Web Conversions (if People-Based Attribution is enabled in your account). For People-Based Attribution, data locker aggregates data daily in one report. This report can be found in Data Locker in the 23rd-hour folder. For example t=web_touch_points/dt=2019-07-19/h=23.
Protect 360 is an AppsFlyer premium solution.
- (optional) In-app events (default=All) Select the in-app events to include .
- Click Apply.
- (optional) Fields (default=All): Select the fields to include in the reports.
Note:
AppsFlyer may add new fields without prior notice. If your parsing process does not support field additions, manually select the required fields. - (optional) Recipients Email list of people to notify when reports are ready. To add more than one recipient, sperate the emails using a comma, for example, user1@example.com, user2@example.com.
- Click Create Bucket.
Bucket credentials
Once the configuration is saved, a dedicated AWS bucket is created. The bucket details appear at the top right-hand corner of the screen. They include the Bucket Name, Home Folder, and credentials for accessing data.
The bucket is only accessible using customer credentials (for security reasons). In addition, all access to the bucket is audited. Note: The above configuration is for Data Locker 2.0. Data Locker 1.0.
Technical details
Permission to configure Data Locker
- Only account admins can configure Data Locker.
Data availability
- We run an hourly batch job that populates the bucket with the relevant report data.
- Each file includes all of the selected apps in the Data Locker configuration.
- Files in Data Locker are available for 30 days.
- Each folder is available for 30 days.
- Data (files and folders) is deleted after 30 days.
Folder structure and format
- Folder structure is: af-ext-reports/<Home Folder>/data-locker-hourly/t=<event-type>/dt=<date YYYY-MM-dd>/h=<Hour h>
- The Home Folder is the Home Folder that appears in the Credentials window (see step 8 in the setup instruction above).
- For example, for the date 2016-08-12 the relevant report appears under:
s3://af-ext-reports/12345678911-acc-1abc234/data-locker-hourly/t=installs/dt=2016-08-12/ - The folder dt=yyyy-mm-dd is split into 25 hourly folders. These folders represent the arrival hour of the event, not the event hour itself. The folder are named h=0, h=1, h=2,...,h=23, and h=late. For example, the folder h=0 contains the events that arrive between 00:00 and 01:00, similarly, the folder h=20 contains the events that arrive between 19:00 and 20:00.
-
In each folder, the data is split between multiple files to avoid large files. Depending on the type of data exported, folders contain up to 1000 files (this number can change without notice). Files are named `part-00000`, `part-00001`, etc.
- In each folder, after all the files are successfully written to that file, an empty file named `_SUCCESS` is generated. This file acts as a flag to indicate that writing to the folder has completed. You should check for the existence of this _SUCCESS flag before reading the data in the folder. Note: There will always be a _SUCCESS flag in a folder even if that folder contains no data files.
Late folder
The Late folder contains events of the preceding day that arrived after 0000 UTC +0 (midnight) midnight and up to 02:00 UTC +0. It also contains the _SUCCESS flag as described in the previous section. Automated processes should look for data in the late folder as is done for all the other folders of the day.
Example
An event is received by AppsFlyer on January 21st at 1:15 AM. The event has a timestamp of January 20th at 18:45. Because this event arrived late, it will be placed in the late folder inside the /dt=2019-20-01/ h=late folder.
File structure and format
- Data Locker files are based on Raw Data Reports V5 (see: Raw Data Reports V5).
- The actual data file is in CSV format but it has no file extension.
- The report files are zipped in .gz format (to make the download process efficient).
- Each file has a header row.
- Values that have a coma in them are contained between double quotes `"`, for example
`"iPhone6,1"`.
Accessing the data
AppsFlyer creates an AWS principle (ARN in Amazon terms) and generates credentials for that principle. A policy is then set allowing the principle to browse and retrieve files from the bucket.
You can access the bucket using AWS command line tools and most FTP clients. To use these tools, retrieve the credentials, AWS Access Key and AWS Secret, from the Credentials section.
Data can be accessed using the following tools, amongst others, as follows:
- AWS CLI
- Amazon S3 Browser should be used on windows. DragonDisk is not fully supported.
Configuring the Amazon S3 browser
Before you begin:
- Install the Amazon S3 Browser.
- In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel as it is needed to perform this procedure.
To configure the Amazon S3 Browser:
- In the S3 browser, Click Accounts > Add New Account.
The Add New Account window opens.
- Complete the fields as follows:
- Account Name: free text.
- Access Key ID: copy the AWS Access Key as it appears in the credentials panel.
- Secret Access Key: copy the Bucket Secret key as it appears in the credentials panel.
- Select Encrypt Access Keys with a password and enter a password. Make a note of this password.
- Select Use secure transfer.
- Click Save changes.
- Click Buckets > Add External Bucket.
The Add External Bucket window opens.
- Enter the Bucket name. The Bucket name has the following format: {Bucket Name}/{Home Folder}. The values needed for bucket name and home folder appear in the credentials window.
- Click Add External bucket.
The bucket is created and displays in the left panel of the window.
You can now access the Data Locker files.
Using AWS CLI
Before you begin:
- Install the AWS CLI on your computer.
- In AppsFlyer, go to Data Locker, retrieve the information contained in the credentials panel as it is needed to perform this procedure.
To use AWS CLI:
- Open the terminal. To do so in Windows, <Windows>+<R>, click OK.
The command line window opens. - Enter aws configure
- Enter the AWS Access Key as it appears in the credentials panel.
- Enter your AWS Secret Key as it appears in the credentials panel.
- Enter eu-west-1
- Press Enter (None)
Use the CLI commands that follow as needed.
In the following commands, the value of {home-folder} can be fond
To list folders in your bucket
aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/Listing files and folders
There are three types of folders in your Data Locker bucket:
- Report Type
t= - Date
dt= - Hour
h=
To list all the reports of a specific report type:
aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/To list all the reports of a specific report type for a specific day:
aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17To list all the reports of a specific report, in a specific hour of a specific day:
aws s3 ls s3://af-ext-reports/{home-folder}/data-locker-hourly/t=installs/dt=2019-01-17/h=23
Available data
For each file, the following information is available:
| Folder | Description | Organic | Non-Organic |
|---|---|---|---|
| clicks | Clicks reports. This type of report is only available in Data Locker. | No | Yes |
| clicks_retargeting | Click reports for clicks coming from retargeting campaigns. This type of report is only available in Data Locker. | No | Yes |
| impressions | Impressions. This type of report is only available in Data Locker. | No | Yes |
| installs | Installs reports. Contains data about both organic and non-organic installs. | Yes | Yes |
| inapps | In-App Events report. Contains data about both organic and non-organic in-app events. | Yes | Yes |
| conversions_retargeting | Retargeting reports for both re-engagements and re-attributions | No | Yes |
| inapp_retargeting | In-App Events resulting from re-attributions and re-engagements | No | Yes |
| sessions | App sessions. This type of report is only available in Data Locker. | Yes | Yes |
| uninstalls | Uninstalls reports. | No | Yes |
| blocked_installs | Protect360 blocked installs. | Yes | |
| blocked_inapps | Protect360 blocked in-app events. | Yes | |
| blocked_clicks | Protect360 blocked clicks. | Yes |
Using reports as data sources
You can use the data from the reports and add it to your own databases. To extract the data and add it to your databases you need to know the report format. Data Locker reports are based on Raw Data Reports. However, the final report format depends on the fields that you choose to include.
Report format
The report format for reports in Data Locker includes all the fields that are available in Raw Data V5 reports.
There is one main difference between Raw Data V5 reports and Data Locker reports. Data Locker reports contain all fields that are available in raw data reports. For example, a V5 installs report contains 90 fields. A Data Locker installs report contains 100 fields. Click here to download a sample file that you can use to compare the two reports.
Tip
Many reports contain valuable data that you can use for campaign optimization and retargeting.
Examples
- Clicks report - the clicks report contains the IDFA or Google Advertising ID. You can use these IDs to retarget users that engage with your ads but fail to install the app.
- Impression report - Like the clicks report, the impression report also contains the IDFA or Google Advertising ID. You can use the impression report to optimize campaigns according to impressions that don't lead to clicks. You can also retarget these users with different ads and in different campaigns.
- Retargeting and Re-attribution report - these reports also contain the IDFA or Google Advertising ID. You can use the IDFA or Google Advertising ID to highlight those users that you manage to retarget. Knowing what users you manage to retarget can help you optimize retargeting campaigns.
Note: To benefit from IDFA or Google Advertising ID as explained above, make sure they are included in all your attribution links.
Hourly reports
Data Locker separates data into hourly folders. The hourly folder represents the processing hour and not the hour that the event occurred. The data is written to Data Locker about six hours after the beginning of the processing.
Example
AppsFlyer receives data for activity between 14:00 and 15:00 on January 17, 2019. At some time after 15:00 AppsFlyer begins processing the data. Due to processing, the data is not written to Data Locker immediately. Thus the data in folder /t=installs/dt=2019-17-01/h=14 is not available on January 17th, 2019 at 15:00 but rather six hours later.
Creating hourly folders
In cases where there is no data for a specific hour, Data Locker creates a folder for that hour. This is to indicate to you that there was no data in that hour. The folder will contain a `_SUCCESS` file which indicates that AppsFlyer has completed writing to this folder. When designing automated processes this should be taken into account, meaning design your data retrieval processes so that they can handle empty hourly folders.
Clicks and impressions of SRNs vs. non-SRNs
- Non-SRNs use AppsFlyer attribution links for clicks and impressions. This provides AppsFlyer with the complete data set of the engagement which is then written to Data Locker.
- SRNs (Self-reporting networks) don't use AppsFlyer attribution links. As a result, only after an app open does the SRN share click and impression information, which is then written to Data Locker. To clarify, AppsFlyer is not aware of clicks and impressions which do not result in an app open. Note: Aggregate data reports include all clicks and impressions even if no app open took place.
Amazon clicks and impressions
Amazon clicks and impressions are not supported. They do not appear in the reports that are stored in Data Locker.
Timezone and currency
Timezone and currency in-app settings don't have an effect on data in Data Locker. Data Locker reports always display the timezone and currency in UTC + 0 and in USD.
