Cost ETL—granular campaign cost in a bucket

• Reports are for all apps included in Cost ETL, per day, per batch. 
• Each time data is written to the bucket, meaning 4 times a day, all available data is written, including the history, updates, and corrections of the previous 6 days, and days 14, 29, and 88. Take this into consideration in your data loading process. 
• Reports structures are detailed in the file fields table. The structures are: 
  • Summary report is less granular (detailed) to enable easier and faster consumption.
  • Dimension reports have a primary dimension that is guaranteed. This means that the dimension is available for all media sources contained in that report. In contrast, if a media source does not provide the primary dimension data, that data is not included in the specific dimension report. Secondary dimensions are included when available. They are not guaranteed.
  • The primary (guaranteed) dimensions available are:
    
    • Geo: Data grouped by country
    • Channel: Media source channel, for example, YouTube in the case of Google and Instagram in the case of Facebook.
  • You should use the dimensions and metrics that best fit your business needs. This may differ depending on the media source.

• Data written to the bucket has the directory and file structure described. View sample file. Note: The sample provided is an Excel file; Cost ETL files are sent to your bucket as parquet files. 
• When Cost ETL completes writing to a directory, a flag is set by creating a success file. This is always the most recent timestamp in the directory. 
• Each time data is written, it includes data for the current day and the previous 6 days (referred to as 7 days in this article), as well as days 14, 29, and 88.
• The number of folders/files is as follows:
  
  • Summary: 4 batch folders per day.
    • Each batch folder contains parquet files with 7 days of data.
  • Dimensions: Each guaranteed dimension contains 4 batch folders per day.
    • Each batch folder contains parquet files containing data with numbering starting from 1.

Example directory structure

Directory structure

/<advertiser bucket name>/cost_etl/version/dt=<yyyy-mm-dd>/b=<n>/
<dimension>/<file name>

File name structure 

part-<number>

This configuration procedure must be performed by an admin user.

Before you start:

• Setting up Cost ETL consists of setting up your AWS bucket (and giving AppsFlyer permission to write data in it), and setting up Cost ETL in AppsFlyer.
• You will need both AWS admin privileges and access to the AppsFlyer UI to complete Cost ETL setup.
• Keep tabs to both AWS and AppsFlyer open during setup. 
• Note: KMS bucket encryption is not supported by AppsFlyer. 

To set up your AWS bucket and Cost ETL:

1. Sign in to the AWS console.
2. Go to the S3 service.
3. To create the bucket:
   1. Click Create bucket.
   2. Complete the Bucket name as follows: Start with the mandatory prefix af-xpend-cost-etl-acc-<af-account-id>- and then add a suffix as free text.
      • Your af-account-id can be found in the AppsFlyer UI as indicated in the following steps 7-9. 
      • See Amazon S3 bucket naming requirements.
   3. Click Create bucket.
4. In AppsFlyer, go to Integrations > Cost ETL.
5. Turn on Cost ETL.
6. In Report schedule, select when you want to receive the first report. Subsequent reports arrive at approximately 6-hour intervals thereafter.
7. Go to Amazon S3 settings.
   

   

8. Select your S3 bucket region from the dropdown.
   If your region isn't displayed, contact your CSM. 
9. Enter your Amazon S3 bucket name.
10. Click Next.
    The bucket policy code snippet displays.
    

    

11. Copy the bucket policy code snippet and paste it into your AWS settings.
    1. In AWS, Select the bucket you created for Cost ETL. 
    2. Go to the Permissions tab.
    3. In the Bucket policy section, click Edit.
    4. The Bucket policy window opens.
    5. Paste the bucket policy snippet into the window.
12. In your AppsFlyer Cost ETL settings, click Next.
    The Validate bucket step displays. 
    
13. Click Validate.
    Verify that Validation successful displays. 
14. Click Next.
15. Select one or more or all apps.  Select all to automatically include apps you add in the future.
16. Click Apply. 
17. Select at least one Guaranteed dimension: Channel and/or Geo.
18. Select at least one additional dimension.
    

    

19. Select at least one metric to be included in the reports. 
    

    

20. Click Apply.

This configuration procedure must be performed by an admin user.

Before you start:

• Setting up Cost ETL consists of setting up your GCS bucket (and giving AppsFlyer permission to write data in it), and setting up Cost ETL in AppsFlyer.
• You will need both GCS admin privileges and access to the AppsFlyer UI to complete Cost ETL setup.
• Keep tabs to both GCS and AppsFlyer open during setup. 

To set up your GCS bucket and Cost ETL:

1. Sign in to the GCS console.
2. Create a bucket.
   Name the bucket as follows:
   • Start with the mandatory prefix af-xpend-cost-etl-acc-<af-account-id>- and then add a suffix as free text.
   • Your af-account-id can be found in the AppsFlyer UI as indicated in the following steps 6-8. 
   • See GCS bucket naming requirements.
3. In AppsFlyer, go to Integrations > Cost ETL.
4. Turn on Cost ETL. 
5. In Report schedule, select when you want to receive the first report. Subsequent reports arrive at approximately 6-hour intervals thereafter.
6. Go to Data destination and select Google Cloud Storage (GCS) bucket.
7. Enter your GCS bucket name.
8. Click Next.
   The AppsFlyer service account displays, to be used to set the GCS permissions.
9. In your GCS console, set the IAM permissions for the bucket:

   1. Add the AppsFlyer service account as a principal to the Cost ETL bucket.

   2. Assign the role Storage Object Admin.
10. In your AppsFlyer Cost ETL settings, click Next.
    The Validate bucket step displays. 
    
11. Click Validate.
    Verify that Validation successful displays. 
12. Click Next.
13. Select one or more or all apps.  Select all to automatically include apps you add in the future.
14. Click Apply. 
15. Select at least one Guaranteed dimension: Channel and/or Geo.
16. Select at least one additional dimension.
    

    

17. Select at least one metric to be included in the reports. 
    

    

18. Click Apply.

In AWS, by default, when AppsFlyer writes objects to your bucket the object owner is AppsFlyer. Depending on your data loading process you might have to change the default ownership to you—the bucket owner.  

To change the ownership of objects in your bucket:

1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

2. In the Buckets list, choose the name of the bucket that you want to enable S3 Object Ownership for.

3. Go to the Permissions tab.

4. Under Object Ownership, click Edit. 

5. Select Bucket owner preferred.

6. Click Save.

When pulling and analyzing your data, it is recommended to pull data for a specific date and batch, or override all previous data for the days that the current batch provides. Otherwise, you may see the same data repeated.

For example, batch 1 on February 20 contains data for Feb 14-20. But, batches written on February 19 also contained data for Feb 14 to Feb 19. Override the data of the previous days received on February 19 with the data received in the most recent February 20 batch.

Not all networks provide data for all dimensions together. Geo and channel data in Facebook are the most common examples. This is why two separate data sets are provided. The geo data set is guaranteed to have geo data and the channel data set is guaranteed to have channel data.

In many cases and for many media sources, the data in the geo and channel sets will be identical. As such, consume one of the data sets (geo or channel), according to what best suits your needs.

If the integration agreement with a given media source doesn't include channel, so that channel is blank, we treat that data as if it contains the channel. 

Cost ETL provides flexible and granular data as deep as can be extracted from the ad network. To extract actionable insight from such potentially huge amounts of data, it is recommended to aggregate the data in a way that best suits your business needs. For example, if you need to understand cost data at the campaign and country-level, use those dimensions.

Not all networks provide data with the same granularity. For example, Facebook does not provide site ID cost data, while Twitter does not provide geo cost data. Be aware of such cases as you aggregate Cost ETL data, and make sure you look at similar data as you compare networks.

Cost ETL provides information regarding all your cost data. Some campaigns provided in Cost ETL do not appear in some AppsFlyer dashboards, for example, data of inactive campaigns, meaning campaigns without any recorded installs. To compare the data, find a specific campaign ID in the overview dashboard and compare it to its cost data in Cost ETL. Learn more about cost data availability