Incrementality guide

At a glance: Incremental lift analysis measures the benefit derived from a remarketing (retargeting) campaign after taking into account organic cannibalization. To calculate lift, in AppsFlyer, you define an audience, select ad network partners to run the campaign, and then enable the ad network partner to receive the audience. AppsFlyer does the rest; we manage participants, test and control groups, partners, metrics, and more. Calculated results display in the Incrementality dashboard.

IncrementalityMetrics_us-en.png

Related reading | Running an incrementality experiment | Audiences

Incrementality

Incrementality analysis is used to make sure that remarketing campaigns have a positive contribution. It considers users who would have converted organically without engaging with a remarketing campaign. The AppsFlyer Incrementality solution has its foundations in the medical science intent-to-treat (ITT) experiment methodology.

Using ITT, researchers randomly split a given population between a control group who are not treated and a test group whom researchers intend to treat. It matters not if treatment is given or not. Researchers measure the efficacy of a given treatment by comparing the results of the groups.

Similarly, in remarketing, the lift metric measures campaign efficacy. It doesn't matter if a user in the target group engages with a campaign (receives treatment) because it is the intent that is relevant in calculating incrementality. 

Calculating lift 

Incrementality_-_Incrementality.png

Statistical significance (p-value)

To make sure that the lift result isn't due to random events or chance, ITT methodology requires that the data be assessed for statistical significance using a p-value indicator.

The p-value indicates that experiment results are more than 90% likely to repeat themselves if you run a replicate of the experiment. A p-value of less than 0.1 (typically ≤ 0.1) is statistically significant. It indicates strong evidence against the null hypothesis—there is less than a 10% probability the null is correct (and the results are random). Therefore, the null hypothesis is rejected and the alternative hypothesis accepted.

Guide to running an experiment

Incrementality dashboard

To use the Incrementality dashboard:

  1. In AppsFlyer, go to Dashboard > Incrementality.
  2. Set filters to display the required metrics.
  3. Use display controls to:
    • Select different metrics
    • Display options according to those described in the following tables. 
Component Description

FilterHeader.png

Audience

Audience selected. Can be used to target more than one app. 

First seen dates

Date on which a user first matched audience rules and was then added to the audience.

Targeted app An app that benefits from the remarketing campaign efforts.
Media sources Ad networks displaying ads to test group members.
Filters
Component Description
Selected metric
Metric Description
Conversion rate 
  • Test: Converting users/Test group users
  • Control: Converting users/Control group users
  • Lift (Test CVR %–Control CVR %)/Control CVR %
  • Difference: Test–Control
Revenue per user for a given event
  • Test: Revenue of converted users/Converting users
  • Control: Revenue of converted users/Converting users
  • Lift (Test revenue per user–Control revenue per user)/ Control revenue per user
  • Difference: Test–Control
  • Media sources: Metric per media source metrics calculated similarly
Events per user for a given event 
  • Test: Number of events of test group users/Test group users =(a)
  • Control: Number of events of control group users/Control group users=(b)
  • Lift: (a–b)/b
  • Difference: a–b
Group size
  • Test: Test group users
  • Control: Control group users
  • Targeted media sources:
    • Distribution of users in %
    • Allocation displays in a hover
Significance

Significance:

  • Measures the probability that the experiment result (lift) was random.
  • AppsFlyer considers a p-value < 0.10 to be significant
  • In general, the larger the test population, the better the significance
Metrics are calculated according to active filters
Component Description
Trend display chart/table form

IncrementatilyChartHeader.jpg

Change the chart/table display with these controls:

  • Group by: 
  • View type
    • Chart
    • Table
  • Calculated as:
    • Cumulative
    • On day
  • Export: Download the table to a CSV file. 
Trends/table

Incrementality raw data reports

Use Incrementality raw data reports to analyze the interaction of users with remarketing campaigns.

Category Report name 

Data Locker folder

First-seen

First-seen users

incrementality_first_seen_users

In-app events

Organic in-app events

incrementality_organic_inapps

In-app events non-organic

incrementality_inapps

In-app events re-attributions

incrementality_inapps_reattr

Sessions

 

 

Organic sessions

incrementality_organic_sessions

Sessions non-organic

incrementality_sessions

Sessions re-attributions

incrementality_sessions_reattr

Uninstalls

Uninstalls (This report is currently not populated) 

incrementality_uninstalls
Incrementality raw-data reports available


Report logic

  • Users participating in the experiment—First-seen report
    • In Audiences, rules are set characterizing users to include in the experiment.
    • When a given user, is identified as matching the rules, the event is recorded in the first_seen report. 
      • Users are allocated randomly to a test or control group indicated by the is_control_group field.
      • Users in the test group are allocated to a media source (pid_destination) for retargeting.
  • User engagement within the app: User engagement with the app during the experiment is recorded in context-specific reports: 
    • Engagement type: Session or in-app event
    • User attribution status when the user is first-seen: Organic, non-organic, re-attribution. For example, in the past, a user installed the app and was attributed to organic. As such, during an experiment, the attribution status is organic.  
  • Uninstalls: Users uninstalling the app during the experiment. Uninstall measurement must be active. 

Data characteristics and fields

Field availability varies according to report type as indicated.

Fields unique to incrementality per report type
Field Description First seen In-app events Sessions Uninstalls
is_control_group If true, the user is part of the control group Y Y Y Y
pid_destination The media source the user is sent to  Y Y Y Y
audience_id Unique identifier Y Y Y Y
joined_audience_date Date user first joined the audience Y Y Y Y
audience_name Audience name (not unique)  Y Y Y Y
tm Hour of day   Y Y Y
timestamp Event time stamp YYYY-MM-DD HH:MM   Y Y  
app_ids App ids associated with the audience rules Y      

 

Other fields in incrementality reports per report type
Field Display name* First seen In-app events Sessions Uninstalls
advertising_id Advertising ID (GAID) Y Y Y Y
android_id Android ID   Y Y  
app_id App ID Y Y Y Y
app_name App name   Y Y  
app_version App version   Y Y  
appsflyer_id Appsflyer ID   Y Y Y
revenue_alt App-specific currency   Y    
bundle_id Bundle ID   Y Y  
country Country code   Y Y Y
currency Currency code   Y Y Y
customer_user_id Customer user ID   Y Y  
brand Device brand   Y Y  
device_category Device category   Y Y  
model Device model   Y Y  
device_model Device model   Y Y  
device_type Device type   Y Y  
event_name Event name   Y Y Y
event_revenue Event revenue   Y    
event_revenue_currency Event revenue currency   Y    
event_revenue_u_s_d Event revenue USD   Y    
event_time Event time   Y Y Y
event_value Event value   Y Y  
idfa IDFA Y Y Y Y
idfv IDFV   Y Y  
imei IMEI   Y Y  
is_purchase_validated Is receipt validated   Y    
os_version OS version   Y Y  
platform Platform   Y Y Y
sdk_version SDK version   Y Y  
* According to raw data specification

Traits and limitations

Trait Remarks 
Ad network access Not available
Agency access Not available
Agency transparency Not applicable
Time zone UTC
Currency  USD
Organic data Yes
Non-organic data Yes
Data freshness

Daily

Historical data

N/A

Team member access Yes, per account permissions.
Was this article helpful?