Incrementality guide

AppsFlyer Incrementality employs 2 alternative methods for calculating lift:

• Intent-to-treat
• Reach-based

While reach-based methodology is preferable for purposes of evaluating campaign performance, its calculation requires the measurement of data points not provided by all ad networks. For these networks, intent-to-treat methodology is used instead.

ITT-based lift

Intent-to-treat (ITT) experimental methodology has its foundations in medical science. Using ITT, researchers randomly split a given population between a control group who are not treated and a test group whom researchers intend to treat. Researchers measure the efficacy of a given treatment by comparing the results of the groups. Whether or not a test group member actually received treatment is not taken into account.

Similarly, in remarketing, the lift metric measures campaign efficacy. It doesn't matter if a user in the test group actually engaged with a campaign (received treatment) because it is the intent that is relevant in calculating incrementality. In other words, results are calculated as if every member of the test group actually engaged with the campaign.

When is it used?

AppsFlyer Incrementality uses ITT-based lift calculation for SRNs.

Although it is not the preferred method for measuring lift in our context, ITT remains a scientifically sound experimental method, and it provides valid results when reach-based calculation is not possible.

Interpreting results

The underlying assumption that all members of the test group were actually reached by a campaign often results in low conversion rates and, as a consequence, low incremental lift.

Reach-based lift

As its name implies, reach-based lift calculation takes into account whether or not a member of the test group was actually reached (served a campaign ad) prior to converting. In addition to requiring additional data, reach-based lift calculation is also more complex in that it requires certain mathematical adjustments to the control group in order to be statistically valid. These adjustments are cumulatively known as the creation of a "counterfactual" control group.

When is it used?

AppsFlyer Incrementality uses reach-based lift calculation for non-SRNs (attribution-link networks).

While it is the preferred method for calculating lift, reach-based calculation requires that ad networks report the following data to AppsFlyer:

• Impression data
• Campaign ID (from the attribution link)

If this data is not reported, incrementality measurement is not available (and displays as N/A on the Incrementality dashboard).

Interpreting results

Since reach-based lift calculation takes into account only test group members that were actually reached by a campaign, networks that are measured using reach-based incrementality will usually show amplified results (positive lift will become much higher, and negative lift will become much lower).

Best practices

Because the 2 different lift calculation methodologies can yield significantly different results, the following best practices are recommended in order to make interpretation more meaningful:

• When creating your experiments, select audiences that are split only among a control group and one or more networks of the same type, either:
  • Attribution-link networks (non-SRNs) for reach-based calculation; or
  • SRNs for ITT-based calculation
• In your initial experiments, consider using an audience split only between a single network and a control group. This makes it less likely that comparisons will be made among 2 or more networks for which different lift calculations are used.

When evaluating the validity of Incrementality experiments, sound scientific methodology requires that we consider statistical significance (the level of confidence that the lift result isn't due to random events or chance). 

On the Incrementality dashboard, significance is reported as a percentage for each media source:

• The percentage indicates the likelihood that the results for that media source would repeat themselves if the experiment were performed again under similar circumstances.
• If statistical significance is determined to be below 60%, the dashboard will display Unreliable results instead of a percentage.
• Significance is affected by audience size, experiment duration, and media source performance.

To access the Incrementality dashboard:

1. In AppsFlyer, go to Dashboards > Incrementality.
2. From the experiments list, click the row of the experiment for which you want to see the dashboard.

The basic logic underlying report structure is as follows:

• Users participating in the experiment—First-seen report
  • In Audiences, rules are set characterizing users to include in the experiment.
  • When a given user, is identified as matching the rules, the event is recorded in the first_seen report. 
    • Users are allocated randomly to a test or control group indicated by the is_control_group field.
    • Users in the test group are allocated to a media source (pid_destination) for retargeting.
• User engagement within the app: User engagement with the app during the experiment is recorded in context-specific reports: 
  • Engagement type: Session or in-app event
  • User attribution status when the user is first-seen: Organic, non-organic, re-attribution. For example, in the past, a user installed the app and was attributed to organic. As such, during an experiment, the attribution status is organic.  
• Uninstalls: Users uninstalling the app during the experiment. Uninstall measurement must be active. 

Field availability varies according to report type as indicated below:

Fields unique to Incrementality (per report type)

Other Incrementality report fields (per report type)