Running an incrementality experiment

At a glance: Run an incrementality experiment to perform lift analysis of remarketing (retargeting) campaigns.


Related reading | Incrementality guide

Incrementality experiment checklist 

Use this checklist to help set up your experiment.

# Description

Define test objectives

Formulate your hypothesis and decide what you want to measure.


Split an audience into test and control groups

In Audiences:

  • Configure the audience
  • Connect it to ad networks
  • Split it into test and control groups
  • Users are sent to the ad networks

During an experiment, don't make any changes to the audience configuration.  Changes to partners, audience name, or other settings will invalidate the statistical analysis.


Launch the test

In the Ad network:

  • Connect the audience to a campaign
  • Launch the campaign

The test group sent to the ad network is exposed to ads. The control group is not exposed to ads.


Analyze the test results

The test results show the incremental value of each group. 


Act on the test results

Based on the findings,

  • Adjust the test and run it again, or
  • Optimize campaigns

Form a hypothesis

An incrementality experiment needs a hypothesis. Use these variables as a guide.

Variables Details and best practices
Hypothesis Base it on the information in this table
Audience Describe the audience
Audiences size

Use an audience of at least 30,000 users and split them between 1-3 ad networks.

Best practice: Find a large user base and a campaign with high conversion rates. A significant statistical base will contribute to the success of the experiment.

Campaign Describe the campaign
Control group % Best practice: 15 to 25%
Test group

Best practice: Use 1-3 networks

Test groups are allocated as a percentage. Decide which ad networks get the test groups.

 Example hypothesis

Variables Details
Hypothesis The rate of test group users—who engage with the remarketing campaign and make a purchase—will be significantly higher than users in the control group. 
Audience Users who completed registration, but didn't make a purchase.
Audience size 100,000
Campaign Campaign that provides a discount voucher to users who make a purchase during a specified period.
Control group % 15% 
Test group Send test group to Network A and Network B.

Add an audience and launch the experiment

It's recommended to let an experiment run for at least 3 weeks; this allows for ongoing user conversions such as in-app events and revenue.

1. Define the audience

Follow these steps and the incrementality tool will begin to record measurements.

  1. Go to Audiences, add an audience.
  2. In the Audiences connect tab:
    • Connect the audience to 1-3 ad network partners.
    • Enable Split audience.
    • Set the size of the control group. Recommended: 15% or more. 

2. Configure campaigns in ad networks

  • On the day the audience is split, configure the campaign in the ad network.
  • Note! Direct the campaign to target users uploaded by AppsFlyer Audiences; don't include users from any other source. 

3. Use the Incrementality dashboard

Review results in the Incrementality dashboard (updated daily) 

Was this article helpful?