Incrementality experiment alerts

At a glance: Understand experiment alerts and get them resolved to ensure accurate experiment results.



Related reading: Running an Incrementality experiment | Incrementality Guide 


Once you've set up an Incrementality experiment, you may come across alerts on the Incrementality dashboard. These alerts are designed to help you identify any issues that could compromise the integrity and accuracy of your experiment results. They provide specific information about the problems and suggest actions that can be taken to address them and ensure precise experiment results. This article provides a comprehensive overview of the different types of alerts, as well as the recommended actions for getting the issues resolved.

Types of alerts

Experiment alerts fall into one of three categories and will appear due to issues with:

  1. Experiment setup
  2. Experiment background environment
  3. Linked campaign performance

Let's take a look at the alerts in detail:

Experiment setup

Audience deletion or control group removal 

Experiments whose connected audiences have been deleted or whose split audience control group has been removed. In both these scenarios, a 404 error will appear when visiting the experiment page.

To resolve this and restore the experiment results:
If the audience was deleted - Please contact your CSM who will have it restored manually.
If the control group was removed - Please split the audience again and set the control group to the exact same percentage as it was before removal.

No impressions (Non-SRNs only)

When Incrementality experiments are done on campaigns run on non-SRN ad networks, AppsFlyer requires user impressions in order to most accurately calculate the incremental impact of those campaigns. Without impressions, the results will simply not display on the experiment page.

This alert describes a situation in which for some reason we're not receiving impressions data for the connected campaign.

Actions to take: Check that you are receiving impressions from the media source and that the tested campaign is working and delivering ads.

More suitable campaign? (SRNs only) 

This alert is relevant to Self Reporting Networks (SRNs) only. This alert appears when a campaign is connected to an experiment but data indicates that there's a campaign that may be better suited for targeting the audience.
A "more suitable campaign" is a campaign, currently active during the experiment time frame, that targets the test group in a more substantial way than the configured campaign (and doesn't target the control group).

Actions to take: Check if the right campaign is linked.

Note: It's also a good idea to make sure the audience is not included as part of the targeting configuration of other campaigns.

Experiment background environment

Background campaign (one or multiple)

This alert appears when there are unrelated, very large campaigns that have a much bigger impact on the audience than the configured campaign in the experiment.

How can this happen?
It's not unusual for unrelated campaigns to exist in the background, but if those campaigns have a bigger budget or operate more intensely, they can overshadow the configured campaign—and subsequently affect the results of the experiment.

Actions to take:

  • Increase the impact of your campaign by raising the budget.
  • Check whether the tested audience overlaps with the audience targeted by the background campaign (can be checked with the Audiences partner overlap feature).
  • Check whether the tested audience overlaps with the unrelated background campaign.
  • Tighten the tested audience segmentation.
  • If actions were taken to minimize the impact of background campaigns, set the experiment start date to after these changes were made.

Note: It is best practice to minimize the targeting of the tested audience with other campaigns.

Linked campaign performance

Control group is targeted 

This warning indicates that the configured campaign is targeting the control group although it's only supposed to target the test group.

How can this happen?

  • The experiment was linked to the wrong campaign.
  • The campaign inside the media source was configured to target users outside of the audience (learn about audience expansion features in Facebook and Google). 
  • Network not enforcing the targeting configuration.

Actions to take:
Please check the common scenarios listed above, and contact your CSM if unsure. In certain cases, depending on the significance of the exposure to the campaign by the control group users, there isn't a fix for this, and a new Incrementality experiment should be created.

If the breach is relatively small, it's enough to make sure the control group targeting has stopped. Track the warning in the experiment’s UI and check to see if the alert still appears after a few days. 


Out-of-audience targeting

The tested campaign is also targeting users outside of the targeted audience test group. Part of the campaign's cost is being spent delivering ads to those unrelated to the tested audience users, and who are not taken into consideration in the Incrementality test.

Therefore, when spending money on unrelated users, metrics that involve the cost of the campaign such as iROAS & CPiA will be biased. They will take into account the entire campaign cost, including the cost it took to present ads to users outside of the audience, but it will not count their conversions.

How can this happen?

  • The experiment was linked to the wrong campaign.
  • The campaign inside the media source was configured to target users outside of the audience (learn about audience expansion features in Facebook and Google). 
  • Network not enforcing the targeting configuration.

Actions to take:
Make sure the campaign's targeting configuration contains only the tested audience and that the campaign's configuration doesn't allow the campaign to target outside of the audience.


Low activity 

The campaign is not delivering ads or is delivering very few ads to the targeted audience.

Actions to take: Investigate the problem on the media partner side to understand why the campaign isn't delivering ads.


Audience overlap

The connected campaign is targeting overlapping audiences. This means that the linked campaign's targeting configuration contains more than one audience.
Additionally, the multiple audiences in the tested campaign configuration overlap. This will result in targeting outside of the tested audience, including targeting the control group of the tested audience.

Actions to take:
Update the campaign's targeting to include the tested audience only and remove any additional audiences.