Announcing CUPED availability in experiment: Ensure reliable results for every test

At Amplitude, our goal is to help our customers build better products by guiding them to clearer insights, trusted data, and faster action.

As the product leader for Amplitude Experiment, our team is committed to guiding our customers to get reliable results from every experiment, faster. Amplitude Experiment helps our customers scale experimentation in order to drive faster innovation across all of their digital products.

As part of that mission, I am extremely excited to announce that we have launched Controlled-experiment Using Pre-Existing Data (also known as CUPED), a powerful statistical technique meant to reduce variance in Amplitude Experiment.

Amplitude Experiment customers can now use CUPED to account for the possibility that the treatment effect may not be the same for all customer or user segments. For example, if you were testing onboarding experiences, novice users may prefer a simplified onboarding process whereas a more experienced user might not. CUPED is a useful tool to identify these sub-groups that could benefit most from this treatment.

What is CUPED and how does it impact A/B testing?

In traditional A/B testing, the average treatment effect is estimated by comparing the average outcomes of a treatment group to a control group. However, this method assumes that the treatment effect is the same for all individuals, which is not always true in practice.

CUPED addresses this limitation by estimating the treatment effect separately for each individual and then aggregating the individual estimates to obtain an overall estimate of the treatment effect.

The CUPED methodology works by first identifying a baseline characteristic (also known as a covariate) that may be related to the treatment effect. This covariate is then used to match individuals in the treatment and control groups based on their propensity score, which is the predicted probability of performing the conversion event based on the covariate.

By matching individuals with similar propensity scores, CUPED corrects for differences in users’ likelihood to convert that stem from factors other than the product change being tested, which reduces the bias in the estimated treatment effect. This is important because some sub-groups are more likely to respond to the treatment than other sub-groups, and these sub-groups may not be equally distributed among treatment and control groups.

Should our team use CUPED for every experiment?

There are a few situations where CUPED is not necessary or will not reduce variance within your tests. CUPED will not be an effective variance reduction technique if:

1. You are only targeting new users in your test.
2. If the event was not instrumented in Amplitude Analytics during the pre-period.

In general, anonymous users can be problematic for CUPED, but with Amplitude’s differentiated approach to seamlessly managing user identity, this is not a problem for Amplitude Experiment customers.

How can I use CUPED in my experiments?

Customers can now toggle on CUPED within their statistical settings under the Analyze tab in Amplitude Experiment. This is also available within Experiment Results.

We are really excited to hear from you about this powerful new statistical technique available to you now in Amplitude Experiment. Want to learn more? Check out a demo of Amplitude Experiment.