Geo holdout testing

What geo testing is

Geo holdout testing measures incremental impact by treating whole geographic regions as the unit of experimentation, instead of individual users. You pick a set of regions, cities, states, or media markets, and split them into a treatment group where ads keep running and a holdout group where ads are paused for the test period. You then compare conversion volume, sales, or another business metric between the two groups.

This approach does not rely on pixels, cookies, or a platform’s internal user-matching. It measures outcomes at the aggregate level, using whatever sales or conversion data you already track by region, which makes it useful when tracking is unreliable or when you want to validate a metric that lives outside any ad platform, like in-store revenue.

How it works

The core requirement is finding regions that are comparable before the test starts. Analysts typically look at historical sales patterns and pick treatment and holdout regions with similar trends, size, and seasonality, so that any difference observed during the test can be attributed to the ad exposure rather than pre-existing differences between markets.

During the test, spend continues normally in treatment regions and is paused or significantly reduced in holdout regions. After the test period, usually a minimum of several weeks to smooth out day-to-day noise, you compare the change in the target metric between the two groups. The difference, adjusted for any pre-test gap between the regions, is the estimated lift.

Why it matters

Geo testing works even when cookie or pixel tracking is degraded, since it does not require identifying individual users across devices. It is one of the few incrementality methods that can validate offline outcomes like retail foot traffic or point-of-sale revenue, which platform-level tests cannot see. It also avoids “contamination,” a problem where individually-randomized tests leak because people in a control group still see an ad meant for someone else in the same household or shared device.

The trade-off is that geo tests are coarser. They need larger regions and longer durations to reach a confident read, and they are more sensitive to regional differences unrelated to advertising, such as a local weather event or a competitor promotion in one market.

How to act on it

Use geo tests for validating incrementality of channels or campaigns where offline conversion matters, or where you distrust pixel-based numbers due to tracking loss. Choose region pairs carefully using historical data rather than intuition, and hold the test period long enough to average out daily noise. When the geo test confirms strong lift, it supports scaling spend with more confidence than attribution data alone provides.

In practice, teams often pair geo testing with platform-level conversion lift studies. Where the two agree, confidence in the incrementality estimate is high. Where they diverge, it usually signals a measurement gap worth investigating further.

Common mistakes

Choosing regions that were never comparable to begin with undermines the whole test, since any observed difference could simply reflect pre-existing gaps. Running the test for too short a period, especially over a holiday or unusual sales event, introduces noise that swamps the actual signal. Ignoring cross-border spillover, where residents of a holdout region are exposed to ads meant for a neighboring treatment region, also weakens results. Treating a geo test result as a fixed truth rather than a periodic check overlooks how market and competitive conditions shift.

How YieldBI helps

YieldBI does not run geo holdout tests itself, but it keeps platform-level attribution and blended ROAS in one place, so a geo test’s regional results are easy to compare against reported performance and spot where the two diverge.