How do you test whether marketing spend is actually incremental?
Turn it off somewhere and watch. A geo holdout splits comparable markets into a test group that keeps spending and a control group that pauses, then compares the difference between them against how those same markets differed before the test. It is the only common method that speaks to cause rather than credit. It costs real revenue while it runs and needs enough volume to see the effect through normal variation.
The question attribution cannot answer
Attribution tells you which touches preceded a sale. It cannot tell you whether the sale would have happened anyway. That gap is the entire reason incrementality testing exists, and it is why a channel can post an excellent cost per lead while contributing almost nothing.
A holdout answers the counterfactual directly by creating one. Some markets get the spend, comparable markets do not, and the difference between them is the lift.
Designing a test that will actually be believed
- Match markets on behavior, not size. Choose control markets whose weekly lead pattern historically tracks the test markets, then verify that tracking over several months of pre-period data.
- Check the pre-period gap. Measure the difference between groups before the test starts. Lift is the change in that gap, not the raw difference during the test.
- Run past the full sales cycle. If jobs close two weeks after the lead, a two-week test measures nothing but the front half.
- Hold everything else still. No new promotions, no crew changes, no other channel adjustments in either group.
- Decide the threshold in advance. Write down what result would change the budget before you see the result.
The failure everyone repeats
The most common bad design is a single-market on-and-off test: pause spend for two weeks, compare against the previous two weeks. That comparison is confounded by weather, seasonality, competitor activity and ordinary week-to-week noise. It produces a number, and the number is unreliable in a direction you cannot predict.
The second most common failure is measuring only the paused channel. If you pause paid search and organic calls rise, total demand may be nearly unchanged. Always measure total leads and total revenue across all channels in both groups, not the channel you turned off.
What holdouts cost, and when they are worth it
Holdouts are not free. You deliberately forgo revenue in the control markets, and you learn something only after weeks of patience. That is a real cost, which is why they should be reserved for the spend lines large enough that a wrong answer is expensive.
Branded search, remarketing and broad awareness spend are the classic candidates, because those are precisely the places where attributed credit and incremental contribution diverge most. Running one properly requires that lead and revenue data already be joined cleanly across markets, which is why testing usually follows integration work rather than preceding it. Once the pipe exists, revenue reporting by market makes the readout routine.
Topics: incrementality · geo holdout · testing · causation
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.