What is the difference between attribution and incrementality?

Attribution assigns credit for a sale to the advertising that touched the buyer. Incrementality measures whether that sale would have happened without the advertising at all. The first is a bookkeeping exercise; the second is a causal question, and only the second answers whether the spend was worth it.

How long should an incrementality test run?

At minimum one full buying cycle for your product, plus the delay before a sale is recorded in your system. For a considered purchase that can be a couple of months. Running it for two weeks because that is what the calendar allowed produces a number nobody should act on.

Is it safe to switch off advertising in a live market?

It carries real cost, which is exactly why it is worth doing across a limited set of areas rather than nationally. Pick regions large enough to read but small enough that the downside is affordable, and treat the lost business as the price of the information.

Can we test without switching anything off?

Partly. Geographic staggering — starting a campaign in some regions weeks before others — gives a weaker but less painful read. It is worth doing where a full holdout is genuinely not acceptable, as long as everyone understands it is the softer evidence.

What if the test shows a campaign is not incremental?

Then you have found money. Usually the right response is not to delete the campaign outright but to reduce it and watch, since some bottom-of-funnel activity defends against competitors rather than creating demand. What you should stop doing is crediting it with results it did not cause.

Performance Marketing

Did the ad cause the sale, or just arrive before it?

Every dashboard answers a question you did not ask — which ad stood closest to the sale. The question you wanted answered is whether that sale would have happened anyway.

7 September 20267 min read

The question the dashboard cannot answer

Your brand-name search campaign has the best numbers in the account. Lowest cost per conversion, highest return, the first thing anyone points at in the review.

It is also the campaign most likely to be buying customers you already had.

Attribution tells you which advertising was nearby when a sale happened. It is genuinely useful for that. What it cannot tell you — not with a better model, not with a longer window, not with a cleverer tool — is whether the sale would have happened without it. Every platform reports on itself, and none of them can report on a version of events in which they were absent.

That second question has a name. Incrementality. And unlike attribution, it has an honest answer.

The test is embarrassingly simple

Turn it off. Somewhere. Not everywhere.

Pick two sets of districts that behave similarly — similar size, similar history, similar dealer presence. Keep the campaign running in one. Switch it off in the other. Change nothing else. Wait long enough for your actual sales cycle to play out, then compare.

If enquiries and sales hold up in the switched-off areas, the campaign was harvesting demand that already existed. If they fall, you have measured something no dashboard could have told you: the part of the result the advertising actually created.

This is the same discipline as proving an activation worked — you need somewhere that did not get the thing, or you are only describing what happened.

Why almost nobody runs it

Because it costs something real, and the cost lands before the answer does.

You are deliberately switching off advertising in a live market, accepting that you may lose business there, in exchange for finding out whether that advertising was doing anything. That is an uncomfortable proposal to put in front of a sales head.

It is also, in most accounts, the cheapest information available. A campaign that is not producing incremental business is one you will fund forever on the strength of a report that was never designed to answer the question.

How to run one without wasting it

Split by geography, not by audience. Splitting an audience inside a platform tends to leak — the same person is reachable through other placements, and the platform is not trying to keep your test clean. Districts do not leak.

Pick a long enough window. Long enough for your real buying cycle, plus the lag before a sale is recorded. A two-week test on a product people consider for two months measures nothing.

Change exactly one thing. No new offer, no price change, no dealer scheme running in one set and not the other. Every extra difference is another explanation you cannot rule out.

Decide what would change your mind before you start. Write down the number that would make you cut the campaign, and the number that would make you double it. Deciding afterwards is how a test quietly becomes a justification.

Accept a rough answer. This will not produce a percentage to two decimal places. It will produce a direction, with real confidence behind it. That is a considerable upgrade on a precise number with no confidence behind it at all.

What usually turns up

Two patterns recur often enough to be worth expecting.

The bottom of the funnel is doing less than it claims. Brand-name search, retargeting, and anything aimed at people already deep in the process tend to show the best attribution numbers and the weakest incrementality. They stand nearest the sale. Standing nearby is not the same as causing.

The top is doing more than it gets credit for. The activity that creates demand — the activation, the signage, the awareness campaign — usually shows poor attribution numbers, because the effect arrives late and through a channel nobody tagged. That is a measurement failure, not a performance failure.

Which means the two things a last-click report will push you to do — cut the top, feed the bottom — are frequently the two things the evidence says not to do.

The uncomfortable summary

Attribution is a map of what was nearby. Incrementality is a measurement of what was necessary.

Most marketing budgets are allocated using the first and defended in the language of the second. Holding a budget to the second is the whole of what honest digital marketing work is. The switch-off test is how you find out which of your campaigns survive the difference — and once you can see clearly, no amount of performance machinery rescues an offer nobody wanted.

Questions people ask

What is the difference between attribution and incrementality?: Attribution assigns credit for a sale to the advertising that touched the buyer. Incrementality measures whether that sale would have happened without the advertising at all. The first is a bookkeeping exercise; the second is a causal question, and only the second answers whether the spend was worth it.
How long should an incrementality test run?: At minimum one full buying cycle for your product, plus the delay before a sale is recorded in your system. For a considered purchase that can be a couple of months. Running it for two weeks because that is what the calendar allowed produces a number nobody should act on.
Is it safe to switch off advertising in a live market?: It carries real cost, which is exactly why it is worth doing across a limited set of areas rather than nationally. Pick regions large enough to read but small enough that the downside is affordable, and treat the lost business as the price of the information.
Can we test without switching anything off?: Partly. Geographic staggering — starting a campaign in some regions weeks before others — gives a weaker but less painful read. It is worth doing where a full holdout is genuinely not acceptable, as long as everyone understands it is the softer evidence.
What if the test shows a campaign is not incremental?: Then you have found money. Usually the right response is not to delete the campaign outright but to reduce it and watch, since some bottom-of-funnel activity defends against competitors rather than creating demand. What you should stop doing is crediting it with results it did not cause.