creative volume reality
How many genuinely different ads can your budget tell apart? Find out before you brief eight creatives.
A result is the event you judge the ads on: a purchase, a lead or a click. A concept is a genuinely different idea, not a colour or headline swap. The cheaper the event, the more results you get, though a cheap event may not predict sales.
Confidence and power
How many concepts one round can read
| How fine a difference | Concepts one round can read | Results each |
|---|
A gap is how much cheaper one concept makes a result than another. Finding smaller gaps needs far more results per concept, so fewer concepts fit.
Your plan for concepts
| How fine a difference | Rounds | Total budget | Plan |
|---|
Each round carries the best concept so far against fresh challengers, so the winner keeps being tested. Rounds run one after another.
How it works
The calculation
With an even split, each concept gets C results. The log of the ratio of two concepts' results has a standard error of about the square root of 2 over C. To see a true gap G at your confidence and power:
Comparing every challenger with the best concept means k - 1 comparisons, so confidence tightens with the Sidak correction. That is why each extra concept costs more than the one before it.
The rounds
If a round reads k concepts, one of them is the current best and k - 1 are new. Testing N concepts in all needs the first round plus enough more to cover the rest: the rounds are the number of new concepts, N - 1, divided by k - 1, rounded up. Jon Loomer has argued for stacking creative variety in phases for smaller budgets, which is the same idea.
Read this before you plan a creative calendar
The platform may not split evenly. In one ad set, delivery favours whichever ad it likes early. Use separate ad sets with equal budgets, or the platform's own split test.
A round that cannot be read still teaches you something. It tells you to test bolder differences or give the test more money.
The cheapest result is not always the best sale. Judging on clicks finds the cheapest click. Check it against what you actually sell.
Results are counted as rare events. This holds when conversions are small against impressions.