Ready in minutes
AI builds your ad from a single prompt
June 24, 2026
A/B testing built the modern digital marketing playbook, and for fifty years television was exempt from it. When one commercial cost $30,000 and the only measurement was a ratings estimate, nobody tested anything; they argued in conference rooms and aired the compromise.
Both barriers are gone. AI generation makes a second variation free and two minutes away, and CTV reports per-creative completion rates, household delivery, and response signals continuously. Which means TV advertising can finally be run like a testing program, and the small businesses doing it are quietly compounding an advantage every month. This guide covers the methodology: what to test, how to structure it, how much data is enough, and the decision rules that keep you honest. For the product-level tips on generating the variations themselves, start with our Adwave AI creative guide; this is the science on top of it.
The cardinal rule of any A/B test is isolation: variations should differ in exactly one meaningful element, so the performance gap has exactly one explanation. Variation B with a different opening, offer, AND voiceover is three tests wearing one trench coat, and it teaches you nothing.
The elements worth isolating, roughly ordered by typical impact:
The opening five seconds. A question versus a bold claim versus a customer's face. Viewers decide whether to keep watching almost immediately, so the open moves completion rates more than anything else.
The core message. Price-led versus quality-led versus trust-led. The biggest strategic lever, and the slowest to read because it works on memory, not just attention.
The call to action. "Visit the website" versus "call today" versus a dated offer; QR-code presence versus none. Reads fastest in direct-response signals.
Tone and pacing. Warm-and-personal versus energetic-and-promotional. Often the difference between a spot that wears out in four weeks and one that runs a quarter; the fundamentals live in what makes a good TV commercial.
Write the hypothesis down before launching, in one sentence: "A question opening will hold attention better than our current claim opening, measured by completion rate." If you can't write that sentence, you don't have a test yet; you have two ads.
Simultaneous rotation (the default). Both variations run in the same campaign, same geography, same window, with delivery split between them. Same audience, same weeks, same external conditions, which controls for everything except the creative. This is the right design for completion-rate questions, and it's operationally trivial: load both spots and let the rotation run.
Sequential flights. Variation A runs for four weeks, then B replaces it for four. Use this only when the response signal you care about can't be split by creative (some conversion proxies can't), and beware its weakness: seasonality rides along. A landscaper testing A in April against B in May isn't testing creative; they're testing spring.
Geo-split. A runs in half your zips, B in the other half, matched for size and demographics. The strongest design for business-outcome questions (calls, visits, bookings by area), and the natural fit for multi-location operators. The cost is complexity: your zips are never perfectly matched, so reserve geo-splits for big strategic questions and run them longer.
TV testing's most common failure is calling winners from noise. Working thresholds:
Two notes on reading that table honestly. Completion rate is your fast, high-volume metric; nearly every test should start there, and many can end there, since attention is the precondition for everything else. And small advertisers should size tests to their delivery: at 30,000 monthly impressions, a two-way rotation hits completion-rate thresholds in two to three weeks, but a four-way split starves every variation. Two variations at healthy volume beat four at noise level, the same concentration logic as the reach-and-frequency math.
Read results in this order, and don't let a lower layer overrule a higher one without a reason:
Completion rate answers "did the creative hold attention?" It's per-creative, high-volume, and arrives first. A variation viewers abandon can't win anything downstream.
Direct response signals (QR scans, landing-page visits) answer "did it move anyone?" Lower volume, slower, but closest to behavior. Tracked per-creative when each variation carries its own code or page.
Branded search and site traffic answer "is it building memory?" Campaign-level rather than per-creative, so they adjudicate sequential and geo-split designs best.
Business outcomes (calls, bookings, sales by area) answer the only question that ultimately matters, on the slowest clock. The full instrumentation list is in our campaign metrics guide.
The hierarchy resolves most apparent contradictions. If B wins completion but A wins QR scans, you've usually learned that B entertains and A converts, which is a message-versus-execution insight worth more than either metric alone: take A's offer and B's opening into variation C.
Five minutes of setup separates a test you can read from a test you'll argue about:
Separate QR codes per variation, each pointing at its own copy of the landing page (or the same page with different URL parameters). Now scans and visits attribute cleanly.
Matching offer language everywhere a variation lands. If spot B promises the consultation and its landing page leads with the discount, you've smeared the test across two variables.
A one-line test log: hypothesis, start date, planned duration, threshold, result. It feels bureaucratic until month six, when it's the most valuable marketing document you own.
Delivery checks weekly. Confirm the rotation is actually splitting impressions evenly; a 70/30 accidental skew quietly invalidates the comparison, and catching it early lets you rebalance instead of rerunning.
A med spa's champion spot opens with the owner speaking to camera; completion runs 93%. Hypothesis: a patient-transformation opening will hold attention better. The challenger is generated in an afternoon, identical from second six onward.
Rotation runs 50/50 across the same five zips. Week one: challenger 95.5%, champion 92.8%, but it's day seven, so nobody calls it. Week two holds: 95.2% versus 93.0% on 9,000 impressions each, a stable 2.2-point gap that clears the threshold. QR scans agree directionally (19 versus 12).
The challenger is promoted. Next month's test takes the new champion's opening as fixed and tests the close: "book online" versus a first-visit offer. The log's first two lines are written, and the spa's ads are now measurably better than they were sixty days ago, at a production cost of zero.
Set the rules before the test starts, because mid-test judgment bends toward whichever spot you secretly prefer:
Call a winner when the leading variation clears the threshold table and holds its lead for a full week of stable delivery. Promote it to champion; retire the loser.
Call a tie when gaps stay inside the noise band past the typical wait. Ties are informative: the element you tested doesn't matter much, so spend the next test on a different element.
Call a foul when delivery skewed (one variation got materially more impressions or better hours) or an external event polluted the window. Rerun rather than rationalize.
Never peek-and-stop. Day-three leads evaporate routinely. Decide the duration in advance and let it run, checking only that delivery stays balanced.
And the meta-rule: one test at a time per campaign. Overlapping tests in the same geography contaminate each other's reads.
The everything-different test. Two unrelated ads compared is a beauty contest, not a test. You'll get a winner and no idea why, which means you can't repeat the win.
Starved variations. Splitting a small campaign four ways guarantees every arm sits in noise forever. Match variation count to delivery volume.
Seasonal confounds in sequential designs. Any before/after comparison inherits whatever else changed: weather, holidays, a competitor's launch. Prefer simultaneous rotation whenever the metric allows.
Frequency entanglement. If adding a variation changes per-household frequency dynamics, you're testing creative and frequency effects at once. Hold total campaign settings constant; vary only the creative split.
Testing trivia. A different background song is rarely worth a test cycle. Spend tests on the hypothesis menu's top items, where wins move real numbers.
A single test is a curiosity; a cadence is an asset. The champion/challenger rhythm that fits most local campaigns:
Your current best spot is the champion, carrying most delivery.
Each month, generate one challenger testing one element (two minutes of work).
Run the rotation to threshold, apply the decision rules, promote or retire.
Log what won and why in a simple running document. Twelve months later you own something no competitor can copy: a tested playbook of what your market responds to.
The businesses that do this don't have better instincts. They've just replaced arguments with evidence, twelve times a year, while the production cost of being wrong sits at zero.
Can small businesses really A/B test TV ads?
Yes, and it's newly practical. Generating a second variation is free and takes about two minutes with Adwave, and CTV dashboards report completion rates and delivery per creative. A campaign with 25,000-30,000 monthly impressions supports a clean two-way test on a two-to-three week read, which puts genuine TV testing inside almost any local budget.
What's the best metric for comparing two TV ad variations?
Start with video completion rate: it's per-creative, accumulates fast, and measures the precondition for everything else, since an ad viewers abandon can't build memory or response. Layer QR scans or per-variation landing pages for a behavioral read, and reserve branded search and business outcomes for bigger, slower strategic tests.
How long should a TV ad A/B test run?
Completion-rate tests typically resolve in one to three weeks once each variation has 5,000-10,000 impressions; response-signal tests want a month; geo-split outcome tests deserve a quarter. The honest answer is "until the thresholds you wrote down beforehand are met," and the discipline of not calling early winners matters more than the exact duration.
Should both ad variations run at the same time or back to back?
Simultaneous rotation is the default because it controls for everything except the creative: same weeks, same households, same weather. Use back-to-back flights only when your key metric can't be split per creative, and treat any sequential result with seasonal suspicion. Geo-splits earn their complexity only for big business-outcome questions.
What should I test first?
The opening five seconds, with everything else held identical. It's the element with the largest, fastest-reading effect on the metric you can measure most cheaply (completion rate), which makes it the ideal first test: you'll see how the whole methodology feels within two weeks, and the winner usually improves every downstream number on its own.
Bottom line: TV creative decisions were the last place in small business marketing where opinion outranked evidence, and the excuse just expired. One hypothesis, one variable, thresholds set in advance, and a new challenger every month. The ads get better, the playbook accumulates, and the conference-room argument quietly retires.
The next challenger is two minutes away. See how Adwave works: generate the variation, set the rotation, and let your customers vote with their attention.
