A/B testing Facebook Ads without fooling yourself

An A/B test means running two versions of an ad at the same time, with a single difference between them, to find out which one gets the better result. A against B. The principle is simple. Getting an answer you can trust is less so.

One difference at a time

This is the rule that makes everything work. If version B changes the hook, the music and the button, and it wins, you don't know why. You have a better ad, but no lesson to reuse on the next one.

A good test asks a precise question: "does opening with the price beat opening with the problem?". The answer then serves every ad that follows.

What to test first

Not everything weighs the same. On Meta today, the creative decides much of who sees the ad and what it costs. That is where tests pay off most:

• The hook, the first seconds of the video. This is the test that makes the biggest difference. Six hook mechanics.
• The angle: price, timing, result, fear of getting it wrong.
• The format: talking to camera, filmed at a customer's home, AI-generated video.
• The audience, after that. It matters, but less than people think since Meta finds the right people itself.

Fine settings, bids and placements, come last. They rarely move the result visibly.

Meta's A/B test, or two ads side by side

Meta offers an A/B testing tool that splits the audience into two groups that don't overlap. It is the cleanest method, but it takes budget and time.

The other way is to let several creatives run in the same ad set and see which ones Meta pushes. It is faster, but Meta quickly picks a favourite, sometimes before the other one has had its chance. Depending on the question, I use one or the other.

What a real test shows: the joinery case

In the joinery case published here, three audiences ran in parallel from day one, with the same creatives. Out of 193 enquiries in two months:

Without this test, you could have bet everything on interests, the option that looks most "targeted", and it turned out to be the most expensive. The full case.

Why many tests lie

• Too few results. Five enquiries against three is not a winner, it's chance. You need enough conversions on each side for the gap to mean something.
• Too short. Monday doesn't look like Saturday. A test stopped after two days mostly measures the calendar.
• The learning phase. Every new ad goes through a few days where Meta is still working out who to show it to. Judging during that period means judging a draft.
• Stopping when you like the number. The most human bias, and the most expensive.

How many results, how many days: it depends on the budget and the cost of a conversion, and it is part of the work I do on each account.

---

Further reading: Ad fatigue and Andromeda, and why Meta wants varied creatives.

Running tests but not sure what to conclude? Book a 30-minute call, and we'll go through your tests together.