"Would those people have bought from us anyway?"

If you own a loyalty program, that question is probably coming in your next budget meeting, if it's not already haunting you in your dreams. Finance will have the deck open and the cost of the program worked out, probably to the penny, and they'll have a pretty good idea of what they would have done with that money instead.

A slide full of enrollment numbers or a dashboard someone screenshotted that morning won't hold up in that room, and you need something that will.

What loyalty program ROI actually means to finance

What finance is really asking about is something called incrementality.

Incremental sales are the purchases that happened because of the program and would not have happened without it. When people talk about loyalty program ROI, that's the number finance actually cares about, because it's the only part of your revenue the program can take credit for. We've written about what loyalty ROI should and shouldn't include before, so this post is about how to consider measuring it.

Enrollment, points issued and points redeemed tell you the program has activity happening, and finance knows a busy program and an effective program can look very different.

Members always tend to look better than non-members, and that comparison can fool you

Almost everyone falls into this one. You pull your members and your non-members, compare what they spend, and the members win by a mile. Great, right?

The problem is that your best customers were always the most likely to join. They already liked you. They were already buying. So when members look better, part of what you're seeing is the program, and part of it is just who signed up. Think about who joins first. It's usually the people already buying from you every month, and of course they spend more than someone who bought once two years ago. They'd spend more even if your program didn't exist, so some of that gap was there before the program ever started. That's what people mean by self-selection.

This is an important piece to make note of, because the difference can be big. Researchers studying seven grocery loyalty programs in the Netherlands found that membership did have a real, positive effect on how much of a household's spending went to that store. But once they accounted for self-selection, the effect was about seven times smaller than the simple member versus non-member comparison suggested [1].

A different study of a convenience store program found that the heaviest buyers were the most likely to claim rewards but didn't change how they bought. The light and moderate buyers were the ones who gradually bought more and became more loyal [2].

So keep that big members-spend-more number, and call it what it is, which is your ceiling. The true effect of your program is somewhere underneath it.

Start with the comparisons you can run today

The first one is the member versus non-member view. Look at repeat rate, purchase frequency and average order value, which is the average amount someone spends each time they buy. Present it as the ceiling, and be clear with finance that that's what it is.

The second is a before and after view on the same people. Take members who joined in a given month and compare what they bought in the months before they joined with what they bought after. Now you're comparing people to themselves, which takes a lot of the self-selection out.

The third is a split by type of buyer. Break your members into light, moderate and heavy buyers based on what they did before they joined, and look at each group separately. If your program works the way it did in that one study, you'd expect to see it most in your light and moderate buyers.

A holdout gets you to the real number, if you set it up right

A holdout group is a small set of members, chosen at random, who don't get a particular offer or experience while everyone else does. Because the groups were picked at random, the gap in what they buy afterward is the lift you created. I touched on control groups in this post on parent companies, and these tests usually go wrong in three places.

The first is picking the success metric after the test is over. If you decide what you're measuring once you've seen the results, you'll find something that looks good. Decide up front whether you're measuring repeat rate, gross profit per member or something else, and commit yourself to measuring that metric.

The second is size and time. Each group needs enough people for a real difference to show up, and the test needs to run long enough to cover your normal buying cycle. Collinson notes that organizations normally hold back between 2% and 7% of new members from program communications, and recommends going above 2% when overall volumes are low [3]. So if you're a smaller program, plan on a bigger share or a longer window.

The third is leakage. If holdout members can see the offer anyway, on your site, in a social post or in an email that went to everyone, your results won't be clean. Check every place the offer shows up before you start. This is why loyalty works best as part of the broader brand strategy. When it's run in isolation, things like this muddy the waters fast.

So what about the sales that happen on the shelf?

If most of your volume sells through retailers, most of your buyers are invisible to you. Your syndicated data shows the units moving, and it stops there. So you can't build a member versus non-member comparison across your whole business, because you only know who one of those groups is.

Receipt scanning changes part of that. A member who uploads a receipt gives you a dated purchase at a specific store, and their next receipt shows you whether they came back. That means you can run a holdout on your receipt members and watch what they do over time.

But I want to be really clear about one thing. The people who scan receipts are usually your most engaged buyers. So your member numbers are going to look even better than your average shopper. Treat them as a ceiling too, and don't present them as your whole market.