What is a holdout group, and why use one for retention?

A holdout group is a random share of your customers that you deliberately leave out of a change, so you can compare them with the customers who got it. It is the only way to say that a retention fix caused the result. Before-and-after charts cannot do that, since prices, seasons and customer mix also move.

What is a holdout group?

You pick a share of customers, say one in five, and make sure they receive none of the new emails, offers or flows. Everyone else gets the change. After a fixed period you compare the two groups on the same metric, using the same definitions.

The group is also called a control group. The idea is simple. The difference between the two groups is what the change did, because everything else, the season, your price, your product, applied to both.

Why is a before-and-after comparison weak?

Because too many things change at once. Suppose your recovery rate rises after you start new failed-payment emails. It might be the emails. It might be that a batch of annual cards renewed, that you changed a price, or that a different customer mix came in this quarter. You cannot tell.

A holdout cancels all of that out. It does not need you to know what else changed.

How big should the holdout be?

In my pilot the holdout is 20% of customers. I chose a share large enough to collect meaningful events in the control group and small enough that most customers get the fixes. The right size for you depends on how many events you get.

The sample pilot plan in my report shows expected events for each arm at the account's current volume. For the fictional company, the failed-payment arm would have about 87 invoices for the treated group and 22 for the holdout. Those are synthetic numbers, but the idea is real: the small arm sets how long the test has to run.

How do I pick who is in it?

Randomly, and in a way that does not change. I use a stable hash of the customer id. That means a customer who is in the holdout today is in the holdout tomorrow, and nobody moves between groups because of what they did. A stable assignment prevents a quiet bias: if you moved people who had just failed a payment into the treated group, the groups would stop being comparable.

Do not pick the holdout by hand. Do not pick it from the customers who seem less engaged. Both bias the result.

What do I measure?

Pick one primary metric before you start, and write it down. I use net revenue per customer, treated against holdout: revenue times gross margin, minus the cost of the offers given. For narrower questions I also track the recovery rate on failed payments, the save rate at cancel and trial-to-paid conversion, all on the same definitions as the report.

A verdict comes from a two-proportion test at 95% for the rate metrics. I report a group with fewer than 30 events as an early signal, not a result.

What mistakes spoil a holdout?

• Peeking and changing the rules. If you look at the result every few days and change the message or the split when you see something, the comparison stops being valid.
• Contaminating the holdout. If holdout customers get the same email from another tool or from support, the groups are no longer different.
• Changing the metric afterwards. Choose it first.
• Stopping too early. A few days of data is noise.
• Ignoring what offers cost. Subtract their cost, or a discount that wins customers will look better than it is.

Does it cost anything to hold customers out?

Yes, and I should say so. The customers in the holdout do not get the fixes. If the fixes work, you gave up some recovered revenue during the test. That is the price of knowing. For me it is also what makes the pilot fair: I am paid 20% of measured uplift, net of offer costs, and uplift is the gap between the two groups. Nobody has to trust my word that it worked.

The pricing is in how a pay-on-uplift retention pilot works.

Can I run one myself?

Yes. Pick the split with a hash of the customer id, write down the metric, apply the change to the treated customers only, and do not touch the rules until the period is up. A spreadsheet is enough. Many small teams skip the holdout because it feels wasteful and then cannot say whether their work paid.

I used the idea for recovery in how to recover failed payments on Stripe subscriptions, and for trials in unconverted trials.

How long should the test run?

Long enough for the smaller arm to collect a useful number of events, and for the effect to show. My pilot runs 90 days. Fewer than about 30 events in a group means an early signal at most. If your volume is low, a longer test is the only fix; making the holdout smaller makes the problem worse, not better.

What does it have to do with the report?

The report is a read of history, and it does not prove that a fix works. It tells you where the money is. The holdout is what proves a fix. If you buy the report for $490 and want to test its three fixes, the next step is a pilot with a holdout, or a version of one you run yourself.