The most dangerous output in analytics is a confident, wrong “why”. A dashboard tells you retention dropped three points. A causal tool goes further and tells you onboarding did it. The second is far more useful, and far more dangerous, because a wrong “why” sends a team to fix the wrong thing with total conviction. So the only question that really matters about any causal claim, ours included, is: how do you know it is right? Most attribution features cannot answer that. Here is how we answer it, and how you should interrogate any causal method before you let it drive a decision.

Why causal attribution is hard to validate

A metric is straightforward to validate. Recompute it, compare, done. A causal claim has no such luxury, because the ground truth is a counterfactual you never get to observe. To know whether onboarding caused a three-point retention drop, you would need to see the same cohort in a world where onboarding did not change, and that world does not exist. Observational data does not contain the answer key.

That is why so many “why” features are, in the strict sense, unfalsifiable. They emit an attribution, it sounds plausible, and there is nothing to check it against. Plausible and unfalsifiable is exactly the combination that gets a team to act on a wrong answer. So validation cannot be an afterthought. It has to manufacture ground truth on purpose. We use two sources, one synthetic and one real.

Injected effects: synthetic ground truth

Start from real data, then inject an effect of known size, sign, and cause. Take a period where little moved, and synthetically degrade onboarding for a defined slice of players by a known amount, leaving UA and core-game untouched. Now you hold the answer key: a move that is entirely onboarding, of a size you chose. Run the decomposition and check three things.

Direction. Does it attribute the move to onboarding, rather than to UA or core-game?
Magnitude. Does the size it reports match the size you injected, or is it systematically off?
Specificity. Inject nothing at all, and it should find nothing beyond noise. A method that “discovers” a driver in data where you planted none is worse than useless.

Two things make this a real test rather than a rigged one. The injection is applied to raw player behavior, upstream of the panel the model reads, not to the model’s own fitted inputs, so recovery is never guaranteed by construction. And because the drivers move together in the base data, attributing the change to onboarding and not to the drivers that correlate with it is genuinely hard, not a formality. Within that, injected effects are cheap, repeatable, and adversarial: you can sweep effect sizes, place the effect in each driver in turn, add measured confounders and see where the method bends. The bar is explicit and public: plant a known cause upstream, and the method has to find that one and not the others.

A/B recovery: real-world ground truth

Injection tests the method against a world you built. A/B recovery tests it against the real one. When a studio has run an actual experiment, the A/B result is a genuine, measured causal effect, the closest thing to a gold standard there is. Run the observational decomposition over the same population and window, and check whether it lands inside the experiment’s confidence interval, on both sign and magnitude. The estimands have to line up for that to mean anything: the natural move we decompose has to be the same change the experiment deliberately made. And a single match could be luck, so what counts is recovery across every experiment a partner has run, not one lucky point.

Each experiment the method recovers widens the set of conditions under which it has been checked. That is evidence its identification assumptions held there. It is not a guarantee the method will be right for a different move with a different confounding structure, a UA-mix shift is not an onboarding change, and we do not present it as one.

The un-testable case is the whole reason the method exists. You cannot A/B test most of what moves a live game. You did not randomize the onboarding tweak, the content patch, and the UA-mix shift that all shipped the same week. A/B recovery is how a method earns the right to answer those un-testable questions: by matching the experiment in the places where an experiment happens to exist.

The precondition: point-in-time correctness

None of this is worth anything if the inputs cheat. A decomposition built on player properties computed with hindsight, spend-to-date that quietly includes tomorrow’s purchases, a segment assigned using the future, will sail through a naive backtest and then lie in production, because the leaked information is not there when you actually need the answer. So the decomposition sits on a point-in-time-correct panel: every comparison resolves player state as of the day it describes, with no future information, correct by construction.

This is also the honest answer to “couldn’t I build this in dbt”. You can build point-in-time-correct properties in dbt if you are careful. But that is the input to the hard part, not the hard part. The causal method on top, and the discipline of proving it recovers known effects, is the work, and it is the work that a hand-rolled model almost never gets held to.

What we do not claim

Retention decomposition is in validation with design partners now, not generally available. We hold it to these tests before it is allowed to answer a real “why”, and we say “validated against ground truth”, never “perfect”: known edge cases are still being refined, and we would rather name them than paper over them.

The method is also built to abstain. When the data cannot support an attribution, too few players, or measured confounders it cannot hold apart from a driver, it says so instead of guessing. It cannot rule out confounders you never recorded, and it does not pretend to: that assumption is stated, not buried. A causal method that never returns “not enough evidence” is not confident, it is uncalibrated, and sooner or later it hands someone a clean-looking wrong answer.

We publish the methodology so you can check it rather than take it on faith. The causal decomposition page has the approach and the launch announcement, with the case studies as we publish them. If you want to watch these checks run against your own retention move, book a demo: 45 minutes, no sales loop, and we will do exactly that on your data.