How much to trust the estimate
What the badge next to an article title on the bandit dashboard means, and why a small sample is not a stable number yet.
TL;DR
The badge answers: can we already trust the estimate for the current reward mode (conversion, depth, or revenue).
It does not say “is this article better than the leader?” It says how likely the number will barely move after another batch of visits.
Same conversion, different sample sizes
| Visits | Purchases | CR | Intuition |
|---|---|---|---|
| 100 | 3 | 3% | Still noisy — tomorrow it may be 2% or 4% |
| 10 000 | 300 | 3% | Mostly settled |
Same 3%, different confidence. That is what the badge measures.
Badge statuses
| Badge | Meaning |
|---|---|
| too little data | Not enough to judge |
| too early · N% | Noisy — do not lean on it yet |
| moderate · N% | Signal exists, uncertainty remains |
| trustworthy · N% | High chance the number holds after more visits |
The percent is the probability that after the check horizon (default 1000 more visits) the estimate shifts by at most the allowed shift (default about 0.5 percentage points for conversion).
Horizon and allowed shift are configurable in bandit settings.
Depends on reward mode
The badge uses the same metric as the Effectiveness column:
- Purchase conversion — stability of conversion
- Depth — stability of mean funnel reward
- Revenue — stability of revenue-mode mean reward
Changing the mode changes both effectiveness and the badge meaning.
How it is computed (plain language)
- The article already has accumulated stats (same Beta used by Thompson Sampling).
- We ask: “if N more visits arrive, where can the estimate move?”
- We compute the probability it stays nearly next to the current value (inside the allowed shift).
More data (and a stabler signal) → higher percent.
Method and references
We use a Bayesian forecast with a Beta–Binomial model (aligned with the bandit’s Beta parameters).
“Allowed shift” follows the idea of a region of practical equivalence (ROPE): movement inside the region counts as practically the same value.
- Beta-binomial distribution (Wikipedia)
- Region of Practical Equivalence (ROPE) — bayestestR
- Kruschke, 2018 — Rejecting or Accepting Parameter Values in Bayesian Estimation
Technical details (optional)
After (N) more visits, future mean (\mu') is random (Beta–Binomial).
Click the badge for a plain-language breakdown with your numbers; (\alpha/\beta) are under “for the curious”.