How Box Office Verdict works

The problem

A newsletter writer tries two subject lines and gets 12 clicks from one and 9 from the other. A shop owner tests two landing pages. A job seeker sends two versions of a resume. All of them want one answer: is the difference real, and what should I do next?

The usual significance test is a poor fit at this size. Low traffic guides suggest around 1,000 visitors and 100 conversions per version before it settles, and checking the result every day quietly inflates false wins: peeking 10 times can push a 1% false positive rate to about 5%. Free Bayesian calculators report the chance each version is best, but stop there. None says how to split traffic next or when it is safe to stop. Google Optimize, the free default, shut down in September 2023.

Beta-Binomial

Treat each version's conversion rate p as unknown. Start from a flat Beta(1, 1) prior, which says every rate from 0% to 100% is equally plausible. After c conversions from n visitors the belief is exactly

That is conjugacy: the Beta prior times the binomial likelihood is another Beta, so updating is two additions. The belief chart draws these curves. The odds strip's 95% range is the middle 95% of each curve, read off the sorted draws below.

Probability to be best

The tool draws 20,000 plausible rates from every version's curve, using a seeded generator so the same numbers always give the same answer. In each round of draws one version comes out on top. The share of rounds each version wins is its chance of being best:

This is a direct answer to the question people actually ask, and it works the same for 2 versions or 6.

Expected loss

Chance to be best ignores how much is at stake. A 70% favourite that could only be worse by a hair is safe to ship; one that could be worse by two points is not. Expected loss measures that: in every round, how far this version falls short of the round's winner, averaged over all rounds.

The decision rule

You set the smallest lift you care about, in percentage points. Call it the bar. Then:

• Too close to matter if every version's expected loss is under the bar. Any choice costs less than you care about.
• Ship the version with the lowest expected loss if its loss is under the bar.
• Keep running otherwise.

Check: 12 conversions from 200 against 9 from 200. Numerical integration of the two Beta curves gives A a 74.43% chance of being best and an expected loss of 0.353 points. The tool's 20,000 draws give 74.5% and 0.348. Both are far above a 0.1 point bar, so the verdict is Keep running, send 75/25.

The next split

While the test runs, each version gets tomorrow's traffic in proportion to its chance of being best. That is exactly what Thompson sampling does one visitor at a time: draw one rate per version, show the highest. Over a day, the shares match the probabilities. Clear losers fade out, and versions that might still win keep enough traffic to prove it.

Why the bandit split

The next split comes from a multi-armed bandit. Picture a cinema with one marquee and five candidate posters, each with a click rate nobody knows. Show the current favourite and you might never discover a better one; keep trying others and you waste visitors. The demo below the tool races three rules for that trade off, measured by regret.

Regret

Regret is how many clicks we lost against an oracle that always shows the best poster. With true rates μ₁ … μ_K, best rate μ* and the arm a_t we chose at step t:

This is expected regret, using the true rates rather than the noisy clicks, so the curves in the demo are smooth. A strategy that keeps exploring forever has regret that grows linearly. Good strategies grow roughly with log T.

Beta-Bernoulli and the Thompson rule

A click is a coin flip with unknown bias μ. Start every poster with a flat Beta(1, 1) prior. After s clicks and f skips the posterior is exactly

That is conjugacy: the Beta prior times the Bernoulli likelihood is another Beta, so an update is two integer increments. The curves in the demo are these densities. Narrow means confident, wide means unsure.

Thompson sampling then does one thing per visitor: draw one sample from every poster's posterior and show the poster with the highest draw. A poster gets shown in proportion to the probability that it is the best. Unsure posters sometimes draw high and get explored; clearly weak ones almost never do. No tuning knob.

The two baselines

ε-greedy shows each poster once, then with probability ε = 0.1 shows a random poster, otherwise the one with the best observed rate s/n. Simple, but it explores at the same rate forever.

UCB1 shows each poster once, then picks the highest optimistic score, where n_i is pulls of poster i and t is total pulls:

Sampling a Beta from scratch

JavaScript has no Beta sampler, so the code builds one. If X ~ Gamma(a) and Y ~ Gamma(b) are independent, then X / (X + Y) ~ Beta(a, b). Gamma draws use Marsaglia and Tsang's method: with d = a − 1/3 and c = 1/√(9d), draw a standard normal x (Box-Muller), set v = (1 + cx)³, and accept dv when log u < x²/2 + d − dv + d log v. For shape below 1 it samples shape + 1 and scales by u^1/a. Simulations use a seeded mulberry32 generator so every run is reproducible.

Check in node, 200,000 draws each: Beta(2, 5) gave mean 0.2856 against 0.2857 and variance 0.02551 against 0.02551; Beta(30, 400) gave 0.0697 against 0.0698. The density curves use a Lanczos log-gamma, and Beta(2, 5) integrates to 1.0000.

Measured results

Default rates 4%, 6%, 9%, 5% and 3%. Average expected regret over 200 seeded runs per strategy, ± one standard error, measured by running this page's own code in node:

At 1,000 visitors Thompson and ε-greedy are tied. By 10,000 Thompson has stopped paying for exploration while ε-greedy still spends a tenth of its traffic at random. UCB1 does poorly here because its bonus assumes rewards spread across 0 to 1; with click rates under 10% the bonus dwarfs the gaps, so it keeps exploring. The random row is exact arithmetic, not simulated.

Limits

Peeking still costs you. Bayesian numbers are not immune to stopping the moment they look good. Decide the bar before you start, and treat a verdict on the first day as provisional.

Yes or no outcomes only. The model counts conversions. It cannot compare revenue, order size or time on page.

Run whole weeks. Visitors on a Sunday are not visitors on a Tuesday, and the model assumes each rate stays fixed. Stop on the same weekday you started.

Keep audiences comparable. If one version got newsletter traffic and the other got ads, the difference is the audience, not the version.

Delayed feedback. Callbacks and purchases arrive late. Count only visitors whose outcome window has closed, or recent visitors will drag every rate down.

Nothing is stored. The numbers live only in the page's link. There is no account and no server.