Machine learning lab · notes

How the movie night arbiter works

Back to the toolMachine Learning LabSource on GitHub

The group problem

Households lose a surprising amount of time choosing what to watch, surveys put it around 110 hours a year, and roughly 38% of couples say they bicker over it. Swipe apps such as Matched, WatchPick and Swyf only surface a film once everyone swiped yes on it. They cannot say how much someone would enjoy a film nobody asked them about.

The group recommender literature splits the job in two. First fill in each person's missing ratings with an ordinary recommender. Then combine the filled in rows with an aggregation rule that decides what fair means.

Filling the gaps

A grid of people by films where most cells are empty. The job is to fill in the blanks well enough that the highest guesses are films that person would actually enjoy. No genres, no plot summaries, only who rated what. This is collaborative filtering: people who agreed before will probably agree again.

Two to six people is far too few to learn from, so each person joins as a new row after 60 seed users who act as a backbone crowd. A household of three trains 63 rows. Films a person rated keep their real rating, every other cell uses the model's prediction.

Aggregation rules

With a full row of scores per person, each candidate film gets an average and a lowest score. The three rules rank on those.

  • Fair, average without misery. Drop any film where someone is predicted below 3, then rank by average. If nothing survives, fall back to the highest lowest score and say so.
  • No veto, least misery. Rank by the lowest score, so the least keen person is as happy as possible.
  • Crowd-pleaser, plain average. Highest mean wins, even if one person sits at 2.

By default films anyone already rated are skipped, since a rating means they have seen it. Each pick carries a confidence label from the rating count of the people whose score is a guess: under 3 ratings is low, under 8 is medium.

The data is synthetic

There is no real shared ratings log here, so the 60 seed users are invented. Each of the 40 films gets a hand written position on three hidden axes: spectacle, darkness and how cerebral it is. Each user gets a random taste on the same three axes, a leniency, and each film a random acclaim.

rating = round(3.3 + leniency + acclaim + 0.8 · (taste · film) + noise), noise ~ N(0, 0.55²), clamped to 1..5

Each user sees each film with a film specific probability between 25% and 65%. A seeded generator makes it deterministic: 1,047 ratings, 43.6% of the grid filled, 10 to 25 ratings per user, at least 12 per film, training mean 3.39. The model never sees the hidden axes. Noise plus rounding sets a floor: even the true noiseless score only reaches 0.613 RMSE on the held-out set.

The model

Every user u and every film i gets a bias and a vector of k numbers. A prediction is the global mean, plus both biases, plus how well the two vectors line up.

r̂ui = μ + bu + bi + pu · qi

Training minimises squared error on known ratings, with an L2 penalty that keeps every parameter small.

L = Σ(u,i) known (rui − r̂ui)² + λ (bu² + bi² + ‖pu‖² + ‖qi‖²)

Each person in the household becomes a row after the 60 seed users, and the same training loop learns their vector alongside everyone else's.

Training loop

Plain stochastic gradient descent, written by hand. Each epoch shuffles the known ratings with a seeded generator and, for each one, nudges the four parameters it touches.

e = rui − r̂ui
bu ← bu + η (e − λ bu)
bi ← bi + η (e − λ bi)
pu ← pu + η (e qi − λ pu)
qi ← qi + η (e pu − λ qi)

Vectors start as small Gaussian noise, biases at zero, μ fixed at the training mean. The page runs one epoch per animation frame so the map moves while the tab stays responsive, and jumps straight to the end when reduced motion is on. Any non finite parameter stops training and flags divergence.

The map is the film vectors projected onto their two main directions, found by power iteration on the uncentred k by k second moment matrix. Axis signs are matched to the previous frame so the map does not flip. Your point is your own vector projected the same way and drawn at a fixed radius, one per person: its direction is what matters, films that lie along it score highest for them.

Hyperparameters

Defaults are k 2, learning rate η 0.02, λ 0.10, 120 epochs. A larger k can memorise more, a larger λ pulls everything toward the bias only model, and a larger η converges faster until it overshoots.

Evaluation

A seeded 20% of the seed ratings (212) is held out and never trained on, leaving 835. All numbers below come from running the same TypeScript in Node, epoch 120 unless noted.

ModelTrain RMSEHeld-out RMSE
Global mean1.0591.076
Bias only (user + film)0.8760.970
Matrix factorization, k 20.5460.753
Noiseless truth (floor)·0.613

Factorization cuts held-out error by 22% against the bias only model and 30% against the global mean. Bias only uses damped averages, a film bias first and then a user bias on the residual.

Setting, η 0.02TrainHeld-out
k 2, λ 0.100.5460.753
k 3, λ 0.100.4840.787
k 4, λ 0.100.4400.790
k 8, λ 0.100.3400.797
k 8, λ 00.1670.993

The data has three true axes, yet k 2 wins. With about 14 training ratings per user the third axis costs more in variance than it buys. Turning λ off at k 8 shows overfitting plainly: train error falls to 0.167 while held-out climbs back to 0.993, after bottoming out at 0.851 on epoch 31.

Failure modes and limits

  • Cold start. With only a few ratings a person's vector stays near zero, so their predictions fall back to how well liked each film is. The confidence label says so.
  • Missing at random by construction. Real people rate what they chose to watch, which skews ratings upward and hides dislikes.
  • One split, one seed. Differences of a few thousandths between settings are within noise for 212 held-out ratings. The held-out set is crowd ratings, so it says nothing direct about how well the model knows your household.
  • The household is a handful of rows among a synthetic crowd and a 40 film catalog, so picks only ever come from those 40. It trains only while the page is open; nothing leaves your browser except to localStorage.
  • The 2D map drops every direction past the first two, so neighbours on the map can still disagree.
  • A high learning rate with low λ can diverge; reset with smaller values.

With real shared data

  • Train on a public log such as MovieLens and compare against these synthetic numbers.
  • Split by time, not at random, so the test asks about the future.
  • Tune k, η and λ on a validation split, with early stopping, and report the test set once.
  • Add implicit feedback (what people watched, not only what they rated) and ranking metrics such as recall at 5.
  • Fold in a new user with a few closed form least squares steps on their row instead of retraining everyone.