The locked holdout I had to run twice — and why both runs are public

I had built one evaluation for ContextLens that was allowed to run exactly once. Its first run came back in line with development on Wikipedia. Its headline Stack Exchange score was wrong in a way no model can be wrong: it covered 472 questions, and not one came from a Technology or Science site — although four of the eleven sites behind that score are about AI, software engineering, quantum computing and the history of science.

The model hadn't forgotten two of its eight topics. One view of the test set had lost them.

ContextLens is a local topic analyser for conversations: each English message gets one of 8 general topics and a primary subtopic out of 28, or uncertain when it's off-topic. This post is about its final test set — the one you only get to look at once — and what I did the day it came back broken: fixed the data offline, proved the model hadn't moved, froze again, ran again with a written reason, and published the broken run next to the good one.

Version 1.0 had test splits. An audit of v1.0 named the problem in four words: no untouched final holdout. Those splits had been computed during development and had appeared in benchmark tables while choices were still being made. A number you have looked at while choosing things is no longer blind.

So for v1.1 I built data that no development script reads, and committed it before the v1.1 model was chosen and trained:

• 374 Wikipedia passages from articles in no version of the corpus. The builder walks every committed version of the label manifest in git history, not just the current one.
• 2,713 Stack Exchange questions created between 2026-01-01 and 2026-09-20. The v1.0 sets were MTEB clustering titles and all-time top-voted API questions, so a 2026 window yields mostly new ones, and any question already in an evaluation set was dropped.
• 4,350 CLINC150 test utterances for off-topic chat, without CLINC's own oos class and five intents that could be on-topic, and the locked half of a Tatoeba set for the language gate.

A manifest records the row count and SHA-256 of the two built files, plus a hash of the CLINC150 and Tatoeba locked halves. Then comes the lock. scripts/freeze.py refuses uncommitted changes to tracked files and writes reports/locked/FREEZE.json: the commit, the runtime settings and the SHA-256 of six files — model config, taxonomy, runtime config, training passages, the locked manifest and the artifact's metadata.json, which itself holds a checksum for every model file, down to each encoder weight file. One hash stands for the whole trained model. evaluate.py --stage locked checks all of it before loading the model:

def check_freeze(model_dir: Path, rerun_reason: str | None) -> dict:
    freeze_path = LOCKED_DIR / "FREEZE.json"
    if not freeze_path.exists():
        raise SystemExit("no reports/locked/FREEZE.json - run scripts/freeze.py before the locked evaluation")
    freeze = json.loads(freeze_path.read_text(encoding="utf-8"))
    now = freeze_fingerprint(model_dir)
    changed = [k for k, v in freeze["files"].items() if now.get(k) != v]
    if changed:
        raise SystemExit(f"configuration changed since the freeze: {changed}")
    results = LOCKED_DIR / "results.json"
    if results.exists() and not rerun_reason:
        raise SystemExit("the locked holdout was already evaluated; pass --rerun-reason to record a second run")
    return freeze

Three refusals: no freeze file, a changed fingerprint, a second run without a reason. The last one reads like a formality. This whole post is about the situation that line was written for, and in the end it never fired.

The Stack Exchange questions serve two evaluations, and that is where the bug lived.

The general view asks whether the model gets the general topic right. Every question from eleven sites is labelled by its site: physics and astronomy are Physics; ai, softwareengineering and quantumcomputing are Technology; hsm (history of science and mathematics) is Science.

The subtopic view asks the same about the subtopic. Its questions come from (site, tag) queries — physics tagged quantum-mechanics is Quantum Mechanics — and four subtopics map to a whole site with no tag at all: AI (ai), software (softwareengineering), quantum computing (quantumcomputing) and history of science (hsm).

A question can be in the general view, the subtopic view, both, or neither. Those are two independent facts. The build script stored them as one value:

# scripts/build_locked_sets.py as it was for run 1 (removed in commit 9613200)
# runs for every (site, tag) query of every subtopic; g is its general topic
row = rows.setdefault(
    key, {**q, "set": "subtopic", "general": g.id, "subtopics": [], "is_ood": False}
)
row["set"] = "subtopic" if row["set"] == "subtopic" or row["general"] == g.id else row["set"]

The general sites were fetched first and their questions marked general; then came the subtopic queries. A question first seen there was marked subtopic, and one already stored as general was flipped to subtopic whenever the topics matched. Either way, a question from one of the eleven sites ended up outside the general view, although its site said it belonged there.

On the six classes that kept any questions, that cost between 31 and 75 percent of them; Physics was left with 164 of its 653. On the four whole-site subtopics it cost every question, because the subtopic query was the site query — same site, same dates, same sort order, no tag. It returned exactly what the general query had stored, and everything flipped: in the run-1 locked file, all 399 questions from ai, softwareengineering, quantumcomputing and hsm carry "set": "subtopic". Those four sites are the whole of Technology and Science in the general view, which is why exactly those two classes went to zero.

Run 1 did grade those questions, just in the other view: the same run, with the same model, scored 317 Technology questions at an F1 of 0.862 in the subtopic view.

Nothing crashed, and the headline wasn't absurd: accuracy 0.735, macro-F1 0.563. That reads like a weak model. It was an average over two empty classes: across the six classes that had questions, macro-F1 was 0.751, and Technology and Science entered the mean as zeros.

The tell is the shape of the data: 472 questions and two classes with no support, where the development set of real questions (ext_dev) has 1,500 Technology and 951 Science questions. And the damage spread. Both off-topic AUROCs use the general view as their in-domain reference, so both moved: 0.942 → 0.961 on assistant chat, 0.838 → 0.886 on questions from off-topic Stack Exchange sites. One broken view, and every number built on it was wrong.

The repair had one constraint: change no question, no label rule and nothing in the model — only which view each question belongs to. Nor did it need a new definition: it restored the rule the v1.0 general set had always used, where a question is in the general view if its site is, and its site gives the label. The subtopic view had never read the set field — it was always every in-taxonomy question with a subtopic — so the general view only needed a flag of its own:

for r in rows:
    r.pop("set", None)
    r["in_general"] = r["site"] in site_general or r["site"] in ood_sites
    r["site_general"] = site_general.get(r["site"])

Off-topic sites join the general view unlabelled, as the negatives for the off-topic evaluation. A new --repair mode re-derives the flags from the saved se_locked.jsonl without a single network request, so no question could be re-fetched, added or lost. On the evaluation side one line changed: the general view became every question with in_general, labelled by its site.

Run 1 didn't disappear. Its results were committed as reports/locked/results_run1_partition_bug.json, next to its raw predictions and its _run1 figures, and D-36 in the decision log explains what happened. Then I froze again. This is the whole difference between the two freezes, with unchanged lines trimmed and hashes cut to twelve characters:

# git show dfc5e5e -- reports/locked/FREEZE.json
-  "frozen_at": "2026-09-24T19:33:30+00:00",
-  "commit": "f0f1b0638887…",
+  "frozen_at": "2026-09-24T19:36:51+00:00",
+  "commit": "96132001674c…",
-    "data/locked/MANIFEST.json": "78f3cb1b90ec…",
+    "data/locked/MANIFEST.json": "30a2f6d55aac…",
     "models/contextlens-topic/metadata.json": "daf72ce77472…"

The commit and timestamp moved; the manifest moved because the view flags inside the locked file changed. The metadata hash did not — and that hash is the model.

Run 2 passed the same fingerprint check, and its results file says why there is a run 2 at all: "run 2 after fixing the locked Stack Exchange view partition bug (D-36); model and configuration unchanged".

The strongest evidence is in the raw predictions. The Wikipedia file and both off-topic files are byte-identical across the runs. All 472 questions that run 1 scored in the general view reappear in run 2 with the same predicted label, the same confidence and the same uncertain flag; only the off-topic score of 100 of them moves, by at most two parts per million — the float noise you get when the same texts are embedded in different batches. Run 2 didn't grade the model differently. It graded 1,151 more questions: the ones run 1 had left out of the general view.

Nothing forced me to publish run 1. It was a data bug, the model never changed, and deleting one file would have left a tidier story. It stayed for three reasons.

It makes the claim checkable. "The model didn't change between runs" is exactly what a careful reader should distrust, because it's also what someone would write after quietly re-running until the number looked better. With both runs committed, nobody has to take my word for it.

It documents the bug better than prose can. Run 1's 472 questions and two empty classes, next to run 2's 1,623 and all eight, show in one table what an exclusive field does to a dataset.

The guards couldn't have stopped me. freeze.py refuses to write a new freeze while reports/locked/results.json exists, and its message says why: "a new freeze would hide that". To freeze again I had to move run 1's results aside; they went into the same commit that fixed the builder, under their new name. That move also disarmed the rerun guard: with no results.json on disk, evaluate.py doesn't ask for a reason. I passed --rerun-reason anyway. The guards stop an accident; they can't stop a decision. What makes run 2 legitimate is the record around it: the archive is a commit, the reason is in the JSON, the decision log has D-36, and KNOWN_ISSUES.md lists "Locked evaluation was run twice" as issue 12, severity low, disclosed.

The seal is strict in small ways, too. After the locked run I renamed the GitHub repository once more, to ContexLens, but the web search still sends the intermediate name, ContexLens-NLP, in its User-Agent (GitHub redirects old names). The string lives in contextlens/config.py, a fingerprinted file, and a cosmetic URL isn't worth breaking a freeze.

A holdout you can quietly re-run is just a validation set with better manners.

A sealed test set certifies the choices made before the seal. Make them on the wrong data, and the lock certifies the wrong answer with a straight face.

ContextLens tunes hyper-parameters on Wikipedia validation but picks model families on validation and on real Stack Exchange questions (ext_dev), because users type short questions, not encyclopedia paragraphs. In this project the second set never had to overrule the first: when I chose the encoder, on the v1.0 labels, the fine-tuned MiniLM led on both. The benchmark rerun on the final corpus — after the freeze, development splits only — shows what the second set is insurance against:

¹ Re-measured on an idle machine; the benchmark run had caught e5-small under load.

On Wikipedia the top two tie, and a tie can't choose: the argmax picks mpnet-base, about five times slower and larger. On real questions the tie breaks, and not only because of fine-tuning — among the frozen encoders alone, mpnet-base drops from first on Wikipedia to behind e5-small and bge-small, models a third its size. It is better calibrated than the fine-tuned MiniLM on both sets, but calibration is the protocol's last tie-breaker, after latency and size, and those two break the other way. The model report puts it in one line: choosing on Wikipedia alone would pick the wrong, slowest model.

These are run 2's numbers, the ones the project reports:

The off-topic gate answers uncertain for 78.1% of CLINC150 assistant chat (AUROC 0.961) but for only 48.5% of questions from off-topic Stack Exchange sites (AUROC 0.886), and a message takes 19.9 ms at the median on a 4-vCPU machine.

The weak spot is in plain view: Science on real questions in the general view scores an F1 of 0.241. All 99 Science questions in that view come from hsm, and 42.4% of them are predicted Physics — "Why Euler didn't include viscosity in equations of fluid dynamics?" goes to Physics at 0.963. Since taxonomy 1.2.0, Science means the scientific enterprise itself — method, research practice, the history of science as such — so that question is arguably Physics, and hsm is reported separately rather than quietly dropped. The project's own rule measures Science on questions in the subtopic view, where it scores 0.420 on 143 questions. What I didn't do is the one thing the lock exists to prevent: see 0.241, go back and tune it up. A second general topic for history-of-a-field questions ("Physics + history") is in the backlog, explicitly not planned for v1.

• It's a lock I hold the key to. The fingerprint proves the files didn't change between freeze and run, not that I never opened the locked files another way. What stands behind it is the public commit history, and history is evidence, not proof: this repository's history was rewritten once, before v1.1, to drop a superseded model artifact, and a commit records that.
• Nothing re-checks it on its own. The CI workflow is manual-only, its steps run locally through scripts/ci.sh, and no test compares the committed files with FREEZE.json.
• The sets are small. With 29 to 60 Wikipedia passages per general topic, single-class numbers carry wide uncertainty, and three subtopics — periodic table, quantum computing, scientific method — have no locked Wikipedia passage at all.
• The headline table doesn't mention run 1. The README's locked table says "evaluated once after the freeze". A reader who never opens D-36 or the known-issues list won't see run 1, and the disclosure should sit next to the number.
• Nothing was re-run for this post. Every number here comes from committed files. The only things I computed while writing were a byte-level comparison of the unchanged prediction files, a row-level match of the 472 questions, and counts over the committed files.

1. Build and commit the locked set before the decisions it will judge.
2. Fingerprint everything that defines the system, including the model's own checksums, and refuse to run on any mismatch.
3. Store group membership as independent flags, and check per-class support before computing a single score.
4. If you have to run again: fix offline, freeze again, archive the broken run, and write the reason into the results even when the tooling doesn't ask.
5. Put the disclosure in the table people quote, next to the number.

The decision log, both results files and the raw predictions of both runs are in the ContexLens repository under Apache-2.0.

A locked holdout isn't a promise that you will never run it twice. It's a promise that if you do, everyone will be able to see why. Break the seal when the data is wrong, never because the number is — and leave the broken run where anyone can read it.