docs/eval-scorer.md was notes-to-self for work already finished. It told the reader to
add a class to eval_harness.py that has been there for months, specified the author's
own GPU by model number, and carried escaped-markdown artifacts from a bad paste. The
feature is real and was used — 9 eval runs, 668 questions, 3536 judgments — so the fix
is documentation, not deletion.
It now explains what coverage scoring measures and why it is worth having alongside
the judge model: the judge catches claims a source contradicts, the cross-encoder
catches claims a source simply does not cover. Includes how to read the number, and
the query that finds the interesting cases — paragraphs the judge passed but the
scorer did not, which is where technically-defensible-but-misleading answers show up.
The scorer defaulted to port 8001, which is also the MCP server's default, so running
both meant one silently failed to bind. Moved to 8005 and documented.
SCORER_ENDPOINT is now in .env.example, and the README has a documentation index —
every file under docs/ was previously unreachable from anywhere in the repo.
Seeded via an explicit allow-list (see /home/lasse/plenum-seed.sh) rather than by
deleting files from a copy, so nothing sensitive can survive by omission.
Excluded: WireGuard backup + client config, the plaintext DB password in admin.py,
Arango credentials in scripts/notes.md, .claude/settings.json, a 113 MB log,
providers.yaml (private endpoint), the Arango/ChromaDB-era scripts, the duplicated
claude-design-system frontend copy, and assorted screenshots and one-off planning docs.
297 tracked files / 49 MB of history -> 161 files / 2.1 MB.
Recovered 14 database migrations that the old .gitignore's `*.sql` rule had been
hiding from version control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lasse Edfast