docs/eval-scorer.md was notes-to-self for work already finished. It told the reader to
add a class to eval_harness.py that has been there for months, specified the author's
own GPU by model number, and carried escaped-markdown artifacts from a bad paste. The
feature is real and was used — 9 eval runs, 668 questions, 3536 judgments — so the fix
is documentation, not deletion.
It now explains what coverage scoring measures and why it is worth having alongside
the judge model: the judge catches claims a source contradicts, the cross-encoder
catches claims a source simply does not cover. Includes how to read the number, and
the query that finds the interesting cases — paragraphs the judge passed but the
scorer did not, which is where technically-defensible-but-misleading answers show up.
The scorer defaulted to port 8001, which is also the MCP server's default, so running
both meant one silently failed to bind. Moved to 8005 and documented.
SCORER_ENDPOINT is now in .env.example, and the README has a documentation index —
every file under docs/ was previously unreachable from anywhere in the repo.
Lasse Edfast
8a0d6b6b3d