plenum/docs/eval-scorer.md

Citation coverage scoring

An optional add-on to the evaluation harness. It measures how well a cited source actually supports the paragraph citing it, filling in eval_judgments.coverage_score.

Everything else in the harness works without it. If the scorer is unreachable the run completes normally and coverage_score is left NULL.

Why it exists

The harness already asks a judge model whether a paragraph is supported by its citations. That is a generative call: slow, and it answers in prose that has to be parsed.

A cross-encoder answers a narrower question — how relevant is this source to this text? — as a single number. It reads the source and the paragraph together and emits one score with no generation at all, so it runs roughly an order of magnitude faster and costs almost no VRAM.

The two measure different things and are worth having together: the judge catches claims a source contradicts, the scorer catches claims a source simply does not cover.

Running it

Any endpoint implementing OpenAI's /v1/score or /v1/rerank will do. With vLLM:

docker run --gpus all -p 8005:8000 --name plenum-scorer \
  vllm/vllm-openai \
  --model BAAI/bge-reranker-v2-m3 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.2

bge-reranker-v2-m3 is multilingual and small — around 2 GB of VRAM, so it coexists with a chat model on one consumer GPU.

Then point the harness at it:

export SCORER_ENDPOINT=http://localhost:8005/v1/score

Port note: the MCP server (make mcp) also defaults to 8001, which is why the example above uses 8005. If you run both, give them different ports.

Verify:

curl -s http://localhost:8005/v1/score \
  -H 'Content-Type: application/json' \
  -d '{"model":"BAAI/bge-reranker-v2-m3",
       "text_1":"Riksdagen beslutade om ny kärnkraft.",
       "text_2":"Vad sa riksdagen om kärnkraft?"}' | head -c 200

A JSON body containing a score means it works. Connection refused means the container is not running, and the harness will skip scoring rather than fail.

Reading the result

coverage_score is 0–1, the sigmoid of the model's logit.

Range	Reading
> 0.8	The source directly supports the paragraph
0.4 – 0.8	Related, but the paragraph may overreach
< 0.4	The citation does not support the claim — worth reading by hand

Low scores are the interesting ones. A run where the judge says "supported" but coverage is low usually means an answer that is technically defensible and practically misleading — exactly the failure this project cares about most.

SELECT q.question, j.paragraph_text, j.coverage_score
FROM eval_judgments j JOIN eval_questions q ON q.id = j.question_id
WHERE j.coverage_score < 0.4 AND j.verdict = 'supported'
ORDER BY j.coverage_score
LIMIT 20;

Implementation

CitationScorer in scripts/eval_harness.py. It probes the endpoint once at startup, warns and disables itself if unreachable, and never fails a run because scoring is unavailable.