docs/eval-scorer.md was notes-to-self for work already finished. It told the reader to add a class to eval_harness.py that has been there for months, specified the author's own GPU by model number, and carried escaped-markdown artifacts from a bad paste. The feature is real and was used — 9 eval runs, 668 questions, 3536 judgments — so the fix is documentation, not deletion. It now explains what coverage scoring measures and why it is worth having alongside the judge model: the judge catches claims a source contradicts, the cross-encoder catches claims a source simply does not cover. Includes how to read the number, and the query that finds the interesting cases — paragraphs the judge passed but the scorer did not, which is where technically-defensible-but-misleading answers show up. The scorer defaulted to port 8001, which is also the MCP server's default, so running both meant one silently failed to bind. Moved to 8005 and documented. SCORER_ENDPOINT is now in .env.example, and the README has a documentation index — every file under docs/ was previously unreachable from anywhere in the repo.main
parent
65bb842cc2
commit
8a0d6b6b3d
5 changed files with 110 additions and 97 deletions
@ -1,94 +1,87 @@ |
|||||||
# **Scorer Integration Guide (RTX 3060 / vLLM)** |
# Citation coverage scoring |
||||||
|
|
||||||
This guide adds a coverage\_score column to your Swedish Parliament evaluation harness using a high-speed "Cross-Encoder" (Reranker) model. |
An **optional** add-on to the [evaluation harness](eval-harness.md). It measures how |
||||||
|
well a cited source actually supports the paragraph citing it, filling in |
||||||
## **1\. How it works** |
`eval_judgments.coverage_score`. |
||||||
|
|
||||||
* **The Model:** BAAI/bge-reranker-v2-m3 acts as a "Scorer." It looks at the Source and the Paragraph simultaneously and outputs a relevance score. |
Everything else in the harness works without it. If the scorer is unreachable the run |
||||||
* **The API:** We use vLLM's /v1/score (or /v1/rerank) endpoint. It is 10x faster than Qwen because it doesn't generate text; it only computes a single mathematical "head". |
completes normally and `coverage_score` is left NULL. |
||||||
* **The GPU:** On your RTX 3060, this runs in the background with very low VRAM usage (\~2GB if quantized). |
|
||||||
|
## Why it exists |
||||||
## **2\. Infrastructure Setup** |
|
||||||
|
The harness already asks a judge model whether a paragraph is supported by its |
||||||
Run this command on your Debian server to start the scorer. vLLM will automatically download the model from HuggingFace on the first run. |
citations. That is a generative call: slow, and it answers in prose that has to be |
||||||
docker run \--gpus all \\ |
parsed. |
||||||
\-p 8001:8000 \\ |
|
||||||
\--name eval-scorer \\ |
A cross-encoder answers a narrower question — *how relevant is this source to this |
||||||
vllm/vllm-openai \\ |
text?* — as a single number. It reads the source and the paragraph together and emits |
||||||
\--model BAAI/bge-reranker-v2-m3 \\ |
one score with no generation at all, so it runs roughly an order of magnitude faster |
||||||
\--device cuda \\ |
and costs almost no VRAM. |
||||||
\--max-model-len 4096 \\ |
|
||||||
\--gpu-memory-utilization 0.2 \\ |
The two measure different things and are worth having together: the judge catches |
||||||
\--trust-remote-code |
claims a source contradicts, the scorer catches claims a source simply does not cover. |
||||||
|
|
||||||
## **3\. Database & Code Integration** |
## Running it |
||||||
|
|
||||||
### **Step A: Update SQL** |
Any endpoint implementing OpenAI's `/v1/score` or `/v1/rerank` will do. With vLLM: |
||||||
|
|
||||||
ALTER TABLE eval\_judgments ADD COLUMN coverage\_score FLOAT DEFAULT 0.0; |
```bash |
||||||
|
docker run --gpus all -p 8005:8000 --name plenum-scorer \ |
||||||
### **Step B: The Python Logic (Add to eval\_harness.py)** |
vllm/vllm-openai \ |
||||||
|
--model BAAI/bge-reranker-v2-m3 \ |
||||||
Add this class to handle the communication with vLLM. Note the use of the sigmoid function to turn the model's "logits" into a 0-1 probability. |
--max-model-len 4096 \ |
||||||
import requests |
--gpu-memory-utilization 0.2 |
||||||
import math |
``` |
||||||
|
|
||||||
class CitationScorer: |
`bge-reranker-v2-m3` is multilingual and small — around 2 GB of VRAM, so it coexists |
||||||
"""Connects to the vLLM /v1/score endpoint.""" |
with a chat model on one consumer GPU. |
||||||
def \_\_init\_\_(self, endpoint: str \= "http://localhost:8001/v1/score"): |
|
||||||
self.endpoint \= endpoint |
Then point the harness at it: |
||||||
|
|
||||||
def get\_score(self, paragraph: str, sources: str) \-\> float: |
```bash |
||||||
"""Calculates a support probability (0.0 to 1.0).""" |
export SCORER_ENDPOINT=http://localhost:8005/v1/score |
||||||
try: |
``` |
||||||
payload \= { |
|
||||||
"model": "BAAI/bge-reranker-v2-m3", |
> **Port note:** the MCP server (`make mcp`) also defaults to 8001, which is why the |
||||||
"text\_1": sources\[:12000\], \# Truncate long sources for speed |
> example above uses 8005. If you run both, give them different ports. |
||||||
"text\_2": paragraph |
|
||||||
} |
**Verify:** |
||||||
response \= requests.post(self.endpoint, json=payload, timeout=5) |
|
||||||
if response.status\_code \== 200: |
```bash |
||||||
\# BGE-Reranker-v2 outputs logits. Sigmoid converts to 0-1. |
curl -s http://localhost:8005/v1/score \ |
||||||
data \= response.json().get("data", \[\]) |
-H 'Content-Type: application/json' \ |
||||||
if data: |
-d '{"model":"BAAI/bge-reranker-v2-m3", |
||||||
raw\_logit \= data\[0\].get("score", \-10.0) |
"text_1":"Riksdagen beslutade om ny kärnkraft.", |
||||||
return 1 / (1 \+ math.exp(-raw\_logit)) |
"text_2":"Vad sa riksdagen om kärnkraft?"}' | head -c 200 |
||||||
return 0.0 |
``` |
||||||
except Exception as e: |
|
||||||
print(f"\[scorer\] Error calling vLLM: {e}") |
A JSON body containing a score means it works. Connection refused means the container |
||||||
return 0.0 |
is not running, and the harness will skip scoring rather than fail. |
||||||
|
|
||||||
### **Step C: Update the Main Evaluation Loop** |
## Reading the result |
||||||
|
|
||||||
In eval\_harness.py, modify the section where you process judgments: |
`coverage_score` is 0–1, the sigmoid of the model's logit. |
||||||
\# Initialize once at start |
|
||||||
scorer \= CitationScorer() |
| Range | Reading | |
||||||
|
|---|---| |
||||||
\# ... inside the paragraph loop ... |
| > 0.8 | The source directly supports the paragraph | |
||||||
try: |
| 0.4 – 0.8 | Related, but the paragraph may overreach | |
||||||
\# 1\. Get the standard LLM verdict (Qwen/etc) |
| < 0.4 | The citation does not support the claim — worth reading by hand | |
||||||
judgments \= judge.verdict(answer, sources\_compact) |
|
||||||
|
Low scores are the interesting ones. A run where the judge says "supported" but |
||||||
\# 2\. Add the quantitative Scorer verdict |
coverage is low usually means an answer that is technically defensible and practically |
||||||
for j in judgments: |
misleading — exactly the failure this project cares about most. |
||||||
p\_text \= j.get("paragraph\_text", "") |
|
||||||
\# The scorer gives a 0-1 confidence that the source supports this paragraph |
```sql |
||||||
j\["coverage\_score"\] \= scorer.get\_score(p\_text, sources\_compact) |
SELECT q.question, j.paragraph_text, j.coverage_score |
||||||
|
FROM eval_judgments j JOIN eval_questions q ON q.id = j.question_id |
||||||
\# 3\. Save to Postgres (Ensure your insert\_judgments helper handles this key) |
WHERE j.coverage_score < 0.4 AND j.verdict = 'supported' |
||||||
insert\_judgments(question\_id, judgments, judge\_model) |
ORDER BY j.coverage_score |
||||||
except Exception as e: |
LIMIT 20; |
||||||
print(f"Error in judge/scorer loop: {e}") |
``` |
||||||
|
|
||||||
## **4\. Verification** |
## Implementation |
||||||
|
|
||||||
To verify it's working without running the whole script: |
`CitationScorer` in [`scripts/eval_harness.py`](../scripts/eval_harness.py). It probes |
||||||
curl http://localhost:8001/v1/score \\ |
the endpoint once at startup, warns and disables itself if unreachable, and never |
||||||
\-H "Content-Type: application/json" \\ |
fails a run because scoring is unavailable. |
||||||
\-d '{ |
|
||||||
"model": "BAAI/bge-reranker-v2-m3", |
|
||||||
"text\_1": "The Riksdag consists of 349 members.", |
|
||||||
"text\_2": "There are 349 politicians in the Swedish parliament." |
|
||||||
}' |
|
||||||
|
|
||||||
*(You should see a high positive score).* |
|
||||||
|
|||||||
Loading…
Reference in new issue