From 8a0d6b6b3d667cb4619a0ddb16fcc692455ca2b7 Mon Sep 17 00:00:00 2001 From: Lasse Server Date: Mon, 3 Aug 2026 13:58:44 +0200 Subject: [PATCH] Rewrite the scorer doc as usage documentation; fix a port collision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/eval-scorer.md was notes-to-self for work already finished. It told the reader to add a class to eval_harness.py that has been there for months, specified the author's own GPU by model number, and carried escaped-markdown artifacts from a bad paste. The feature is real and was used — 9 eval runs, 668 questions, 3536 judgments — so the fix is documentation, not deletion. It now explains what coverage scoring measures and why it is worth having alongside the judge model: the judge catches claims a source contradicts, the cross-encoder catches claims a source simply does not cover. Includes how to read the number, and the query that finds the interesting cases — paragraphs the judge passed but the scorer did not, which is where technically-defensible-but-misleading answers show up. The scorer defaulted to port 8001, which is also the MCP server's default, so running both meant one silently failed to bind. Moved to 8005 and documented. SCORER_ENDPOINT is now in .env.example, and the README has a documentation index — every file under docs/ was previously unreachable from anywhere in the repo. --- .env.example | 4 + README.md | 16 ++++ docs/eval-harness.md | 4 +- docs/eval-scorer.md | 181 +++++++++++++++++++--------------------- scripts/eval_harness.py | 2 +- 5 files changed, 110 insertions(+), 97 deletions(-) diff --git a/.env.example b/.env.example index 4fe37a7..11b8deb 100644 --- a/.env.example +++ b/.env.example @@ -69,6 +69,10 @@ PLENUM_DATA_DIR= # Re-read prompt files on every call. Development only. # PROMPTS_RELOAD=1 +# Optional: citation coverage scoring for the eval harness (docs/eval-scorer.md). +# Unset means scoring is skipped; the harness still runs. +# SCORER_ENDPOINT=http://localhost:8005/v1/score + # ── Deep research tuning ────────────────────────────────────────────────────── # All optional; the defaults in backend/services/research/ are sensible. Listed # here so they are discoverable rather than only findable by grep. diff --git a/README.md b/README.md index a4bae50..7b92906 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,22 @@ frontend/ React + TypeScript + Vite deploy/examples/ systemd units and an nginx site, with placeholders ``` +## Documentation + +| | | +|---|---| +| [docs/SETUP.md](docs/SETUP.md) | Step-by-step install: database, chat model, embeddings, data | +| [docs/PORTING.md](docs/PORTING.md) | Adapting to a parliament other than Sweden | +| [docs/SCHEMA.md](docs/SCHEMA.md) | What every table and column means | +| [docs/deep-research.md](docs/deep-research.md) | How the background research agent works | +| [docs/sources-system.md](docs/sources-system.md) | How citations are tracked and verified | +| [docs/multi-provider.md](docs/multi-provider.md) | Letting users bring their own model API key | +| [docs/shadow-communicator.md](docs/shadow-communicator.md) | The parallel commentary shown while the model works | +| [docs/eval-harness.md](docs/eval-harness.md) | Measuring whether answers are actually grounded | +| [docs/eval-scorer.md](docs/eval-scorer.md) | Optional cross-encoder scoring for citations | +| [SECURITY.md](SECURITY.md) | Model-authored SQL, API keys, chat privacy | +| [CONTRIBUTING.md](CONTRIBUTING.md) | How to work on this | + ## Configuration All settings are environment variables, documented in `.env.example`. Two paths let a diff --git a/docs/eval-harness.md b/docs/eval-harness.md index 61efec5..428b200 100644 --- a/docs/eval-harness.md +++ b/docs/eval-harness.md @@ -90,7 +90,7 @@ Three tables (see `_postgres/migrations/add_eval_tables.sql`): ### `coverage_score` (cross-encoder grounding signal) -A 0–1 probability from a `BAAI/bge-reranker-v2-m3` cross-encoder served locally on port 8001 +A 0–1 probability from a `BAAI/bge-reranker-v2-m3` cross-encoder served locally on port 8005 via vLLM. For each paragraph, **all** cited source texts are concatenated (up to 28 000 chars / ≈7 000 tokens) and scored against the paragraph as a single call. The raw logit is converted via sigmoid so 0.5 = neutral, >0.7 = likely grounded, <0.3 = likely hallucinated. @@ -101,7 +101,7 @@ Start the scorer (first run downloads the model to `$HOME/models`): ```bash nohup vllm serve BAAI/bge-reranker-v2-m3 \ - --port 8001 \ + --port 8005 \ --download-dir $HOME/models \ --gpu-memory-utilization 0.2 \ --max-model-len 8192 \ diff --git a/docs/eval-scorer.md b/docs/eval-scorer.md index fa88d8e..83d888a 100644 --- a/docs/eval-scorer.md +++ b/docs/eval-scorer.md @@ -1,94 +1,87 @@ -# **Scorer Integration Guide (RTX 3060 / vLLM)** - -This guide adds a coverage\_score column to your Swedish Parliament evaluation harness using a high-speed "Cross-Encoder" (Reranker) model. - -## **1\. How it works** - -* **The Model:** BAAI/bge-reranker-v2-m3 acts as a "Scorer." It looks at the Source and the Paragraph simultaneously and outputs a relevance score. -* **The API:** We use vLLM's /v1/score (or /v1/rerank) endpoint. It is 10x faster than Qwen because it doesn't generate text; it only computes a single mathematical "head". -* **The GPU:** On your RTX 3060, this runs in the background with very low VRAM usage (\~2GB if quantized). - -## **2\. Infrastructure Setup** - -Run this command on your Debian server to start the scorer. vLLM will automatically download the model from HuggingFace on the first run. -docker run \--gpus all \\ - \-p 8001:8000 \\ - \--name eval-scorer \\ - vllm/vllm-openai \\ - \--model BAAI/bge-reranker-v2-m3 \\ - \--device cuda \\ - \--max-model-len 4096 \\ - \--gpu-memory-utilization 0.2 \\ - \--trust-remote-code - -## **3\. Database & Code Integration** - -### **Step A: Update SQL** - -ALTER TABLE eval\_judgments ADD COLUMN coverage\_score FLOAT DEFAULT 0.0; - -### **Step B: The Python Logic (Add to eval\_harness.py)** - -Add this class to handle the communication with vLLM. Note the use of the sigmoid function to turn the model's "logits" into a 0-1 probability. -import requests -import math - -class CitationScorer: - """Connects to the vLLM /v1/score endpoint.""" - def \_\_init\_\_(self, endpoint: str \= "http://localhost:8001/v1/score"): - self.endpoint \= endpoint - - def get\_score(self, paragraph: str, sources: str) \-\> float: - """Calculates a support probability (0.0 to 1.0).""" - try: - payload \= { - "model": "BAAI/bge-reranker-v2-m3", - "text\_1": sources\[:12000\], \# Truncate long sources for speed - "text\_2": paragraph - } - response \= requests.post(self.endpoint, json=payload, timeout=5) - if response.status\_code \== 200: - \# BGE-Reranker-v2 outputs logits. Sigmoid converts to 0-1. - data \= response.json().get("data", \[\]) - if data: - raw\_logit \= data\[0\].get("score", \-10.0) - return 1 / (1 \+ math.exp(-raw\_logit)) - return 0.0 - except Exception as e: - print(f"\[scorer\] Error calling vLLM: {e}") - return 0.0 - -### **Step C: Update the Main Evaluation Loop** - -In eval\_harness.py, modify the section where you process judgments: -\# Initialize once at start -scorer \= CitationScorer() - -\# ... inside the paragraph loop ... -try: - \# 1\. Get the standard LLM verdict (Qwen/etc) - judgments \= judge.verdict(answer, sources\_compact) - - \# 2\. Add the quantitative Scorer verdict - for j in judgments: - p\_text \= j.get("paragraph\_text", "") - \# The scorer gives a 0-1 confidence that the source supports this paragraph - j\["coverage\_score"\] \= scorer.get\_score(p\_text, sources\_compact) - - \# 3\. Save to Postgres (Ensure your insert\_judgments helper handles this key) - insert\_judgments(question\_id, judgments, judge\_model) -except Exception as e: - print(f"Error in judge/scorer loop: {e}") - -## **4\. Verification** - -To verify it's working without running the whole script: -curl http://localhost:8001/v1/score \\ - \-H "Content-Type: application/json" \\ - \-d '{ - "model": "BAAI/bge-reranker-v2-m3", - "text\_1": "The Riksdag consists of 349 members.", - "text\_2": "There are 349 politicians in the Swedish parliament." - }' - -*(You should see a high positive score).* \ No newline at end of file +# Citation coverage scoring + +An **optional** add-on to the [evaluation harness](eval-harness.md). It measures how +well a cited source actually supports the paragraph citing it, filling in +`eval_judgments.coverage_score`. + +Everything else in the harness works without it. If the scorer is unreachable the run +completes normally and `coverage_score` is left NULL. + +## Why it exists + +The harness already asks a judge model whether a paragraph is supported by its +citations. That is a generative call: slow, and it answers in prose that has to be +parsed. + +A cross-encoder answers a narrower question — *how relevant is this source to this +text?* — as a single number. It reads the source and the paragraph together and emits +one score with no generation at all, so it runs roughly an order of magnitude faster +and costs almost no VRAM. + +The two measure different things and are worth having together: the judge catches +claims a source contradicts, the scorer catches claims a source simply does not cover. + +## Running it + +Any endpoint implementing OpenAI's `/v1/score` or `/v1/rerank` will do. With vLLM: + +```bash +docker run --gpus all -p 8005:8000 --name plenum-scorer \ + vllm/vllm-openai \ + --model BAAI/bge-reranker-v2-m3 \ + --max-model-len 4096 \ + --gpu-memory-utilization 0.2 +``` + +`bge-reranker-v2-m3` is multilingual and small — around 2 GB of VRAM, so it coexists +with a chat model on one consumer GPU. + +Then point the harness at it: + +```bash +export SCORER_ENDPOINT=http://localhost:8005/v1/score +``` + +> **Port note:** the MCP server (`make mcp`) also defaults to 8001, which is why the +> example above uses 8005. If you run both, give them different ports. + +**Verify:** + +```bash +curl -s http://localhost:8005/v1/score \ + -H 'Content-Type: application/json' \ + -d '{"model":"BAAI/bge-reranker-v2-m3", + "text_1":"Riksdagen beslutade om ny kärnkraft.", + "text_2":"Vad sa riksdagen om kärnkraft?"}' | head -c 200 +``` + +A JSON body containing a score means it works. Connection refused means the container +is not running, and the harness will skip scoring rather than fail. + +## Reading the result + +`coverage_score` is 0–1, the sigmoid of the model's logit. + +| Range | Reading | +|---|---| +| > 0.8 | The source directly supports the paragraph | +| 0.4 – 0.8 | Related, but the paragraph may overreach | +| < 0.4 | The citation does not support the claim — worth reading by hand | + +Low scores are the interesting ones. A run where the judge says "supported" but +coverage is low usually means an answer that is technically defensible and practically +misleading — exactly the failure this project cares about most. + +```sql +SELECT q.question, j.paragraph_text, j.coverage_score +FROM eval_judgments j JOIN eval_questions q ON q.id = j.question_id +WHERE j.coverage_score < 0.4 AND j.verdict = 'supported' +ORDER BY j.coverage_score +LIMIT 20; +``` + +## Implementation + +`CitationScorer` in [`scripts/eval_harness.py`](../scripts/eval_harness.py). It probes +the endpoint once at startup, warns and disables itself if unreachable, and never +fails a run because scoring is unavailable. diff --git a/scripts/eval_harness.py b/scripts/eval_harness.py index a7d2b89..0b49a20 100644 --- a/scripts/eval_harness.py +++ b/scripts/eval_harness.py @@ -319,7 +319,7 @@ def _fetch_full_talks(talk_ids: List[str]) -> Dict[str, str]: _SCORER_COMBINED_MAX_CHARS = 28_000 -_SCORER_ENDPOINT = os.environ.get("SCORER_ENDPOINT", "http://localhost:8001/v1/score") +_SCORER_ENDPOINT = os.environ.get("SCORER_ENDPOINT", "http://localhost:8005/v1/score") _SCORER_MODEL = "BAAI/bge-reranker-v2-m3"