recovery.sh migrated data out of ArangoDB. This project abandoned Arango before the
extraction, and no one cloning plenum has an Arango server to migrate from — it was
carried over by inertia.
scripts/deploy_debate_embeddings.sh and deploy_summary_embeddings.sh were one-off
rollout scripts for features whose columns are now part of the base schema, so a fresh
install already has them.
scripts/correct_arguments.py was a single corrective pass over a specific extraction
error made by a specific small model on Swedish speeches from 2002 onward. Not
reusable, and not something a new deployment should run.
_postgres/rename_map.py and scripts/generate_rename_migration.py produced the
Swedish-to-English migration. That migration is written, applied, and its rollback
verified; the generator has no remaining use and the map described a schema that no
longer exists.
Root __init__.py was empty and made the repository look like an importable package,
which it is not.
scripts/test_auth_e2e.py moved to tests/ — it is a test, and putting it in scripts/
implied it was an operational tool.
Kept deliberately: the eval harness and its analysis scripts, which are a documented
feature rather than leftovers, and the embed/summarise/debate pipeline scripts, which
any deployment needs after an ingest run.
database_query passed model-written SQL straight to a connection that commits. The
corpus reaches the model's context, and in a parliament anyone able to speak can get
text into the corpus, so that SQL has to be treated as untrusted input.
Two layers now apply. Statements must begin with SELECT or WITH, and multi-statement
payloads are rejected — that is how a write gets smuggled in behind a leading SELECT.
Then the query runs inside SET TRANSACTION READ ONLY, so PostgreSQL rejects writes
itself even if the first check is wrong. Verified both independently: five attack
shapes refused by the guard, and INSERT/DROP still refused when the guard is bypassed
entirely and the database layer is called directly.
share_insight gets the same treatment. It re-executes SQL stored in saved
conversations, which is no more trustworthy than freshly generated SQL.
Neither layer replaces running against a SELECT-only role, and SECURITY.md still says
so — it now describes what the application does rather than what it fails to do.
Tool rename completed as a clean break, no aliases, since there are three users and
six sessions to preserve: arango_search -> search_speeches (it has queried Postgres
since the Arango migration and the name was pure debt), search_motions ->
search_documents, fetch_motion -> fetch_document.
Also renamed fetch_documents -> fetch_speeches. It fetches speeches despite the name,
and sitting next to the new fetch_document it was a trap the model would fall into.
Note for anyone with an MCP client configured against the old names: riksdagen_mcp
exposes these, so the names it advertises have changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The orchestrator prompt inlines the database schema so the model can write SQL. It
still listed talks, anforandetext and intressent_id, so every generated query would
have failed against the renamed database. This was the one place where excluding
prompts/ from the rename was wrong: those files hold SQL identifiers as well as
Swedish prose.
A blanket substitution corrupted the prose — "ett annat parti" became "ett annat
party", "fel talare" became "fel speaker_name" — because parti and talare are
ordinary Swedish words as well as column names. Reverted and applied only to the
English schema listing plus the handful of genuine identifier references.
Golden snapshots regenerated deliberately, since the prompts really did change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
talks -> speeches ("talks" reads as conference talks to everyone outside this
project), motions -> documents with a new doc_type column so bills, written
questions and committee reports can share the table later, and 79 columns from
anforandetext -> text, intressent_id -> person_id, valkrets -> constituency,
lydelse -> text, and so on.
_postgres/rename_map.py is the single source of truth. The migration, its rollback,
and the code rewrite are all derived from it, so they cannot drift apart. Hand-
writing a rollback is how you end up with one that fails halfway through.
Two identifiers could not be renamed mechanically and were done by reading the
queries: dok_id means the protocol document in talks but the primary key in
motions, and year stays a calendar year in speeches while becoming session_year in
documents. The latter is aliased in SQL so the JSON field stays `year` and the
frontend contract is unchanged.
Values are never translated. Bifall and Avslag stay as published; parliament.yaml
glosses them. A research tool must not silently rewrite the record.
The migration is guarded by an existence check, so the same file is a no-op on a
fresh database and does the work on an existing one — one schema definition in the
world. ALTER TABLE ... RENAME is catalog-only, so the millions of HNSW-indexed
vectors are untouched.
The trigger functions are recreated explicitly, because plpgsql bodies are stored
as opaque text and do not follow renames: they would have compiled fine and then
failed at the next INSERT. They now read the text-search config from a database
setting rather than hardcoding 'swedish'.
Verified: migration round-trips to a byte-identical schema across columns, indexes
and triggers; re-running is a no-op; triggers repopulate search_vector with working
Swedish stemming on INSERT and UPDATE; and the renamed code runs real searches
against a migrated database holding 5,000 rows of production data, with plain,
prefix and exclusion syntax all working. Frontend type errors went from 9 to 7 —
the rename fixed two and introduced none.
Not yet done, and tracked: legacy compatibility views, the shim for SQL replayed
from saved snapshots, tool-name aliases, and updating docs to the new names.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sixteen system prompts lived as module-level string constants across chat.py,
board.py, synthesis.py, trip.py and llm_tools.py. They are the main thing a fork
for another parliament has to rewrite, and editing them meant editing Python.
They now live under prompts/sv/ as Markdown. Placeholders use string.Template
($name), not str.format: three of these prompts embed literal JSON braces that
str.format raises on, and safe_substitute leaves an unknown placeholder alone
rather than killing a live chat turn over a typo.
Technical configuration that had leaked into the prompt text is now templated:
websearch_to_tsquery('swedish', ...) became '$fts_config', "Answer in Swedish"
became "Answer in $answer_language", and the åäö preservation rule became
$preserve_characters. Domain vocabulary from parliament.yaml is available too, so a
prompt can say $speech_plural and read naturally in any language.
PROMPTS_RELOAD=1 re-reads the files per call, so prompt iteration no longer needs
a server restart.
The prompts themselves stay Swedish. That is the intended design: a fork writes
prompts/<lang>/ in its own language, and the loader falls back through
<lang>/ -> shared -> en/.
Verified by snapshotting all sixteen constants before the move and asserting
equality after: 16/16 render byte-identically, including after parameterization —
which is what proves the templating substitutes exactly what was there before.
tests/test_prompts_golden.py keeps that guarantee going forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeded via an explicit allow-list (see /home/lasse/plenum-seed.sh) rather than by
deleting files from a copy, so nothing sensitive can survive by omission.
Excluded: WireGuard backup + client config, the plaintext DB password in admin.py,
Arango credentials in scripts/notes.md, .claude/settings.json, a 113 MB log,
providers.yaml (private endpoint), the Arango/ChromaDB-era scripts, the duplicated
claude-design-system frontend copy, and assorted screenshots and one-off planning docs.
297 tracked files / 49 MB of history -> 161 files / 2.1 MB.
Recovered 14 database migrations that the old .gitignore's `*.sql` rule had been
hiding from version control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lasse Edfast