The magic number 6 is the length of the literal "speeches/" prefix these
four call sites strip off an ArangoDB-style id — spelling it out makes that
relationship explicit and survives a future rename of the prefix without a
silent off-by-N bug. Also swap response.statusText for response.status in
fetchTalk's error message: statusText is frequently empty (HTTP/2 drops the
reason phrase), so the numeric code is the more reliable thing to show.
This mirrors a fix already made directly on rixdagen-prod's production
branch (commit 9a87111) that hadn't been brought upstream yet.
packages/llm/client.py:_read_stream yielded token-by-token deltas but was
dead code — nothing called generate(stream=True). The chat tool loop always
blocked on the final answer generation, so a ~40s answer arrived as a single
SSE event and read as a hang, even though the "card stack" already streamed
tool-selection progress.
The final answer is structurally just whichever tool-loop iteration returns
content with no tool_calls — there's no way to know in advance which
iteration that'll be, and a turn can carry short narration before a tool
call in the very same response. So streaming has to be speculative:
- packages/llm/client.py: _read_stream becomes StreamAccumulator, which both
streams live (thinking/content/tool_call) deltas *and* reconstructs a full
ChatCompletionMessage (content, tool_calls, reasoning_content) once the
stream ends. Tool-call argument fragments are accumulated by index, same
pattern every OpenAI-compatible streaming client needs. Downstream tool
execution code needed zero changes because of this — it just consumes the
reconstructed message instead of the SDK's blocking return value. Also
hardened <think>/</think> detection against a tag split across chunk
boundaries (was a per-chunk substring check; now a carry-over buffer) —
latent before since nothing exercised streaming for real.
- backend/services/streaming_answer.py (new): AnswerStreamFilter holds
buffered content back until it's implausible to still be one-line
narration (this codebase already caps narration at 150 chars elsewhere),
and separately holds back any trailing bracket run that could still grow
into a [src:ID]/[[src:ID]] marker — citation validity isn't known until
parse_and_renumber_citations runs on the complete text, so a raw marker,
hallucinated or not, must never reach the client even for a frame.
run_streaming_iteration drives one tool-loop iteration through this filter
and emits "answer_delta" events; if a tool_call shows up after content
already cleared the gate (narration grew unusually long before the model
pivoted), it emits one "answer_delta_retract" so the UI doesn't strand a
stale preview.
- backend/services/chat.py, mp_chat.py: thread an explicit stream_answer
flag through to _run_tool_loop's single generate() call site, rather than
inferring streaming from event_callback being non-None — eval logging's
ConversationRecorder.wrap() always returns a non-None callable, even for
the plain blocking /api/chat endpoint, so that inference would have been
wrong. Only stream_chat_response (the two SSE routes) sets it; /api/chat
is untouched.
- Frontend: both SSE consumers (ChatPanel.tsx's card stack, MpChatPanel.tsx's
simpler turn list) grow "answer_delta"/"answer_delta_retract" handling.
The live preview renders as plain, React-escaped text (no
dangerouslySetInnerHTML, no markdown parsing mid-stream — that's a
one-shot full-document transform, unsafe on partial input) and is fully
replaced by the authoritative, citation-renumbered/person-linked/
language-polished HTML once the terminal "answer" event lands. Both event
types were previously-unhandled and silently ignored, so the backend and
frontend halves are independently safe to deploy.
Added tests/test_llm_client_stream.py and tests/test_streaming_answer.py
covering the reconstruction (content, tool-call fragment accumulation,
split <think> tags, truncated-stream handling) and the filter (gate timing,
citation-marker safety across chunk boundaries, the retract path, error
propagation). 5 pre-existing failures in test_prompts_golden.py/
test_provenance.py are unrelated (reproduce identically on main).
The README opened with "It runs in production as rixdagen.se", which reads as though
this repository is that website — or that cloning it gets you a copy of it. Neither is
true: rixdagen.se is a fork with its own configuration and server setup, and plenum is
software you run against your own parliament.
Now says so, and explains why Sweden is here at all: it is the instance the project is
tested against, so its config, ingest adapter and prompts ship as the worked example to
copy from.
Most people arriving here will set this up with an assistant, and the guide for that
was buried in a table two thirds down the page. It is now the first thing after the
introduction, with the starting prompt inline so it can be copied without opening
another file.
Also says what the assistant will do first — run the doctor — so the reader knows the
machine gets inspected before they are asked anything.
A fork for Hungary would have looked Swedish. Only party colours were configurable —
the warm paper, the institutional blue and the Garamond serif were hardcoded, and that
palette was drawn from riksdagen.se deliberately. Nothing said so anywhere. A `theme:`
block now sets those as CSS custom properties at runtime, the same mechanism party
colours already used, so no stylesheet needs editing. README and PORTING.md both say
plainly that the default is not neutral.
scripts/doctor.py reports what a machine can and cannot do: Python and dependencies,
PostgreSQL with its extensions and text-search setting, schema and row counts, the chat
endpoint, whether that model can actually call tools, the embedding dimension against
the configured one, GPU, sudo, and which ports are free. It changes nothing and exits
non-zero on failures so a script can gate on it. Tool calling is checked separately
because a model can converse perfectly and still never call a tool, which here means
confident answers with no sources.
docs/ASSISTANT-SETUP.md is a prompt plus the interview an assistant should run: ask the
doctor before asking the user, one section at a time, always recommend a default, never
invent a parliament's colours or data URL, and verify each stage with real output
rather than an exit code.
Verified against the live deployment: 26 checks OK, no failures.
The README told people to clone. For anyone adapting this to their own parliament
that is wrong: they cannot push, so their work is stranded on one machine, and
pulling in later updates is awkward. docs/YOUR-PARLIAMENT.md covers forking, and the
habit that decides whether future updates are painless — add files rather than edit
them, so git never has to merge the same file twice.
Writing it meant walking through a Norwegian adaptation for real, which found two
things that would have stopped a user on their first attempt.
A relative PARLIAMENT_CONFIG resolved against the current directory, so
`PARLIAMENT_CONFIG=parliament.no.yaml` worked when you happened to be standing in the
repository and failed everywhere else, including under systemd. Relative paths now
resolve against the repository, for PROMPTS_DIR and CONTENT_DIR too.
Worse, YAML 1.1 reads unquoted `no`, `yes`, `on`, `off`, `y` and `n` as booleans.
Norway is the worst possible case: `country: NO` and `prompt_language: no` both became
false, and the symptom was a TypeError deep inside a path join that named nothing
relevant. Text fields are now coerced back, and both parliament.yaml and the guide say
to quote such values, since the repair cannot recover the original capitalisation.
Verified by building a Norwegian fork the way the guide describes — adding
parliament.no.yaml, prompts/no/ and an adapter, editing nothing — and confirming the
app loads it. Sweden is unaffected and production still serves.
The README's documentation index is now grouped by what you are trying to do, with a
sentence on each entry rather than a bare filename.
docs/eval-scorer.md was notes-to-self for work already finished. It told the reader to
add a class to eval_harness.py that has been there for months, specified the author's
own GPU by model number, and carried escaped-markdown artifacts from a bad paste. The
feature is real and was used — 9 eval runs, 668 questions, 3536 judgments — so the fix
is documentation, not deletion.
It now explains what coverage scoring measures and why it is worth having alongside
the judge model: the judge catches claims a source contradicts, the cross-encoder
catches claims a source simply does not cover. Includes how to read the number, and
the query that finds the interesting cases — paragraphs the judge passed but the
scorer did not, which is where technically-defensible-but-misleading answers show up.
The scorer defaulted to port 8001, which is also the MCP server's default, so running
both meant one silently failed to bind. Moved to 8005 and documented.
SCORER_ENDPOINT is now in .env.example, and the README has a documentation index —
every file under docs/ was previously unreachable from anywhere in the repo.
recovery.sh migrated data out of ArangoDB. This project abandoned Arango before the
extraction, and no one cloning plenum has an Arango server to migrate from — it was
carried over by inertia.
scripts/deploy_debate_embeddings.sh and deploy_summary_embeddings.sh were one-off
rollout scripts for features whose columns are now part of the base schema, so a fresh
install already has them.
scripts/correct_arguments.py was a single corrective pass over a specific extraction
error made by a specific small model on Swedish speeches from 2002 onward. Not
reusable, and not something a new deployment should run.
_postgres/rename_map.py and scripts/generate_rename_migration.py produced the
Swedish-to-English migration. That migration is written, applied, and its rollback
verified; the generator has no remaining use and the map described a schema that no
longer exists.
Root __init__.py was empty and made the repository look like an importable package,
which it is not.
scripts/test_auth_e2e.py moved to tests/ — it is a test, and putting it in scripts/
implied it was an operational tool.
Kept deliberately: the eval harness and its analysis scripts, which are a documented
feature rather than leftovers, and the embed/summarise/debate pipeline scripts, which
any deployment needs after an ingest run.
providers.template.yaml was a Raycast AI template. It named Raycast in its comments
and used api_keys/abilities/context — none of which provider_registry.py reads, while
omitting user_api_key and server_api_key_env, which it requires. Anyone copying it
got a file the application ignored. Replaced with one matching the code, covering
vLLM, Ollama, OpenRouter, Berget, OpenAI and Gemini.
docs/SETUP.md is written to be executed rather than skimmed: what the four external
dependencies are and which are optional, how to choose between providers, and a
verification command after every step that calls the real thing. Chat and tool calling
are checked separately, because a model can hold a conversation perfectly and still
never call a tool — which in this project means confident answers with no sources, the
one failure mode that matters most here.
Embeddings get their own section because they are a separate endpoint from chat and
conflating the two is the most common setup mistake, and because the dimension is
load-bearing: changing the model means re-embedding the whole corpus.
Ends with a symptom/cause/fix table covering the traps found during the production
cutover — the EnvironmentFile parsing difference, app.fts_config not surviving a
restore, and .env being unsourceable in bash.
Verified: every verification block in the guide was run against the live deployment.
The first version read git log through a shell while-loop with a tab IFS, which make
mangled — every commit came out as an 'ambiguous argument' error. git's own --grep and
--invert-grep search the full commit message, including the trailer, and need no
parsing at all.
Upstream: no — tooling for forks.
A deployment run as a fork will sometimes fix something in production first, because
production is what is broken. Those fixes are easy to forget, and a forgotten one is
how the two copies quietly diverge.
Commits carry an 'Upstream: yes' or 'Upstream: no — reason' trailer, and
`make upstream-pending` lists both what is marked and still owed, and what carries no
marker at all so the decision is visible rather than skipped.
Upstream: no — this is tooling for forks, but it belongs in the shared Makefile so
every deployment gets it rather than reinventing the wrapper.
systemd's environment-file parser is stricter than python-dotenv: it does not
accept spaces around '=' and handles quoting differently. A real .env with
'LLM_MODEL =' produces a key with a trailing space, which then reads as unset — the
service starts and fails later on a missing model rather than at startup.
The app already loads .env itself via python-dotenv, resolved against
WorkingDirectory, so the directive was redundant as well as harmful. Found while
cutting rixdagen.se over; the predecessor's unit carried a comment warning about
exactly this, which the generated example had dropped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EOF
Site copy now resolves through CONTENT_DIR, alongside PARLIAMENT_CONFIG and
PROMPTS_DIR. All three let a deployment keep its own wording, branding and contact
details outside the repository, so they never appear in a diff against upstream —
which is what keeps a production fork mergeable.
parliament.yaml's site.title was "rixdagen.se". That is the deployment's identity,
not Sweden's data, so upstream now ships a neutral default. The Swedish user guide
keeps its content but loses the personal contact details and site-specific wording;
a deployment ships its own guide by pointing CONTENT_DIR at it.
Fixes /api/guide, which still read ../user-guide.md after that file moved to
content/sv/ and had been returning 404.
Deliberately kept: the author metadata in pyproject.toml and the copyright line in
LICENSE, which are attribution rather than configuration, and the README's note that
this runs in production as rixdagen.se, which is provenance worth stating.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This project moved to Postgres some time ago, but ~380 lines of SQL-to-AQL
translation and AQL syntax validation were still sitting in utils.py, reachable
from nothing. utils.py is now just TextChunker, which is the only part anything
imports. Also drops a stray prompt draft that had been left in backend/services/,
and the _normalize_arango_search_args name, which never touched Arango.
Worth knowing separately: the Arango server is still running and still
authenticating, so the riksdagsgruppen account is a live door into a system nothing
here uses any more. Deleting that account is simpler than rotating its password.
The env_manager account is a different one and other projects still depend on it.
.gitignore now excludes .claude/ entirely rather than just settings.local.json.
settings.json can carry credentials inside allowlisted command strings, which is
exactly how a live database password ended up committed to the predecessor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`python -m ingest.cli fetch|load|sync` now exists — README, PORTING.md, the systemd
example and pyproject's console script have all been referencing it.
This replaces six scripts that the rename had left genuinely broken: the speech
loader's INSERT listed `year, year` as two columns, and both loaders read the source
JSON using plenum's column names rather than the Riksdag's field names, so a sync
would have written empty rows without erroring. Rewriting on top of the adapter was
safer than repairing them, because the adapter makes the source-name/column-name
boundary explicit rather than leaving it to be remembered.
Two source quirks found by running it over real archives rather than reasoning about
it: a few records carry a bare string where a child element belongs, which took the
whole load down on an AttributeError; and party codes appear in both cases, so
`parties` held {S,s} and any filter on 'S' would have silently missed half the
documents. Both are handled in the adapter, where source messiness belongs.
Also adds doc_type to schema.sql. It was only in the migration, so a fresh install
lacked a column the loader writes.
Verified end to end against real archives: 400 speeches and 400 documents ingested
into a clean database, producing 3,742 authors and 2,229 proposals, with search
vectors built by the triggers and full-text queries returning hits.
Post-load steps (debate ids, embeddings, summaries) still run separately via
scripts/debates.py and scripts/make_embeddings.py; wiring them into `sync` is
tracked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ingest/adapters/riksdagen.py is the only module that knows what the Riksdag calls
its fields. Everything downstream works in plenum's column names, which is what
makes another parliament a matter of writing one file rather than editing the
application.
Writing it surfaced a bug the rename introduced: the loaders read the *source* JSON,
whose keys are the Riksdag's Swedish field names, and the mechanical pass renamed
those reads as if they were column names. doc.get("anforandetext") became
doc.get("text"), doc.get("talare") became doc.get("speaker_name"), and so on. The
next sync would have written empty speeches without erroring. This is exactly the
confusion an adapter exists to prevent, since it makes the source-name/column-name
boundary explicit instead of implicit.
The adapter captures what this source actually does, verified against real files:
repeated elements arrive as an object when there is one and a list when there are
several; null is serialised as the string "None"; speech text is HTML; and the
speaker field carries the party in parentheses, which has to come off or names stop
joining to the member register. Output was checked field by field against what
production stores for the same record.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
database_query passed model-written SQL straight to a connection that commits. The
corpus reaches the model's context, and in a parliament anyone able to speak can get
text into the corpus, so that SQL has to be treated as untrusted input.
Two layers now apply. Statements must begin with SELECT or WITH, and multi-statement
payloads are rejected — that is how a write gets smuggled in behind a leading SELECT.
Then the query runs inside SET TRANSACTION READ ONLY, so PostgreSQL rejects writes
itself even if the first check is wrong. Verified both independently: five attack
shapes refused by the guard, and INSERT/DROP still refused when the guard is bypassed
entirely and the database layer is called directly.
share_insight gets the same treatment. It re-executes SQL stored in saved
conversations, which is no more trustworthy than freshly generated SQL.
Neither layer replaces running against a SELECT-only role, and SECURITY.md still says
so — it now describes what the application does rather than what it fails to do.
Tool rename completed as a clean break, no aliases, since there are three users and
six sessions to preserve: arango_search -> search_speeches (it has queried Postgres
since the Arango migration and the name was pure debt), search_motions ->
search_documents, fetch_motion -> fetch_document.
Also renamed fetch_documents -> fetch_speeches. It fetches speeches despite the name,
and sitting next to the new fetch_document it was a trap the model would fall into.
Note for anyone with an MCP client configured against the old names: riksdagen_mcp
exposes these, so the names it advertises have changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The orchestrator prompt inlines the database schema so the model can write SQL. It
still listed talks, anforandetext and intressent_id, so every generated query would
have failed against the renamed database. This was the one place where excluding
prompts/ from the rename was wrong: those files hold SQL identifiers as well as
Swedish prose.
A blanket substitution corrupted the prose — "ett annat parti" became "ett annat
party", "fel talare" became "fel speaker_name" — because parti and talare are
ordinary Swedish words as well as column names. Reverted and applied only to the
English schema listing plus the handful of genuine identifier references.
Golden snapshots regenerated deliberately, since the prompts really did change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README, PORTING.md and SCHEMA.md all described Swedish column names that no longer
exist. SCHEMA.md is rewritten around the English names and now explains why the
awkward ones are what they are: speaker_name rather than speaker (English "Speaker"
is the presiding officer), session_label rather than term (the European Parliament
uses term for its five-year cycle), document_proposals for a concept English has no
single word for.
Also records two things learned while verifying the migration: pg_dump does not
capture database-level settings, so a restore silently loses app.fts_config and
searches come back empty with no error; and saved chat snapshots contain SQL written
against the old column names, which is why the rename cannot be a clean break.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
talks -> speeches ("talks" reads as conference talks to everyone outside this
project), motions -> documents with a new doc_type column so bills, written
questions and committee reports can share the table later, and 79 columns from
anforandetext -> text, intressent_id -> person_id, valkrets -> constituency,
lydelse -> text, and so on.
_postgres/rename_map.py is the single source of truth. The migration, its rollback,
and the code rewrite are all derived from it, so they cannot drift apart. Hand-
writing a rollback is how you end up with one that fails halfway through.
Two identifiers could not be renamed mechanically and were done by reading the
queries: dok_id means the protocol document in talks but the primary key in
motions, and year stays a calendar year in speeches while becoming session_year in
documents. The latter is aliased in SQL so the JSON field stays `year` and the
frontend contract is unchanged.
Values are never translated. Bifall and Avslag stay as published; parliament.yaml
glosses them. A research tool must not silently rewrite the record.
The migration is guarded by an existence check, so the same file is a no-op on a
fresh database and does the work on an existing one — one schema definition in the
world. ALTER TABLE ... RENAME is catalog-only, so the millions of HNSW-indexed
vectors are untouched.
The trigger functions are recreated explicitly, because plpgsql bodies are stored
as opaque text and do not follow renames: they would have compiled fine and then
failed at the next INSERT. They now read the text-search config from a database
setting rather than hardcoding 'swedish'.
Verified: migration round-trips to a byte-identical schema across columns, indexes
and triggers; re-running is a no-op; triggers repopulate search_vector with working
Swedish stemming on INSERT and UPDATE; and the renamed code runs real searches
against a migrated database holding 5,000 rows of production data, with plain,
prefix and exclusion syntax all working. Frontend type errors went from 9 to 7 —
the rename fixed two and introduced none.
Not yet done, and tracked: legacy compatibility views, the shim for SQL replayed
from saved snapshots, tool-name aliases, and updating docs to the new names.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AGPL text was fetched from a community license-template repository. Checked it
against GitHub's choosealicense corpus: the operative terms are identical across
both, and the only differences were the copyright placeholder (intentionally filled
in) plus URL schemes and one line wrap in the appendix example notice.
Normalised to the current canonical form. A license file is not somewhere to trust
a single unverified source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AGPL-3.0-or-later, with a README note offering other terms on request for
newsrooms and public-interest projects.
README rewritten in English, replacing a 55 KB Swedish document that described a
Sweden-only tool. It states plainly what is and is not yet country-agnostic rather
than overselling: the schema still carries Swedish column names, and docs/SCHEMA.md
translates them until the rename lands.
docs/PORTING.md is the guide this whole effort exists for — what a Bulgarian, UK or
EU deployment actually has to write. It is honest about the parts that are real work
(the ingest adapter, data quality, the untranslated UI) and about the assumption
underneath the data model, so nobody invests in an adapter for a parliament that does
not fit.
SECURITY.md documents the database_query issue rather than leaving deployers to find
it, and gives the read-only role to run against. .env.example documents all 48
environment variables, which were previously discoverable only by grep.
CI runs ruff, applies schema.sql to a clean pgvector database, runs the tests, builds
the frontend, and scans for secrets. The schema step exists because that file drifted
from production once already. tsc is non-blocking until the nine inherited type
errors are fixed.
Makefile rewritten around a documented setup/dev/test/lint flow, and carries the
check-fork-divergence target the production fork uses to prove it differs from
upstream only under deploy/prod/.
Also removed the personal GitHub Copilot instructions file.
Verified: no secrets, no personal contact details, and no private IPs or hostnames
in shipped source; backend still imports.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sixteen system prompts lived as module-level string constants across chat.py,
board.py, synthesis.py, trip.py and llm_tools.py. They are the main thing a fork
for another parliament has to rewrite, and editing them meant editing Python.
They now live under prompts/sv/ as Markdown. Placeholders use string.Template
($name), not str.format: three of these prompts embed literal JSON braces that
str.format raises on, and safe_substitute leaves an unknown placeholder alone
rather than killing a live chat turn over a typo.
Technical configuration that had leaked into the prompt text is now templated:
websearch_to_tsquery('swedish', ...) became '$fts_config', "Answer in Swedish"
became "Answer in $answer_language", and the åäö preservation rule became
$preserve_characters. Domain vocabulary from parliament.yaml is available too, so a
prompt can say $speech_plural and read naturally in any language.
PROMPTS_RELOAD=1 re-reads the files per call, so prompt iteration no longer needs
a server restart.
The prompts themselves stay Swedish. That is the intended design: a fork writes
prompts/<lang>/ in its own language, and the loader falls back through
<lang>/ -> shared -> en/.
Verified by snapshotting all sixteen constants before the move and asserting
equality after: 16/16 render byte-identically, including after parameterization —
which is what proves the templating substitutes exactly what was there before.
tests/test_prompts_golden.py keeps that guarantee going forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything that made this Sweden-only in code now lives in one configuration file:
party list and colours, chamber activity types, the Postgres text-search
dictionary, the party-code pattern used to verify attributions, identifier shapes,
source URLs, the person-photo URL template, embedding model and dimension, and the
site copy.
info.py is deleted. Its party_colors_lighten table is gone too — the tint is now
computed from the party colour, so there is no second palette to keep in sync. Its
dead select_columns and css strings go with it.
The stylesheet no longer carries per-party rules. It had --party-M through
--party-NYD plus twenty [data-party="..."] selectors, which no amount of
configuration could adapt: CSS cannot read a dict. App.tsx now publishes the
configured palette as custom properties from /api/meta, and one color-mix rule
covers every party in any country.
The author's email and Twitter handle are out of the source entirely. Explainer
copy moved to content/sv/*.md, and contact details come from `site.contact`, which
upstream ships empty for a deployment to fill in via its own PARLIAMENT_CONFIG.
Two startup assertions added, both for failures that are otherwise silent: a
text-search config mismatch returns near-zero rows with no error, and a vector
column that disagrees with embeddings.dimension fails deep inside pgvector with a
message that never mentions configuration.
Verified: five searches spanning phrase, exclusion and prefix syntax — exercising
all three tsquery builders that changed — return byte-identical payloads against
production, with differing hit counts and first hits proving the queries really
differ. Frontend builds, and tsc reports the same 9 pre-existing errors as before,
none new.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
schema.sql had drifted badly enough that a newcomer running it got a database the
code partly fails against. Missing entirely: error_log, llm_events, and the four
eval_* tables. Missing columns: talks.dok_id (written by every ingest run),
talks.summary_embedding, talks.arguments/arguments_corrected/tagging_failed,
debates.summary_embedding, and four chat_snapshots columns including llm_messages.
Six indexes were undocumented, among them both HNSW summary-embedding indexes.
Also removed `*.sql` from .gitignore in the previous commit, which is why 14 of the
16 migration files could be recovered at all.
This has to be right before the rename lands, since everything downstream treats
schema.sql as the definition of truth.
Verified by building a scratch database from schema.sql and diffing it against
production: 251 columns across 22 tables and all 66 indexes identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project could only run from /home/lasse/riksdagen: 22 files hardcoded that
path, mostly as an `os.chdir(...)` + `sys.path.append(...)` prelude. Those now
derive the root from the file's own location via bootstrap.py, and bulk data
resolves against PLENUM_DATA_DIR.
requirements.txt was from March 2023 and listed Streamlit, altair and pydeck while
omitting fastapi, uvicorn, psycopg2 and pgvector. Replaced with a pyproject.toml
derived from the actual import graph, plus ruff and pytest config.
Postgres opened its connection pool in __init__, so `import backend.app` failed
outright without a reachable database — breaking test collection and any tooling
that merely imports the app. The pool is now opened on first use behind a lock.
systemd units and an nginx site move to deploy/examples/ with __PROJECT_ROOT__ and
__DOMAIN__ placeholders. Real values belong in the deployment's own deploy/prod/,
which is a path upstream never writes to, so the two cannot conflict on merge.
Removed: scripts/migrate_keys.py (an already-executed Arango migration referenced
nowhere) and a dead `import talks2db` in download_talks.py that pointed at a file
excluded from this repo.
Verified: `import backend.app` now succeeds with neither a database nor any network,
and all rewritten scripts import from a foreign working directory with their module
docstrings intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_llm called env_manager.set_env() at import time, which connected to a private
ArangoDB to fetch secrets. That single line meant a fresh clone could not start,
regardless of what else was configured. Both packages also lived in separate
private repos and were gitignored here, so the code shipped without them.
packages/llm/ is 780 lines against _llm's 1750. Dropped as unused by this project
(measured, zero call sites): token counting and message trimming, image/vision
handling, make_summary, the ollama-specific paths, the query/user_input/context
argument style, and the self-mutating provider_quirks.json cache.
Kept and reworked:
- tools.py, the docstring -> JSON-schema tool registry, which has no equivalent
in the cuj-fup client and which llm_tools.py depends on entirely.
- The provider quirks that actually matter: vLLM-only extra_body fields stripped
for hosted providers, enable_thinking disabled at template level when think is
off, reasoning models (o1/o3/o4/gpt-5) switched to max_completion_tokens.
Adopted from cuj-fup's client: LLMConfig as a dataclass instead of 20 constructor
kwargs, the SDK's native max_retries instead of hand-rolled backoff, and error
messages that name the likely cause.
Fixes a latent bug: Optional[list[str]] parameters were advertised to the model as
strings, because get_origin(Optional[X]) is Union, so neither the schema mapping
nor the list coercion in execute_tool fired. `parties`, `people` and `focus_ids`
were all affected.
Also drops a dead SELECT-only guard in execute_tool that keyed on a parameter name
(`sql_query`) that no tool has ever used. Real SQL hardening is tracked separately.
Verified: `import backend.app` succeeds with all external network blocked and zero
outbound connection attempts; all 12 tools register; live vLLM calls confirmed for
plain generation, structured output via format=, tool execution, and the
error-returns-a-string contract that call sites branch on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeded via an explicit allow-list (see /home/lasse/plenum-seed.sh) rather than by
deleting files from a copy, so nothing sensitive can survive by omission.
Excluded: WireGuard backup + client config, the plaintext DB password in admin.py,
Arango credentials in scripts/notes.md, .claude/settings.json, a 113 MB log,
providers.yaml (private endpoint), the Arango/ChromaDB-era scripts, the duplicated
claude-design-system frontend copy, and assorted screenshots and one-off planning docs.
297 tracked files / 49 MB of history -> 161 files / 2.1 MB.
Recovered 14 database migrations that the old .gitignore's `*.sql` rule had been
hiding from version control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lasse Server
