`python -m ingest.cli fetch|load|sync` now exists — README, PORTING.md, the systemd
example and pyproject's console script have all been referencing it.
This replaces six scripts that the rename had left genuinely broken: the speech
loader's INSERT listed `year, year` as two columns, and both loaders read the source
JSON using plenum's column names rather than the Riksdag's field names, so a sync
would have written empty rows without erroring. Rewriting on top of the adapter was
safer than repairing them, because the adapter makes the source-name/column-name
boundary explicit rather than leaving it to be remembered.
Two source quirks found by running it over real archives rather than reasoning about
it: a few records carry a bare string where a child element belongs, which took the
whole load down on an AttributeError; and party codes appear in both cases, so
`parties` held {S,s} and any filter on 'S' would have silently missed half the
documents. Both are handled in the adapter, where source messiness belongs.
Also adds doc_type to schema.sql. It was only in the migration, so a fresh install
lacked a column the loader writes.
Verified end to end against real archives: 400 speeches and 400 documents ingested
into a clean database, producing 3,742 authors and 2,229 proposals, with search
vectors built by the triggers and full-text queries returning hits.
Post-load steps (debate ids, embeddings, summaries) still run separately via
scripts/debates.py and scripts/make_embeddings.py; wiring them into `sync` is
tracked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
talks -> speeches ("talks" reads as conference talks to everyone outside this
project), motions -> documents with a new doc_type column so bills, written
questions and committee reports can share the table later, and 79 columns from
anforandetext -> text, intressent_id -> person_id, valkrets -> constituency,
lydelse -> text, and so on.
_postgres/rename_map.py is the single source of truth. The migration, its rollback,
and the code rewrite are all derived from it, so they cannot drift apart. Hand-
writing a rollback is how you end up with one that fails halfway through.
Two identifiers could not be renamed mechanically and were done by reading the
queries: dok_id means the protocol document in talks but the primary key in
motions, and year stays a calendar year in speeches while becoming session_year in
documents. The latter is aliased in SQL so the JSON field stays `year` and the
frontend contract is unchanged.
Values are never translated. Bifall and Avslag stay as published; parliament.yaml
glosses them. A research tool must not silently rewrite the record.
The migration is guarded by an existence check, so the same file is a no-op on a
fresh database and does the work on an existing one — one schema definition in the
world. ALTER TABLE ... RENAME is catalog-only, so the millions of HNSW-indexed
vectors are untouched.
The trigger functions are recreated explicitly, because plpgsql bodies are stored
as opaque text and do not follow renames: they would have compiled fine and then
failed at the next INSERT. They now read the text-search config from a database
setting rather than hardcoding 'swedish'.
Verified: migration round-trips to a byte-identical schema across columns, indexes
and triggers; re-running is a no-op; triggers repopulate search_vector with working
Swedish stemming on INSERT and UPDATE; and the renamed code runs real searches
against a migrated database holding 5,000 rows of production data, with plain,
prefix and exclusion syntax all working. Frontend type errors went from 9 to 7 —
the rename fixed two and introduced none.
Not yet done, and tracked: legacy compatibility views, the shim for SQL replayed
from saved snapshots, tool-name aliases, and updating docs to the new names.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
schema.sql had drifted badly enough that a newcomer running it got a database the
code partly fails against. Missing entirely: error_log, llm_events, and the four
eval_* tables. Missing columns: talks.dok_id (written by every ingest run),
talks.summary_embedding, talks.arguments/arguments_corrected/tagging_failed,
debates.summary_embedding, and four chat_snapshots columns including llm_messages.
Six indexes were undocumented, among them both HNSW summary-embedding indexes.
Also removed `*.sql` from .gitignore in the previous commit, which is why 14 of the
16 migration files could be recovered at all.
This has to be right before the rename lands, since everything downstream treats
schema.sql as the definition of truth.
Verified by building a scratch database from schema.sql and diffing it against
production: 251 columns across 22 tables and all 66 indexes identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeded via an explicit allow-list (see /home/lasse/plenum-seed.sh) rather than by
deleting files from a copy, so nothing sensitive can survive by omission.
Excluded: WireGuard backup + client config, the plaintext DB password in admin.py,
Arango credentials in scripts/notes.md, .claude/settings.json, a 113 MB log,
providers.yaml (private endpoint), the Arango/ChromaDB-era scripts, the duplicated
claude-design-system frontend copy, and assorted screenshots and one-off planning docs.
297 tracked files / 49 MB of history -> 161 files / 2.1 MB.
Recovered 14 database migrations that the old .gitignore's `*.sql` rule had been
hiding from version control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lasse Edfast