`python -m ingest.cli fetch|load|sync` now exists — README, PORTING.md, the systemd
example and pyproject's console script have all been referencing it.
This replaces six scripts that the rename had left genuinely broken: the speech
loader's INSERT listed `year, year` as two columns, and both loaders read the source
JSON using plenum's column names rather than the Riksdag's field names, so a sync
would have written empty rows without erroring. Rewriting on top of the adapter was
safer than repairing them, because the adapter makes the source-name/column-name
boundary explicit rather than leaving it to be remembered.
Two source quirks found by running it over real archives rather than reasoning about
it: a few records carry a bare string where a child element belongs, which took the
whole load down on an AttributeError; and party codes appear in both cases, so
`parties` held {S,s} and any filter on 'S' would have silently missed half the
documents. Both are handled in the adapter, where source messiness belongs.
Also adds doc_type to schema.sql. It was only in the migration, so a fresh install
lacked a column the loader writes.
Verified end to end against real archives: 400 speeches and 400 documents ingested
into a clean database, producing 3,742 authors and 2,229 proposals, with search
vectors built by the triggers and full-text queries returning hits.
Post-load steps (debate ids, embeddings, summaries) still run separately via
scripts/debates.py and scripts/make_embeddings.py; wiring them into `sync` is
tracked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ingest/adapters/riksdagen.py is the only module that knows what the Riksdag calls
its fields. Everything downstream works in plenum's column names, which is what
makes another parliament a matter of writing one file rather than editing the
application.
Writing it surfaced a bug the rename introduced: the loaders read the *source* JSON,
whose keys are the Riksdag's Swedish field names, and the mechanical pass renamed
those reads as if they were column names. doc.get("anforandetext") became
doc.get("text"), doc.get("talare") became doc.get("speaker_name"), and so on. The
next sync would have written empty speeches without erroring. This is exactly the
confusion an adapter exists to prevent, since it makes the source-name/column-name
boundary explicit instead of implicit.
The adapter captures what this source actually does, verified against real files:
repeated elements arrive as an object when there is one and a list when there are
several; null is serialised as the string "None"; speech text is HTML; and the
speaker field carries the party in parentheses, which has to come off or names stop
joining to the member register. Output was checked field by field against what
production stores for the same record.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lasse Edfast