`python -m ingest.cli fetch|load|sync` now exists — README, PORTING.md, the systemd
example and pyproject's console script have all been referencing it.
This replaces six scripts that the rename had left genuinely broken: the speech
loader's INSERT listed `year, year` as two columns, and both loaders read the source
JSON using plenum's column names rather than the Riksdag's field names, so a sync
would have written empty rows without erroring. Rewriting on top of the adapter was
safer than repairing them, because the adapter makes the source-name/column-name
boundary explicit rather than leaving it to be remembered.
Two source quirks found by running it over real archives rather than reasoning about
it: a few records carry a bare string where a child element belongs, which took the
whole load down on an AttributeError; and party codes appear in both cases, so
`parties` held {S,s} and any filter on 'S' would have silently missed half the
documents. Both are handled in the adapter, where source messiness belongs.
Also adds doc_type to schema.sql. It was only in the migration, so a fresh install
lacked a column the loader writes.
Verified end to end against real archives: 400 speeches and 400 documents ingested
into a clean database, producing 3,742 authors and 2,229 proposals, with search
vectors built by the triggers and full-text queries returning hits.
Post-load steps (debate ids, embeddings, summaries) still run separately via
scripts/debates.py and scripts/make_embeddings.py; wiring them into `sync` is
tracked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lasse Edfast
e2b71c64b1