plenum

0 Tags

1.4 MiB

Author	SHA1	Message	Date
Lasse Edfast	e2b71c64b1	Add the ingest pipeline and CLI, replacing the ad-hoc scripts `python -m ingest.cli fetch\|load\|sync` now exists — README, PORTING.md, the systemd example and pyproject's console script have all been referencing it. This replaces six scripts that the rename had left genuinely broken: the speech loader's INSERT listed `year, year` as two columns, and both loaders read the source JSON using plenum's column names rather than the Riksdag's field names, so a sync would have written empty rows without erroring. Rewriting on top of the adapter was safer than repairing them, because the adapter makes the source-name/column-name boundary explicit rather than leaving it to be remembered. Two source quirks found by running it over real archives rather than reasoning about it: a few records carry a bare string where a child element belongs, which took the whole load down on an AttributeError; and party codes appear in both cases, so `parties` held {S,s} and any filter on 'S' would have silently missed half the documents. Both are handled in the adapter, where source messiness belongs. Also adds doc_type to schema.sql. It was only in the migration, so a fresh install lacked a column the loader writes. Verified end to end against real archives: 400 speeches and 400 documents ingested into a clean database, producing 3,742 authors and 2,229 proposals, with search vectors built by the triggers and full-text queries returning hits. Post-load steps (debate ids, embeddings, summaries) still run separately via scripts/debates.py and scripts/make_embeddings.py; wiring them into `sync` is tracked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	80771ff4aa	Rename the schema and code from Swedish to English talks -> speeches ("talks" reads as conference talks to everyone outside this project), motions -> documents with a new doc_type column so bills, written questions and committee reports can share the table later, and 79 columns from anforandetext -> text, intressent_id -> person_id, valkrets -> constituency, lydelse -> text, and so on. _postgres/rename_map.py is the single source of truth. The migration, its rollback, and the code rewrite are all derived from it, so they cannot drift apart. Hand- writing a rollback is how you end up with one that fails halfway through. Two identifiers could not be renamed mechanically and were done by reading the queries: dok_id means the protocol document in talks but the primary key in motions, and year stays a calendar year in speeches while becoming session_year in documents. The latter is aliased in SQL so the JSON field stays `year` and the frontend contract is unchanged. Values are never translated. Bifall and Avslag stay as published; parliament.yaml glosses them. A research tool must not silently rewrite the record. The migration is guarded by an existence check, so the same file is a no-op on a fresh database and does the work on an existing one — one schema definition in the world. ALTER TABLE ... RENAME is catalog-only, so the millions of HNSW-indexed vectors are untouched. The trigger functions are recreated explicitly, because plpgsql bodies are stored as opaque text and do not follow renames: they would have compiled fine and then failed at the next INSERT. They now read the text-search config from a database setting rather than hardcoding 'swedish'. Verified: migration round-trips to a byte-identical schema across columns, indexes and triggers; re-running is a no-op; triggers repopulate search_vector with working Swedish stemming on INSERT and UPDATE; and the renamed code runs real searches against a migrated database holding 5,000 rows of production data, with plain, prefix and exclusion syntax all working. Frontend type errors went from 9 to 7 — the rename fixed two and introduced none. Not yet done, and tracked: legacy compatibility views, the shim for SQL replayed from saved snapshots, tool-name aliases, and updating docs to the new names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	0be70f9307	Move country-specific values into parliament.yaml Everything that made this Sweden-only in code now lives in one configuration file: party list and colours, chamber activity types, the Postgres text-search dictionary, the party-code pattern used to verify attributions, identifier shapes, source URLs, the person-photo URL template, embedding model and dimension, and the site copy. info.py is deleted. Its party_colors_lighten table is gone too — the tint is now computed from the party colour, so there is no second palette to keep in sync. Its dead select_columns and css strings go with it. The stylesheet no longer carries per-party rules. It had --party-M through --party-NYD plus twenty [data-party="..."] selectors, which no amount of configuration could adapt: CSS cannot read a dict. App.tsx now publishes the configured palette as custom properties from /api/meta, and one color-mix rule covers every party in any country. The author's email and Twitter handle are out of the source entirely. Explainer copy moved to content/sv/*.md, and contact details come from `site.contact`, which upstream ships empty for a deployment to fill in via its own PARLIAMENT_CONFIG. Two startup assertions added, both for failures that are otherwise silent: a text-search config mismatch returns near-zero rows with no error, and a vector column that disagrees with embeddings.dimension fails deep inside pgvector with a message that never mentions configuration. Verified: five searches spanning phrase, exclusion and prefix syntax — exercising all three tsquery builders that changed — return byte-identical payloads against production, with differing hit counts and first hits proving the queries really differ. Frontend builds, and tsc reports the same 9 pre-existing errors as before, none new. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	d36ace941b	Add pyproject, remove hardcoded paths, make the DB pool lazy The project could only run from /home/lasse/riksdagen: 22 files hardcoded that path, mostly as an `os.chdir(...)` + `sys.path.append(...)` prelude. Those now derive the root from the file's own location via bootstrap.py, and bulk data resolves against PLENUM_DATA_DIR. requirements.txt was from March 2023 and listed Streamlit, altair and pydeck while omitting fastapi, uvicorn, psycopg2 and pgvector. Replaced with a pyproject.toml derived from the actual import graph, plus ruff and pytest config. Postgres opened its connection pool in __init__, so `import backend.app` failed outright without a reachable database — breaking test collection and any tooling that merely imports the app. The pool is now opened on first use behind a lock. systemd units and an nginx site move to deploy/examples/ with __PROJECT_ROOT__ and __DOMAIN__ placeholders. Real values belong in the deployment's own deploy/prod/, which is a path upstream never writes to, so the two cannot conflict on merge. Removed: scripts/migrate_keys.py (an already-executed Arango migration referenced nowhere) and a dead `import talks2db` in download_talks.py that pointed at a file excluded from this repo. Verified: `import backend.app` now succeeds with neither a database nor any network, and all rewritten scripts import from a foreign working directory with their module docstrings intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	843db25211	Replace _llm and colorprinter with self-contained packages _llm called env_manager.set_env() at import time, which connected to a private ArangoDB to fetch secrets. That single line meant a fresh clone could not start, regardless of what else was configured. Both packages also lived in separate private repos and were gitignored here, so the code shipped without them. packages/llm/ is 780 lines against _llm's 1750. Dropped as unused by this project (measured, zero call sites): token counting and message trimming, image/vision handling, make_summary, the ollama-specific paths, the query/user_input/context argument style, and the self-mutating provider_quirks.json cache. Kept and reworked: - tools.py, the docstring -> JSON-schema tool registry, which has no equivalent in the cuj-fup client and which llm_tools.py depends on entirely. - The provider quirks that actually matter: vLLM-only extra_body fields stripped for hosted providers, enable_thinking disabled at template level when think is off, reasoning models (o1/o3/o4/gpt-5) switched to max_completion_tokens. Adopted from cuj-fup's client: LLMConfig as a dataclass instead of 20 constructor kwargs, the SDK's native max_retries instead of hand-rolled backoff, and error messages that name the likely cause. Fixes a latent bug: Optional[list[str]] parameters were advertised to the model as strings, because get_origin(Optional[X]) is Union, so neither the schema mapping nor the list coercion in execute_tool fired. `parties`, `people` and `focus_ids` were all affected. Also drops a dead SELECT-only guard in execute_tool that keyed on a parameter name (`sql_query`) that no tool has ever used. Real SQL hardening is tracked separately. Verified: `import backend.app` succeeds with all external network blocked and zero outbound connection attempts; all 12 tools register; live vLLM calls confirmed for plain generation, structured output via format=, tool execution, and the error-returns-a-string contract that call sites branch on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	09c1b3d79c	Initial import from rixdagen Seeded via an explicit allow-list (see /home/lasse/plenum-seed.sh) rather than by deleting files from a copy, so nothing sensitive can survive by omission. Excluded: WireGuard backup + client config, the plaintext DB password in admin.py, Arango credentials in scripts/notes.md, .claude/settings.json, a 113 MB log, providers.yaml (private endpoint), the Arango/ChromaDB-era scripts, the duplicated claude-design-system frontend copy, and assorted screenshots and one-off planning docs. 297 tracked files / 49 MB of history -> 161 files / 2.1 MB. Recovered 14 database migrations that the old .gitignore's `*.sql` rule had been hiding from version control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago

6 Commits (e2b71c64b163004487d58b4965157c7429270661)