plenum

0 Tags

1.4 MiB

Author	SHA1	Message	Date
Lasse Server	7ea92c9366	Stream the final chat answer instead of one blocking blob packages/llm/client.py:_read_stream yielded token-by-token deltas but was dead code — nothing called generate(stream=True). The chat tool loop always blocked on the final answer generation, so a ~40s answer arrived as a single SSE event and read as a hang, even though the "card stack" already streamed tool-selection progress. The final answer is structurally just whichever tool-loop iteration returns content with no tool_calls — there's no way to know in advance which iteration that'll be, and a turn can carry short narration before a tool call in the very same response. So streaming has to be speculative: - packages/llm/client.py: _read_stream becomes StreamAccumulator, which both streams live (thinking/content/tool_call) deltas and reconstructs a full ChatCompletionMessage (content, tool_calls, reasoning_content) once the stream ends. Tool-call argument fragments are accumulated by index, same pattern every OpenAI-compatible streaming client needs. Downstream tool execution code needed zero changes because of this — it just consumes the reconstructed message instead of the SDK's blocking return value. Also hardened <think>/</think> detection against a tag split across chunk boundaries (was a per-chunk substring check; now a carry-over buffer) — latent before since nothing exercised streaming for real. - backend/services/streaming_answer.py (new): AnswerStreamFilter holds buffered content back until it's implausible to still be one-line narration (this codebase already caps narration at 150 chars elsewhere), and separately holds back any trailing bracket run that could still grow into a [src:ID]/[[src:ID]] marker — citation validity isn't known until parse_and_renumber_citations runs on the complete text, so a raw marker, hallucinated or not, must never reach the client even for a frame. run_streaming_iteration drives one tool-loop iteration through this filter and emits "answer_delta" events; if a tool_call shows up after content already cleared the gate (narration grew unusually long before the model pivoted), it emits one "answer_delta_retract" so the UI doesn't strand a stale preview. - backend/services/chat.py, mp_chat.py: thread an explicit stream_answer flag through to _run_tool_loop's single generate() call site, rather than inferring streaming from event_callback being non-None — eval logging's ConversationRecorder.wrap() always returns a non-None callable, even for the plain blocking /api/chat endpoint, so that inference would have been wrong. Only stream_chat_response (the two SSE routes) sets it; /api/chat is untouched. - Frontend: both SSE consumers (ChatPanel.tsx's card stack, MpChatPanel.tsx's simpler turn list) grow "answer_delta"/"answer_delta_retract" handling. The live preview renders as plain, React-escaped text (no dangerouslySetInnerHTML, no markdown parsing mid-stream — that's a one-shot full-document transform, unsafe on partial input) and is fully replaced by the authoritative, citation-renumbered/person-linked/ language-polished HTML once the terminal "answer" event lands. Both event types were previously-unhandled and silently ignored, so the backend and frontend halves are independently safe to deploy. Added tests/test_llm_client_stream.py and tests/test_streaming_answer.py covering the reconstruction (content, tool-call fragment accumulation, split <think> tags, truncated-stream handling) and the filter (gate timing, citation-marker safety across chunk boundaries, the retract path, error propagation). 5 pre-existing failures in test_prompts_golden.py/ test_provenance.py are unrelated (reproduce identically on main).	6 days ago
Lasse Edfast	ac181022e5	Remove the last ArangoDB remnants; exclude .claude/ wholesale This project moved to Postgres some time ago, but ~380 lines of SQL-to-AQL translation and AQL syntax validation were still sitting in utils.py, reachable from nothing. utils.py is now just TextChunker, which is the only part anything imports. Also drops a stray prompt draft that had been left in backend/services/, and the _normalize_arango_search_args name, which never touched Arango. Worth knowing separately: the Arango server is still running and still authenticating, so the riksdagsgruppen account is a live door into a system nothing here uses any more. Deleting that account is simpler than rotating its password. The env_manager account is a different one and other projects still depend on it. .gitignore now excludes .claude/ entirely rather than just settings.local.json. settings.json can carry credentials inside allowlisted command strings, which is exactly how a live database password ended up committed to the predecessor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	bad2ad89e3	Refuse writes from model-authored SQL; finish the tool rename database_query passed model-written SQL straight to a connection that commits. The corpus reaches the model's context, and in a parliament anyone able to speak can get text into the corpus, so that SQL has to be treated as untrusted input. Two layers now apply. Statements must begin with SELECT or WITH, and multi-statement payloads are rejected — that is how a write gets smuggled in behind a leading SELECT. Then the query runs inside SET TRANSACTION READ ONLY, so PostgreSQL rejects writes itself even if the first check is wrong. Verified both independently: five attack shapes refused by the guard, and INSERT/DROP still refused when the guard is bypassed entirely and the database layer is called directly. share_insight gets the same treatment. It re-executes SQL stored in saved conversations, which is no more trustworthy than freshly generated SQL. Neither layer replaces running against a SELECT-only role, and SECURITY.md still says so — it now describes what the application does rather than what it fails to do. Tool rename completed as a clean break, no aliases, since there are three users and six sessions to preserve: arango_search -> search_speeches (it has queried Postgres since the Arango migration and the name was pure debt), search_motions -> search_documents, fetch_motion -> fetch_document. Also renamed fetch_documents -> fetch_speeches. It fetches speeches despite the name, and sitting next to the new fetch_document it was a trap the model would fall into. Note for anyone with an MCP client configured against the old names: riksdagen_mcp exposes these, so the names it advertises have changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	80771ff4aa	Rename the schema and code from Swedish to English talks -> speeches ("talks" reads as conference talks to everyone outside this project), motions -> documents with a new doc_type column so bills, written questions and committee reports can share the table later, and 79 columns from anforandetext -> text, intressent_id -> person_id, valkrets -> constituency, lydelse -> text, and so on. _postgres/rename_map.py is the single source of truth. The migration, its rollback, and the code rewrite are all derived from it, so they cannot drift apart. Hand- writing a rollback is how you end up with one that fails halfway through. Two identifiers could not be renamed mechanically and were done by reading the queries: dok_id means the protocol document in talks but the primary key in motions, and year stays a calendar year in speeches while becoming session_year in documents. The latter is aliased in SQL so the JSON field stays `year` and the frontend contract is unchanged. Values are never translated. Bifall and Avslag stay as published; parliament.yaml glosses them. A research tool must not silently rewrite the record. The migration is guarded by an existence check, so the same file is a no-op on a fresh database and does the work on an existing one — one schema definition in the world. ALTER TABLE ... RENAME is catalog-only, so the millions of HNSW-indexed vectors are untouched. The trigger functions are recreated explicitly, because plpgsql bodies are stored as opaque text and do not follow renames: they would have compiled fine and then failed at the next INSERT. They now read the text-search config from a database setting rather than hardcoding 'swedish'. Verified: migration round-trips to a byte-identical schema across columns, indexes and triggers; re-running is a no-op; triggers repopulate search_vector with working Swedish stemming on INSERT and UPDATE; and the renamed code runs real searches against a migrated database holding 5,000 rows of production data, with plain, prefix and exclusion syntax all working. Frontend type errors went from 9 to 7 — the rename fixed two and introduced none. Not yet done, and tracked: legacy compatibility views, the shim for SQL replayed from saved snapshots, tool-name aliases, and updating docs to the new names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	b8a775ca27	Move prompts out of Python into prompts/ Sixteen system prompts lived as module-level string constants across chat.py, board.py, synthesis.py, trip.py and llm_tools.py. They are the main thing a fork for another parliament has to rewrite, and editing them meant editing Python. They now live under prompts/sv/ as Markdown. Placeholders use string.Template ($name), not str.format: three of these prompts embed literal JSON braces that str.format raises on, and safe_substitute leaves an unknown placeholder alone rather than killing a live chat turn over a typo. Technical configuration that had leaked into the prompt text is now templated: websearch_to_tsquery('swedish', ...) became '$fts_config', "Answer in Swedish" became "Answer in $answer_language", and the åäö preservation rule became $preserve_characters. Domain vocabulary from parliament.yaml is available too, so a prompt can say $speech_plural and read naturally in any language. PROMPTS_RELOAD=1 re-reads the files per call, so prompt iteration no longer needs a server restart. The prompts themselves stay Swedish. That is the intended design: a fork writes prompts/<lang>/ in its own language, and the loader falls back through <lang>/ -> shared -> en/. Verified by snapshotting all sixteen constants before the move and asserting equality after: 16/16 render byte-identically, including after parameterization — which is what proves the templating substitutes exactly what was there before. tests/test_prompts_golden.py keeps that guarantee going forward. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	0be70f9307	Move country-specific values into parliament.yaml Everything that made this Sweden-only in code now lives in one configuration file: party list and colours, chamber activity types, the Postgres text-search dictionary, the party-code pattern used to verify attributions, identifier shapes, source URLs, the person-photo URL template, embedding model and dimension, and the site copy. info.py is deleted. Its party_colors_lighten table is gone too — the tint is now computed from the party colour, so there is no second palette to keep in sync. Its dead select_columns and css strings go with it. The stylesheet no longer carries per-party rules. It had --party-M through --party-NYD plus twenty [data-party="..."] selectors, which no amount of configuration could adapt: CSS cannot read a dict. App.tsx now publishes the configured palette as custom properties from /api/meta, and one color-mix rule covers every party in any country. The author's email and Twitter handle are out of the source entirely. Explainer copy moved to content/sv/*.md, and contact details come from `site.contact`, which upstream ships empty for a deployment to fill in via its own PARLIAMENT_CONFIG. Two startup assertions added, both for failures that are otherwise silent: a text-search config mismatch returns near-zero rows with no error, and a vector column that disagrees with embeddings.dimension fails deep inside pgvector with a message that never mentions configuration. Verified: five searches spanning phrase, exclusion and prefix syntax — exercising all three tsquery builders that changed — return byte-identical payloads against production, with differing hit counts and first hits proving the queries really differ. Frontend builds, and tsc reports the same 9 pre-existing errors as before, none new. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	843db25211	Replace _llm and colorprinter with self-contained packages _llm called env_manager.set_env() at import time, which connected to a private ArangoDB to fetch secrets. That single line meant a fresh clone could not start, regardless of what else was configured. Both packages also lived in separate private repos and were gitignored here, so the code shipped without them. packages/llm/ is 780 lines against _llm's 1750. Dropped as unused by this project (measured, zero call sites): token counting and message trimming, image/vision handling, make_summary, the ollama-specific paths, the query/user_input/context argument style, and the self-mutating provider_quirks.json cache. Kept and reworked: - tools.py, the docstring -> JSON-schema tool registry, which has no equivalent in the cuj-fup client and which llm_tools.py depends on entirely. - The provider quirks that actually matter: vLLM-only extra_body fields stripped for hosted providers, enable_thinking disabled at template level when think is off, reasoning models (o1/o3/o4/gpt-5) switched to max_completion_tokens. Adopted from cuj-fup's client: LLMConfig as a dataclass instead of 20 constructor kwargs, the SDK's native max_retries instead of hand-rolled backoff, and error messages that name the likely cause. Fixes a latent bug: Optional[list[str]] parameters were advertised to the model as strings, because get_origin(Optional[X]) is Union, so neither the schema mapping nor the list coercion in execute_tool fired. `parties`, `people` and `focus_ids` were all affected. Also drops a dead SELECT-only guard in execute_tool that keyed on a parameter name (`sql_query`) that no tool has ever used. Real SQL hardening is tracked separately. Verified: `import backend.app` succeeds with all external network blocked and zero outbound connection attempts; all 12 tools register; live vLLM calls confirmed for plain generation, structured output via format=, tool execution, and the error-returns-a-string contract that call sites branch on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago
Lasse Edfast	09c1b3d79c	Initial import from rixdagen Seeded via an explicit allow-list (see /home/lasse/plenum-seed.sh) rather than by deleting files from a copy, so nothing sensitive can survive by omission. Excluded: WireGuard backup + client config, the plaintext DB password in admin.py, Arango credentials in scripts/notes.md, .claude/settings.json, a 113 MB log, providers.yaml (private endpoint), the Arango/ChromaDB-era scripts, the duplicated claude-design-system frontend copy, and assorted screenshots and one-off planning docs. 297 tracked files / 49 MB of history -> 161 files / 2.1 MB. Recovered 14 database migrations that the old .gitignore's `*.sql` rule had been hiding from version control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago

8 Commits (291dcebcb8d5e3f1d9ba034b016fa0b42d018760)