lasse
/
plenum
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update docs to the renamed schema

Browse Source 
README, PORTING.md and SCHEMA.md all described Swedish column names that no longer
exist. SCHEMA.md is rewritten around the English names and now explains why the
awkward ones are what they are: speaker_name rather than speaker (English "Speaker"
is the presiding officer), session_label rather than term (the European Parliament
uses term for its five-year cycle), document_proposals for a concept English has no
single word for.

Also records two things learned while verifying the migration: pg_dump does not
capture database-level settings, so a restore silently loses app.fts_config and
searches come back empty with no error; and saved chat snapshots contain SQL written
against the old column names, which is why the rename cannot be a clean break.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
main
Lasse Edfast 1 week ago
parent 80771ff4aa
commit a3388835aa
3 changed files with 93 additions and 70 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 8
      README.md
  2. 7
      docs/PORTING.md
  3. 148
      docs/SCHEMA.md

8
README.md
Unescape View File

@ -35,9 +35,11 @@ Adapting to another parliament means writing a config file, an ingest adapter, a
set of prompts in your language — not editing the application. See set of prompts in your language — not editing the application. See
[docs/PORTING.md](docs/PORTING.md). [docs/PORTING.md](docs/PORTING.md).
The database schema still uses Swedish column names (`anforandetext`, `intressent_id`, The database schema is country-neutral English: `speeches`, `documents`,
`yrkanden`). Renaming those to neutral English is planned and tracked; until then, `document_proposals`, `person_id`, `constituency`. Concepts that have no stable
[docs/SCHEMA.md](docs/SCHEMA.md) gives the translations. English equivalent — the Swedish *yrkande* or *riksmöte* — carry neutral column names
while the country's own word lives in `parliament.yaml`. See
[docs/SCHEMA.md](docs/SCHEMA.md).
## Requirements ## Requirements

7
docs/PORTING.md
Unescape View File

@ -89,9 +89,10 @@ through `/api/meta` and need no code change.
The search engine, chat orchestration, tool-calling, citation and attribution The search engine, chat orchestration, tool-calling, citation and attribution
checking, the deep-research agent, authentication and the encrypted chat store are checking, the deep-research agent, authentication and the encrypted chat store are
all country-neutral. So is the database schema — although its column names are still all country-neutral. So is the database schema: `speeches`, `documents`,
Swedish (`anforandetext`, `intressent_id`, `yrkanden`). They are just names; nothing `document_proposals`, `person_id`, `constituency`. Where a concept has no stable
reads meaning from them. [docs/SCHEMA.md](SCHEMA.md) translates them. English equivalent, the column name is neutral and your own word for it goes in
`vocabulary:`. See [docs/SCHEMA.md](SCHEMA.md).
## A caution about scope ## A caution about scope

148
docs/SCHEMA.md
Unescape View File

@ -1,113 +1,133 @@
# Database schema # Database schema
The definitive version is [`_postgres/schema.sql`](../_postgres/schema.sql), which is The definitive version is [`_postgres/schema.sql`](../_postgres/schema.sql), verified
verified against the production database. This page explains it. against the production database. This page explains it.
Column names are still Swedish, inherited from the original Riksdag-only project. Names are country-neutral English. Where a parliamentary concept has no stable
They are only names — nothing in the application derives meaning from them — but they English equivalent — the Swedish *yrkande*, *riksmöte* or *replik* — the column takes
are an obstacle to reading the code, so a rename to neutral English is planned. Until a neutral name and the country's own word lives in `parliament.yaml` under
then, this is the translation. `vocabulary:`, where prompts pick it up. Data values are never translated.
## Core tables ## Core tables
### `talks` — speeches in the chamber ### `speeches` — what was said in the chamber
One row per speech. The primary key `id` comes from the source; note the separate
`dok_id`, which identifies the *protocol document* the speech appeared in.
| Column | Meaning | | Column | Meaning |
|---|---| |---|---|
| `id` | Primary key, from the source record's `id` field | | `id` | Primary key, from the source record |
| `anforande_id` | The source's own UUID, preserved from an earlier key migration | | `source_speech_id` | The source's own UUID, kept from an earlier key migration |
| `anforandetext` | The speech text | | `text` | The speech itself |
| `avsnittsrubrik` | Heading of the agenda item | | `section_title` | Heading of the agenda item |
| `anforande_nummer` | Position within the debate | | `sequence` | Position within the debate |
| `kammaraktivitet` | Activity type code; see `activity_types` in `parliament.yaml` | | `activity_type` | Activity code; see `activity_types` in `parliament.yaml` |
| `talare` | Speaker name as printed. Not the presiding officer — "speaker" in the plain sense | | `speaker_name` | Who spoke, as printed. Not the presiding officer — plain `speaker` would be ambiguous in a parliamentary schema, which is why the column is not called that |
| `parti` | Party code | | `party`, `person_id` | Party code; person, referencing `people` |
| `intressent_id` | Person id, references `people` | | `date`, `source_datetime`, `year`, `session_year` | Date, raw source string, calendar year, and the parliamentary session's start year |
| `datum`, `dok_datum`, `year`, `period` | Date, raw source datetime, calendar year, session start year | | `source_doc_id`, `related_doc_id`, `source_doc_number`, `source_record_id` | Source document references. `source_doc_id` is the protocol document the speech appeared in — not this row's own key |
| `dok_id`, `rel_dok_id`, `dok_nummer`, `hangar_id` | Source document references | | `title` | Debate title |
| `titel` | Debate title | | `debate_id` | Groups speeches into a debate, `"{date}:{index}"` |
| `debate` | Debate grouping key, `"{date}:{index}"` | | `is_reply` | A short right-of-reply intervention. The Swedish *replik* carries procedural standing this name does not capture; nothing downstream depends on it |
| `replik` | Whether this is a rebuttal (a short right-of-reply intervention) |
| `summary`, `tags`, `arguments` | LLM-derived | | `summary`, `tags`, `arguments` | LLM-derived |
| `summary_embedding` | Embedding of `summary`, for debate-level semantic search | | `summary_embedding` | Embedding of `summary`, for debate-level semantic search |
| `search_vector` | Full-text vector, maintained by trigger | | `search_vector` | Full-text vector, maintained by trigger |
### `motions` — member-submitted documents ### `documents` — what members submitted
The Swedish *motion* is a formal proposal submitted by members. Each contains one or A *document* is anything a member formally puts to the chamber. `doc_type` says which
more numbered *yrkanden* — operative demands — which is what people actually search kind; today everything is `'motion'`, and the column exists so bills, written
and cite. questions and committee reports do not each need their own table later.
| Column | Meaning | | Column | Meaning |
|---|---| |---|---|
| `dok_id` | Primary key | | `doc_id` | Primary key |
| `rm` | Session label, e.g. `"2022/23"` (Swedish *riksmöte*) | | `doc_type` | Kind of document; defaults to `'motion'` |
| `beteckning` | Number within the session | | `session_label` | Which annual session, e.g. `"2022/23"`. Not called *term*: the European Parliament uses that for its five-year cycle |
| `subtyp` | Whether submitted by an individual, a committee group, or a whole party | | `designation` | Number within the session |
| `organ` | Committee the document was referred to | | `subtype` | Whether submitted by an individual, a committee group, or a whole party |
| `titel`, `undertitel`, `text` | Content | | `committee` | Committee it was referred to |
| `forslag_text` | The proposal texts concatenated — high-signal, weighted separately in search | | `title`, `subtitle`, `text` | Content |
| `proposals_text` | The proposal texts concatenated — high-signal, weighted separately in search |
| `parties`, `author_names` | Denormalised authorship, in signing order | | `parties`, `author_names` | Denormalised authorship, in signing order |
| `forslag`, `bilagor` | Raw proposal and attachment records from the source | | `proposals_raw`, `attachments` | Raw records from the source |
| `has_text` | False for documents that exist only as scanned PDFs | | `has_text` | False for documents that exist only as scanned PDFs |
| `session_year` | Derived from `session_label` |
### `document_proposals` — the operative demands inside a document
### `motion_yrkanden` — individual proposals within a document The unit people actually search and cite. The Swedish *yrkande* is a numbered,
formally worded demand; English has no single word for it, hence the neutral name.
| Column | Meaning | | Column | Meaning |
|---|---| |---|---|
| `lydelse` | The proposal's wording | | `text` | The proposal's wording |
| `utskottet` | The committee's recommendation | | `number`, `ordinal` | Its number as published, and its 0-based position |
| `kammaren` | The chamber's decision, e.g. `Bifall` (approved) / `Avslag` (rejected) | | `committee_recommendation` | What the committee recommended |
| `behandlas_i` | Which report handled it | | `chamber_decision` | What the chamber decided, e.g. `Bifall` / `Avslag` |
| `handled_in` | Which report handled it |
Values are stored exactly as published and never translated. `decisions:` in Values stay exactly as published. `decisions:` in `parliament.yaml` glosses them, so
`parliament.yaml` glosses them so prompts can explain them. prompts can explain them without the record being rewritten.
### `people` — the register of members ### `people` — the register of members
`intressent_id` is the source's person id and the join key used everywhere. `person_id` is the source's identifier and the join key used throughout.
`valkrets` is the electoral district; its meaning varies by country — a single-member `constituency` means different things by country — a single-member seat in the UK, a
constituency in the UK, a multi-member district in Sweden — and nothing in the code multi-member district in Sweden, a national list in the European Parliament — and
assumes either. nothing in the code assumes any of them.
### `chunks`, `motion_chunks` — passages with embeddings ### `speech_chunks`, `document_chunks` — passages with embeddings
Text split into passages, each with a `vector(N)` embedding under an HNSW cosine Text split into passages, each with a `vector(N)` embedding under an HNSW cosine
index. `N` must match `embeddings.dimension` in `parliament.yaml`; the application index. `N` must match `embeddings.dimension` in `parliament.yaml`; the application
checks this at startup because a mismatch otherwise fails deep inside pgvector. checks this at startup, because a mismatch otherwise fails deep inside pgvector with
a message that never mentions configuration.
### `debates` — aggregated debates ### `debates`
One row per debate, keyed `"{date}:{index}"`, holding a synthesised summary and the One row per debate, keyed `"{date}:{index}"`, with a synthesised summary and the list
list of speeches it covers. of speeches it covers.
## Application tables ## Application tables
Already country-neutral: `users` and `auth_tokens` (zero-knowledge auth), Country-neutral already: `users` and `auth_tokens` (zero-knowledge auth),
`chat_sessions` and `chat_snapshots` (encrypted chat storage and shared snapshots), `chat_sessions` and `chat_snapshots`, `research_boards`, `research_threads`, `jobs`,
`research_boards`, `research_threads`, `jobs`, `job_events` (deep research), `job_events`, `error_log` and `llm_events`, and the `eval_*` tables.
`error_log` and `llm_events` (observability), and the `eval_*` tables.
Two notes: Two notes:
`chat_snapshots.llm_messages` holds the full tool-call history so a forked snapshot `chat_snapshots.llm_messages` holds the full tool-call history so a forked snapshot
resumes with the model's context. Stat cards replay the SQL stored there, which is resumes with the model's context. Stat cards replay the SQL stored there, which is why
why renaming a column is not a clean break — old SQL has to keep working. the rename cannot be a clean break — SQL written against the old column names is still
sitting in saved conversations.
`session_type` is `'general'` or `'mp'`. The `'mp'` value means a chat conducted in `session_type` is `'general'` or `'mp'`. Read `'mp'` as "member". The values were left
the persona of a member; read it as "member", not as anything UK-specific. alone deliberately: rewriting live rows and a CHECK constraint would buy nothing, and
the display label comes from `parliament.yaml`.
## Full-text search ## Full-text search
Every search vector is built with the configuration named by `language.fts_config`, Search vectors are built with the configuration named by `language.fts_config`, read
read at runtime from `current_setting('app.fts_config')` inside the triggers. Set it at runtime inside the triggers via `current_setting('app.fts_config')`. Set it per
per database: database:
```sql ```sql
ALTER DATABASE plenum SET app.fts_config = 'swedish'; ALTER DATABASE plenum SET app.fts_config = 'swedish';
``` ```
This only takes effect for new sessions, so restart the API after changing it. It only takes effect for new sessions, so restart the API after changing it.
Note that `pg_dump` does **not** capture database-level settings. Restoring a dump
onto a fresh server loses this one, and the symptom is searches quietly returning
nothing rather than any error.
## The rename
The schema was originally Swedish, inherited from the Riksdag-only predecessor.
[`_postgres/rename_map.py`](../_postgres/rename_map.py) records every old-to-new pair
and generates both the migration and its rollback, so the two cannot drift apart.
The migration is guarded by an existence check: a no-op on a database created from
the current `schema.sql`, and the real thing on one created before the rename — so a
single file serves both a fresh install and an existing deployment.
`ALTER TABLE ... RENAME` is catalog-only in PostgreSQL, so no data moves and no index
is rebuilt, regardless of table size.

Write Preview
Loading…
Cancel
Save