plenum/docs/multi-provider.md

# Guide for adding multi-provider LLM support in `rixdagen`

This document is intended to be pasted into an assistant session in VS Code so it can implement the change.

## What is true in the repo right now

The current code is still wired around **one global endpoint**, even though the repo also contains a more flexible provider config template.

### Verified repo facts

1. `backend/services/chat.py` creates `smart_llm`, `fast_llm`, and `communicator_llm` using the same `base_url=os.getenv("LLM_DIRECT_URL")`.
2. `FAST_MODEL` and `SMART_MODEL` are selected from environment variables (`LLM_MODEL_FAST`, `LLM_MODEL_SMART`), but both still flow through that same single endpoint.
3. `providers.template.yaml` already describes a multi-provider design where each provider has:
   - an `id`
   - a `name`
   - a `base_url`
   - optional `api_keys`
   - a list of `models`
   - optional `additional_parameters`
   - per-model `abilities`
4. `providers.template.yaml` explicitly says you can add any **OpenAI API compatible** provider to the `providers` list, and also notes that the OpenAI API is not a standard, so compatibility is not guaranteed.
5. The template also already shows a provider with **multiple API keys** and a `litellm` example, which means the intended design already assumes routing different models to different upstreams.

## Main conclusion

The repo already points toward the right architecture, but the runtime path is not there yet.

Right now, adding more entries to `providers.yaml` alone is **not enough**. The backend still behaves like this:

- choose model alias from env
- send all LLM traffic to one global `LLM_DIRECT_URL`

To support `vLLM`, `berget.ai`, `openai.com`, and potentially other OpenAI-compatible providers cleanly, the app needs to move to this model:

- choose a logical model alias for each role (`smart`, `fast`, `embeddings`, maybe `communicator`)
- resolve that alias through `providers.yaml`
- derive:
  - provider
  - protocol
  - base URL
  - API key env var
  - upstream model ID
  - abilities / extra parameters
- build the client per resolved model

## What needs to be implemented

### 1. Make `providers.yaml` the source of truth for model routing

Do not keep endpoint selection centered on `LLM_DIRECT_URL`.

Instead:

- keep environment variables for **which logical model alias** to use
- move endpoint/model/provider metadata into `providers.yaml`

Recommended env vars:

- `LLM_MODEL_SMART`
- `LLM_MODEL_FAST`
- `LLM_MODEL_EMBEDDINGS`
- `LLM_MODEL_COMMUNICATOR` (optional, if you want it to diverge later)
- provider secrets such as `OPENAI_API_KEY`, `BERGET_API_KEY`, `LLM_VLLM_API_KEY`, etc.

### 2. Add a provider resolver layer

Create a small module, for example `backend/services/llm_provider_registry.py` or `_llm/provider_registry.py`, with responsibilities like:

- load `providers.yaml`
- validate it
- index models by alias or logical name
- resolve one requested model into a runtime configuration object

Recommended shape:

```python
@dataclass
class ResolvedModel:
    alias: str
    provider_id: str
    provider_name: str
    protocol: str
    base_url: str
    api_key: str | None
    api_key_env: str | None
    upstream_model_id: str
    abilities: dict[str, Any]
    additional_parameters: dict[str, Any]
```

Functions to add:

```python
def load_provider_config(path: str = "providers.yaml") -> dict: ...
def resolve_model(alias: str) -> ResolvedModel: ...
def get_api_key_from_env(env_name: str | None) -> str | None: ...
```

### 3. Refactor `_llm.LLM` to use resolved provider config

Today the calling code passes `model=...` and `base_url=...` directly.

Change that so `_llm.LLM` can accept either:

- a logical alias and resolve internally, or
- an already-resolved provider config object

Preferred direction:

```python
LLM(model_alias="smart")
```

or:

```python
resolved = resolve_model(os.getenv("LLM_MODEL_SMART", "smart"))
LLM(resolved_model=resolved)
```

The key point is that `_llm.LLM.generate(...)` should no longer rely on one app-wide endpoint.

### 4. Remove the single-endpoint assumption from `ChatService`

Refactor `backend/services/chat.py` so each role resolves independently.

Current behavior:

- `smart_llm` -> `LLM_DIRECT_URL`
- `fast_llm` -> `LLM_DIRECT_URL`
- `communicator_llm` -> `LLM_DIRECT_URL`

Target behavior:

- `smart_llm` -> resolved from `LLM_MODEL_SMART`
- `fast_llm` -> resolved from `LLM_MODEL_FAST`
- `communicator_llm` -> resolved from `LLM_MODEL_COMMUNICATOR` or fallback to `LLM_MODEL_SMART`

So the code should look more like:

```python
smart_alias = os.getenv("LLM_MODEL_SMART", "smart")
fast_alias = os.getenv("LLM_MODEL_FAST", smart_alias)
communicator_alias = os.getenv("LLM_MODEL_COMMUNICATOR", smart_alias)

self.smart_llm = LLM(model_alias=smart_alias, system_message=ORCHESTRATOR_SYSTEM, temperature=0.2)
self.fast_llm = LLM(model_alias=fast_alias, system_message=WORKER_SYSTEM, temperature=0.05)
self.communicator_llm = LLM(model_alias=communicator_alias, system_message=ORCHESTRATOR_SYSTEM, temperature=0.3)
```

### 5. Keep a fallback path for the current vLLM setup

Do not break the current local workflow.

Recommended fallback logic:

- if `providers.yaml` exists and the requested alias resolves, use it
- otherwise fall back to the current legacy env path:
  - `LLM_DIRECT_URL`
  - `LLM_API_KEY`
  - `LLM_MODEL`

That lets the migration happen incrementally.

### 6. Distinguish protocol from provider

This matters for future-proofing.

Do **not** assume every upstream will behave exactly like local vLLM.

Add a provider field like:

```yaml
protocol: openai
```

Even if you only implement `openai` first.

That keeps the design open for future support of providers that may require slightly different auth, payloads, streaming behavior, or tool-calling semantics.

## Recommended `providers.yaml` structure

Use the existing repo template as the starting point and extend it slightly.

Suggested structure:

```yaml
providers:
  - id: vllm
    name: Local vLLM
    protocol: openai
    base_url: http://localhost:8000/v1
    api_keys:
      vllm: LLM_VLLM_API_KEY
    models:
      - id: meta-llama/Meta-Llama-3.1-70B-Instruct
        alias: smart_local
        name: Smart Local
        provider: vllm
        context: 131072
        abilities:
          temperature:
            supported: true
          system_message:
            supported: true
          tools:
            supported: true

  - id: openai
    name: OpenAI
    protocol: openai
    base_url: https://api.openai.com/v1
    api_keys:
      openai: OPENAI_API_KEY
    models:
      - id: gpt-4.1-mini
        alias: fast_openai
        name: GPT-4.1 mini
        provider: openai
        context: 1000000
        abilities:
          temperature:
            supported: true
          system_message:
            supported: true
          tools:
            supported: true

  - id: berget
    name: Berget
    protocol: openai
    base_url: REPLACE_WITH_VERIFIED_BERGET_BASE_URL
    api_keys:
      berget: BERGET_API_KEY
    models:
      - id: REPLACE_WITH_VERIFIED_MODEL_ID
        alias: smart_berget
        name: Berget Smart
        provider: berget
        context: REPLACE_WITH_VERIFIED_CONTEXT
        abilities:
          temperature:
            supported: true
          system_message:
            supported: true
          tools:
            supported: REPLACE_WITH_VERIFIED_VALUE
```

## Important implementation details

### Alias resolution

The app should refer to models by local aliases, not raw provider model IDs.

That means env vars should point to aliases such as:

```bash
LLM_MODEL_SMART=smart_berget
LLM_MODEL_FAST=fast_openai
LLM_MODEL_COMMUNICATOR=smart_local
```

### API key lookup

The provider file should contain the env var **name**, not the secret itself.

Example:

```yaml
api_keys:
  openai: OPENAI_API_KEY
```

Runtime should do:

```python
env_name = provider.api_keys[model.provider]
api_key = os.getenv(env_name)
```

### Additional provider parameters

The template already supports `additional_parameters` at provider level.

Preserve and pass those through when calling the upstream API. This is especially useful because many OpenAI-compatible providers expose non-standard extensions.

### Abilities gating

Use the `abilities` block to avoid sending unsupported settings.

Examples:

- do not send tools if `tools.supported` is false
- do not send temperature if unsupported
- do not rely on reasoning-specific settings unless the model explicitly supports them

This is important because the template itself warns that OpenAI compatibility is imperfect.

### Streaming and tool calling

You should assume that compatibility varies most in these areas:

- streaming chunk format
- tool/function calling schema
- system message handling
- vision payload format
- reasoning fields

So build adapters conservatively and fail loudly with clear messages.

## Suggested migration plan

### Phase 1: provider registry

- add loader + schema validation for `providers.yaml`
- resolve aliases to provider configs
- keep legacy env fallback

### Phase 2: wire `ChatService`

- replace `LLM_DIRECT_URL` usage in `backend/services/chat.py`
- resolve each model independently

### Phase 3: update `_llm`

- centralize client creation
- add support for provider-level `additional_parameters`
- add abilities filtering

### Phase 4: test with three providers

Test combinations like:

1. all-local
   - smart -> vLLM
   - fast -> vLLM
   - communicator -> vLLM

2. mixed
   - smart -> Berget
   - fast -> OpenAI
   - communicator -> vLLM

3. cloud-only
   - smart -> OpenAI
   - fast -> OpenAI
   - communicator -> OpenAI

## Acceptance criteria

The change is done when all of the following are true:

1. I can switch providers **without code changes**, only by editing `providers.yaml` and env vars.
2. `smart`, `fast`, and `communicator` can point to different providers.
3. The current local vLLM setup still works.
4. Unsupported capabilities are filtered based on model abilities.
5. Failures clearly say which alias/provider/model failed and why.
6. Secrets are only read from environment variables, never committed into YAML.

## External documentation notes that must be verified before implementation

I cannot verify the live OpenAI and Berget documentation from this environment because web access is disabled in this session.

That means the VS Code assistant should **verify these exact facts from the docs before finalizing the patch**:

### OpenAI

Verify from current OpenAI docs:

- the recommended Python SDK import and client initialization pattern
- whether the intended integration path should use `responses` or `chat.completions`
- exact base URL shape for the public API
- auth header expectations
- tool-calling support for the models you intend to use
- streaming response shape if this app streams tokens

### Berget

Verify from `https://docs.berget.ai/quickstart`:

- the exact OpenAI-compatible `base_url`
- the exact auth mechanism / env var examples
- the exact Python example for OpenAI SDK usage
- which model IDs are documented for chat usage
- whether tools/function calling are supported
- whether reasoning / vision / embeddings are separately documented

### One OpenAI-compatible provider in general

Use one documented compatibility layer as a reference point when implementing adapters. The goal is to confirm what the **minimum common denominator** really is across providers.

At minimum, verify:

- expected `/v1` pathing
- supported request schema for chat completions or responses
- tool/function call representation
- streaming format
- how unsupported fields are handled

## Instructions for the coding assistant

Implement this as a minimal, low-risk refactor.

### Required changes

1. Add a provider registry module.
2. Add YAML loading and validation.
3. Refactor `backend/services/chat.py` so each LLM instance resolves independently.
4. Refactor `_llm.LLM` to build clients from resolved provider config.
5. Preserve backwards compatibility with the current env-only setup.
6. Add clear error messages for missing aliases, missing API keys, and unsupported features.

### Nice-to-have changes

1. Add a startup validation command that prints configured aliases and providers.
2. Add unit tests for alias resolution.
3. Add one integration test for local vLLM and one mocked OpenAI-compatible provider.
4. Add comments explaining why compatibility must be treated as provider-specific, not assumed.

## Suggested commit breakdown

1. `add provider registry and providers.yaml loader`
2. `refactor chat service to resolve llm aliases per role`
3. `refactor _llm client creation to use provider config`
4. `add compatibility fallback for legacy env config`
5. `add tests for provider resolution and errors`

## Final note

The most important design change is this:

**Stop routing all models through one global `LLM_DIRECT_URL`.**

Everything else is secondary. Once alias -> provider -> endpoint resolution exists, adding `berget.ai`, `openai.com`, or other OpenAI-compatible providers becomes a configuration problem instead of a code fork.