Tag: Branch: Tree: main

main

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from 'main'

${ noResults }

Commit Graph

2 Commits (main)

Author	SHA1	Message	Date
Lasse Server	7ea92c9366	Stream the final chat answer instead of one blocking blob ... packages/llm/client.py:_read_stream yielded token-by-token deltas but was dead code — nothing called generate(stream=True). The chat tool loop always blocked on the final answer generation, so a ~40s answer arrived as a single SSE event and read as a hang, even though the "card stack" already streamed tool-selection progress. The final answer is structurally just whichever tool-loop iteration returns content with no tool_calls — there's no way to know in advance which iteration that'll be, and a turn can carry short narration before a tool call in the very same response. So streaming has to be speculative: - packages/llm/client.py: _read_stream becomes StreamAccumulator, which both streams live (thinking/content/tool_call) deltas *and* reconstructs a full ChatCompletionMessage (content, tool_calls, reasoning_content) once the stream ends. Tool-call argument fragments are accumulated by index, same pattern every OpenAI-compatible streaming client needs. Downstream tool execution code needed zero changes because of this — it just consumes the reconstructed message instead of the SDK's blocking return value. Also hardened <think>/</think> detection against a tag split across chunk boundaries (was a per-chunk substring check; now a carry-over buffer) — latent before since nothing exercised streaming for real. - backend/services/streaming_answer.py (new): AnswerStreamFilter holds buffered content back until it's implausible to still be one-line narration (this codebase already caps narration at 150 chars elsewhere), and separately holds back any trailing bracket run that could still grow into a [src:ID]/[[src:ID]] marker — citation validity isn't known until parse_and_renumber_citations runs on the complete text, so a raw marker, hallucinated or not, must never reach the client even for a frame. run_streaming_iteration drives one tool-loop iteration through this filter and emits "answer_delta" events; if a tool_call shows up after content already cleared the gate (narration grew unusually long before the model pivoted), it emits one "answer_delta_retract" so the UI doesn't strand a stale preview. - backend/services/chat.py, mp_chat.py: thread an explicit stream_answer flag through to _run_tool_loop's single generate() call site, rather than inferring streaming from event_callback being non-None — eval logging's ConversationRecorder.wrap() always returns a non-None callable, even for the plain blocking /api/chat endpoint, so that inference would have been wrong. Only stream_chat_response (the two SSE routes) sets it; /api/chat is untouched. - Frontend: both SSE consumers (ChatPanel.tsx's card stack, MpChatPanel.tsx's simpler turn list) grow "answer_delta"/"answer_delta_retract" handling. The live preview renders as plain, React-escaped text (no dangerouslySetInnerHTML, no markdown parsing mid-stream — that's a one-shot full-document transform, unsafe on partial input) and is fully replaced by the authoritative, citation-renumbered/person-linked/ language-polished HTML once the terminal "answer" event lands. Both event types were previously-unhandled and silently ignored, so the backend and frontend halves are independently safe to deploy. Added tests/test_llm_client_stream.py and tests/test_streaming_answer.py covering the reconstruction (content, tool-call fragment accumulation, split <think> tags, truncated-stream handling) and the filter (gate timing, citation-marker safety across chunk boundaries, the retract path, error propagation). 5 pre-existing failures in test_prompts_golden.py/ test_provenance.py are unrelated (reproduce identically on main).	5 days ago
Lasse Edfast	843db25211	Replace _llm and colorprinter with self-contained packages ... _llm called env_manager.set_env() at import time, which connected to a private ArangoDB to fetch secrets. That single line meant a fresh clone could not start, regardless of what else was configured. Both packages also lived in separate private repos and were gitignored here, so the code shipped without them. packages/llm/ is 780 lines against _llm's 1750. Dropped as unused by this project (measured, zero call sites): token counting and message trimming, image/vision handling, make_summary, the ollama-specific paths, the query/user_input/context argument style, and the self-mutating provider_quirks.json cache. Kept and reworked: - tools.py, the docstring -> JSON-schema tool registry, which has no equivalent in the cuj-fup client and which llm_tools.py depends on entirely. - The provider quirks that actually matter: vLLM-only extra_body fields stripped for hosted providers, enable_thinking disabled at template level when think is off, reasoning models (o1/o3/o4/gpt-5) switched to max_completion_tokens. Adopted from cuj-fup's client: LLMConfig as a dataclass instead of 20 constructor kwargs, the SDK's native max_retries instead of hand-rolled backoff, and error messages that name the likely cause. Fixes a latent bug: Optional[list[str]] parameters were advertised to the model as strings, because get_origin(Optional[X]) is Union, so neither the schema mapping nor the list coercion in execute_tool fired. `parties`, `people` and `focus_ids` were all affected. Also drops a dead SELECT-only guard in execute_tool that keyed on a parameter name (`sql_query`) that no tool has ever used. Real SQL hardening is tracked separately. Verified: `import backend.app` succeeds with all external network blocked and zero outbound connection attempts; all 12 tools register; live vLLM calls confirmed for plain generation, structured output via format=, tool execution, and the error-returns-a-string contract that call sites branch on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	1 week ago