You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ollama backend never sets reasoning_effort, so local reasoning models (nemotron, deepseek-r1, qwq) can blow --api-timeout on real extraction chunks #2932
Benchmarking Claude/GLM-5.2/nemotron-3-super:120b-a12b-q4_K_M (via --backend ollama, served locally through Ollama) on identical corpora, Nemotron's extraction was consistently much sparser than the other two backends, and on a larger multi-file corpus one file (README.md) was dropped entirely after repeated timeout+bisect retries (5 files → 2 → 1, each hitting the client's 600s --api-timeout, with nothing left to bisect at 1 file).
No JSON parsing errors were involved — this is not the #2882 narration/parsing issue, which is already fixed and working correctly.
Root cause (confirmed empirically, not guessed)
Direct POST /v1/chat/completions against nemotron-3-super:120b-a12b-q4_K_M on the same Ollama host, same prompt (real source file + the extraction instructions), varying only reasoning_effort:
Call
content length
reasoning length
completion_tokens
default (reasoning_effort omitted)
3,299 chars
17,442 chars
5,234
reasoning_effort: "high"
2,092 chars
9,154 chars
2,807
reasoning_effort: "low"
3,463 chars
17,543 chars
5,244
Findings:
choices[0].message has a separate reasoning field (Ollama's OpenAI-compat naming) alongside content. content itself is never empty or sparse in any of the three calls — each held valid, parseable JSON with genuine extraction — so this isn't the LLM responses not in JSON format: instructions instead of data #2882 recovery path, and it isn't a case of the answer hiding in the wrong field either.
Ollama's OpenAI-compat endpoint does respect reasoning_effort for this model. "high" roughly halves both the reasoning length and total completion tokens compared to omitting it or passing "low".
graphify/llm.py's BACKENDS["ollama"] config never sets a reasoning_effort key — only BACKENDS["gemini"] does ("reasoning_effort": "low"). So for every --backend ollama call, cfg.get("reasoning_effort") is None and the field is never sent, regardless of which model is actually being served.
Because the reasoning narration shares the same request's wall-clock and the client's --api-timeout (default 600s), a model that reasons at length before answering can burn most or all of that budget on narration alone. On a real extraction chunk (not the isolated single-file diagnostic above), that pushed a 5-file chunk into the adaptive bisect-and-retry path, and it still failed at a single file with no smaller unit to fall back to — the whole file was dropped from that run.
Why this isn't ollama-backend-wide
Ollama serves many non-reasoning models (the backend's own default is qwen2.5-coder:7b), so a blanket default would be wrong for most --backend ollama users. The bug is specifically that reasoning-tuned local models get no reasoning_effort treatment at all, whereas the same family is already recognized elsewhere in the codebase — _THINK_BLOCK_RE's own comment names nemotron, deepseek-r1, qwq as the models known to narrate a <think> block.
Repro
Serve a reasoning-tuned model locally, e.g. ollama pull nemotron-3-super:120b-a12b-q4_K_M.
graphify extract <a corpus with 3+ semantic files> --backend ollama --model nemotron-3-super:120b-a12b-q4_K_M --max-concurrency 1
On a large enough chunk, watch it bisect and potentially drop a file, with GRAPH_REPORT.md showing far fewer nodes/edges than the same corpus run through another backend (Gemini, GLM-5.2, Claude) gets.
Suggested fix
Opening a PR shortly: resolve reasoning_effort per-call for the ollama backend (env var override via GRAPHIFY_OLLAMA_REASONING_EFFORT, falling back to "high" for models matching the same reasoning-family list _THINK_BLOCK_RE already recognizes, and None/unchanged otherwise) rather than leaving it unset for every model served through Ollama.
What happened
Benchmarking Claude/GLM-5.2/
nemotron-3-super:120b-a12b-q4_K_M(via--backend ollama, served locally through Ollama) on identical corpora, Nemotron's extraction was consistently much sparser than the other two backends, and on a larger multi-file corpus one file (README.md) was dropped entirely after repeated timeout+bisect retries (5 files → 2 → 1, each hitting the client's 600s--api-timeout, with nothing left to bisect at 1 file).No JSON parsing errors were involved — this is not the #2882 narration/parsing issue, which is already fixed and working correctly.
Root cause (confirmed empirically, not guessed)
Direct
POST /v1/chat/completionsagainstnemotron-3-super:120b-a12b-q4_K_Mon the same Ollama host, same prompt (real source file + the extraction instructions), varying onlyreasoning_effort:contentlengthreasoninglengthcompletion_tokensreasoning_effortomitted)reasoning_effort: "high"reasoning_effort: "low"Findings:
choices[0].messagehas a separatereasoningfield (Ollama's OpenAI-compat naming) alongsidecontent.contentitself is never empty or sparse in any of the three calls — each held valid, parseable JSON with genuine extraction — so this isn't the LLM responses not in JSON format: instructions instead of data #2882 recovery path, and it isn't a case of the answer hiding in the wrong field either.reasoning_effortfor this model."high"roughly halves both the reasoning length and total completion tokens compared to omitting it or passing"low".graphify/llm.py'sBACKENDS["ollama"]config never sets areasoning_effortkey — onlyBACKENDS["gemini"]does ("reasoning_effort": "low"). So for every--backend ollamacall,cfg.get("reasoning_effort")isNoneand the field is never sent, regardless of which model is actually being served.Because the reasoning narration shares the same request's wall-clock and the client's
--api-timeout(default 600s), a model that reasons at length before answering can burn most or all of that budget on narration alone. On a real extraction chunk (not the isolated single-file diagnostic above), that pushed a 5-file chunk into the adaptive bisect-and-retry path, and it still failed at a single file with no smaller unit to fall back to — the whole file was dropped from that run.Why this isn't ollama-backend-wide
Ollama serves many non-reasoning models (the backend's own default is
qwen2.5-coder:7b), so a blanket default would be wrong for most--backend ollamausers. The bug is specifically that reasoning-tuned local models get noreasoning_efforttreatment at all, whereas the same family is already recognized elsewhere in the codebase —_THINK_BLOCK_RE's own comment namesnemotron, deepseek-r1, qwqas the models known to narrate a<think>block.Repro
ollama pull nemotron-3-super:120b-a12b-q4_K_M.graphify extract <a corpus with 3+ semantic files> --backend ollama --model nemotron-3-super:120b-a12b-q4_K_M --max-concurrency 1GRAPH_REPORT.mdshowing far fewer nodes/edges than the same corpus run through another backend (Gemini, GLM-5.2, Claude) gets.Suggested fix
Opening a PR shortly: resolve
reasoning_effortper-call for theollamabackend (env var override viaGRAPHIFY_OLLAMA_REASONING_EFFORT, falling back to"high"for models matching the same reasoning-family list_THINK_BLOCK_REalready recognizes, andNone/unchanged otherwise) rather than leaving it unset for every model served through Ollama.