Making Honcho Cheaper Without Making Memory Worse

I only wanted to make my self-hosted Honcho setup cheaper. The plan sounded simple: find a cheaper model, update a few environment variables, restart the containers, and lower my OpenRouter bill.

Honcho would be the memory provider for my Hermes Agent. It would store conversations, extract observations, and give Hermes persistent context across sessions.

The model swap turned into a production reliability investigation. I compared models, replayed failed payloads, debugged structured JSON output, checked hidden reasoning tokens, explained an OpenRouter usage spike, and changed Honcho’s Dream settings.

Prices and benchmark values here were checked on July 18, 2026. OpenRouter pricing and provider availability can change, so treat the numbers as a dated snapshot.

The short version

The final setup is role-specific:

Honcho role	Final choice
Deriver	Gemma 4 31B
Dialectic	DeepSeek V4 Flash
Summary	DeepSeek V4 Flash
Dream deduction	DeepSeek V4 Flash
Dream induction	DeepSeek V4 Flash
Embedding	OpenAI Text Embedding 3 Small

Gemma writes structured memory. DeepSeek searches, reasons over, and summarizes that memory.

DREAM__DOCUMENT_THRESHOLD=100
DREAM__MIN_HOURS_BETWEEN_DREAMS=24

I kept the same Dream history depth and tool budget. So this change reduces frequency, not depth.

The setup I started with

My Honcho deployment was self-hosted on v3.0.9. The stack was pretty standard:

Service	Job
FastAPI app	Main Honcho API
Deriver worker	Background memory extraction
PostgreSQL + pgvector	Message, observation, and vector storage
Redis	Queue/backend coordination

I was preparing Hermes Agent to use Honcho as its active memory provider. Raw messages would be stored asynchronously, embeddings would go into pgvector, and Dialectic would search and reason over that memory. The original LLM setup was simple, maybe too simple:

Gemma 4 31B handled every Honcho language-model role. OpenAI Text Embedding 3 Small handled embeddings.

Gemma handled Deriver, all five Dialectic levels, Summary, Dream deduction, and Dream induction. Embeddings used:

EMBEDDING_MODEL_CONFIG__MODEL=text-embedding-3-small
EMBEDDING_VECTOR_DIMENSIONS=1536

The goal was specific: reduce model cost without making Honcho’s memory less useful. That constraint shaped the whole investigation.

Honcho is not one model job

Honcho has several LLM workloads with different failure risks.

Component	What it does	What matters most
Embedding	Converts messages and observations into vectors	Stable embedding space, low cost, correct dimensions
Deriver	Extracts structured observations from conversations	Valid JSON, instruction following, consistency
Dialectic	Searches and reasons over memory	Tool use, reasoning, context handling
Summary	Compresses sessions	Coherence and factual retention
Dream deduction	Builds higher-level conclusions from observations	Reasoning, tool use, deduplication
Dream induction	Finds patterns across observations	Long-context synthesis and restraint

The Deriver carries the highest risk. It reads raw messages and writes structured long-term observations. Malformed or empty JSON can leave a batch with no derived observations, even when the raw conversation still exists.

Dialectic searches memory and reasons over the results. A model can handle Dialectic well and still be risky as a Deriver. The model reading memory does not need to be the model writing it.

Why I did not touch embeddings

Text Embedding 3 Small was already cheap. At the time I checked, it cost about $0.02 per million tokens and matched the 1,536 dimensions in my database schema.

Changing embedding models is not a normal model swap. Existing vectors from one embedding model are not semantically compatible with vectors from another model, even when both use the same dimensions. A proper migration would mean re-embedding stored messages and observations. The possible savings were small. The migration risk was real. So embeddings stayed exactly where they were.

Gemma vs DeepSeek looked obvious at first

On paper, DeepSeek V4 Flash looked like the easy win. The live OpenRouter catalog showed better price and better benchmark numbers than Gemma 4 31B:

Model	Input price per million	Output price per million	Intelligence	Coding	Agentic
Gemma 4 31B	$0.22	$0.55	29.4	43.4	14.4
DeepSeek V4 Flash	about $0.09 to $0.098	about $0.18 to $0.196	40.3	56.2	31.1

At those catalog prices, DeepSeek looked cheaper by about:

Direction	Expected saving
Input	55.5%
Output	64.4%

It also supported the capabilities Honcho needed:

• response_format
• structured_outputs
• tools
• tool_choice
• configurable reasoning

I checked other candidates too:

Model	Why I did not pick it
GPT-OSS 120B	Cheaper input, but much lower measured intelligence, coding, and agentic scores
Ling 2.6 Flash	Very cheap, but benchmark scores were too low for reasoning-heavy memory work
MiMo V2.5	More expensive than DeepSeek while scoring lower in intelligence and agentic performance
MiniMax M3	Better scores, but 3.33x higher input cost and 6.67x higher output cost
Nex-N2 Mini	Much cheaper, supports tools and structured outputs, but independent benchmark data was missing

At that moment, DeepSeek V4 Flash looked like the best cost-performance option. So I tried the obvious thing.

The first migration looked fine

The first migration replaced Gemma 4 31B with DeepSeek V4 Flash across all nine LLM roles. Embeddings stayed unchanged. Synthetic checks passed:

• Pydantic structured-output parsing worked.
• Tool calling worked for Dialectic memory search.
• OpenRouter accepted the required parameters.
• Dialectic retrieved current conversation context correctly.

That looked good. Too good, apparently. During deployment, I noticed something else: the API was healthy, but the Deriver container was not actually running. It was stuck in the Created state. That distinction matters more than it sounds.

What still works	What breaks quietly
API requests	New structured observations
Raw message storage	Summaries
Searching old memory	Dream processing
Existing Dialectic context	Vector reconciliation

Honcho can look partially healthy while the background memory pipeline is not forming new memory. I started the Deriver worker and added an unless-stopped restart policy. Once it came alive, it started processing the backlog. Then the real problem showed up.

Synthetic success was not enough

The first real Deriver batches with DeepSeek produced observations. Then later batches failed with this:

Repair failed: Expecting value: line 1 column 1
Deriver generated zero observations
Observation Count: 0

That is the kind of log that makes me pause. Not because it is loud. Because it is quiet. It did not look like a hard API failure. Honcho continued processing the queue. But the result was zero observations. For a memory system, that is dangerous. So I rolled back only the Deriver:

Role	Model
Deriver	Gemma 4 31B
Every other LLM role	DeepSeek V4 Flash

That was not the final conclusion yet. It was just the conservative move while I figured out what actually happened.

Replaying the real failure

I replayed the failed production payload using Honcho’s real PromptRepresentation schema. My earlier direct test had used an 8,192-token output limit and OpenAI SDK parsing. It succeeded twice on DeepSeek and once on Gemma. But production was not using that exact shape. Production had a much smaller output budget:

DERIVER.MAX_INPUT_TOKENS = 25,000
LLM.DEFAULT_MAX_TOKENS = 2,500
DERIVER.MODEL_CONFIG.max_output_tokens = unset

Because the Deriver-specific output limit was unset, Honcho fell back to 2,500 completion tokens.

Then I found the real difference. DeepSeek V4 Flash defaults to high reasoning when the request does not explicitly disable it. Gemma 4 31B supports optional reasoning, but defaults to reasoning disabled. So DeepSeek was spending the completion budget on hidden reasoning. And sometimes it had no room left for the actual JSON.

The failure pattern

Here is what the failed responses looked like:

Provider	Completion tokens	Reasoning tokens	Final content	Finish reason	Result
Baidu	2,500	2,500	0 chars	length	Invalid JSON
AtlasCloud	2,501	2,500	0 chars	length	Invalid JSON
DigitalOcean	Not exhausted	0	3,954 chars	stop	Valid JSON, 19 observations

So the model was not too dumb for the task. It understood the task. The problem was token allocation. On failing endpoints, DeepSeek used the entire completion budget for hidden reasoning and returned no final content. No content means no JSON. No JSON means no observations. And if this happens inside the Deriver, your memory pipeline quietly loses derived memory.

Why Honcho did not recover

Honcho’s OpenAI-compatible backend reads the final message.content and tries to repair it into the expected Pydantic model. But when the provider returned empty content, there was nothing useful to repair. The result became an empty representation.

1. The provider returned reasoning-only output.
2. Final content was empty.
3. JSON repair had no content to recover.
4. Honcho created an empty representation.
5. The queue item completed with zero observations.

This was not mainly a structured-output problem. It was the interaction between:

• a small output cap
• default reasoning behavior
• OpenRouter provider routing
• empty final content
• repair behavior that did not turn this into a hard failure

That combination is exactly why real production replay matters.

The fixes I tested

I tested several possible DeepSeek fixes against the exact failed payload.

Fix	Result
Add strict: true	Did not solve the token problem. If all tokens go to reasoning, there is still no JSON to validate.
Use require_parameters	Did not make automatic routing reliable. Providers could still handle reasoning controls differently.
Set reasoning_effort=none without provider pinning	Better, but AtlasCloud handled it inconsistently. Some requests still used reasoning and truncated.
Increase output limit to 8,192	More successful, but slower and more expensive. Still not reliable enough for Deriver.
Pin Baidu and disable reasoning	Best DeepSeek result, but required provider-specific routing support.

The strongest DeepSeek test used:

{
  "provider": {
    "only": ["baidu"],
    "allow_fallbacks": false
  },
  "reasoning_effort": "none",
  "max_tokens": 2500
}

That produced valid structured output in three exact-payload replays:

Test	Reasoning tokens	Observations	Result
1	0	47	Valid
2	0	18	Valid
3	0	18	Valid

The cost comparison was interesting:

Model and route	Cost
DeepSeek, Baidu, no reasoning	$0.000371
Gemma 4 31B, same payload	$0.000422

So yes, DeepSeek could work as a Deriver. But only with a provider pin and reasoning disabled. Operationally, that meant depending on one provider continuing to honor the reasoning control and remain available. Honcho’s backend also did not forward arbitrary OpenRouter provider-routing parameters from its model override config, so I would need a source patch before making that safe. For an 11.9% saving on that tested payload? Not worth it. The Deriver is the part that writes long-term memory. I do not want modest savings there if the failure mode is silent.

The final model split

The final model decision became conservative on purpose.

Honcho role	Final model	Why
Deriver	Gemma 4 31B	Reliable structured JSON, reasoning disabled by default, less provider-sensitive
Dialectic minimal	DeepSeek V4 Flash	Stronger reasoning and tool use at lower token prices
Dialectic low	DeepSeek V4 Flash	Same
Dialectic medium	DeepSeek V4 Flash	Same
Dialectic high	DeepSeek V4 Flash	Same
Dialectic max	DeepSeek V4 Flash	Same
Summary	DeepSeek V4 Flash	Good compression and lower cost
Dream deduction	DeepSeek V4 Flash	Stronger reasoning and tool use
Dream induction	DeepSeek V4 Flash	Better synthesis over longer context
Embedding	Text Embedding 3 Small	Cheap, stable, compatible with existing vectors

The final environment looked like this:

DERIVER_MODEL_CONFIG__MODEL=google/gemma-4-31b-it

DIALECTIC_LEVELS__minimal__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__low__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__medium__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__high__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__max__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash

SUMMARY_MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DREAM_DEDUCTION_MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DREAM_INDUCTION_MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash

EMBEDDING_MODEL_CONFIG__MODEL=text-embedding-3-small
EMBEDDING_VECTOR_DIMENSIONS=1536

Gemma writes memory. DeepSeek reads, reasons, summarizes, and runs Dream over that memory.

That split preserved most of the savings because eight of the nine LLM roles moved to DeepSeek. But the most reliability-sensitive role stayed on Gemma.

Then OpenRouter usage spiked

After the migration, Honcho’s OpenRouter usage looked much higher than usual. The Honcho-specific key showed $0.130060 daily usage and $0.178172 lifetime usage.

That one day was 73% of the key’s lifetime usage. My first suspicion was DeepSeek reasoning. Wrong again. The logs pointed somewhere else: Dream processing dominated the bill.

Four Dream jobs ate most of the bill

The expensive part was four Dream jobs:

Job	Input tokens	Output tokens
Deduction 1	350,533	10,093
Induction 1	135,038	5,344
Deduction 2	164,974	8,860
Induction 2	183,468	6,142
Total	834,013	30,439

At DeepSeek’s then-current price, the spending breakdown looked like this:

Category	Cost	Share
Dream jobs	$0.087699	67.4%
Diagnostic probes	$0.015127	11.6%
Normal Honcho activity	$0.027234	20.9%

The main bill was not the Deriver. It was Dream.

Why Dream got so expensive

The Dream runtime settings were:

Setting	Value
Enabled	true
Document threshold	50
Minimum hours between Dreams	8
History token limit	16,384
Maximum tool iterations	20
Enabled type	omni

At first glance, a 16,384-token history limit sounds bounded. But Dream is a tool-using agent loop. That means the cost is cumulative across iterations:

1. Iteration 1 sends the system prompt and history.
2. Iteration 2 sends those again, plus the first response and tool result.
3. Later iterations include the earlier turns and more tool results.
4. The final billed input is the sum of every iteration.

The active context can stay within its limit, while total billed input grows much larger. In this case, the four Dream specialists made 61 tool calls. So yes, the context limit was doing its job. But the bill still grew because the loop kept sending accumulated context. Fun. Painful, but fun.

Why it happened that day

The Deriver worker had been inactive. When I started it, Honcho had to catch up:

1. The worker processed queued conversations from multiple sessions and workspaces.
2. It created many explicit observations.
3. Observation collections crossed the 50-document Dream threshold.
4. Honcho scheduled deduction and induction work.
5. Dream agents ran multi-turn tool loops over accumulated memory.
6. Summary and vector reconciliation also caught up.

After the backlog finished, the queues were clear. So part of the spike was a one-time catch-up event. But it could happen again, because every collection was allowed to run Dream after 50 new observations and eight hours. That felt too eager for my usage.

Tuning Dream without making memory worse

Dream does not store raw conversations or extract immediate facts. The Deriver handles explicit memory:

The user prefers concise technical answers.

The user wants lower AI cost without reducing reliability.

Dream works at a higher level. It may combine multiple observations into something like:

The user accepts cost savings only when production reliability is preserved.

So reducing Dream frequency does not stop Honcho from storing raw messages, embeddings, explicit observations, or summaries. It only delays higher-level consolidation. There are several knobs, but they carry different risk:

Tuning change	Core memory impact	Higher-level memory impact	Risk
Increase document threshold	None	Conclusions arrive later	Low
Increase minimum interval	None	Conclusions may be less fresh	Very low
Reduce tool iterations	None	Shallower consolidation and deduplication	Medium
Reduce history token limit	None	More recency bias, weaker old-pattern detection	Medium
Disable Dream	Explicit memory remains	Deduction and induction stop	High

So I chose the boring option. Reduce frequency. Keep depth.

The final Dream settings

I increased the document threshold from 50 to 100. I also increased the minimum interval from 8 hours to 24 hours.

DREAM__DOCUMENT_THRESHOLD=100
DREAM__MIN_HOURS_BETWEEN_DREAMS=24

These stayed unchanged:

• Dream enabled: true
• History token limit: 16,384
• Maximum tool iterations: 20

That means when Dream runs, it still has the same history depth and tool budget. It just waits for more evidence and runs no more than once per collection per day. I could have gone harder:

• threshold 150 or 200
• fewer tool iterations
• smaller history window
• Dream disabled entirely

But for the first production pass, I did not want to mix too many changes. If memory quality got worse, I wanted to know why.

Deployment notes

Before editing the environment file, I backed it up. Docker Compose reads env_file values when containers are created, so a normal restart was not enough. The API and Deriver had to be recreated.

cd /path/to/honcho
docker compose config --quiet
docker compose up -d --force-recreate api deriver

PostgreSQL and Redis were left intact. I also fixed the file permissions because the environment file contains provider credentials:

chmod 600 .env .env.backup-robust-20260718T054815Z

Small thing. But worth doing.

Verification

I checked the deployment at several layers. Service health:

API health endpoint: 200 OK
API container: healthy
Deriver worker: running
PostgreSQL: healthy
Redis: healthy
Restart policy: unless-stopped

Runtime settings from Honcho’s live Pydantic configuration:

Deriver: google/gemma-4-31b-it
Summary: deepseek/deepseek-v4-flash
Dream deduction: deepseek/deepseek-v4-flash
Dream induction: deepseek/deepseek-v4-flash
Dream document threshold: 100
Dream minimum interval: 24
Dream history limit: 16,384
Dream maximum tool iterations: 20
Embedding: text-embedding-3-small
Embedding dimensions: 1,536

All five Dialectic levels also resolved to DeepSeek V4 Flash. Functional checks:

Minimal Dialectic request: HONCHO_DIALECTIC_OK
Post-deployment Deriver batch: nonzero observation count
Hermes memory provider: installed, active, available

Post-deployment log scan:

Structured-output repair failures: 0
Zero-observation warnings: 0
Tracebacks: 0
Fatal errors: 0

Queue state:

Dream: processed
Summary: processed
Representation: processed
Reconciler: processed
Webhook: processed

I did not force a Dream run just to test the new 24-hour cadence. That would have created another big token bill. And honestly, that would defeat the whole point of this exercise.

What changed in practice

The final system separates memory writing from memory reasoning.

Job	Model
Write structured memory	Gemma 4 31B
Reason over memory	DeepSeek V4 Flash
Summarize sessions	DeepSeek V4 Flash
Run higher-level Dream consolidation	DeepSeek V4 Flash, less often
Keep semantic search stable	Text Embedding 3 Small

This is more reliable than picking one model for everything. It also keeps most of the expected savings, because DeepSeek still handles eight of the nine LLM roles. Dream still runs with full depth. It just runs less aggressively.

The lessons

Benchmarks do not prove workflow compatibility

DeepSeek had better scores and lower token prices than Gemma. It also passed synthetic structured-output tests. Production still exposed a silent Deriver failure mode. A useful model test needs the real prompt, real schema, real output limit, and real provider-routing behavior.

Completion tokens include reasoning

This was the big gotcha. A low output cap may look like a cost control, but reasoning models share that budget between hidden reasoning and final content. If hidden reasoning uses the whole budget, your app receives no answer. For structured JSON jobs, that can become ugly fast.

OpenRouter provider routing matters

One model name on OpenRouter can route to several providers. Those providers may differ in quantization, latency, parameter handling, reasoning controls, and structured-output behavior. Testing one endpoint does not validate the whole route.

Empty structured output should be treated as failure

For a nonempty Deriver batch, empty final content should be observable and retryable. Honcho’s repair path can turn malformed output into an empty representation. A future hardening patch should distinguish a real no-observation result from a provider response that ended at the token limit with no final content.

Background workers need their own health checks

A healthy API did not mean the memory pipeline was healthy. The Deriver worker needed its own process check, startup verification, and nonzero observation test.

Agent loops can dominate token bills

Per-model pricing was not the main cost driver that day. Four Dream jobs consumed 834,013 input tokens and 30,439 output tokens.

The practical saving came from changing how often Dream could run, not from chasing a slightly cheaper Deriver.

Future experiments

There is still room to reduce cost, but I would test these carefully.

Gemma 4 26B A4B for Deriver

Gemma 4 26B A4B looked interesting at $0.10 per million input tokens and $0.30 per million output tokens.

Compared with Gemma 4 31B, that is about:

• 54.5% lower input cost
• 45.5% lower output cost

But the Deriver needs exact production-schema testing, repeated provider checks, and real queue verification before I would trust it.

Nex-N2 Mini for low-risk roles

Nex-N2 Mini was much cheaper than DeepSeek and supported tools plus structured outputs. The missing part was independent quality data. If I test it, I would start with summaries or an isolated Dialectic canary. Not production Deriver traffic.

OpenRouter provider routing support in Honcho

Honcho’s OpenAI backend could forward a request-level OpenRouter provider object from model overrides. That would allow a tested provider allowlist for specific roles. The patch should include regression tests for:

• provider parameter forwarding
• reasoning_effort=none
• token-limit finish handling
• empty final content
• malformed JSON
• model fallback behavior

Dream cost guardrails

Dream could also use better spending controls:

• per-workspace Dream budgets
• max cumulative input tokens per Dream run
• tool-iteration cost limits
• better prompt caching across tool turns
• alerts when Dream input exceeds a threshold
• separate usage labels for Deriver, Summary, Dialectic, and Dream

Final configuration

Here is the final config in one place:

# Structured memory extraction
DERIVER_MODEL_CONFIG__MODEL=google/gemma-4-31b-it

# Memory reasoning
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__low__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__medium__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__high__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DIALECTIC_LEVELS__max__MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash

# Session compression and higher-level memory
SUMMARY_MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DREAM_DEDUCTION_MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash
DREAM_INDUCTION_MODEL_CONFIG__MODEL=deepseek/deepseek-v4-flash

# Stable semantic index
EMBEDDING_MODEL_CONFIG__MODEL=text-embedding-3-small
EMBEDDING_VECTOR_DIMENSIONS=1536

# Less frequent Dream runs without reducing depth
DREAM__DOCUMENT_THRESHOLD=100
DREAM__MIN_HOURS_BETWEEN_DREAMS=24

The takeaway

The cheapest model was not automatically the cheapest system. DeepSeek V4 Flash was the better cost-performance choice for reasoning, summaries, and Dream. But Gemma 4 31B stayed as the Deriver because it returned structured memory reliably without hidden reasoning eating the final-output budget. The biggest saving did not come from replacing the Deriver. It came from noticing that Dream agent loops had consumed 834,013 input tokens in four jobs, then reducing how often those jobs could run. That feels like the real lesson here. If a system has multiple model roles, do not tune it like one model call. Find the part that writes state. Protect that first. Then tune the expensive loops around it.