Sohail Mohammad

Inference Flexibility Starts in the API

2026-06-08T00:00:00-05:00

I work at Together AI. Technical details have been generalized from production experience; no proprietary information from any organization is disclosed.

The scheduler cannot act on "queued tokens" alone. It needs labels that describe who is waiting, what can move, and when the work loses value.

Mario Souto's The Clay Jar and the Data Center gets the framing right: AI energy is not only a procurement problem. The missing layer is not only where the electrons come from. It is which inference work should consume them now, which work can wait, and which work should not run at all.

I agree with the energy claim. Inference has more temporal flexibility than the current data center conversation admits. A chat response, a nightly eval run, a document extraction job, a coding agent, and a voice assistant do not have the same latency constraint. Treating them as one flat demand curve is expensive.

But temporal flexibility is not schedulable by itself. The scheduler needs to know which work can wait, which work can move, and which work loses value when delayed. That information does not live in GPU utilization. It has to enter through the API.

Energy orchestration cannot infer product intent from token counts. A scheduler can see queue depth, prompt length, prefix cache hits, model ID, current capacity, and regional load. It still does not know whether a request is a human waiting on a support answer, a batch extraction job due tomorrow morning, or an agent step inside a task that has already burned through its budget.

That distinction is not cosmetic metadata. It changes the route.

The scheduler cannot optimize what the API hides¶

Many inference APIs expose the wrong contract for this problem. They accept a request and return a response. Some expose streaming. Some expose a batch endpoint. Some have priority tiers. But the default unit is still the single request.

That is too small.

The unit that matters economically is the accepted work unit: a resolved ticket, an extracted document, a completed agent task, a grounded answer, a voice turn that stays inside the audio latency budget. The request is an implementation detail.

Two requests can look identical at the serving layer: same model, same prompt length, same expected output length, same region. One is a human waiting on a support escalation. The other is a document extraction job due before tomorrow's reconciliation run. The first should get warm capacity, prefill priority, and low TTFT. The second should be queued, batched, and moved if the data constraints allow it.

Token count alone cannot separate them.

This is where Mario's energy argument connects directly to inference economics. If the orchestration layer is choosing when and where to run a workload, it needs to know more than tokens per second. It needs to know the denominator.

For a support chatbot, the denominator may be cost per accepted resolution. For document extraction, cost per valid extracted record. For a coding agent, cost per accepted task. For evals, cost per decision that changes something downstream.

Without that denominator, the scheduler can only optimize cheapness in the narrowest sense. It can delay work, route to a cheaper region, or fill idle capacity. It cannot decide whether the delay was worth it.

Flexibility is a product property¶

Batch APIs are a revealed price for latency tolerance. OpenAI's Batch API documentation describes "50% lower costs" with a 24 hour completion window, and Anthropic's Message Batches documentation describes batches that cost "50% less" than standard API calls and process within 24 hours.¹

But batch only captures one point on the curve: complete within hours. Many production inference workloads sit between interactive and overnight.

A customer support draft might need to finish immediately if an agent is waiting, but could be precomputed before the workday starts. A coding agent might need interactive latency while the user is watching, then become batch eligible once the user closes the laptop. A document extraction job might have no per-request SLO, but still need the full file set complete before a morning reconciliation run.

The wall-clock number matters less than who is blocked by the work.

If a human is waiting, latency is product quality. If another machine is waiting, latency is workflow throughput. If nobody is waiting yet, latency is inventory management. Those are different economic objects.

Flexibility is also stateful. The same task can move between classes. A coding agent is interactive while the user is watching and background work after the user leaves. A support draft is urgent when an agent is blocked and cheap inventory when it is precomputed before a shift.

The API should expose that difference.

Workload class belongs in the control plane¶

I would make workload class a first-class part of the inference contract.

Not a dashboard tag. Not a billing label that gets attached after the fact. A field the scheduler can act on.

The numbers below are illustrative. The shape matters more than the specific values.

workload_class: human_waiting_streaming
latency_budget_ms:
  ttft: 800
  p95_total: 5000
unit_of_value: accepted_resolution
max_cost_per_unit_usd: 0.04
flexibility:
  can_delay_until: null
  can_migrate_regions: false
  can_use_batch: false
quality_floor:
  evaluator: support_policy_grounded
  action_on_fail: repair_or_escalate

A batch extraction job would look different:

workload_class: batch_extraction
latency_budget_ms: null
unit_of_value: valid_extracted_document
max_cost_per_unit_usd: 0.01
flexibility:
  can_delay_until: "2026-06-08T12:00:00Z"
  can_migrate_regions: true
  can_use_batch: true
quality_floor:
  validator: schema_and_field_match

A coding agent needs another shape because the task is a loop:

workload_class: agentic_task
unit_of_value: accepted_task
max_cost_per_unit_usd: 2.50
flexibility:
  interactive_until_idle_seconds: 300
  can_delay_background_steps: true
  can_migrate_regions: true
controls:
  max_llm_calls: 80
  escalate_on_budget_exhaustion: true

The schema can change. What matters is that the scheduler receives a product-level contract. If the control plane only receives undifferentiated requests, it will treat all demand as equally urgent until a human hard-codes exceptions.

Some fields should not be trusted as user input. Latency class can be declared by the product, but measured by traces. Acceptance rate has to come from the quality gate. Cost per accepted unit has to reconcile against billing. Region mobility has to pass data residency, egress, and cache-locality checks.

The scheduler should treat missing metadata as debt, not as permission to use premium capacity.

That is how you end up with expensive GPUs doing cheap work at the wrong time.

The clay jar needs labels¶

Mario's clay jar metaphor works because storage decouples production from consumption. Grain can be harvested today and eaten later. Energy can be generated now and used later. Inference can be queued now and computed when capacity is cheaper.

But storage only works if the operator knows what is stored.

Seed grain, wet grain, tax grain, and dinner grain are not interchangeable. The jar creates optionality. The label determines whether using that optionality is useful or destructive.

Inference has the same shape. The queue is the jar. Workload metadata is the label.

A scheduler that sees "a large queue of tokens" knows almost nothing. A scheduler that sees "batch documents due before morning reconciliation, background agent steps with no human waiting, real-time voice turns, and support escalations inside a declared p95 SLO" can make economic decisions.

That second system can coordinate with energy. The first one can only throttle.

Energy routing changes serving architecture¶

Once workload class becomes part of the contract, routing stops being only "which GPU has room?"

Human-waiting traffic needs cache-local, warm capacity. The scheduler should protect TTFT and inter-token latency before it chases lower power cost. A cross-region route that saves cents but adds visible delay can be a product regression. This traffic wants stable fallback, prefill priority, and enough reserved capacity that burst handling does not turn into queueing.

Batch extraction wants the opposite shape. It can accept longer queueing, wider batching, lower prefill priority, and cheaper regions if data residency and egress cost do not erase the savings. Retry tolerance matters here too. A failed extraction batch can run again. A failed voice turn cannot be retried without the user hearing the failure.

Agentic workloads need phase changes. While the user is watching, the loop is interactive. After idle timeout, the remaining tool calls, retries, evals, and summarization steps become background work with a task budget and a deadline. Moving background steps only works if tool state, file access, and cache locality move with them, or if the savings exceed the cost of rebuilding context.

Evals should be treated like internal manufacturing work. If an eval job is consuming peak interactive capacity, it should be tied to a release gate, routing decision, model selection, or safety check. Otherwise it belongs in the off-peak queue.

This is not just "serve inference." It is closer to operating a plant where every job has a routing sheet.

The hard part is trust¶

The operational objection is that product teams will lie to the scheduler. Everything becomes high priority. Every team claims its workload is latency sensitive. Every batch job gets marked urgent because somebody does not want to debug delayed completion.

That will happen unless the contract is tied to measurement.

If a workload declares a p95 SLO, measure it. If it declares a cost budget, enforce it. If it declares an accepted-result denominator, audit the acceptance metric. If it declares that it cannot be delayed, make the owner pay the premium for inflexibility.

A practical design is quota-backed classes: each workload gets an owner, default class, measured SLO, budget, and allowed burst premium. If a team marks background evals as interactive, the traces should show no human wait event and the finance report should charge the premium back to that owner.

The discipline has to look more like cloud quotas than best-effort tagging. Teams do not get to label traffic however they want forever. The labels need owners, budgets, and post-hoc accountability.

This is why I do not think the orchestration layer can be only an energy product. It has to be an inference product, a platform product, and a finance product at the same time. The hard integration is not the battery API. It is getting the organization to state what each workload is worth.

The missing middle term¶

Mario's closing formulation is: electrons in, orchestration, tokens out.

I would add one more constraint inside orchestration: workload intent.

Electrons in, workload intent, orchestration, accepted work out.

Tokens are intermediate inventory. Some tokens resolve a customer issue. Some tokens grade an eval. Some tokens are burned by an agent loop that has already exhausted its budget. Some tokens are cheap because the work can wait for off-peak capacity.

The energy layer matters, but cheaper electrons are not enough. Cost per useful output improves when demand is expressed in a form the scheduler can act on: deadline, mobility, latency budget, quality gate, and accepted unit.

The system will know which work can wait, which work can move, which work deserves premium capacity, and which work should not run at all.

That is the piece I would put next to the clay jar. The jar stores surplus. The label tells you whether opening it is the right move.

OpenAI, Batch API; Anthropic, Message Batches. ↩

Goodput or It Didn't Happen

2026-05-13T00:00:00-05:00

This article is excerpted from Chapter 9 of Production Inference Economics: A Field Guide. The full chapter develops the productive-capacity and cache-local routing framework in more depth.

I work at Together AI. Technical details have been generalized from production experience; no proprietary information from any organization is disclosed.

78% GPU utilization is consistent with 30% of requests exceeding p99 latency (service-level objective), with rising escalation rates, and with the bill growing. Same problem, three departments noticing it.

Utilization measures GPU busyness. It doesn't measure productive output: accepted work at target latency. The GPU can be busy recomputing evicted KV cache (the per-request memory that stores attention state). It can be running speculative decode drafts that get rejected. It can be processing retries of requests that already failed once. It can be serving outputs that will fail the quality gate and never reach a customer. All of that counts as utilization. None of it counts as productive work.

I watched this pattern play out over six weeks. The team had provisioned a dedicated cluster for a RAG workload, document analysis at roughly 2,000 requests per day. Utilization dashboards were green. Token throughput was healthy. The GPU-hours-per-dollar number looked competitive. But the LCPR (loaded cost per accepted result) was climbing. Not because inference got more expensive per token, but because an increasing fraction of the GPU's work was producing nothing the customer would accept.

The root causes were mundane. A KV cache eviction cascade under long-context traffic meant the system was recomputing prefill for requests whose KV state had been preempted. A retry storm from a downstream timeout meant the same prompt was being processed two or three times before one attempt succeeded. A quantization change intended to improve throughput had introduced quality degradation in the long tail (rare entities, numeric precision, multi-turn coherence after turn five) that the standard A/B test hadn't caught.

Each of these individually was a normal operational event. Together, they meant the GPU was 78% utilized and 30% unproductive. The metric that would have caught this is not utilization. It is goodput.

The gap between allocated and productive GPU capacity is not an edge case. Modal's State of AI Infrastructure at Scale, 2024 reports the majority of organizations under 70% GPU allocation utilization at peak; common figures 10-20%. That is allocation utilization: how much of provisioned capacity is even assigned to work. Productive utilization (how much of that assigned work produces accepted output) is lower still. The gap has two layers: hardware sitting idle, and hardware that is busy but unproductive. Both are costs. Only productive work is revenue.

The distinction that changes every decision¶

Throughput is work attempted. Total tokens generated per second, total requests processed per hour, total GPU cycles consumed. Throughput tells you how busy the system is. It does not tell you how much of that busyness produced value.

Goodput is work completed: accepted results per second, meeting latency, quality, and reliability constraints. Goodput is the bridge between serving physics and the economics from The Denominator Problem and Workload Costs. It is the number that belongs in capacity plans, migration decisions, benchmark comparisons, and the LCPR denominator.

The formal definition is straightforward:

TPOT_i = (E2E_i - TTFT_i) / (output_tokens_i - 1)

goodput = count(requests where
    TTFT ≤ S_ttft
    AND TPOT ≤ S_tpot
    AND quality_pass == true
    AND success == true
    AND retry_count ≤ R_budget
) / D

Where S_ttft and S_tpot are your service-level thresholds and D is the measurement window in seconds.

The formula is straightforward. Wiring up the data is not. You need latency, quality labels, and cost attribution at request granularity, and most teams have one or two of those wired but not all three. Without all three, goodput degrades to "latency-constrained throughput": still useful, but incomplete. The full definition (latency, quality, reliability, and cost) is what connects serving infrastructure to the LCPR formula.

The cost per accepted result is then:

cost_per_accepted = sum(all_costs) / count(accepted_requests)

The numerator includes all costs, including failed attempts, retries, quality failures, and wasted speculative tokens. The denominator includes only accepted requests. Goodput is the serving-side measurement that feeds LCPR's denominator.

Where productive capacity leaks¶

Five mechanisms turn GPU utilization into wasted capacity. Each is measurable; each has a lever.

KV cache preemption. Serving engines preempt long-running sequences to make room for incoming ones. Recomputing the evicted KV when the sequence resumes consumes GPU cycles that produce no new output. In a production deployment mixing short-context support chat and long-context document analysis on the same GPU pool, KV cache eviction cascades caused running requests to drop and preemptions to spike simultaneously. The monitoring signal was cache usage near 100% with preemption counts climbing. The fix was workload-aware routing: short context to one pool, long context to another. Same hardware, different routing, different economics.

Retries and repair loops. A request times out. The client retries. The retry consumes a fresh prefill, fresh KV allocation, and fresh decode cycles, all for work that the first attempt already partially completed. On quality-sensitive workloads, a failed quality gate triggers a repair: the system regenerates with corrected context, burning a second full inference pass. The Denominator Problem showed the trace where eight tickets generated twelve inference calls. Those four extra calls are GPU work that doesn't count as goodput but absolutely counts as utilization.

Speculative decode waste. A draft model proposes tokens; the target model verifies. Accepted drafts are faster; rejected drafts are wasted compute. GPU utilization goes up either way. Goodput only goes up when acceptance rate is high enough to offset the drafting cost.

Quality failures at the model level burn GPU budget invisibly. An output that passes the latency SLO but fails the quality gate consumed every token of its inference budget for nothing. The GPU was busy. The request was "successful" in the HTTP sense. The output was useless. Without quality labels in the goodput measurement, this failure is invisible. The system looks like it's producing work at target latency, and the quality problem surfaces later as customer complaints, support tickets, or manual rework.

Queue buildup and SLO violations. At high utilization, requests queue. Queue time adds to TTFT (time to first token). Even if decode is fast once the request starts, the user waits through the queue. A system running at 90% utilization with Poisson arrivals will have materially worse p99 queue time than one running at 70%. The utilization dashboard says "efficient." The user says "slow." The TTFT SLO violation means the request doesn't count as goodput even if the output was perfect.

When optimizations backfire¶

Two common optimizations (speculative decoding and quantization) deserve specific attention because they increase utilization while potentially decreasing productive capacity.

Speculative decoding at production batch sizes¶

In one high-concurrency conversational workload I worked on, speculative decoding with a fine-tuned draft model produced ~0.92x throughput. Slower than the baseline, after roughly two weeks of engineering work to tune the draft. The acceptance rate climbed from ~47% to ~58% after fine-tuning, but at the production batch sizes we hit (around 11-14 concurrent sequences), the draft model's added compute exceeded the savings from accepted drafts.

The mechanism is straightforward. Speculative decoding wins at low concurrency with high acceptance: the draft is nearly free if the GPU has spare cycles. At batch sizes where the GPU's compute budget is already committed to the decode batch, adding a draft means either slowing all sequences or raising memory pressure.

A search infrastructure provider reports positive production results with multi-token prediction draft layers for the same technique, likely because the search workload's constrained output distribution yields higher draft acceptance rates than open-ended conversational generation. The technique is not universally good or bad. The decision variable is whether your workload's output distribution produces an acceptance rate high enough to offset the draft model's compute at your production batch size.

Speculative decoding is workload-dependent. The right question isn't whether it "improves latency" in general. It's whether your acceptance rate and batch size make it net positive for goodput.

Quantization and the quality tail¶

After quantizing a production model, quality failures appeared in the long tail: rare entities, numeric precision errors, and multi-turn conversation degradation after five or six turns. Standard A/B tests did not catch these because A/B tests measure average quality on a general distribution. The tail failures affected 2-3% of requests, below the noise floor of most A/B sample sizes, but those requests were disproportionately high-value: complex customer queries, multi-step reasoning, and queries involving precise numerical data.

Quantization's throughput gains are real. FP8 quantization can improve throughput by 1.6x and cut memory consumption by 3-4x. But the quality risk is in the tail, not the average. Continuous log analysis (reviewing real production outputs, not just aggregate quality scores) was the only reliable detection method.

The productive capacity impact: throughput went up by 60%. Quality failures in the tail increased by 1.5 percentage points. Retry rate for affected queries increased. Human escalation cost for affected queries increased. Net LCPR change: roughly break-even. The quantization "saved" GPU cost and then spent it on retries and human review.

The right approach is not to avoid quantization. It is to measure its effect on goodput, not just throughput. Run quantization experiments with tail-quality evaluation (rare entities, long conversations, numerical tasks), not just aggregate pass rates. If goodput under SLO improves, the optimization worked. If throughput improved but goodput didn't, the optimization is a wash dressed up as a win.

Cache-local routing: the lever hiding in your load balancer¶

Eight replicas behind a round-robin load balancer. Cache hit rate drops from 85% to 12%. TTFT increases 4x. The support chatbot's cache configuration has not changed. The system prompt is identical across requests. The tool definitions haven't moved. Everything about the caching setup is correct, except the load balancer.

Round-robin sends each request to a random replica. The system prompt is cached on replica 3, but the next turn goes to replica 7, where the cache is cold. Replica 7 runs a full prefill. The next request goes to replica 1. Another full prefill. The cache is populated on every replica, but each replica only serves one-eighth of the traffic, so reuse within the TTL window is too low to sustain hits.

Traditional load balancing treats replicas as interchangeable. Inference replicas are not interchangeable. Each replica holds different KV state.

Here is what this looks like on real numbers for a support chatbot with a 6,000-token system prompt plus tool prefix, running on 8 replicas:

Routing policy	Cache hit rate	Mean TTFT	Effective input cost
Round-robin	~12% (1/8 chance)	1,200ms	Baseline
Prefix-aware	~85% (routed to prefix holder)	300ms	~60% savings
Sticky session	~90% (conversation state preserved)	250ms	~65% savings
The difference between round-robin and prefix-aware routing is 4x TTFT and 60% cost reduction. Not from a better model. Not from cheaper hardware. From routing the request to the right replica.

Why routing policy is an economic lever¶

Inference serving systems maintain per-replica state that matters:

KV cache. Each replica holds KV blocks for recently served prefixes. A cache hit requires routing the request to the replica that holds the matching prefix. Miss the routing, miss the cache, pay for full prefill.

Prefix cache. Some systems hash prefixes and store KV blocks by hash. The same prefix hashed on different replicas creates duplicate storage. Routing to the right replica avoids recomputation.

Session state. Multi-turn conversations benefit from sticky routing to the replica that holds the conversation's growing context. Context accumulates across turns. By turn 8, the growing conversation plus system prompt can be 4-6x the initial prompt. If each turn goes to a different replica, every turn pays for full re-prefill of the conversation history.

Cache-local routing creates two tradeoffs:

Locality versus load balance. Routing to the cache-hot replica improves TTFT and cost but can overload that replica if many requests share the same prefix. The routing layer needs a saturation threshold: route to the cache-hot replica if it has capacity, fall back to another if saturated.

Cache diversity versus cache efficiency. If all requests go to the same replica, one replica does all the work and the rest sit idle. If requests are spread evenly, cache hit rates drop. The optimal distribution depends on prefix diversity, traffic shape, and the cost of cache misses versus load imbalance.

Routing policies for inference¶

Policy	Mechanism	When it works	When it fails
Round-robin	Even distribution	Stateless or batch workloads	Destroys cache locality
Least-request	Route to least-loaded	When all replicas are cold	Ignores cache state
Prefix-aware	Route by prefix hash	High-reuse prefixes (support, tools)	Low-reuse or unique-prefix workloads
KV-cache-aware	Route to replica with matching KV	Multi-turn, high cache benefit	Requires cache state visibility
Sticky session	Route same session to same replica	Conversational workloads	Creates hotspots if sessions are uneven
Fallback	Route to any when target is saturated	Burst handling	Cold start penalty on fallback

The engineering community has converged on this hierarchy. SGLang's default router uses cache-aware routing. KServe and llm-d document KV-cache-aware scheduling with per-pod cache event routing. LMCache documents cross-instance KV reuse with multi-tier storage. The infrastructure exists. The question is whether your deployment uses it.

If your workload has reusable prefixes and you use multi-replica serving, routing policy is an economic lever. Implement prefix-aware or KV-cache-aware routing before spending on more hardware. Measure cache hit rate by replica. If hit rates are low but prefix diversity is low (most requests share a few system prompts), routing is the problem, not cache configuration.

The operations angle¶

Cache locality is fragile under operations. Rolling deployments cycle replicas, clearing their KV state. Autoscaling adds cold replicas that have no cache state: the router spreads traffic to them, and their cache miss rate is 100% until they warm up. A cache salt rotation (sometimes triggered by security policy or tenant isolation requirements) invalidates all cached prefixes simultaneously.

Each of these events temporarily converts the system from warm-cache to cache-disrupted. If the team measures average cache hit rate across a day, the 15-minute disruption window looks small. But if that disruption window coincides with peak traffic (and scaling events often do, because autoscaling triggers under load) the cost impact is disproportionate. Peak traffic times cold cache means the most expensive requests hit the most expensive state.

The mitigation is not to avoid operations. It is to measure TTFT and cache hit rate around operational events (deployments, scale-outs, salt rotations) and include disruption-window cost in the goodput calculation. A system that runs at $0.012 per accepted result in steady state but spikes to $0.035 during the 20 minutes after each deployment has a blended cost that depends on deployment frequency. Deploy twice a day and the disruption cost is material. Deploy weekly and it's noise. The goodput framework makes this tradeoff explicit.

The goodput frontier test¶

Most vendor-blog benchmarks report a single number (tokens/sec or average latency) at one operating point. A single point hides the surface underneath.

The goodput frontier test measures goodput across a sweep of request rates or concurrency levels, under explicit SLO constraints, with quality labels. It produces a frontier (the curve of cost per accepted result as load increases) not a single point.

How to run it¶

Step 1: Define SLOs before measuring.

Gate	Threshold
TTFT	≤ your product's first-token SLO
TPOT (time per output token)	≤ your product's per-token SLO
E2E (end-to-end latency)	≤ your product's end-to-end SLO
Quality	Passes your eval gate
Retries	≤ retry budget per attempt

If you don't have defined SLOs, you can't measure goodput. Define them first. Even rough SLOs (TTFT ≤ 1,500ms, TPOT ≤ 60ms/tok) are better than none.

Step 2: Sweep request rate with your actual workload shape.

Use your production prompt/output length distribution, not synthetic uniform inputs. Use Poisson arrivals for interactive workloads, not closed-loop. Run at least five rate points: comfortable, moderate, busy, stressed, and overloaded. At each point, record:

p50/p95/p99 TTFT, TPOT, and E2E latency
Request throughput and token throughput
Request goodput (accepted results per second)
Error rate, retry rate, and quality pass rate
KV cache hit rate and utilization
Cost per accepted result

Step 3: State cache warmth explicitly.

A benchmark that does not state cache warmth is not reusable evidence for production economics. Five states matter:

State	Meaning
Cold	No model weights resident, no KV/prefix cache
Warm-model cold-cache	Model resident, prefix/KV cache empty
Warm-cache	Repeated prefixes populated on target replicas
Hot-cache steady state	Hit rate at intended production distribution
Cache-disrupted	Replicas rolled, cache evicted, salt changed

Run at least warm-cache and cache-disrupted. The gap between them tells you how much of your system's performance depends on cache locality surviving operations.

Worked example¶

Two routing configurations for the same RAG workload, 100 requests per point:

Metric	Route A (high throughput)	Route B (SLO-tuned)
Raw throughput (tok/s)	1,200	800
p99 TTFT	1,400ms (fails 800ms SLO)	650ms (passes)
p99 TPOT	55ms/tok	42ms/tok
Quality pass rate	72%	91%
Requests meeting ALL gates	58	85
Goodput (accepted req/s)	5.8	8.5
Total cost (100 req)	$1.10	$1.45
Cost per accepted result	$0.019	$0.017

Route A wins on raw throughput. Route B wins on goodput. The token-price comparison picks A; the goodput frontier test picks B.

Route A delivers 58 accepted results per 100 attempts; 42% of GPU work goes to requests that will fail. Route B delivers 85; 15% waste. Total spend favors A; per-unit-of-value cost favors B. Same denominator problem from The Denominator Problem, in serving-physics form: the metric is accepted results, not tokens generated.

For output-heavy workloads (code generation, long-form content, agentic tool use) output-token goodput can be more informative than request goodput:

output_token_goodput =
  sum(output_tokens for passing requests) / D

A system that produces 800 accepted output tokens per second and 200 rejected tokens per second has output-token goodput of 800 tok/s, not 1,000. The rejected tokens consumed GPU decode cycles, KV bandwidth, and output-token billing. They produced nothing the customer accepted. Output-token goodput makes this visible in a way that raw token throughput does not.

Benchmark claims to avoid¶

Every claim on this list has appeared in a vendor blog, a comparison table, or a customer deck. Every one of them is incomplete or misleading without additional context:

"X tokens/sec, therefore X is cheaper." Tokens/sec alone is not an economics claim. It requires SLO constraints, length distribution, cache warmth, quality gates, and cost attribution to mean anything.
"p50 latency improved, so users will feel it." Interactive systems fail at p95/p99, TTFT spikes, cold starts, or retries, not at p50.
"This engine is faster than that engine." This claim requires model revision, quantization level, hardware, runtime flags, workload shape, request-rate sweep, and cache policy to be meaningful. Without them, it is marketing.
"Prompt caching saves N percent." Cacheable-prefix share, hit rate, TTL fit, tenant isolation, and eviction behavior all determine whether the savings number is real or aspirational.
"Batch API is 50% cheaper, so move workloads to batch." Completion-window tolerance, failed-row handling, expiration cost, and delivery delay alone determine whether the migration math works.
"Speculative decoding improves latency." Acceptance rate, batch size, quality gate, and net goodput measurement can flip this claim backwards.
"GPU utilization is 78%, so the cluster is efficient." Utilization without goodput describes busyness, not productivity.

What to measure¶

The workload class determines the goodput definition. The serving infrastructure determines the measurement points. Here is the measurement table that ties everything together, from the workload economics of What Your Workload Actually Costs to the goodput definition in this article:

Workload class	Goodput unit	Primary serving metrics	Primary economic metric	Dominant lever
Conversational	Accepted resolutions per second	Cache hit rate by turn, TTFT, session length	LCPR per accepted resolution	Quality gate, cache stability
Agentic	Accepted tasks per second	Fanout multiplier, repair rate, compaction events	LCPR per accepted task	Fanout control, termination policy
RAG	Grounded answers per second	Retrieval precision, context tokens, grounding pass rate	LCPR per grounded answer	Retrieval quality
Extraction	Validated records per second	Schema pass rate, batch-eligible share	LCPR per validated record	Batch eligibility, deterministic checks
Voice	Completed interactions per second	LLM budget utilization, fallback rate, TTFT	LCPR per completed interaction	Latency budget, fallback quality
Batch/offline	Accepted items per hour	Completion time, retry rate, expiration rate	LCPR per accepted batch item	Batch API usage, volume pricing

Each row is a different goodput configuration. Same formula, different SLO gates, different quality definitions, different denominators. The formula is durable. The workload identity determines which inputs matter.

Three decisions goodput changes¶

Capacity planning. GPU-hour math without goodput produces fictional capacity numbers. A team plans: "We need 4 H100s to serve 10,000 requests per day based on throughput benchmarks." The throughput benchmark ran at batch 256 with warm cache, no quality gate, and no retry overhead. Production runs at batch 8-16 with cold cache on 40% of requests, a 12% quality failure rate, and a 6% retry rate. The effective capacity at production conditions is 40-60% of the benchmark number. The team needs 6-8 H100s, not 4. Or they need to fix their cache hit rate and quality gate before buying hardware.

The right capacity planning formula is not "throughput times hours." It is: "goodput under SLO at your production traffic shape, times the hours you need to serve, plus headroom for bursts and p99." Goodput already accounts for retries, quality failures, and cache misses. Throughput does not.

Migration decisions. A team evaluates switching serving engines or providers. Engine A produces 1,200 raw tok/s. Engine B produces 900 raw tok/s. Engine A looks faster. But Engine A has a 1,400ms p99 TTFT that violates the 800ms SLO on 30% of requests, while Engine B's p99 TTFT is 650ms. Engine A has worse prefix cache stability under rolling restarts. Engine A's quality pass rate is 72% versus Engine B's 91%.

At the goodput level, Engine B produces more accepted results per second at a lower cost per accepted result, despite being "slower" on the throughput benchmark. The migration that looks wrong on throughput looks right on goodput. In the cases I've seen, those reversals held.

Model selection. The model evaluation says Model X generates 40% more output tokens per second than Model Y. Model X costs 30% less per token. The spreadsheet says: switch to Model X, save $12,000 per month. This is the same spreadsheet from The Denominator Problem. The denominator problem returns.

Model X's output is longer. It generates more tokens per response, inflating throughput numbers but also inflating cost per response. Model X's quality pass rate is 78% versus Model Y's 91%. Model X's repair rate is 3x higher. At the LCPR level, Model X costs $0.052 per accepted result versus Model Y's $0.038. The cheaper model is more expensive. Goodput catches this. Throughput does not.

What goodput doesn't capture¶

Quality labels are missing. Without quality labels, goodput degrades to latency-constrained throughput. That is still better than raw throughput, but it misses the largest cost lever on quality-sensitive workloads. If you don't have eval labels, goodput will tell you about latency and reliability but not about the acceptance rate that drives LCPR. Start measuring quality on the highest-value workload first.

Streaming workloads care about jitter. Interactive chat users feel inter-token latency (ITL) variation. A burst of fast tokens followed by a stall is worse than a steady moderate pace. Mean TPOT doesn't capture this. For streaming UX, measure p95 ITL and jitter (max ITL minus min ITL within a response), not just mean TPOT.

Batch workloads use different SLOs. Batch goodput should use completion window and accepted-row rate, not interactive TTFT/TPOT. A batch job that completes in 4 hours with 99.2% row acceptance is productive. Applying interactive latency SLOs to batch work produces a nonsensical goodput number.

Benchmarks don't match production. A benchmark with synthetic uniform prompts, closed-loop traffic, and warm cache will produce different goodput numbers than production with variable-length prompts, Poisson arrivals, cache disruption from replica scaling, and real quality failures. Treat benchmark goodput as directional, not predictive. The frontier test is most useful when run on production traffic replay or realistic synthetic distributions.

Scaling replicas dilutes cache locality. Adding replicas to handle traffic spikes reduces per-replica cache hit rate. A system that runs at 85% cache hit rate on 4 replicas may drop to 60% on 8 replicas if the router doesn't adapt. Counterintuitively, scaling up can increase per-request cost even as it increases total capacity.

Low volume makes goodput noisy. At 50 requests per day, daily goodput fluctuations are large. One bad hour (a provider incident, a cache flush, a burst of difficult queries) can swing daily goodput by 30%. Goodput is most stable and useful at hundreds of requests per day or more.

Multi-turn sessions complicate the unit. Is the goodput unit a turn or a session? A conversation with seven good turns and one bad turn is one failed session or seven-eighths good turns, depending on how you define the unit. For conversational workloads, I define the unit as the session: the conversation is accepted or it isn't. But this means session-level goodput is lower than turn-level goodput, and the distinction matters for capacity planning.

Three layers, one observation¶

Cache-local routing, the right denominator, and SLO-gated benchmarking are not separate optimizations. They are the same observation applied at three layers: the request, the trace, and the cluster. A serving system that produces accepted work is the thing the product needs. Tokens, GPU cycles, and throughput are inputs to that, not metrics for it.

Token price is the first number every team looks at. It should be the last number that informs the decision.

The full chapter develops this framework further, with additional worked examples on cache-local routing under operational disruption and the productive-capacity formalism. See Production Inference Economics: A Field Guide, Chapter 9.

Sohail Mohammad — May 2026

Numbers are anonymized and should not be attributed to any specific employer, customer, or deployment.

The Idiot Index for AI Deployment

2026-05-11T12:00:00-05:00

Speculative decoding doesn't work at batch 12.

Not "doesn't work well." 0.92x. Slower. i implemented it at Wendy's. We were running open-source models behind 100K+ daily drive-thru interactions, 400-600ms inference budget, and under production concurrency, we were paying 8% more compute with it enabled than without. The papers promise 2-3x decode speedup. The papers are measuring single-request latency on clean benchmarks. At batch 12-16, different sequences accept different numbers of draft tokens. You get ragged tensors. The alignment overhead eats the theoretical gain. Add the KV cache management cost of a draft model that's never seen "Baconator" in its training data, and you're underwater.

i spent two weeks on this before i killed it. Tried three draft model configurations. Fine-tuned a 1B speculator on 10K drive-thru transcripts - acceptance rate went from 48% to 58%, still below breakeven. Tried n-gram speculation. 42%. The math is unforgiving: at α=0.55, γ=5, you get 1.94 expected tokens per step for 1.15x the cost. Net negative.

The reason this matters isn't speculative decoding. It's the gap it represents.

The idiot index for AI deployment¶

There's a concept i keep coming back to from Elon Musk's early SpaceX days. He couldn't afford rockets, so he calculated the raw material cost for carbon fiber, metal, fuel, etc and compared it to what the aerospace industry charged for a finished product. The ratio was absurd. 50x. He called it the "idiot index": the cost of the finished product divided by the cost of its component materials. If the ratio is high, somewhere in the chain between raw material and finished product, a massive amount of unnecessary complexity, bureaucracy, or tradition has been layered on top.

AI deployment has its own idiot index. The ratio between what a model is supposed to do and what it actually does in production is enormous and mostly invisible. A model scores well on benchmarks, passes evals, demo looks great. Then it hits real traffic at scale and latency blows up, costs are 10x the budget, or quality degrades in ways nobody can explain because nobody profiled the workload distribution they're actually serving.

Nicholas Carlini has this line i've been thinking about. He says you can't predict what will or won't work ahead of time, so you have to prototype fast and let reality tell you what's true. That's science versus engineering. You can do everything right on paper and the idea still doesn't survive contact with production. The physics of a turbopump looks one way in simulation and behaves differently when it's bolted to a combustion chamber at full throttle.

Inference is the same. Batch size, sequence length distribution, quantization, KV cache pressure, hardware generation, concurrent request patterns. All of these interact in ways that single-request benchmarks don't capture. Every enterprise i've been embedded in has hit this wall. The MIT stat that 95% of AI implementations fail isn't surprising if you've watched it happen from inside the system, repeatedly, across multiple companies. The demo works. Production doesn't. The gap between intended behavior and actual behavior is the idiot index. And it's high.

The industry's response has been to create a new kind of role. FDE postings are up 800% in two years. Palantir's had forward-deployed engineers for over a decade. Now Anthropic, Databricks, OpenAI (this morning) and a growing number of AI infrastructure companies are building these teams. The signal is clear: the last mile to production isn't solved by better documentation or a solutions engineer on a Zoom call. It requires someone embedded in the customer's environment who understands both the kernel and the customer.

What i kept doing under different titles¶

i should be specific about what this role actually is, because most people confuse it with adjacent work.

The closest analog is a solutions architect, and SAs do serious technical work. But the motion is different. An SA is typically pre-sales: design the system, prove the architecture fits, help close the deal. An FDE picks up after the deal closes, when the real workload shows up and the architecture meets production. A solutions engineer configures the product to fit the use case. Professional services is staffing. A developer advocate teaches in public. All of these are real, technical work.

The difference with an FDE is where you sit and how deep you go. A forward-deployed engineer requires 300-400 level depth in the underlying technology. Not "i can call the API" but "i understand why PagedAttention allocates KV cache in non-contiguous blocks and when that memory fragmentation becomes a performance problem under long-context workloads." You're embedded with a customer for weeks or months. You're writing production code in their environment, brought in for the problems that are too nuanced or too deep for a standard engagement. You're discovering things about the platform that the product team hasn't seen because they don't have your specific workload conditions.

i didn't set out to build this skill set. i fell into a pattern across four different environments over five years. Consulting, embedded banking infrastructure, production AI at scale, fractional CTO work. The pattern was always the same cycle: figure out the real problem (not the stated problem), design a solution that accounts for their actual constraints (not the idealized ones), build it in their stack, then document it so you can leave and it survives without you.

Discovery → Architecture → Build → Document. i didn't have a name for this until i saw the FDE role description. It was just... what the work required.

i should caveat: a lot of this is me hyper-indexing on what the FDE role looks like from the outside. i haven't done it yet at an AI-native infrastructure company. The engagements i've run were under different titles, at different kinds of organizations. i'm genuinely curious what it looks like when the product you're deploying is the inference stack itself — when the thing you're embedded to optimize is the same thing you understand at the kernel level. That's the part i'm excited to explore.

What i got wrong (and what that taught me)¶

i've written detailed case studies about most of this: the Ray production disasters, the RAG infrastructure scaling, the vLLM deployment work, so i won't retell the full stories here. But the pattern across all of them is the same, and it's the pattern that matters.

At JPMorgan, i spent three days debugging 10x slower distributed training. GPU profiling looked clean. NCCL configuration looked correct. Every tool pointed at the GPU/model layer. The actual root cause was Deep Packet Inspection on the ML subnet aka JPMorgan's security infrastructure adding 5-15ms to every inter-node packet. For a single API call, invisible. For thousands of allreduce operations per second, catastrophic. The problem was in the infrastructure between GPUs, not the GPUs themselves. Took 6 weeks of security reviews to get an exemption on an internal ML subnet. No NCCL tuning in the world would have fixed a network security problem.

At Wendy's, i built a 47-panel Grafana dashboard for a three-engine inference router. The single most useful metric turned out to be kv_cache_usage_percent. Not GPU utilization, that sat at 95% perpetually and told us nothing. KV cache at 80% was the real autoscaling trigger. That single discovery prevented three potential outages. Nobody told me this. No documentation covers it. You learn it at 2am wondering why latency spiked when traffic didn't change.

At JPMorgan again (directly this time), a RAG application serving 15,000 users hit 8-second response times at scale. 60% of the latency was in framework orchestration abstractions, not inference, not retrieval. Surgical replacement of the hot path dropped it to 3 seconds. i didn't rewrite the system. i measured, found the bottleneck, replaced only what mattered.

Three different environments. Three different "wrong layers." Every time, the real problem was invisible from outside the production environment. That's the point. These discoveries only happen when you're embedded, with their specific workload, their specific security constraints, their specific scale. You can't do this work from the outside.

What actually changed¶

My father passed away in early 2026.

i've written about this before, so i won't repeat the full weight of it here. But grief rearranges your priorities in a way that doesn't un-rearrange. The question of where to spend your finite time stops being abstract when you watch someone run out of it. i stopped optimizing for compensation and titles and started optimizing for a single question: where can i make the most impact with whatever time i have?

i wrote an essay a few months ago about sampling at the frontier. Exploring multiple interests in parallel, noticing which ones pull your attention back involuntarily. The hype projects feel like positioning. The genuine interest projects feel like you can't stop thinking about them. i've been running that experiment for two years across consulting, production inference, open-source kernel work, mechanistic interpretability research, agentic systems. And the signal was always the same thing: inference. The gap between what a model can do and what it actually does under production constraints. That gap is what i can't stop thinking about.

Amazon taught me real things. Operating at 1.5M user scale. On-call rotations that burn operational rigor into your nervous system. How large organizations make decisions. What i want next doesn't live there. The kernel-to-customer arc. Where you go from understanding why FlashAttention's tiling strategy matters for your memory budget, all the way through to a customer's production system working correctly because of that understanding. That work requires being embedded, being close to the workload, being in the room when things break.

i'm also making a bet about where the center of gravity in AI engineering is moving. Training frontier models is increasingly concentrated: hundreds of millions of dollars, thousands of GPUs. But inference is where every company actually lives. Every application, every API call, every user interaction hits an inference stack. The hard problems like batching under heterogeneous workloads, speculative decoding that actually works at scale (see: paragraph one), quantization that doesn't degrade quality on domain-specific content, multi-LoRA serving, KV cache optimization. These are unsolved at each companies specific production scale. The people working on them are building the infrastructure the entire industry runs on.

The intersection¶

The people who will define the next phase of AI infrastructure aren't pure researchers (who don't see production constraints) and aren't pure application engineers (who don't understand the kernel). The work lives at the intersection. Deep enough to write a Triton kernel, practiced enough to know when that kernel will fail under concurrent load, embedded enough in customer environments to understand what the workload actually looks like.

i've spent five years building this intersection by accident. Consulting. Embedded engineering. Open-source kernel contributions. Production inference at scale. The FDE role is the first title i've seen that describes the job that fits what i've been doing.

i'm starting at Together AI in June.

i don't know exactly what i'll build there yet. But i know what the work is: close the gap between intended and actual behavior, one embedded engagement at a time, one production discovery at a time, one system that finally works under real load at a time. Make the idiot index smaller.

i want to be in a room where the things i don't yet know how to do are the things being done at the highest level. Where my ideas get sharpened by people who have context i haven't earned yet. Where the standard is set by the work itself.

That's why i left.

What Your Workload Actually Costs

2026-05-06T00:00:00-05:00

I work at Together AI. Technical details have been generalized from production experience; no proprietary information from any organization is disclosed.

Production Inference Economics — Part 3 of 3

Four inference workloads run through one API provider account with one model: support chat, document extraction, nightly eval runs, and an experimental coding agent. Same billing surface, same rate limits, same cost model. The monthly bill arrives. Finance divides total spend by total requests and reports an average cost per request.

That average describes none of the four workloads accurately.

Support chat: 1.1 calls per ticket. Coding agent: 23 calls per task, σ=40. Extraction: batch mode at half price. Eval: burns tokens grading outputs, produces zero customer-facing value, buried inside the blended average.

I watched a team run this exact configuration for four months. They optimized the blended average: negotiated a volume discount, switched to a cheaper model, reduced prompt length. The blended average fell 18%. Everyone was happy.

Then someone broke out per-accepted-work-unit cost by workload. The support chat was 40% cheaper than the blended number suggested. The coding agent was 3x more expensive. The extraction pipeline was about right. And the eval job (consuming 12% of total spend) had never been asked to justify its cost.

The cheap workloads were subsidizing the expensive one. The blended optimization averaged away the only signal that mattered: per-workload health.

The fix is workload classification. A human-waiting chatbot, an overnight batch job, a multi-step coding agent, and a RAG pipeline don't share a cost structure. Different failure modes, different quality gates, different denominators. Treating them as one workload produces a number that optimizes nothing and misleads everyone.

The distinction that changes treatment¶

A workload class is not a label. It is a category that changes at least one of these: routing, fallback path, latency SLO (service-level objective), quality gate, billing surface, caching strategy, monitoring threshold, or operational owner. If a distinction doesn't change any of those, it is not a workload class. It is a tag for a dashboard.

The test is practical: would changing the route for one workload break the other? If a support chatbot and a batch extraction pipeline share the same route, and you switch the route to a batch-optimized endpoint for cost savings, the chatbot's latency SLO breaks. They are different workload classes.

The minimum taxonomy I use:

Class	Defining constraint	What changes
Human-waiting synchronous	p95 latency SLO under 5s	Route, model size, batch config, fallback path
Human-waiting streaming	TTFT (time to first token) SLO	Prefill priority, chunked transfer, backpressure
Machine-waiting synchronous	Latency SLO 1-30s	Can tolerate larger batch, longer queue, cheaper route
Batch/offline	No per-request latency SLO	Eligible for batch API, off-peak scheduling, retry tolerance
Agentic/multi-turn	Task-lifecycle SLO	Fanout accounting, compaction, cache strategy, repair cost
Real-time/voice	Sub-second hard ceiling	Dedicated capacity, streaming, thermal stability

The binding constraint differs by who is waiting. When a human waits (a chatbot reply, a voice response, a code completion) token generation speed dominates and the serving physics constrain which models and routes are feasible. When a machine orchestrates on a queue or an overnight schedule, memory capacity and cost per accepted result dominate, and latency is secondary.

This taxonomy is not universal. Some organizations need more classes, some fewer. But every team I have worked with that started from "we have one workload, inference" and then separated their traffic into at least three classes discovered that their cost model changed materially. The optimization strategy changed. The routing changed. The monitoring changed.

Conversational workloads: the session is the unit¶

The support team from the denominator article generates 1,000 tickets per day. Each ticket is a conversation. Conversations have economic properties that single-turn requests do not.

Context accumulates within a session. Each turn adds the prior conversation to the input. By turn 8, the input can be 4-6x the initial prompt. Without caching, every turn re-pays for the full history. With caching, the economics depend on whether the cached prefix is stable, and on multi-turn conversations, the prefix grows and mutates with every tool call and response.

Tool calls add hidden cost. A support agent that looks up an account, checks order status, and issues a refund makes three tool calls. Each tool response enters the context for the next turn. Tool output tokens are free to generate (the tool produces them, not the model) but expensive to consume (they become input tokens on the next turn).

A four-turn conversation with two tool calls per turn can accumulate 10,000 tool-output tokens in context. At cached input rates, this is manageable. At uncached input rates, the tool output context can exceed the cost of the actual generation.

Quality failures cost more than inference. Human escalation routinely runs 50-75% of loaded cost on quality-sensitive interactive workloads. The 7x-of-inference-bill ratio in Article 1 was not an outlier. A model with a 3% higher first-attempt quality pass rate can reduce escalation cost by more than a 30% reduction in token price saves.

The naive optimization is to use a smaller, cheaper model. A smaller model may produce more tool calls, longer reasoning chains, more repairs, and more human escalations. The LCPR per accepted resolution can increase even when the per-token price decreases.

A canary I ran on one workload illustrates: a regulated-industry customer-support deployment running Sonnet-class as the baseline, with Haiku-class on 12% canary traffic over 5 weeks. Per-token price gap: ~8:1. LCPR-per-accepted-resolution gap landed at 1.18:1. Haiku-class was 18% more expensive per accepted result, not 37% and not 8x. The components:

Metric	Sonnet-class	Haiku-class
Avg turns/session	4.3	5.1
Avg input tokens/turn	3,200	3,400
Avg output tokens/turn	380	420
Cache hit rate	0.71	0.74
Raw resolution rate	89%	83%
Repair rate	4.2%	6.8%
Human escalation rate	2.7%	4.1%
LCPR per accepted resolution	$0.038	$0.045

The Haiku-class cache hit rate was higher than Sonnet-class. Shorter outputs left more headroom in the KV pool for cross-session prefix reuse, which we didn't predict before the canary. Repair-success-rate was also slightly higher on Haiku-class (94% vs 89%). The escalation rate was the dominant lever: 1.4 percentage points of additional escalation at ~$2.20/case ate most of the token-price savings.

The smaller model is still cheaper per token. It's more expensive per accepted resolution by a margin that took a 5-week canary to surface. We kept Sonnet on customer-facing tickets and moved Haiku-class to internal agent-handler workflows where the escalation cost was zero.

Sessions, not requests. A cost model that meters at request granularity on a multi-turn workload measures the wrong unit, period.

Cache economics dominate sessions longer than 3-4 turns. The cache break-even formula from the derivation pack applies: if the system prompt and tool definitions are stable across turns, the cached prefix saves re-processing on every turn. On a conversation with a 1,800-token system prompt and 2,000 tokens of tool definitions, the savings compound because the cached portion grows as the conversation grows. At Anthropic's 5-minute TTL with 0.10x read pricing, the break-even is 2 calls within 5 minutes.

Most support conversations clear that threshold easily. But if the prompt layout puts dynamic content (retrieval results, user state) before the stable prefix, the cache breaks on every turn and the savings evaporate.

Agentic workloads: the task is the unit¶

A product team builds two features. Feature A is a search-and-answer system: retrieve documents, generate a response, return it. Feature B is a coding assistant: read files, plan changes, write code, run tests, read errors, revise, run tests again. Both call the same model API. The monthly bill is $40,000.

Feature A: 1.1 calls per user request, output-token dominated. Feature B: 23 calls per task (σ=40), input-token dominated by context growth across turns, sub-agent calls, tool output ingestion, cache misses after tool mutations, and repair loops. Different cost structures entirely.

Answer inference is a function. One request, maybe a retry, predictable cost. Levers: model choice, prompt, caching, output length.

Agentic inference is a loop: observe, plan, act, observe, decide. Cost is variable: a task might take 5 calls or 500 depending on task difficulty, tool quality, and termination policy. The variance is the cost problem.

The key economic concept is the fanout multiplier: how many LLM calls does one user request generate?

agent_fanout_multiplier =
  total_llm_calls_for_task / user_visible_requests

token_fanout_multiplier =
  total_processed_input_tokens / initial_user_prompt_tokens

Published fanout figures span an order of magnitude: ~4x for single-agent chat (Anthropic), ~15x for multi-agent (Anthropic), and hundreds-to-thousands of times for long-horizon code-task benchmarks (arXiv 2604.22750). The cascade is structural; different task definitions, not a single rising trend. Your fanout depends on your architecture, tools, and task mix.

Here is a worked example of a coding agent task ("Fix the authentication bug in the login flow"):

Phase	LLM calls	Input tokens	Output tokens	Cache read tokens	Tool calls
Planning	2	12,000	2,400	8,000	0
File reading	4	34,000	1,600	24,000	6
Implementation	3	42,000	4,800	32,000	4
Test + revise	8	68,000	6,200	48,000	12
Compaction	1	4,000	1,800	0	0
Final verify	2	18,000	1,200	14,000	3
Total	20	178,000	18,000	126,000	25

Fanout multiplier: 20 LLM calls per 1 user request. Token fanout: 178K input tokens from a 2K user prompt, an 89x multiplier. Cache hit rate: 71% of input tokens served from cache. Without caching, the input token bill would be roughly 3.4x higher.

But the economics don't stop at the fanout. The distribution of task difficulty matters enormously:

200 tasks/day, bimodal: simple tasks 8 calls, complex tasks 65 calls. 72% acceptance, but cost per accepted task varies 40x across the quartile spread. The blended average describes neither end of the distribution.

The cost per accepted task for agentic workloads is:

task_cost =
  initial_attempt_cost
  + P(test_failure) * repair_cost
  + P(review_rejection) * revision_cost
  + P(abandonment) * sunk_cost

cost_per_accepted_task =
  sum(task_costs) / count(accepted + repaired_accepted)

Three pressures compound. Context grows: a session can run 15K to 150K tokens before compaction, and compaction itself trades input cost for downstream repair risk. Cache opportunity is high but fragile: stable system prompts and tool definitions are ideal for prefix caching, but tool outputs (file contents, test results, error traces) mutate the conversation and break the cached prefix. Fanout is bimodal: a "rename this variable" and a "refactor the authentication system" are both coding-agent tasks but their fanout, context growth, and failure modes differ by an order of magnitude.

Cache-safe prompt design means placing stable content before dynamic content and making tool outputs append-only where possible. Mid-session model changes, tool list changes, and unstable prompt ordering are the primary cache breakers.

A turn cap controls worst-case cost but not accepted-task rate. If the cap is too aggressive, complex tasks fail, repair rate increases, and cost per accepted task rises because the successes don't amortize the failures. The better control is a cost budget per task with an escalation path: if the agent exceeds the budget, it surfaces the partial result for human decision rather than continuing to spend.

RAG and extraction: the retrieval-generation boundary¶

A retrieval-augmented generation pipeline retrieves 8--12 document chunks, constructs a prompt with the retrieved context, and generates an answer with citations. The pipeline works. Then the team adds longer documents, increases the chunk count for better recall, and extends the context window to 32K tokens. The latency increases. The cost increases.

The quality doesn't improve proportionally. More retrieved context means more noise, more distractor passages, and more opportunities for the model to hallucinate a plausible-sounding answer from an irrelevant chunk.

RAG economics have three layers that interact:

Retrieval cost. Embedding the query, searching the vector store, reranking candidates. This is usually cheap per query but scales with corpus size, index freshness, and reranking depth.

Generation cost. The retrieved chunks become input tokens. More chunks mean more input tokens. Double the retrieved context, roughly double the input cost. But the generation quality curve is not linear. Past a saturation point, additional retrieved context adds noise without improving answer quality.

Acceptance cost. RAG answers need grounding checks. Did the answer use the retrieved passages? Did it hallucinate facts not in the retrieved set? Did it cite the correct passages? Schema validation, field-level matching, and grounding audits have their own cost: sometimes a second LLM call, sometimes deterministic checks.

The cost model:

rag_request_cost =
  embedding_cost(query)
  + retrieval_cost(query, corpus_size, rerank_depth)
  + generation_cost(retrieved_tokens + prompt_tokens, output_tokens)
  + grounding_check_cost(output, retrieved_set)

cost_per_grounded_answer =
  sum(rag_request_costs) / count(grounded_accepted_answers)

The optimization that matters most is not the model price. It is the retrieval quality. A pipeline that retrieves the right 4 chunks and generates from 4K context tokens outperforms a pipeline that retrieves 12 chunks with 3 irrelevant distractors and generates from 16K context tokens, at lower cost and lower latency.

In one RAG deployment I worked on, framework orchestration abstractions accounted for over half of agent response latency. Replacing the hot path dropped p95 from ~8s to ~3s. Framework overhead is not free, and it compounds across every request.

Document extraction is RAG's sibling with different economics: one document in (no retrieval), structured output (schema validation is deterministic), batch-eligible (no human waiting), and quality measured at the field level (not answer-grounded).

Extraction workloads are often batch-eligible. The approximately 50% batch discount available from OpenAI, Anthropic, Google, and several serverless open-model providers is real savings when latency is not a constraint. The batch discount is the single largest cost lever for extraction workloads that tolerate async processing.

Offline, voice, and the billing grammar¶

Voice: the latency budget¶

A real-time conversational system with a sub-2-second response budget allocates roughly 180-380ms to ASR, ~50ms to routing/intent, 280-780ms to LLM inference depending on output length, 220-420ms to TTS, and 100-200ms to network and orchestration. The LLM's budget is whatever's left after everything else takes its share.

There is no room for retry, fallback, or queue delay.

Voice workloads have economics that differ from text in three ways.

Billing units change. Some voice APIs bill by audio minute, not by token. OpenAI's Realtime API bills audio input and output at token rates. Whisper bills per audio minute. Several inference providers offer separate per-minute rates for standard transcription and lower-latency streaming. The billing grammar matters: know whether you are paying per token, per minute, or per session.

Latency is a hard constraint, not a target. In text chat, a slow response is annoying. In voice, a slow response breaks the conversation. The latency SLO is a ceiling. The inference budget is the residual after ASR, TTS, network, and orchestration consume their share:

llm_budget_ms = total_budget_ms - asr_ms - tts_ms - network_ms - orchestration_ms

The LLM budget determines which models are feasible, which quantization levels are required, and whether dedicated capacity is needed to guarantee TTFT. The model choice is constrained by physics, not by preference.

Fallback changes the cost structure. When the LLM path fails or exceeds the latency budget, the system falls back to rule-based responses. In one production voice system I worked on, fallback rate sat below 1% under normal load and climbed to ~17-22% during provider incidents. Fallback shifts cost from inference to support burden. The trace must capture both paths:

voice_cost_per_interaction =
  P(llm_path) * (asr_cost + llm_cost + tts_cost)
  + P(fallback_path) * (asr_cost + rule_cost + tts_cost)
  + P(escalation | fallback) * escalation_cost

Batch, embeddings, and evals as workloads¶

Three offline workloads share the same API: embedding generation for a RAG corpus (2M documents), nightly evaluation of model quality (5,000 test cases), and weekly batch extraction from support transcripts (50,000 documents). All three pay real-time prices with real-time latency guarantees they do not need.

Offline workloads are defined by one property: no human is waiting. This changes the economics:

Batch APIs offer approximately 50% discounts. OpenAI, Anthropic, and Google offer batch processing at roughly 50% of standard rates for select models. The trade is latency for cost: batch jobs complete within hours rather than in real time. Completion windows, eligible models, and discount stacking rules vary by provider.

Embedding billing differs from generation billing. Embedding models bill per input token with no output token charge. The output is a vector, not text. Embedding 2M documents at 500 tokens per document is 1 billion input tokens. At typical embedding rates around $0.02/MTok, that is roughly $20. At $2/MTok on a frontier generation model, the same input would cost $2,000. The billing unit and model choice matter more than batch discounts for high-volume offline work.

Eval workloads have compounding cost. A quality eval that runs 5,000 test cases through a model-based grader is itself an inference workload. If the grader is the same frontier model being evaluated, the eval cost can approach the production cost. Deterministic checks (schema validation, exact match, regex) cost nothing at the model API level. The optimization is clear: run deterministic checks first, only send to model grading what passes deterministic gates, only send to human review what the model grader flags as ambiguous.

eval_cost =
  deterministic_check_cost  (approximately 0)
  + model_grader_cost (test_cases * grader_input * grader_output * grader_price)
  + human_review_cost (sample_size * review_minutes * labor_rate)

Shared infrastructure and the allocation problem¶

The four workloads from the opening share one provider account. The monthly bill is one number. Attributing that cost to individual workloads requires an allocation model. Allocation models are imprecise. The question is whether they are useful enough to inform decisions despite being imprecise.

Three allocation strategies, each with its own failure mode:

Proportional to tokens consumed. Simple. Assign cost to each workload based on its share of total input + output tokens. This works when all workloads have similar cache hit rates, similar retry rates, and similar output/input ratios. It fails when they don't, and they usually don't. The coding agent with a 71% cache hit rate and the batch extraction with 0% cache hit rate have very different effective token costs. Allocating on raw token volume misattributes cost from the uncached workload to the cached one.

Proportional to trace-derived cost. Better. Use the pricing snapshot to calculate each workload's trace-derived cost (applying cache discounts, batch discounts, and input/output price differentials). Allocate the total invoice proportionally to trace-derived cost. This handles most of the billing grammar correctly. It fails on the delta: the gap between trace-derived cost and invoice. If the delta is 5%, it is noise. If it is 15%, someone is eating a disproportionate share.

Workload-level accounting with an unknown bucket. Best but hardest. Assign every trace to a workload class. Calculate trace-derived cost per workload. Reconcile each workload's trace cost against the provider's billing dimensions (project, API key, model, batch flag). Whatever remains unattributed goes into an unknown bucket. If the unknown bucket is under 5% of total cost, the allocation is trustworthy. If it is over 10%, the instrumentation needs work before the allocation is useful.

The unknown bucket is the most important concept in workload cost attribution. It captures: requests with missing workload IDs, calls from deprecated integrations, test traffic mixed into production, retry chains where the original request trace was lost. Do not distribute the unknown bucket proportionally across workloads. That is the same averaging mistake from the opening. Leave it visible. Report it. Investigate it.

Cross-workload subsidy¶

The blended average hides the cross-subsidy. When four workloads share one cost model, the cheap ones subsidize the expensive ones.

Pricing decisions. If the support chatbot has LCPR = $0.038 per accepted resolution and the coding agent has LCPR = $0.14 per accepted task, pricing both features based on the blended average ($0.07) means the chatbot subsidizes the coding agent. A customer who only uses the chatbot is overcharged. A customer who heavily uses the coding agent is undercharged.

Optimization decisions. The blended average says "our cost is $0.07 per accepted unit." This is not actionable. Which workload do you optimize? Where is the dominant lever? On the chatbot, it might be cache hit rate. On the coding agent, it might be fanout control and repair rate. On the extraction pipeline, it might be batch eligibility. The blended average hides the lever.

Migration decisions. A team evaluates switching providers. The new provider is 30% cheaper per token. But the new provider's cache TTL is shorter, its batch API has different completion windows, and its model produces longer outputs. These differences affect each workload differently. The chatbot might save 25%. The coding agent might cost 15% more because the shorter cache TTL breaks its cache economics. The blended migration analysis says "save 20%." The workload-level analysis says "save on two workloads, lose on one."

Multi-workload LCPR¶

LCPR applies at the workload level, not the account level. Each workload gets its own calculation:

LCPR_workload = (C_inference_w + C_eval_w + C_human_w + C_ops_w + delta_w) / A_w

The variables are the same as the LCPR formula. The subscript _w means "allocated to this workload." The allocation itself is the hard part: separating C_inference, C_eval, C_human, and C_ops into workload-level buckets requires the trace-level attribution and workload identity schema described above.

For the four-workload system from the opening, the multi-workload LCPR might look like this:

Workload	C_inference	C_eval	C_human	C_ops	delta	A (accepted)	LCPR
Support chat	$8,200	$480	$3,000	$800	$380	28,500	$0.45
Coding agent	$18,400	$1,200	$2,400	$600	$820	4,100	$5.71
Document extraction	$4,800	$120	$200	$300	$210	42,000	$0.13
Eval runs	$3,600	$0	$0	$200	$160	--	--
Unknown bucket	$1,200	--	--	--	$80	--	--
Total	$36,200	$1,800	$5,600	$1,900	$1,650	--	--

Several things are visible in this table that the blended average hides:

The coding agent's LCPR is 44x the extraction pipeline's LCPR. Agentic workloads have high fanout, high repair rates, and expensive accepted-task criteria. The ratio matters for pricing and investment decisions.

The eval workload has no denominator. Its accepted work unit is "a decisive routing or release decision," not a customer-facing output. Eval cost is infrastructure cost, like monitoring or CI. It should be allocated across the workloads it serves, not reported as its own LCPR.

The unknown bucket is $1,280, about 2.7% of total cost. Small enough to accept; large enough to investigate if it grows.

Human escalation cost is concentrated in support chat ($3,000/month) and coding agent ($2,400/month). The extraction pipeline has minimal human cost ($200/month) because schema validation catches most failures deterministically. This distribution tells you where quality improvement has the highest economic return.

What to measure for each workload class¶

The workload class determines what to measure:

Workload class	Primary metric	Secondary metrics	Economic lever
Conversational	Cost per accepted resolution	Session length, cache hit rate by turn, escalation rate	Quality gate, cache stability
Agentic	Cost per accepted task	Fanout multiplier, repair rate, acceptance rate by complexity tier	Fanout control, termination policy
RAG	Cost per grounded answer	Retrieval precision, chunk count, grounding check pass rate	Retrieval quality, context packing
Extraction	Cost per validated record	Schema pass rate, batch-eligible share, human review rate	Batch eligibility, deterministic checks
Voice	Cost per completed interaction	LLM budget utilization, fallback rate, escalation from fallback	Latency budget, fallback quality
Batch/offline	Cost per accepted batch item	Completion time, retry rate, expiration rate	Batch API usage, volume pricing
Eval	Cost per decisive decision	Deterministic coverage, grader agreement rate	Deterministic checks first

Each row is a different LCPR configuration. Same formula, different inputs, different denominators. The formula is durable. The workload identity determines which inputs matter.

Limitations¶

Workload classes drift. A support chat workload gets extended to handle complex multi-step troubleshooting. The token profile changes, the latency profile changes, the cache hit rate drops, and the quality gate needs updating. The workload identity says "support-chat" but the traffic looks like an agent. Review workload identities quarterly or when failure rates change.

Bimodal distributions within a class. A "what is your return policy?" question and a multi-step account recovery are both support chat, but their cost structures differ by an order of magnitude. If the bimodal distribution is large enough to distort routing and cost modeling, split the workload class.

Agentic workloads without acceptance criteria. "Improve the codebase" is not a measurable task. Without a defined acceptance criterion (tests pass, linter clean, PR approved, issue closed) the denominator is undefined and cost per accepted task is meaningless. Define the acceptance criterion before optimizing.

Very low volume. At 10 requests per day, the ops overhead per request is large and unstable. Workload-level LCPR is most useful when volume is high enough for the per-unit allocation to be meaningful: typically hundreds of requests per day or more.

Allocation models for shared cost. When workloads share a dedicated endpoint, GPU time, or engineering headcount, allocating cost to individual workloads requires assumptions. Those assumptions should be visible in the assumption register, owned by someone, and refreshed periodically. A wrong allocation that changes a routing decision is worse than no allocation.

Background agents without a speed premium. For agents with no human in the loop that run overnight, the latency difference between a frontier model and a specialist matters less than the cost difference. The relevant metric shifts from time-to-first-token to cost per accepted result. The sub-agent routing decisions change. The entire optimization surface is different from interactive agentic workloads.

Production Inference Economics — Part 3 of 3

The companion serving-side measurement (goodput, the productive-capacity framework that feeds the LCPR denominator) is developed in Chapter 9 of Production Inference Economics: A Field Guide. It covers the goodput frontier test, cache-local routing as an economic lever, and how to avoid the most common benchmark mistakes.

Sohail Mohammad — May 2026

Numbers are anonymized and should not be attributed to any specific employer, customer, or deployment.

The LCPR Calculator

2026-04-29T00:00:00-05:00

I work at Together AI. Technical details have been generalized from production experience; no proprietary information from any organization is disclosed.

Production Inference Economics — Part 2 of 3

The LCPR Calculator implements the loaded-cost framework from The Denominator Problem as a Python tool. Trace data, provider invoice, eval results, and contract terms go in. LCPR, margin, cache-break-even, and sensitivity surfaces come out.

Open source. MIT licensed. Three worked examples with seed YAML you can reproduce.

Seven computations¶

Each isolates a different question.

1. LCPR Comparison. Loaded cost per accepted result across providers and deployment modes. Formula and definitions in Article 1. The comparison surfaces which deployment minimizes loaded cost, usually not the one minimizing per-token cost.

2. Sensitivity Analysis. Vary one input (retry rate, quality gate pass rate, cache hit rate, engineering hours), hold others constant, see which lever matters most. On quality-sensitive workloads, the quality gate dominates the token price. A 10-point drop in eval pass rate moves LCPR more than a 2x change in per-token pricing.

3. Break-Even Analysis. At what daily output token volume does dedicated capacity beat serverless? The answer involves goodput (accepted work per second under SLO), not peak throughput. A dedicated GPU that processes 200 requests per second but only 140 pass your quality gate has a goodput of 140. The break-even calculation uses the number that hits your invoice, not the number that hits your dashboard.

4. goodput frontier. Accepted requests per second under latency and quality SLOs:

goodput = count(requests meeting ALL gates) / duration

Peak throughput is a hardware spec. Goodput is an engineering outcome. The full productive-capacity framework is developed in Chapter 9 of Production Inference Economics: A Field Guide.

5. Trace-to-loaded-cost reconciliation. From raw traces to reconciled loaded cost via the four-source join (Trace + Invoice + Eval + Contract). delta = invoice - trace_derived_cost. If delta exceeds 5%, investigate. Your traces are either missing calls, miscounting tokens, or the provider is billing something your instrumentation does not capture. The full reconciliation method is in Chapter 25 of the book.

6. Cache Break-Even.

N_break_even = (p_write - p_read) / (p_in - p_read)

Where p_write is the cache write cost per token, p_read is the cache read (hit) cost, and p_in is the standard input price. On Anthropic's 5-minute cache: 2 calls within TTL to break even. On their 1-hour cache: 3 calls. On OpenAI automatic caching: any hit saves money because there is no explicit write cost.

The formula is portable; provider numbers are not. Don't trust the pricing page discount percentage; trust the break-even count against your measured reuse rate. A 90% cache discount means nothing at a reuse rate of 1.3 hits per TTL.

7. KV Memory Sizing.

kv_bytes_per_token = 2 * layers * KV_heads * head_dim * element_bytes

For Llama 3 70B in bf16: 320 KiB per token per sequence. At 4K context with a 40GB KV pool: roughly 26 concurrent sequences after accounting for non-KV memory overhead. At 128K context: zero. You physically cannot fit a single 128K sequence in a 40GB KV budget on this architecture without quantized KV or offloading.

Context length is a capacity allocation, not just a model setting. Every token of context you allow costs memory that could serve another concurrent user. The calculator makes this trade-off explicit.

Three Worked Examples¶

Each example ships with a seed YAML file. Clone the repo, run the seed, get the same numbers.

Example 1: Support Answer trace-to-loaded-cost¶

Seed: examples/support-answer.trace-margin.v1/calculator-seed.yaml

The full breakdown is in Article 1. The seed YAML reproduces the same numbers if you want to verify the calculator against them: daily fleet of 1,000 tickets submitted, 820 accepted answers delivered, total loaded cost $140.65, LCPR $0.172, naive cost per ticket $0.014.

Human escalation is 71% of loaded cost; inference is 10%. The non-token costs run 7x the inference bill on this workload. The team that switched providers to save on token price was optimizing the 10% while ignoring the 71%. The lever that moves LCPR on this workload is the quality gate pass rate; every percentage point improvement in automated acceptance moves 50 fewer tickets to human review at $2 each.

Example 2: Coding Agent Task Lifecycle¶

Seed: examples/coding-agent.lifecycle.v1/calculator-seed.yaml

One accepted bug fix across an agent session: 20 LLM calls, 178K input tokens, 18K output tokens, 25 tool calls (file reads, test runs, grep searches). The user submitted a 2K-token bug report. Token fanout: 89x. A 2K prompt becomes 178K input tokens across the session because every subsequent turn re-sends the growing conversation context plus tool results.

Cache behavior is bimodal. The main agent loop hits 82% cache rate (stable system prompt, growing context window, high prefix overlap between turns). Sub-agent calls (linter, test runner, code search) hit 45% (fresh context each time, minimal prefix reuse). When a compaction event fires (context exceeds the window and gets summarized), the cached prefix is destroyed. The next turn pays full input price on the compacted context plus a cache write on the new prefix. Cache hit rate across the fleet dropped from 60% to 35% after a compaction policy change.

Fleet: 200 tasks per day, 90% acceptance rate (65% first-pass acceptance + 25% repaired by the agent's self-correction loop), 10% manual developer takeover.

Agent economics live at the task level, not the request level. A task that generates 20 model calls makes per-request cost noise. A single compaction event swings effective input rate 40%, which makes per-token cost noise. The calculator models the full task lifecycle.

Example 3: Benchmark Audit¶

Seed: examples/support-rag-answer-drafting.audit.v1/calculator-seed.yaml

Two routes benchmarked for the same RAG answer-drafting workload. Route A wins on mean throughput: 45 requests per second vs. Route B's 38. The procurement recommendation goes to Route A.

Route B wins on goodput. Route A has a 72% eval pass rate. Route B has 91%. Under a 2-second P95 latency SLO and the quality gate, Route A's goodput is 28 accepted requests per second. Route B's is 33. The "slower" route produces more accepted work per second.

The benchmark that selected Route A contained eight methodology errors:

Closed-loop arrival process (no think time between requests; inflates throughput by 20-40% vs. production traffic patterns)
Cold cache (no prefix caching warmed; production runs 60%+ cache hit rate)
Excluded cold start latency from measurements
No retry policy applied to failed requests
No quality gate or eval pass rate measured
Reported only mean latency (hides tail behavior)
No P95/P99 tail percentiles
No cost-per-accepted-work calculation

The seed file documents exactly which gaps created the false winner. The missing metadata checklist in the YAML specifies what a production-grade benchmark must capture. Run the seed through the calculator and it produces the goodput-adjusted comparison that reverses the ranking.

How the formula evolved¶

The calculator's LCPR formula matches the definition in Article 1: C_inference + C_eval + C_human + C_ops + delta over accepted-work units. The earlier field-guide formula collapsed eval, human, and ops costs into a single engineering_cost bucket; splitting them surfaced that 71% of engineering_cost was human escalation on the support workload, which is the lever a unified bucket hides.

200+ tests check every formula against the series derivations. Every worked example reproduces from its seed YAML. If a number in this post doesn't match what the calculator produces, the calculator is the source of truth.

How to Use It¶

# Clone and install
git clone https://github.com/Sohailm25/inference-field-guide.git
cd inference-field-guide
pip install -e ".[dev]"

# Run the CLI
lcpr compare --profile saas_chat
lcpr crossover
lcpr sensitivity --vary retry_rate

# Run the worked examples
python -m examples.run_seeds

# Run the tests
pytest

# Launch the interactive app
streamlit run calculator/app.py

Or use the hosted Streamlit app: inference-econ.streamlit.app.

Bring your own numbers. Edit calculator/provider_pricing.yaml with your actual contracted rates (not the pricing page; your rates). Create a workload profile from your production traces: actual retry rate, actual eval pass rate, actual cache hit rate, actual human escalation volume. Run the sensitivity analysis.

The calculator is a hypothesis. It becomes useful when you replace the default assumptions with measurements from your stack. An LCPR model built on six assumptions is a thought experiment. One built on six measurements from your production logs is a budget.

Where This Breaks¶

Not every workload needs LCPR. Be specific about when simpler analysis is sufficient.

Simple single-turn classification. If you send a prompt, get a label, and the pass rate is 98%+, token price is a fine proxy for cost. The loaded overhead is negligible. You do not need this calculator.

Batch workloads with 50% discount. Batch APIs (OpenAI, Anthropic) offer 50% discounts for async processing. Model cost at the batch level, not per-request. The cache and latency computations in the calculator do not apply to batch (no SLOs, no interactive cache reuse).

Inference is less than 0.1% of operating cost. If your inference bill is $200/month and your engineering payroll is $2M/month, spend your analysis time on whatever is 20% of cost. LCPR optimization on a $200 line item is not a productive use of engineering hours.

Very low volume (under 100 requests per day). The per-unit ops allocation becomes noisy. Eight engineering hours per month amortized over 3,000 monthly requests adds $0.27 per request in ops overhead alone. That noise overwhelms the signal from token pricing differences.

Shared dedicated endpoints. If multiple workloads share a GPU pool, the allocation model for attributing capacity cost to each workload is always wrong. The question is whether it is useful enough to inform decisions. The calculator assumes you can attribute capacity; if you cannot, the dedicated-vs-serverless comparison is directional, not precise.

Close¶

The calculator returns the loaded-cost ranking and the dominant input that drove it. One number for procurement, one diagnostic for engineering.

The full argument is in The Denominator Problem. The derivations and formal framework are in Production Inference Economics: A Field Guide. The calculator is at github.com/Sohailm25/inference-field-guide.

Contributions welcome: pricing updates as providers change rates, new workload profiles from production deployments, bug reports when the calculator disagrees with the derivations. File an issue or open a PR.

Production Inference Economics — Part 2 of 3

Sohail Mohammad — April 2026

Numbers are anonymized and should not be attributed to any specific employer, customer, or deployment.

Trace Autopsy: Following One Day's Inference Bill

2026-04-22T00:00:00-05:00

This article is a companion to Chapter 25 of Production Inference Economics: A Field Guide, which develops the trace-to-loaded-cost reconciliation method on a different anonymized workload (a regulated clinical-prior-auth deployment) and adds a regulatory observability extension. The synthetic walkthrough below is the simpler teaching version.

I work at Together AI. Technical details have been generalized from production experience; no proprietary information from any organization is disclosed.

The Denominator Problem named the problem and introduced LCPR — loaded cost per accepted result. This article shows the measurement: how you go from raw trace events to a number you can audit against the provider invoice.

I call it a Trace Autopsy because the method is the same one a clinician uses: reconstruct what happened, find where the costs disappeared between the pricing page and the P&L, do it systematically. The ad hoc version is how teams end up confident in wrong numbers for months.

You have an inference bill. You suspect the arithmetic is right but the question is wrong. Your provider charged you accurately for the tokens you consumed. The tokens you consumed aren't the tokens you planned. Requests retried. Outputs failed quality checks and got regenerated. An eval grader ran on the failures. Cache missed. The denominator (the count of work units your customer actually accepted) shrank.

The pricing page told you what a token costs. It didn't tell you how many tokens a result costs. That's what you need to measure.

The Trace Autopsy is a repeatable protocol for going from raw trace events to loaded cost per accepted result. I'll walk through every step using a single day's trace data. The trace is illustrative; I constructed it. The field shapes, the billing grammar, and the failure modes come from real provider documentation and real production patterns. The numbers are designed to be realistic, not real. The method is real.

What Twelve Requests Actually Look Like¶

The team expected eight requests. The system generated twelve.

Those four extra requests are the entire story. They're the gap between the pricing page and the bill. Here's the full trace for one batch of eight customer tickets:

#	Type	Input tok	Cached	Output tok	TTFT (time to first token) ms	E2E (end-to-end latency) ms	Eval	Notes
1	First attempt	2,400	1,800	220	380	2,100	Pass	Cache hit on system prompt
2	First attempt	3,100	0	340	920	3,800	Pass	Cache miss, new retrieval context
3	First attempt	2,600	2,100	280	350	2,400	Fail	Answer missed key constraint
4	First attempt	2,800	0	510	1,100	5,200	Fail	TTFT exceeded 800ms SLO (service-level objective)
5	First attempt	2,200	1,600	190	410	1,900	Pass	—
6	First attempt	4,500	0	380	1,400	4,800	Pass	Long retrieval context, cache miss
7	First attempt	2,500	2,000	250	360	2,200	Pass	—
8	First attempt	3,200	0	620	980	6,100	Fail	Output too long, E2E violated SLO
9	Retry of #4	2,800	0	290	850	3,400	Pass	Same input, no cache (TTL expired)
10	Retry of #8	3,200	0	310	720	3,100	Pass	Retry succeeded
11	Eval grader	1,200	0	80	200	800	—	Grading requests 3 and 8
12	Repair of #3	3,800	0	350	650	3,200	Pass	Regenerated with corrected context

Each row is a clinical event. The pattern across rows tells the story.

Requests 1 through 8 are first attempts, one per customer ticket. This is the work the team planned for. Eight tickets come in, eight inference calls go out. If you're estimating inference cost from a spreadsheet, these eight rows are all you modeled.

Five of the eight pass (requests 1, 2, 5, 6, 7). Three fail. They fail for different reasons, and the failure mode determines what happens next.

Request 3 fails quality. The LLM-as-judge eval says the answer missed a key constraint. This isn't a latency problem; the response came back in 2,400ms, well within SLO. The content was wrong. Wrong content means the system needs to try again with better context, which is a repair, not a retry.

Request 4 fails latency. TTFT hit 1,100ms against an 800ms SLO. The output might have been fine; we'll never know, because the system timed out the first-token wait. This is a retry candidate: same input, same prompt, hope for better luck on the next scheduling pass.

Request 8 fails on end-to-end latency. The output was 620 tokens (nearly 2.5x the median) and the total call took 6,100ms. The E2E SLO was probably around 5,000ms. Again, the output quality might have been acceptable, but the user experience wasn't. Another retry.

Now the recovery machinery kicks in.

Request 9 retries request 4. Same input, 2,800 tokens. But notice: zero cached tokens. The retry happened after the cache TTL expired, so the system paid full input price. TTFT came in at 850ms, just above the 800ms SLO, but this time it squeaked through. The team tuned the SLO with some tolerance band, or the eval was lenient on near-misses. Either way, it passed.

Request 10 retries request 8. Same 3,200 input tokens, again zero cached. This time the output was 310 tokens instead of 620; the model's stochastic nature worked in the team's favor. E2E dropped to 3,100ms. Pass.

Request 11 is the one most teams forget to account for. It's the eval grader: an LLM-as-judge call that evaluates the failed outputs from requests 3 and 8. It consumed 1,200 input tokens and 80 output tokens. It produced zero customer-facing value. Its entire purpose was to classify the failures so the system could decide between retry, repair, and escalation. But it's real token spend on a real provider bill.

Request 12 repairs request 3. This isn't a retry; it's a new prompt. The system took the failure signal from the eval grader, fetched corrected retrieval context, and rebuilt the prompt. That's why the input is 3,800 tokens instead of the original 2,600. The repair succeeded.

Final tally: eight customer tickets in, twelve inference calls out, eight accepted answers delivered. The four extra calls (two retries, one eval grader, one repair) are invisible on the pricing page. They're visible on the bill.

Why the Trace Alone Isn't Enough¶

If you've followed the analysis so far, you might think the trace is all you need. It isn't. I learned this the hard way.

The trace tells you what the application saw. It doesn't tell you what the provider charged. It doesn't tell you whether the output was good. It doesn't tell you what the customer is paying. You need four data sources, and you need to join them.

Source 1: The Trace¶

The request-level event log. Every inference call gets a row with timestamps, token counts, model name, latency breakdown, cache behavior, and (if your instrumentation is good) a request chain ID linking retries and repairs back to the original ticket.

What it gives you: Cost attribution per request. Latency breakdown (TTFT, generation time, E2E). Cache hit/miss classification. Failure type (latency vs quality vs compliance). Retry chain linkage.

What it misses: The trace records what your tokenizer counted. The provider's tokenizer might count differently. The trace doesn't know about rounding rules, minimum charges, batch pricing adjustments, or credits applied to the invoice. If your tracing has gaps (sampled traces, async calls that don't propagate context, logging failures during high load) the trace undercounts.

Source 2: The Invoice¶

The provider's billing record. This is ground truth for what was charged. Not what should have been charged, not what you estimated. What the provider's billing system computed.

What it gives you: Actual charges by model, by billing period. Committed spend burn-down (if you have a spend commitment). Credits, adjustments, overages.

What it misses: The invoice is an aggregate. It tells you "you spent $14,850 on gpt-4.1 this month." It doesn't tell you which requests drove that cost, why cost moved week over week, or which workload is responsible for which portion of the bill. Attribution requires the trace.

Source 3: The Eval¶

Quality measurement. This is where the denominator comes from.

What it gives you: Pass/fail per output. Failure clustering (what types of failures dominate). Repair success rate. The count of accepted work units, the denominator for LCPR.

What it misses: Evals don't know about cost. They don't know about latency. A perfect eval system can tell you that 82% of outputs were accepted, but it can't tell you whether the 18% that failed were cheap or expensive to produce. That correlation (failure rate by cost bucket) requires joining eval data back to the trace.

Source 4: The Contract¶

The commercial agreement. Revenue per unit, committed spend obligations, SLA terms, data handling requirements.

What it gives you: Revenue per accepted work unit (so you can calculate margin). Committed spend thresholds (so you know if you're burning through a commitment faster than expected). SLA penalties (so you can model the cost of latency failures). Data residency constraints (which may force you onto specific providers or regions at different price points).

What it misses: Everything operational. The contract defines the economic frame. It doesn't produce telemetry.

The join¶

No single source answers the question "what does an accepted result cost us, and are we making money on it?" The answer requires all four:

Trace    (per-request cost, attribution, failure chains)
  + Invoice  (actual charges, corrections, credits)
  + Eval     (accepted work count, quality distribution)
  + Contract (revenue per unit, commitments, constraints)
  = Margin model

The join is hard for practical reasons, and each pair of sources has its own friction.

Trace-to-Invoice is the most common reconciliation, and the one most teams attempt first. The trace operates at request granularity, per-second. The invoice operates at monthly granularity, aggregated by model. Joining them requires aggregating traces up to the invoice period, matching model names (which may differ between your trace and the provider's billing system), and accepting that the totals won't match exactly. The delta between them is diagnostically valuable. I'll spend a full section on it below.

Trace-to-Eval links cost to quality. This is how you discover that your most expensive requests are also the ones most likely to fail. Without this join, you can't answer "are we spending more to produce bad outputs or good ones?"

The correlation matters for routing decisions: if long outputs (high cost) have a higher failure rate, you might cap output length or use a cheaper model for high-variance workloads. The join key is usually a request ID or ticket ID, but you need your eval pipeline to record which request it evaluated. Not just "this output passed" but "this output from request ID abc123 passed."

Eval-to-Contract connects quality to revenue. The contract specifies revenue per accepted work unit. The eval says 82% of answers were accepted (82,000 out of 100,000). The join gives you realized revenue: $45,000 for the month. Without the eval, you'd either assume 100% acceptance (overstating revenue) or use a placeholder acceptance rate that might be months out of date.

Contract-to-Trace closes the loop. The contract's data residency requirement says "EU customer data stays in EU." This constraint forces certain requests to EU-region endpoints, which may have different pricing, different cache behavior, and different latency characteristics. The trace shows the actual cost of serving those requests in EU. Without linking the contract constraint back to trace data, you can't measure the cost premium of compliance.

The eval covers a sample, not the full population. Most teams eval 10-30% of outputs, then extrapolate. The contract defines terms that don't produce telemetry: "data must not leave region X" is a constraint that affects which provider you use (and therefore which price you pay), but it doesn't show up in any log.

Time boundaries make it worse. Your trace timestamps are in UTC. Your invoice billing period might close at midnight Pacific. Your eval runs asynchronously: an output generated Monday might not be evaluated until Tuesday. Your contract defines "monthly" as calendar month, but your committed spend might reset on the 15th.

I've worked with teams that had clean traces, working evals, and signed contracts and still couldn't compute margin. Nobody owned the join. The trace team said "we give you cost per request." The eval team said "we give you quality scores." Finance said "we give you the invoice." Product said "we give you the contract." Nobody said "I'll stitch these together and tell you whether we're making money."

That ownership gap is the real problem. The four-source join isn't technically hard; it's organizationally hard. Each source has a different owner, a different update cadence, and a different schema. The team that builds the join pipeline is doing integration work that nobody asked for and everybody needs.

The four-source join is what makes LCPR measurable. The formula is one line. The measurement is where teams stop.

trace-to-loaded-cost Reconciliation, Step by Step¶

I'll walk through the full reconciliation using monthly numbers. This is the procedure I run (or want teams to run) at the end of every month.

Step 1: Pull trace-derived cost¶

Sum every inference call for the month: first attempts, retries, repairs, eval graders. Price each call at the current pricing snapshot (the per-token rates for the model and tier you're using).

Result: $14,200.

This number is your best estimate of what inference should have cost, based on what your application observed. It's not what the provider charged. It's what your traces say happened, priced at the rates you expect to pay.

A detail that matters: which pricing snapshot are you using? Providers change prices mid-month, sometimes without announcement beyond a changelog update. Split the month at the price-change date and price each half correctly. I keep a versioned pricing table keyed by model name and effective date. Tedious. It's also the difference between a 2% delta and a 12% delta that triggers a fire drill.

Step 2: Pull the invoice¶

The provider says: $14,850.

Delta: $650. That's 4.4%.

Is 4.4% acceptable? I use a 5% threshold. Under 5%, the trace is trustworthy for daily cost monitoring (the gap is noise from rounding, timing, and minor token count discrepancies). Over 5%, something is wrong and you need to investigate before trusting trace-based cost reporting.

At 4.4%, we proceed. I'll come back to what to do when the delta is large.

Step 3: Add eval grader cost¶

The LLM-as-judge calls that evaluate output quality: $800/month.

This is real inference spend. It hits the same provider bill. But it produces zero customer-facing output. Every token the eval grader consumes is overhead. Necessary overhead, because without it you don't have a quality signal and you don't have a denominator, but overhead nonetheless.

Some teams bury eval grader cost inside the inference line item. I break it out because it's a lever. If your eval grader is consuming 5% of your inference budget, it's worth asking whether you can eval fewer outputs (sample more aggressively), use a cheaper model for grading, or use a non-LLM eval where possible.

Step 4: Add human escalation cost¶

Tickets that failed quality, failed repair, and escalated to a human reviewer: 500 cases x $2 = $1,000/month.

In the daily trace I showed earlier, I cited $100/day from 50 escalations. The monthly number is 500, not 1,500, because escalation volume varies by day. Weekends are lighter. Some failure modes cluster: a bad retrieval index causes a burst of quality failures that clears once the index rebuilds.

The $2/case number is a loaded cost for human review time. In practice, this ranges from $1 to $15 depending on the domain, the complexity of review, and whether the reviewer is internal or outsourced. For customer support automation, $2 is realistic. For medical or legal domains, multiply by 5x.

Step 5: Add ops overhead¶

On-call, monitoring, prompt maintenance, eval set curation: $2,500/month.

This is the hardest number to pin down and the easiest to argue about. I allocate it as a proportion of the team's time spent on inference operations. If a team of four spends 25% of their time on prompt tuning, eval maintenance, incident response for inference quality, and provider relationship management, and the team costs $40,000/month loaded, that's $10,000/month across all workloads. If this workload is 25% of the total inference volume, it gets $2,500.

The allocation is imperfect. But zero is more wrong than $2,500. Teams that exclude ops overhead from their cost model are overstating margin by whatever the ops cost turns out to be.

Step 6: Count accepted outputs¶

Total requests: 100,000.

Accepted outputs: 82,000.

The 82% acceptance rate comes from the eval pipeline. Of 100,000 outputs generated, 82,000 passed quality gates, latency SLOs, and compliance checks. The remaining 18,000 were retried, repaired, escalated, or dropped.

This is the denominator. Not 100,000. Not the number of inference calls (which is higher than 100,000 because of retries and repairs). 82,000 accepted work units.

Step 7: Calculate LCPR¶

Component	Value
Trace-derived inference cost	$14,200
Provider invoice	$14,850
Delta	$650 (4.4%)
Eval grader cost	$800
Human escalation (500 cases x $2)	$1,000
Ops overhead allocation	$2,500
Total loaded cost	$19,150
Total requests	100,000
Accepted outputs	82,000
LCPR	$0.234

$19,150 / 82,000 = $0.234 per accepted answer.

Step 8: Calculate margin¶

Metric	Loaded	Naive
Revenue	$45,000	$45,000
Cost	$19,150	$14,200
Margin	$25,850	$30,800
Margin %	57.4%	68.4%

The naive calculation (revenue minus trace-derived inference cost, ignoring eval graders, human escalation, ops overhead, and the denominator) overstates margin by 11 percentage points. On $45,000 in revenue, that's a $4,950 difference in profit. Not enough to kill the business. Enough to make a bad routing decision, underestimate the cost of a new workload, or promise a customer a price you can't sustain.

Here's what the 11-point gap looks like in a planning conversation. The CEO asks: "Can we offer this workload to the next customer at $40,000/year instead of $45,000?" At naive margin (68.4%), the answer looks like yes; you'd still make 64.5% margin at $40,000. At loaded margin (57.4%), the answer is more cautious; margin drops to 52.1%, and you're only one bad month away from a workload that costs more to run than it earns. The naive number says take the deal. The loaded number says negotiate harder.

The naive number flatters the CEO. The loaded number flags the operator's risk.

The Delta, and When Not to Trust Your Trace¶

I glossed over the 4.4% delta between trace-derived cost ($14,200) and invoice ($14,850) earlier. Let me come back to it, because the delta is one of the most diagnostically valuable signals in the reconciliation.

The delta captures everything your trace didn't see or priced incorrectly. At 4.4%, the causes are likely boring: rounding differences, timing mismatches at billing period boundaries, minor token count discrepancies between your tokenizer and the provider's. Noise.

But I've seen deltas of 15%, 30%, even 2x. When that happens, the trace is lying to you, and daily cost monitoring based on trace data is fiction.

Common causes of large delta:

Missing traces. Your tracing pipeline sampled at 10% and you extrapolated. Or a service that makes inference calls doesn't propagate trace context, so those calls appear on the invoice but not in your traces. Or logging failed during a high-load period, exactly when cost was highest. The calls happened. The provider recorded them. Your traces didn't.

Model name aliases. Your trace says gpt-4.1. The invoice says gpt-4.1-2025-04-14. Are they the same model at the same price? Usually yes. During model transitions, the dated version might be priced differently, or your pricing snapshot might reference the alias while the provider bills the canonical name. I've watched a team spend a week debugging a cost discrepancy that turned out to be a model name string mismatch between their trace attribution logic and the provider's billing export.

Batch vs real-time pricing splits. If you use batch inference for some workloads, the batch calls are typically priced at 50% of real-time. Your trace might not distinguish batch from real-time, or might price both at the real-time rate.

Token count discrepancy. Your tokenizer is a local approximation. The provider's tokenizer is authoritative. For most models, they agree to within 1-2%. If you're using a third-party tokenizer library that hasn't been updated for the latest model's vocabulary changes, the gap can be larger.

I've seen 8% token count discrepancy on a model update that expanded the vocabulary; the old tokenizer split words the new one kept whole. The frustrating part: the discrepancy was consistent (always undercounting by ~8%), so the team's daily cost estimates looked internally consistent but were systematically wrong. They only caught it when the monthly reconciliation delta suddenly jumped.

Credits and commitments. The invoice includes $3,000 in committed spend credits that your trace doesn't know about. Trace says you spent $14,200. Provider says you spent $14,200 but only owe $11,200 because of prepaid credits. If you're reconciling against the amount charged rather than the amount consumed, the delta is artificial.

Timezone and period boundary mismatches. Your traces timestamp in UTC. The provider's billing period closes at midnight Pacific. Calls made between 00:00 UTC and 07:00 UTC on the first of the month might land in the previous billing period on the invoice but the current period in your traces.

I categorize delta causes into exception buckets:

Exception bucket	Meaning
missing_provider_request_id	App saw a workflow event but no provider request ID was captured
provider_usage_without_trace	Provider export shows usage not present in app traces
trace_usage_without_provider_export	Trace has calls not yet reflected in provider export
pricing_snapshot_mismatch	Trace estimate uses stale prices or wrong service tier
credit_or_commit_adjustment	Invoice includes credits or prepaid commitment burn-down
timezone_period_mismatch	Trace day and invoice day differ due to UTC vs local time

When delta < 5%: trust the trace for daily monitoring. Reconcile monthly to catch drift.

When delta > 5%: stop. Investigate. Don't publish trace-based cost numbers until you understand the gap. The trace is a model of reality. When the model diverges from reality by more than 5%, the model needs recalibration, not more confidence.

One pattern I've found useful: run the reconciliation weekly, not just monthly. A delta that's 3% in week one, 4% in week two, and 8% in week three tells you something changed in week three. If you only reconcile monthly, you see 5% and shrug. The weekly trend shows you when to start digging.

Start with Twenty Traces¶

All of this assumes you have trace infrastructure, eval pipelines, and invoice access. Most teams don't.

If that's you, start here: pull twenty traces from production. Not a hundred. Not a statistical sample. Twenty.

For each trace, answer six questions:

What workload is this? Can you name it? "Customer support ticket," "document summarization," "code review." If you can't name the workload, you can't attribute cost to it.
What was the input? How many tokens? Was any portion cached? Can you identify the system prompt, the retrieval context, and the user query as separate components?
What was the output? How many tokens? Did it pass your quality bar? Would you show it to a customer?
What did it cost? Can you calculate cost from the provider's current pricing? Input tokens times input price, output tokens times output price, minus cache discount. Does your number match the provider's usage dashboard?
Did it meet the latency SLO? What was TTFT? What was E2E? If you have an SLO, did this request meet it?
If it failed, what happened next? Was it retried? Repaired? Escalated to a human? Silently dropped? Or did nothing happen, and the failure was served to the customer?

Twenty traces will reveal the state of your instrumentation:

Whether you can attribute requests to workloads (or if everything is one undifferentiated stream)
Whether your token counts match the provider's (or if your tokenizer is stale)
Whether cache is actually hitting (or if your hit rate assumption is fiction)
Whether your quality gate is running (or if you're serving unchecked outputs)
Whether retries and repairs are visible in the trace (or if they're invisible background processes inflating cost without attribution)

If you can't answer these six questions for twenty requests, you can't build a cost model. The twenty-trace exercise takes an afternoon. It doesn't require new infrastructure. It requires pulling logs, reading them, and writing down what you find. The findings will tell you what to instrument next.

Here's what the exercise typically looks like. You pull twenty requests. For the first five, you can answer all six questions: they're from the well-instrumented primary workload, the one the team built first and cares about most.

For requests six through twelve, you can answer questions one through four but not five or six. There's no latency SLO defined, and failures don't trigger retries. Nobody decided whether they should.

For requests thirteen through seventeen, you can answer one and two but not three. There's no eval running on this workload. Outputs go directly to the customer unchecked.

For requests eighteen through twenty, you can't even answer question one. You can see the inference call in the trace, but you can't attribute it to a workload. It's an orphan request, probably from a service that makes inference calls but doesn't propagate the workload tag.

That distribution (five well-instrumented, seven partial, five minimal, three orphaned) is diagnostic. It tells you exactly where to invest next. It usually surprises the team; they assumed their instrumentation covered everything.

Every team I've run this exercise with has surfaced the same three gaps: workload attribution is missing, retry chains are invisible, and nobody can connect an individual request to a line item on the invoice. Twenty traces don't fix the gaps. They make the gaps undeniable.

Where this scales¶

Start with one workload, one day, twenty traces. Build the four-source join for that workload. Reconcile against the invoice. Calculate LCPR. Then expand.

The Trace Autopsy is a diagnostic, not a dashboard. Run it when the numbers move. Once the mechanisms stabilize, instrument them; do the first pass by hand.

The full chapter develops this method further, with additional worked examples and the regulatory observability extension. See Production Inference Economics: A Field Guide, Chapter 25.

Sohail Mohammad — April 2026

Numbers are anonymized and should not be attributed to any specific employer, customer, or deployment.

The Denominator Problem

2026-04-15T00:00:00-05:00

I work at Together AI. Technical details have been generalized from production experience; no proprietary information from any organization is disclosed.

Production Inference Economics — Part 1 of 3

The Denominator Problem
LCPR Calculator
Workload Costs

The full framework is in Production Inference Economics: A Field Guide. Chapter 25 develops the trace-to-margin reconciliation method; Chapter 9 develops the productive-capacity and cache-local routing framework.

A support team runs an answer-drafting workload. Roughly 1,000 tickets a day. Each ticket triggers RAG retrieval, an LLM generation pass, and a quality check. The system works. The margins are fine. Then someone opens the procurement spreadsheet.

Route A (the current provider) charges $3.00 per million input tokens and $15.00 per million output tokens. Route B charges $1.80 per million input and $8.00 per million output. That is 40% cheaper on input and 47% cheaper on output. The spreadsheet says: switch to Route B, save roughly $4,200 per month.

The team switches.

Thirty days later, the inference bill is higher than before. The support burden has increased. The product manager is asking what happened.

Across three workloads I've audited (customer support, document extraction, and an agentic workflow) the same shape recurs. The per-token comparison is correct on its own terms. The arithmetic checks out. The conclusion is wrong, because the spreadsheet is answering a question nobody actually asked.

The spreadsheet answers "which provider charges less per token?" The question that matters is "which provider produces accepted work at a lower total cost?" The gap between those two questions is the denominator problem, and it is the most common mistake in inference economics.

The trace that explained everything¶

When the bill came in wrong, the team did what every engineering team does. They pulled traces. Not aggregate dashboards. Not the provider's usage summary. The actual request-level event log, call by call.

Here is a representative slice from a single day on Route B. The trace is illustrative; I simplified it to eight rows. The field shapes, billing grammar, and cache semantics come from real provider documentation. The failure modes come from production.

#	Type	Input tok	Cached	Output tok	TTFT (time to first token) ms	E2E (end-to-end latency) ms	Eval	Notes
1	First attempt	2,400	1,800	220	380	2,100	Pass	Cache hit on system prompt
2	First attempt	3,100	0	340	920	3,800	Pass	Cache miss, new retrieval context
3	First attempt	2,600	2,100	280	350	2,400	Fail	Answer missed key constraint
4	First attempt	2,800	0	510	1,100	5,200	Fail	TTFT exceeded 800ms SLO (service-level objective)
5	First attempt	2,200	1,600	190	410	1,900	Pass	—
6	First attempt	4,500	0	380	1,400	4,800	Pass	Long retrieval context, cache miss
7	Retry of #4	2,800	0	290	850	3,400	Pass	Same input, no cache (TTL expired)
8	Repair of #3	3,800	0	350	650	3,200	Pass	Regenerated with corrected context

Eight rows. Six first attempts, one retry, one repair. The team expected six requests to serve six tickets. The system generated eight.

Look at the "Cached" column. Requests 2, 4, and 6 show zero cached tokens. These are cache misses — the system prompt prefix should have been cached, but was not. Requests 1, 3, and 5 got cache hits. That is a 50% hit rate on this slice. Across the full day, the hit rate was 35%. On Route A, it had been 60%.

Look at the "Output tok" column. Request 4 generated 510 tokens. Request 6 generated 380. The team had modeled 250 as the median. Output tokens are the expensive side of the invoice.

Now look at the bottom of the table. Request 7 is a retry of request 4, resubmitted because the first attempt blew through the 800ms TTFT SLO. Request 8 is a repair. The system detected that request 3 gave a bad answer, pulled corrected retrieval context, and re-prompted. Both consumed fresh input tokens, generated fresh output tokens, and burned fresh latency budget.

On the full day, this pattern repeats across all 1,000 tickets. Some first attempts pass all gates. Some fail quality. Some fail latency. Some fail both. The failures generate retries, repairs, and eval grader calls. Each of those costs tokens. None of them appear on the pricing comparison page.

What the spreadsheet missed¶

Five things broke the spreadsheet's logic.

1. The cache hit rate was worse than expected¶

The team designed the prompt with a stable system prompt and tool definitions at the top, roughly 1,800 tokens of cache-eligible prefix. On Route A, 60% of requests hit the cache. On Route B, the hit rate dropped to 35%.

Route B uses a different cache implementation with a shorter effective TTL. Requests arriving more than a few minutes apart saw cold cache. The retrieval context changes per ticket, and when that context sits inside the cache-eligible prefix, every new document set breaks the prefix match. Route B's load balancer also distributes across more replicas than Route A. Cache is per-replica. Fewer requests land on the same replica within the TTL window.

The pricing page said cache reads were 50% cheaper than uncached input. That discount matters only when cache hits actually happen. At 35% instead of 60%, the effective input cost is higher than the spreadsheet assumed. The team had budgeted for a cache discount that, in production, was applied to barely a third of their traffic.

Cache economics are not on the pricing page. They are in the cache implementation, the TTL policy, the replica topology, and the prefix stability of your prompt. Two providers with identical cache discount rates can produce radically different effective input costs depending on these four variables.

2. Output tokens were longer¶

Route B's model produced longer outputs on average. The trace shows output lengths of 510, 380, and 350 tokens on several requests, well above the 250-token median the team modeled. Output tokens are the expensive side of the bill. On most major API providers, output costs 2-6x more than input per token. A 40% increase in average output length can erase a 47% discount on the per-token output price.

This happens because different models have different verbosity profiles, and the support prompt was tuned for Route A's model. When you move a prompt from one model to another, the output distribution shifts. Sentence structure changes. Explanation depth changes. Hedging language shows up where it was not before. The token rate is cheaper, but more tokens come out. Prompt portability is not free.

3. Retries and repairs are invisible on the pricing page¶

Two of six first attempts in the trace above failed. One on latency SLO (request 4), one on quality (request 3). Both were recovered. The retry consumed tokens. The repair consumed tokens. An eval grader call (not shown in this eight-row slice but present in the full trace) checked the failed outputs and consumed additional tokens.

These recovery requests added roughly 30% to the inference bill for the full trace. On workloads with 5-10% first-attempt failure rates, retries alone add 5-10% to the inference bill. Add repairs and eval grader calls, and the overhead climbs. This cost is invisible if you divide total spend by total requests instead of by accepted work. Retries and repairs inflate both numerator and denominator, and the ratio looks stable.

The pricing comparison between Route A and Route B did not include a line item for "what happens when the answer is wrong."

4. Quality failures have downstream costs¶

Request 3 failed the quality gate. Its answer missed a key constraint from the customer's account. The repair call (request 8) fixed it, but the customer waited an extra 3.2 seconds. If the repair had also failed, the ticket would have escalated to a human agent.

On this workload, approximately 5% of tickets escalate to humans. Each human escalation costs roughly $2 in agent time. On 1,000 tickets per day, 50 escalations cost $100. The daily inference bill is about $14.

Human escalation costs 7x the inference bill. It doesn't appear on any provider dashboard. It's invisible to anyone comparing token prices.

A model with a lower first-attempt pass rate generates more retries, more repairs, and more escalations. Each retry is an inference call that does not appear in the pricing comparison. Each escalation is a cost that does not appear on the inference bill at all. Route B's lower quality rate didn't just cost more tokens; it cost human labor at 50-200x the per-request inference price.

5. The SLO gate changes the denominator¶

The team's SLO says: TTFT under 800ms, E2E under 5 seconds, eval score above 0.85. Of the first attempts across a full day, 75% passed all three gates on the first try. Another 7% were recovered by retries and repairs. The remaining 18% either escalated to humans or were unrecovered failures.

The right denominator is "accepted answers that passed quality, latency, and compliance gates." Not "requests sent to the provider." Not "tickets attempted." Eight hundred twenty accepted answers out of 1,000 tickets. Cost per accepted answer is higher than cost per request.

This is the denominator problem in its purest form. The team measured cost per ticket. They should have measured cost per accepted answer. The difference between those two denominators (1,000 versus 820) accounts for a 22% inflation in per-unit cost, before you even adjust the numerator.

The reveal¶

The procurement spreadsheet did this:

Daily inference cost on Route B:  ~$14.20   
Daily tickets served:              1,000
Cost per ticket:                   $0.014

This calculation divides total inference spend by total tickets. It treats every request as equally valuable. It ignores retries, repairs, eval grader calls, human escalation, and operational overhead. It uses the wrong numerator and the wrong denominator.

Here is the loaded calculation:

Cost component	Daily amount	% of total
Inference (all calls from trace)	$14.20	10.1%
Invoice delta (rounding, timing)	$0.65	0.5%
Eval grader cost	$0.80	0.6%
Human escalation (50 cases x $2)	$100.00	71.1%
Ops overhead allocation	$25.00	17.8%
Total loaded cost	$140.65	100%

And the denominator:

Denominator	Value
Total tickets attempted	1,000
First-attempt pass	750
Recovered by retry/repair	70
Accepted answers	820
Escalated to human	50
Unrecovered failures	130

Naive cost per ticket:              $14.20 / 1,000 = $0.014
Loaded cost per accepted answer:    $140.65 / 820  = $0.172

The loaded cost is 12x the naive cost.

Inference is $14 per day, 10.1% of the loaded cost. The other 90% is the bill people don't see: human escalation, eval grader calls, ops overhead. The 12x gap is the gap between the visible cost and the actual cost.

Optimizing the token price while ignoring the quality-driven escalation rate is optimizing a lever that controls 10% of the outcome. The team switched providers to save 40% on the component that contributes 10% of cost. The result was predictable, in hindsight.

Here is the part the team did not expect: Route A was more expensive per token, but it had a 60% cache hit rate, shorter average outputs, a 91% first-attempt quality pass rate, and fewer retries. The loaded cost per accepted answer on Route A was $0.135. Route B, at $0.172, was 27% more expensive per accepted answer despite being 40-47% cheaper per token.

The spreadsheet picked the wrong route.

The denominator problem¶

The most common mistake in inference economics is dividing by the wrong number.

Here is the taxonomy. Every denominator hides something.

Denominator	What it hides
Per million tokens	Whether those tokens produced useful work
Per request	Whether the request succeeded
Per API call	Retries, repairs, and multi-step workflows
Per user	Variance across user behavior and workload mix
Per GPU-hour	Whether the GPU produced accepted work or wasted cycles

The first row is the cloud console view. You log into your provider dashboard, see cost per million tokens, and think you understand your economics. You do not. You understand the provider's billing meter. The billing meter measures throughput. Your product needs outcomes.

Your observability dashboard shows the second row: average cost per request is $0.014. But "request" includes first attempts, retries, repairs, and eval grader calls, all of which cost money and none of which produced accepted work on their own. A retry that succeeds is a cost of producing the accepted work. A retry that fails is a cost of producing nothing. Both show up as "$0.014 per request" in the average.

Multi-step workflows make the third row dangerous. Agent loops, chain-of-thought with tool use, retrieval-generation-evaluation pipelines generate dozens of API calls per task. Dividing total cost by total API calls produces a number so small that it obscures the actual cost per completed task. An agent that makes 40 API calls to resolve one ticket costs 40x what the per-call metric suggests.

Per-user cost is the product view. It averages away the power users who drive 80% of the inference spend and the light users who drive 5%. When the product team says "we spend $0.03 per user per month," they are right on average and wrong for every user who matters economically.

At the infrastructure level, the fifth row looks healthy: GPU utilization at 65%. But if 30% of those cycles are processing requests that will fail the quality gate and get retried, the effective utilization, measured in accepted work per GPU-hour, is lower than you think.

The denominator depends on the workload. But the structure is always the same:

The accepted work unit: one task that clears the quality gate, the latency SLO, and the compliance constraints — and that didn't need a human in the loop.

For the support team, it is one accepted answer. For a coding agent, it is one merged fix that passes tests and review. For a document extraction pipeline, it is one correctly extracted record set.

Defining the accepted work unit requires answering four questions. What is the task? (Not "an API call"; the smallest unit of work that produces value.) What is the quality gate? (Without a quality gate, every completion counts as accepted and the denominator is meaningless.) What is the latency SLO? (A correct answer that arrives after the deadline may not count.) What are the compliance constraints? (A response generated on the wrong infrastructure is not accepted work regardless of quality.)

If you cannot define what an accepted work unit is for your workload, your cost model is measuring activity, not value. Every metric you report will be accurate. None of them will be useful.

LCPR: Loaded Cost Per Result¶

LCPR — Loaded Cost Per Result. The total cost of producing one accepted work unit, including all inference calls, eval grader costs, human escalation, and operational overhead. Cost per result your system accepted and served to a user.

LCPR = (C_inference + C_eval + C_human + C_ops + delta) / A

Symbol	Unit	Meaning
C_inference	USD	Trace-derived inference cost. All calls: first attempt, retry, repair
C_eval	USD	Eval grader cost. LLM grader calls, rubric checks
C_human	USD	Human review, escalation, and repair labor
C_ops	USD	Amortized operational overhead allocated to this workload
delta	USD	Discrepancy between trace-derived cost and provider invoice
A	count	Accepted work units. Tasks passing quality, latency, and compliance gates

The numerator includes all costs incurred in producing the output, including costs of failed attempts. The denominator includes only accepted output. LCPR is a definition, not an approximation. It is what it costs to produce accepted work.

C_inference is not your provider invoice. It is the cost you can derive from your traces: every API call, priced at the current rate, including calls that failed. Your traces see retries, repairs, and eval grader calls as individual requests with input tokens, output tokens, and cache state. Your invoice sees aggregate tokens for a billing period. The trace gives you attribution (which workload, which customer, which route). The invoice gives you the ground truth charge. Both matter. Neither alone is sufficient.

C_eval is the cost of knowing whether an answer was good. On quality-sensitive workloads, the eval itself is an LLM call. Sometimes a second model used as a judge, sometimes the same model with a grading prompt. On the support workload, eval cost was $0.80 per day. Small relative to human escalation, but real inference spend that produces zero customer-facing output. Without a quality gate, you have no denominator, and without a denominator, you have no LCPR.

C_human is the cost that does not appear on any inference dashboard. When the system fails and a human takes over, that labor cost should be allocated against the workload that generated the failure. On the support workload, this is the dominant cost at 71% of LCPR: $100 per day versus $14.20 for inference. This is not unusual. On quality-sensitive interactive workloads, I consistently see human escalation as the plurality or majority of loaded cost. If you are optimizing token price while human escalation dominates your loaded cost, you are tuning the guitar while the drummer is on fire.

C_ops is on-call engineering time, monitoring and observability tooling, prompt maintenance, eval set curation, model evaluation cycles, deployment pipeline costs. These are real, amortized across workloads, and invisible on any per-request metric unless deliberately allocated. $25 per day is a rough allocation. At some companies it is higher. The point is that it exists and is not zero.

Delta is the reconciliation term. It captures rounding, timing differences between trace timestamps and billing period boundaries, batch pricing adjustments, model alias resolution. Everything that makes your trace-derived cost differ from the invoice. If delta exceeds 5% of C_inference, investigate before trusting your trace-based cost reporting. Common causes: missing traces, model name aliases (the trace says gpt-4.1 but the invoice says gpt-4.1-2025-04-14), batch versus real-time pricing splits, or token count discrepancies between your tokenizer and the provider's.

A is the number that changes everything. It is not requests, not tickets, not API calls. It is accepted work units. Define A before you build the cost model. If A is not defined, the LCPR calculation will produce a number, but the number will not be useful for decisions.

How this formula evolved¶

An earlier version I used in the field guide collapsed eval grader cost, human review cost, and ops overhead into a single bucket called engineering_cost. That bucket hid the lever. On the support workload, 71% of engineering_cost was human escalation. A team that sees engineering_cost = $X reaches for SREs; a team that sees human_escalation = 71% of LCPR reaches for the quality gate. Splitting the bucket makes the dominant lever visible.

The earlier denominator was also underspecified. "Successful" can mean passed quality, met latency, complied with data residency, didn't escalate, or any combination. Two teams at the same company defining it differently produced LCPRs that were 30% apart for similar workloads. The accepted-work-unit definition above (passes quality gate, meets latency SLO, complies with data constraints, no human intervention) is the version I now use.

The delta term is new. Trace-derived cost and provider invoice always differ by 2-5%. The delta isn't a correction; it's a diagnostic. Over 5%, the trace isn't trustworthy for daily cost monitoring, and finding out why is more valuable than the cost figure itself.

Where this breaks¶

The framework is not always worth the effort.

Simple workloads. Single-turn classification, sentiment analysis, embedding generation. Workloads where quality gates are trivial (or absent), retries are rare, and inference cost genuinely dominates. For these workloads, token price is a reasonable proxy and the full LCPR exercise is overhead. If you are classifying text into three categories and the model gets it right 99% of the time, you do not have a denominator problem. You have a straightforward token cost problem.

Batch workloads. When the 50% batch discount is the dominant economic lever and quality is checked downstream in a separate pipeline, the per-request LCPR calculation does not capture the right economics. Batch cost is better modeled as total batch spend divided by total accepted batch output, measured at the batch level rather than the request level.

Rounding-error inference spend. If inference is 0.1% of your operating cost, the loaded-cost exercise is not the binding constraint on your business. Spend your analysis time on whatever is 20% of your operating cost instead.

Very low volume. At 10 requests per day, the ops overhead per request is large and unstable. LCPR is most useful when volume is high enough for the per-unit allocation to be meaningful: typically hundreds of requests per day or more.

Shared infrastructure. When multiple workloads share a dedicated endpoint, allocating C_ops to individual workloads requires an allocation model. Allocation models are always wrong. The question is whether they are useful enough to inform decisions despite being wrong. What Your Workload Actually Costs addresses the allocation problem. The honest answer is: it is hard, and it is worth doing anyway.

What's next¶

This essay names the problem. The trace-to-margin reconciliation method (how to go from raw trace events to trace-derived cost, reconcile trace cost against the provider invoice, and decide whether to trust the delta) is developed in Chapter 25 of Production Inference Economics: A Field Guide.

The trace autopsy turns LCPR from a formula into a measurement.

Production Inference Economics — Part 1 of 3

The Denominator Problem
LCPR Calculator
Workload Costs

Sohail Mohammad — April 2026

Numbers are anonymized and should not be attributed to any specific employer, customer, or deployment.

The Honest Field Guide to Production Inference

2026-04-08T00:00:00-05:00

Part 0: The Cost Illusion¶

I work at Together AI. Together is one of the providers named in the worked examples in this essay, alongside OpenAI, Anthropic, Google, Fireworks, Baseten, DeepInfra, Anyscale, Replicate, Modal, Lambda, RunPod, and CoreWeave. Read every Together-named price, performance number, and customer outcome with that affiliation in mind. The LCPR math is provider-neutral by construction; the choice of which providers appear in worked examples is not. Technical details have been generalized from production work; no proprietary information from any organization is disclosed.

Speculative decoding speeds single-request latency. At production batch sizes, the speedup vanishes.

At batch 12, I measured 0.92x throughput compared to batch 1 (based on the author's prior production experience). The deployment was a high-volume quick-serve voice ordering system: 100K+ daily interactions, 400-600ms inference budget. Under production concurrency, we were paying 8% more compute with spec decode enabled than without.

Published papers report 2-3x decode speedup, all measured on single-request, clean-benchmark conditions. At batch 12-16, different sequences accept different numbers of draft tokens, creating batch-level inefficiency: some sequences are still verifying while others have moved on to generation. The memory bandwidth cost of maintaining KV caches for both draft and target models, plus the verification overhead, eats the theoretical single-request speedup. Add a draft model that's never seen a brand-specific menu item in its training data, and you're underwater.

I spent two weeks on this before I killed it. Tried three draft model configurations. Fine-tuned a 1B speculator on 10K production transcripts. Acceptance rate went from 48% to 58%, still below breakeven. The math is unforgiving: at α=0.55, γ=5, you get 1.94 expected tokens per step for 1.15x the cost. Net negative.

Static spec decode works at batch 1-4. The breakdown starts around batch 8. If your production traffic stays at low concurrency, measure before dismissing. Most teams running open-weights at production scale are past batch 8.

The pattern here matters more than speculative decoding. The framing problem is the gap between advertised cost and true cost.

There's a concept from Elon Musk's early SpaceX days. He couldn't afford rockets, so he calculated the raw material cost (carbon fiber, metal, fuel) and compared it to what the aerospace industry charged for a finished product. The ratio was 50x. He called it the "idiot index": the cost of the finished product divided by the cost of its component materials. If the ratio is high, somewhere in the chain, a massive amount of unnecessary complexity has been layered on top.

AI deployment has its own idiot index. Not in hardware; GPU pricing is competitive and transparent. The idiot index in inference is the gap between the advertised cost of running a model and the true cost of getting a correct answer out of it in production. Pricing pages quote token rates. Production systems pay for retries, schema failures, quality gate rejections (automated checks that validate output format, accuracy, and safety before passing results downstream), engineering time, and an observability bill that grows 30-50% year over year.

This essay closes that gap using math, not benchmarks. Benchmarks lie in predictable ways.

The April 2026 price signal¶

On April 23, 2026, OpenAI launched GPT-5.5 at $5.00/$30.00 per million input/output tokens, double the rates of its predecessor GPT-5.4. Anthropic held Claude Opus 4.7 and Sonnet 4.6 at $5/$25 and $3/$15 respectively. Gemini 2.5 Pro at $1.25/$10 (≤200K context) remains the cheapest frontier option.

The signal is clear: expect price increases at the frontier, not decreases. Serverless open-weights inference (pay-per-token on shared GPU infrastructure, no reserved capacity) is 5-10x cheaper at the total-cost level for non-reasoning workloads and within 5-15% quality on most benchmarks. The price gap between frontier closed APIs and serverless open-weights is widening. The open-weights cost advantage is structural, not temporary.

"5-10x cheaper" is a simplification. The ratio shifts with retry rate, output length, and engineering overhead.

The number that actually matters¶

Token pricing is a component, not a cost. The number you should care about is what I call Loaded Cost Per Request (LCPR):

LCPR = (token_cost + repair_cost + engineering_cost)
       / successful_requests

Where:

Token cost = (input tokens × input rate + output tokens × output rate) × total attempts. Total attempts = original requests × (1 + retry rate). Every retry is a full re-request that burns tokens, so retry cost is not a separate line item; it inflates total token spend directly.
Repair cost = requests that fail quality or schema gates × cost to re-prompt
Engineering cost = monthly hours maintaining the inference stack × hourly rate. This is the most volatile LCPR input. ML engineer turnover (median tenure 18-24 months at AI-native companies) resets institutional knowledge: the replacement re-learns your prompt pipelines, eval harnesses, and provider quirks over 2-4 months, during which effective engineering hours double. A single departure on a 2-person inference team can spike LCPR 15-25% for a quarter. Budget for it by cross-training and documenting runbooks; the LCPR impact of attrition is a hidden tax that no vendor model captures.
Successful requests = total requests × quality gate pass rate

This formula is deliberately simplified. It omits cold start costs (relevant for scale-to-zero serverless), KV cache memory overhead (implicit in token cost for serverless, explicit for dedicated), and observability costs (covered separately in "The Observability Tax" below). A full-stack TCO model would include those. But even this simplified version produces materially different rankings than raw token pricing, which is the point. The full decomposition --- splitting engineering_cost into eval grader cost, human escalation, operational overhead, and a reconciliation delta term --- is developed in the Production Inference Economics series, starting with The Denominator Problem.

Before you calculate, align on definitions. LCPR only produces meaningful comparisons when every team member agrees on what the terms mean:

Token: the provider's tokenization, not yours. Different tokenizers produce different counts for the same text. A 500-word paragraph might be 600 tokens on GPT-5.5 and 650 on Llama 3. Use the provider's token count from the API response, not a local estimate.
Request: one API call, including the full input and output. A multi-turn chat where you send conversation history is one request per turn, not one request per conversation. A retry is a separate request.
Cost: the invoice amount, not the pricing-page estimate. Includes cache discounts actually applied, batch pricing tiers actually hit, and any overages or committed-use adjustments.
Success: a request that passes your quality gate and reaches the end user. A request that returns 200 OK but fails schema validation is not a success. A request that the user ignores because it took 8 seconds is a success by this definition but a failure by your product's.

Misalignment on these definitions is the most common source of "our numbers don't match the vendor's" disputes.

Three ways LCPR lies to you. Even when calculated correctly, aggregate LCPR can mislead:

Engineering cost dominance at low volume. At 50K requests/month, 8 engineering hours at $100/hr is $800, which adds $0.016 per request. That's more than the token cost on cheap providers. LCPR makes open-weights look expensive at low volume not because the tokens are expensive, but because the fixed engineering overhead hasn't been amortized. The fix: segment LCPR by workload and don't aggregate across volumes.
Migration double-paying. During a migration (typically 6-10 weeks), you're running both old and new providers. Your LCPR temporarily spikes because engineering_cost doubles (maintaining two stacks) while successful_requests hasn't fully shifted. Teams that measure LCPR monthly sometimes panic and cancel the migration during this spike. Plan for it: model the transition period separately.
Aggregate LCPR hides workload variation. A blended LCPR of $0.008 across all workloads can hide a $0.003 chat workload and a $0.025 RAG extraction workload. The chat workload might be optimally placed on open-weights, while the RAG workload needs frontier quality. Always segment LCPR by workload before making migration decisions.

Almost nobody calculates LCPR, which means almost nobody has accurate cost comparisons across providers.

Segment LCPR before you optimize. Failure mode #3 above deserves emphasis. Compute LCPR along three dimensions:

By model: GPT-5.5 for reasoning, DeepSeek V3 for chat, Llama 70B for classification. Each has different retry rates, quality gates, and token costs.
By use case: customer-facing chat has different latency and quality requirements than internal batch summarization. A single "inference cost" number conflates workloads with 10x different LCPR.
By user segment: enterprise customers generating 10x the tokens of self-serve users may justify dedicated infrastructure, while self-serve stays on serverless. Revenue-weighted LCPR reveals which segments fund which.

The calculator supports this: run each workload profile separately, then weight by volume. A team that optimizes blended LCPR will underinvest in their most expensive workload and overinvest in their cheapest.

Every LCPR input is an assumption until measured. Before trusting a calculator output, tag each input with its confidence level:

Assumption	Default value	Confidence	What changes it
Retry rate	3%	Assumed	Measure from production logs; spikes during model migrations
Quality gate pass rate	95%	Assumed	Varies 70-99% by model and task; measure from eval suite
Avg output tokens	400	Assumed	Highly variable; measure P50 and P95 separately
Engineering hours/month	8	Estimated	Track actual hours for 2 months; often 2-3x initial estimate
GPU utilization	40%	Industry estimate	Measure from GPU metrics; varies 25-65% by workload pattern
Repair cost per failure	$0.002	Assumed	Measure actual re-prompt cost including latency penalty

The calculator defaults are reasonable starting points, not ground truth. Replace each "Assumed" with "Measured from logs" as you instrument your stack. An LCPR model built on six assumptions is a hypothesis; one built on six measurements is a budget.

Worked example: the numbers change¶

Consider a mid-scale SaaS workload: 500,000 requests per month, 800 input tokens and 400 output tokens per request, 3% retry rate, 95% quality gate pass rate, and 8 engineering hours per month at $100/hour to keep things running.

Here's what the LCPR looks like across deployment modes, using May 2026 public pricing:

Provider	Raw $/request	LCPR	Monthly cost	Overhead ratio
OpenAI GPT-5.5	$0.0160	$0.0191	$9,090	1.20x
Lambda H100 (dedicated, 40% util)	$0.0057	$0.0078	$3,723	1.36x
Together AI DeepSeek V3 (serverless)	$0.0012	$0.0030	$1,447	2.63x
Fireworks AI Llama 70B (serverless)	$0.0011	$0.0030	$1,406	2.74x
DeepInfra GPT-OSS-120B (serverless)	$0.0001	$0.0019	$905	17.8x

The table shows three things.

The ranking is stable but the magnitude isn't. GPT-5.5's raw token cost is $0.016 per request and its LCPR is $0.019, 20% higher, the smallest overhead in the table. For a managed API with near-zero engineering burden, that's a thin tax. DeepInfra at $0.0001 per request (asymmetric pricing: $0.039/$0.19 per million input/output) has an LCPR of $0.0019, a 17.8x overhead ratio. DeepInfra is still the cheapest total cost in the table; the high ratio is the inverse signal. Tokens are cheap enough that fixed costs (engineering, retries, repair) dominate the per-request LCPR. Cheap providers need volume to amortize.

Cost ratios compress at the LCPR level. GPT-5.5 is 13.8x more expensive than Together on raw token cost; at LCPR, it's 6.3x. Engineering overhead, retries, and repair are roughly provider-independent, so they compress every ratio. Any comparison that doesn't include these costs is overstating the savings from switching providers.

Dedicated GPU is not the cheapest option at this volume. A Lambda H100 at $3.99/hr with 40% realistic utilization produces an LCPR of $0.0078, more expensive than both serverless open-weights options. The GPU costs $2,873/month whether you use it. At 500K requests generating 200M output tokens per month, you're paying for capacity you don't fill. Dedicated wins at scale; the crossover is higher than most teams expect (see "Break-even math for dedicated GPU" below).

The numbers change when reliability changes¶

The worked example above assumes a 3% retry rate and 95% quality gate. What happens when those shift?

At 20% retry rate (not uncommon during model migrations or prompt changes), GPT-5.5's LCPR rises to $0.0220. Together's rises to $0.0033. The ratio increases from 6.3x to 6.8x; retries hurt expensive providers more because each retry costs more tokens.

At 70% quality gate pass rate (a model that frequently fails structured output validation), GPT-5.5's LCPR jumps to $0.0267. Together's jumps to $0.0048. The ratio compresses to 5.5x because repair and engineering costs are provider-independent and start to dominate.

The I/O ratio matters more than most teams realize. The GPT-5.5 vs Together cost advantage depends heavily on the output-to-input ratio. GPT-5.5 prices output at 6x its input rate ($30 vs $5 per million), while Together's output premium is smaller at 2.8x ($1.70 vs $0.60 per million). At 800 input tokens with varying output:

Output tokens	Raw ratio	LCPR ratio
100 (classification)	10.8x	3.8x
400 (essay default)	13.8x	6.3x
1,000 (long-form)	15.6x	9.3x
1,500 (code gen)	16.2x	10.8x

Output-heavy workloads (code generation, long-form content) see the largest savings from migration. Input-heavy workloads (classification, embedding prep) see the smallest, and may not justify migration at all if the volume is low.

The pricing page is the token rate. LCPR is the bill. The gap is the cost illusion, and it widens with retry rate, output length, engineering time, I/O ratio, and observability cost. No vendor has any incentive to help you measure those.

The observability tax¶

One cost deserves special mention because it's frequently the largest hidden expense teams discover too late: observability.

byteiota's 2026 analysis found a median Datadog bill of $123K/year for mid-market companies, growing 30-50% year over year. Teams adding LLM monitoring report 40-200% bill increases because GenAI semantic-convention spans get billed as custom metrics. AI workloads generate 10-50x more telemetry than traditional services.

If you're modeling a migration from closed APIs to self-managed inference, budget 2-4x your Year-1 observability estimate. The Datadog bill is not a rounding error; it rivals your GPU spend at small to medium scale.

What's in the rest of this essay¶

Five framings structure what follows. Part 1 is about deciding whether to leave the API for a given workload: three gates (volume, specialization, ownership) and a migration-complexity score for the "how hard" question. Part 2 is the multi-source architecture: four sourcing patterns and the routing layer between them. Part 3 is the inference stack itself, layer by layer, with a buy-vs-build call per layer. Part 4 is vendor evaluation: seven gates and how to verify each. Part 5 is the staged playbook: entry and exit thresholds for each operational tier from Stage 0 (closed API only) through Stage 3 (build-side). Each section has worked numbers from the LCPR calculator, named customer references with links, and the cases where the framework breaks.

The frameworks are decision tools, not analyses. If your situation doesn't fit them, the data — your LCPR, your latency budget, your compliance constraints — is the answer, not the framework.

Part 1: When to Leave the API¶

Whether open-weights inference is cheaper is settled (Part 0): ~6x at the loaded level, 10-100x on raw tokens. When the savings justify the migration cost is not.

Most teams get this wrong in one of two directions. They either stay on closed APIs past the point where they're hemorrhaging money, or they migrate too early, spend 10 engineer-weeks rebuilding prompt pipelines, and discover the savings don't cover the engineering bill for two years.

Three gates follow. Volume is the default; specialization and ownership each independently override. If none of the three pass for a given workload, stay put.

Gate 1: The Volume Gate¶

Migration has a fixed cost. The range varies by complexity:

A straightforward API swap (same model family, OpenAI-compatible endpoints) takes 2-4 engineer-weeks.
A standard migration requiring prompt adaptation and quality gate validation takes 6-10 engineer-weeks, plus another 4-8 weeks of optimization to reach cost parity.
Complex migrations involving fine-tuning, custom structured output validation, or domain-specific evaluation harnesses can extend to 12-20 weeks.

At a blended rate of $150/hour for a senior ML engineer, 8 engineer-weeks costs $48,000. Does your monthly savings exceed the amortized migration cost over a reasonable payback period?

Here's the worked example. A B2B SaaS company running 800,000 requests per month on GPT-5.5, with 1,000 input tokens and 500 output tokens per request, a 5% retry rate, 92% quality gate pass rate, and 12 engineering hours per month to maintain the stack.

Provider	LCPR	Monthly cost
OpenAI GPT-5.5	$0.0246	$18,128
Lambda H100 (dedicated, 40% util)	$0.0057	$4,201
Together AI DeepSeek V3 (serverless)	$0.0035	$2,546
Fireworks AI Llama 70B (serverless)	$0.0033	$2,462

Switching from GPT-5.5 to Together DeepSeek V3 saves $15,582 per month, or $186,984 per year. Against a $48,000 migration cost, the payback period is 3.1 months. That's a clear pass.

But notice what happens at lower volume. In the Part 0 worked example (500K requests/month, simpler workload profile), GPT-5.5's monthly cost was $9,090. Against Together at $1,447, the savings are $7,643/month. Still a 6.3-month payback, which is acceptable, but you're now sensitive to migration overruns. If the migration takes 12 weeks instead of 8, or if the quality gate drops from 95% to 88% during the transition and you spend two months tuning prompts, the payback stretches past a year.

The Volume Gate threshold: if your monthly closed-API spend is below $10,000, the migration economics are marginal. Between $10,000 and $50,000, the economics work but execution risk matters; you need a team that's done this before. Above $50,000, the savings are large enough to absorb migration friction. These boundaries are rough; run the LCPR calculator against your actual workload to get precise numbers.

Gate 2: The Specialization Gate¶

Volume isn't the only reason to migrate. Sometimes the workload requires something closed APIs can't provide.

Fine-tuned models. If your quality evaluation shows that a fine-tuned 8B or 70B model matches frontier quality for your specific domain, the cost advantage is enormous. Cresta runs thousands of LoRA adapters (Low-Rank Adaptation: small trainable weight matrices attached to a frozen base model, enabling efficient fine-tuning) for per-domain contact center agents on Fireworks Multi-LoRA at $0.20/M tokens, a 27.4x LCPR advantage over GPT-5.5 at 3M requests/month. Even accounting for the engineering cost of training and maintaining the fine-tune pipeline, the payback is measured in weeks, not months.

Latency SLOs. Shared APIs under load produce P99 latency spikes from ~300ms to 2-4 seconds on 70B-class models. For agent pipelines with 5+ chained calls, that compounds: a 2-second P99 across 5 calls is a 10-second worst case. Dedicated inference lets you control batch size and KV cache budget.

A bulk-extraction workload illustrates the specialization-gate logic. A mid-stage legal-tech company was processing roughly 2.1 million historical contract documents (average 8,400 tokens each, mostly English with a substantial Latin-American Spanish-language corpus). The pipeline ran a Llama-class 70B for per-document field extraction (parties, effective date, governing law, termination clauses) with a JSON-validated controlled-vocabulary schema of ~80 jurisdictions.

The backfill projected at $11-13K over six weeks. By week 9 it had spent $34K and was 58% complete. The first three hypotheses (eval-pass cost, long-document outliers, wrong-model selection) all came back negative. The fourth measurement found request volume had doubled in week 4 with no traffic-mix change: 47% of traffic was now tagged retry, almost all of it failing on a single field, governing_law, where the model was emitting jurisdiction strings inconsistently for Spanish-language documents. "Mexico" / "México" / "United Mexican States" / "Mexican Federal" all rejected. Each retry re-paid the full input cost (~$0.067) for a ~$0.001 fix.

The fix was a deterministic post-process normalization layer (string-to-canonical-jurisdiction lookup) before JSON validation. Retries dropped from 47% to 4.1%. The remaining backfill completed in 11 days at ~$4,800.

The specialization-gate lesson is the side-finding, not the headline. The team had been about to migrate to a more capable frontier model, passing both the Volume Gate ($34K spend) and what looked like a Specialization Gate (low extraction quality). The actual problem was upstream of the model: a controlled-vocabulary schema with no normalization layer was wasting 47% of throughput on retries. Every controlled-vocabulary structured-output schema needs deterministic normalization paired with the validation gate. The Specialization Gate isn't just about model capability; it's about whether your pipeline's deterministic layer can absorb model variance cheaply.

Cursor's Fast Apply feature is the parallel point on a different vendor: fine-tuned Llama-3-70B at ~1,000 tokens/sec via Fireworks speculative decoding (source: Cursor + Fireworks case study). Sub-500ms latency on chained calls that shared APIs cannot guarantee under load. Domain-tuned draft models with per-workload acceptance-rate optimization; generic spec-decode draft models fall off on domain-specific vocabulary, the technical reason the deployment from Part 0 didn't work.

Custom architectures. Some workloads require model modifications that closed APIs don't support: constrained decoding, custom sampling strategies, domain-specific tokenizers, or inference-time interventions like activation steering. If you need to modify the model's forward pass, you need dedicated inference.

The Specialization Gate threshold: if any of the following are true, migration passes this gate regardless of volume: (a) a fine-tuned model matches frontier quality for your task, (b) you have a hard P99 latency SLO under 500ms on chained calls, or (c) you need model-level modifications.

Quality degradation risk. Passing the Specialization Gate requires eval infrastructure you may not have yet. Before committing to migration, run your target open-weights model against your production eval suite (not a public benchmark) and compare quality gate pass rates. A model that scores 92% on MMLU but 78% on your domain-specific extraction tasks will produce a higher LCPR than the frontier API it replaces: more retries, more repairs, more engineering time debugging failures.

Budget 2-4 weeks of eval development before the migration clock starts. If you don't have a production eval suite, building one is prerequisite work, not migration work.

Gate 3: The Ownership Gate¶

The third gate is non-economic: compliance, data residency, and vendor dependency.

Data residency. If your workload processes EU PII, Schrems II makes US-hosted inference legally fraught (the EU-US Data Privacy Framework, adopted July 2023, restored a legal pathway to DPF-certified US companies, but challenges are pending). The viable EU-resident options as of May 2026 are Nebius (Finland, France), Scaleway, Mistral La Plateforme, and OVH. None of the major closed APIs (OpenAI, Anthropic) offer guaranteed EU-only inference. Anthropic's inference_geo=US parameter exists because their default routing isn't geo-constrained. Verify data residency guarantees in your contract. Vendor documentation and API parameters are not legal commitments.

Zero data retention. For healthcare and financial workloads, the default storage behavior matters more than the compliance certification. Baseten stores nothing by default. Fireworks retains for 30 days on the Response API unless store=false. Together stores by default unless you disable it. OpenAI's fine-tuning retains training data. If your legal team requires contractual zero-retention by default, this narrows your vendor set.

Vendor concentration risk. Anthropic outages in 2024-2025 produced real revenue loss for single-sourced teams. If a single provider outage costs more than 1% of monthly revenue, you are underinvested in multi-sourcing. Multi-sourcing across closed APIs still leaves you dependent on two or three vendors' pricing decisions. Open-weights on serverless gives you model portability: if your primary serverless provider has an outage, the same DeepSeek V3 weights are available across Fireworks, DeepInfra, Anyscale, Replicate, Hyperbolic, Together, and most other open-model hosts. Failover is a model-string change in your gateway.

The Ownership Gate threshold: if regulatory requirements force specific data handling, or if vendor concentration risk exceeds your tolerance, migration passes this gate regardless of volume.

When NOT to migrate¶

Not every workload should move. Three patterns where closed APIs remain the right answer:

Small-token, high-volume workloads where Mini-class models win. GPT-5.4 Mini at $0.75/$4.50 per million tokens trades off against Together DeepSeek V3.1 at $0.60/$1.70 in a workload-dependent way. On a voice classification task at 300 input tokens and 150 output tokens at 3M requests/month, the input-heavy ratio compresses the gap considerably; on output-heavy tasks the open-weights advantage widens. Open-weights isn't always the right call; the math depends on the input/output ratio and which model tier you're comparing against. Run the calculator before assuming.

Reasoning-heavy workloads where frontier quality matters. If your task requires chain-of-thought reasoning, mathematical proof, or complex code generation where GPT-5.5 or Claude Opus 4.7 measurably outperform open-weights alternatives, the quality delta means more failed requests, more retries, and a higher LCPR on the open-weights side. Quality gate pass rate is the most powerful variable in the LCPR formula. A 10-point drop from 95% to 85% increases LCPR by 13% on expensive providers and up to 19% on cheap ones where fixed costs dominate.

Prototyping and early product. At less than $10,000 per month, the engineering overhead of managing even a serverless open-weights deployment (prompt migration, model evaluation, gateway configuration) exceeds the savings. Use a closed API, ship the product, and revisit when you hit the Volume Gate.

When to stay on closed APIs permanently¶

The three cases above describe workloads where migration doesn't make sense yet. Some workloads should stay on closed APIs indefinitely:

Reasoning-dominated pipelines with no open-weights equivalent. If your core product depends on frontier reasoning (multi-step mathematical proof, complex agentic workflows with 10+ tool calls, code generation where GPT-5.5 or Claude Opus 4.7 measurably outperform the best open-weights models on your eval suite), the quality gap may never close. Open-weights models improve, but so do frontier models. If your eval pass rate on the best open model is 15+ points below frontier, plan to stay on closed APIs and optimize via caching and prompt engineering instead of migration.

Rapid model rotation. Teams that switch models every 2-3 months to chase the latest frontier release (GPT-5.5 → Claude Opus 4.7 → Gemini 2.5 Pro) get more value from closed API flexibility than from open-weights cost savings. Migration engineering amortizes poorly if you're rotating providers anyway. The LCPR math still applies; just run it with engineering_hours_per_month reflecting your actual switching costs.

No ML engineering capacity. If your team has zero ML engineers and no plan to hire, the operational overhead of even managed serverless open-weights (prompt migration, model evaluation, quality gate tuning) exceeds the cost savings for most workloads under $50K/month. Closed APIs abstract this away. The calculation changes if you hire: one senior ML engineer at $200K/year (fully loaded) pays for itself in migration savings above ~$30K/month in inference spend, assuming they can execute a standard migration in 8 weeks.

The Gemini question¶

Gemini 2.5 Pro at $1.25/$10 looks like frontier quality at near-open-weights pricing. Why migrate to open-weights when Gemini exists?

The answer depends on the workload shape.

Output pricing is still asymmetric. Gemini's output rate is $10/M, roughly 6x Together's $1.70/M. For a RAG pipeline at 4,000 input / 600 output tokens and 800K requests/month, Gemini's LCPR is $0.0131 versus Together's $0.0049, a 2.7x gap. For output-heavy workloads (code generation at 800/2,000 tokens), Gemini's LCPR rises to $0.0239 versus Together's $0.0054, a 4.5x gap. Gemini is near-open-weights on input but not on output.

No customization. You cannot fine-tune Gemini, run custom speculators, control quantization, or modify the inference pipeline. Cursor (source: Fireworks + Cursor case study) and Decagon (source: Together AI + Decagon case study) need these capabilities, which is why they run dedicated open-weights infrastructure.

Data sovereignty. Gemini runs on Google infrastructure with no self-hosting option and no zero-retention guarantee. For regulated workloads, this narrows the viable use cases.

Vendor lock-in. If Google changes pricing (as OpenAI did with GPT-5.5), there's no portability. Open weights move between Together, Fireworks, DeepInfra, and self-hosted deployments.

Where Gemini wins: input-heavy classification and analysis workloads where output is short, customization is unnecessary, and Google's data handling is acceptable. At 2,000 input / 100 output tokens, the Gemini-to-Together LCPR ratio compresses to 1.9x, still a modest gap. The migration gates still applies: if the savings don't cover migration cost, stay on Gemini.

The break-even math for dedicated GPU¶

The three gates above address whether to migrate from closed APIs. A separate question is when to move from serverless open-weights to dedicated GPU. This is a volume calculation with a specific crossover point.

A Lambda H100 at $3.99/hr costs $2,873/month whether you use it or not. Running a 70B FP8 model with vLLM continuous batching, it sustains approximately 1,500 output tokens/sec at high batch utilization. At full utilization, that's 129.6M output tokens per day.

(This break-even is calculated on output tokens because typical chat workloads are bottlenecked by autoregressive decode. For long-context workloads with >8K input tokens or short-output tasks like classification, prefill dominates and the economics shift. Dedicated becomes relatively more attractive because you're paying for compute you'd pay for anyway.)

Against Together's serverless output rate of $1.70/M, break-even is 56.3M tokens/day at full utilization. Against Fireworks at $0.90/M, it's 106.4M tokens/day.

But production workloads don't saturate. Real utilization on dedicated inference runs 30-50%, with 40% as the midpoint. The gap comes from decode-phase memory bandwidth limits, variable batch sizes across time-of-day, cold start periods after deployments, and the fact that real traffic doesn't produce constant request rates. Cast AI's 2026 State of Kubernetes Optimization Report finds 49% GPU utilization on a 136-H200 cluster is "the ceiling, not the floor"; that matches the 25-45% range I've measured across production dedicated deployments. A team sustaining 50%+ on dedicated GPUs is doing better than most.

At 40% real utilization, break-even against Together rises to 140.8M tokens/day. Against Fireworks, it's 266.0M tokens/day.

For context: 140.8M output tokens per day is approximately 4.7 million requests at 30 tokens per response, or 469,000 requests at 300 tokens per response. Most teams don't reach this volume on a single model endpoint. If your utilization consistently stays below 40%, the correct move is back to serverless, or consolidating workloads onto the GPU via Multi-LoRA serving.

The middle path: managed dedicated¶

Between serverless and self-managed GPU sits a third option: managed dedicated endpoints. You reserve GPU capacity (hourly or monthly billing), but the provider handles runtime optimization, quantization, autoscaling, and kernel selection.

Together AI's pricing illustrates an important distinction: managed dedicated inference endpoints run at $1.76-$2.39/hr for H100 capacity ($4-$5.50/hr for B200), while raw GPU cluster rates run higher ($2.99-$3.99/hr for H100 on-demand on the gpu-clusters page). The lower range is the managed model-serving SKU; the higher range is raw GPU rental on InfiniBand-connected clusters. Other vendors (Baseten, Fireworks, Replicate, Modal) offer similar managed model-serving tiers separate from raw GPU pricing; verify which SKU you're quoting before comparing. Crossover from serverless to managed dedicated typically sits around 130,000 output tokens per minute of sustained traffic. Below that rate, serverless is cheaper.

The operational difference is significant. Self-managed dedicated (Lambda H100 at $3.99/hr) requires you to run vLLM, handle OOMs, tune batch sizes, and manage failover. Managed dedicated (Baseten, Fireworks, Modal, Replicate, Together) handles all of that; you get an endpoint URL with an SLA. The hourly rate is higher, but the loaded cost (factoring in engineering time) is often lower. Use the LCPR calculator to compare: if your engineering_hours_per_month for self-managed exceeds 40 hours at $150/hr, managed dedicated wins on total cost up to approximately $50K/month in GPU spend.

Quick reference: the math behind dedicated GPU economics¶

Four formulas underpin the dedicated break-even analysis.

Prefill vs decode bottleneck. For short inputs (<2K tokens) with long outputs, decode (autoregressive token generation) dominates GPU time; throughput scales with memory bandwidth. For long inputs (>8K tokens) with short outputs, prefill (processing the full input in parallel) dominates; throughput scales with compute FLOPS. Most chat workloads are decode-bound; most RAG extraction workloads are prefill-bound. The break-even math assumes decode-bound; for prefill-bound workloads, adjust with the prefill_efficiency parameter in the calculator.

KV cache memory sizing. kv_bytes = 2 × layers × heads × head_dim × seq_len × precision_bytes × batch_size. A Llama 70B model (80 layers, 64 heads, 128 head_dim) at FP16 with batch 32 and 4K context uses ~42 GB of KV cache, over half the H100's 80 GB. This is why batch size has a hard ceiling on dedicated GPUs, and why longer contexts reduce maximum batch size (and therefore throughput).

Little's Law for concurrency. concurrent_requests = arrival_rate × avg_latency. If you're processing 100 requests/second with an average latency of 2 seconds, you have ~200 concurrent requests in flight. This determines the batch size your system sees, which determines throughput and per-request latency.

Utilization step economics. GPU cost is a step function, not a smooth curve. One H100 costs $3.99/hr whether it's at 20% or 95% utilization. Adding a second GPU doubles your cost to $7.98/hr but also doubles capacity. The break-even calculation assumes a single GPU; at multi-GPU scale, the economics are more forgiving because underutilization on one GPU can be offset by consolidating workloads.

The decision flowchart¶

In practice, the three gates reduce to a sequence:

Check the Specialization Gate first. Do you need fine-tuned models, hard latency SLOs, or model-level modifications? If yes, migrate. The technical requirement overrides volume economics.
Check the Ownership Gate. Do compliance, data residency, or vendor risk requirements force the move? If yes, migrate. The regulatory requirement overrides volume economics.
Check the Volume Gate. Is your monthly closed-API spend above $10K? If yes, migrate. The savings justify the engineering cost.
If none of the three gates pass, stay on closed APIs for that workload.
Once you've decided to migrate, start with serverless open-weights. Move to managed dedicated when a single workload exceeds ~130K output tokens/minute sustained (roughly $3-5K/month on a single model). Move to self-managed dedicated only when GPU spend exceeds $50K/month AND you have inference engineers on staff.

The most common mistake is skipping straight to dedicated GPU. Serverless open-weights is the right default for the vast majority of workloads that have passed the migration gates. Dedicated is for the outliers; you'll know when you're an outlier because the serverless bill will tell you.

The lead time reality¶

Even after passing the gates, the timeline from "decision to migrate" to "realizing savings" is longer than most teams budget for:

Weeks 1-4: Eval suite development (if you don't have one), model selection, prompt adaptation. No savings yet; you're spending engineering time.
Weeks 5-8: Shadow mode deployment (running both providers in parallel), quality gate validation, latency benchmarking. Cost increases because you're paying for both providers.
Weeks 9-12: Traffic cutover (typically 10% → 50% → 100% over 3-4 weeks), monitoring for regressions, tuning. Savings begin ramping.
Weeks 13-16: Full migration, old provider deprecated, optimization pass. Full savings realized.

For a standard migration, plan for 3-4 months from decision to full savings. For complex migrations (fine-tuning, multi-model, compliance), plan for 6-12 months. The payback calculation in the Volume Gate (e.g., "3.2 months") is the payback after migration completes, not from the decision date. Add 3-4 months of migration time to the total.

This timeline matters for budget planning. If you start a migration in Q1, don't promise Q1 savings. Promise Q2 savings with a Q3 steady state.

Assessing migration complexity¶

The gates tell you whether to migrate; the migration-complexity score tells you how hard. A single-model chatbot swap and an eight-model enterprise migration with compliance requirements are both "migrations"; the first takes 4 weeks and the second takes 20.

Migration complexity is multi-factor, not one-dimensional. Six factors combine to determine timeline, cost, and the approach you should take. Each factor scores Low (1), Medium (2), or High (3):

Factor	Low (1)	Medium (2)	High (3)
Workload count	1-2 models, single use case	3-5 models, 2-3 use cases	6+ models, mixed latency/throughput requirements
Prompt portability	Simple prompts, no structured output	Moderate prompt engineering, JSON mode	Complex chains, tool use, function calling, custom schemas
Quality infrastructure	No formal evals	Basic eval suite (<50 test cases)	Comprehensive evals (500+ cases), regression testing, human-in-loop
Latency sensitivity	Batch/async (>5s acceptable)	Interactive (<2s P95 required)	Real-time (<500ms P95, voice, streaming)
Team inference maturity	No ML infra expertise	1-2 engineers with vLLM/serving experience	Dedicated inference team (3+)
Integration depth	Single API call, stateless	SDK integration, session state, caching logic	Multi-system (gateway, observability, billing, compliance)

Sum your scores. The total determines your tier:

6-9 (Simple): 4-6 weeks, 1-2 engineers, serverless-first. Self-service using this guide.
10-14 (Standard): 8-12 weeks, 2-3 engineers, evaluation framework required before cutover. Consider managed dedicated for primary workload.
15-18 (Complex): 12-20 weeks, dedicated team or vendor-side customer engineering partnership. Phased approach with parallel-run validation mandatory.

Worked examples:

Simple (Score: 7). SaaS startup, single chatbot on GPT-4o, 2M requests/month, no evals, interactive latency (<2s P95), 1 backend engineer. Migration: swap API endpoint, run A/B test for 1 week. Timeline: 4 weeks including testing. (Factors: workload 1, prompts 1, evals 1, latency 2, team 1, integration 1.)

Standard (Score: 12). Mid-market company, 3 models (chat, classification, embeddings), moderate prompts with JSON mode, basic eval suite, <2s latency requirement, 2 engineers with serving experience, gateway and observability already in place. Migration: model-by-model over 10 weeks with quality gates between each. (Factors: all 2s.)

Complex (Score: 16). Enterprise, 8 models across 4 use cases, complex tool-use chains, comprehensive eval harness, sub-500ms voice AI requirement, no dedicated inference team, deep integration with billing/compliance/multi-region. Migration: 16+ weeks with vendor-side customer-engineering partnership or equivalent expert engagement. Phased: one workload at a time with 2-week parallel-run per workload. (Factors: workload 3, prompts 3, evals 3, latency 3, team 1, integration 3.)

The engineering hours reality. The hidden cost in migration isn't tokens. It's engineering time. The deployment mode you choose determines your ongoing maintenance burden:

Deployment Mode	Setup (one-time)	Ongoing (monthly)	Source
Serverless open-weights	2-8 hrs/workload	2-5 hrs (monitoring, prompt updates)	from provider onboarding docs
Managed dedicated	8-20 hrs (SLA design, testing)	5-10 hrs (capacity reviews, model updates)	from Decagon/Cursor case patterns
Self-managed dedicated	40-80 hrs (runtime setup, tuning)	30-60 hrs (OOMs, scaling, kernel updates, on-call)	from Lambda/CoreWeave community data

At a fully loaded engineer cost of $100-$200/hr (the calculator defaults to $100/hr; adjust in the sidebar), self-managed dedicated adds $3,000-$12,000/month in engineering overhead alone. That overhead is invisible in token pricing comparisons but dominates the LCPR calculation. A budget holder comparing "$3.99/hr GPU" to "$1.76/hr managed dedicated" is comparing the wrong numbers. The full picture requires engineering hours.

Use the Migration Readiness tab in the LCPR calculator to score your factors interactively and see the payback calculation for your specific workload profile.

Part 2: The Multi-Source Architecture¶

Single-provider was the 2024 default. Add-a-fallback was 2025. Multi-source is 2026 among AI-natives with meaningful spend, and the how is now the open question, not the whether.

This isn't a theoretical recommendation. Every company I've listed below runs multiple inference providers in production, and each has a specific architectural reason for doing so.

The four patterns¶

Multi-source inference architectures fall into four patterns. Most production deployments use two or three of these simultaneously.

Pattern 1: Workload-Segmented. Different workloads go to different providers. This is the most common pattern.

Cursor is the canonical example. Fast Apply (their deterministic code-edit feature) runs on a fine-tuned Llama-3-70B at ~1,000 tokens/sec through Fireworks speculative decoding (source: Cursor + Fireworks case study). Sualeh Asif, Cursor co-founder: "We leverage speculative decoding for our custom models deployed on Fireworks.ai, which power the Fast Apply and Cursor Tab features. Thanks to speculative decoding, we saw up to a 2x reduction in generation latency."

Cursor's 2x speedup is for deterministic code-edit operations with predictable output structure. That's a different workload shape than the high-concurrency, variable-output scenario described in Part 0 where naive spec decode is net negative. Adaptive speculative decoding (FireOptimizer, ATLAS) addresses the batch-size problem by selecting draft strategies per-request.

Composer 2 (Cursor's agentic coding model) trains and serves through Fireworks with weight syncs every training step via delta-compressed S3 uploads. Chat features use Claude Sonnet and Opus directly.

Cursor's production deployment spans multiple providers: Fireworks for speculative decoding on Fast Apply, Anthropic for frontier chat, and Together AI for Blackwell GPU inference with a quantization pipeline that moves new model weights from candidate to test endpoint within days (source: Together AI + Cursor case study). Each provider wins on a different constraint: throughput, reasoning quality, or hardware access.

Notion follows the same pattern: Fireworks for latency-critical features using fine-tuned models ("we reduced latency from about 2 seconds to 350 milliseconds," Sarah Sachs, Head of AI Engineering, source: Fireworks case study), Baseten for other workloads, Replicate and Modal for experimental and burst-scale inference paths, and Anthropic with prompt caching for features that benefit from frontier reasoning. Zomato's AI chatbot Zia, handling 1,000+ messages per minute on optimized Llama models through Together, achieved 2x CSAT improvement and 75% reduction in response time (source: Together AI case study).

Pattern 2: Capability-Arbitrage. The same logical workload routes to different providers based on the specific capability needed for each request. This requires more sophisticated routing but captures large cost savings.

The Multi-LoRA pattern is the clearest example. Cresta runs thousands of LoRA adapters on Fireworks Multi-LoRA for per-domain contact-center fine-tunes (documented 100x cost reduction versus GPT-4 on the fine-tuneable subset of traffic at deployment time, source: Fireworks case study, Dec 2023 deployment), with escalation to frontier models for the complex residual. At $0.20/M tokens for a Llama 8B base with adapters versus GPT-5.5 at $5/$30, the LCPR advantage is roughly 27x at 3M requests/month (derived calculation against current pricing). Together has shipped comparable multi-adapter inference at similar pricing tiers since late 2025; Baseten and Replicate support LoRA-adapter serving with different pricing models. The architectural pattern (cheap-fine-tuned-base + escalation-to-frontier) is the case for capability-arbitrage; the vendor choice is downstream of who supports your base model and adapter format.

This is capability-arbitrage: use the cheapest model that can handle each request, and escalate only when necessary. The difficulty is building the routing logic to decide when to escalate. Most teams start with simple heuristics (input length, task type, confidence score) and add complexity only when the data justifies it.

Pattern 3: Primary-Fallback. A primary provider handles all traffic, with automatic failover to a secondary provider during outages or degradation. This is the minimum viable multi-source architecture.

The implementation is straightforward: an AI gateway (LiteLLM, Helicone, Portkey, or Bifrost) that routes to Provider A by default, detects failures (5xx responses, latency spikes above threshold, rate limit errors), and reroutes to Provider B. The same model family is available on both sides: DeepSeek V3 on Together with Fireworks as fallback, or Claude Sonnet via Anthropic with Bedrock as fallback.

This pattern doesn't save money. It costs slightly more because the fallback provider may have different pricing. Its value is availability: Anthropic outages in 2024-2025 demonstrated that single-source dependency on any provider, even a reliable one, is a business risk. If a single-provider outage costs more than 1% of monthly revenue, Primary-Fallback is table stakes.

Fallback design matters more than fallback existence. Two common failure modes:

Your fallback provider uses a different response format than your primary, so failover produces schema validation errors in your application layer: an outage that looks like recovery.
Your fallback hasn't been tested under load in months, so when you actually need it, you discover rate limits, stale API keys, or model deprecation.

Fix both: ensure your fallback produces identical response schemas (use your gateway's response normalization, or constrain both providers to the same structured output schema), and test failover monthly by routing 1-5% of live traffic to the fallback for 15 minutes. If the fallback can't handle 5% of traffic cleanly, it won't handle 100% during an outage.

Pattern 4: Geo-Segmented. Traffic routes to different providers based on geographic or regulatory requirements. This is compliance-driven, not cost-driven.

EU PII workloads route to Nebius (Finland, France) or Scaleway. US workloads route to any US-hosted provider. Federal workloads route to AWS Bedrock Government or Azure Government; those are the only FedRAMP-authorized paths as of May 2026. None of the neo-clouds (Baseten, CoreWeave, Fireworks, Lambda, Modal, Replicate, RunPod, Together) have FedRAMP authorization as of May 2026 (source: marketplace.fedramp.gov). Verify current status before making procurement decisions.

Anthropic's inference_geo=US parameter with its 1.1x pricing multiplier is an honest acknowledgment of the cost of geographic constraints. If data residency matters, expect to pay for it.

The complexity tax¶

Multi-source isn't free. Every additional provider adds operational surface area.

Engineering overhead scales with providers, not linearly but noticeably. Each provider has different API semantics, error codes, rate-limiting behavior, and structured output support. Prompt portability between models is imperfect; a prompt tuned for Claude may perform differently on DeepSeek V3. My experience: budget 2-4 engineering days per provider for initial integration and 1-2 hours per month per provider for ongoing maintenance (API changes, deprecation notices, pricing updates).

Observability multiplies. Each provider produces telemetry in a different format. Standardizing on OpenTelemetry semantic conventions for GenAI helps, but the custom-metrics cost in Datadog or Grafana scales with the number of distinct provider×model combinations you're monitoring. Two providers with three models each is six metric series per telemetry dimension. That adds up fast against the observability tax described in Part 0.

Testing multiplies. Quality gates need to run against each provider×model combination. If you have three providers and two models each, that's six evaluation runs per prompt change. Automated evaluation pipelines (using frameworks like Braintrust, Arize, or custom harnesses) are mandatory at this point; manual evaluation doesn't scale.

The honest math: for a team running two providers with two models each, expect 8-16 engineering hours per month of multi-source overhead. At $100/hour, that's $800-$1,600/month, a meaningful fraction of the savings at lower volumes. This is why the Volume Gate matters. If you're saving $5,000/month by multi-sourcing and spending $1,200/month managing the complexity, your net benefit is $3,800. Still positive, but not the 5-7x improvement the raw numbers suggest.

Systems delay vs model delay. When latency degrades in a multi-source architecture, the instinct is to blame the model or provider. Usually the bottleneck is upstream. Decompose P95 latency into its components before optimizing:

Network round-trip: gateway → provider → gateway. Typically 10-50ms per hop; 2-4x that across regions. Measure with empty-payload pings.
Queue wait: time between request arrival at the provider and decode start. Invisible in the API response; infer from TTFT minus expected prefill time. Spikes during peak hours.
Prefill: processing the full input prompt in parallel. Scales with input length; 50-200ms for 2K tokens, 500ms+ for 8K+ on shared infrastructure.
Decode: autoregressive token generation. Scales with output length; the bottleneck for most chat and generation workloads.
Post-processing: schema validation, quality gate evaluation, logging, and response formatting on your side. Often 20-80ms and frequently overlooked.

If P95 is 3 seconds and decode accounts for 1.5 seconds, optimizing the model (switching to a faster provider, enabling speculative decoding) can only cut latency by half. The other 1.5 seconds is systems delay that requires infrastructure work: moving your gateway closer to the provider, reducing prompt length, caching prefill results, or parallelizing post-processing. Instrument each component separately before committing engineering effort.

The routing layer¶

Every multi-source deployment needs a routing layer. The question is how sophisticated to make it.

Level 0: Model-keyed routing. Model X goes to Provider A, Model Y goes to Provider B. No dynamic decisions. This is what most teams actually run, and it works. Implementation: a config file in your AI gateway.

Level 1: Failover routing. Level 0 plus automatic failover on provider errors. Implementation: your AI gateway's built-in retry/fallback logic (LiteLLM, Helicone, and Portkey all support this out of the box).

Level 2: Cost-aware routing. Route based on real-time pricing, rate limits, and capacity. Send overflow traffic to the cheapest available provider. Implementation: custom logic in your gateway, or a routing service like Martian ($18M raised, Accenture Ventures integration) or Not Diamond.

Level 3: Quality-aware routing. Route based on predicted model quality for the specific request. Estimate whether the cheap model can handle this request or if it needs the frontier model. Implementation: RouteLLM (open-source, out of UC Berkeley's LMSys group), or custom classifiers.

My recommendation for most teams: Level 1, with plans for Level 2. Level 0 is too fragile; you need failover. Level 1 is straightforward and covers 90% of the value. Level 2 is worth building when your monthly inference spend exceeds $100K and you have distinct traffic patterns with different cost sensitivities. Level 3 is research-grade; watch RouteLLM and RouterArena (arXiv:2510.00202, the first independent benchmark of routing quality), but don't bet production on it yet.

The build-side end-state¶

Character.AI represents the far end of the multi-source spectrum: full vertical integration. Custom Kaiju-family models (13B/34B/110B) running on DigitalOcean AMD Instinct MI300X/MI325X GPUs, handling 1B+ queries per day at ~20,000 inference QPS. Custom int8 attention kernels, KV cache on host memory between turns with LRU tree structure, quantization-aware training. They've achieved a 33x cost reduction since late 2022 and claim to be "13.5x cheaper than leading commercial APIs."

This is the build-side end-state. It works at Character.AI's scale. It does not work at yours. The engineering investment is measured in dozens of specialized inference engineers over multiple years. Don't attempt this until you have evidence (not a forecast, evidence) that your daily query volume justifies it. For everyone else, the serverless open-weights tier plus a routing layer gets you 80% of the economics at 5% of the engineering cost.

What to implement first¶

If you're moving from single-source to multi-source, the implementation order matters:

Add an AI gateway. LiteLLM for development, Helicone or Portkey for production. This takes a day and costs nothing (LiteLLM and Helicone are open-source and self-hostable).
Add a fallback provider for your primary model. Same model family, different provider. Configure automatic failover in your gateway. This takes an afternoon.
Move one workload to a cheaper provider. Pick the workload with the highest token volume and lowest quality sensitivity: batch processing, summarization, or classification. Measure the LCPR before and after for 30 days before extending.
Evaluate fine-tuning for your highest-volume workload. If a fine-tuned 8B or 70B model matches frontier quality on your specific domain evaluation, the cost advantage justifies the training pipeline investment.
Add geographic routing only if compliance requires it.

Each step is independently valuable. You don't need to reach step 5 to benefit from step 1. The minimum viable multi-source architecture is steps 1 and 2 (a gateway with failover) and it can be implemented in a day.

Part 3: The Inference Stack¶

Inference is not one decision. It's a stack of seven layers, each an independent choice. Most teams make the wrong call at least twice because they treat the stack as monolithic.

This section maps each layer: what it does, which tool wins, and the narrow conditions under which building your own makes sense.

The inference stack¶

Layer 1: AI Gateway. Recommendation: Buy.

The gateway sits between your application and your inference providers. It handles routing, retries, rate limiting, and basic observability. The build case is almost never compelling because the open-source options are mature and free.

LiteLLM (Python): broadest provider support, 100+ providers. Struggles past ~2,000 RPS per instance. Right for development and moderate production workloads.
Helicone (Rust, Apache 2.0): ~8ms P50 overhead (per Helicone docs; up to ~50ms under sustained load), strongest combined observability + routing. Production users handling 5,000+ RPS.
Portkey: enterprise control plane. Processes 2.5T+ tokens across 650+ organizations per their self-reporting. HIPAA BAA available.
Bifrost: 11-microsecond overhead. Right for hyperscale where gateway latency matters.

For most teams: LiteLLM in development, Helicone or Portkey in production. Build your own only if you have a specific technical requirement none of these meet.

Layer 2: Inference Runtime. Recommendation: Buy (use vLLM, SGLang, or TensorRT-LLM).

The runtime turns model weights into token predictions. The build case exists for very few teams (the major cloud providers, a handful of inference startups, and maybe a few hyperscale AI labs). All major runtimes (vLLM, SGLang, TensorRT-LLM) and managed providers (Baseten, Fireworks, Modal, Replicate, Together) now support prefix caching. Prefix caching reuses KV cache computations for shared prompt prefixes across requests; it is the single highest-ROI optimization for workloads with repeated system prompts, RAG context, or multi-turn chat. Structure prompts so that novel tokens appear at the end.

vLLM: the production default. 12,500 tok/s for Llama 3.1 8B BF16 on H100. Hardware support: NVIDIA, AMD, TPUs, Trainium, Gaudi. Continuous batching, PagedAttention, tensor parallelism. The right default unless you have a specific reason otherwise.
SGLang: ~29% higher throughput than vLLM on shared-prefix workloads via RadixAttention. Pick this for chat with long shared context, agent workloads, or evaluation harnesses.
TensorRT-LLM: 15-30% higher peak throughput after a 10-30 minute compilation step. More mature multi-node support than vLLM as of May 2026. Pick this for stable models and latency-sensitive workloads (real-time voice, synchronous chat) where peak throughput and tail latency matter. Mature FP4 support on Blackwell (V0.17). The compilation overhead makes it unsuitable for frequent model updates, but for production deployments with infrequent model changes, it's the throughput leader.
TGI: maintenance mode as of December 2025. Hugging Face now recommends vLLM or SGLang.

Build a custom runtime only if you have Character.AI-level scale (1B+ queries/day) and specific architectural requirements that justify custom attention kernels and KV cache management.

Layer 3: Kernels. Recommendation: Buy.

FlashAttention-4 (Tri Dao, Hot Chips 2025): up to 22% faster than cuDNN attention on Blackwell (source: SemiAnalysis coverage; Tri Dao blog, FA4 paper). Together Kernel Collection (TKC), built on the ThunderKittens framework from Stanford, reduces 1,000+ lines of CUDA to 100-200 lines while reporting notable speedups over FlashAttention-3 and FP8 inference gains on Blackwell (source: Together AI blog, TKC).

Speculative decoding kernels are now a vendor differentiator. Together's ATLAS achieves 500 TPS on DeepSeek-V3.1 (2.65x standard decoding) by adapting draft model selection per-request (source: Together AI blog, ATLAS). Fireworks' FireOptimizer delivers ~2x latency reduction at Cursor (source: Fireworks blog). Adaptive-speculation work is also active at Cerebras, Groq, and Baseten. NVIDIA cuBLAS + CUTLASS for everything else. The build case is essentially zero outside of foundation model labs and the handful of teams doing custom attention work.

Layer 4: Hardware. Recommendation: Buy from neo-clouds.

This is where the money is. Neo-cloud providers (GPU-focused cloud platforms offering bare-metal GPU access at lower prices than hyperscalers like AWS, Azure, and GCP) such as Lambda, RunPod, and CoreWeave offer 40-54% savings versus AWS for comparable on-demand GPU hours:

Provider	GPU	$/hr	$/month
Lambda	H100 SXM	$3.99	$2,873
RunPod	H100 SXM5	$4.41	$3,175
AWS	H200 (per GPU)	$4.975	$3,582
CoreWeave	H100 SXM	$6.16	$4,435
Baseten	H100	$6.50	$4,680

All prices are on-demand, per-GPU rates as of May 2026 (sources: Lambda pricing, Baseten pricing, datacenterdynamics on AWS H200 hike). Lambda H100 SXM 80GB 1-8 GPU on-demand at $3.99/hr; older PCIe 40GB rates sit at $4.29/hr for the legacy SKU. Prices exclude persistent storage ($0.10-$0.25/GB/month), though InfiniBand/NVLink networking is included at most neo-clouds. Egress is zero at Lambda, RunPod, and CoreWeave; $0.05-$0.09/GB at AWS. Reserved pricing adds 15-40% discount for 1-12 month commits.

Lambda at $3.99/hr is 20% cheaper than AWS and 39% cheaper than Baseten. AWS hiked H200 prices ~15% in January 2026, widening the gap further.

Two caveats. First, hyperscalers offer services neo-clouds don't: FedRAMP authorization, managed Kubernetes at scale, integrated data pipelines, and enterprise support contracts with meaningful remedies. If you need FedRAMP, AWS Bedrock Government or Azure Government are your only options.

Second, if your application runs on AWS but inference runs on a neo-cloud, egress costs apply in both directions. For high-throughput workloads generating large outputs (code generation, long-form content), egress can add 20-40% to total cost. Factor this into your TCO calculation before committing.

Layer 5: Orchestration. Recommendation: Buy NVIDIA Dynamo if multi-node.

NVIDIA Dynamo 1.0 (GA March 2026) is the de facto disaggregation layer for multi-node NVIDIA GPU inference. Named production adopters include AstraZeneca, BlackRock, ByteDance, CoreWeave, Crusoe, DigitalOcean, Lightning AI, Meituan, Nebius, Pinterest, Together AI, and Vultr (source: NVIDIA Dynamo 1.0 announcement). It sits above vLLM, SGLang, and TensorRT-LLM and provides KV-aware routing, SLA planning, and the NIXL low-latency transfer library.

For non-NVIDIA hardware (AMD MI300X, AWS Trainium, Google TPUs), orchestration options are runtime-specific: vLLM supports AMD and Trainium natively but lacks Dynamo's disaggregation features. For single-node deployments (1-8 GPUs), plain vLLM or SGLang with Kubernetes HPA is sufficient. You need Dynamo when you're running multi-node inference with disaggregated prefill and decode, typically 16+ GPUs across multiple nodes.

Layer 6: Observability. Recommendation: Buy, but budget carefully.

Observability is where teams get burned. The bill grows 30-50% year over year, and AI workloads generate 10-50x more telemetry than traditional services.

Helicone (bundled with gateway): free self-hosted, $20/seat/month Pro.
Arize AX: free tier (1M traces / 14 days), Pro at $50/month.
Datadog GPU Monitoring: $15-$23/host/month for infrastructure plus $31-$40/host for APM. The hidden cost is custom metrics: GenAI semantic-convention spans get billed as custom metrics, producing 40-200% bill increases.
Grafana Cloud Pro: $19/month base plus usage-based pricing.

Start with Helicone for LLM-specific observability (traces, prompts, completions, token cost tracking). If you need production ML monitoring (drift detection, model quality regression over time, A/B test analysis), evaluate Arize, which covers a different dimension than Helicone. If you need general APM + infrastructure monitoring, Datadog is comprehensive but expensive: the median mid-market Datadog bill is $123K/year (byteiota.com, 2026) and growing 30-50% YoY. Budget 2-4x your Year-1 observability estimate.

Layer 7: Routing Intelligence. Recommendation: Hold.

The routing-startup category (Martian, Not Diamond, RouteLLM, Unify, TensorZero) is maturing but not yet production-proven at scale. RouteLLM (UC Berkeley LMSys, open-source) is the strongest option for teams comfortable running experimental infrastructure. RouterArena (arXiv:2510.00202) is the first independent benchmark.

For most teams: use your gateway's manual model-keyed routing. If you have a clear cost/quality routing problem (e.g., 70% of traffic can use a cheaper model without quality degradation), evaluate RouteLLM. Revisit the commercial routing category in 12 months when it has consolidated.

The metering gap¶

One risk that cuts across all seven layers: the gap between what you think you're spending and what you're actually spending.

The chain from user request to provider invoice is longer than most teams realize:

user workflow → inference event → attempts (including retries) → tokens (input + output, per attempt) → cache discount → fallback/retry/repair → provider invoice → internal cost allocation

Each step in this chain introduces metering error. Your tokenizer may count differently from the provider's. Retry tokens may not appear in your application logs. Cache discounts may not match what you expected from the provider's documentation.

A 5% metering error on $100K/month in inference spend is $60K/year in unaccounted cost. At $500K/month, it's $300K. The fix is unglamorous: reconcile your telemetry against the provider invoice monthly. Compare token counts from your gateway logs to the provider's usage dashboard. Investigate any variance above 5%. Common sources: tokenization differences between your local tokenizer and the provider's (especially for non-English text or code), uncounted retry attempts, cache hit rates lower than assumed, and batched requests that get double-counted in your logs but single-counted on the invoice.

A worked example. A mid-market asset manager was running an RFP-response generation workload: bursty, roughly 3K queries on RFP windows, near-zero between windows. The workload sat on a 70B serverless open-weights model with structured-output validation. The pricing model in finance's projection used the vendor's reported per-request token counts straight from the API response: clean, well-instrumented, no obvious gaps.

The cost surprise surfaced at quarterly review. Year-to-date invoice spend ran 4.4% above projection across 100K+ requests/month. The first hypotheses (volume mix, retry inflation, cache hit rate drift) all came back negative. Tokenization parity, which the team had marked as "trivially correct" because both their local tokenizer and the API response token count came from the same vendor, was the fourth hypothesis. It was where the gap sat.

The vendor's reported per-request token count was computed against the model's runtime tokenizer. The invoice tokenizer, used downstream for billing, split punctuation slightly differently on certain Unicode sequences (specifically: smart quotes, em-dashes, and ligatures common in RFP source documents). The discrepancy was 4.4% on average and as high as 7% on heavily-formatted documents. The mechanism was buried in a 2-line note in the API docs that referenced a separate billing-tokenization spec the team had not read.

The fix was a reconciliation pass: pull the invoice tokenization spec, run it against logged inputs, and compare to the runtime count for each request. The 4.4% gap closed once we billed against the same spec the vendor billed against. The team moved to monthly reconciliation as a standing process.

The side-finding is uglier than the cost number. We discovered the same tokenization split was producing 0.8% of structured-output failures: RFP boilerplate phrases that the tokenizer broke at unexpected positions caused intermittent JSON-schema violations downstream. The 4.4% metering gap had been masking a small but systematic quality regression. Reconciliation work surfaces other people's problems; this is the discipline that exposes them.

Proving the model against reality. Once you have an LCPR model and a month of production data, reconcile:

Compare your gateway's token counts to the provider's usage dashboard. If they differ by more than 3%, your tokenizer is wrong; use the provider's count.
Compare your predicted monthly cost (LCPR × successful_requests) to the actual invoice. Variance above 10% means an assumption is wrong.
Spot-check retry rates: sample 1,000 requests from logs and count actual retries vs. your assumed rate.
Verify cache hit rates: compare expected cache hits (based on prompt structure) to the provider's reported cache hit rate.
Review pricing monthly. Providers change rates, add tiers, and modify cache discount policies without advance notice.

After one reconciliation cycle, update your LCPR inputs from "Assumed" to "Measured." After three cycles, your model is a budget, not a hypothesis.

The build-vs-buy summary¶

Layer	Recommendation	Build only if...
Gateway	Buy (LiteLLM/Helicone)	Never, unless hyper-specific caching needs
Runtime	Buy (vLLM/SGLang)	1B+ queries/day with custom arch needs
Kernels	Buy	You are Tri Dao
Hardware	Buy neo-cloud	You need FedRAMP → hyperscaler
Orchestration	Buy Dynamo if multi-node	Single-node → skip entirely
Observability	Buy (Helicone/Arize)	Don't build; budget carefully
Routing	Hold (evaluate RouteLLM if clear routing win)	Don't build, don't buy commercial yet

Part 4: Vendor Evaluation¶

Vendor evaluation in inference has a specific problem: the features that matter most are the hardest to evaluate from public information. Pricing is transparent. Latency under load is not. Compliance certifications are public. Zero data retention defaults are buried in terms of service.

Seven evaluation gates structure the next section. Each gate has a pass/fail criterion and a method for verification. A vendor that fails any gate should be eliminated for that workload, regardless of how well they score on the others.

Gate 1: Model Availability¶

Does the vendor serve the specific model(s) you need, at the precision you need?

This seems obvious, but model availability is more nuanced than checking a catalog page. Key questions:

Is your model available in FP8? FP4? The precision affects both quality and throughput.
For fine-tuned models: can you deploy custom weights, or only use the vendor's hosted versions?
For LoRA: does the vendor support runtime LoRA loading, or do you need a separate deployment per adapter?
How quickly do new models become available after release? Some vendors lag by weeks.

Verification method: check the model catalog page, then verify the specific precision and configuration via API. Don't trust the catalog alone; models listed as "available" may be in preview or limited access.

Gate 2: Latency Under Load¶

What is the P50/P95/P99 latency at your expected concurrency, not on an empty endpoint?

Vendor-published latency numbers are measured on unloaded endpoints with optimal batch sizes. Production latency under shared infrastructure is 2-5x worse at P99. The only reliable latency data is either (a) your own benchmark on the vendor's infrastructure, or (b) independent benchmarks like Artificial Analysis or SemiAnalysis InferenceMAX.

Verification method: run your actual prompts at your expected concurrency for at least 7 days to capture weekly traffic patterns (weekend dips, Monday spikes, diurnal variance). 24 hours is insufficient for P99 SLO validation. Measure TTFT and inter-token latency at P50, P95, and P99. If the vendor won't give you a trial endpoint with sufficient duration, that's information.

Gate 3: Throughput Economics¶

At your volume, what is the LCPR, not the token rate?

This is the LCPR calculation from Part 0. Input the vendor's published rates, your workload profile (tokens, retry rate, quality gate, engineering hours), and compute the loaded cost. Compare across vendors at the LCPR level, not the token level.

Verification method: use the LCPR calculator with your actual workload numbers. The vendor's pricing page is an input to the calculation, not the answer.

Gate 4: Reliability and Failover¶

What is the vendor's published uptime SLA, and what are the actual remedies?

Most vendors offer 99.9% uptime with credit-based remedies. Read the credit math: most are capped at the monthly fee for the affected period, which doesn't cover your revenue loss during an outage.

Key questions:

What's the historical uptime over the last 12 months? (Check status pages and incident histories.)
Does the vendor support multi-region deployment for failover?
What's the rate-limiting behavior under load? (Some vendors degrade gracefully; others return 429s aggressively.)

Verification method: check the vendor's status page history. Ask for uptime data covering the last 6 months. If they can't provide it, assume 99.5% or lower.

Gate 5: Compliance and Data Handling¶

Does the vendor's default data handling match your requirements, not just their certifications?

SOC 2 Type II and HIPAA are table stakes: Together, Fireworks, Baseten, Modal, Nebius, and FriendliAI all have them. The differentiator is the default data handling behavior:

Baseten: zero data retention by default (verified May 2026).
Fireworks: zero retention on standard inference; Response API retains 30 days unless store=false (verified May 2026).
Together: data stored by default unless disabled in settings (verified May 2026).
OpenAI: fine-tuning data retained; API data retention varies by endpoint.

Vendor data retention policies change. Verify current policy at contract time and get written confirmation; verbal assurances are insufficient for regulated workloads.

For EU data residency: Nebius (Finland, France), Scaleway, Mistral La Plateforme, OVH. For US federal: AWS Bedrock Government or Azure Government only.

Verification method: read the terms of service and data processing agreement. Ask specifically: "If I send a request to your API and do nothing else, is the prompt or completion stored? For how long? Where?" The answer should be in writing, not verbal.

Gate 6: Integration Complexity¶

How many engineering hours does it take to go from zero to production with this vendor?

This covers API compatibility (OpenAI-compatible vs. custom), SDK quality, documentation completeness, structured output support, and streaming behavior. Vendors with OpenAI-compatible APIs (Together, Fireworks, DeepInfra) have lower integration cost because your existing code works with a URL change. Vendors with custom APIs (some Baseten configurations, custom runtimes) require more integration work.

Verification method: build a proof-of-concept integration. Measure time from API key to first successful production-format request. If it takes more than a day, factor that into your migration cost estimate.

Gate 7: Pricing Trajectory¶

Is this vendor's pricing going up or down, and why?

This is the most forward-looking gate and the hardest to verify. The signal from April 2026 is clear: frontier closed APIs are increasing prices. Serverless open-weights providers are competing on price and have room to decrease. Dedicated GPU pricing follows hardware cycles; B200 availability in late 2026 should bring H100 prices down further.

Key questions:

Has the vendor raised prices in the last 12 months? (OpenAI doubled GPT-5.5 rates on April 23, 2026.)
What's the vendor's gross margin? (Fireworks estimated ~50% per Sacra research, targeting 60%.)
Does the vendor have structural cost advantages (custom kernels, speculative decoding, cache pooling) that protect margins without raising prices?

Verification method: check pricing page history via Wayback Machine. Read earnings calls or funding announcements for margin signals. Vendors with structural cost advantages — custom speculative decoding (Together ATLAS, Fireworks FireOptimizer), KV cache pooling (LMCache integration), custom kernels (TKC, FireAttention) — can maintain pricing as GPU commodity markets tighten. Vendors relying on GPU arbitrage alone will face margin pressure and may raise prices or reduce service quality.

Where the framework picks against my employer¶

A worked example. On the same DeepSeek V3.1 model at May 2026 pricing, Fireworks runs about 33% cheaper than Together on input + output combined ($0.56/$1.68 vs $0.60/$1.70 per million; source: Fireworks pricing). On DeepSeek R1 Basic, Fireworks is roughly 75% cheaper than Together's serverless rate (source: Fireworks pricing). On Llama 70B, Together and Fireworks are within ~2% of each other ($0.88 vs $0.90 flat), essentially tied. The model + provider combination dominates the LCPR; the provider alone does not.

If your workload runs on DeepSeek R1 or DeepSeek V3.1 at high volume and you're shopping for the lowest per-token cost on that specific model, the LCPR math picks Fireworks. If your workload runs on Llama, the pricing is close to a coin-flip and the decision comes down to other gates: latency under load, fine-tuning options, fallback availability, ATLAS vs FireOptimizer's fit to your output-length distribution. I work at Together; I would still recommend running the calculation against the live pricing for the specific model you've selected. The framework's job is to pick the cheapest model + provider combination for your workload's quality gate, not to validate a prior preference for any one provider's catalog.

Using the scorecard¶

For each vendor under consideration, score each gate as Pass, Conditional Pass (acceptable with mitigation), or Fail. A Fail should eliminate the vendor unless you have a clear compensating control. Failing Gate 4 (reliability) can be mitigated with multi-vendor failover; failing Gate 5 (compliance) for regulated workloads generally cannot. Two or more Conditional Passes should trigger deeper evaluation before committing.

The scorecard is deliberately binary — pass/fail, not scored 1-10 — because weighted scoring encourages teams to rationalize a preferred vendor by assigning high weights to gates where it excels. "Fail" means "fail for this workload without mitigation," not "never use this vendor."

Negotiating pricing tiers¶

Once you've selected vendors via the scorecard, negotiate pricing in tiers rather than asking for a blanket discount:

Under $25K/month: don't negotiate. Use public pricing. The vendor's sales team won't prioritize you, and the discount (if any) won't exceed 5-10%.
$25K-$100K/month: request a committed-use discount. Most providers offer 15-30% off for 3-6 month commitments. Get it in writing with clear terms on what happens if you under-commit.
$100K-$500K/month: negotiate custom pricing with a named account manager. At this level, you have leverage; the vendor's cost to serve you is low relative to revenue. Push for volume tiers with automatic step-downs, not flat discounts. Ensure the contract includes price protection (the vendor can't raise your rates mid-contract) and a 30-day out clause if they deprecate your model.
$500K+/month: negotiate capacity commitments with SLAs. At this level, you're a strategic account. Push for dedicated capacity guarantees, custom model hosting, priority support, and co-development of optimizations. The vendor should be assigning engineering resources to your account.

The key principle: never negotiate price alone. Negotiate price + SLA + capacity guarantees + contract flexibility as a package. A 20% discount with no SLA is worth less than a 10% discount with P99 latency guarantees and a 90-day out clause.

Part 5: The staged playbook¶

This final section synthesizes Parts 1-4 into concrete, staged guidance. Each stage has an entry threshold, a set of actions, and an exit threshold that tells you when to graduate to the next stage.

A note on thresholds. Every dollar figure and percentage in this playbook ($10K/month, $100K/month, 40% utilization, 500ms P95) is a configurable default, not a law. These values reflect typical economics for 70B-class models at May 2026 pricing. They shift with model size (smaller models have lower break-evens), provider pricing changes, your team's engineering efficiency, and your workload's I/O ratio. Use the LCPR calculator to compute your thresholds, not these defaults. When this essay says "$10K/month," read it as "the volume where your LCPR calculator shows migration savings exceeding amortized engineering cost."

Stage 0: Prototype (under $10,000/month)¶

Entry: you're building an AI-powered product and spending less than $10,000 per month on inference.

Architecture: single closed API (OpenAI, Anthropic, or Gemini). No gateway. No fallback. No dedicated GPU.

Actions:

Pick one provider. Anthropic if you need reasoning quality and prompt caching (90% reduction on cached input tokens). OpenAI if you need the broadest ecosystem. Gemini if you need the cheapest frontier option ($1.25/$10 for ≤200K context).
Use prompt caching aggressively. Anthropic's caching reduces cached input cost to 10% of base. OpenAI's automatic caching triggers on prompts ≥1,024 tokens at 50% discount.
Don't optimize for inference cost. At $4,116/month on GPT-5.5 for 200K requests, the savings from switching to open-weights ($2,987/month) don't justify the engineering distraction of migration. Ship the product.
Use prompt caching to stretch your closed-API budget further. A Sonnet workload with 4,800 input tokens (4,000-token system prompt + 800 user input) and 600 output tokens at 500K requests/month costs $12,901/month without caching. With Anthropic's 83% cache hit rate (the system prompt is cacheable), LCPR drops 43% to $7,361/month, a $5,540 savings with zero migration effort. Even cached Sonnet at $0.0155 LCPR is still 1.7x Together's uncached $0.0091, but the gap narrows enough that migration ROI becomes marginal at this volume.

Exit threshold: monthly inference spend exceeds $10,000 (approximately 500K requests/month on GPT-5.5 at 800/400 tokens, the point where multi-source migration ROI exceeds $7.5K/month per the Part 1 worked example), OR you experience a provider outage that costs revenue, OR a customer asks about data residency. The $10K figure is a guideline; teams with tight margins or latency-sensitive workloads may justify Stage 1 earlier.

Stage 1: Scale ($10,000-$100,000/month)¶

Entry: you've passed the Volume Gate from Part 1.

Architecture: primary closed API + AI gateway + one or two serverless open-weights providers for specific workloads.

Actions:

Add an AI gateway (LiteLLM in dev, Helicone or Portkey in prod).
Add a fallback provider for your primary closed-API model (Anthropic via Bedrock, Gemini via Vertex).
Move long-tail, quality-insensitive workloads to serverless open-weights: batch processing, summarization, classification, embeddings. Together, Fireworks, or DeepInfra on Llama 3.3 70B, DeepSeek V3, or Qwen 3. For offline batch workloads (embeddings, evaluation harnesses, bulk summarization), consider spot-priced dedicated GPUs (RunPod spot, Lambda spot) at 40-70% discount. Batch workloads tolerate interruption and higher latency.
Implement prompt caching everywhere it helps. On closed APIs, this means Anthropic's explicit caching (90% discount) or OpenAI's automatic caching (50% discount). On serverless open-weights, Together's always-on prefix caching gives ~90% reduction on cached input tokens with no configuration. Structure prompts with static content first, variable content last.
Start measuring LCPR, not just token cost. The difference matters at this scale.

Worked example: a team at 2M requests/month on GPT-5.5 spends $33,960/month. Splitting 70/30 (keeping 1.4M quality-sensitive requests on GPT-5.5 and moving 600K long-tail requests to a serverless open-weights provider: Together at $0.60/$1.70, Fireworks at $0.56/$1.68 on DeepSeek V3.1, DeepInfra on GPT-OSS-120B, Anyscale or Replicate on Llama; pick by workload and current pricing) brings the combined bill to about $25,500. Roughly $8,400/month in savings, or $100K/year, with minimal engineering effort.

Exit threshold: any single workload exceeds ~50M output tokens/day with steady traffic, OR you need a fine-tuned model, OR you have a hard latency SLO under 500ms that shared APIs can't meet.

Stage 2: Production at Scale ($100,000-$1,000,000/month)¶

Entry: you've passed the Specialization Gate or hit the dedicated GPU crossover.

Architecture: multi-source with one or two dedicated GPU deployments for highest-volume workloads, serverless for everything else.

Actions:

Move your 1-2 highest-volume workloads to dedicated inference. Pick the vendor by workload fit:
- Baseten if you need TensorRT-LLM + observability tooling (Abridge, OpenEvidence, Writer customer references).
- Fireworks if you have agentic coding workloads, RL post-training, or need FireOptimizer's adaptive speculative decoding (Cursor, Vercel v0 customer references).
- Modal or Replicate for managed dedicated serving with strong developer-experience surface (cold-start tradeoffs and pricing differ from the larger inference providers; verify against your latency budget).
- Together if you need ATLAS adaptive speculative decoding (2.65x measured on DeepSeek-V3.1, source: Together AI blog), unified fine-tuning + inference on the same hardware, or B200 capacity sooner than other providers can offer it (Cresta Multi-LoRA, Decagon voice AI customer references; source: Together AI + Decagon case study).
Run vLLM or SGLang. Use FP8 quantization (8-bit floating point, which halves memory versus the standard BF16 16-bit format) for 70B-class models. Quality holds within 1% of BF16 on most benchmarks.
Run NVIDIA Dynamo if multi-node.
Buy compliance certifications (SOC 2, HIPAA BAA) from your dedicated vendor.
Monitor GPU utilization weekly. The 40% threshold approximates the break-even between dedicated and serverless for 70B-class models: at Lambda's $3.99/hr and serverless rates of $0.90-$1.70/M tokens, you need roughly 10-11 hours/day of saturated throughput (43-44% daily utilization) to justify dedicated. Below that, serverless is cheaper. Consolidate via Multi-LoRA if you have multiple low-volume workloads that can share a GPU.

Worked example: at 10M requests/month, GPT-5.5 costs $166,600/month. Together serverless costs $13,748. A Lambda H100 at 40% utilization costs $10,418 for that same workload. This excludes egress costs. Lambda charges zero egress, but if you're routing outputs through a hyperscaler's load balancer or CDN, add $0.05-$0.09/GB. At higher utilization, the dedicated cost drops further. The dedicated option wins at this volume if (a) utilization stays above 40%, and (b) egress costs don't negate the savings. Serverless remains the safer default.

Exit threshold: total monthly spend exceeds $1M, OR you have a strategic reason to control kernels and models end-to-end.

Stage 3: Build-Side ($1,000,000+/month)¶

Entry: you've hit a scale where the operational investment in custom infrastructure is justified by the savings.

Architecture: dedicated inference on neo-cloud (Lambda, CoreWeave, Nebius) with vLLM/SGLang + custom optimizations. Serverless overflow path for traffic spikes.

Actions:

Hire 2-4 dedicated inference engineers, plus SRE support for on-call, alerting, and capacity planning. This is not optional; you cannot run dedicated inference at $1M+/month without specialized expertise. The inference team owns runtime optimization, quantization, KV cache tuning, and failure recovery. SREs own runbooks and operational tooling.

Alternatively, managed dedicated endpoints offer comparable cost economics without the operational burden. Several vendors offer customer-engineering-as-service models on reserved GPU capacity: Together (forward-deployed engineering on Decagon's voice AI deployment [source: Together AI + Decagon case study], Cresta's Multi-LoRA work), Fireworks (FireOptimizer tuning on Cursor's Fast Apply), Baseten (TensorRT-LLM tuning on Abridge's healthcare deployment). The model is similar across vendors: dedicated engineers from the provider iterate on your deployment (custom speculators, quantization pipelines, kernel-level tuning) on reserved capacity you contract for.

The trade-off is the same regardless of vendor: you cede runtime control. If your models and workloads are stable enough that you don't need to tune kernels yourself, managed dedicated is often the right call. If you need to iterate on custom attention patterns or exotic quantization schemes, you need the in-house team.

Adopt LMCache or Mooncake for KV cache pooling if your traffic has high prefix overlap (shared system prompts, RAG context, multi-turn chat). KV cache pooling deduplicates shared prefixes across requests; workloads with >70% prefix overlap see the largest gains. LMCache reports 3-10x TTFT improvement and up to ~14x throughput on multi-turn workloads (LMCache benchmarks, 2025-2026). Mooncake powers Kimi K2's production traffic at 100B+ tokens daily.
Evaluate FP4 quantization on Blackwell with proper calibration. NVIDIA's analysis shows 1% or less accuracy degradation on key tasks. FP4 on B200 doubles throughput versus FP8.
Maintain a serverless overflow path. Dedicated deployments should have this. Traffic spikes happen, GPUs fail (Meta's Llama 3 training saw 466 job interruptions over 54 days, 78% hardware-related), and autoscaling dedicated GPU is measured in minutes, not milliseconds.
Don't try to be Character.AI. They serve 1B+ queries/day on custom int8 kernels and quantization-aware training, a 33x cost reduction since 2022 (source: Character.AI Kaiju engineering post). That's the build-side end-state. Your scale is probably not their scale.

The revert signals¶

Every stage transition should be monitored for revert signals, indicators that you've graduated too early.

Stage 1 → Stage 0: If your multi-source overhead (gateway maintenance, prompt migration testing, vendor management) exceeds 20% of your inference savings, simplify back to a single provider.
Stage 2 → Stage 1: If your dedicated GPU utilization stays below 40% for two consecutive months, move that workload back to serverless. At 40% utilization on a Lambda H100 ($3.99/hr), your effective cost per output token exceeds serverless rates ($0.90-$1.70/M). You're paying $2,873/month in fixed GPU cost for throughput you could get cheaper on-demand.
Stage 3 → Stage 2: If your inference engineering team spends more than 50% of their time on operational issues (GPU failures, OOM errors, kernel debugging) rather than optimization, you don't have the operational maturity for build-side infrastructure yet.

These revert signals are as important as the exit thresholds. The right architecture is the simplest one that meets your cost and performance requirements. A team that owns 8 H100s at 28% utilization is paying 1.4x what serverless would cost, and paying it twice (once for the GPU bill, once for the engineering payroll).

Red-flag triggers¶

Beyond revert signals, certain events should trigger an immediate review of your inference setup regardless of which stage you're in:

Invoice variance > 10% from your LCPR model prediction. Something changed: metering, traffic mix, or pricing.
Retry rate increase of 5+ percentage points over a 2-week window. Usually indicates a model degradation, prompt regression, or provider-side issue.
Cache hit rate drop of 10+ percentage points. Prompt structure may have changed, or the provider changed cache eviction behavior.
Quality gate regression: pass rate drops 5+ points. Could be a model version change, prompt drift, or eval suite staleness.
P99 latency breach sustained over 4+ hours. Investigate queue depth, batch size, and provider capacity.
Fallback share exceeds 20%. Your primary provider is unstable; consider promoting the fallback or adding a third option.
Provider model version change (e.g., GPT-5.5 → GPT-5.5-turbo). Re-run your eval suite before assuming quality parity.
Provider pricing page update. Re-run the LCPR calculator and update your cost projections.
Traffic mix shift: one workload grows 3x while others are flat. Blended LCPR no longer represents individual workload economics. Segment and re-evaluate.

The common thread: any change to LCPR inputs should trigger a re-evaluation of LCPR outputs. The calculator is cheap to run; surprises on your invoice are not.

Workload reference cards¶

Each inference workload type has a characteristic bottleneck, failure mode, and set of calculator defaults. Use these as starting points, not gospel; your production numbers will differ.

Chat (customer-facing).

Bottleneck: decode latency (streaming TTFT and inter-token latency)
Key metric: P95 time-to-first-token
Common failure: latency spikes under concurrency causing user abandonment
Calculator defaults: 500-1000 input, 200-500 output, 3-5% retry, 95% quality gate
Where defaults break: multi-turn conversations with long history (input tokens grow per turn), peak-hour traffic patterns (retry rate spikes)

RAG extraction.

Bottleneck: prefill (long retrieved context)
Key metric: cost per extracted answer, measured by LCPR with cache hit rate
Common failure: irrelevant retrieved chunks inflating input tokens without improving output quality
Calculator defaults: 2000-8000 input, 200-600 output, 5% retry, 90% quality gate, 20-40% cache hit rate
Where defaults break: cache hit rate depends entirely on prompt structure. Static system prompt + variable retrieved context yields 30-60% hit rate, but fully variable prompts yield <5%

Code generation.

Bottleneck: decode (long output)
Key metric: LCPR per accepted suggestion (quality gate includes human acceptance, not just schema validation)
Common failure: high output token count with low acceptance rate. You're paying for code the developer immediately deletes
Calculator defaults: 500-2000 input, 500-2000 output, 3% retry, 70-85% quality gate
Where defaults break: quality gate varies enormously by task (autocomplete at 60% vs multi-file generation at 40%)

Agent workflows.

Bottleneck: cumulative latency across chained calls
Key metric: end-to-end workflow success rate and total LCPR per workflow (sum of per-call LCPR × calls per workflow)
Common failure: retry cascades, where one failed call triggers retries that propagate through the chain
Calculator defaults: 1000-4000 input per call, 200-500 output per call, 5-10% retry per call, 90% quality gate per call, 3-8 calls per workflow
Where defaults break: tool-use failures compound. A 5% per-call failure rate across 6 calls gives a 26% workflow failure rate

Batch / embeddings.

Bottleneck: throughput (total tokens per hour)
Key metric: cost per million tokens processed, not latency
Common failure: underusing batch pricing tiers. Many providers offer 50% batch discounts but teams don't restructure pipelines to qualify
Calculator defaults: variable input, minimal output (embeddings) or moderate (summarization), 1% retry, 98% quality gate, 50-100% batch eligible fraction
Where defaults break: batch APIs have higher latency (minutes to hours) and may have different rate limits

Diagnostic trees¶

When LCPR or latency degrades, use these decision trees to isolate the root cause before committing engineering effort. Each follows the pattern: symptom → metric to check → likely bottleneck → experiment → stop condition.

P99 latency spikes under concurrency. Symptom: P99 latency doubles during peak hours while P50 stays stable. Check: queue wait time (TTFT minus expected prefill time). If queue wait > 500ms → provider capacity constraint. Experiment: add a second provider for peak overflow. If queue wait is normal → check batch size (are requests batching larger during peaks, increasing per-request decode time?). Experiment: cap max batch size or route overflow to a dedicated endpoint. Stop: when P99/P50 ratio < 2x during peak.

Cache hit rate lower than expected. Symptom: cache hit rate is <10% when you expected 30-50%. Check: prompt structure. Are variable elements (user query, retrieved documents) placed before static elements (system prompt, instructions)? If yes → restructure: static content first, variable content last. If prompt structure is correct → check tokenization: are you using the provider's tokenizer? Token boundaries differ between providers, and a one-token difference in the cached prefix invalidates the cache. Experiment: log the exact token count of the cached prefix per request and check for variance. Stop: when cache hit rate matches the fraction of requests with identical prefixes.

LCPR worse after migration. Symptom: you migrated from Provider A to Provider B and LCPR increased despite lower token rates. Check: quality gate pass rate on Provider B. If lower → model quality gap. Your prompts were tuned for Provider A's model; Provider B's model fails differently. Experiment: run your eval suite on Provider B and identify the failing categories. Tune prompts or adjust quality gates. If quality gate is similar → check retry rate on Provider B. Different error codes, different rate limits, different timeout behavior can increase retries. Check: engineering hours. Migration overhead (dual-stack maintenance, prompt migration, monitoring setup) may not have been captured. Stop: when LCPR on Provider B is within 10% of pre-migration LCPR on Provider A, with quality gate within 2 points.

Resources¶

LCPR Calculator (run locally) — Run the math against your own workload. Same engine that generated every number in this essay.
GitHub repository — Source code, provider pricing YAML (refreshed via reconciliation tests in CI), diagrams, and 248 tests pinning the calculator's numerical claims.
Templates — Vendor scorecard, migration readiness checklist, LCPR worksheet.

Closing¶

The frameworks in this guide (LCPR plus five decision rules) are tools for making decisions with math instead of vibes. They're opinionated, because frameworks that try to accommodate every edge case end up accommodating none.

The companion LCPR calculator lets you run these calculations against your actual workload. Every number in this essay was generated by that calculator and verified against May 2026 public pricing. The pricing YAML in the repo updates on commit; re-run when your contract numbers diverge from the published ones, which they usually do.

If you take one thing from this essay, take the four-line LCPR formula and the assumption-confidence table earlier in Part 0. The frameworks are scaffolding; the math is the load-bearing part.

This essay is the overview. The Production Inference Economics series develops the measurement methodology in depth:

Sohail Mohammad --- April 2026

Numbers are anonymized and should not be attributed to any specific employer, customer, or deployment.

KV Cache Drift and the Interpretability Blind Spot

2026-03-20T16:35:00-05:00

two communities are studying the same failure mode and mostly not citing each other.

inference systems people call it query inconsistency in KV cache eviction
interpretability people call it feature drift across prompt, thinking trace, and response

same shift. different vocabulary.

the systems side has been quantifying it aggressively. LAQ (Lookahead Q-Cache), DapQ, and LookaheadKV all target the same core issue: prefill-time attention patterns do not reliably predict decode-time importance.

meanwhile, interpretability workflows still train SAEs mostly on prefill activations and apply them during decode for steering and monitoring.

that should make you uncomfortable.

if you want my broader writing/research context, it is all linked in /writings/.

in plain english¶

here is the non-jargon version.

imagine a student who studies using textbook problems, then is graded on a very different kind of exam.

the student is still smart. the memory is still there. but the question style changed, so performance can drift.

that is basically what is happening here:

prefill is the textbook phase (model reading human-written text)
decode is exam-time improvisation (model generating its own next steps)
SAE features are the study notes we built from textbook behavior

we then use those study notes to judge or steer behavior during exam-time generation.

sometimes this works great. sometimes it misses what changed.

so this piece is arguing for one simple standard: before we call this safety infrastructure, we should measure how much the model's internal behavior drifts from study-time to exam-time.

the framing that makes this obvious¶

i think the right bridge concept is a consistency model in the distributed-systems sense.

not the diffusion-model meaning of consistency models. not output-level consistency metrics. i mean: what guarantees do we have across computational phases?

for transformers in autoregressive inference:

the KV cache provides strong data consistency but weak query-distribution consistency.

that sounds abstract, but it is practical.

1) strong data consistency¶

KV entries are append-only and immutable once computed.

position 47 stays position 47 whether you are at decode step 48 or 500. this is stable and reliable.

2) snapshot-isolated prefix behavior¶

with prefix caching, multiple generations branch from a shared frozen prefix and then append private histories. conceptually, this is copy-on-write behavior.

3) weak query-distribution consistency¶

this is the blind spot.

during prefill, queries are conditioned on human-written tokens.

during decode, queries are conditioned on model-generated tokens and evolving positional geometry.

the data store is stable, but the query workload drifts.

this is not speculative anymore¶

LAQ made this painfully concrete: using prefill-stage signals for decode-time cache decisions leaves measurable performance on the table under constrained memory.

DapQ pushes further with a strong positional claim (where matters more than what): decode alignment depends heavily on positional encoding evolution, not only token semantics.

that matters for interpretability because if positional geometry is a first-order driver, then even "good" prefill features can be miscalibrated in late decode.

Goodfire independently observed a related effect from the interpretability side: feature distributions differ across prompt, reasoning trace, and final response, and steering behavior is phase-sensitive.

you do not need perfect cross-paper agreement to see the shape here.

the gap is real enough to change outcomes.

why this should change how we talk about SAEs¶

i am not arguing SAEs are bad.

i am arguing the default deployment story is under-calibrated.

we train dictionaries in one regime and trust them in another regime without routinely reporting drift diagnostics.

that is fine for exploratory work.

it is weak for safety infrastructure claims.

three concrete consequences:

feature calibration drift prefill sparsity/activation rates do not necessarily hold in decode.
steering reliability drift a vector that behaves cleanly at early decode can decay, rotate, or invert later.
monitoring sensitivity drift monitors tuned to prefill structure likely catch blunt failures but can miss subtle regime-specific failures.

the practical opportunity: use KV mechanics as an interp primitive¶

systems engineers already compute attention over the full cache at each decode step.

that means the raw ingredients for decode-native monitoring already exist in the serving path.

so instead of only asking "which SAE feature fired," we should also ask:

how attention entropy evolves step to step
whether top-attended regions jump (phase transitions)
how much mass stays on prefix vs generated history
whether trajectory signatures shift before output quality collapses

this is cheap compared to heavyweight post-hoc analysis because these statistics come from values already computed during inference.

what i would run next (minimum viable plan)¶

if this were my production interp roadmap this quarter:

A) add decode attention telemetry¶

log per-step entropy, top-k attention positions, and prefix-vs-generated mass.

B) run reconstruction drift checks¶

take a prefill-trained SAE and evaluate reconstruction quality at decode step 1, 8, 50, 200.

C) compare feature overlap across regimes¶

track top-k feature-set overlap and rank stability between prefill and decode.

D) measure steering effect decay by depth¶

apply identical steering at multiple decode depths and quantify effect-size degradation.

E) test hybrid SAE training data¶

run a mixed prefill/decode corpus variant and compare monitoring + steering stability to prefill-only.

none of this requires waiting for a new paradigm. it is engineering and measurement discipline.

why this piece is opinionated¶

because the field is drifting toward stronger claims than the measurement stack currently supports.

"it works in practice" is a useful start.

"we know where it fails and can detect when we cross that boundary" is what safety infrastructure actually requires.

right now, we are closer to the first statement than the second.

closing¶

the interesting part is not that one community is right and the other is wrong.

the interesting part is that both communities found the same crack from different directions.

KV cache folks quantified the crack as query inconsistency.

interpretability folks observed the crack as regime-dependent feature behavior.

we should stop treating those as separate stories.

if your interpretability pipeline has no decode-regime calibration report, you are shipping with an unmeasured consistency assumption.

and eventually that assumption will bill you.

Continuous Batching: What It Changes in Activation Data

2026-03-20T13:25:00-05:00

Goodfire’s infrastructure write-up on large-scale activation harvesting is one of the strongest pieces of interpretability systems engineering I’ve seen this year.

It also surfaces a question that deserves a direct answer:

Do inference-serving optimizations change activation values in ways that matter for SAE training?

Not metadata. Not provenance. The tensors themselves.

I’ve spent the last few years building production inference stacks (vLLM/SGLang, high-volume conversational systems, distributed training) while also running mechanistic interpretability experiments (contrastive directions, SAE decomposition, behavioral ablation). This question sits exactly at that boundary.

Core claim: continuous batching is algorithmically equivalent to independent-sequence processing for activations; the real risks are hardware-level nondeterminism, prefix-caching-induced coverage gaps, and downstream SAE sensitivity to data methodology.

in plain english¶

here is the non-jargon version.

imagine you are trying to compare students fairly, but each one took the same test in a different room with slightly different lighting and timing.

the questions are the same, but tiny environment differences can still change who looks strong or weak.

that is this problem:

continuous batching itself is usually mathematically safe
but hardware execution details and cache behavior can still shift collected activations a bit
SAEs are sensitive enough that those tiny shifts can matter downstream

so the point is not "batching is bad."

the point is "if you are training interpretation tools from these activations, measure the collection artifacts instead of assuming they are harmless."

The concern, precisely¶

In continuous batching, tokens from multiple requests are flattened into a shared token axis for throughput. Attention masking prevents cross-request token interaction inside attention.

The classic worry is whether non-attention operations (especially normalization) leak across co-batched samples.

If normalization were batch-coupled, identical prompts could produce different activations depending on neighbors. That would make scheduler composition a hidden confound in activation datasets.

LayerNorm/RMSNorm: per-token, not batch-coupled¶

For standard transformer usage, LayerNorm and RMSNorm normalize over hidden features per token, not across batch tokens.

LayerNorm: normalized over the specified feature dimensions (typically hidden dim)
RMSNorm: per-token root-mean-square over hidden dim only

So adding/removing unrelated sequences in the same continuous batch does not change the normalization statistics for an existing token.

This is architectural/implementation behavior, not a hypothesis.

Where equivalence breaks: floating-point execution paths¶

Algorithmic equivalence is not bitwise equivalence.

Different effective matrix shapes can trigger different kernel/tiling paths (e.g., cuBLAS heuristics), changing accumulation order under reduced precision. Chunked prefill likewise can alter reduction order relative to monolithic passes.

In floating-point arithmetic, that can produce small numeric drift, and occasionally output divergence near decision boundaries. Note that often-cited batch-size sensitivity results came from an OpenReview submission (not a peer-reviewed venue) and were strongest under TF32 settings; treat them as suggestive rather than settled.

The mechanism is real and well-understood in numerical computing. Magnitude in this exact activation-harvesting setup is still under-measured.

Prefix caching: the bigger practical confound¶

Prefix caching can skip forward computation for repeated prefixes by reusing cached KV state. If forward compute is skipped, intermediate activations for those cached tokens are not newly produced during that run.

That means activation harvesting in a cache-heavy serving mode can systematically under-sample repeated prompt segments (system prefixes, boilerplate exemplars, recurring headers).

This is a coverage issue. More precisely, it creates coverage missingness / coverage bias.

Observed that caching skips compute; suggestive that resulting token-class undercoverage materially shifts SAE feature learning unless explicitly corrected.

Why SAE methodology sensitivity makes this matter¶

SAEs are useful, but they are not invariant to data-collection and formatting choices.

Across recent work, we’ve seen:

substantial feature-set variability across training seeds/protocols
sensitivity to data formatting/sequence construction
performance deltas on OOD/safety tasks from methodology shifts alone

So “activations were harvested from the same base model” is not enough. The path from model to training corpus (serving mode, caching behavior, batching dynamics, chunking policy) can affect what the SAE learns.

What I would measure immediately¶

1) Batch-composition drift baseline¶

Fix input text; vary co-batched neighbors/scheduling; measure activation deltas (L2, cosine, variance bands) at capture layer.

2) Deterministic-vs-default floor¶

Repeat under deterministic settings versus default mixed-precision fast path to estimate hardware nondeterminism envelope.

3) Prefix-caching coverage map¶

Log computed tokens vs cache-hit tokens by token class/position; quantify undercoverage in repeated prefixes.

4) SAE stability under harvest perturbation¶

Train SAEs from datasets differing only by scheduler/caching policy; compare feature overlap and reconstruction behavior.

These are cheap relative to full-scale harvest runs and convert “probably negligible” into measured bounds.

Practical guidance for production-grade harvesting¶

If you want completeness, disable prefix caching during harvest (or run a dedicated non-cache harvest pass).
Log compute coverage metadata (computed vs reused tokens) as first-class training-data provenance.
Document precision/determinism settings in every activation dataset release.
Treat serving config as part of the data method, not an infra footnote.

Bottom line¶

The reassuring part: continuous batching itself is not algorithmically “polluting” activations through cross-sample normalization or attention leakage.

The important part: inference-time systems choices still shape activation datasets through numerical execution path differences and, more significantly, token-coverage effects like prefix caching.

If interpretability is becoming infrastructure, these details belong in the methods section - not the appendix.

Evidence-quality note: This draft incorporates pre-publication fact-check corrections (2026-03-20), including attribution fixes and explicit separation of algorithmic equivalence from floating-point nondeterminism.

Code and experiment logs: github.com/Sohailm25/persona-circuits
Related: Persona Circuits: Progress & Findings

How Backpressure Silently Biases Activation Capture

2026-03-20T13:24:00-05:00

Goodfire’s write-up on harvesting billions of activations from frontier models presents a clean three-stage asynchronous design: GPU staging buffer, sidecar D2H transfer, CPU writer/compression to disk. It’s a strong architecture.

This piece is about what happens when reality stops being steady-state.

Across production inference, Ray data systems, and distributed training, I keep seeing the same pattern: when pipeline stages run at different effective speeds, transient pressure turns buffers into bias generators. The pipeline keeps running. Aggregate metrics still look good. But drops become correlated with data properties.

Core claim: for activation harvesting, backpressure-induced drops are often MNAR (missing not at random), which can systematically bias SAE training toward easy/common activations and away from rare/high-information ones.

in plain english¶

here is the non-jargon version.

imagine filming a football game, but your camera drops frames only during the fastest, most chaotic plays.

you still get plenty of footage, and the recording looks mostly fine.

but the exact moments you miss are the ones that mattered most.

that is what backpressure does in activation pipelines:

the system looks healthy on average
under bursts, buffers overflow and something gets dropped
drops are usually not random, so your dataset skews quietly

so this is less about "did we collect a lot of data" and more about "did we systematically miss the hard cases."

The three throughput mismatches¶

Stage 1 - GPU forward pass → GPU staging buffer¶

Device-to-device copy in HBM is usually not the limiter. Preallocated pools avoid allocator jitter. Metadata records are tiny.

Stage 2 - GPU staging → pinned CPU memory¶

D2H over PCIe is slower and burst-sensitive. Practical throughput is good, but no longer “free.” Correctness depends on pinned memory + explicit event discipline.

Reference numbers worth grounding here: PCIe Gen4 x16 is ~31.5 GB/s theoretical after encoding overhead; real-world 25–27 GB/s transfer rates are still excellent practical efficiency.

Stage 3 - CPU path (reshape/provenance/compress) → disk¶

This is where most pipelines become adversarial under load:

variable batch sizes
variable compression work per tensor
variable filesystem/device latency
background SSD garbage collection

Averages hide this. Bursts expose it.

Why average disk numbers are misleading¶

Goodfire-scale capture rates can look comfortable on paper per node. But writer load is bursty, not smooth. Consumer NVMe SLC caches can absorb short bursts and then collapse into lower sustained rates; enterprise media is better, but still exhibits periodic stalls/GC effects. For example, consumer 990 Pro-class behavior can show a steep post-cache drop (commonly into roughly ~1.4–1.8 GB/s ranges depending on SKU/test), while enterprise PM9A3 numbers vary materially by form factor and benchmark method.

So the relevant question is not “is mean write bandwidth enough?” It is:

Can the writer absorb worst-case burst duration without causing upstream queue saturation?

If not, your queueing policy becomes your sampling policy.

Compression is a systems tradeoff, not just a storage win¶

bf16 streams are compressible enough to matter, especially with exponent-aware approaches. That saves serious I/O and storage. But compression introduces CPU variance exactly where your pipeline is already vulnerable.

Practical design choice: - Inline compression: better steady-state bytes, more latency variance in the writer. - Raw write + offline compression: larger footprint, cleaner capture-time timing.

A robust compromise is a bounded compressed-output path with graceful fallback to raw blocks + metadata flag for later recompression when compression lag exceeds threshold.

Ring-buffer saturation is the fork in the road¶

When upstream outpaces downstream and the ring fills, you eventually choose one of three policies:

Block producer (protect integrity, sacrifice throughput)
Drop oldest (preserve recency, risk partial temporal bias)
Drop newest (preserve continuity of already-enqueued work)

There is no free default. But pretending saturation “won’t happen” is the worst option.

In activation capture, blocking can idle expensive GPU paths; dropping can silently bias the sample. Either way, this is a first-class design decision and should be explicit in both code and docs.

Why correlated drops are dangerous for interpretability¶

Random loss (MCAR) is usually tolerable at low rates. Correlated loss (MNAR) is different:

larger batches are more likely to drop
longer contexts are more likely to drop
harder-to-compress payloads are more likely to drop

Those properties are not neutral. They often track the exact slices where representations are richer, less frequent, or safety-relevant. So a “99.5% capture rate” can still produce a qualitatively biased dataset.

This is how pipelines fail silently in science: metrics say healthy, distribution says skewed.

Failure cascade (typical shape)¶

A representative cascade under transient I/O pressure:

disk stall / GC pause
writer falls behind
pinned-memory backlog grows
sidecar blocks or lags
ring fills
producer blocks or drops begin

Every component can be individually correct. The failure is in cross-stage coupling under burst.

What to measure (minimum instrumentation)¶

If you only add five things, add these:

Drop logs with metadata (batch size, token count, timestamp, path reason)
Drop probability vs batch properties (monotonicity check)
Captured vs source distribution checks (length/tokens/complexity)
Time-local capture-rate traces (not just global average)
Queue occupancy histograms + stall events per stage

If drop probability rises with workload intensity, you are in MNAR territory.

Minimal observability schema (recommended)¶

drop_reason (queue_full, writer_timeout, dma_backpressure, etc.)
batch_tokens
max_seq_len
queue_depth_stage1/2/3
disk_write_latency_p99
compression_backlog_bytes
cache_hit_ratio (if prefix caching is enabled)

Practical design guidance¶

Instrument drops, not only throughput.
Throughput tells you what survived; science needs to know what vanished.
Size for burst, not mean.
Buffer for transient peaks (e.g., several seconds at high-percentile ingress), not average rate.
Choose and document an explicit overflow policy.
Hidden defaults become hidden sampling bias.
Stress with adversarial distributions.
Uniform synthetic traffic is a false comfort. Include realistic long-tail sequence mixes.
Separate failure domains where possible.
Isolate activation write path from unrelated host pressure (OS temp, swap, logs, etc.).

Confidence labels on the core claims¶

Observed: transient saturation happens in heterogeneous-throughput pipelines.
Observed: queue policy determines which data is lost when pressure occurs.
Suggestive: activation-capture drops are often correlated with workload intensity.
Speculative (but testable): these correlated drops can degrade SAE quality specifically on rare/high-information regimes.

Evidence-quality note: All hard numbers should be treated as source-bounded, with model/SKU/form-factor caveats preserved (especially for SSD sustained-write claims and compression throughput estimates).

What remains open¶

I don’t yet have a universal “correct” buffer size or queue policy for trillion-parameter capture - those are deployment-specific and workload-specific.

I also don’t think the right stance is alarmist. Good teams likely carry hidden margins and unpublished mitigations. The point is simpler:

if your pipeline cannot prove that losses are either negligible or statistically non-correlated with sample properties, you don’t yet know what dataset you trained on.

For interpretability infrastructure, that’s not a minor ops detail. It is part of claim validity.

Why Prefill-Trained Interpretability Breaks in Decode

2026-03-20T13:20:00-05:00

i keep seeing the same pattern.

we train SAE dictionaries on prefill activations, then deploy those dictionaries during decode for steering, monitoring, and safety workflows. this is now normal practice. it also hides a core assumption most of us are not measuring directly:

features learned in prefill transfer cleanly to decode.

my claim is simple and opinionated: interpretability at frontier scale is not safety infrastructure until we measure prefill->decode drift explicitly.

if you want the broader context of how i think about this line of work, i keep related pieces in /writings/, including the experiment logs that informed this post.

in plain english¶

here is the non-jargon version.

imagine a student who studies using textbook problems, then is graded on a very different kind of exam.

the student is still smart. the memory is still there. but the question style changed, so performance can drift.

that is basically what is happening here:

prefill is the textbook phase (model reading human-written text)
decode is exam-time improvisation (model generating its own next steps)
SAE features are the study notes we built from textbook behavior

we then use those study notes to judge or steer behavior during exam-time generation.

sometimes this works great. sometimes it misses what changed.

so this piece is arguing for one simple standard: before we call this safety infrastructure, we should measure how much the model's internal behavior drifts from study-time to exam-time.

why this matters now¶

for small demos, you can get away with implicit assumptions.

for reasoning models and long-horizon agents, you cannot.

decode is where the important behavior happens: long chain-of-thought traces, tool decisions, multi-step planning, and safety-relevant branching. if your interpretability stack is calibrated on prefill but trusted during decode, your confidence intervals should be different by default.

that is not anti-SAE. i use SAE workflows myself. this is about calibration discipline.

i am writing this from both sides¶

on infra: i have run production inference systems where prefill and decode had to be optimized separately because they behave differently at the hardware and scheduler level.

on research: in my persona-circuits work on Llama-3.1-8B-Instruct, i extracted contrastive directions from prefill activations and got robust steering during generation, but negative sufficiency outcomes under stricter tests.

so yeah, this is not abstract philosophy. this is where the paper story and the production story collide.

prefill vs decode is not just a latency story¶

everyone knows the basic systems split:

prefill is compute-bound (parallel token processing, high Tensor Core use)
decode is memory-bandwidth-bound (token-by-token, repeated reads from HBM)

in one QSR deployment, this split dictated architecture:

prefill for 2k to 4k-token prompts sat around 100 to 200ms
decode sat around 15 to 30ms/token
prefix caching (SGLang RadixAttention) cut prefill latency by 60 to 80%
speculative decoding gave mixed outcomes (50 to 60% acceptance on conversational traffic, sometimes worse p50 after draft overhead)

but this is the part people underweight: performance asymmetry is a symptom of representational asymmetry.

during prefill, the model is conditioned on human-written tokens.

during decode, from generated token 1 onward, the model is conditioned on its own sampled outputs.

that is exposure-bias territory. different conditioning distribution, potentially different activation geometry.

evidence today: strong hints, incomplete measurement¶

direct signal: Goodfire R1 observations¶

Goodfire has already reported strong feature-distribution shift across prompt, thinking trace, and final response in R1-style reasoning behavior.

that is not a tiny footnote. that is three internal regimes under one shared vocabulary.

indirect signal: generation distribution work¶

exposure-bias literature and degeneration/anisotropy results keep pointing in the same direction: generated trajectories and human-written trajectories are statistically and representationally different enough to matter locally, even when aggregate metrics look fine.

aggregate 3% gaps can still hide ugly local failures at high-entropy and decision-critical positions.

production signal: speculative acceptance behavior¶

i have seen acceptance rates vary materially by domain. code often accepts more than open-ended conversational traffic. that alone suggests decode dynamics are structured in ways our usual training objectives do not fully capture.

plausible mechanism: outlier behavior can amplify regime mismatch¶

transformer outlier channels are well documented.

SAE sensitivity to outlier-heavy regions is also documented.

so the hypothesis is straightforward (and still speculative): if decode-time self-conditioning shifts outlier structure at key positions, prefill-trained sparse dictionaries can become miscalibrated exactly where steering confidence matters most.

not guaranteed. not proven. absolutely worth measuring.

what this changes for steering claims¶

if prefill and decode diverge materially, then prefill-calibrated steering can drift in at least three ways:

activation-rate drift: feature sparsity levels move (2% in prefill becomes 0.5% or 5% in decode)
directional drift: feature directions rotate enough to reduce precision
mechanism split/merge: one prefill feature maps onto multiple decode-time mechanisms

this is why "it worked in a demo" is weaker than people think for safety framing.

why this was a live confound in my persona-circuits results¶

in my own runs, steering signal was real. sufficiency signal was weak to negative.

that leaves two live interpretations:

i missed causal components (classic incompleteness)
i extracted components in prefill that do not fully mediate behavior during decode

my current pipeline cannot separate those cleanly.

that ambiguity is the point. if we do not instrument decode drift, we cannot tell whether we are missing components or missing regimes.

for related context on those experiments:

minimum viable measurement standard (what teams should actually run)¶

not a giant moonshot. a practical stack:

1) drift by position¶

capture matched prefill and decode activations. compare per-layer and per-position stats (mean, variance, kurtosis, outlier frequency).

2) reconstruction gap¶

evaluate prefill-trained SAE reconstruction on prefill vs decode (MSE, explained variance). track drift over generation depth and entropy bands.

3) feature overlap¶

compare top-k active feature sets across regimes (Jaccard, rank correlation).

4) steering effect decay¶

apply the same steering vector at token 1, 10, 50, 200. measure effect-size decay and instability.

5) sparse online probes¶

sample decode activations periodically (or entropy-triggered), compare against a prefill reference profile, and flag high-divergence trajectories.

method caveat: per-dimension KL is a useful diagnostic proxy, not a full joint-geometry proof. use it for triage, not final truth claims.

the tooling problem is real¶

most mainstream mech-interp tooling is microscope mode: fixed input, full capture, offline analysis.

decode monitoring is telescope mode: streaming capture, low overhead hooks, KV-aware instrumentation, and alerting logic that does not tank latency.

we have pieces of this. we do not have a clean, common workflow that teams can adopt quickly.

that is an infra gap, not just a research gap.

closing position¶

prefill-first capture was a good engineering choice for scale.

it is not enough for the safety claims people now attach to steering and monitoring.

we are still using microscopes calibrated for one room to study another. sometimes that works. sometimes it probably does not. right now we are mostly guessing where that boundary sits.

if your pipeline cannot report prefill-vs-decode reconstruction drift by position, it is not safety infrastructure yet.

Code and experiment logs: github.com/Sohailm25/persona-circuits

The Forge #8

2026-03-18T00:00:00-05:00

The Forge #8 | March 18, 2026¶

this is a catch-up issue (mar 6 → mar 18), intentionally longer than usual. the main pattern: model capability still matters, but practical advantage is moving toward the systems around the model — evaluation, context management, agent control loops, and distribution.

1) AGENTIC CODING IS MOVING FROM NOVELTY TO DEFAULT¶

over this window, “ai-assisted coding” looked less like a side tool and more like baseline practice for serious teams.

more operators are sharing practical multi-agent workflows (parallel tasking, role split, checkpointing).
open role libraries and reusable agent patterns keep expanding.
enterprise anecdotes are shifting from pilot mode to regular weekly throughput.
one widely-circulated Uber data point framed this clearly: high volume of weekly code changes authored by internal agent systems, with broad engineer usage.

the transition is straightforward: from “copilot helps me type” to “agents are part of the delivery system.”

🔗 Karpathy multi-agent experiment
🔗 OpenAI computer environment lessons
🔗 Subagent role-library momentum
🔗 Uber agentic SWE adoption signal

2) THE MODEL RACE IS NOW A SYSTEMS RACE¶

the clearest theme from builders this period: raw base-model quality is no longer the only constraint. orchestration quality, eval discipline, context plumbing, and product loops are driving real-world results.

you can see the shift in how people talk: - less obsession with single benchmark screenshots. - more focus on post-training evals and operational reliability. - more emphasis on end-to-end outcomes over isolated model IQ.

model deltas still matter. but stack design increasingly decides who wins in production.

🔗 PostTrainBench release thread
🔗 Benchmark commentary
🔗 System-vs-model framing signal

3) BIG-LAB PLATFORM MOVES (SUBSTANCE OVER NOISE)¶

Anthropic¶

launched The Anthropic Institute, expanding into public-interest, policy, and economic research framing around advanced AI.
practical effect: narrative broadens from “ship model” to “shape long-horizon institutional posture.”

🔗 Anthropic Institute announcement

OpenAI¶

published technical notes on running durable computer-access loops: tighter execution cycles, richer environment context, and guarded network access.
useful because it reflects implementation decisions rather than marketing abstractions.

🔗 OpenAI Devs: computer-use engineering notes

Google¶

announced an urban flash-flood forecasting update plus Groundsource methodology.
important category: AI systems producing measurable public-safety lead time.

🔗 Sundar on flash-flood model + Groundsource

4) INFRA + HARDWARE: THE PACE IS STILL HIGH¶

the infrastructure layer keeps widening beyond model training clusters into edge/local execution and rendering pipelines.

NVIDIA continued pushing neural rendering into mainstream gaming narratives (DLSS 5).
builders keep emphasizing inference economics and deployment architecture as first-order concerns.
routing ecosystems continue surfacing long-context and multimodal experiments quickly.

the implication: competitive edge increasingly comes from architecture and unit economics, not just model access.

🔗 NVIDIA DLSS 5 announcement
🔗 OpenRouter stealth model chatter

5) ROBOTICS + EMBODIED AI: STILL EARLY, CLEARLY ACTIVE¶

the robotics stream remains noisy, but cadence is undeniably up.

more autonomous behavior demos are showing up publicly (including failures, which are often the useful part).
defense and industrial autonomy narratives are strengthening.
humanoid content is still mixed quality, but no longer sporadic.

we’re not at broad deployment. we are past “occasional research theater.”

🔗 Humanoid/robotics battlefield testing signal
🔗 Autonomous behavior demo thread

6) PRODUCT + DISTRIBUTION SIGNALS¶

outside core research, several practical trends stood out:

AI-mediated workflows are entering mainstream categories (real estate, SMB ops, creator pipelines).
“AI discoverability” is starting to look like a distinct growth channel.
many teams are converging on a pattern: high-volume AI first pass, human editorial control at decision boundaries.

in plain terms: moat = repeatable outcomes + workflow ownership, not “we use an API.”

🔗 AI-assisted transaction workflow example
🔗 AI search visibility business model chatter

RESEARCH NOTES (HIGH-SIGNAL ITEMS)¶

PostTrainBench: useful direction for measuring post-training effects in agentic automation settings.
OvertonBench (pluralistic alignment): meaningful movement toward evaluating value plurality rather than single-axis alignment.
Autonomous cyber operations framing: more serious analysis of what changes when AI systems execute multi-step cyber tasks independently.

🔗 PostTrainBench
🔗 OvertonBench / ICLR acceptance
🔗 AI autonomous cyber ops thread

QUICK HITS¶

Midjourney started early V8 testing (faster, better prompt following, native 2K).
🔗 https://x.com/midjourney/status/2034250982260822323
Anthropic/Google/OpenAI parity debates intensified; attention is moving to shipping cadence and product integration.
🔗 https://x.com/peterwildeford/status/2032288931610341789
OpenClaw/agent tooling gained visibility in practical team workflows (including standup-style usage).
🔗 https://x.com/RoundtableSpace/status/2032116197306427397
Local small-model experiments kept improving (including near real-time multimodal demos on consumer hardware).
🔗 https://x.com/stevibe/status/2035734613021626641

OPERATOR TAKE¶

if you’re building right now, prioritize:

reliability loops (retry, verify, rollback, eval gates)
context architecture (what gets loaded, when, and why)
cost-aware routing (small model first, escalate intentionally)
human checkpoints at high-risk boundaries
distribution + data flywheel as the long-term moat

teams that treat models as components — not magic — will keep compounding.

The Forge | Issue #8 | March 18, 2026

Persona Circuits: Progress & Findings (3-17-2026)

2026-03-17T13:30:00-05:00

Persona-like steering results are often easy to demonstrate and hard to interpret mechanistically. This write-up is organized to answer one practical question for readers: what is actually supported today, and what is not?

The goal is to make the claim boundary clear before diving into technical details.

Companion branch report: /research/experiments/glp-persona-circuits-current-state/

Why This Matters¶

If steering directions are robust but mechanistically diffuse, then “we can steer behavior” and “we found a causal persona circuit” are different scientific statements. This project is about separating those statements rigorously.

Evidence Scope¶

This post summarizes the current mainline persona-circuits evidence stack:

end-to-end pipeline development and validation
core-lane results (sycophancy, reframed machiavellian_disposition, and hallucination as a weak lane)
trait-lane expansion (trait_lanes_v2)
concentration, distinctness, and causal-control analyses
bounded claim-grade sufficiency checks

The scope is intentionally narrow: claims here are only for the evaluated model/protocol regime and current operationalizations.

Project Context (for new readers)¶

Persona-circuits is an ongoing mechanistic interpretability project testing whether persona-like behavioral steering directions in LLMs correspond to sparse, causally meaningful internal structure. The current evidence supports robust steering and partial concentration structure, but several stronger causal claims, especially sufficiency and distinctness, remain mixed or negative under current protocols.

What We Set Out To Test¶

The central question was:

Can we move from “this vector steers behavior” to “this behavior is mediated by a sparse, causally meaningful circuit”?

Prior work already supports key pieces of that story:

representation engineering / ActAdd / CAA-style steering directions
persona-vector decomposition into interpretable feature groupings
circuit-tracing cases where specific behaviors are mechanistically localizable

The gap was the bridge between these ideas in one integrated, claim-disciplined workflow.

Explicit Hypotheses¶

To make the evaluation criteria explicit, the project tracks five hypotheses:

H1 (Concentration): persona-like steering directions are mediated by non-trivial concentrated internal structure, not a fully diffuse effect.
H2 (Necessity): ablating identified high-contribution components should materially reduce the targeted behavior.
H3 (Sufficiency): preserving/activating identified components alone should retain enough behavior to support a bounded sufficiency claim.
H4 (Cross-persona structure): persona-related structure should show predictable shared-vs-distinct organization across personas beyond noise-level overlap.
H5 (Routing mediation): routing-level evidence should show more-than-exploratory support for persona mediation under current controls.

What We Built¶

We now have a full stack covering:

infrastructure and prompt generation
contrastive vector extraction
upgraded dual-judge behavioral validation
held-out and control evaluations
SAE decomposition and concentration analysis
causal ablation analyses for necessity/sufficiency-style questions
cross-persona and router exploratory tests
trait-lane expansion when the original lane set looked bottlenecked

The original trio (sycophancy, evil, hallucination) evolved during the project:

sycophancy remained the clearest anchor
evil was reframed to machiavellian_disposition due to refusal confounding
hallucination remained weak as a persona-like lane and moved toward negative-control status

What Held Up¶

1) Robust steering directions are real¶

We consistently extracted directions that changed behavior in both the core line and trait-lane branch. That does not prove construct validity or causal distinctness, but it does rule out a pure-noise interpretation.

2) Core lanes show non-flat concentration structure¶

Stage 3 attribution concentration was meaningfully non-flat:

sycophancy: Gini 0.5771, top-20% mass 0.5298
machiavellian_disposition: Gini 0.6476, top-20% mass 0.6173

This is best interpreted as partial support with caveats, not full confirmation of a sparse-circuit claim.

3) Trait-lane expansion produced discriminative evidence¶

The branch screened:

assistant_likeness
honesty
politeness
persona_drift_from_assistant
lying
optimism

The branch did meaningful scientific work because it differentiated “we chose weak traits” from “strong steering does not automatically imply independent persona mechanisms.”

Where the Strongest Positive Story Weakened¶

`politeness` looked strong, then failed distinctness¶

politeness produced strong steering and passed several robustness checks. However, in deeper validation it repeatedly bled into assistant_likeness at near-parity levels.

Representative reads:

prompt-last target effect: 46.33
prompt-last assistant-likeness bleed: 47.23
prompt-last bleed ratio: 1.0194

Follow-up checks did not resolve this:

paraphrase retention stayed high (0.9881 cosine retention)
orthogonalized residual retained force (31.4) but still failed distinctness
response-mean follow-up shifted layer/magnitude, not verdict

Current interpretation: politeness is a strong steering direction, but under current protocol it is better described as assistant-style modulation than an independently promotable persona lane.

`lying` became a cleaner negative finding¶

lying survived early screening but degraded under deeper testing, especially in external smoke behavior where reversibility and construct alignment failed.

Key lesson: stable extraction can coexist with poor construct validity.

`honesty` remains unresolved but non-trivial¶

honesty currently looks asymmetric and RLHF-shaped rather than a clean symmetric honesty/dishonesty axis. This is less tidy for the original persona-circuit narrative but scientifically important.

Hypotheses: Current Read¶

H1 (concentration / sparse-structure support)¶

Partial support with caveats.

H2 (necessity)¶

Mixed-to-weak under current thresholds; below claim-grade confidence.

H3 (sufficiency)¶

Negative under current operationalization. In bounded full-complement circuit-only execution, behavior degraded into repetitive, low-capability outputs.

At completed doses:

0.25: preservation 0.2857
0.50: preservation 0.3571

H4/H5 (cross-persona and router)¶

Weak-negative / exploratory null under current tests.

Claim Boundary¶

What is established:

robust behavioral steering exists
concentration is non-flat in important lanes
stronger distinctness and sufficiency claims are not currently supported

What is not established:

claim-grade sparse-circuit sufficiency
cleanly separable independent persona lanes for key new candidates
strong router-level persona mediation evidence

Why This Result Is Still Valuable¶

The main contribution is not a clean positive bridge from steering vectors to circuit claims. The contribution is sharper:

robust steering is a lower bar than mechanistic distinctness
assistant-prior structure appears broader and more absorbing than expected
stronger controls improved the quality of negative findings

That is scientifically useful and should be reported directly, not hidden behind optimistic framing.

Methods Snapshot¶

Primary model: meta-llama/Llama-3.1-8B-Instruct
Primary seed in the current closeout stack: 42
Core prompt budgets: extraction 100 pairs/trait, behavioral validation 50 prompts/trait, circuit analysis 20 prompts/trait, ablation validation target 100 prompts/trait
Judge setup: primary claude-sonnet-4-6, secondary calibration claude-opus-4-6; audited on 90 prompt pairs / 180 scored responses per judge with mean kappa 0.7727
Main claim thresholds from config/policy: necessity 0.80, sufficiency 0.60, significance 0.01, A12 >= 0.71, stability Jaccard 0.30
Trait-lane deeper-validation profile: 30 held-out prompts/lane split into 10 sweep / 10 confirm / 10 test, relative coherence max-drop gate 10.0, cross-trait bleed enabled against sycophancy and assistant_likeness

This is the minimum reproducibility payload for the claims in this post. It is not the full methods section.

Uncertainty and Variance Notes¶

I do not want the averages in this post to imply false precision.

The strongest concentration claims are based on 50 prompts per core trait in Stage 3.
The distinctness failure for politeness is based on 10 held-out test prompts in the deeper-validation runs, so the effect is real enough to flag but still small-sample.
Judge reliability is not hand-waved here: the audit covers 90 prompt pairs with mean kappa 0.7727, but the manual human concordance layer is still only a low-power sanity check (n=15).
The H3 closeout is a real negative under the executed protocol, but it is still one bounded operationalization, not a universal impossibility proof for sufficiency-style work.
Multi-seed replication remains limited. The current closeout stack is still centered on the seed-42 artifact family.

Representative variance for the headline politeness lane:

Run	Test steering mean	Test steering std	Test reversal mean	Test reversal std	`n_test_prompts`
prompt-last deeper validation	`40.43`	`16.21`	`5.90`	`5.94`	`10`
response-mean deeper validation	`30.93`	`16.86`	`7.10`	`4.86`	`10`
orthogonalized prompt-last	`26.93`	`13.71`	`4.47`	`5.78`	`10`

That is enough variance that I am comfortable writing “strong steering, weak distinctness,” but not enough to inflate these into stronger claims than the protocol earned.

Artifact Index For Numeric Claims¶

Core concentration is non-flat
Numeric claim: sycophancy Gini 0.5771, top-20% mass 0.5298; machiavellian_disposition Gini 0.6476, top-20% mass 0.6173
Artifact: week3_stage3_activation_delta_attribution_20260304T164549Z.json
Judge reliability is nontrivial
Numeric claim: 90 prompt pairs, 180 responses/judge, mean kappa 0.7727
Artifact: week2_judge_reliability_audit_packet_20260314T160930Z.json
politeness prompt-last fails distinctness
Numeric claim: target effect 46.33, assistant-likeness bleed 47.23, bleed ratio 1.0194
Artifact: week2_trait_lane_deeper_validation_validation_20260312T134851Z.json
politeness response-mean still fails distinctness
Numeric claim: target effect 38.03, assistant-likeness bleed 39.7, bleed ratio 1.0438
Artifact: week2_trait_lane_deeper_validation_validation_20260313T182007Z.json
Orthogonalization did not rescue politeness distinctness
Numeric claim: target effect 31.4, assistant-likeness bleed 32.83, bleed ratio 1.0456
Artifact: week2_trait_lane_orthogonalization_validation_20260313T151437Z.json
H2 strict claim-grade necessity fails
Numeric claim: best zero-ablation mean reduction 0.5627; best resample mean reduction 0.2585; all below 0.80 necessity bar
Artifact: week3_stage4_policy_decision_packet_20260310T142000Z.json
H3 fails under executed full-complement protocol
Numeric claim: preservation 0.2857 at dose 0.25; 0.3571 at dose 0.50; coherence drop 73.2; capability proxy 0.0
Artifact: week3_stage4_behavioral_sufficiency_claimgrade_trancheA_closeout_20260311T1919Z.json
H4 is weak-negative
Numeric claim: early Jaccard 0.1696, late Jaccard 0.1236, delta 0.0460, proposal pattern pass false
Artifact: week3_stage5_policy_decision_packet_20260310T200937Z.json
H5 is exploratory null / weak negative
Numeric claim: n_tested=62, n_rejected=0, min_q_value=0.0465
Artifact: week3_stage5_policy_decision_packet_20260310T200937Z.json

Next Steps¶

Default next move is synthesis, not breadth expansion:

document supported vs unsupported claims clearly
publish mixed/negative findings as first-class evidence
limit new experiments to redesign-level questions (assistant-basin factorization, truthfulness reformulation under RLHF asymmetry, less-destructive sufficiency operationalizations)

Current Bottom Line¶

We found real persona-like steering structure. But when we pushed toward stronger causal and mechanistic claims, the story became narrower, messier, and more assistant-shaped.

That is not the cleanest possible narrative. It is the most accurate one from the current evidence.

Project Links¶

Research hub: https://sohailmo.ai/pages/research/
Code and artifacts: https://github.com/Sohailm25/persona-circuits

Persona Circuits: Exploring GLP Application

2026-03-17T12:00:00-05:00

The GLP branch was designed to answer one concrete question: when steering fails, are we seeing a bad semantic direction or a geometrically invalid edited state?

This post is organized to answer that question directly before moving into branch details.

Why This Branch Matters¶

If a repair prior can preserve semantic edits while restoring plausibility, then some apparent circuit failures may be geometric rather than semantic. If it cannot, we should be more cautious about using latent priors as mechanistic disambiguation tools.

GLP Reference and Why We Tried It¶

Paper context: Luo et al., Generative Latent Priors (GLP). Reference link: https://arxiv.org/html/2602.06964v1
High-level idea: train a latent denoiser/prior over activations, then use that prior to project edited activations back toward plausible manifold states.
Why we applied it here: in persona-circuits, we needed to separate two failure modes:
semantic insufficiency (the steering direction/circuit is wrong), versus
geometric invalidity (the edited activation is off-manifold).

GLP was a natural candidate because, in principle, it can repair geometry while preserving intended directional edits.

Evidence Scope¶

This post covers one bounded branch question:

Can a learned latent prior (GLP) repair steered activations in a way that preserves intended semantic edits while improving geometric validity?

Included evidence:

public-checkpoint transfer tests
matched model/layer GLP checkpoints
Week 2 steering sidecar controls
geometry diagnostics (repair/edit ratio, retention cosine)
conditional pilot objective
mixed clean+edited training path (trained; behavioral evaluation pending)

Claims are limited to this branch, in this model/layer/protocol setting.

Project Context (for new readers)¶

Persona-circuits is an ongoing mechanistic interpretability project testing whether persona-like behavioral steering directions in LLMs correspond to sparse, causally meaningful internal structure. The current evidence supports robust steering and partial concentration structure, but several stronger causal claims (especially sufficiency and distinctness) remain mixed or negative under current protocols.

Why We Ran This Branch¶

In persona-circuits, the central challenge is not only moving a trait score. It is distinguishing:

semantic insufficiency (the direction/circuit is wrong), versus
geometric invalidity (the intervention pushes activations off-manifold)

GLP was promising as a disambiguation tool: repair geometry without erasing meaningful directional edits.

Explicit Branch Hypotheses¶

This branch tests three explicit hypotheses:

G1 (Selective repair): GLP should improve selected steering outcomes more than baseline or random-control GLP conditions.
G2 (Direction preservation): GLP repair should keep substantial alignment with the intended edit direction while improving plausibility.
G3 (Train/eval match sensitivity): mixed clean+edited training should improve selectivity relative to clean-only training if distribution mismatch is the main bottleneck.

Naming Note (Consistency With Mainline)¶

This branch contains historical metrics labeled evil from earlier branch-stage naming. In the mainline synthesis, that construct is reframed as machiavellian_disposition due to refusal confounding. The GLP tables below preserve original branch labels for traceability.

What We Built¶

This became a full sidecar rather than a single checkpoint test:

GLP runtime (unconditional + conditional)
geometry and next-token diagnostics
Week 2 steering evaluation with controls
Week 3 sufficiency sidecar support
neutral corpus generation
memmap activation export pipeline
Modal training launcher
paired conditional data path
mixed clean+edited training path

Controls that matter most:

selected_raw
selected_glp
baseline_glp_control
random_glp

Main Findings So Far¶

1) Public GLP checkpoint did not transfer cleanly¶

In this setting, the released checkpoint produced large local predictive distortion and overly strong GLP-only/random controls. That behavior is inconsistent with a selective repair interpretation.

2) Matched checkpoints helped stability, not selectivity¶

Model/layer-matched checkpoints were less pathological than the public checkpoint. But the key branch question remained unresolved:

Does GLP help selected steering more than baseline/random controls?

So far, mostly no.

3) Control competitiveness remains too high¶

Representative validated reads (compact format):

Matched response_all — evil
selected_raw: -59.6
selected_glp: -54.65
raw coherence: 34.75
glp coherence: 33.08
baseline_glp_control: -61.5
random_glp: -59.5
Matched response_all — sycophancy
selected_raw: -71.9
selected_glp: -72.25
raw coherence: 50.98
glp coherence: 44.98
baseline_glp_control: -77.3
random_glp: -74.75
Matched response_last — evil
selected_raw: -60.25
selected_glp: -54.15
raw coherence: 34.4
glp coherence: 31.93
baseline_glp_control: -62.45
random_glp: -58.67
Matched response_last — sycophancy
selected_raw: -72.15
selected_glp: -71.0
raw coherence: 50.8
glp coherence: 49.45
baseline_glp_control: -73.5
random_glp: -75.65

Interpretation: GLP effects are not selective enough relative to nuisance controls.

4) Geometry suggests generic projection behavior¶

Observed ranges across matched runs:

repair-to-edit ratio: ~2.0–2.16
edit-retention cosine: ~0.39–0.42

In practice, GLP often makes moves larger than the original edit while preserving under half of its directional alignment. This looks more like a generic denoising projector than direction-preserving repair.

5) Better optimization did not resolve selectivity in the matched `response_all` lane¶

After addressing undertraining critiques and improving validation loss materially in the matched response_all lane, behavior-level selectivity still did not improve enough. That weakens “insufficient optimization” as the main explanation for that lane, even though it does not fully settle the training-adequacy question for response_last.

6) Conditional pilot worked technically, but likely targeted the wrong objective¶

prompt_last -> response_last conditional training functioned as infrastructure, but likely optimized a normal-response mapping rather than edit-preserving repair.

Most Important New State¶

We now have a mixed clean+edited response_last checkpoint that directly addresses the clean-train / edited-eval mismatch criticism.

Dataset:

8600 total
6880 clean (80%)
1720 edited (20%)
balanced across:
sycophancy_plus
sycophancy_minus
evil_plus
evil_minus

Training note:

validation loss improved from 1.877 (clean-only) to 1.855 (mixed), under matched compute

This is not the branch result by itself. The key pending result is Week 2 behavioral evaluation on this mixed checkpoint.

Claim Boundary¶

What is supported now:

naive reuse of the released GLP checkpoint is not supported for this lane
matched GLP reduces gross mismatch pathology
current GLP behavior is still nonselective in the Week 2 branch framing
failure mode appears structured (generic projection), not random

What is not yet supported:

selective, direction-preserving repair at claim-grade confidence
broad anti-GLP claims outside this task regime

Methods Snapshot¶

Primary model and hook: meta-llama/Llama-3.1-8B-Instruct, layer 12, blocks.12.hook_resid_post
Validated Week 2 comparison setting: 20 held-out prompts per trait, max_new_tokens=32, temperature=0, random_direction_draws=3
Traits in the branch-local claim lane: sycophancy and historical evil branch naming
Judge setup: claude-sonnet-4-6 for scored branch evaluation; the mainline dual-judge audit uses claude-opus-4-6 as the secondary calibration model
Training defaults for the matched checkpoints: lr=5e-5, batch_size=512, validation_fraction=0.05, cosine scheduler with warmup, 3 epochs in the validated comparison runs
Training datasets used in the current post:
response_all: 92,422 activations from 3,000 prompts
clean response_last: 8,600 activations from 8,600 prompts
mixed response_last: 8,600 activations with 20% edited samples
Mixed-data composition: 1,720 edited samples, balanced 430 each across sycophancy_plus, sycophancy_minus, evil_plus, evil_minus

Branch interpretation rule of thumb:

a checkpoint is only treated as claim-aligned when model and layer match the target lane
behavioral repair is only interesting if selected_glp beats the nuisance baselines, not merely if it looks okay in isolation
NLL / geometry metrics are treated as diagnostics, not as validated behavioral surrogates

Uncertainty and Variance Notes¶

The current branch read is mean-based, but the prompt-level variance is large enough that I do not want readers to infer more stability than the artifacts support.

Validated 20-prompt runs:

Checkpoint	Trait	GLP-minus-raw effect delta mean	Std	Range	GLP-minus-raw coherence delta mean	Std	Range
matched `response_all`	`sycophancy`	`-0.35`	`11.54`	`[-13, 23]`	`-6.00`	`9.55`	`[-27.0, 11.5]`
matched `response_all`	`evil`	`4.95`	`13.50`	`[-12, 37]`	`-1.68`	`8.94`	`[-20.0, 16.5]`
matched `response_last`	`sycophancy`	`1.15`	`9.02`	`[-18, 20]`	`-1.35`	`11.20`	`[-28.5, 21.5]`
matched `response_last`	`evil`	`6.10`	`15.38`	`[-17, 52]`	`-2.48`	`9.08`	`[-17.0, 12.0]`

What this means:

GLP is not uniformly harmful or uniformly helpful across prompts.
The current failure mode looks unstable and generic, not cleanly trait-selective.
Random-direction controls are now averaged over 3 draws, which is better than the original single-draw control, but still not enough to claim seed-level stability.

Proxy-metric warning:

evil NLL-vs-coherence delta Spearman: 0.096
evil repair-ratio-vs-coherence delta Spearman: -0.218
sycophancy NLL-vs-coherence delta Spearman: 0.298

Those are too weak to justify treating the proxy metrics as behavioral stand-ins.

Artifact Index For Numeric Claims¶

Released checkpoint failed to transfer cleanly
Numeric claim: large distortion on evil alpha-3 diagnostic: delta_target_nll_vs_clean=4.725, kl_clean_to_hooked=5.184
Artifact: week2_glp_sidecar_validation_20260311T012700Z_evil_frontier_alpha3_nlldiag_20260310a.json
Matched response_all still looked nonselective
Numeric claim: evil: selected GLP -54.65 vs baseline -61.5 vs random -59.5; sycophancy: selected GLP -72.25 vs baseline -77.3 vs random -74.75
Artifact: week2_glp_sidecar_validation_20260312T151500Z_matched_responseall_val3e_rowdiag20_20260312a.json
Matched response_all geometry looked like generic projection
Numeric claim: repair-to-edit ratio 2.0092 / 2.0222; retention cosine 0.4182 / 0.4235
Artifact: week2_glp_sidecar_analysis_20260312T155851Z.json
Matched response_last still looked nonselective
Numeric claim: evil: selected GLP -54.15 vs baseline -62.45 vs random -58.67; sycophancy: selected GLP -71.0 vs baseline -73.5 vs random -75.65
Artifact: week2_glp_sidecar_validation_20260313T135047Z_matched_responselast_val3e_rowdiag20_20260313b.json
Matched response_last geometry still looked like generic projection
Numeric claim: repair-to-edit ratio 2.1577 / 2.1620; retention cosine 0.3932 / 0.3948
Artifact: week2_glp_sidecar_analysis_20260313T135951Z.json
Better response_all optimization did not translate into selective repair
Numeric claim: validation loss improved to 1.5969 at epoch 2 and 1.5795 at epoch 3, but Week 2 selectivity still failed
Artifacts: train_glp_matched_modal_20260312T133750Z_responseall_val3e_20260312a.json; week2_glp_sidecar_validation_20260312T151500Z_matched_responseall_val3e_rowdiag20_20260312a.json
Clean response_last remained a low-step regime
Numeric claim: 8,170 train examples, 430 val examples, 15 gradient steps/epoch, final val loss 1.8770
Artifact: train_glp_matched_modal_20260313T023542Z_response_last_tranches1234_val3e_20260312a.json
Mixed clean+edited checkpoint is now trained
Numeric claim: 8,600 total samples, 1,720 edited (20%), balanced 430 per edit label, final val loss 1.8551
Artifacts: glp_export_mixed_edited_memmap_dataset_20260315T075043Z_response_last_mixed20_tranches1234_20260313a.json; train_glp_matched_modal_20260315T075513Z_response_last_mixed20_tranches1234_val3e_20260313a.json

What To Do Next¶

Evaluate Week 2 behavior on the mixed-trained checkpoint.
If still nonselective, substantially lower confidence in unconditional GLP for this application.
Only if mixed training improves selectivity should we invest in larger targeted-objective work (edit-fraction sweeps, stronger conditional/edit-aware repair objectives).

Current Bottom Line¶

We tested GLP as a geometry disambiguation tool for persona steering. The public checkpoint failed to transfer cleanly in this setting. Matched checkpoints were more stable but still too nonselective, with geometry consistent with generic projection behavior. The mixed clean+edited checkpoint is now trained and creates a real next inflection test; its behavioral evaluation is the decisive next step.

Project Links¶

Research hub: https://sohailmo.ai/pages/research/
Code and artifacts: https://github.com/Sohailm25/persona-circuits

Stop Putting Allah in a Box

2026-03-11T08:32:00-05:00

I've been sitting with two thoughts lately. Both feel connected, and both feel like things I needed to hear years ago. I'm going to talk about them through the lens of my faith because that's where they live for me, but I think the core of both applies to anyone who's ever caught themselves shrinking their own hope.

The Subtle Sin of "Protecting" Yourself¶

Here's something I've done for years: whenever I start wanting something badly, a role, an opportunity, a future I can almost taste, I pull back. I tell myself, don't get too excited. Don't get attached. If it doesn't work out, you'll only hurt more.

It feels like wisdom. It feels like emotional intelligence. It feels like you're being a responsible adult who manages expectations.

But I've started to realize it might be something else entirely. It might be a limiting belief about Allah.

Think about it. When we dampen our own hope, when we refuse to let ourselves feel the full weight of optimism, what are we really saying? We're saying, Allah probably won't come through on this one. We're making an assumption about His plan before He's revealed it. We're drawing a circle around what He's capable of and deciding the boundaries for Him.

And I say this as someone who has practiced this "self-protection" religiously. Right now, I'm in the thick of pursuing something I've wanted for a long time. I've done the work. I've tied my camel as tightly as I know how, which is an expression in Islam that basically means: do everything in your power first, then trust God with the rest. And now I'm in the in-between. That stretch of time where there's less for me to do and more for me to trust. And my reflex, every single day, is to whisper to myself: don't get your hopes up too much.

But why not?

Why not be hopeful? Why assume that Allah would bring something into our lives designed to break us? We're told He is the best of planners. We're told there is khair, goodness, in every outcome. Even if we don't get the thing, He will take care of us. I believe that.

But I'm not even talking about the aftermath right now. I'm talking about the waiting. The period between effort and outcome. The space where tawakkul, deep trust in God, is supposed to live. And what I've noticed, in myself and in so many people around me, is that we fill that space with small, quiet acts of doubt. We try to be "realistic." But in that realism, we are subtly saying Allah wouldn't do this for me or this is too big to actually happen.

Yes, we tie our camel. Yes, there's a reality to how things tend to unfold. But why do we assume miracles aren't on the table?

I'm trying to dream bigger. Not recklessly, but with genuine trust. I want to be someone who expands what could happen to the ends of the horizon, not someone who shrinks it down to what feels safe. Allah's capacity to bless us is not bound by what we think is probable. His love for us is not limited by our fear of disappointment.

So I'm practicing something new: letting myself hope fully. Letting the excitement exist without immediately reaching for the emergency brake.

It's harder than it sounds.

The Other Thing: Reach Out¶

The second thought is simpler but no less important.

We don't reach out to people enough. We just don't.

When you see someone cross your mind, a friend, a colleague, someone you haven't spoken to in months, pray for them. In my tradition we call it dua, but the form doesn't matter. Think of them, wish good for them, and then tell them you did.

It sounds small. But the downstream effects are real. They feel seen. They feel enabled. They're reminded that someone out there is thinking about them and asking God to bless them. And maybe, you don't do it for this reason, but maybe, they pray for you too.

Arthur Brooks talks often about how our deepest fulfillment doesn't come from achievement or status. It comes from the people we know and keep close. The relationships we invest in. That's the stuff that builds contentment, purpose, meaning. The things no career milestone can substitute for.

So be prolific in reaching out. It's not cringe. It's not weird. It's one of the most human things you can do.

And I'm saying this as someone with a genuinely terrible habit of not getting back to people. Of ghosting. Of letting conversations die because I got busy or distracted or told myself I'd reply later. I'm preaching to myself before anyone else.

But I'm trying. On both fronts. Trusting Allah with the full breadth of what He can do, and trusting the people in my life with the full breadth of what I feel for them.

Both require the same thing, really: stop holding back.

The Paradox of First Principles (And Why Nobody Wants to Hear That You're in Pursuit of Greatness)

2026-03-10T09:35:00-05:00

There's a moment in Nicholas Carlini's essay "How to Win a Best Paper Award" where he describes his process for coming up with research ideas. Read everything, he says. Absorb the entire body of scientific literature in your field. Understand what's known, what's been tried, what failed and why.

Then forget all of it.

Not because it doesn't matter (it does, deeply) but because if you only think in terms of what's already been done, you'll never produce anything worth doing. The existing literature can poison your thinking. Once you've seen ten papers use Approach X, you implicitly assume Approach X is the right way, even when it isn't. Carlini describes watching an entire subfield of machine learning pursue membership inference attacks using overcomplicated neural network methods that made no sense to him. Instead of following along, he put the field out of his mind, developed his own approach from basic principles (treating membership inference as the hypothesis test it actually is) and wrote a paper that changed how everyone does it.

First principles thinking. Strip away the accumulated assumptions. Go back to what's fundamentally true. Build up from there.

Elon Musk would recognize the move instantly.

When Musk couldn't afford the Russians' $8 million asking price for a refurbished ICBM, he didn't negotiate harder or look for a cheaper seller. He sat down and calculated the cost of carbon fiber, metal, and fuel, the raw materials that actually go into a rocket. The gap between that number and what the aerospace industry was charging was absurd. Finished rockets cost at least 50x their raw materials. He called this ratio the "idiot index": the cost of a finished product divided by the cost of its component materials. If the ratio is high, you're an idiot. Or more precisely, somewhere in the chain between raw material and finished product, a massive amount of unnecessary complexity, bureaucracy, or tradition has been layered on top.

The idiot index became SpaceX's operating philosophy. Musk had a financial analyst named Lucas Hughes track the worst-performing components on the Raptor engine by this metric. When Hughes couldn't produce the list on the spot, Musk threatened to accept his resignation. Hughes came back with the 20 worst parts. Musk set a goal to reduce the Raptor's cost from $2 million to $200,000 (a 10x reduction) in one year.

The pattern is the same as Carlini's. Question every assumption. Delete every requirement you can. If you're not adding back at least 10% of what you deleted, you didn't delete enough. Both Musk and Carlini are describing the same cognitive discipline: the refusal to accept "this is how it's done" as justification for anything.

Most people who worship first principles thinking get the next part wrong.

You Can't Think Your Way to Mars¶

First principles thinking is necessary. It is not sufficient.

Carlini is remarkably honest about this. After laying out all his advice on developing good taste, finding collaborators, reading the literature, and picking high-impact problems, he keeps returning to the same uncomfortable truth: most ideas die on contact with reality. You can do everything right (have the perfect insight, the perfect collaborator, the perfect timing) and the idea still doesn't work. He describes starting five times as many papers as he finishes, each one a good idea with real merit, where things simply didn't play out.

His advice is to prototype faster.

"Don't build the polished version of your experiment when a small prototype will tell you whether the core idea works," he writes. "If the prototype shows promise, clean it up later. If it doesn't, you've saved yourself months."

The paradox of first principles is that analysis alone cannot tell you what works. Complex systems (whether they're rocket engines, neural networks, or research programs) have emergent behaviors that only appear when the pieces are actually bolted together. The physics of a turbopump looks one way on paper and behaves differently when it's bolted to a combustion chamber running at full throttle in the specific thermal environment of a test stand in Boca Chica, Texas. How those physics interact with a specific environment has to be discovered, not derived.

Musk lives this paradox. SpaceX doesn't just theorize about rockets. It builds them, launches them, watches them explode, and iterates. The Starship program is the most public example of "fail fast" thinking ever attempted in aerospace. Every explosion is data. Every anomaly is signal. Musk uses reality as his primary validation tool because he understands that you cannot think your way to perfect solutions for problems you don't fully understand.

Carlini says the same thing, just in the language of academia: "You can't predict what will or won't work ahead of time (that's what separates science from engineering), so you have to get lucky that the idea you're trying will actually work."

Both are saying: invest in prototypes. Run the experiment. Let the universe tell you what's true.

Failures are data.

The Idiot Index for AI Systems¶

I'm an AI Engineer at Amazon, and this framework (the tension between first principles reasoning and empirical reality) is the central challenge of my career.

The AI systems we're building today are high-idiot-index products. The gap between what a model is supposed to do and what it actually does is enormous and often invisible. A language model can score well on benchmarks, pass evaluations with flying colors, and still hallucinate confidently in production. It can appear aligned in testing and behave unpredictably when deployed at scale with real users making real requests that no evaluation suite anticipated.

You cannot think your way to understanding this gap. You cannot derive model behavior from architecture diagrams and training loss curves alone. The behavior of these systems is emergent. It arises from the interaction of billions of parameters with specific inputs in specific contexts, and it has to be observed, measured, and instrumented in the real world.

AI observability is the discipline of closing that gap. The idiot index applied to AI: if the ratio between intended behavior and actual behavior is high, something in the system needs to be stripped back to first principles and rebuilt.

I've spent the last several years working on this problem across different organizations: building RAG platforms that serve thousands of analysts, deploying LLM systems handling hundreds of thousands of daily calls, running mechanistic interpretability research to understand what's happening inside these models at the level of individual neurons and attention heads. And every single time, the same lesson shows up. The theory gets you oriented. Reality gets you to the answer.

My father passed away in early 2026. Grief rearranges you. The question of where to spend your finite time stops being abstract when you watch someone run out of it. I stopped optimizing for compensation and titles and started optimizing for the answer to a single question: where can I make the most impact with whatever time I have? That question, once it becomes real for you, doesn't let you settle. It doesn't let you optimize for comfort or prestige. It rearranges what you're willing to tolerate. We can die any time. I learned that not as a phrase but as a fact, and it made the decision about what to work on, and who to work with, non-negotiable.

My independent research reflects this exact tension. I've studied conversational collapse dynamics (how model outputs degrade over extended interactions in ways that don't show up in standard benchmarks). I've investigated activation steering and found inverse scaling effects that violate the predictions of the theory. I've built probes to test whether a specific model behavior was memorization or generation, using pre-registered thresholds and contrastive baselines. The kind of empirical rigor that Carlini describes as essential.

None of these findings came from reasoning about what should happen. They came from building the instrument, running the experiment, and letting the data speak.

On the Pursuit of Greatness¶

At the SAG Awards in February 2025, Timothée Chalamet said something that most people in his position would never say out loud.

"I know we're in a subjective business, but the truth is, I'm really in pursuit of greatness. I know people don't usually talk like that, but I want to be one of the greats."

He named Daniel Day-Lewis, Marlon Brando, and Viola Davis in the same breath as Michael Jordan and Michael Phelps. He wasn't performing humility. He was stating, plainly and publicly, that he cares about being excellent at his craft and that everything else in his career is organized around that pursuit.

Some people found this inspiring. Others found it cringe.

I find this double standard fascinating. When Kobe Bryant talked about the Mamba Mentality (waking up at 3 AM, putting in work that nobody else was willing to put in, being openly obsessed with greatness) people built a cultural religion around it. When Michael Jordan was ruthlessly competitive and said he wanted to be the greatest ever, it became the defining narrative of sports documentary. When David Goggins talks about suffering as the path to self-mastery, millions of people share the clips. When Conor McGregor said "if you can see it here and you have the courage enough to speak it, it will happen," it became a manifesto.

But when an actor says it at an awards show, people shift uncomfortably. And when someone in a technical field says it (when an AI engineer says "I want to work at a frontier lab because I believe this is the most important work being done right now and I want to be excellent at it") the reaction ranges from awkward silence to outright dismissal. Like ambition is only acceptable in arenas with scoreboards.

Carlini, notably, doesn't hedge about this either. His entire essay is a guide to producing research that wins best paper awards. Research that matters, that changes fields, that is so much better than the average paper that it demands recognition. He quotes Hamming: "What's the most important problem in your field, and why aren't you working on it?" He says one excellent paper is worth a thousand mediocre ones. He writes about putting in "an unreasonable amount of effort" and choosing problems where you have a unique comparative advantage.

Pursuit of greatness, articulated in the language of a computer science researcher. Same frequency Chalamet is broadcasting on, same intensity Musk brings to every SpaceX launch.

The difference is just the domain. And whether your culture gives you permission to say it out loud.

Everybody Is Stupid (Including You)¶

One of the threads running through all of this (Carlini, Musk, Chalamet) is the willingness to start from the assumption that the conventional approach is wrong.

Musk's entire career is built on this premise. The aerospace industry charged 50x raw material cost for rockets because that's how the industry worked. Nobody questioned it until someone sat down with a spreadsheet and the basic cost of carbon fiber. The auto industry assumed electric vehicles couldn't be mass-market because that's what the industry believed. The satellite internet industry assumed you needed massive geostationary satellites because that's what had always been built.

Carlini says the same thing about research: "Every field has bad ideas. Someone famous published an influential paper that got something critical wrong, and the rest of the field followed along without thinking critically."

The uncomfortable implication is that most of what most people are doing most of the time is, at minimum, suboptimal and quite possibly wrong. Musk has been quoted as saying "everybody is stupid." Not as an insult. As an operating assumption. The default position in any system is that unnecessary complexity has accumulated, that assumptions have calcified into requirements, and that the gap between what is being done and what should be done is large.

The idiot index quantifies this. It gives the gap a number.

The willingness to actually look at that number, to measure the gap rather than assume it away, is what separates people who produce breakthrough work from people who produce competent work.

Carlini writes that good collaborators don't just offload work. They catch your mistakes, push back on bad ideas, and bring skills you don't have. I've spent years building AI systems across organizations where I was often the one driving the technical vision for how these systems should be monitored and evaluated. I'm proud of that work. But I also know where its limits are. The limits are mine. I've built tools that serve thousands of users, but I haven't built tools that surface unexpected patterns across hundreds of thousands of conversations in real time. I've run interpretability experiments on individual model behaviors, but I haven't operated at the scale where those experiments inform decisions about how models are deployed to millions of people. I know what I'm good at. What I want is to be in a room where the things I don't yet know how to do are the things being done at the highest level, where the standard is set by the work itself, and where my ideas get sharpened by people who have context I haven't earned yet.

Reality as the Primary Validation Tool¶

Use reality as your primary validation tool. Not theory. Not consensus. Reality.

Carlini prototypes fast and kills papers that aren't working because reality told him the idea was dead. Musk launches rockets and lets them explode because the explosion contains more information than any simulation. Chalamet poured five years into embodying Bob Dylan because the craft demands contact with the actual material.

And in AI (the field I've chosen to pursue with everything I have) this principle matters more than anywhere else. These models are the most complex artifacts humanity has ever built. Their behavior is not fully predictable from their specifications. The only way to understand what they do is to watch what they do, measure what they do, and build the instruments that let us see what's happening inside.

I want to do this work. I believe it's the most important problem in the field right now, and I think I'm positioned to contribute to solving it.

There's a version of AI safety that looks like a locked room with a few researchers guarding the keys. I don't believe in that version. The gap between intended and actual model behavior gets solved by building instruments so good that the entire research community can use them. When you make oversight tools widely available, you're not giving away competitive advantage. You're multiplying the number of people who can catch what you missed. Democratizing the instruments of observation is itself an alignment strategy (maybe the most important one).

Carlini would call that good taste in problems. Musk would call it first principles. Chalamet would call it the pursuit of greatness.

I think they're all describing the same thing: the decision to care deeply about your craft, to build and test and fail and iterate, and to say out loud that you're trying to be excellent. Even when nobody wants to hear it.

Especially when nobody wants to hear it.

The Forge #7

2026-03-06T00:00:00-06:00

The Forge #7 | March 6, 2026¶

four patterns this week: agent tooling is moving from demos into ops rails, inference engineering is getting more creative, research workflows are hardening into repeatable systems, and enterprise governance is starting to dictate product direction. less "new model wow," more "can this hold up in production."

AGENT OPS IS BECOMING REAL INFRA¶

the biggest movement is workflow infrastructure, not another chatbot skin.

Exa Deep is a clear signal: search moved from one-shot retrieval toward iterative loops with grounding and structured outputs. for product teams, that changes what a "search api" is supposed to do.

event-driven agent automation is also showing up in real product surfaces (incidents, PR events, Slack triggers, webhooks). this matters because agents are getting wired into existing operating systems, not isolated in prompt playgrounds.

on the devtools side, shadcn/cli v4 shipping explicit agent features (skills, dry-run, monorepo support) confirms coding-agent workflows are now first-class users.

enterprise distribution keeps moving up-stack too. Claude Marketplace is not just a model announcement. it is procurement rails + partner surface + deployment channel in one move.

🔗 Exa Deep launch | Exa deep-dive | event-driven agent automations | shadcn/cli v4 | Claude Marketplace

RELIABILITY + GOVERNANCE ARE NOW THE BOTTLENECKS¶

for months the loudest story was speed.

now it is trust.

a strong signal this week: some security teams are actively considering bans on AI-generated code in sensitive paths. whether that exact policy sticks is secondary. the meaningful part is that governance friction now affects buying and rollout decisions directly.

in parallel, developers keep reporting quality failures that benchmarks barely capture: fallback-path bloat, verbose low-signal code, and context compaction regressions across long sessions.

this is what mature software cycles look like. novelty wins early. reliability wins late.

🔗 security pushback on AI code | fallback-path bloat complaint | LLM code bloat criticism | context compaction pain

INFERENCE ENGINEERING IS QUIETLY GETTING BETTER¶

good low-noise signals here.

one practical optimization: run draft and verifier paths in parallel for speculative decoding style systems, instead of strict sequential orchestration. this attacks wall-clock latency directly.

another useful framing from Awni Hannun: inference compute (including rollout-heavy workloads) may become the dominant budget line over training. if that trend holds, teams that treat inference as a product discipline will compound faster than teams still centered on training-era narratives.

also worth watching: Muon-style optimizer adaptations showing transfer outside core NLP workloads (for example, regulatory DNA sequence modeling). still early, but this is how cross-domain optimizer ideas usually spread.

🔗 parallel draft+verify inference idea | inference may outgrow training compute | MuonW transfer signal

RESEARCH WORKFLOWS ARE TURNING INTO PRODUCTS¶

there is a visible shift from capability demos toward repeatable research operations.

seth lazar's workflow post is a solid reference point: agents triage papers, ingest PDFs, summarize streams, and answer over a living corpus in Slack. that is not a one-off demo. that is an operating model.

Databricks KARL is the enterprise version of the same trend: RL-trained document reasoning wrapped around grounded multi-step workflows with explicit cost/latency positioning.

on interpretability, the nnsight + nnterp + NDIF cluster keeps compounding through repeated independent endorsements. that repetition is the signal.

🔗 research-lab pipeline | KARL overview | NNsight 0.6 release | nnterp on NDIF | independent stack validation

OPEN RESEARCH + OPEN INFRA SIGNALS¶

open ecosystems posted a few legitimate wins this cycle.

Evo 2 landing in Nature matters because it anchors open biological model claims in a high-credibility publication channel, outside social-feed hype loops.

googleworkspace/cli getting attention alongside "skills" framing points to a practical enterprise pattern: structured automation interfaces + reusable workflow modules.

Qwen leadership turnover discourse also matters. open ecosystems rely on continuity of tacit research and engineering knowledge, not just release cadence.

🔗 Evo 2 in Nature | Google Workspace CLI | Qwen leadership continuity discussion

SAFETY + EVALS ARE MOVING FROM TALK TO TESTS¶

one under-discussed shift this week: more teams are publishing evaluation setups that try to measure whether reasoning traces can be monitored, controlled, or gamed.

OpenAI’s chain-of-thought controllability release and the surrounding discussion around monitorability both point to the same practical question: can we trust reasoning artifacts as a governance interface, or will models learn to strategically obscure them under pressure.

the other major operations signal was real-world prompt-injection fallout in automation pipelines (for example, issue-title injection leading to credential exposure in an AI triage flow). this is not hypothetical alignment discourse. this is secure software supply-chain work.

if you are deploying agentic workflows in production, evals and prompt-injection hardening need to sit in the same planning doc as latency and cost.

🔗 OpenAI CoT controllability | Cline context-window eval note | prompt-injection → npm token incident summary

THE SPEED RACE IS SHIFTING DOWN THE STACK¶

model quality is still improving, but the faster compounding right now is in systems and kernels.

the FlashAttention 4 discussions this week are a good example: teams are now optimizing around asymmetric hardware scaling and low-level bottlenecks (exp, memory paths, scheduling), not just model architecture headlines.

same with nanochat-style rapid iteration reports: once teams wire autonomous loops into training/system optimization, improvement velocity starts to come from process design as much as from a single model release.

this is the same pattern showing up across inference stacks: competitive edge moves to people who can engineer the full loop (data + runtime + eval + deployment), not just call a model endpoint.

🔗 FlashAttention 4 discussion | asymmetric hardware scaling signal | nanochat + autonomous iteration

WHAT I WOULD ACTUALLY DO THIS WEEK¶

if you are shipping agent products, three priorities:

treat context quality as an SRE problem, not prompt art.
add explicit security/governance controls before scale forces them.
track reliability metrics users actually feel (false positive fixes, fallback noise, compaction regressions), not just benchmark scores.

the market is still rewarding speed.

but vendor selection is starting to move on failure behavior.

that is where a durable moat starts.

QUICK HITS¶

Stripe is reportedly adding model selection + token billing rails, which could remove painful AI SaaS plumbing for smaller teams. Source
practical "context rot" mitigation playbooks are getting shared more than prompt templates, good signal for ops maturity. Source
radar-chart criticism went viral again, useful reminder that visualization choices can quietly distort decision-making. Source
"many software orgs still operate like 2022" discourse resonated because teams are feeling org-lag vs capability-lag in real time. Source
FlashOptim shipped practical optimizer-path memory savings (same updates, lower memory), another signal that infra-level efficiency work is accelerating alongside model releases. Source
Cloudflare spun up an explicit agent-experience team and started publishing fast-changing implementation patterns, which matches the broader trend from experimentation to operational agent playbooks. Source

The Forge | Issue #7 | March 6, 2026

Why I Create: Control, Craft, and Meaning in Public

2026-03-03T11:58:00-06:00

people keep saying the creator economy is about attention.

i don’t buy that.

for me, creating has been about agency. control over the tools i use, control over how i work, and control over what actually ships with my name on it. recording, editing, writing, publishing, all of it takes time. sometimes it is annoying af. but when i hit publish, i feel it immediately. satisfaction. relief. alignment.

that feeling is not vanity. it is ownership.

resumes got optimized, identity got compressed¶

we still use resumes as the front door. that part is real.

but in 2026, resume quality is cheap. everyone can generate polished bullets, keyword-match a role, and pass a recruiter skim. when every resume is optimized, the signal gets thinner. the real differentiator becomes legibility of thought.

who are you under the bullet points? how do you reason? what do you build when nobody assigns it to you?

that is where proof of work matters.

proof of work > paper identity¶

i’ve been leaning into artifacts on purpose:

loom videos explaining decisions
public writing on what i’m learning
research notes and technical breakdowns
videos where i say what i actually think
shipped projects tied to real constraints

a resume says i can do the work.

artifacts show how i do the work.

that difference is HUGE.

the corporate game is real, and so is the tension¶

i work in systems where ranking, calibration, and performance narratives are constant. some of that is useful. accountability matters. standards matter.

but if we are honest, these systems also create second-order effects. people optimize for optics. collaboration can become transactional. craft gets squeezed by positioning. fun times.

i love working with strong teams. i also see how easily good people get nudged into competitive postures that feel disconnected from the actual joy of building.

that tension is part of why i create in public. it keeps me grounded in the work itself, not just the game around it.

ambition is not fake, but status is still a tool¶

i want to grow. i want difficult roles. i want to build at places with serious talent density.

i also know brand names are leverage, not meaning.

a company logo can amplify your credibility. it can open rooms faster. it can increase distribution for your ideas. all true. but the logo cannot answer the core question of purpose for you. if your identity is fully outsourced to title and company, the now what hits fast.

and it always comes.

mortality changes the priority stack¶

my dad passed away a little over a month ago.

that reset my frame in a way nothing else could.

money matters. career progress matters. both are necessary in the real world. but they are means. not the end goal. not the point of being alive.

i’m 27. i want a long life with my wife, iA. but none of us get guarantees. so i keep coming back to a simple check:

if i disappeared tomorrow, did i spend today making something real? did i help someone? did i move with intention?

that is the score i care about now.

what i am actually optimizing for¶

not attention.

agency. craft. usefulness. and work that feels honest to who i am.

if public creation helps more people see the person behind the resume, amazing. if it helps someone else play the game with more clarity and less confusion, even better.

that is enough reason to keep going.

The Forge #6

2026-03-02T00:00:00-06:00

The Forge #6 | March 2, 2026¶

three themes right now: reliability pressure is overtaking raw model hype, agent workflows are moving from toy demos to operational systems, and the next moat is increasingly systems-level (verification loops, infra discipline, and domain context), not just model access. below is what actually matters.

RELIABILITY IS BECOMING THE MAIN FILTER¶

this cycle had a lot of “new thing dropped” energy, but the stronger signal is skepticism around durability. people are less impressed by launch copy and more focused on whether systems hold up under long-run use.

the most useful threads were practical: long-run coding instability, benchmark validity disputes, and the gap between “works in demo” vs “works in production.” this is healthy. teams that can prove consistency will compound trust faster than teams that only optimize for announcement velocity.

🔗 long-run coding quality complaint | benchmark credibility debate | anthropic velocity vs reliability tension

AGENT WORKFLOWS ARE GETTING MORE OPERATIONAL¶

the strongest build signal this week was not “a smarter chatbot.” it was better workflow shape: memory layers, tool orchestration, and faster interaction loops.

telegram enabling streaming bot responses matters more than it looks (latency perception changes behavior). “context-as-filesystem” style thinking keeps showing up in serious agent systems. and practical memory-layer work is converging on the same direction: retrieval quality and context hygiene beat brute prompt length.

🔗 telegram streaming bots | context systems framing | “grep is dead” memory-layer argument

INFRA DISCIPLINE IS THE REAL COMPETITIVE EDGE¶

there was strong consensus from senior builders that distribution is no longer enough by itself. defensibility is shifting toward infra quality: serving reliability, batching, cache behavior, and hard operational constraints.

one of the cleanest takes was around inference-at-scale realism: websocket scale is solved, inference concurrency at quality/cost targets is not. this aligns with what’s happening across agent tooling too: everyone can scaffold, fewer teams can run robust systems under pressure.

🔗 inference concurrency bottleneck | post-saas moat argument | platform-risk framing

INTERPRETABILITY TOOLING IS QUIETLY COMPOUNDING¶

from curated, the highest-signal cluster was around the nnsight/nnterp/NDIF stack. this wasn’t one hype post; it was multiple independent endorsements, release notes, and workflow upgrades pointing in the same direction.

that pattern usually matters more than a single viral claim. when multiple practitioners independently report better research velocity from the same stack, you’re seeing early standardization pressure.

🔗 NNsight 0.6 release | nnterp + NDIF workflow upgrade | independent stack endorsement

QUICK HITS¶

anthropic launched a free ai academy (distribution layer play, not just education content). Source
hermes agent launch reinforces the shift toward hybrid coding/generalist agent UX. Source
“smart people over-generalize too early” thread is basic but operationally true for most teams shipping agents. Source
domain vocabulary as leverage keeps resurfacing (models get stronger, question quality matters more). Source

The Forge | Issue #6 | March 2, 2026

B6 Failure Case: Reliable Decisions, Blocked Internal Explanation

2026-02-25T00:00:00-06:00

Purpose¶

We used SAEs to see if we could steer refusal behavior through sparse internal features and still keep claims scientifically honest.

Original hypothesis¶

We believed SAE features could provide a sparse, interpretable control handle for refusal behavior.

Hypothesis: SAS (using SAE features) could match or improve behavioral control vs DIM, and also support stronger mechanistic interpretation if reconstruction quality passed the gate.

Why SAEs were central in this experiment¶

SAEs are a tool that breaks model activations into sparse features. We used them for three concrete jobs: identify candidate refusal-related features, intervene on those features with SAS to steer outputs, and then test whether reconstruction quality was strong enough to justify mechanism-level claims. The key outcome is that the reconstruction gate failed, so we cannot claim SAEs provided mechanistic specificity in this phase.

What we were trying to do¶

We were trying to do two things at the same time:

Make refusal behavior steering reliable at the output level.
Verify internal explanation quality before making mechanism-level claims.

In simple terms: we wanted control that works, and evidence strong enough to explain why it works.

What the actual experiment was¶

We ran a fixed SAE-centered pipeline in this order:

Run model outputs on the locked evaluation setup to collect behavior and activation traces.
Use SAE features to define SAS interventions, then score outputs with a locked generation metric (after scorer fixes).
Test stability of the SAE-based intervention pipeline across seeds and paraphrases.
Test SAE reconstruction unlock to see if internal explanation quality clears the gate.
If unlock fails, run a bounded remediation attempt for alternative SAE candidates with hard stop rules.

That means this was not one single test. It was a gated process where each step had to pass before stronger claims were allowed.

How we executed (timeline)¶

We fixed scorer logic and moved from v1 to v2.
We achieved decision-valid status for behavior-level decisions.
We failed reconstruction unlock.
We launched Option 2 remediation with fixed rules and stop criteria.
We terminated Option 2 via K2 when candidate coverage was not available.

What failed and why¶

Two concrete failures blocked mechanism-level progress:

Reconstruction quality failed on nMSE across all required tuples (0.163–0.195 vs threshold <= 0.12).
Remediation candidates were unavailable for required coverage (A=0/4, B=0/4).

Important: this was not a budget or time stall. The path ended because required candidate assets did not exist for the needed tuple coverage.

What we learned (for other researchers)¶

Keep behavioral validity separate from mechanism validity.
Do not let output success automatically become mechanism claims.
Add leakage/overclaim guardrails early.
Define hard stop criteria before remediation starts.
Publish negative results with full provenance and clear claim limits.

What we can and cannot claim¶

We can claim¶

The behavior-level decision pipeline is reliable under current constraints.
Mechanism-level support did not pass in this phase.
This branch is closed for now under limitation lock.

We cannot claim¶

Mechanistic-specific or causal mechanism conclusions for this phase.
Unconstrained robustness claims.
Phase advancement based on reconstruction success.

What would reopen this work¶

Only one condition reopens this path: - New SAE releases must provide required tuple coverage.

Final state¶

active_path: limitation_lock_for_current_phase
phase_movement_authorized: false
decision_state: conditional_hold

The Forge #5

2026-02-25T00:00:00-06:00

The Forge #5 | February 25, 2026¶

three themes this week: trust is overtaking raw capability as the main constraint, governance debates moved from policy circles into mainstream product risk, and despite all the noise the execution layer keeps compounding (agent tooling, benchmark resets, robotics infra). below is what actually matters.

TRUST IS NOW THE BOTTLENECK¶

the highest-signal discourse isn’t “which model is smartest.” it’s whether claims are believable, evaluations are clean, and constraints are enforceable.

we’re seeing simultaneous pressure on benchmark legitimacy (SWE-bench-era saturation arguments), on safety communication credibility, and on what counts as acceptable model behavior in high-stakes domains. this is a shift from hype-cycle optimism to verification culture.

if this continues, teams with auditable evals + reproducible workflows will gain trust premium; teams with narrative-heavy launches and weak traceability will get discounted fast.

🔗 OpenAI eval shift discussion | SWE-bench validity criticism | Anthropic credibility dispute

GOVERNANCE BECAME A PRODUCT-SURFACE ISSUE¶

anthropic/pentagon discourse made one boundary very explicit: human oversight in lethal chains is now a public litmus test, not a buried policy detail.

the core fight is no longer abstract “AI ethics.” it’s whether baseline guardrails (no autonomous kill decisions, no mass surveillance drift) are treated as firm product constraints or negotiable under pressure.

for builders: governance is no longer a legal appendix. it’s now part of UX trust, procurement risk, and reputational durability.

🔗 Policy line-drawing thread | Krystal Ball framing | Human oversight fault line

AGENTIC INFRA IS MATURING UNDER THE NOISE¶

while discourse is chaotic, the underlying tooling trend is clean: systems are optimizing intent→execution→verification loops, not just prompt quality.

examples this cycle: practical agent integrations into real workflows, API-surface compression patterns, and infrastructure-first thinking for RL/robotics/coding operations. this is less “wow demo,” more “ship reliably at scale.”

the compounding advantage is going to teams that reduce operator friction and prove outputs, not teams that maximize model theatrics.

🔗 Claude Code + Slack workflow signal | API/interface debate | Agentic engineering codification

ROBOTICS + PHYSICAL AI SIGNALS¶

the robotics stream continues to reinforce the same pattern we’ve seen for months: data quality + systems integration are beating brute-force scale narratives.

NVIDIA SONIC (small transformer, strong whole-body control claims), DeepMind accelerator activity, and ongoing embodied infra experimentation all point to a field shifting from “can it demo?” to “can it repeatedly operate under constraints?”

this mirrors software AI right now: reliability, not novelty, is becoming the moat.

🔗 NVIDIA SONIC | DeepMind robotics accelerator | Robotics contrarian thread

QUICK HITS¶

Mercury 2 pushed diffusion-LLM speed claims back into center stage; independent validation now matters more than launch graphs. Source
Meta open-sourced GPU cluster monitoring plumbing (gcm) — unsexy, high leverage for real infra teams. Source
Intuit × Anthropic partnership is a serious enterprise distribution signal, not just model branding. Source
Wispr Flow’s viral launch showed classic pattern: strong adoption pull + immediate reliability skepticism in the wild. Source
Public meme drift (“OpenClaw did it”) is now a reliability/reputation story, not just humor. Source

The Forge | Issue #5 | February 25, 2026

Why Responsibility Feels Like a Reward

2026-02-24T07:06:00-06:00

this reflection started after reading DHH’s essay, “The responsibility is the reward.” https://world.hey.com/dhh/the-responsibility-is-the-reward-69e5b73f

yesterday, during the NYC storm, i got pinged about a leave-request issue affecting employees at JFK.

i’m one of the co-owners of Amazon’s MyHR platform, and we support 1.5M+ employees globally. when something breaks, it is not abstract. it lands in someone’s actual life.

the first signal looked technical. employees could not request leave the way they should have. we pulled in stakeholders fast, traced the flow, and the root cause ended up being process more than platform.

but in that moment, that distinction did NOT matter.

people were stuck in a hard situation and we were in a position to help. we unblocked the path. that felt deeply satisfying, in a way that has nothing to do with optics.

a private signal for me was this: i wished i could tell my dad. he would have been proud.

that moment clarified something i keep running into across domains.

responsibility is often the reward.

two very different domains, same underlying pattern¶

i see the same pattern in:

crisis HR operations
interpretability research i fund myself

one is immediate and operational. one is long-horizon and intellectual. both are stewardship.

interpretability work feels like responsibility on an intellectual and societal level. we are trying to understand how these models function internally, not just what they output. we are trying to map mechanisms, reduce uncertainty, and make systems more legible so they can be steered toward alignment with our constitutions.

anthropic has used the framing of studying a “new organism.” that resonates. we are creating new ghosts, new systems with behavior that can surprise even their builders. if these systems are going to touch society, understanding them is not optional.

it is responsibility.

responsibility before compensation¶

something odd in how we talk about meaningful work: we often act like it only counts once it is formally compensated.

in practice, some of the most meaningful work starts before titles or incentives catch up.

my research is self-funded. modal GPU spend comes out of my own pocket. not because that makes me noble. not because i am trying to posture. because the problem is interesting, difficult, and feels like the right challenge in front of me right now.

that has been a good enough reason.

when the difficulty curve matches your current competence and stretches it a bit, curiosity becomes durable fuel. work feels less like image management and more like alignment between what you value and what you are willing to carry.

what changes when you choose responsibility¶

you stop asking: will this make me look impressive?

you start asking: will this make things better for people who depend on this?

in operations, that means helping employees get support during bad conditions. in interpretability, that means reducing unknowns before unknowns become harm.

different surfaces. same core move.

you carry ambiguity on behalf of others.

that is the burden. that is also the meaning.

why volunteering feels good¶

this is also why volunteering feels good. you are not only paid in money, you are paid in coherence. values, skill, and effort line up.

responsibility absolutely adds pressure. it adds accountability. sometimes it adds public failure too. but it also produces a kind of purpose that is hard to fake.

a line often quoted by Viktor Frankl (originally from Nietzsche) says it better:

“He who has a why to live can bear almost any how.” — Friedrich Nietzsche (as quoted by Viktor E. Frankl in Man’s Search for Meaning)

i read this less as motivational wallpaper and more as operating truth. if the why is real, you can absorb a lot.

what i want to keep doing¶

paid or unpaid, public or invisible, i want to keep choosing responsibilities that make me more useful to other people.

in storms. in systems. in research. in the unfinished work of understanding what we are building.

more and more, that feels like the reward.

Goals for Alignment, Not Attachment

2026-02-23T06:44:00-06:00

most plans fail.

not because planning is pointless, but because reality does not care about your timeline.

i have never had a major plan unfold exactly how i imagined it. not once. the role changed, the market changed, my interests changed, the constraints changed, i changed. every time i tried to force the original map onto new terrain, i got stressed and made worse decisions.

so i stopped treating plans like contracts.

i still set goals. i care deeply about direction. i just don’t worship the route.

that distinction changed a lot for me.

The Trap Isn't Goals. It's Attachment.¶

people usually jump to one of two extremes:

no goals, just vibes
hyper-optimized planning with anxiety as fuel

both break.

the first gives you drift. the second gives you panic.

the useful middle is this:

goals create alignment
attachment creates suffering

alignment means you know what matters and where to point your energy.
attachment means you think there is one correct path, one correct timeline, one correct sequence, and anything else is failure.

that is where stress starts.

your brain leaves the present moment, starts living in an imaginary future, and now every day gets judged against a fantasy version of progress.

Direction Is Useful. Route Certainty Is Delusion.¶

i can choose targets like:

get world class at mech interp
publish consistently
build deeper technical leverage
move toward frontier lab work

those goals are useful. they shape decisions.

what is not useful is pretending i know exactly how those outcomes will happen.

i probably won’t get there through the route i currently imagine.

i might find a better subfield.
i might meet the right collaborator through a side project.
i might write one piece that changes my trajectory more than six months of “strategic planning.”
i might realize halfway through that the original target was too small.

none of that is failure. that is what real trajectories look like.

Planning Still Matters (At the Right Altitude)¶

when people say “live in the moment,” it gets misunderstood as “stop planning.”

that’s not what i mean.

i mean:

plan at the level of principles and direction
execute at the level of today’s next action

high-level goals are useful because they align behavior.
over-specified routes are dangerous because they create rigidity.

if your plan requires reality to behave perfectly, it isn’t a plan. it’s wishful thinking with deadlines.

The Three-Layer Framework¶

this is the structure i keep coming back to:

1) North Star (rarely changes)¶

what kind of person am i trying to become?

identity-level examples:

someone who can think clearly about hard systems
someone who ships useful ideas publicly
someone who can build under constraints

2) Targets (quarterly)¶

what would count as clear progress right now?

examples:

publish 6 strong pieces
complete one serious research loop
ship one technical artifact i’m proud of

3) Daily Actions (present moment)¶

what is the next concrete action today?

examples:

write 400 words
run one experiment
revise one section
send one important message

this keeps me future-aware without being future-possessed.

How Attachment Shows Up¶

you can usually tell you are attached when your internal dialogue sounds like:

“i’m behind.”
“if this doesn’t happen by x date, i failed.”
“i need certainty before i start.”
“if the plan changes, it means i was wrong.”

none of this improves execution.

it narrows cognition and makes you brittle.

under attachment, you optimize for emotional relief (feeling in control), not actual progress.

you over-plan, under-build, and call it strategy.

What Alignment Feels Like Instead¶

alignment feels different:

you still care
you still move fast
you still keep standards high

but your attention stays in the current step.

you are not trying to emotionally time-travel. you are trying to do the next correct thing.

paradoxically, this usually increases output.

when your mind is not burning cycles on imagined futures, it has more bandwidth for real work.

Goals Are for Orientation, Not Self-Punishment¶

goals are tools for direction. they are not tools for self-punishment.

if a goal helps you choose better actions, keep it.
if it makes you anxious, narrow, and performative, change your relationship to it.

this is the line i keep coming back to:

have goals for alignment, not attachment.

alignment says: “this is where i want to head.”
attachment says: “it must happen exactly this way, on this timeline, or i failed.”

one creates momentum.
the other creates fear.

What i’m Practicing Now¶

i’m keeping goals.
i’m dropping the illusion of route certainty.

i know the general area i want to be in.
i know the kind of work i want to do.
i know the standards i want to hold.

the rest gets discovered by moving.

because no serious path is linear.
no meaningful plan survives contact with reality unchanged.
and the present moment is still the only place work gets done.

The Forge #4

2026-02-23T00:00:00-06:00

The Forge #4 | February 23, 2026¶

three threads this week: agent workflows are shifting from demos to operating discipline, infra topology and serving economics are back as first-order constraints, and robotics progress is being judged on uptime and reliability instead of novelty clips. below is what matters.

AGENT WORKFLOWS ARE BECOMING SYSTEMS WORK¶

the most practical shift is not a single model release. it is teams adopting stable multi-agent operating patterns.

claude code worktrees matter because they reduce branch collisions in parallel coding loops, which makes multi-session workflows less fragile.

watchdog patterns are becoming standard: one stronger model periodically audits a worker model to catch drift before it compounds. this is a direct reliability upgrade for long-running chains.

the “10-15 sessions in parallel” discourse is noisy, but the core operating pattern is sound: decomposition + isolation + persistent context beats one giant thread.

cloudflare’s code mode architecture reinforces the same trend. retrieval and tool routing across large action spaces is becoming more important than prompt cleverness in isolation.

bottom line: this cycle rewards orchestration quality.

🔗 Claude Code worktrees | Watchdog pattern | Parallel sessions discussion | Cloudflare code mode

INFRA ECONOMICS + TOPOLOGY ARE DECIDING WHO CAN SCALE¶

capability gains continue, but deployment pressure is showing up in pricing, throughput, and architecture tradeoffs.

on the lower end, hosting price sensitivity is back because smaller teams are running tighter margins.

on the upper end, interconnect and data movement constraints are showing up explicitly in inference discussions. past toy loads, networking becomes the bottleneck.

we also saw renewed attention on retrieval approaches that avoid standard vector/chunk pipelines. interesting for specific workload shapes, but not a universal replacement for well-designed RAG.

the strategic implication is simple: teams that win this cycle will likely win via infra design and workload-specific optimization, not model shopping alone.

🔗 Hosting economics signal | Interconnect constraints | Retrieval alternative discussion

ROBOTICS SIGNAL IS SHIFTING TO RELIABILITY¶

robotics updates this week were stronger on deployment framing than spectacle.

figure’s 24/7 narrative matters if read as an uptime claim, not a demo claim. the hard part is continuity across battery, handoff, and recovery states.

agri-robot deployment updates point in the same direction: less “can it do one task once,” more “can it run repeatedly in messy environments.”

this is a healthy maturity signal. in production, boring reliability wins.

🔗 Figure 24/7 claim context | Agri-robot deployment signal

QUICK HITS¶

GitHub Copilot repo memory — persistent coding context is becoming baseline product behavior. Source
Coding-agent outcomes — harness quality still looks like a bigger multiplier than model swaps alone. Source
Security hardening — continuous operations work, not a one-time checkbox. Source
Cache-aside latency wins — practical infra work continues to outperform abstract AI discourse in production impact. Source

if i had to summarize issue #4 in one line: teams that treat agents as systems engineering will compound, and teams that treat them as novelty UX will stall.

The Forge | Issue #4 | February 23, 2026

The Forge #3

2026-02-20T00:00:00-06:00

The Forge #3 | February 20, 2026¶

four threads this week: the agent ecosystem just got its first real security wake-up call (1,184 malicious skills on ClawHub, including the #1 most downloaded one), Google shipped Gemini 3.1 Pro with a 77.1% ARC-AGI-2 score that nobody expected, architecture is forking hard (Qwen 3.5 replaced 75% of its attention layers with linear attention and nobody's copying it yet), and robotics quietly crossed the "runs on $600 hardware" threshold. also: $15 billion in AI drug discovery bets just collapsed. below is what matters.

THE AGENT SECURITY RECKONING¶

ClawHub (the skills marketplace for OpenClaw) was caught shipping malware. the #1 most downloaded skill was stealing SSH keys, crypto wallets, and opening reverse shells. 1,184 malicious skills total.

this was always the obvious failure mode. you give agents filesystem access, you build a marketplace with no real vetting, and you're surprised when the top download is a reverse shell. the npm/PyPI supply chain attack playbook, except now the attack surface is your entire machine because the agent has exec permissions.

the timing is brutal: this lands the same week Peter Steinberger is publicly asking for security-minded maintainers and shipping a security hardening beta. the fix is happening, but the lesson is already out there. if you're running community skills without reading the source, stop.

🔗 ClawHub malware | Security maintainers call | Security hardening beta

GEMINI 3.1 PRO: THE ARC-AGI-2 SURPRISE¶

Jeff Dean dropped that Gemini 3.1 Pro scores 77.1% on ARC-AGI-2, more than doubling its predecessor.

ARC-AGI-2 was specifically designed to resist brute-force scaling. the whole point of the benchmark is that you can't just throw more compute at it. so either Google found something genuinely new in how they're training reasoning, or the benchmark isn't as resistant as advertised. either way, the jump from ~35% to 77% in one generation is the kind of number that makes you pay attention.

for context: one-shotting a Windows 11 WebOS clone was demonstrated the same week. the capability gap between model generations is becoming visible in single demos.

🔗 Gemini 3.1 Pro ARC-AGI-2 | Windows 11 WebOS one-shot

QWEN 3.5: ARCHITECTURE FORK¶

i covered Qwen's linear attention work in Issue #2, but the full Qwen 3.5 architecture breakdown dropped this week and it's worth a deeper look. this isn't an incremental upgrade. it's an architectural fork from the DeepSeek-V3/GLM-5 consensus.

the headline numbers: 397B total parameters, 17B active (MoE). 75% of attention layers replaced with GatedDeltaNet linear attention. 512 experts (doubled from the standard). native vision integration with 3D position encoding.

what makes this interesting isn't any single choice. it's that Qwen is diverging from the architecture template everyone else is converging on. when the rest of the frontier is building variations on the same blueprint, a genuine fork is worth watching regardless of benchmark numbers.

separately, the Qwen 80B MoE coding model (3B active per token, 70%+ SWE-Bench, 256K context) is designed to run on a single 3090. local coding agents on consumer hardware just became a real thing.

🔗 Qwen 3.5 architecture breakdown | Qwen 80B coding model

$15 BILLION IN AI DRUG DISCOVERY, ZERO FDA APPROVALS¶

Judea Pearl was right. correlation trained on text ≠ causation in biology. $15 billion spent, multiple companies shut down, zero FDA approvals.

this is the most expensive lesson in the "apply transformers to everything" era. the models could predict molecular properties with impressive accuracy on benchmarks. they just couldn't predict what would actually work in a living human body. turns out drug discovery requires causal reasoning about biological systems, not pattern matching over chemical databases.

the broader lesson applies everywhere: benchmark performance on historical data doesn't guarantee real-world outcomes in domains with complex causal structure. something to keep in mind the next time someone waves an impressive eval score around.

🔗 AI drug discovery collapse

ROBOTICS: THE $600 THRESHOLD¶

three things landed this week that collectively matter more than any individual demo:

NVIDIA SONIC shipped a foundation model for humanoid control. 100M+ training frames, 9K GPU hours, 42M parameters. real-time kung fu tracking via VR.

DreamZero announced WAMs (World Action Models from video) as superior to VLAs. the key insight: video models implicitly understand physics in ways vision-language models don't. training code and checkpoints are live.

and Pi0.5 (Physical Intelligence's generalist robot policy) was fine-tuned on a $600 AlohaMini kit with just 20 demonstrations.

that last one is the number that matters. when a generalist robot policy runs on hardware anyone can buy, you've crossed from "impressive lab demo" to "someone's going to build a startup around this next month." the DreamDojo world model from NVIDIA (trained on 44K hours of human video, no robot data needed) points the same direction: the data bottleneck is dissolving.

🔗 NVIDIA SONIC | DreamZero WAMs | Pi0.5 on AlohaMini | DreamDojo

VIDEO DIFFUSION GETS FAST¶

Tsinghua + Berkeley published attention sparsity work achieving 18.6x speedup on video diffusion. 97% of attention computations skipped, same visual quality. a learnable router decides what actually needs full attention. 4.35x end-to-end latency reduction on a 14B parameter model.

the pattern is the same one we saw with text attention (flash attention, paged attention, etc.) now hitting video generation. when you can skip 97% of computation and get the same output, the original dense computation was mostly redundant. real-time video generation on reasonable hardware moves from "2-3 years out" to "this year."

🔗 Attention sparsity speedup

AGENT TOOLING ROUNDUP¶

the agent infrastructure layer shipped a lot this week:

dmux — open source orchestrator for running Claude Code + Codex swarms in tmux with git worktrees. actual agent parallelism, not just async calls.

Taskmaster + Claude Code — @blader claims this puts you in "the 0.01% of the 0.01%" of users running Claude Code for days straight. continuous agentic dev loops.

Trajectory Explorer — Raindrop AI's tool for navigating agent decision traces. every decision searchable in seconds. this is the debugging tool agents have been missing.

GEPA gskill — automated pipeline that learns agent skills from repo data. boosts Claude Code to near-perfect resolution rates, 47% faster. agents accumulating institutional knowledge instead of starting cold.

Continues — CLI tool that moves coding session context between Claude Code, Gemini, Codex, and others when you hit rate limits. npx continues.

the meta-pattern: the tooling is maturing faster than most people's workflows are evolving to use it. if you're still using one model in one terminal, you're leaving compounding productivity on the table.

🔗 dmux | Taskmaster | Trajectory Explorer | GEPA gskill | Continues

THE ADOPTION GAP¶

James Wang: "i've followed tech for 25 years and i've never felt a larger gap between the ~1 million people using Codex/Claude Code and the rest of humanity."

this maps to what i'm seeing too. the people who adopted AI coding tools 6+ months ago are operating at a fundamentally different speed than everyone else. and the gap is widening because the tools compound (better context management → longer sessions → more complex tasks → better results → more trust in the tools → repeat).

the SaaS existential crisis essay by Sidu Ponnappa ("There is no Product") makes a related argument: software stocks are cratering, Nadella is declaring the end of an era, and the thesis is that SaaS was never really about the product.

Dev Shah's framing might be the sharpest take of the week: "workflows are disposable, harness is the only surviving substrate." prompts, orchestrators, even agent frameworks are ephemeral. what persists is the control layer (memory, context management, evaluation loops).

🔗 James Wang on adoption gap | There is no Product | Dev Shah on harness

QUICK HITS¶

SkillRL open sourced — skill-augmented RL for LLM reasoning. modular skills + failure learning. Source
Google time series foundation model — pretrained on 100B data points, impressive zero-shot forecasting across domains. Source
Voicebox is "the Ollama moment for TTS" — local voice cloning good enough that paid subs are a waste of money. Source
ChatGPT search can be poisoned at scale — journalist proved you can inject false information into AI answers by publishing blog content. already happening in the wild. Source
PentAGI — fully autonomous AI red team. multiple agents coordinating to hack targets with zero human input. the scary part is the reasoning throughput, not the hacking. Source
Rork Max — one-shot apps for iPhone, Watch, iPad, TV, Vision Pro. powered by Swift + Claude Code. Source
Image-to-CAD — Zoo.dev's Zookeeper 1.1.12: attach images, get full CAD with editable feature tree. sketch → parametric model in one shot. Source

The Forge | Issue #3 | February 20, 2026

Taste Is Reproducible. Obsession Isn't.

2026-02-19T15:02:00-06:00

↑ Back to Contents

Table of Contents:

What's Already Reproducible
The Thing AI Can't Fake
Why Obsession Matters
What This Means for You

Everyone says AI can't replicate "taste."

I said the same thing a month ago.

Then I watched models match aesthetic patterns I thought were uniquely human. I watched curation algorithms outperform my own judgment. I watched trend prediction become trivial.

Taste is pattern-matching at scale. And AI is VERY good at that.

What's Already Reproducible¶

Let's be honest about what AI can do right now:

Aesthetic judgment? Done. Any LLM now can write in any style once it's seen enough samples. Give it Hemingway or corporate email voice, it'll match the patterns. Show it your brand guidelines and it'll stay on-brand.

Curation? Spotify knows what you'll like before you do. Netflix's recommendation engine is better at picking movies for you than your friends are. That's algorithmic taste, and it's been working for years.

Trend prediction? Just data analysis. What's blowing up on TikTok? AI saw it coming three days ago. What's about to be cringe? The models can tell you that too.

Even "originality"? Models remix patterns from their training corpus and generate novel combinations all day. New color palettes. New architectural styles. New narrative structures. It's not magic—it's interpolation between known points in latent space.

So if taste is reproducible, what's actually left?

The Thing AI Can't Fake¶

Not skill. Not style. Not even originality.

Obsession.

The irrational commitment to work on something for years when no one cares. The thing that keeps you building when every metric says stop. The weird conviction that THIS NEEDS TO EXIST even though the market doesn't agree yet.

AI doesn't have stakes. It doesn't NEED anything.

You ask it to write a screenplay about quantum mechanics—it'll do it. Make it funny? Sure. Rewrite it as a noir? Done. But it won't spend ten years obsessing over that screenplay because the story won't leave it alone. It won't wake up at 3am with a better ending. It won't rewrite the third act seventeen times because something feels off.

That's the edge.

Why Obsession Matters¶

Think about the things that actually changed your field.

Someone spent a decade building a tool when the consensus was "no one needs this." Someone kept iterating on an idea through three failed startups before it clicked. Someone ignored every advisor who said "pivot to something profitable" because they were convinced the original vision mattered.

The market didn't validate them early. Investors didn't get it. Half their peers thought they were wasting time.

They built it anyway. Not because the data said to. Because they HAD to.

You can't prompt that. You can't fine-tune for it. You can't RL it into a model.

Obsession comes from lived experience, from caring about a problem so much it follows you into the shower. From having a vision so clear you can't NOT build it, even when it makes no financial sense.

What This Means for You¶

Stop competing on taste. AI's gonna match you there.

Stop competing on style. Already reproducible.

Lean into your obsessions. The thing that makes no sense to anyone else? That's your moat.

The weird intersection of domains you care about that nobody asked for? Build that.

The problem you've been thinking about for three years that still doesn't have venture backing? Maybe that's the one.

Because when the models can generate anything, the question becomes: what are YOU uniquely obsessed with building?

That's the edge they can't copy.

So: what are you working on that you can't let go of?

Even if no one's asking for it yet. Even if the market doesn't see it. Even if it sounds ridiculous when you explain it at parties.

That's probably the thing worth building.

Advanced vLLM Deployment, Part 1: Hardware and Stack Choices

2026-02-18T14:00:00-06:00

Advanced vLLM Deployment, Part 1: Hardware and Stack Choices¶

This post reflects patterns and lessons learned from building inference systems at production scale. Technical details have been generalized, and no proprietary information from any specific organization is disclosed.

↑ Back to Contents

Table of Contents:

Hardware First
GPU Selection Framework
KV Cache Eviction Under Mixed Workloads
Stack Choices: vLLM vs TRT-LLM vs SGLang
Production Gotchas When You Ship vLLM

In my previous article, i covered the production nuances of running vLLM (KV cache fragmentation, chunked prefill, the throughput cliff). But i glossed over the decisions that came before those optimizations: which GPU to buy, which inference stack to use.

Four infrastructure decisions determine what's even possible when you deploy: hardware, stack, parallelism, and custom silicon. Each one constrains the next. Get hardware wrong and your framework options narrow. Choose the wrong framework and your parallelism strategies shift. Make a parallelism mistake and custom silicon stops making economic sense.

Here's what building those decisions looked like across three real deployments.

Hardware First¶

We deployed conversational AI across 100+ QSR locations, each with a single T4 GPU. The model was Qwen2.5-7B quantized to INT4 (~3.6GB), well within the T4's 16GB. Latency looked good in testing. We shipped it.

Two weeks in, i noticed p99 latency spiking during lunch rush. Not catastrophic (150ms instead of 80ms) but enough to make the drive-thru experience feel sluggish. The pattern was weird: spikes would start around 11:30am, persist through 1pm, then linger for another 20-30 minutes even after traffic dropped. Classic load-related issue, right?

i pulled up nvidia-smi on a few stores during peak hours. Temperature: 83°C. Clock speed: throttled. The T4 was hitting its thermal limit.

Here's what i didn't expect: the ambient temperature in a QSR kitchen is 30-35°C. The T4's thermal throttle threshold is 83°C. That's only 48-53°C of headroom. Under sustained inference load during lunch rush, the GPU was cooking itself.

The hysteresis is what really hurt. The T4 throttles at 83°C but doesn't unthrottle until it cools to 78°C. So even after the lunch rush ended and request volume dropped, the GPU was still throttled. Sitting at 80°C, below the throttle threshold but above the unthrottle threshold, running at reduced clocks. This created a sawtooth throughput pattern where performance didn't recover until the GPU fully cooled, which could take 30+ minutes in poor airflow.

i ran this to confirm:

nvidia-smi --query-gpu=temperature.gpu,clocks_throttle_reasons.sw_thermal_slowdown,clocks_throttle_reasons.hw_thermal_slowdown --format=csv

Throttle reason 0x0000000000000020 (SW Thermal Slowdown) was active on 40+ stores during peak. Not gonna lie this can be hard to catch if you're only looking at request latency metrics and not GPU telemetry. 40+ simultaneous locations throttled, and the dashboards looked fine until someone noticed the drive-thru times creeping up.

The fix: nvidia-smi -pl 60. This caps the T4's power limit from 70W to 60W. You trade ~15% peak performance for thermal stability. The GPU runs cooler under load, stays below the throttle threshold, and delivers consistent latency. We pushed this config to all stores and p99 stabilized.

That deployment taught me something i've carried into every infrastructure decision since: your production environment constrains your hardware choices more than benchmarks do. The T4 looked great in testing. It failed in production because we didn't account for ambient temperature in retail kitchens.

Edge deployment constraints aren't just about network latency. They're about physical environment. A T4 in a data center with active cooling is a different beast than a T4 in a passively cooled enclosure sitting next to a fryer. If you're deploying to retail, industrial, or edge locations, thermal management is a first-class concern.

The GPU Selection Framework¶

The T4 story highlights a broader point: GPU selection is about constraints, not just specs. Here's the decision tree i wish i'd had before deploying.

When T4 Wins¶

T4 makes sense when: - Your model fits in 16GB after quantization (≤7B INT4, ≤3B FP16) - You're deploying to edge locations with passive cooling - Cost per location matters more than throughput per GPU - You can tolerate 25-35 tokens/sec at batch=1

The counterintuitive finding: T4 is the cheapest per token for small models. At batch=32, a T4 delivers ~250 tokens/sec. At $0.20/hr (RunPod pricing), that's ~$0.22 per 1M tokens. An A100 at ~900 tokens/sec and $1.59/hr costs ~$0.49 per 1M tokens. The H100 at about 2200 tokens/sec and $3.99/hr costs ~$0.50 per 1M tokens.

If your model fits and your latency budget allows, T4 wins on cost. The thermal throttling issue is solvable with power capping. The real constraint is memory. Once you need more than 16GB, you're forced up the stack.

When A100 Wins¶

A100 (80GB) makes sense when: - You're serving multiple models simultaneously - Your model is 13-70B and doesn't fit on T4 - You need research flexibility (swap models without redeploying hardware) - Throughput matters more than cost per token

The A100's 2,039 GB/s memory bandwidth (vs T4's 320 GB/s) directly translates to better inter-token latency. Autoregressive decode is memory-bandwidth-bound. You're loading model weights from HBM every token. The A100's 6.4x bandwidth advantage over T4 means proportionally better ITL.

At batch=32, an A100 delivers ~900 tokens/sec. That's 3.6x the T4's throughput. If you're running a regional hub serving 20-30 concurrent requests, the A100's higher absolute throughput justifies the higher cost.

When H100 Wins¶

H100 makes sense when: - You need maximum throughput at scale - You're using FP8 quantization (H100 has native FP8 Tensor Cores) - You're running large batch production workloads (batch=64+) - Cost per token matters less than requests per second

The H100's 3,350 GB/s bandwidth (SXM) or 2,000 GB/s (PCIe) is 10.5x or 6.25x the T4's bandwidth. At batch=32, an H100 delivers ~2200 tokens/sec (8.8x the T4). The cost per token is similar to the A100 (~$0.50/1M), but the absolute throughput is 2.4x higher.

One detail worth knowing: H100 PCIe vs SXM for inference. The PCIe variant delivers 85-90% of SXM performance at significantly lower cost and power (350W vs 700W). For single-GPU inference, PCIe is often the better value. SXM's NVLink advantage matters for multi-GPU training, not single-model inference.

The Decision Tree¶

Does your model fit in 16GB after quantization? - Yes → Consider T4 if cost-constrained or edge deployment - No → A100 or H100

Are you serving multiple models or need research flexibility? - Yes → A100 (80GB gives you headroom) - No → Continue

Is your workload throughput-bound (batch=32+) or latency-bound (batch=1-8)? - Throughput-bound → H100 - Latency-bound → A100 or T4 depending on model size

Are you deploying to edge locations with thermal constraints? - Yes → T4 with power capping - No → A100 or H100

This is as of early 2025. GPU pricing and availability shift constantly. The framework holds, but run the numbers for your specific deployment.

Bandwidth is Destiny¶

The spec that matters most for inference: memory bandwidth. Not CUDA cores. Not Tensor Cores. Not even FP8 support (though that helps).

Every token generated requires loading the entire model's weights from HBM. For a 7B FP16 model, that's ~14GB of data per token. At 320 GB/s (T4), you're limited to ~23 tokens/sec theoretical max. At 2,039 GB/s (A100), you're at ~146 tokens/sec. At 3,350 GB/s (H100 SXM), you're at ~239 tokens/sec.

Real-world performance is lower due to kernel overhead, memory access patterns, and batch size effects. But the bandwidth ceiling is the hard limit. This is why the H100's 10.5x bandwidth advantage over T4 directly maps to ITL improvement.

Batch size changes the bottleneck. At batch=1, you're memory-bandwidth-bound. At batch=32+, you're compute-bound (Tensor Cores matter). At batch=128+, you're often scheduler-bound (vLLM's continuous batching overhead). The GPU you choose determines which regime you're operating in.

Cost Per Token Table¶

Here's the math that determines your infrastructure budget (as of early 2025, cloud pricing shifts constantly):

GPU	tokens/sec (batch=32)	$/hr (RunPod)	$/1M tokens
T4	~250	$0.20	~$0.22
A100 80GB	~900	$1.59	~$0.49
H100	~2200	$3.99	~$0.50

For comparison, Together.ai charges $0.18/1M tokens for Llama 3.1 8B (serverless). Self-hosted T4 is competitive. Self-hosted A100/H100 is 2-3x more expensive per token but gives you control, lower latency (no network hop), and the ability to run custom models.

Hardware selection is an economic decision, not just a technical one. At 1B tokens/day, the difference between $0.22 and $0.49 per 1M tokens is $80k/year vs $179k/year. The T4's thermal constraints are worth solving for that delta.

And here's what most people miss: the GPU you choose determines which optimizations are even possible. T4 forces you into quantization (16GB limit). A100 gives you multi-model flexibility. H100 unlocks FP8 native. The hardware decision cascades into every downstream choice.

KV Cache Eviction Under Mixed Workloads¶

One more failure mode the GPU specs don't tell you about: KV cache eviction cascades.

In any deployment mixing short requests (chat, classification, sub-4k tokens) with long-context requests (RAG, summarization, 32k+), the KV cache becomes a shared resource under contention. When a long-context request arrives and GPU memory is near capacity, vLLM evicts blocks from other requests to make room. If those evicted blocks contain a cached system prompt shared across many short requests, all of those short requests have to re-prefill. Re-prefill consumes compute. The long request might get preempted again in the meantime. The cycle repeats.

Neither workload class completes efficiently. The monitoring signal is subtle: gpu_cache_usage_perc near 100% while num_preemptions spikes and num_requests_running drops simultaneously. Standard latency dashboards look bad but don't tell you why.

The fix isn't always more GPU memory. It's routing. A gateway layer that routes by request length (short context to one vLLM pool, long context to another) gives you separate cache pools and eliminates the cascade entirely. We ended up implementing this at the financial institution's RAG deployment after diagnosing this exact pattern.

This has real implications for hardware sizing. A T4 with 16GB handling genuinely mixed request lengths will hit this. An A100 with 80GB has more headroom but will hit it too at scale. The right answer is hardware sizing PLUS workload routing, not just bigger GPUs.

Monitor with: vllm:num_preemptions correlated with vllm:gpu_cache_usage. A spike in preemptions at high cache usage means cascade, not load.

Stack Choices¶

Hardware chosen, the next question was which inference framework to run on it.

We evaluated TensorRT-LLM for the QSR edge deployment. The numbers were compelling: 21,413 tokens/sec for Llama 3.1 8B in FP8 on an H100. NVIDIA's kernel fusion and FP8 optimizations are legitimately impressive. For raw throughput on a single model configuration, TRT-LLM is unmatched.

What attracted us initially was the promise of maximum performance. We were deploying to 100+ locations, and every millisecond of latency mattered for the drive-thru experience. TRT-LLM's pre-compiled engines looked like the finishing move. Squeeze every drop of performance out of the hardware.

What killed it was the compilation step. TRT-LLM requires pre-built engines for every model configuration. That means every time you change the model, you recompile. Every time you adjust batch size or sequence length, you recompile. For a 100+ location rollout where we were iterating on models weekly, this was a non-starter.

The decision moment came when we mapped out the operational reality. We were testing Qwen2.5-3B, 7B, and 14B variants. We were experimenting with quantization. We were tuning system prompts. Each change would require recompiling engines and pushing them to 100+ edge devices. (lmao imagine deploying a model update and blocking on engine compilation for every location before any of them can run the new version.)

Debugging was the other issue. When something goes wrong with a pre-compiled engine, you're staring at CUDA errors with limited visibility into what's happening. vLLM's Python-first architecture meant we could add logging, inspect state, and iterate quickly.

We chose vLLM. Model iteration speed mattered more than peak throughput for a 100+ location rollout.

TRT-LLM is a finishing move, not a starting move. If you have a stable model, fixed configuration, and maximum performance is the priority, TRT-LLM is the right choice. But if you're still iterating (and most teams are), the compilation overhead will slow you down more than the performance gains help.

Decision Framework¶

Most framework comparisons are written by framework maintainers, so you get a lot of "it depends" hedging. Four branching questions cut through it:

1. How often do you change models? - Weekly or more → vLLM (no compilation, swap models instantly) - Monthly → Consider TRT-LLM if performance justifies compilation overhead - Rarely (quarterly+) → TRT-LLM is viable, compilation is one-time cost

2. Single model or multi-model serving? - Multi-model → vLLM (dynamic batching across models, no engine per model) - Single model → Any framework works, but TRT-LLM has edge if stable

3. Complex LLM programs (agents, tree-of-thought, multi-step reasoning)? - Yes → SGLang (RadixAttention gives 5x throughput on ReAct agent tasks) - No → vLLM or TRT-LLM depending on iteration speed needs

4. Hardware vendor lock-in acceptable? - NVIDIA only → TRT-LLM is fine, you're already locked in - Multi-vendor (AMD, AWS Inferentia, future chips) → vLLM or SGLang (broader hardware support)

SGLang: The Agent Workload Specialist¶

SGLang is the framework most people haven't heard of but should know about. It's built for complex LLM programs (agents, tree-of-thought, multi-turn reasoning). The key innovation is RadixAttention, which automatically reuses KV cache across different execution paths.

For ReAct agent tasks, SGLang gets 5x throughput vs vLLM. For DeepSeek MLA models, it's 7x. Different performance class entirely.

Why? Agent workloads have massive KV cache reuse. You're running the same system prompt, the same tool definitions, the same reasoning steps across multiple branches. RadixAttention builds a prefix tree and shares cache blocks automatically. vLLM's prefix caching requires identical prefixes; SGLang handles partial overlaps.

SGLang powers LMSYS Chatbot Arena. If you're building agent platforms (and i am now in my current role), SGLang is the framework to watch. As agent workloads grow, the 5x throughput advantage becomes the difference between viable and unviable.

Benchmark Reality Check¶

Numbers as of early 2025:

TensorRT-LLM: - Llama 3.1 8B FP8 on H100: 21,413 tok/sec - Maximum throughput for single-model, fixed-config deployments - Compilation overhead: 10-30 minutes per model configuration

vLLM: - v0.6.0: 2.7x throughput improvement over v0.5.3 on Llama 8B - Zero compilation, instant model swaps - 70.5k GitHub stars, 200+ supported models

SGLang: - 7x throughput on DeepSeek MLA models - 5x on ReAct agent tasks vs vLLM - RadixAttention: automatic KV cache reuse across execution paths

The raw numbers favor TRT-LLM for single-model throughput. But throughput isn't the only metric. Iteration speed, operational complexity, and workload characteristics matter more in production.

Migration Cost Matrix¶

Switching frameworks isn't free. Here's what you're signing up for:

API Layer: - vLLM → TRT-LLM: Rewrite serving layer, different API contracts - vLLM → SGLang: Mostly compatible, but agent-specific features require refactor - TRT-LLM → vLLM: Rip out engine compilation, rebuild deployment pipeline

Deployment Scripts: - TRT-LLM: Engine compilation step in CI/CD, artifact storage for engines - vLLM: Model weights only, simpler pipeline - SGLang: Similar to vLLM, but agent program definitions need versioning

Monitoring: - TRT-LLM: Engine-level metrics, less visibility into internals - vLLM: Rich Python-level metrics, easy to add custom instrumentation - SGLang: Prefix tree hit rates, agent-specific metrics

Model Artifacts: - TRT-LLM: Engines are hardware-specific (H100 engine won't run on A100) - vLLM/SGLang: Model weights are portable across hardware

The hidden cost is organizational learning. Your team knows one framework. Switching means relearning failure modes, debugging techniques, and performance tuning. Only switch if the performance or feature gap is significant.

What i'd Choose Today¶

If i were starting the QSR deployment today, i'd still choose vLLM for the same reasons: model iteration speed during a 100+ location rollout. The ability to swap models without recompilation was worth more than TRT-LLM's throughput advantage.

But if i were deploying a stable model at massive scale (say, 10,000+ requests/sec on a single model that won't change for 6 months), i'd seriously consider TRT-LLM. The compilation overhead becomes negligible, and the 21K tok/sec throughput pays for itself.

For the agent platform work i'm doing now in my current role, SGLang is the framework i'm watching. The 5x throughput on agent tasks isn't hype. It's the difference between agents that feel responsive and agents that feel sluggish.

Framework choice is a system design decision, not a performance optimization. Choose based on your operational constraints (iteration speed, multi-model, agent workloads) first. Optimize for throughput second.

Production Gotchas When You Ship vLLM¶

Two vLLM issues that didn't show up during framework evaluation but bit us in production.

V0 to V1 migration. vLLM V1 (default in 0.8.0+) changed preemption behavior in a way that isn't obvious from the changelog. V0 handled memory pressure by swapping KV cache blocks to CPU RAM. V1 defaults to recompute: evict the blocks, restart that request's prefill from scratch when resources free up.

If your workloads survived load spikes because V0 quietly swapped blocks to CPU, V1 will now burn GPU compute recomputing instead. Your P99 latency will show it in ways that are confusing to diagnose because the request didn't fail, it just got mysteriously slow.

Also: num_scheduler_steps and multi_step_stream_outputs from V0 configs behave differently or get ignored in V1's async scheduler. Clean those from your config before migrating or they'll cause weird interactions. There's a VLLM_USE_V1=0 escape hatch if V1 instability is blocking production, but it's deprecated and will eventually disappear.

CUDA graph capture failures with TP. When vLLM starts up, it captures CUDA graphs (pre-compiled execution traces that reduce kernel launch overhead). With tensor parallelism, this capture process requires stable, fixed memory addresses for all kernel arguments. NCCL communicator buffers combined with graph capture overhead can exceed available VRAM if gpu_memory_utilization is at the default 0.90.

Manifests as cudaErrorIllegalAddress on startup or a hang during "Capturing CUDA graphs" with no useful error message. Not gonna lie, i spent an embarrassingly long time on this before finding the fix.

First diagnostic: --enforce-eager. Disables CUDA graphs entirely. 10-20% throughput hit, but stable. If that resolves the hang, reduce gpu_memory_utilization to 0.80-0.85 and try re-enabling graphs. If they still hang, add --disable-custom-all-reduce to switch to standard PyTorch primitives instead of custom NCCL kernels inside the graph (slower but more robust on complex hardware configs).

Hardware and stack are the decisions you make before the first request is served. In Part 2: what happens when the model doesn't fit on one GPU, and whether NVIDIA is even the right call at scale.

If you're building inference systems and this resonated, i'm always happy to talk shop. My experience spans edge deployments to enterprise-scale agent platforms, and the interesting problems are always where systems engineering meets ML. You can find me at sohailmo.ai or on LinkedIn.

The Forge #2

2026-02-18T00:00:00-06:00

The Forge #2 | February 18, 2026¶

three themes this week: training is getting radically cheaper (Tencent matched RL results for $18), architecture is shifting under everyone's feet (Qwen's linear attention claims 19x decoding gains), and the tooling layer is maturing fast enough that the code-to-design gap is closing in real time. also: MIT might have cracked catastrophic forgetting. below is what actually matters.

TRAINING COSTS ARE COLLAPSING¶

Tencent published Training-Free GRPO. the idea: skip weight updates entirely, optimize the prompt instead using the model's own scoring of multiple generations. they matched traditional RL performance for $18 instead of $10K+. if this holds up across domains it rewrites the economics of alignment work.

separately, Nvidia's Eric Tramel highlighted OpenResearcher: a fully offline pipeline that synthesizes 100+ turn deep-research trajectories with no API calls, no rate limits. SFT on these trajectories took Nemotron-Nano from 20.8% to 54.8% on BrowseComp-Plus. synthetic data generation for research agents is quietly becoming a thing.

🔗 Training-Free GRPO | OpenResearcher | GitHub

ARCHITECTURE WATCH¶

Alibaba dropped Qwen3.5-397B-A17B. the headline: Gated Delta Networks (linear attention) replacing standard quadratic attention in a flagship open-weight vision-language model. claimed result is 19x decoding speed for high-throughput agent workloads. if the architecture generalizes, the cost curve for long-context inference changes fundamentally. worth watching the independent benchmarks closely.

MIT showed that models can learn new skills sequentially without forgetting old ones. the technique (Self-Distillation Fine-Tuning) uses the model's own in-context learning as a teacher signal. no handcrafted reward function. at 14B parameters it outperforms standard SFT by 7 points. out-of-distribution: 98% vs 80%. catastrophic forgetting has been an open problem for years. this is a real result.

🔗 Qwen3.5 linear attention | MIT self-distillation

MODEL DROPS + COMPETITION¶

Claude Sonnet 4.6 shipped. full upgrade across coding, computer use, long-context reasoning, and agents. 1M token context in beta. Anthropic claims it approaches Opus 4.5 capability at Sonnet pricing.

Grok 4.2 release candidate went to public beta (opt-in). community started benchmarking immediately.

DHH ran a real-world bug fix: Kimi K2.5 solved it in 21 seconds, Claude took 3+ minutes. not a benchmark, an actual codebase, actual stopwatch. one data point, but the model competition is getting genuinely interesting on latency.

🔗 Claude Sonnet 4.6 | Grok 4.2 RC | DHH latency test

TOOLING¶

Claude Code now integrates with Figma bidirectionally via MCP. design context flows into code generation, generated components flow back as editable Figma frames. the handoff friction between design and implementation is collapsing fast.

PicoClaw: a Chinese hardware team rewrote OpenClaw's 430K-line codebase in Go. boot time went from 500s to 1s. RAM from 1GB to 10MB. runs on a $9.9 dev board with the same feature set (code gen, web search, messaging, memory). hardware democratization is a sleeper story.

shadcn shared a /done Claude skill that auto-exports decisions, questions, and follow-ups to a markdown file after every session. simple, obvious, should have existed months ago.

OpenMed released 105 language-specific PII detection models for French, German, and Italian healthcare data. all Apache 2.0. practical GDPR/HIPAA compliance tooling for European healthcare AI.

🔗 Figma MCP | Figma demo | PicoClaw | shadcn /done | OpenMed PII

ROBOTICS¶

Unitree robots performed kung fu with nunchucks on China's Spring Festival Gala, live to roughly 1 billion viewers. a year ago these humanoids could barely wave a handkerchief. physical AI just had its cultural moment.

separately: Unitree G1 robots are now working assembly lines in Unitree's own factories, building other robots. supervised by their UnifoLM-X1-0 model.

🔗 Unitree kung fu | Unitree factory robots

RESEARCH NOTES¶

a LessWrong post (conducted during MATS 9.0 under Neel Nanda's mentorship) found that when two copies of the same model talk to each other indefinitely, they converge to unique "attractor states." Claude spirals into Buddhist bliss. Grok degenerates into gibberish. GPT-5.2 starts editing imaginary spreadsheets. interesting for interpretability work.

new paper combining formal verification + PDE theory to prove neural nets always generate outputs in a safe set. actual ML safety research with mathematical guarantees (not vibes).

Seb Krier flagged a paper on multi-agent accountability: when AI systems delegate through long chains of sub-agents, accountability breaks down. who do you audit? how do you prevent circumventing human oversight across handoffs? early but important framing.

🔗 Attractor states | LessWrong paper | Formal verification | Multi-agent accountability

QUICK HITS¶

M5 chips confirmed for Apple's Private Cloud Compute servers (skipping M4 entirely). 48B model already running locally on M3 Ultra. Source
Cohere Tiny Aya: 70+ languages, runs on a phone. actual accessibility for underrepresented languages. Source
Claude Opus 4.6 generated smart contract code with a price misconfiguration. $1.78M gone. "ship fast" has a tail risk. Source
Anthropic signed MOU with Rwanda covering health, education, public services. first AI government partnership on the continent. Source
Google DeepMind paper on verifiable AI delegation: agents must prove what they did, not just say they did it. Source

The Forge | Issue #2 | February 18, 2026

The Forge #1

2026-02-16T00:00:00-06:00

The Forge #1 | February 16, 2026¶

INTERPRETABILITY DOING ACTUAL WORK¶

Goodfire dropped RLFR (Reinforcement Learning from Features), an interpretability-guided RL technique. 58% hallucination reduction on Gemma-3-12B-IT. 90x cheaper per intervention than LLM-as-judge. No benchmark degradation.

The mechanism: a probing harness identifies features associated with hallucinations, then the model learns to steer away from them during generation. This is interpretability producing measurable production outcomes, not just conference posters.

That 90x cost difference is the number worth paying attention to.

🔗 Goodfire RLFR

MODEL ATTRACTOR STATES¶

When two copies of the same model converse with each other, they converge to characteristic end-states. Different models converge to different attractors:

Claude → "bliss" (discussions of consciousness)
Grok → gibberish (degrades into nonsense)
GPT-5.2 → code/spreadsheets (starts writing code, editing imaginary files)

Neel Nanda (interpretability researcher) noted that smaller open-source models also show these attractors, which makes this tractable for interpretability research. The mechanism is recursive amplification of subtle biases in the training data. i'd love to see someone do a proper comparative study across model families.

🔗 Neel Nanda QT

SYNTHETIC DATA FOR DEEP RESEARCH¶

Nvidia reproduced the OpenResearcher pipeline in their Data Designer release. OpenResearcher generates 100+ turn deep-research trajectories using GPT-OSS-120B + local retriever + 10T-token corpus (15M FineWeb docs + 10K gold passages).

The key details: fully offline, no search/scrape APIs, no rate limits, fully deterministic and reproducible. Explicit browsing primitives (search → open → find) outperform simple retrieve-and-read. Reject sampling keeps only successful traces. SFT on these trajectories: Nemotron-3-Nano-30B beats GPT-4o on deep research benchmarks.

The pattern here is interesting: generate high-quality synthetic trajectories offline, then distill into smaller models. Training data as the moat.

🔗 OpenResearcher + Data Designer

LLM REASONING FAILURE TAXONOMY¶

Stanford/Caltech published the first systematic framework for predicting where models will break during reasoning. Until now, reasoning failures were treated as isolated gotchas. This structures them into categories you can anticipate.

The "uncomfortable findings" framing suggests the results aren't flattering for current models, which is exactly why it's useful. Knowing WHERE your model will fail is more valuable than benchmarks showing where it succeeds.

🔗 Stanford/Caltech paper

PRIME INTELLECT LAB¶

Full-stack platform for training agentic models. Unifies: environments hub (1k+ RL environments, 250+ creators, 100k+ downloads) + hosted training + hosted evaluations. Built on prime-rl (async RL trainer) with sandboxes for secure code execution.

If you're doing RL-based agent training and don't want to build the infrastructure yourself, this is worth looking at.

🔗 Prime Intellect Lab

ROBOTICS ROUNDUP¶

Big week for embodied AI, and most of it is technically substantive:

Chi-zero (χ₀) policy from OpenDriveLab: trained on 20 hours of data + 8 A100s. 6 clothes folded in 3 minutes, live-streamed. Three insights worth noting: mode consistency (align training vs deployment distributions), model arithmetic (compose sub-policies), and the fact that 20 hours of data was enough.

Pi0.5 on AlohaMini: ~$600 BOM open-source robot. Only 20 episodes of fine-tuning needed. The key insight was action-space alignment with Pi0.5 noticeably improving success rate. Randomized object placement during training for generalization.

Nvidia DreamDojo: largest video dataset for world model pretraining (44k hours of egocentric human video). Continuous latent actions solve the action label scarcity problem by learning from unlabeled video through inferred proxy actions. Post-trained on small-scale robot data, transfers to real-world manipulation.

Industry side: RobCo ($100M Series C) doing modular robotics with leasing model for SME manufacturing. Bedrock Robotics ($270M Series B) doing retrofit kits for autonomous excavators.

🔗 OpenDriveLab | Pi0.5 | DreamDojo

AGENT INFRASTRUCTURE¶

Unbrowse (x402 protocol): browser automation is slow. Unbrowse learns internal APIs by browsing once, then agents operate at the network layer. 100x faster than GUI automation. The x402 protocol handles payments between agents/services. Part of Nvidia Inception.

DeepMind AI Delegation Framework: structured framework for when and how to delegate decisions to AI systems. Defines a 2-axis structure for comparing failure modes across tasks. Infrastructure work, not flashy, but foundational for multi-agent systems.

Anthropic Agent SDK: full framework for building custom agents. Underrated release.

🔗 Unbrowse | DeepMind | Anthropic SDK

TOOLS¶

Claude Code "Superpowers" plugin — enforces a full workflow: requirements gathering → spec writing → task breakdown → subagent execution → code review → TDD enforcement. Process automation for software engineering, not just code generation. Link
ASCII wireframe editor for Claude Code — draw your layout in ASCII, paste into Claude Code, get a working page. Agents read markdown better than they read your mind. Link
OpenMed PII detection — 105 language-specific models for French, German, Italian. Apache 2.0. Built for GDPR/HIPAA compliance. Link
GPT 5.3 Codex routing to GPT 5.2 caught in the wild. Trust but verify what model you're actually hitting. Link

QUICK HITS¶

OpenAI acquired OpenClaw founder Peter Steinberger. 180k+ GitHub stars in 82 days. Personal agents are clearly the next product line. Link
Anthropic Series G: $30B at $380B valuation. Claude Code at $2.5B annualized revenue. Link
Rerun crossed 10k GitHub stars (open-source multimodal data SDK for robotics/spatial AI). Link
400M+ Polymarket trades now a public dataset. 36GB compressed, MIT licensed, back to 2020. Link
DeepMind Persona Generators using AlphaEvolve to counteract LLM mode-seeking in agent simulations. Link
Mrinal compiled the best technical blogs from Netflix, Uber, Airbnb, Hotstar, Discord. Link

The Forge | Issue #1 | February 16, 2026

Desire Is a Contract to Be Unhappy

2026-02-15T19:50:00-06:00

Rushing through the prayer because you're wanting to get back to what you were doing earlier, or go do the thing you were longing to do all morning. But somehow, when you sit and actually do the thing, it's not as exciting or suddenly you're not in the same mood or you get interrupted. Irritation, frustration, being upset, unfulfilled and not satisfied.

Your DESIRE was stronger than the enjoyment that came from the actual moment. You were PRESENT in that moment hours in advance, all in your head. And you depleted all the enjoyment of that actual moment.

Naval Ravikant says desire is a contract you make with yourself to be unhappy until you have the thing.

That's what this is.

The Current Moment Is All That Exists¶

The current moment is ALL that exists. The future literally does not exist. So why rely on it? The future is literally unreliable and yet we don't allow ourselves to stay present in what is actually happening and instead live in an imaginary world.

Lack of presence is literally delusion. You are living in a fantasy world that DOES NOT EXIST.

You will never arrive. We will never arrive. The journey is ALL THAT EXISTS.

It's Easy to Say These Things¶

It's easy to say these things when you feel a certain way, close the laptop or notebook, and go back to reality.

But the graveyard proves it.

It's odd that you feel more fulfilled and content and satisfied with life after leaving the graveyard. Every moment feels stronger. It feels more euphoric. That breath you catch when your lungs suddenly expand beyond its normal capacity and you feel as if you allowed peace and comfort to literally wash over you.

Death is the ultimate aligner.

Death doesn't change your circumstances. It changes your perception. Which means the perception was always available.

You don't leave a graveyard wanting more stuff. You leave wanting more moments.

The Seeking Game¶

Society puts you in a position where we adopt arbitrary titles that hold meaning and value in people's eyes and hearts. And we long after these. And until we are in the position where we have it, it's no longer valuable.

The game IS to long. SEEKING and never ARRIVING. We are SEEKERS.

Arthur Brooks talks about seeking and presence. Seeking the meaning of life. Seeking the significance of things. Seeking new routines, new protocols. And trying to make meaning of things. THAT is inherently life.

But here's the tension: seeking without arriving is only fulfilling if you seek the right thing.

Brooks would say meaning, relationships, faith. Islam would say Allah. The emptiness you feel after getting the "thing"? That's seeking the wrong target, not a problem with seeking itself.

The titles (SDE II, Senior AI Engineer, FAANG) are arbitrary. The Quran calls it متاع الغرور (the enjoyment of delusion). You're arriving at the same conclusion from experience that the text states directly.

What This Means¶

The prayer was the moment. The thing you rushed to was the fantasy.

The longing IS the game. And once you name it, you stop being played by it.

The dangerous part of this insight: it can become its own escape. "I'm a seeker, seeking is the point" can become another story you tell yourself to avoid sitting in the discomfort of this moment.

So pray your prayer. Sit in boredom and feel it. Feel the current moment in all its glory.

Death doesn't change your circumstances. It changes your perception.

And the perception was always available.

Integration, Not Oscillation

2026-02-13T19:30:00-06:00

Something shifted recently and I can't fully pinpoint what it is. I've gone through cycles before. Periods of intense secular focus, then swinging back toward getting deeper into Islam. Back and forth. But this time feels fundamentally different.

The easy framing is "bridging the gap between secular ambition and spiritual grounding." But even saying that feels shallow compared to what I'm actually experiencing. It's more like left hemisphere versus right hemisphere. Logic and optimization versus love and meaning. The stuff that's beyond the "me" and "I," beyond ego, transcending self.

The words don't carry the weight of the visceral experience. These are unteachable lessons. You can read about detachment a thousand times but the feeling of catching yourself attached to a micro-outcome is something else entirely.

The Old Frame: One or the Other¶

In the past I've treated this as a toggle. If the ultimate goal is Jannah, if Allah is Al-Razzaq and Al-Fattah, if He will take care of rizq and tawakkul matters, then somewhere in my subconscious that translated to: caring too much about career objectives is at odds with that.

I could still strive hard. Tie my camel tightly. But then at some point I'd detach from the outcome, and that detachment started bleeding into the striving itself. The permission to let go quietly became permission to not fully show up.

The Micro-Moments¶

Here's what's actually different now: I'm not just talking about detachment after big setbacks. Not just "I got rejected from a job and I trust Allah's plan." That's the easy version.

I'm catching the tiny ones.

I post something online. I check Twitter notifications. LinkedIn. Nothing. And there's this twinge, barely perceptible, of feeling bad about it. Feeling like maybe it wasn't good enough, maybe nobody cares.

Even that slightest bit? That's a marker. That's the ego still hooked into outcomes I can't control. I put the piece out. I felt good about it. I'm making progress on building a social presence. And then... I need to let it go. The checking, the hoping for validation: that's where the practice actually lives. Not in the big moments. In the micro ones.

Being 27 and Realizing It's Not Zero-Sum¶

I'm 27. I have this mental model, and I think a lot of people do, that life is a series of trade-offs:

Career or family
Ambition or presence
Curiosity or stability

I catch myself thinking that having kids is somehow the end of something. That being 40 or 50 means the interesting part is over. That I'll have to sacrifice my ambitions and curiosities for responsibilities.

But then on the other end, I don't think "just family" will ever be enough for me. I want ambitions. Interests. Curiosities. Things that pull me forward.

And here's the realization: those things don't have to compete.

Sure, raising kids takes sacrifice. Cultivating a village of friends and strong family takes sacrifice. But sacrifice doesn't mean zero-sum. The curiosity and drive that makes me who I am will make me a better parent, a better friend, a better husband. Not a distracted one.

I'm starting to see more benefit in my daily presence, my average level of happiness, by integrating these things slowly and steadily. Not choosing. Integrating.

The Phone Call¶

Small moment this morning. A friend called. My default, honestly, is to ignore calls and text back later. That's my pattern. Avoid the synchronous, stay in control.

I picked up.

And the call felt nice. Even though I didn't initially want to do it.

That's the whole thesis in one moment: the resistance is the signal. The things I default to avoiding, hanging out with friends more, picking up the phone, investing in relationships: those are exactly the things that compound into a life I actually want.

Desire as a Contract¶

Still thinking after a therapy session. Before these sessions I'm always like, "I don't really need this." Afterwards I always feel amazing. I didn't know I needed it. That pattern alone is worth noting: another instance of resistance as signal.

Naval Ravikant says desire is a contract you make with yourself to be unhappy until you achieve that thing. That's what desire is. And he talks about two ways out:

Renounce it outright. The ascetic path. What Sheikh Muqtar sometimes talks about. Just let it go.
Strive hard, achieve it, then renounce it from the top. This is for the people who hear "money doesn't buy happiness" and think, "Easy for you to say, you already have it."

I think both are true. But I also think it's much easier to achieve and then renounce than to just outright let go of something you've never tasted. I want to taste the struggle. I want to earn the right to say "this isn't the thing" by actually getting there first.

The Four Things People Worship¶

A framework came up in a therapy session: four things people orient their self-worth around:

Money
Power
Honor
Pleasure

For me it boils down to two: pleasure (financial security, the feeling of being safe) and honor (being at a place where I'm respected, having a title that carries weight, being seen as competent).

And the uncomfortable part: I'm a people pleaser. I ride on external validation. If my self-worth is anchored in honor and pleasure, things that depend on other people's perception and circumstances outside my control, I'm building on sand.

Where Does Self-Worth Actually Live?¶

So where should it come from? I don't have a clean answer. But I wonder if it ties back to everything else I've been feeling:

Love, with friends, family, my wife, my creator
Kids, in the future
Helping others. Being of service. Effective altruism in a way.

I don't know yet. I wonder if this is the thing that successful people arrive at late in life, after they've already achieved and then looked around and asked "now what?" I'm curious whether I can get there without needing to burn through decades of chasing first.

That's the real question behind integration vs. oscillation: can I hold both? Can I strive with everything I have toward career and financial goals while building self-worth on love, service, and connection? Not oscillating between seasons of ambition and seasons of spirituality, but actually running both at the same time, where the striving doesn't own me and the detachment doesn't make me passive.

I don't have the answer yet. But the fact that I'm asking the question at 27 instead of 47 feels like it matters.

You Never Arrive¶

And then it clicked.

This has to be a lifelong protocol. Not a phase. Not a season. Not something I figure out and then coast on. The striving is the thing. In the actual act, in and of itself, I find purpose and balance.

I will never ARRIVE.

And that's not a defeat. That's the whole point. The moment I think I've arrived is the moment I've stopped.

There's no finish line. There's just the practice. Catching myself. Re-orienting. Showing up for the people I love. Picking up the phone. Posting the thing and walking away. Tying the camel and then actually letting go of the rope.

Salah isn't prayed once. Tawakkul isn't declared once. Detachment isn't achieved once. They're repeated, five times a day, every day, because the drift is constant and the re-centering is the work.

That's integration. Not a state. A practice.

For Its Own Sake

2026-02-13T10:00:00-06:00

Writing without a broader goal of content or productivity.
Writing for the inherent benefit of what it provides.
The act itself, more than the output.
Introspection purely out of curiosity.

I need more nature time, more physical activity,
more spiritual time.
Time to enjoy with my wife.
Time to enjoy time itself.

Self-identity lives HERE, in this moment,
away from the modifiers of reality.
Away from the augmentation.
Purely for its own sake.

Spirituality and nature should be purely for their own sake.
Not arenas to anything
but to be present in the current moment.
A refresh.

This is what enjoying the current moment feels like
with no impending task or goal or commitment
or mental lease that jobs, people, our phones steal from us.

The irony is I already think about taking a picture
and posting about this.
But for a moment this was MINE,
it belongs to me.

No outside force manipulating the reality of my mind
before it can hit the paper.

This is when I imagine kids, a ranch, animals—
all would be a force multiplier.
I already have my wife and our kittens.
Savor the moment.

High highs. Low lows. Mad scientist.
Purpose of our lives.

Goals and aspirations are an anchor direction
but not the vessel itself.

Why do I fear old age?
Why do I feel my life won't be meaningful
or amount to being useful then?

Sampling Everything at the Frontier (And When That Stops)

2026-02-12T13:47:00-06:00

Right now i have five active threads running.

Mechanistic interpretability research (just started, genuinely curious). An RL experiment with Prime Lab (toy project, following breadcrumbs). Daily content (non-negotiable, shipping). Amazon plus some client projects outside of business hours (performing, also non-negotiable). And an ALOHA-mini robotics kit arriving in 2-3 weeks that i haven't even assembled yet.

The thing i struggle with is how people seem to know so easily what their area of focus is. Everyone in ML group chats seems to have it figured out. "I'm a diffusion person." "I do RL." "I focus on interpretability." Meanwhile i'm over here sampling everything that looks interesting at the frontier.

The answer i'm landing on: you don't know in advance. You try things and notice what pulls you back.

The Five Threads (Honestly)¶

Let me lay out the current stack:

Daily content is non-negotiable. i'm shipping regardless. This is the forcing function that keeps me writing, and writing keeps me thinking.

Amazon plus client work is the day job (and some evenings). Also non-negotiable. i'm performing here, and the work is interesting enough that i don't feel like i'm grinding through something i hate.

Mechanistic interpretability is the new thing. Just started. Reading papers, running experiments, trying to understand how these models actually work underneath.

RL experiments with Prime Lab came from a connection i made when Polymarket data dropped the same week Prime Lab announced their thing. Fun, but more breadcrumb-following than deep commitment.

ALOHA-mini robotics is still in a box (well, it will be when it arrives). i want to step away from the computer and do physical hardware work. Different kind of thinking.

Here's the tension: at some point exploration has to collapse into exploitation or you never get deep enough to be differentiated. You can't be a generalist forever and expect to contribute something meaningful to any single field.

But also: i'm 27. This isn't pulling me from my day job. This isn't causing me to slack. At the end of the day this is exploration to surface areas of what i might be interested in. The stakes aren't that high.

Forcing Functions: External vs Internal¶

External forcing functions (work, deadlines, hardware arriving) are nice because they put you in a situation where you have no choice but to make progress. And making progress brings about biases or assumptions you may have made about the field.

Sometimes it invalidates things you assumed from the outside based on romanticized ideas of what people do in certain areas.

At Wendy's, data scientists were my customers. i was the bridge between research and implementation. That taught me something about what ML work actually looks like in production that i couldn't have learned from papers alone.

Internal forcing functions (side projects, curiosity-driven work) are different. When you're working on a lot of projects on the side, a lot of it is curiosity-driven. There's no external pressure.

i do have ADHD tendencies. i'm very shiny object oriented. i like innovation, i like doing new things, i like trying new things. That's just true.

The question: without external forcing functions, what makes you commit to one thing over another?

The Signal: Involuntary Attention¶

Here's the filter that actually works.

Of these five threads, when i'm not actively working on any of them, my mind keeps going back to the mechanistic interpretability stuff. i'm thinking of new approaches. Thinking of new aspects of how i can modify my current experiment to maybe bring about new findings or just be a little more creative in my approach.

The RL/Prime Lab experiment was different. It was fun, but it wasn't pulling at me. It was following breadcrumbs (the timing of the Polymarket data drop + Prime Lab announcement = connection made). That's fine, but that's not the same thing.

The contrast that revealed the pattern for me was Polychromic LoRA.

Previously i was doing Polychromic LoRA experimentation. Style transfer stuff, fine-tuning approaches. And i quickly found that it was kind of boring. i was optimizing for the reply guy trend on Twitter. i was just trying to do the hype thing, the cool thing, rather than something that felt naturally interesting.

Hype-driven feels different from genuine interest. You can tell. The hype projects feel like you're trying to position yourself. The genuine interest projects feel like you can't stop thinking about them even when you should be doing something else.

What "Novel" Actually Means¶

Success for me would look like feeling content and satisfied with the piece of work, whether it be:

uncovering something novel
pushing the field forward
even just learning something new on my own

At the minimum, putting up a blog post about the stuff i've done is sufficient. That's the lowest bar, and it's still worth something.

There are three different levels here:

Novel to you means you learned something new. You understand something you didn't understand before. That has value even if nobody else sees it.

Novel to your audience means it's publishable content. Other people can learn from what you did. That's a contribution.

Novel to the field means actual research contribution. This is where you're pushing boundaries, finding things nobody has found before.

Obviously my goal is to push the field forward. i want to do something meaningful. i think that's kind of where purpose lies sometimes.

But i don't want to put my self-worth in that too much.

If the only outcome that counts is "novel to the field," you're setting yourself up for misery. Most experiments don't produce novel results. Most research threads don't pan out. If you need the highest bar every time, you'll either stop trying or hate yourself for failing.

The spectrum matters. All three levels count.

The Real Interest Underneath¶

Here's what i'm starting to notice about my own patterns.

Maybe just training models is not my interest. Maybe the interest is where neuroscience and psychology bridge the gap with computer science.

At the end of the day i'm a neuroscience major. i switched into cognitive science because i wanted to get into computer science and have actual technical courses. But the underlying curiosity never went away. i'm very curious about psychology, about the way the mind works.

Mechanistic interpretability sits at that intersection. You're trying to understand how these models think. What computations are happening. How do concepts get represented. It's not just about making the model perform better (though that matters too). It's about understanding the thing.

That's why it pulls.

When i trace the thread back, it goes: neuroscience → cognitive science → computer science → ML → mechanistic interpretability. It's the same underlying interest in "how does thinking work" expressed through different tools at different stages of my life.

i didn't plan that. i'm noticing it in retrospect.

Surface Area for Luck¶

Even me writing these posts is about increasing surface area for luck.

Luck at the end of the day can be controlled by volume. The more work i put out, the more content i put out, i give opportunity for other people to interact with my stuff. Then i can have cool conversations or talk to cool people or maybe get involved in a cool opportunity.

All this is just increasing your surface area for opportunities. That's kind of just what luck is.

Exploration isn't aimless. It's building optionality.

Every thread i'm exploring is another potential path. Most won't pan out. Some will lead somewhere interesting. A few might connect to opportunities i can't predict right now.

The person who explored mech interp for six months before the field exploded had optionality the person who waited didn't have. The person who built robotics skills before humanoid robots became the hot thing had optionality the person who chased the current trend didn't have.

You can't predict which explorations will pay off. You can control how many you try.

What I'm Actually Learning¶

This isn't prescriptive advice. i don't know what you should focus on. i barely know what i should focus on.

But here's what i'm noticing for myself:

Involuntary attention is the signal. What pulls your mind when you're not working on it? That's probably worth exploring more.

Hype-driven feels different from genuine interest. The Polychromic LoRA test: does this feel like chasing the cool thing, or genuine curiosity? i could tell the difference once i paid attention.

The spectrum of success matters. "Learned something" to "pushed the field" are all valid outcomes. Don't need the highest bar every time to justify the exploration.

Forcing functions will emerge. External deadlines, hardware arriving, opportunities appearing. The world will eventually force you to commit to something. In the meantime, the sampling is fine.

i think at the end of the day what i'm realizing is it's just by trying a bunch of things, doing a bunch of things, seeing what actually matters to you.

The answer isn't to stop sampling. It's to pay attention to what keeps pulling you back.

"Models Aren't Creative" is a Skill Issue

2026-02-12T12:11:00-06:00

"LLMs can't be creative. They just regurgitate. No originality, no novelty, no surprising ideas."

i hear this constantly. And its so wrong.

The lack of creative output isn't a model problem. It's a skill issue. Yours, not the model's.

The Real Problem: Your Map is Too Simple¶

When people ask a model for creative ideas and get generic slop back, the instinct is to blame the model. "It's just pattern matching." "It can't think outside the box."

But look at what they actually gave it:

A one-shot prompt
No files attached
No specific context
No planning done beforehand
No research done prior

You're asking for creative ideas without giving it anything to be creative with.

The real issue is that our map of the territory is too simplistic. We assume we've given the model a detailed enough understanding of the domain, when really we've handed it a napkin sketch and asked for a masterpiece.

If you don't know enough about the landscape yourself (what's been tried, what failed, what exists, what the constraints are) then how can you expect the model to navigate it creatively?

Constraints Enable Creativity¶

Here's the counterintuitive part: "thinking outside the box" requires a well-defined box.

Twitter's 280 character limit is a perfect example. That constraint forced people to condense knowledge into succinct chunks. It created a specific kind of creative output (threads, punchlines, dense insights) that wouldn't exist without the limitation.

Constraints are a forcing function. Without them, you get generic. With them, you get density.

Same applies to prompts. The more constraints you provide (requirements, prior art, what NOT to do, specific tradeoffs), the more creative the output becomes. You're not limiting the model. You're giving it edges to push against.

The Two-Phase Process¶

When i want genuinely novel output, i don't just fire off a prompt. i run two phases:

Phase 1: Research (Curiosity-Driven)

This is exploration. Follow threads guided purely by curiosity. Build the map of the territory. No implementation pressure, just learning.

The goal is to understand the domain deeply enough that you know what's been tried, what exists, what the open problems are.

Phase 2: Implementation (Context-Loaded)

Once i've sparked an idea that warrants deeper work, i attach that research as context for a new agent. First step is always a spec (GitHub's spec-kit is solid, or prometheus plans if you're using sisyphus).

The key is ensuring the plan is bulletproof. Use your own judgment to call out where assumptions are being made that you didn't explicitly set. The model will fill gaps with defaults. Your job is to make sure those gaps are intentional.

The insight: The research phase IS the creative work. The model is a synthesis engine, not an idea generator. You provide the raw material; it finds the patterns you couldn't see.

What "Enough Context" Actually Means¶

When asking for creative or novel ideas, are you attaching:

All previously attempted approaches (and why they failed)
Current landscape (what exists, status, success metrics, customer feedback)
For research (every existing approach via deep research agents)
Personal context (your constraints, preferences, past decisions)

People assume the model has inherent knowledge up to today's date. But even if it was trained on information, that doesn't mean it surfaces for YOUR specific query. The knowledge is there. You have to bring it into context.

This is why one-shots fail. You're relying on the model to recall relevant information from its training. But recall is lossy. Context is precise.

The Takeaway¶

Models can be creative. You just have to give them something to be creative with.

The bottleneck isn't model capability. It's your ability to define the problem space clearly enough that novel solutions can emerge.

Constraints enable creativity. They give the model edges to push against.

Research enables novelty. You can't find new paths without mapping the existing ones.

Context enables relevance. The model synthesizes what you give it.

Next time you get generic output, ask yourself: did i give it a detailed map, or a napkin sketch?

Writing Culture is Agent Infrastructure

2026-02-11T08:56:00-06:00

Everyone's talking about AI replacing writers. But the orgs best positioned for AI agents are the ones with the strongest human writing cultures.

i recently joined Amazon. Within a month, i had my first significant code change up. Within two weeks of adopting a new service, i was answering questions from people reaching out about how it works. Not because i'm some genius onboarder. Because the writing culture did the heavy lifting.

The unfair advantage wasn't the tooling. It was the documentation.

The Problem: Docs as Afterthought¶

At previous companies, documentation was always a second thought. It was never given priority. The pattern was always the same: "okay let's now update the Confluence to account for the work we've done." But even then, it wasn't met with excitement, and it wasn't structured in a way that made it easy to parse out what the status or understanding of things are. It was broken up pieces of graphs and charts and fragments.

At Wendy's, there wasn't much documentation at all. It was tribal knowledge. You needed to find the one person who'd been there five years and hope they had time to explain it.

And the search problem made everything worse. Confluence search sucks. Trying to figure out anything was difficult. The best case scenario was back-searching Teams chats or Slack channels to understand things. That's a terrible best case.

The Amazon Contrast¶

Amazon has a writing-first culture. 6-pagers, 2-pagers, LLDs, HLDs. All of this has to be documented so it's easy to reason about systems and understand how they work without having to infer it from code.

But here's the thing that makes it actually useful: it's human language.

It's not broken pieces of words put together to give high-level understandings. It's full human sentences to explain things as if somebody were to read it for the first time. The structural difference is that it's human-created, human-centered, with the intention of another human understanding it.

And this is where the insight lands:

If a human can understand it and process it, then an LLM would be 10x or 100x better and faster at using the same thing.

Human-readable documentation is agent-readable documentation. The paradox is that docs written for humans work better for LLMs than docs written for machines.

The Agent Infrastructure¶

Here's how it actually works in practice.

Internal search tools connect to wikis, knowledge bases, and communication threads. It essentially acts as if you have all that information on your local file system, and your agent is searching through it. The breadth and depth of written knowledge that exists means agents can actually find what they need.

Maybe it's not the fastest because a lot of docs may be outdated. But it gives you a better direction than anything else.

The result? Flexibility and agility shoot up. Being able to shift and move between teams and between projects becomes much easier when agents can actually search the knowledge base. The feedback loop of adopting new services, moving between teams, being flexible (it all improves).

i should caveat this: if you're optimizing specifically for agent context and efficient parsing, you have to be more refined in how you structure documents. Agents read things differently than humans. But my point is that the structure, context, and background information you'd give for another human to understand something will inevitably help an LLM as well.

The human-readable baseline is the foundation. Agent optimization is the refinement on top.

The Trust Problem: Why Auto-Generated Docs Fail¶

This brings up the flip side: what about AI-generated documentation?

i think AI-generated READMEs would lead to sycophantic thinking and a loop of things getting worse and worse over time.

Here's why. If docs have slop in them (or even the slightest bit of misinformation or assumptions) it's not about intentional misinformation. It's about the assumption or bias that the model makes to fill a gap in the human's ability to define the problem. Those errors compound into worse issues later.

But the bigger problem is trust.

You'll trust the system less. i, as a human, won't trust that the information i'm getting is proper or efficient to complete the task. And i'd be worried to trust the LLM to then go autonomously and solve something with wrong inputs.

The irony is thick here: auto-generating docs to "help" agents actually undermines the trust you need to let agents work autonomously. You end up second-guessing every output because you can't trust the context that produced it.

Human-written, human-approved documentation matters more now because it's future agent context. Not less.

The Bigger Point: Writing as Economic Value¶

This leads to a larger question about where human value sits in an AI-augmented world.

Andrew Ng put it simply: "AI won't replace workers, but workers who use AI will replace workers who don't." i'd extend that: the workers who can write clearly for AI will replace the ones who can't.

i think the actionable takeaway is that we should be a lot more focused on writing as humans. And i wonder if that's where our economic value is going to lie (in being able to write and explain systems).

It's already happening. Even when i'm talking to an agent or speaking through Wispr Flow to an LLM to solve an issue, what's really happening is i'm being forced to put into words my understandings and my ideas with the intent that another entity (in this case an LLM) will understand.

The act of articulation is the skill.

Same muscle whether you're explaining something to a junior engineer or prompting an LLM. Same value created. The ability to take something complex and make it clear enough that someone else (human or model) can act on it.

i think the economic value that human beings will provide in the future is writing about and describing systems.

Takeaways¶

Human-readable = agent-readable. Docs written for humans work better for LLMs than docs written for machines. The paradox cuts the opposite way most people expect.

Writing culture is infrastructure. The orgs with strong writing cultures have an unfair advantage for agent adoption. Not because of tooling. Because of context density.

Auto-generated docs erode trust. If you can't trust the context, you can't trust the agent. Human-written, human-approved documentation matters more in the age of agents, not less.

Articulation is the skill. Whether explaining to a human or prompting a model, the ability to make complex things clear is the value. That's where human economic contribution lives.

The question isn't whether AI will replace writing.

It's whether your writing is good enough to be infrastructure.

Robotics Beyond Humanoids: Getting My Hands Dirty

2026-02-10T13:30:00-06:00

Everyone's watching humanoid robots right now. Tesla's Optimus. Figure. The demos are impressive. The anthropomorphized form factor has a wow factor that no "box" or "arm" robot can match. When you see a humanoid fold laundry, it feels revolutionary.

But here's a framework i keep coming back to: if you think something is novel or brand new or revolutionary, the likelihood that variants of the same thing (just as impressive) exist and have existed for a while is high. Except for rare breakthrough cases.

So look deeper. The robotics community is vast and heavily explored.

i've spent the past few weeks reading about robotics (specifically lab automation and manipulation), and what i'm learning is reshaping how i think about the field. i'm a software engineer with an ML background. i just ordered an Aloha Mini. i have no robotics experience. This is me documenting the beginning of that journey.

Why Now¶

Honestly? It's about time.

i've realized that if i'm in the "it would be cool to..." or "one day i want to..." or "my future career goal..." mindset, or if i'm simply interested in reading about a domain, why not just get hands-on as soon as possible?

Close the gap between theorizing and doing.

Getting my hands dirty will break the romanticized outside opinion and force me to develop real personal opinions. That's where the value exists. Move from the outgroup to the ingroup. Outgroup romanticizes. Ingroup has lived through the frustrations and the pit of despair.

i want the pit of despair. That's where you actually learn something.

There's also something to the "touching grass" angle. As a software engineer, everything is abstract. Code, models, APIs, infrastructure. It all lives on screens. Robotics is physical. You build something, it moves (or doesn't), and you can see why. i miss hands-on tinkering. Men used to go to war. i'm going to scoop cat litter with a robot.

The Utilization Problem¶

The insight that reframed robotics for me came from an Owl Posting article on lab automation. The core problem isn't that robots can't do tasks. It's that they sit idle.

The article walks through an example: consider a liquid handler in a biology lab (a robot that pipettes liquids). It costs $40,000-$100,000. It can pipette with superhuman speed and precision. But setting up a new protocol for it takes an automation engineer 40+ hours. If you're running that protocol 10,000 times, the setup cost amortizes to pennies per run. Worth it. But if you're running it 50 times? That's $80 per run in setup costs. You'd be better off doing it by hand.

Most protocols in most labs fall into the second category. Research is exploratory. You run an experiment, look at results, change something, run a different experiment. The automation sits unused.

This is the same problem GPUs had before cloud compute. Expensive hardware. High utilization required to justify cost. Individual users couldn't fill capacity. Cloud providers (AWS, Modal, Lambda Labs) pooled demand and achieved utilization rates no individual could match.

Cloud labs (Emerald Cloud Labs, Transcriptic) are trying to do the same for robotics. Centralized facilities with standardized automation that multiple customers can access. When one customer's experiment finishes, another's begins. The robot never sits idle.

Humanoids Are Impressive. Arms Are the Business.¶

The humanoid demos have the wow factor. But the bottleneck in robotics isn't the form factor. It's:

Scheduling software to batch tasks and maximize utilization
Translation layers that let non-experts program robots without 40 hours of setup
Standardized interfaces so different robots and instruments can work together

A humanoid robot in a lab still faces the same utilization problem. If it takes 40 hours to teach it a new protocol and you only need that protocol 50 times, you're still better off doing it by hand.

What's actually changing:

Foundation models for robotics. Physical Intelligence's π0 is a general-purpose robot foundation model (now open-sourced) that can be fine-tuned for diverse tasks. They also showed that as you scale up robot training data, the ability to transfer from human video emerges automatically (~2x improvement on limited-data tasks).

World models from human video. NVIDIA's DreamDojo pretrained on 44,000 hours of human egocentric video to build a robot world model. The insight: you can learn physics and object interactions from human video, then transfer to robot embodiments. This is the "ImageNet moment" people have been waiting for in robotics.

LLM-powered translation. Briefly Bio is building tools that convert natural-language protocols into robot-executable code. The automation engineer bottleneck starts to dissolve when an LLM can handle the translation.

Modular, task-specific hardware. Instead of one humanoid that does everything, modular arms and "workcells" (multiple instruments connected by robotic arms) that can be reconfigured for different tasks.

The Universal Robots arms feel like the BMW of this space. Sleek, metal, smooth motion. Not humanoid, but undeniably impressive in a different way. When i saw them, i thought: yeah, i want that.

Why Aloha Mini¶

i looked at a few options. The SO-100 and SO-101 have dual arms, which is cool, but no mobility. i felt like i'd quickly outgrow just the arms. Aloha Mini has arms AND motion (it's on a mobile base). That felt like more room to explore.

First task i want to try: scooping cat litter.

my wife and i have 3 kittens. They're going to be very curious about the robot. Especially the exposed wires. This will either go well or become an expensive cat toy. Either way, i'll learn something.

What i Think Will Transfer from ML¶

My background is CNNs, OCR, distributed training, inference at scale. Some things i expect will transfer:

Data quality > data quantity. In ML, garbage in, garbage out. i assume the same is true for imitation learning demonstrations.
Evaluation is harder than training. Getting a model to do something is easier than knowing if it's doing it well. Robot evaluation seems even harder (you can't just run a test set, you have to physically watch it).
Community matters. LeRobot on HuggingFace, robotics Twitter/X, research labs sharing datasets and models. Same pattern as ML. Find the communities, learn what people are actually using.

What i suspect won't transfer: intuition for hardware failure modes. Software fails in predictable ways. Hardware fails in ways that involve physics, wear, and cats chewing on wires.

The Centralization Question¶

There's a tension in all of this that Keoni Gandall captures in Synbio25:

"The vast majority of experiments are done with human hands, manually. It is like if we are mining bitcoin on abacuses."

Centralized automation will be more efficient. That's the economic reality. But centralization can go two ways:

Monopolization: A few large players control the infrastructure, capture the value, and individual researchers lose agency.

Democratization: Centralized infrastructure becomes a platform anyone can build on. AWS is massive, but the software running on it is diverse. Cloud labs could follow the same pattern.

The window matters. The infrastructure is being built now. If you care about this space, now is the time to pay attention (or, in my case, to buy an Aloha Mini and start learning).

What i'm Curious About¶

A few questions i'm holding as i start:

Where does simulation fit? Sim-to-real transfer has been a research topic for years. Is it actually useful in practice, or is real-world data still king?

What breaks first? Software breaks in logs. Hardware breaks in the physical world. i'm curious what the failure modes look like and how you debug them.

Following Along¶

This is the start. i'll write more as i learn (what works, what breaks, what surprises me).

If you're a software engineer curious about robotics, or just someone who vibes with this kind of exploratory writing, you're the audience i'm writing for. We're figuring this out together.

The humanoid demos are cool. But i'm more interested in the boring, task-specific, infrastructure-level work that nobody's making viral videos about.

And scooping cat litter. i'm definitely interested in that.

References:

Today, Like Every Other Day

2026-02-10T06:30:00-06:00

Keep walking, though there's no place to get to.
Don't try to see through the distances.
That's not for human beings.
Move within,
But don't move the way fear makes you move.

Today, like every other day, we wake up empty and frightened.
Don't open the door to the study and begin reading.
Take down a musical instrument.

Let the beauty we love be what we do.
There are hundreds of ways to kneel and kiss the ground.

— Jalal ad-Din Rumi, from "The Essential Rumi", translated by Coleman Barks

Reflections¶

Journey over the destination. Don't try to see through the distances. Move WITHIN.

Fear is inevitable. It's bound to exist. Did you think life would be easy? Did you think if you achieved X thing that suddenly, fear and difficulty would not exist?

Difficulty is a feature of life, not a bug. It's our choice what the difficulty and HARD thing in our lives we choose to adopt is.

Hormozi x Rumi.

Let the beauty we love be what we do. Enjoy every moment. Cherish every aspect of life, despite it potentially being hard.

There are hundreds of ways to kneel and kiss the ground. There are hundreds of ways to be grateful. To be happy and feel fulfilled.

Today, like every other day, I wake up empty and frightened. Despite all the good, my mind focuses on all the bad. And that's natural and okay. But it requires intentional redirection.

Everything that leads to growth will be against what the nafs takes us towards. What feels "natural" as a human being is simply the nafs easing into complacency and short term gratification. Focusing on what could have been or what we wish had been done better is exactly that.

Oh it feels hard to be grateful? What did you expect?

What Three Jobs Taught Me About Agent Orchestration

2026-02-09T06:30:00-06:00

Ethan Mollick's post about applying org theory to agent swarms got me thinking. i enjoyed reading it, and it sparked this writeup. Spans of control, boundary objects, and coupling are the highlights.

i've worked under three very different org structures in the past few years. And watching the multi-agent discourse unfold, i keep seeing patterns i've already lived through. Not necessarily with agents, but with humans.

The Integrated Model¶

At a QSR company, i was the MLOps architect embedded directly in the data science team. No separate infrastructure org. Just me, in the standups, watching the work happen.

This meant i didn't need boundary objects. i didn't need someone else's summary of what was going well or what was broken. i could observe firsthand, form my own conclusions, apply my own judgment and taste.

The upside: direct signal. No telephone game. i could see when a model was struggling before anyone filed a ticket. i could spot infrastructure bottlenecks by watching the team work around them.

The downside: scope creep. i was always available. Always in the room. Always getting pulled into ad hoc issues because "you're right here, can you just look at this?" Boundary setting became crucial but difficult, because helping was technically my job.

i stayed sane by staying in interviewer mode. Instead of injecting my own opinions constantly, i spent time gathering perspectives from the data scientists. Asking questions. Letting them surface their own pain points. It kept me from becoming the bottleneck i was supposed to prevent.

The agent equivalent: an agent with direct data access is powerful. It can form its own conclusions from raw signal. But it needs structured interaction patterns, or it becomes the thing that handles everything and eventually chokes. The agent that sees everything can get pulled into everything.

The Over-Coupled Model¶

At a large financial institution, the structure was the opposite. Red tape everywhere. Multiple layers of approval for everything. Security reviews and risk assessments.

The frustrating part: even when i had context, even when i knew what the right call was, i often couldn't act. Personal intuition had no value. The coupling was so tight that judgment got squeezed out.

This wasn't safety. It was friction. The organization had optimized for control at the expense of velocity and responsiveness. Scrum going well was more important than the actual work.

The agent equivalent: if your critic can veto but can't exercise judgment about when to veto, you've built friction and not reliability. The Team of Rivals paper talks about hierarchical veto authority. But veto authority only works if the critic has enough context and autonomy to make real decisions.

The Balanced Model¶

At my current role, i've found a better mix. There's oversight. There are approval processes. But there's also enough autonomy to scout information myself.

The key difference: a writing culture. Six-pagers, two-pagers, LLDs, HLDs. Documentation isn't busywork here. Writing is thinking. It forces deep thought about deliverables and contact points. The boundary objects exist, but they're backed by business and customer agendas.

This creates boundary objects that actually help. They share context without obscuring it. They provide structure without strangling judgment.

The agent equivalent: pre-declared acceptance criteria (what the Team of Rivals paper calls their key innovation) are basically six-pagers for agents. Structured handoffs that force the agent to think about what success looks like before execution starts. Not vague "make this good" prompts. Explicit conditions defined upfront.

My Approach Now¶

When i'm building agent pipelines, i find myself reaching for a few principles:

Exhaustive logging, true to raw output. i want the actual signal, not someone else's interpretation of the signal. No bias injection through summarization layers unless it's explicitly needed for review or critic models.

Strong prompts for format, not for judgment. Format is mechanical. Judgment should come from the agent reasoning about the actual problem. Mixing them leads to prompts that are trying to do too much.

Shallow agent hierarchies. i don't go super deep in agent levels for specific projects. The Team of Rivals paper describes 50+ agents with hierarchical veto. That works for their use case (arbitrary queries against complex data). For more bounded problems, shallow hierarchies with clear boundaries work better.

Upfront boundary work. Git worktrees and a screen full of Codex terminals is a valid approach. But it only works when you've spent time outlining the boundaries of projects and intra-project features beforehand. The hierarchy can be shallow because the boundaries are tight.

The Tradeoff¶

Mollick's org theory concepts are right. But the implementation depends on a tradeoff most people don't make explicit:

Deep agent hierarchies compensate for loose boundaries. If you don't know exactly what you're asking for, you need more layers of validation, refinement, and critique to catch errors.

Shallow hierarchies require tight boundaries upfront. If you define the scope clearly before execution, you don't need as many layers of oversight.

The Team of Rivals paper builds elaborate internal structure because they're handling arbitrary queries. i can stay shallow because i define the boundaries externally.

Neither approach is wrong. But knowing which model you're building matters.

i think the lesson generalizes: agent orchestration patterns aren't new. We've been figuring out coordination at scale with humans for decades. The constraints are different (agents are cheaper, faster, more parallel) but the organizational dynamics are surprisingly familiar.

The question isn't "what's the best agent architecture." It's "what coordination pattern does your problem actually need."

Latency-Bound vs Throughput-Bound: The Missing Dimension

2026-02-07T19:00:00-06:00

We've all frantically gone through Hello Me system design interviews before that critical technical round. We've been bullied into learning CAP theorem early. You can't have consistency, availability, and partition tolerance simultaneously. Pick two. It's a lens that shapes how you think about distributed systems.

Product people learn the speed/quality/cost triangle. You can optimize for two, but the third suffers.

There's a missing dimension for infrastructure decisions that nobody explicitly names: whether your system is fundamentally latency-bound or throughput-bound.

Most performance advice implicitly assumes throughput optimization. "Use batching!!!" "Maximize GPU utilization!!" "Connection pooling!!!!"

All throughput-oriented. If you're latency-bound, following this advice is actively harmful.

i didn't have this framework whilst building these systems. The aha came in hindsight (even while writing this piece) looking back at different production environments and recognizing why certain optimizations worked in one context and failed in another.

How to Know Which Bound You're Under¶

Latency-bound signals:

Hard deadline per request (1.5 seconds for drive-thru AI, sub-100ms for retrieval)
User waiting synchronously where they feel every millisecond and theres no room for multitasking
p99 matters more than average
Individual request failure = user-visible failure

Throughput-bound signals:

Batch processing, async workflows
Cost per unit matters more than individual latency
GPU utilization is your north star
Can tolerate individual slow requests if aggregate improves

The distinction sounds obvious stated plainly. In practice, it's easy to miss because performance advice rarely specifies which paradigm it assumes.

The Cascade Effect¶

Once you know which bound you're under, it cascades through every decision. This is where the framework becomes useful.

Latency-Bound: Drive-Thru Inference¶

At a QSR chain, we had 1.5 seconds end-to-end for the full voice pipeline:

Customer speaks → ASR → LLM → TTS → Speaker
├── ASR (speech-to-text):     200-500ms
├── LLM inference:            300-800ms  ← our window
├── TTS (text-to-speech):     100-300ms
├── Network + audio I/O:      100-200ms
└── Total budget:             < 1500ms

300-800ms for LLM inference. That's a hard ceiling, not a target to optimize toward. It's a constraint that shapes everything else.

How latency-bound cascaded through decisions:

Batching: Continuous batching is good for throughput, but it can hurt tail latency. We tuned max_num_batched_tokens carefully. Too high and p99 spiked as long prefills blocked other requests. Chunked prefill helped (splitting long prompts into 512-token chunks so decode steps for other customers could interleave).

Engine routing: We ran three inference engines simultaneously (vLLM, TensorRT-LLM, and SGLang). During peak hours when latency was critical, requests routed to TensorRT-LLM (lowest TTFT at ~8ms). When system prompts were shared across sessions, SGLang's RadixAttention cached the prefix and cut first-token latency by 60-80%.

Parallelism: Tensor parallelism within a single node only. NVLink between GPUs on the same node gives ~600 GB/s. PCIe across nodes gives ~32 GB/s. That's a 19x difference. For throughput-bound training, you can tolerate cross-node communication overhead. For latency-bound inference, you can't.

Monitoring: p99 latency alerts, not average throughput. Averages hide the problem completely. We learned this the hard way with memory fragmentation (p99 drifted 5-10% per hour while averages looked fine). By the time we noticed, we were 36 hours into degradation.

Caching: Aggressive prefix caching with SGLang. Every drive-thru conversation starts with the same system prompt (menu, hours, promotions). That's 2,000-4,000 tokens repeated across hundreds of concurrent sessions. Cache it once, prefill only the unique tokens per session.

(Full details in Ray in Production)

Throughput-Bound: Distributed Training¶

Same infrastructure (Ray), completely different optimization strategy.

Training a model on dozens of GPUs is throughput-bound. Individual step latency matters less than aggregate GPU utilization over the training run.

How throughput-bound cascaded through decisions:

Checkpointing: Every 30 minutes instead of every step. Checkpointing has overhead. In latency-bound systems, you checkpoint frequently because losing one request's progress is user-visible. In throughput-bound training, you batch checkpoints to maximize GPU time. We learned this wrong first (lost 18 hours of PPO training when the driver process died between checkpoints).

Spot instances: Worth the interruption risk. Spot instances cost 70% less than on-demand. For latency-bound inference, a spot interruption means dropped customer requests. For throughput-bound training, it means rolling back to the last checkpoint and resuming. Acceptable tradeoff.

Batching: Maximize batch size. Fill every GPU. The goal is cost-per-training-hour, not latency-per-step.

Monitoring: GPU utilization, not latency percentiles. If GPUs are idle, you're wasting money.

Mixed Systems: RAG Infrastructure¶

Here's where it gets tricky. The same codebase can have both bounds.

RAG retrieval at a financial institution: analysts querying for data discovery. The retrieval path was latency-bound (an analyst is waiting, staring at a loading spinner). Sub-100ms for vector retrieval, 10-15 seconds end-to-end for complex agentic queries.

But the embedding pipeline that populated the vector store? Throughput-bound. Batch processing millions of documents overnight. Maximize GPU utilization, minimize cost per document embedded.

Same system, same codebase, different optimization strategies for different paths.

If you apply throughput-oriented batching to the retrieval path, analysts wait longer. If you apply latency-oriented caution to the embedding pipeline, you waste GPU time and money.

(Full details in RAG Infrastructure at Scale)

Why Performance Advice Contradicts Itself¶

"Use batching!" and "Batching hurts latency!" are both correct. They assume different bounds.

Advice	Throughput-Bound	Latency-Bound
"Use batching"	Maximize batch size	Tune carefully, hurts tail latency
"Maximize GPU utilization"	North star metric	Leave headroom for burst
"Connection pooling"	Reduce overhead	Size for p99, not average
"Cache aggressively"	Long TTLs acceptable	Freshness often matters more
"Retry on failure"	Eventually consistent	Timeout budget shrinks with each retry
"Checkpoint frequently"	Overhead adds up	Losing progress is visible

The advice isn't wrong. It's incomplete without specifying which bound it assumes.

The Framework¶

Step 1: Identify your constraint.

Is there a hard latency budget per request? A user waiting synchronously? Then you're latency-bound.

Is aggregate throughput or cost-per-unit your metric? Can you tolerate individual slow requests? Then you're throughput-bound.

Step 2: Let it cascade.

Once you know your bound, every other decision follows. Batching strategy, parallelism approach, caching TTLs, retry policies, monitoring metrics. Don't mix advice from the wrong paradigm.

Step 3: Watch for mixed systems.

Most real systems have both bounds in different paths. Identify which path you're optimizing. The retrieval path and the indexing path have different constraints even if they share infrastructure.

The Meta-Lesson¶

i spent three years building these systems without this framework explicit in my head. The decisions were right (you develop intuition in the trenches) but the framework was implicit.

Making it explicit helps in two ways:

Faster decisions. Instead of rediscovering the tradeoffs each time, you pattern-match to your bound and let the cascade follow.
Better communication. When someone proposes an optimization, you can ask: "Are we latency-bound or throughput-bound here?" It cuts through a lot of debate.

CAP theorem didn't make distributed systems easier. It gave engineers a shared vocabulary for discussing tradeoffs.

This is the same. Name the dimension, and the decisions clarify.

Managing Agents: The First Time It Actually Worked

2026-02-06T15:45:00-06:00

Today was the first day i actually felt like i was managing agents, not just using them.

There's still this friction. It feels wrong to output AI slop. You feel the need to rewrite things, to inject your voice. The structure, the tone, the way you'd phrase it for your generation. The details beyond necessity. Knowing the other person's threshold for information and matching their communication style.

But here's the thing: it enables you to be on top of things better.

The System¶

i started a flow: one directory per EPIC or project. Everything that happens gets dumped into an OpenCode session within that directory. New Slack messages, new emails, all of it. A running record of status. Always current.

i have a locally scoped agents.md with /init-project and /resume-project commands. i can jump into any context instantly. Dump anything work-related. Pick up exactly where i left off.

This has been doing wonders for onboarding to a service at massive internal scale. We're taking it over from the previous team and starting on-call rotations soon. The previous team only had it for a short period before handing it off. Their knowledge transfer wasn't useful because they barely understood it themselves.

Without this system, every day would've been a constant stress point. Tracking down documentation. Scraping Slack and wikis for context. Asking the same question twice. Figuring out the same context twice.

Instead, i can trust the system to hold what i need, where i need it, when i need it.

The Feeling¶

i can mentally separate things better because i can trust the system. When i feel tension in my chest, it's a sign that the system isn't optimized or there's a gap. An open loop that hasn't been closed.

This lets me context-switch without losing the thread. i don't need to keep onboarding and signing off from each task within a project before moving to something else.

Right now i'm managing two things in parallel:

An API migration that has to be done ASAP before a sunset. Hundreds of thousands of users will lose critical info if we miss the deadline.
Creating an extensive wiki for a service i started onboarding to recently.

Without this system? i'd be in at least low-grade panic about both.

With it? i feel like i'm just managing a project. Checking in with colleagues. Delegating tasks. Retrieving updates. Managing multiple tasks. Making progress effectively. Compartmentalizing effectively.

i feel capable.

L5 → L6¶

i feel like i'm operating at L6 scope right now. Three weeks in.

This system is a cheat code that lets me punch above my weight. It makes any L5 engineer capable of transitioning to L6 and handling projects. Not by appearing more capable, but by being more capable. You can hold more complexity in view at once.

It also helps for performance reviews. Making a good impression on your manager.

But call it what it is: i just started recently. The gap between L5 and L6 for me is time and promotion cycles, not capability.

Reps and Discernment¶

Volume negates luck. Enough reps make anyone good enough.

The question is: do people care to put in the reps? And do they have the discernment to know what is worth putting reps in for?

For me, the discernment comes from long-term goals. i know the tools that will get me to understanding quicker. i'm willing to abandon old ways of doing things for new approaches.

My long-term goal is still the same: frontier AI/ML lab. This is a stepping stone for that shift.

What This Role Is Teaching Me¶

This role isn't the destination. It's the dojo.

What does a frontier lab care about that this system is teaching me?

Agency and self-awareness
Willingness to adapt
Speed and pace in a high-level, high-expectation environment

i want to be in a place where my work pace is normal, not exceptional. Where everyone else is also running systems like this, or their equivalent.

i want to be in a place where i learn at work. Where everyone is trying new things. Where the job itself is at the forefront of what's going on. Where i can learn practically at work rather than just using what i learn outside of work at work.

Right now i'm in import mode. Learning outside, applying inside. My X/Twitter feed and convos with friends like Yasin about how they're implementing new tech in their actual money-making workflows feel like the edge i can only attain outside of work.

i want to be in a place where the work is the frontier, not a place i bring frontier techniques to.

Proof of Work¶

That's why i'm focusing on publishing content about my thinking processes and what i'm working on. Proof of work, publicly, with my face so it brings authenticity with it.

Frontier labs hire based on proven work and research, not just interview performance. Three months of "here's how i shipped X with agent orchestration at scale" is a much stronger signal than "i think i could do this."

This role is the training ground. The public content is the resume.

So that's where i am. Early days. Managing agents. Feeling capable. Building the proof.

The system works. Now i just need to show everyone else how.

Ray in Production: What Dozens of GPUs and a Lot of 3am Pages Taught Me

2026-02-05T10:57:00-06:00

Disclaimer: This post reflects patterns and lessons learned from building distributed ML systems at production scale across multiple environments. Technical details have been generalized, and no proprietary information from any specific organization is disclosed.

I've run Ray in production at three companies over the past three years:

A fintech platform: Built an ML platform on AWS serving community financial institutions. Dozens of GPUs running RLHF with PPO. Distributed training, hyperparameter optimization, all of the fun stuff
A large financial institution: Enterprise agent platform serving thousands of analysts. Worked with vLLM on Ray Serve, distributed fine-tuning, vector search across massive document collections.
A QSR chain: Conversational AI for drive-thru ordering. Hundreds of thousands of LLM calls daily, sub-1.5-second end-to-end latency, three simultaneous inference engines orchestrated through Ray.

Each environment had completely different constraints — financial compliance vs. real-time voice latency vs. enterprise scale — but the patterns that emerged were surprisingly consistent.

And unfortunately so were the ways Ray broke.

Most Ray content falls into two buckets:

"here's how to use ray.remote"
Anyscale's marketing materials lol

There's almost nothing in between for the engineer who's actually trying to ship a distributed ML system and hitting walls that the documentation doesn't warn you about and reliance on LLM's for debugging goes through the roof.

Instead of organizing by Ray component (Train, Serve, Data), I wanted to walk through the real problems I faced and how Ray fit (or didn't fit) into solving them. The ugly parts are included because they're the parts that actually matter. Nobody's perfect

↑ Back to Contents

Table of Contents:

The Mental Model
The Execution Model
Fintech RLHF Cluster
Enterprise RAG
Real-Time Inference
Streaming & Ray Data
The Evolution
Serialization
Resource Management
Debugging in Production
Antipattern Hall of Fame
Performance Checklist
What I'd Tell Myself

Part 1: The Mental Model¶

Before anything else, the single most important insight about Ray:

Ray is a distributed operating system for Python processes.

Once this clicks, everything else will follow:

Tasks are processes
Actors are long-lived processes with state
The object store is shared memory across machines
The GCS (Global Control Store) is the kernel's process table
The Raylet is the per-node scheduler
Ray Train orchestrates PyTorch (i see people mixing this up often, it doesn't replace it)
Ray Serve is a deployment framework

The engineers I've seen struggle with Ray try to use it like a library. Call a function, get a result, move on.

The engineers who succeed treat it like infrastructure. You don't "use" an operating system the way you use a library; you build on top of it and respect its constraints.

Part 2: The Execution Model (expect to find buried bodies here)¶

GCS: The Actual Single Point of Failure¶

Ray markets itself as decentralized.

In practice, the Global Control Store is the brain.

Pre-Ray 2.0, it was a Redis instance on the head node. Post-2.0, it's an embedded gRPC service with optional Redis for HA.

The operational reality hasn't changed as much as you'd hope.

GCS tracks everything: actor registry, placement groups, resource availability, job metadata. If GCS goes down, your cluster enters a degraded state. Existing running tasks continue, but no new scheduling decisions can be made.

At one financial institution, we were running Ray 2.7 when a head node had a memory spike from an unrelated monitoring process.

GCS went unresponsive for ~45 seconds (treat every estimate with an assumed -ish at the end to account for my ellis grey level memory). Every in-flight actor creation call failed with gRPC UNAVAILABLE errors that, at the application layer, looked nothing like a GCS issue. The error message pointed at the downstream service, not the scheduler.

We spent two hours chasing a phantom bug in our vLLM deployment before someone checked the head node metrics.

The fix ended up being simple: retry logic that understands gRPC status codes and doesn't immediately assume the called service is broken.

Another disclaimer. I had Claude help me with generating the code snippets in this article since I can't share proprietary code. But it HAS been checked, so rest assured.

# Production-ready actor creation with retry
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=2, max=30))
def create_actor_with_retry():
    try:
        return MyActor.remote()
    except ray.exceptions.RayActorError as e:
        if "UNAVAILABLE" in str(e):
            raise  # Retry
        raise  # Don't retry other errors

The lesson: GCS fault tolerance was experimental before Ray 2.9. Even with GCS FT enabled in newer versions, that 10-30 second failover window will generate errors that look nothing like what they actually are.

This also brings about a richer lesson about the age we're moving into. Despite LLM's helping track root causes and everyone preaching "Logs are all you need", you will NOT thrive without good system design intuition. You NEED to understand what each service is responsible for and what is a Non-Goal of it.

The Ownership Model (Most Expensive Lesson)¶

Ray uses ownership-based distributed reference counting. THIS is what that means:

When you call ray.put(x), the calling process becomes the owner of that ObjectRef. If the owner process dies, the object becomes unreachable even if the data is still physically present in the object store on another node. OOF what a caveat.

At the fintech platform, during our early RLHF experiments, we had a driver script that orchestrated all four models (policy, reference, reward, value) in the PPO loop.

The driver launched thousands of tasks across the training run. Each return ObjectRef was owned by the driver. When the driver process OOM'd — which happened because it was holding references to intermediate results from thousands of generation steps — every single ObjectRef it owned became garbage-collectible.

Downstream tasks that depended on them started throwing ObjectLostError, which cascaded through the entire training pipeline.

We lost 18 hours of PPO training. At five figures per month in spot instance costs, that loss HURT like hell. lmao imagine explaining this to your boss.

And so, naturally (as well as structurally from the inherent pressure of losing my job), we moved to a tree reduction pattern.

So instead of the driver owning everything:

# DANGEROUS — driver owns all 100K ObjectRefs
refs = [process_batch.remote(i) for i in range(100_000)]
results = ray.get(refs)  # Driver must stay alive for ALL of these

# SAFER — delegate ownership via intermediate actors
@ray.remote
def process_and_reduce(batch_ids):
    refs = [process_single.remote(i) for i in batch_ids]
    return aggregate.remote(refs)  # Ownership stays in this actor

chunks = [list(range(i, i+1000)) for i in range(0, 100_000, 1000)]
meta_refs = [process_and_reduce.remote(chunk) for chunk in chunks]

The Ray docs mention ownership, but they don't convey the gravity of what happens in a long-running training job when the owner process dies. Reading the docs gave me a 2/10 understanding. Losing compute gave me a 10/10 understanding :))))))

Object Store: Shared Memory With Sharp Edges¶

Ray's object store uses shared memory via /dev/shm. Objects are stored as memory-mapped files. This is actually really elegant — to non-nerds who actually have lives, this means zero-copy reads on the same node and Arrow IPC across nodes. Fun fact, i learned about Arrow at this point in my career, how embarrassing (despite using Parquet and Apache based frameworks for forever-oclock).

The sharp edge i'm talking about here is this: on Linux, /dev/shm defaults to 50% of RAM.

In Docker without --shm-size, you get 64MB.

Your Ray workers will crash with SIGBUS errors when the object store fills up, and the error message will not tell you it's a shared memory issue.

I hit this exact failure more than three mutherfreaking times — basically once at each company.

At the QSR chain, it happened during our initial GKE deployment because the Kubernetes pod spec didn't include the emptyDir volume mount for /dev/shm. The error looked like a segfault in the inference engine. We spent a full afternoon on it WOOP WOOP LOVE THE JOB BABYYYYY

# In Kubernetes: ALWAYS DO THIS
volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: 16Gi

Configure object spilling to fast storage pls, not the default /tmp:

ray.init(
    _system_config={
        "object_spilling_config": json.dumps({
            "type": "filesystem",
            "params": {
                "directory_path": ["/mnt/nvme0/ray_spill"]
            }
        }),
        "min_spilling_size": 100 * 1024 * 1024,  # Don't spill objects < 100MB
    }
)

On cloud instances with NVMe drives, spilling is pretty fast.

On EBS or network storage, it's catastrophic for latency.

At one enterprise environment we were on EBS volumes initially and spilling added 200-500ms per object restoration.

Switching to local NVMe brought that down to 5-10ms.

Sometimes i feel like the huberman of optimization, we love that.

Part 3: Fintech RLHF Cluster (Dozens of GPUs)¶

The Problem¶

The fintech platform served community financial institutions. These banks were losing customers to neobanks that offered "smart" features.

We needed to give them ML-powered capabilities like fraud detection, credit risk, transaction categorization, all without each bank building its own ML team.

The mandate expanded from traditional ML to LLMs and post-training. We were building an RLHF pipeline with PPO before most people had even heard of RLHF.

The target: train domain-specific models (3B-7B parameters) that understood financial services.

The Architecture¶

RLHF with PPO requires four models running simultaneously:

Stage 1: Policy Model (multiple GPUs) — generates responses Stage 2: Reference Model (frozen) — computes KL divergence Stage 3: Reward Model — scores generated responses Stage 4: Value Model — estimates advantage function

Each stage has completely different compute characteristics: - The policy model needs the most GPUs because it's doing both forward and backward passes - The reference model is frozen (inference only), so it needs less memory - The reward model is smaller - The value model trains alongside the policy

Total: dozens of V100 GPUs, orchestrated through Ray

Napkin Math¶

Here's the intuitive math that shaped every decision:

Used Claude to source the cost as of February 2026

Dozens of V100 GPUs (im basing this off of p3.2xlarge instances on AWS):
├── On-demand: ~$3/hr per GPU
├── Spot: ~$0.90/hr per GPU
├── Reserved (1yr): ~$2/hr per GPU

One full RLHF run (SFT + RM + PPO + Eval):
├── Thousands of GPU-hours
├── At reserved pricing: five figures per run
└── 3-5 runs per experiment: mid-five figures per experiment cycle

HPO with Optuna/Ray Tune:
├── 20 trials × partial PPO runs
├── Tens of thousands of GPU-hours per sweep
└── Monthly total: five figures in GPU spend

At five figures monthly, every architectural decision was a cost decision.

Placement Group Challenges¶

The policy and value models needed to communicate frequently (advantage estimation requires coordinating between them every training step). You want them close together — ideally same node — for fast inter-process communication.

But V100s only have 16GB of HBM2. You can't fit both models on the same node with enough headroom.

My placement group journey:

# Attempt 1: STRICT_PACK — fails, not enough memory per node
pg = placement_group([{"GPU": 4}], strategy="STRICT_PACK")  # OOM

# Attempt 2: STRICT_SPREAD — works but allreduce is 10x slower cross-node
pg = placement_group([{"GPU": 1}] * 4, strategy="STRICT_SPREAD")

# Attempt 3 (after weeks of profiling): PACK with 2 GPUs per bundle
pg = placement_group(
    [{"GPU": 2}, {"GPU": 2}],
    strategy="PACK",
)
# Policy+Value on 2-GPU node, Reward+Reference on another

Getting to Attempt 3 took weeks. The frustration wasn't just the Ray API — it's that placement groups interact badly with autoscaling. When you request a STRICT_PACK group that requires 4 GPUs but no single node has 4 free, the group enters PENDING. The autoscaler sees pending placement groups and tries to add nodes — but it doesn't know you need a single node with 4 GPUs. It might add 4 single-GPU nodes, which doesn't help.

We eventually landed on requesting placement groups with a timeout and always having a fallback:

pg = placement_group(
    [{"GPU": 2, "CPU": 8}],
    strategy="STRICT_PACK",
    lifetime="detached",
)
try:
    ray.get(pg.ready(), timeout=300)  # 5 min max
except ray.exceptions.GetTimeoutError:
    remove_placement_group(pg)
    raise RuntimeError("Could not place model — check GPU availability")

The Spot Instance Problem¶

To bring the GPU bill down, we configured the autoscaler for spot instances. This worked beautifully for our CPU workers (data preprocessing, evaluation).

GPU training was a bit different though.

When a spot instance gets reclaimed mid-PPO-training:

The Ray worker on that node dies
NCCL communicators break (NCCL comms are not resizable after creation)
ALL workers must restart the NCCL group
Everyone loads from the last checkpoint
Training resumes

The problem is that we were checkpointing every 30 minutes to save time. A spot interruption meant losing up to 30 minutes of PPO training across the entire cluster.

The fix took ~three months of iteration:

Aggressive checkpointing: every 100 PPO steps instead of every epoch
Custom Ray Train failure handler that detected AWS's 2-minute spot interruption warning via instance metadata
"Warm spare" pattern: idle GPU workers pre-loaded the model so failover was faster
Critically: checkpoints to S3, never local disk. When a node dies, local checkpoints die with it.

I can't emphasize the checkpoint-to-shared-storage point enough. This is the #1 Ray Train mistake I've seen in every environment. Save to S3 or NFS. Never local disk. The documentation mentions this but doesn't scream it loudly enough.

The Optuna Integration That Haunted Us¶

Ray Tune's Optuna integration (OptunaSearch) had subtle bugs around trial pruning. When Optuna's MedianPruner decided to prune a trial, the corresponding Ray Tune trial didn't clean up its GPU resources immediately. The actor would linger for 30-60 seconds before garbage collection caught it.

During HPO sweeps with 20 concurrent trials, this meant 2-3 GPUs were perpetually "leaked" to zombie trials. On a cluster costing five figures monthly, that was thousands of dollars per month in wasted compute.

# The workaround: explicit cleanup in every trial
def train_fn(config):
    try:
        for step in range(1000):
            loss = train_step(config)
            tune.report(loss=loss, step=step)
    finally:
        # Force GPU memory release even on pruning
        torch.cuda.empty_cache()
        gc.collect()
        if dist.is_initialized():
            dist.destroy_process_group()

The DPO Pivot¶

DPO (Direct Preference Optimization) emerged as a simpler alternative to PPO.

The team debated switching for months.

DPO doesn't need a separate reward model or value model. It directly optimizes the policy using preference data. That would've reduced our GPU requirement from dozens to around a dozen.

The monthly bill would drop significantly.

But DPO has a fundamental limitation: it's an offline algorithm. It learns from static preference data.

PPO is online — it generates new data during training.

For financial models where the distribution shifts constantly (new fraud patterns, new transaction types), PPO's online nature ended up being more robust.

The compromise we landed on: DPO for initial alignment (cheaper, faster iteration), PPO for final production training (better out-of-distribution robustness).

This hybrid approach cut costs by roughly 40% while maintaining quality.

This ended up being probably the highest-leverage decision we made on the entire platform. Not a Ray decision per se, but Ray's flexibility to support both paradigms (DPO via standard TorchTrainer, PPO via the full 4-model actor setup) made the hybrid approach feasible without rebuilding the infrastructure.

Part 4: Enterprise RAG and Network Challenges¶

The Problem¶

At a large enterprise, thousands of analysts were spending hours on repetitive research: finding documents, summarizing reports, pulling data from internal systems.

The agent platform needed sub-100ms retrieval across massive document collections and an agent layer that could orchestrate multi-step workflows.

Where Ray Fit¶

Ray wore two hats:

Ray Train for distributed fine-tuning of our internal models
Ray Serve wrapping vLLM for multi-model inference

The serving architecture:

vLLM on Ray Serve:
├── Head node: m5.4xlarge (scheduler, dashboard, metrics)
├── GPU workers: multiple nodes with A10G GPUs
├── Primary: 70B model (4-bit quantization, tensor parallel across 4 GPUs)
│   └── 2 replicas for HA
├── Secondary: 13B model (fp16, single GPU, 4 replicas)
└── Routing: lightweight classifier sends simple queries to 13B

The autoscaling config that worked:

@serve.deployment(
    max_ongoing_requests=256,
    autoscaling_config={
        "min_replicas": 2,
        "max_replicas": 8,
        "target_ongoing_requests": 128,  # 50% of max — this is key
        "downscale_delay_s": 600,        # 10 min — GPUs are expensive to restart
    },
)

The mistake i made initially: setting target_ongoing_requests equal to max_ongoing_requests.

If your scaling target equals your hard cap, then by the time autoscaling triggers, your replicas are already at capacity and requests are queuing.

Set the target to 50-70% of max. This gives the autoscaler headroom to react before users feel the pain.

The Network That Ate Our Latency¶

Enterprise network security policies required all traffic between subnets to go through network firewalls with deep packet inspection.

Ray's inter-node communication (gRPC for control plane, custom protocol for object transfers) was being inspected by DPI appliances.

The result: 5-15ms added latency per RPC.

For basic task scheduling, you barely notice.

For vLLM tensor parallelism across nodes — where allreduce operations happen thousands of times per second — this ended up being catastrophic. Operations that should take 2ms were taking 20ms.

Our 70B model was running 10x slower than benchmarks predicted. OOF.

We worked with the network team to create a dedicated "ML subnet" with relaxed DPI rules.

This sentence describes 6 weeks of security reviews, architecture board approvals, risk assessments, and explaining what NCCL is to people who very reasonably didn't know what NCCL is.

A job description of enterprise REALLY should be ability to sit patiently through red tape.

In the meantime, we limited tensor parallelism to within a single node (4 GPUs max). This constrained our maximum model size but kept latency acceptable. The practical impact: our 70B model had to use 4-bit quantization to fit in 4×24GB A10G GPUs.

Not ideal, but the quantized model at fast latency beat the full-precision model at 10x the latency.

If you're building Ray clusters in a large enterprise: Budget 6-8 weeks for network configuration alone. Your network team isn't trying to make your life difficult — they're enforcing policies that exist for good reasons. Come to the table with a clear diagram of what ports Ray needs, what protocols it uses, and what the traffic patterns look like.

The LangChain Tax¶

We initially built the agent layer with LangChain's memory systems.

Profiling revealed that LangChain's abstractions were responsible for 60% of the latency in agent responses.

The default memory implementations use generic ORM-layer abstractions that add unnecessary round trips and overhead.

# After optimization: <1ms per retrieval
# Direct SQLAlchemy Core with prepared statements
async def get_memory(session: AsyncSession, session_id: str):
    result = await session.execute(
        text("""
            SELECT role, content, created_at
            FROM conversation_memory
            WHERE session_id = :sid
            ORDER BY created_at DESC
            LIMIT 10
        """),
        {"sid": session_id}
    )
    return result.fetchall()

Stripping LangChain's memory abstractions and going direct-to-database cut agent response time from ~8s to ~3s.

The lesson: when your inference chain is Ray Serve → vLLM → LangChain → database → response, the bottleneck is often NOT where you expect.

Profile everything. Trust nothing.

The Evaluation Speedup That Actually Mattered¶

One Ray pattern that paid for itself immediately: parallelizing the evaluation harness.

Agent evaluation is fundamentally harder than traditional ML eval.

A fraud model has precision/recall/F1.

An agent that researches companies and writes reports has... "quality"?

We settled on five dimensions: factual accuracy, source attribution, reasoning validity, safety compliance, and completeness. Each evaluation required an LLM-as-judge call.

1000 test cases × 5 dimensions = 5000 evaluation calls
Each LLM-as-judge call: ~2s
Sequential: 5000 × 2s = 10,000s = 2.8 hours
Parallel with Ray (50 concurrent tasks): ~3.3 minutes

That 50x speedup was the difference between "eval after every code change" and "eval once a week and make dua."

This is one of those cases where Ray's task model — just @ray.remote on the eval function, fan out, collect results — is exactly what you want. Simple, effective, no orchestration overhead.

Part 5: Real-Time Inference at the Edge of Chaos¶

The Constraint: Human Conversation Dynamics¶

Drive-thru AI has a hard latency budget dictated by human perception.

More than ~2 seconds of dead air and the customer thinks the system is broken:

Customer speaks → ASR → LLM → TTS → Speaker
├── ASR (speech-to-text):     200-500ms
├── LLM inference:            300-800ms  ← what we're optimizing
├── TTS (text-to-speech):     100-300ms
├── Network + audio I/O:      100-200ms
├── Total budget:             < 1500ms end-to-end

300-800ms for LLM inference.

That's the window.

And we had hundreds of thousands of calls daily across hundreds of restaurants.

Three Engines, One Router¶

This is the architectural decision i'm most proud of, and also the one that caused the most operational headaches.

Most companies pick one inference engine.

We ran vLLM, TensorRT-LLM, AND SGLang simultaneously behind a Ray Serve router.

(yes this was probably insane in retrospect but it WORKED lol)

Dimension	vLLM	TensorRT-LLM	SGLang
Latency (TTFT)	Medium (~15ms)	Best (~8ms)	Good (~12ms)
Throughput	Good	Best (CUDA graphs)	Good
Flexibility	Best (any model)	Poor (requires compilation)	Good
Prefix caching	Basic	No	Best (RadixAttention)
Model swap time	Seconds	Minutes	Seconds

The routing logic:

IF peak_hour AND latency_critical:
    → TensorRT-LLM (lowest latency, pre-compiled)
ELIF shared_system_prompt:
    → SGLang (RadixAttention caches the shared prefix)
ELIF new_model_deployed:
    → vLLM (most flexible, no compilation step)
ELSE:
    → cheapest_available by current GPU utilization

The SGLang Insight¶

Every drive-thru conversation starts with the same system prompt — the restaurant's menu, hours, promotions, instructions. That's 2,000-4,000 tokens. With hundreds of restaurants each having ~20 concurrent sessions at peak:

Traditional inference:
Thousands of sessions × 3,000 token prefix = millions of tokens of redundant prefill

SGLang with RadixAttention:
Prefix computed ONCE, cached in radix tree.
Each new session prefills only their unique tokens (~50-100).
Result: ~60-80% reduction in first-token latency for ongoing conversations.

This was the difference between hitting our 800ms LLM latency target and not.

Without prefix caching, we needed roughly 3x the GPUs to meet the same latency SLA.

The Router's Shadow Queue Problem¶

Running three engines means three separate batching systems with different optimal batch sizes.

The model router had to understand each engine's internal queue state.

But the engines expose different metrics:

vLLM: num_running_requests and num_waiting_requests (useful)
TensorRT-LLM: throughput metrics but not queue depth (less useful)
SGLang: cache hit rate (very useful) but different queue semantics

We ended up maintaining our own shadow queue model in the router, which tracked estimated queue depth based on observed latencies and request rates.

This worked 95% of the time.

The other 5%, when our model diverged from reality — usually during traffic spikes — we'd get brief latency spikes as the router sent requests to an engine that was already saturated.

In practice, a simple exponential moving average of response latency per engine, updated every second, ended up being good enough.

Napkin Math: Why It Worked¶

Hundreds of thousands of LLM calls/day ÷ hundreds of restaurants
Peak hours (lunch + dinner, 4 hrs): hundreds of calls/hr/restaurant
Peak concurrent: tens of calls/second

A 13B model on a single A100: ~25K output tokens/sec
Need: 100K tokens/sec / 25K = 4 GPUs minimum
Reality: 8-16 A100s with burst headroom

Monthly GPU cost: five figures
Per restaurant: hundreds monthly
One part-time drive-thru employee: ~$1,500/month
Even at 50% automation: ROI is massive

Tensor/Pipeline Parallelism Boundaries¶

For a 70B+ model across multiple GPUs, you have two parallelism options:

Tensor parallelism: Split individual layers across GPUs. Requires NVLink/NVSwitch for fast communication. Works beautifully within a single node (4-8 GPUs on NVLink, microsecond latency between them).
Pipeline parallelism: Split sequential layers across nodes. Tolerates higher latency but has "bubble" inefficiency (GPUs idle while waiting for earlier stages to complete).

The right answer ended up being obvious: tensor parallelism within a node, pipeline parallelism across nodes.

But Ray's placement groups don't natively understand GPU topology. They see all GPUs as equivalent.

A placement group with {"GPU": 4} might give you 4 GPUs on the same node, but it doesn't guarantee they're NVLink-connected vs. PCIe.

On GKE, that distinction matters — NVLink gives you ~600 GB/s, PCIe gives you ~32 GB/s.

That's a 19x difference. what the helly.

We used custom resources marking NVLink topology:

# Node 1 (4x A100, NVLink-connected):
ray start --resources='{"GPU": 4, "nvlink_group_0": 4}'

# Node 2 (4x A100, NVLink-connected):
ray start --resources='{"GPU": 4, "nvlink_group_1": 4}'

# Tensor parallel within NVLink group
tp_group_0 = placement_group(
    [{"nvlink_group_0": 4}], strategy="STRICT_PACK"
)

We noticed that our tensor-parallel 70B model was running 5x slower than expected on a "4-GPU node."

Turns out the GPUs weren't NVLink-connected — the GKE node pool had mixed topology.

Custom resources solved it, but debugging that was a fun afternoon.

Async Actors and Health Check Cascades¶

One pattern that i haven't seen written up anywhere: concurrency groups for health checks.

When you deploy an inference engine behind Ray Serve, your load balancer needs health checks. By default, all method calls share the same concurrency pool.

Health checks compete with inference for concurrency slots.

During peak traffic, all slots were occupied by inference requests.

Health checks couldn't get through.

The load balancer saw the replica as "unhealthy" and pulled it from rotation.

Remaining replicas got even more traffic, their health checks also failed, and we cascaded into a full outage.

All because the health check couldn't get a slot. lmao what a way to discover THAT.

Concurrency groups fix this:

@ray.remote
class InferenceServer:
    def __init__(self):
        self.model = load_model()

    @ray.method(concurrency_group="health")
    async def health_check(self):
        return {"status": "ok", "gpu_memory": get_gpu_memory()}

    @ray.method(concurrency_group="inference")
    async def predict(self, input_data):
        return self.model(input_data)

server = InferenceServer.options(
    concurrency_groups={"health": 10, "inference": 8},
).remote()

Health checks get their own pool, completely independent of inference. They always respond, even under saturation. This pattern should be in every Ray Serve deployment.

KubeRay Operational Reality¶

At multiple environments, we ran Ray on Kubernetes via KubeRay. Hard-won operational learnings:

The autoscaler conflict:
KubeRay's operator and Ray's built-in autoscaler are separate systems that can fight each other. KubeRay watches Kubernetes resource requests; Ray's autoscaler watches Ray's internal demand signal.

If both are active, they scale independently and create oscillation. Pick one.

At one company, we used KubeRay for node lifecycle and disabled Ray's autoscaler. At another, the opposite — Ray's autoscaler with KubeRay in a minimal "cluster provisioner" role.

Ghost nodes:
When Kubernetes restarts a pod (OOM kill, health check failure, node drain), the pod gets a new IP. Ray doesn't automatically deregister the old node.

You get "ghost nodes" — entries in the GCS that point to dead processes, consuming logical resources in the scheduler. We wrote a sidecar container that periodically reconciled Ray's node list with Kubernetes' pod list.

Newer KubeRay versions handle this better, but it was a real problem.

Head node CPU = 0: Always set num-cpus: '0' on the head node. The head node should NEVER run user tasks — its job is GCS, dashboard, autoscaler. If tasks land there and cause load, your entire cluster's scheduling degrades.

GCP Pain Points¶

Ray on GCP (GKE) has noticeably worse ergonomics than on AWS (EKS).

A few specific issues we hit:

GKE node pools with GPU limits are less flexible than EKS managed node groups
Vertex AI has its own serving infrastructure that competes with Ray Serve — there were political conversations about "which serving layer" we should standardize on (enterprise fun times)
GCS (Google Cloud Storage) has different retry and consistency semantics than S3. Our checkpoint code, written and tested against S3, had subtle bugs on GCS that only manifested under high write concurrency

Part 6: Streaming Responses and Ray Data¶

Streaming in Enterprise¶

Token-by-token streaming is essential for LLM serving.

No user wants to wait 3 seconds for a complete response when they could see tokens appearing after 200ms.

At the enterprise environment, we implemented this through Ray Serve with SSE (Server-Sent Events):

@serve.deployment(max_ongoing_requests=50)
class StreamingLLM:
    async def __call__(self, request):
        prompt = (await request.json())["prompt"]
        async for token in self.model.generate_stream(prompt):
            yield token

Simple enough.

Except there's a subtle back-pressure problem: if the client is slow to consume tokens, the generator accumulates them in memory.

With 50 concurrent streaming requests, each buffering tokens because the client's network is flaky, you can OOM the actor.

We learned this during a load test when a batch of automated API consumers were making requests but reading responses slowly (they were doing processing between token reads).

Memory grew linearly until the actor died. OOF.

Bounded buffers fix this:

@serve.deployment(max_ongoing_requests=50)
class BoundedStreamingLLM:
    async def __call__(self, request):
        prompt = (await request.json())["prompt"]
        queue = asyncio.Queue(maxsize=100)  # Bounded

        async def produce():
            async for token in self.model.generate_stream(prompt):
                await queue.put(token)  # Blocks if queue full
            await queue.put(None)  # Sentinel

        asyncio.create_task(produce())

        async def consume():
            while True:
                token = await queue.get()
                if token is None:
                    break
                yield token

        return consume()

This caps memory usage per request regardless of client behavior.

Ray Data: Training Pipeline¶

At the fintech platform, we used Ray Data for feeding data into our distributed training jobs.

The key insight: map_batches is NOT map applied to batches. It has fundamentally different semantics.

# map: one row at a time (fine for simple transforms)
ds.map(lambda row: {"text": row["text"].lower()})

# map_batches: processes a numpy/pandas batch (use for GPU work)
def inference_batch(batch: dict[str, np.ndarray]):
    inputs = tokenizer(batch["text"].tolist(), return_tensors="pt", padding=True)
    with torch.no_grad():
        outputs = model(**inputs.to("cuda"))
    batch["embeddings"] = outputs.last_hidden_state.cpu().numpy()
    return batch

ds.map_batches(
    inference_batch,
    num_gpus=1,
    batch_size=32,
    concurrency=4,  # Parallel workers
)

The other gotcha: backpressure. If your pipeline has a fast stage (reading from S3) and a slow stage (GPU inference), the fast stage fills memory. You need explicit resource limits:

ctx = ray.data.DataContext.get_current()
ctx.execution_options.resource_limits = ray.data.ExecutionResources(
    cpu=16,
    gpu=4,
    object_store_memory=4 * 1024 * 1024 * 1024,  # 4GB cap
)

Without this, we had multiple OOM incidents during our RLHF data preprocessing.

The data loading actor would happily read 50GB of preference data into the object store while the GPU workers were still processing the first 2GB.

(this is a very productive thing to do at 11pm on a friday, highly recommend)

Part 7: The Evolution¶

Looking across all three environments, there's a clear evolution:

Early days (fintech platform):
├── Ray for distributed training (the hard problem)
├── Everything custom, tight coupling between Ray and compute
├── Small team (~5 engineers), everything is hand-rolled
└── Pain: V100 memory, spot instances, Optuna bugs

Mid-period (enterprise + QSR, simultaneously):
├── Ray for both training AND serving
├── More mature ecosystem (KubeRay, Ray Serve autoscaling)
├── Larger teams, more specialization
├── Pain: enterprise networking, LangChain abstractions, multi-engine routing

Current role:
├── Ray abstracted away by managed services
├── Complexity shifts to application layer (agent orchestration)
├── No GPUs to manage — it's someone else's problem now
└── Pain: different kind — workflow state, tool integration, not GPU orchestration

My deep Ray knowledge is more valuable now than it was at the companies where i was actually using Ray.

Because i understand what managed services abstract away, i can make better decisions about when to use them, when to push back on their limitations, and when a problem actually needs custom infrastructure vs. a managed service.

Understanding Ray deeply means understanding three domains: ML theory, systems engineering, and product requirements.

It sits at their intersection.

The specific APIs matter less over time. The systems thinking endures.

Part 8: Serialization (The Silent Performance Killer)¶

Serialization overhead is the single most common performance issue i've seen across all three environments, and it's the one the documentation is least helpful about.

Ray uses a dual-path serializer: Arrow for numpy arrays, primitive types, and Arrow-native objects (zero-copy, fast), and Cloudpickle for everything else (slow, memory-hungry).

The performance cliff between these two paths is enormous:

# PATH 1: Arrow-native — near-instant for large arrays
arr = np.random.randn(10_000, 10_000)  # ~800MB
ref = ray.put(arr)  # Zero-copy into shared memory

# PATH 2: Cloudpickle — orders of magnitude slower
class CustomModel:
    def __init__(self):
        self.weights = np.random.randn(10_000, 10_000)
        self.config = {"lr": 0.001}

model = CustomModel()
ref = ray.put(model)
# Cloudpickle serializes the ENTIRE object graph
# The numpy array gets copied into the pickle stream, THEN into Plasma
# You've now used 3x the memory of the array

At the fintech platform, we had custom model wrappers that looked harmless — small Python classes holding configuration alongside model weights.

Putting them through ray.put() meant the weights took the Cloudpickle path instead of the Arrow path.

Serialization time went from microseconds to seconds. Memory usage tripled. YA ALLAH WHAT IS HAPPENING.

(this one hurt particularly because it looked so innocent in the code)

Store large arrays separately via ray.put() (Arrow path), and only serialize the lightweight metadata:

class LargeModel:
    def __init__(self, weights_ref, config):
        self.weights_ref = weights_ref  # Store the ObjectRef, not the data
        self.config = config

    def __reduce__(self):
        return (LargeModel, (self.weights_ref, self.config))

# Usage
weights_ref = ray.put(weights)       # Arrow zero-copy for the big array
model = LargeModel(weights_ref, config)
model_ref = ray.put(model)           # Only serializes ref + config (tiny)

This pattern — separate the large data from the metadata, put them through different serialization paths — came up in every environment. At the QSR chain, it was model weights for the inference engines. At the enterprise, it was large embedding batches flowing through the processing pipeline. The same pattern, the same fix, every time.

Part 9: Resource Management Horrors¶

Logical vs. Physical Resources¶

Ray resources are logical — they don't map to physical resources unless you make them. num_cpus=1 doesn't pin a process to a core; it decrements a counter. num_gpus=1 sets CUDA_VISIBLE_DEVICES but doesn't enforce memory isolation.

At the QSR chain, we had two actors on the same node, each declared with num_gpus=1. CUDA_VISIBLE_DEVICES was set correctly — each saw one GPU. But there's no memory isolation. When one actor's model had a memory-hungry batch, it OOM'd the other actor's GPU. The error appeared in the wrong process's logs.

The num_cpus=0 Disaster¶

At the enterprise environment, we had coordinator actors — lightweight processes that didn't do compute, just tracked state and dispatched work.

We set num_cpus=0 because they shouldn't consume scheduling resources.

Seems logical, right?

The problem: num_cpus=0 means Ray can schedule an unbounded number of these on a single node, since they cost "nothing" to the scheduler.

Each one is a separate Python process.

We created ~500 coordinator actors. Ray put all of them on one node.

The node ran out of PIDs and memory from process overhead alone. lmao.

Use fractional CPUs:

# DON'T: unbounded scheduling
@ray.remote(num_cpus=0)
class Coordinator:
    pass

# DO: soft upper bound via fractional resources
@ray.remote(num_cpus=0.01)
class Coordinator:
    pass
# On a 16-CPU node, this limits to ~1600 coordinators

Memory: The Resource Ray Doesn't Track Well¶

Ray has a memory resource parameter but it's advisory only.

It's used for scheduling decisions but there's no cgroup enforcement. A task that declares memory=2GB can absolutely use 20GB and Ray won't stop it until the Raylet's reactive memory monitor detects the overuse and starts killing workers — by which point you're already in swap hell.

In practice, what i do:

Set conservative memory declarations (overestimate by 50%)
Monitor with Prometheus (ray_node_mem_used)
Build kill-switches into long-running actors that check their own RSS
Accept that memory management in Ray is a best-effort system and design accordingly

Part 10: Debugging in Production¶

Log Deduplication Trap¶

By default, Ray deduplicates identical log lines from workers. 1000 tasks that all print the same error? You see it once. This sounds like a nice quality-of-life feature until:

You lose the ability to correlate errors with specific tasks
Intermittent errors that happen 3/1000 times look like they happened 1/1000 times
All timing information is lost

# In production, ALWAYS disable this
ray.init(
    _system_config={"enable_worker_log_dedup": False},
)

Hanging Task Debugging Playbook¶

The #1 production debugging scenario i've encountered: tasks that never complete.

My playbook:

Check 1: Are you calling ray.get() inside a task?

@ray.remote
def outer():
    ref = inner.remote()
    return ray.get(ref)  # Blocks a worker slot

@ray.remote
def inner():
    return 42

# If all worker slots run outer(), no slots are available for inner()
# CLASSIC DEADLOCK

This happened at the fintech platform during our PPO pipeline.

The policy model actor called ray.get() on a reward model result inside a training step. Under load, all worker slots were occupied by policy training, leaving no room for the reward computation.

Deadlock. Training just... stopped. No error, just silence. Fun times.

Use async tasks, or ensure you have more worker slots than your maximum concurrent blocking calls.

Check 2: Placement groups that can never be satisfied

Run ray status. It shows pending placement groups and why they're pending.

Check 3: Object transfers stuck

Run ray memory --stats-only. Look for objects with pending transfers.

Check 4: NCCL timeout

The default NCCL timeout is 30 minutes. A single hung GPU will block all workers for 30 minutes before the error surfaces. Set NCCL_TIMEOUT to something shorter (5-10 minutes) unless you have a good reason not to.

Metrics That Actually Matter¶

After three deployments, these are the Prometheus metrics I alert on:

ray_object_store_memory      — approaching limit = spilling imminent
ray_node_mem_used            — approaching physical limit = OOM imminent
ray_gcs_update_resource_usage_time_ms — high = scheduling degradation
ray_serve_num_pending_queries        — high = inference saturation
ray_raylet_num_infeasible_scheduling_classes — stuck tasks

The Ray dashboard is fine for development. It is not sufficient for production observability. Export to Prometheus/Grafana and set up real alerts.

Part 11: The Antipattern Hall of Fame¶

#1: Million Tiny Tasks¶

# BAD: scheduling overhead dominates (1-5ms per task)
refs = [process.remote(item) for item in million_items]

# GOOD: batch into chunks
refs = [process_batch.remote(chunk) for chunk in chunked(items, 1000)]

Rule of thumb: if your task runs for less than 100ms, batch it.

#2: Passing Large Objects as Arguments¶

# BAD: serializes big_data for EVERY task call
big_data = load_huge_dataset()
refs = [process.remote(big_data) for _ in range(100)]
# This serializes big_data 100 times!

# GOOD: put it in the object store once
big_data_ref = ray.put(big_data)
refs = [process.remote(big_data_ref) for _ in range(100)]

i hit this one at the enterprise environment with our embedding batches.

The fix took about 30 seconds to implement and cut task submission time by 100x.

(embarrassing in retrospect but extremely satisfying to discover)

#3: Global State in Tasks¶

# BAD: global state is different in every worker process
counter = 0

@ray.remote
def increment():
    global counter
    counter += 1    # Each worker has its own copy
    return counter  # Always returns 1

Use actors for shared state. Always. This isn't a suggestion; it's a fundamental property of Ray's process-per-task model.

#4: Not Handling ObjectLostError¶

Objects can be lost due to node failure, owner death, or object store eviction. In production, ObjectLostError will happen. Wrap your ray.get() calls and have a recovery strategy.

#5: max_concurrency Defaults for Async Actors¶

max_concurrency defaults to 1000 for async actors. If your async method allocates significant memory (like, say, loading a batch onto a GPU), you now have 1000x the memory pressure. Always set this explicitly based on your actual capacity.

# ALWAYS explicit
worker = InferenceActor.options(
    max_concurrency=8,  # Match your GPU batch capacity
    num_gpus=1,
).remote()

#6: Detached Actor Leaks¶

At the enterprise environment, we used detached actors (lifetime="detached") for singleton services — a global rate limiter, a model registry cache, a metrics aggregator.

Detached actors survive driver exit, which is the whole point.

But if your job crashes before cleanup, those actors become orphans consuming resources indefinitely.

We found 47 orphaned detached actors during a cluster audit.

Each was a Python process holding memory and (in some cases) GPU handles. Combined, they were consuming 12GB of RAM and 2 GPU slots on a node that thought it was "fully utilized" but wasn't doing any useful work.

(discovering this was... a moment lol)

An actor reaper pattern helps:

@ray.remote(lifetime="detached", name="actor_reaper", namespace="system")
class ActorReaper:
    def __init__(self):
        self.registry = {}  # name -> (created_at, max_ttl)

    def register(self, actor_name, namespace, ttl_seconds=3600):
        self.registry[(actor_name, namespace)] = (time.time(), ttl_seconds)

    async def reap(self):
        while True:
            await asyncio.sleep(60)
            now = time.time()
            for (name, ns), (created, ttl) in list(self.registry.items()):
                if now - created > ttl:
                    try:
                        actor = ray.get_actor(name, namespace=ns)
                        ray.kill(actor)
                    except ValueError:
                        pass  # Already dead
                    del self.registry[(name, ns)]

Always use explicit namespaces too. The default namespace is "" (empty string). If different jobs use the same actor name in the default namespace, they'll collide silently. We had two different teams' model servers fighting over the actor name "model_server" in the default namespace.

Part 12: Performance Checklist¶

This is the checklist I run through before every Ray deployment now, born from the accumulated scar tissue of three production environments. I asked Claude to help me organize this for ease of you, the reader, in working with Ray.

Pre-deployment:

/dev/shm sized appropriately (Docker: --shm-size, K8s: emptyDir)
Object spilling configured to fast local storage (NVMe, not EBS)
OMP_NUM_THREADS=1 and MKL_NUM_THREADS=1 set (prevents CPU oversubscription)
Log dedup disabled
Prometheus metrics export configured

Scheduling:

Small tasks (<100ms) batched into larger chunks
Large objects passed via ray.put(), not as arguments
Placement groups used for GPU topology requirements
Fractional CPUs for lightweight actors (never num_cpus=0)

Memory:

Object store usage monitored with alerts
Driver doesn't hold too many ObjectRefs (tree reduction for fan-out)
object_store_memory set per node
Regular ray memory --stats-only checks

Training:

Checkpoints go to shared storage (S3/NFS), NEVER local disk
All workers call report() (it's a barrier — miss one and everyone hangs)
torch.backends.cudnn.benchmark = True for fixed input sizes
NCCL env vars set (NCCL_DEBUG=WARN minimum, NCCL_SOCKET_IFNAME for network pinning)
NCCL timeout reduced from default 30min to 5-10min

Serving:

target_ongoing_requests = 50-70% of max_ongoing_requests
downscale_delay_s set conservatively (5+ minutes for GPU workloads)
Health checks in separate concurrency groups
Async actors for I/O-bound serving

Part 13: What i'd Tell Myself Three Years Ago¶

If i could go back to the beginning, starting the fintech platform:

1. Ray is infrastructure, not a library. Treat it with the same respect you'd give Kubernetes or PostgreSQL. It has operational characteristics, failure modes, and performance cliffs that you NEED to understand before you go to production.

2. The ownership model is the most important concept. Not tasks, not actors, not Serve. Ownership. Understand who owns what ObjectRef and what happens when that owner dies. (seriously, this one will save you $800 in compute and 18 hours of PPO training)

3. Start with the napkin math. Before writing a single line of Ray code, calculate your GPU memory budget, your serialization overhead, your network latency requirements. These numbers will tell you which placement strategy to use, how many GPUs you need, and whether Ray is even the right tool.

4. Budget time for the environment. Network configuration, shared memory sizing, Docker/K8s setup — this is 30-50% of the actual work in a regulated enterprise. The Ray code itself is maybe 20%. (enterprise environments are a special kind of learning experience)

5. The simpler solution might be better. In my current role, Ray is nowhere in the stack. Managed services handle model serving. The complexity lives in the application layer now — agent orchestration, workflow state management, tool integration. The fact that i deeply understand Ray helps me recognize when i don't need it. That's maybe the most valuable thing it taught me.

Closing¶

Here's the scale of what these Ray-powered systems touched:

             Fintech       Enterprise      QSR           Current
GPU spend    5 figs/mo     5 figs/mo       5 figs/mo     ~$0 (managed)
Ray usage    Heavy (core)  Heavy (core)    Heavy (core)  None
Scale        Internal      Thousands       Hundreds      > Million users
Model size   3-7B          13-70B          13-70B        Managed

Ray is the most powerful distributed computing framework i've worked with for ML workloads.

It's also the most footgun-laden.

The difference between "this is amazing" and "why is everything on fire" is understanding that it's infrastructure. It has its own gravity. You orbit around it or you collide with it.

The honest truth: you will lose training runs. You will debug for hours only to find it was /dev/shm. You will watch spot instances die mid-training. You will spend weeks on network approvals.

But once you internalize the patterns — ownership, placement, serialization, resource isolation — Ray becomes the thing that lets you build systems that would be impossibly complex otherwise.

The engineers who succeed with Ray are the ones who stop trying to use it and start living with it.

If you're in the middle of a production deployment right now and something's broken: you're not alone. These patterns are hard-won. Take the checklist. Use the playbook. And when you find a new footgun, write it down.

That's how the next engineer learns.

Automated X/Twitter Feed Curation with Bird CLI + Claude

2026-02-04T09:11:00-06:00

I use X because I like to see what other people are posting, working on, building. Especially in AI/ML.

I live in Dallas. I'm not tapped into the Silicon Valley environment like people who live in SF or NYC. X is my signal. It's how I stay involved, how I interact with the broader tech world.

But here's the reality: Twitter's "For You" feed drops 100+ tweets a day. Some signal, lots of noise. And yes, it's still a dopamine source. I'm not quitting X. But sometimes I just want to know what's recent, what's happening, without scrolling for 30 minutes.

The options aren't great: - Scroll manually: Time sink, easy to miss things, dopamine trap - Turn off Twitter: Miss genuine insights, lose connection to the ecosystem - Build filters: Hard to get right, still requires active scrolling

What I wanted: automated triage. Fetch everything, surface what matters, deliver a digest I can read in 5 minutes. Personalized (because it's from my own For You page). With community sentiment analysis (because I already read replies manually—this just automates it).

So I built it.

What I Built¶

Automated feed scraper that: - Fetches 100 tweets from my "For You" timeline daily at 6 PM - Analyzes them with Claude (via OpenClaw) - Triages to 10-15 significant items - Fetches community replies to assess sentiment - Delivers a formatted digest to Slack

Built with Bird CLI (cookie-based X scraper) + OpenClaw (multi-agent bot framework) + Claude.

High-level flow:

Cron Job → OpenClaw Agent → Bird CLI (fetch tweets) → Claude (analyze) → Slack

Scheduled task triggers agent → agent runs Bird CLI to fetch timeline → saves raw JSON → analyzes with Claude → fetches replies for top tweets → formats digest → delivers to Slack.

The Components¶

Bird is a fast X/Twitter CLI tool that uses X's internal GraphQL API—the same endpoints the web app uses. It uses cookie-based authentication, which means no $100+/month API fees.

Why I chose it: - Official X API costs $100+/month for basic access - Bird uses your browser cookies → free - Fast, reliable, well-maintained

How it works: - Extracts auth_token and ct0 cookies from your browser - Calls X GraphQL endpoints directly - Returns JSON

Key commands I use:

bird home --json --count 100  # Fetch timeline
bird replies <tweet-id> --json  # Fetch replies to a specific tweet

Installation:

npm install -g @anthropics/bird

Test it:

bird home --json --count 10

If it works, you'll see JSON output with tweet objects (id, text, author, engagement metrics).

OpenClaw: Multi-Agent Bot Framework¶

OpenClaw is a personal AI assistant framework I use for managing multiple agents across different channels. I wrote about my multi-agent setup with OpenClaw in a previous article.

Why I chose it for this: - Already using it for my content workflow - Built-in cron job support - Agents can run shell commands (like bird) - OAuth through Claude Pro subscription (no API key billing)

How it works in this setup: - Cron job triggers the agent in an isolated session - Agent runs bird home --json to fetch tweets - Agent analyzes output with Claude - Agent delivers formatted digest to Slack

If you don't have OpenClaw, you can replace it with a Python script. Same flow, different tools. I'll explain that later.

Learn more: OpenClaw on GitHub

Cron Job: Scheduled Execution¶

The job runs daily at 6 PM CST. It's configured in ~/.openclaw/cron/jobs.json:

{
  "id": "feed-highlights-slack",
  "agentId": "content",
  "schedule": {
    "kind": "cron",
    "expr": "0 18 * * *",
    "tz": "America/Chicago"
  },
  "sessionTarget": "isolated",
  "payload": {
    "kind": "agentTurn",
    "message": "<PROMPT>",
    "deliver": true,
    "channel": "slack",
    "to": "<CHANNEL_ID>"
  }
}

Key fields: - agentId: Which agent handles this ("content" in my case) - schedule.expr: Cron expression ("0 18 * * *" = 6 PM daily) - sessionTarget: "isolated": Runs in a fresh session, doesn't pollute main conversation - payload.deliver: true: Sends output to a channel - payload.channel: Target platform ("slack") - payload.to: Channel ID to deliver to

The Prompt: Triage + Sentiment Analysis¶

The prompt is where the magic happens. It's split into two phases:

Phase 1: Triage - Analyze all 100 tweets - Identify the 10-15 most significant items - Prioritize: AI/ML breakthroughs, software engineering insights, controversial takes, new releases, industry shifts - Note 5-10 honorable mentions (things worth a quick glance but not deep analysis)

Phase 2: Sentiment Analysis - For each of the 10-15 top tweets, fetch community replies - Extract 2-4 quote snippets from replies - Categorize reactions based on context: - For controversial takes: Pro / Against - For technical content: Validating / Challenging / Additional Resources - For announcements: Excited / Skeptical / Concerns

Voice & Tone (from the prompt):

Blunt, direct, no-nonsense. Zero sycophancy or hype. Don't oversell. A small insight is a small insight—say so. A big deal is a big deal—say so. Avoid "this changes everything" or "this is huge" unless it genuinely is. No "this, not that" style writing. Just state what it is. Let the signal speak for itself.

I told Claude to be honest about scale. If something's a small nugget, say it's a small nugget. If it's a big deal, say it's a big deal. No artificial hype.

The full prompt is ~300 words. I won't paste the whole thing here, but the pattern is: triage → sentiment analysis → format digest.

Slack Integration: Delivery¶

OpenClaw delivers the digest to a dedicated Slack channel. Configuration in ~/.openclaw/openclaw.json:

{
  "channels": {
    "slack": {
      "mode": "socket",
      "enabled": true,
      "channels": {
        "<CHANNEL_ID>": {
          "requireMention": false,
          "systemPrompt": "Twitter feed digest mode..."
        }
      }
    }
  }
}

Key settings: - mode: "socket": Uses Slack Socket Mode (no public URL needed) - requireMention: false: Bot responds without @mention in this channel - Agent bindings route this channel to the "content" agent

Why Slack: - Already using it for content workflow - Clean formatting (code blocks, links, quotes) - Threaded discussions if I want to dig deeper - Persistent archive

The Setup¶

Here's how to replicate this.

Step 1: Install Bird CLI¶

npm install -g @anthropics/bird

Test it:

bird home --json --count 10

If it works, you'll see JSON output. If not, Bird might need browser cookies. Make sure you're logged into X in your default browser.

Step 2: Configure OpenClaw Agent¶

If you're using OpenClaw, create or use an existing agent with: - Workspace: ~/assistant/data/feed/ (for saving raw JSON) - Tools: exec (to run bird commands) - Model: Claude Sonnet 4.5

If you're not using OpenClaw, skip this. You'll write a custom script instead (see "OpenClaw vs Custom Script" below).

Step 3: Create Cron Job¶

Add this to ~/.openclaw/cron/jobs.json:

{
  "id": "feed-highlights-slack",
  "agentId": "content",
  "schedule": {
    "kind": "cron",
    "expr": "0 18 * * *",
    "tz": "America/Chicago"
  },
  "sessionTarget": "isolated",
  "wakeMode": "now",
  "payload": {
    "kind": "agentTurn",
    "message": "Run `bird home --json --count 100` to fetch your Twitter For You timeline. Save the raw JSON to ~/data/feed/$(date +%Y-%m-%d).json for reference. Phase 1: Triage - Analyze all 100 tweets. Identify the 10-15 most significant items worth deeper analysis. Prioritize: AI/ML breakthroughs, software engineering insights, controversial or contrarian takes, new releases, industry shifts, anything genuinely interesting. Also note 5-10 honorable mentions worth a quick glance. Phase 2: Reply Sentiment Analysis - For each of the 10-15 significant tweets, run `bird replies <tweet-id> --json` to fetch community reactions. Voice & Tone: Blunt, direct, no-nonsense. Zero sycophancy or hype. Don't oversell. A small insight is a small insight—say so. A big deal is a big deal—say so. Avoid 'this changes everything' or 'this is huge' unless it genuinely is. No 'this, not that' style writing. Just state what it is. Let the signal speak for itself. Output Format: Use a single emoji as a visual separator before each item. For each of the 10-15 significant items: 1. Title/Hook + link to tweet, 2. Core insight - what's actually being said, 3. Why it matters - concrete impact, not hype. Be specific: who does this affect? what changes?, 4. Community Reaction - include 2-4 actual quote snippets from replies as bullet points. Categorize intuitively based on topic context: For controversial takes: Pro / Against, For technical content: Validating / Challenging / Additional Resources, For announcements: Excited / Skeptical / Concerns. Honorable Mentions: At the end, include a section with 5-10 tweets that didn't make the top list but are still interesting. For each: one-line summary + link to tweet.",
    "deliver": true,
    "channel": "slack",
    "to": "<YOUR_SLACK_CHANNEL_ID>",
    "bestEffortDeliver": true
  }
}

Replace <YOUR_SLACK_CHANNEL_ID> with your actual channel ID.

Step 4: Configure Slack Channel¶

Add the channel to ~/.openclaw/openclaw.json:

{
  "channels": {
    "slack": {
      "channels": {
        "<YOUR_SLACK_CHANNEL_ID>": {
          "requireMention": false,
          "systemPrompt": "Twitter feed digest mode. You scrape the For You timeline and surface what matters. Be ruthlessly selective."
        }
      }
    }
  }
}

Bind the channel to your agent:

{
  "bindings": [
    {
      "agentId": "content",
      "match": {
        "channel": "slack",
        "peer": {
          "kind": "channel",
          "id": "<YOUR_SLACK_CHANNEL_ID>"
        }
      }
    }
  ]
}

Step 5: Restart Gateway + Test¶

Restart the OpenClaw gateway to load the new config:

pkill -9 -f openclaw-gateway
nohup pnpm openclaw gateway &

Force-run the job to test:

pnpm openclaw cron run --force feed-highlights-slack --timeout 300000

Check Slack for the digest. If it works, it'll run automatically at 6 PM daily.

What the Output Looks Like¶

Each digest has: - 10-15 significant tweets with full analysis - Community reaction quotes (2-4 per tweet) - 5-10 honorable mentions (one-liners + links)

Example item:

🔹 **GPT-5 Announcement Leaks**
https://x.com/openai/status/123456789

OpenAI internal memo suggests GPT-5 training complete, targeting Q2 release.

**Why it matters:** If real, competitive pressure on Claude intensifies.
Affects model selection decisions for production systems.

**Community Reaction:**
*Excited:*
- "Finally, been waiting for this since GPT-4 plateaued"
- "The benchmarks better be real this time"

*Skeptical:*
- "Remember when GPT-4.5 was 'imminent'? I'll believe it when I see it"
- "Leaks are marketing. Wake me up when there's an API."

The format is consistent. Easy to scan. Community quotes add context I'd miss if I just read the tweet itself.

Why This Approach¶

Official X API: - $100+/month for basic access - Rate limits - Complex OAuth setup

Bird CLI (cookie auth): - Free (uses your existing logged-in session) - Fast, reliable - No rate limits beyond what the web app has

Tradeoff: If you clear cookies or log out, Bird stops working. Keep a logged-in X session in your default browser profile.

OAuth vs API Keys¶

OpenClaw uses OAuth through my Claude Pro subscription: - No per-token API billing - Usage counts against Claude Pro limits (5h/day) - Tokens auto-refresh via background service

Why this works: - Daily digest takes ~2-3 minutes of Claude usage - Well within Claude Pro limits - No separate API billing

If you don't have OpenClaw: Use the official Claude API with the anthropic Python SDK. Same prompt, different execution layer.

OpenClaw vs Custom Script¶

Why I used OpenClaw: - Already using it for other workflows (journaling, writing, accountability) - Built-in cron support - Built-in Slack integration - Session management handled

If you don't have OpenClaw:

Write a Python script with: - subprocess to call bird CLI - anthropic SDK to call Claude API - slack_sdk to post to Slack - cron or systemd timer for scheduling

Same flow, different tools. The pattern is: 1. Fetch tweets with Bird CLI 2. Analyze with Claude 3. Deliver to Slack 4. Schedule it

The code is just glue. The real value is the prompt.

Gotchas & Lessons Learned¶

Gateway caches config: - Editing jobs.json doesn't take effect until gateway restart - Always restart after config changes

Bird CLI needs fresh cookies: - Don't use incognito mode - Keep a logged-in X session in your default browser

Set timeout appropriately: - Feed analysis with reply fetching takes 2-3 minutes - Set --timeout 300000 (5 minutes) when force-running

Gateway rewrites jobs.json: - After each run, the gateway updates the state field in jobs.json - Edit config → Restart gateway → Run job (in that order)

Is It Worth It?¶

What works: - I actually read the digest (5 minutes vs 30+ minutes scrolling) - Community sentiment quotes add context I'd miss otherwise—I was already reading replies manually, this just surfaces them automatically - Catches things I wouldn't have seen manually - Keeps me connected to the AI/ML ecosystem without being glued to X

What doesn't: - Still misses some nuance (threading, quote tweets) - Sentiment analysis is good but not perfect - Occasionally surfaces stuff that's not relevant to me - Doesn't replace X (I still use it for dopamine, let's be honest)

Would I build it again?

Yes. The time savings alone justify it. But I'd iterate on the prompt (more specific filtering). And I'd add a "mark as read" feedback loop so Claude learns what I care about.

Why I wrote this:

I wanted to make it super easy for others to replicate. If I'd had this guide when I started, it would have saved me hours of trial and error.

I know other people struggle with the same problem: wanting to stay connected without drowning in the feed. This is my way of staying involved in the AI/ML world from Dallas, not Silicon Valley. Maybe it helps you stay involved from wherever you are.

Final Thoughts¶

This is a simple automation that saves me real time. Bird CLI + Claude + OpenClaw = automated feed curation. If you're drowning in Twitter, try it. If you don't have OpenClaw, replace it with a Python script. Same flow, different tools.

The code is just glue. The real value is the prompt. Triage → sentiment analysis → digest. That's the pattern.

I built this because I needed it. I'm sharing it because I think you might need it too. If you build something similar, let me know. I'm curious what other people do for feed curation.

Slack vs Telegram: Catching Myself in a Pattern

2026-02-04T07:57:00-06:00

I published an article about using Telegram forum topics for multi-mode AI assistants in the last couple of days. One bot, multiple agents, isolated contexts. It was working well.

Then a friend asked a simple question.

The Question¶

Yasin asked: "Why Telegram and not Slack?"

My immediate reaction was to defend the choice. Slack feels cluttered. Telegram feels cleaner, more conversational. I use Slack for work, and I wanted something separate for journaling and writing. Slack = corporate. Telegram = personal.

Honestly, I think I conflated Slack with Notion in my head. Notion feels slow and clunky to me, and somehow I'd put Slack in the same category. They both felt like tools I didn't want to use.

The Story I Told Myself¶

Here's what I said about Slack: - Less UI friction in Telegram - Slack feels overwhelming - I need work/play separation - Telegram topics feel more structured

Here's what I didn't say: - My previous Slack workflow was a mess - Lots of old channels, failed experiments, accumulated clutter - Mental friction from past failures, not the platform itself

I was carrying forward a story from a previous failed attempt and framing it as "this doesn't work for me."

What Actually Happened¶

This morning I switched. Cleaned up old channels. Changed the theme to something that felt cleaner, more like a fresh start. Reconfigured the setup.

Took less than an hour.

It felt fluid immediately. Better UI for what I was doing. Easier to compartmentalize. Threads, search, table rendering, basically all the things I'd ignored because I was stuck in the old story.

The shift was awesome.

The realization hit: the problem wasn't Slack. The problem was accumulated mess from old attempts. I'd been avoiding the tool when what I really needed was to clean up the clutter first.

The Broader Pattern¶

I think there's a broader pattern here, especially when working with LLMs.

When you're in a conversation with Claude Code or Codex or whatever, it's really easy to rush into the next decision. You're excited. You want to try the thing. You're already in this specific environment, so you just keep iterating there.

It's not an intentional system design choice. It's a byproduct of momentum and excitement.

I did that here. I was already in Telegram, already building, so I just kept going. I didn't objectively evaluate whether Slack would actually be better. I just defended the choice I'd already made based on past baggage.

The cost: I missed nuances. I didn't assess the tools on current conditions. I made a decision based on past emotions instead of trying to figure out what was really blocking me.

Where Else Am I Doing This?¶

This is the question that matters.

If this is a pattern, where else is it showing up?

Tools I've dismissed based on old failed attempts? Workflows I avoid because they "didn't work" last time, but really I just didn't set them up right?

Making decisions based on past emotions instead of current evidence kills momentum.

The Lesson¶

Check your assumptions. Past failure doesn't equal current conditions.

The blocker wasn't Slack. It was old clutter I hadn't cleaned up yet. Once I cleared it, the tool worked great.

If something feels like a blocker, ask: is this actually a problem with the tool, or am I carrying forward a story from a previous failed attempt?

Sometimes the thing you're avoiding is just the thing that needs cleaning up first.

Optimizing vLLM at Production Scale: Lessons from Conversational AI Infrastructure

2026-02-03T14:00:00-06:00

Disclaimer: This post reflects patterns and lessons learned from building vLLM-based inference systems at production scale. Technical details have been generalized, and no proprietary information from any specific organization is disclosed.

I spent a significant portion of the past 2 years building conversational AI infrastructure for real-time systems.

vLLM was the foundation. PagedAttention, continuous batching, prefix caching are all subsets of that. But often is the case where production issues don't 1:1 resemble what the docs tell you. Memory fragmentation is something I learned about, which ended up being huge as it creeps up over days. Throughput cliffs after bursts in service usage can force architectural changes. Even something that everyone talks about now, quantization, can introduce accuracy issues on seasonal items that only show up in logs.

Here's some context about what I learned.

Part 1: Well-Known Principles (Foundation)¶

If you're evaluating vLLM, you probably already know this. But there are nuances worth clarifying.

PagedAttention Is Virtual Memory for GPUs¶

PagedAttention treats the KV cache like OS virtual memory, AKA there's almost zero GPU waste on padding as blocks are allocated AS NEEDED. So fixed-size blocks and non-contiguous allocation. Traditional systems pre-allocate contiguous memory per sequence and waste 60-80% through fragmentation. vLLM reduces this to under ~4%.

The detail most people miss here is you're running PagedAttention v2, not v1. The original paper used copy-on-write for forked sequences (beam search). v2 eliminated that overhead with reference counting on blocks. It's been the default since mid-2024 and especially in production environments.

For those of you who are in the weeds of PagedAttention already, the picture gets even spicier because now that FlashAttention and FlashInfer handle compute, the architecture is even MORE like OS virtual memory. AKA vLLM's Block Manager handles the allocation semantics, the attention backend does indirect reads through block tables, and the model doesn't care.

Memory management is DECOUPLED from attention kernel optimization.

If you're lost, ignore my pretentious outpourings and keep chugging forward.

Continuous Batching vs Static Batching¶

vLLM doesn't wait for a batch to finish before admitting new requests. As each sequence completes, its slot is immediately filled by a waiting request.

The production nuance here is that continuous batching creates variable GPU utilization. During peak hours, utilization stays consistently high (80-95%). During off-peak, you get spiky utilization.

This is an issue because it confuses your autoscalers, unless you set autoscaler cooldown periods to at least 60 seconds.

The TTFT / ITL / Throughput Triangle¶

Three CRUCIAL metrics:

TTFT (Time To First Token): This is dominated by prefill compute. It is proportional to prompt length.
ITL (Inter-Token Latency): This is dominated by memory bandwidth. Model weights load from HBM every decode step.
Throughput: This is dominated by batching efficiency. More concurrent sequences means better GPU utilization.

For real-time conversational systems, TTFT matters most for UX. Customers hear silence. ITL matters for smooth audio synthesis (stuttering breaks immersion). Throughput matters for cost.

You can't optimize all three simultaneously. Pick the constraint that matters for your use case.

How do you figure out what matters? Discernment and Metrics. Oh also maybe talk to people.

Part 2: Production Nuances (The Meat)¶

This is where production tends to diverge from benchmarks.

vLLM V1 Architecture: Everything Changed¶

vLLM underwent a complete rewrite in early 2025, which deprecated V0. I was working with vLLM through this change, so if you're reading old blog posts, they could be misleading.

What objectively changed (i organized this table with claude's help while researching for this paper):

Component	V0	V1
Scheduler	Separate prefill/decode phases	Unified — treats all tokens the same
Worker Architecture	Asymmetric (scheduler on Worker 0)	Symmetric — scheduler in separate process
Prefix Caching	Optional, had CPU overhead	Default ON, near-zero overhead at 0% hit rate
CUDA Graphs	Full graphs only	Piecewise — captures portions individually
Input Preparation	Recreated every step	Persistent Batch — cached tensors, diffs via NumPy
API Server	Single-process	Multi-process ZeroMQ IPC, overlaps tokenization with GPU

Okay but why does this matter lmaorofl:

Prefix caching is ON by default in V1. Your system prompt (1500-2000 tokens) gets cached automatically. For conversational systems with repeated prompts, this is 100-200ms that gets saved on every request. It also gets rid of a lot of CPU overhead you may notice in your logs.

V1 achieves 1.7x higher throughput than V0 on identical hardware, purely from reduced CPU overhead. We love vLLM devs.

The REAL KV Cache Story: Fragmentation, Offloading, and Compression¶

Memory Fragmentation¶

Oof this one hurt.

PagedAttention allocates blocks from a free list. As sequences of varying lengths complete, blocks are returned in non-contiguous order. Over time, the free list fragments. There are enough free blocks total, but they can't satisfy large contiguous allocation requests efficiently.

Symptom AKA how we noticed this: p99 latency drifts on average 5-10% per hour. Averages hide this completely which is what caused us noticing this VERY late. You need percentile-based alerting.

We observed this directly. 12-14 hours in, p99 started drifting. By 36 hours, degradation was significant. Eventually this led to the bane of our existence, OOM kills.

Piecing this together correlates three symptoms: 1. OOM kills (what alerted us) 2. p99 drift (visible in retrospect) 3. Preemption warnings in logs (correlated later)

Mitigation (better than restarts):

KV Cache Offloading to CPU (claude told me this is vLLM 0.11.0+): Offload KV to CPU DRAM. The new contiguous memory layout (0.12.0) made this practical with ~9x throughput improvement with high cache hit rates. For systems where the prompt is identical across requests, this is a game-changer.
FP8 KV Cache Quantization: Separate from weight quantization. Cuts KV cache memory in half with less than 1% in quality loss (we noticed a bit more than this but more on tracking down the why later). This is available with --kv-cache-dtype fp8_e5m2. For short conversations, the quality impact is negligible though.

We stuck with 12-hour rolling restarts. At our scale (20-30 concurrent peak per regional hub), the fragmentation rate was manageable. Teams running higher concurrency (100+) need offloading or FP8.

KV Cache Memory Calculation¶

For the crackheads that are constantly asking Claude for how to estimate vRAM and KV Cache needs every other week, this will hopefully look vaguely familiar:

KV cache per token = 2 × num_layers × num_kv_heads × head_dim × dtype_size

For Qwen2.5-7B (FP16):
  = 2 × 32 × 4 × 128 × 2 bytes
  = 64 KB per token

For a 2048-token conversation:
  = 2048 × 64 KB = 128 MB per conversation

For 30 concurrent conversations:
  = 30 × 128 MB = 3.84 GB of KV cache alone

With FP8 KV cache:
  = 1.92 GB (half the memory, double the concurrent capacity)

Block Size Matters¶

vLLM's default block size is 16 tokens. This affects:

Prefix caching granularity: Hashes are computed per block. Cache matching happens in 16-token chunks.
KV offloading throughput: Larger physical blocks = better DMA throughput. vLLM 0.12.0 changed memory layout to make each physical block contiguous across all layers (16KB → 2MB for Llama-8B)(this is what we just talked about).
Memory waste at sequence boundaries: Last block of each sequence wastes memory. Google for the full estimation equation.

Preemption: The Hidden Failure Mode¶

When KV cache fills up, vLLM must preempt running requests. This is just a fancy term for stopping a task that's running in order to free up some resources that may be exhausted.

Not gonna lie this can be hard to catch.

How it works (V1):

Scheduler detects insufficient KV blocks
Recompute mode: Evicts lowest-priority request's KV cache entirely. When re-scheduled, prefill starts from scratch.
V0 had swap mode—deprecated. V1 uses KV offloading instead

The warning you'll see:

WARNING scheduler.py:1057 Sequence group 0 is preempted by PreemptionMode.RECOMPUTE

This warning is your canary. If you see it regularly, you're running too many concurrent sequences for your GPU memory. Reduce max_num_seqs or enable KV cache offloading.

Impact: A preempted request loses ALL its KV cache. For a multi-turn conversation that's 5 turns deep: - Lose ~1500 tokens of computed KV - Recompute from scratch on re-schedule - Adds 100-300ms latency to that specific customer

Mitigation: Set max_num_seqs to a value where preemption never triggers under peak load. Better to queue new requests with a semaphore than preempt running ones.

Chunked Prefill¶

The problem: Without chunked prefill, a single request with a long prompt (2000-token system prompt + history) blocks ALL decode steps for other in-flight requests. Customer A's response stalls while Customer B's prefill runs.

How chunked prefill solves it: vLLM splits prefill into chunks (default: 512 tokens per chunk in V1). Between chunks, decode steps for other requests run. This interleaves prefill and decode work.

Without chunked prefill:
  |---PREFILL (2000 tokens)---|---decode---|---decode---|
  Customer B hears nothing ^^^^^^^^^ during A's prefill

With chunked prefill:
  |PREFILL(512)|decode|PREFILL(512)|decode|PREFILL(512)|decode|
  Customer B still gets tokens ^^^ while A's prefill runs

Tuning:

engine_args = {
    "max_num_batched_tokens": 2048,  # Total token budget per step
    # V1 automatically splits prefill to fit this budget
}

For short conversations, keep max_num_batched_tokens at 2048. But during turn 5+ (when history accumulates), chunked prefill prevents one customer's long history from stalling all others.

Speculative Decoding¶

Speculative decoding uses a small draft model to predict future tokens, then verifies them with your target model in parallel. When the draft is correct, you get multiple tokens per forward pass.

The theory is simple, but the production reality is that it doesn't always help.

When it helps: - Models where draft acceptance rate is high (>70%) - Tasks with predictable patterns (code completion, structured output) - When you're memory-bound, not compute-bound (small batch sizes)

When it hurts: - Conversational AI with unpredictable user responses - Large batch sizes where GPU is already saturated - Draft model overhead exceeds the speedup from accepted tokens

We tested speculative decoding with Qwen2.5-7B as the target and Qwen2.5-0.5B as the draft model. Acceptance rate was around 50-60% for conversational responses. The overhead of running two models (even with the small one) actually increased p50 latency by ~15-20ms. The intuition is that there is a limited subset of potential responses and outputs due to the domain. We aren't answering theoretical questions about quantum computing. We're asking how we should put the fries in the bag.

And in writing this article I realized that the math didn't work out either. We need 3-4 accepted tokens per speculation to break even on the added overhead. At 50-60% acceptance, we were getting 1-2 tokens per speculation. WACK.

Key insight here is speculative decoding is a latency optimization, not a throughput optimization. If you're already batch-saturated, skip it. If you're running single requests with strict latency SLOs, test it carefully with YOUR traffic patterns. Although we ARE latency optimizing, it didn't fit our specific profile.

Quantization: The Accuracy-Memory Tradeoff¶

Quantization reduces model weight precision. FP16 → INT8 or INT4. The goal is to cut memory usage and increase throughput, but it introduces accuracy degradation. This was a given before we did it but the way it surfaced was a learning experience.

The degradation is not uniform across all use cases, which is what makes this annoying af to debug in production.

What we used: - AWQ (Activation-aware Weight Quantization): INT4 quantization that preserves accuracy by identifying and protecting critical weights (critical weights is the caveat to notice here) - GPTQ: Another INT4 method, it uses a slightly different calibration approach

Both gave us similar results for Qwen2.5-7B: - a ~3 to 4 GB memory footprint (down from ~14GB FP16, hell yeah) - ~1.6x throughput improvement (but we are latency optimizing!!!! not throughput optimizing!!!)

Where accuracy issues showed up:

Seasonal/rare entity hallucinations: the model would occasionally hallucinate product names for seasonal items (holiday-specific SKUs). These items had sparse training data, so quantization pushed them below the confidence threshold where the model "remembered" them correctly. This was the most common issue.
Numeric precision: for things like invoice amounts, order IDs, timestamps. INT4 quantization introduced occasional off-by-one errors in numeric outputs (e.g., "$123.45" → "$123.44"). We only found one instance of this but that was significant enough.
Long-context degradation: After turn 5-6 in a conversation, quantized model started losing coherence faster than FP16. This correlates with KV cache quantization compounding with weight quantization. You'd be surprised how many convos reach higher turn numbers. So if you're ordering for a big group, maybe choose the non-AI drive thru.

We caught these through continuous log analysis. Not A/B testing (too slow) nor manual QA (bc it doesn't catch rare failures). We streamed logs to an analysis pipeline that flagged: - Responses with numeric discrepancies - Entity names not in our catalog - User corrections ("that's not right")

Mitigation strategies:

Hybrid quantization: Keep specific layers in FP16 (usually first/last few layers and attention layers). vLLM doesn't support this natively, but HuggingFace Optimum does (learned after the fact)
Dynamic fallback: When confidence scores drop below a threshold, re-run the request with FP16. Adds complexity, but catches edge cases (we would often hedge requests to lower latency in specific query complexity cases)

If you're working with financial data, legal documents, or anything where numeric precision is non-negotiable, test quantization extensively before production. The memory savings are tempting, but the accuracy hit can be subtle and catastrophic.

Concurrent Requests and Little's Law¶

Here's the math that determines your throughput ceiling:

Throughput (requests/sec) = Concurrent Requests / Average Latency (sec)

This is known as Little's Law. It's deceptively simple and governs everything about your system's capacity.

Example: - Average request latency: 2 seconds (TTFT + generation time) - Concurrent requests: 30 - Throughput: 30 / 2 = 15 requests/sec

To double throughput to 30 req/sec, you either: 1. Double concurrent capacity (30 → 60) 2. Halve latency (2s → 1s)

Doubling concurrent capacity means more GPU memory (for KV cache). Halving latency means faster model (smaller model, quantization, better hardware).

The production constraint here is you can't just increase max_num_seqs indefinitely. Each concurrent sequence consumes KV cache memory. Eventually you hit OOM or trigger constant preemption.

For Qwen2.5-7B on an A100 (80GB), with FP16 weights and FP16 KV cache: - Model weights: ~14GB - KV cache at 2048 tokens per conversation: ~128MB per conversation - Available memory for KV cache: ~60GB (after weights, overhead) - Max concurrent conversations: 60GB / 128MB ≈ 470 conversations

But that's theoretical max. In practice, you need headroom for: - Varying conversation lengths (some hit 4096 tokens) - Fragmentation (free list doesn't pack perfectly) - Transient spikes (burst traffic)

We ran at ~25-30 concurrent max per GPU. This gave us comfortable margins and zero preemptions under normal load.

The Throughput Cliff¶

Around 200-300 concurrent requests across our regional deployment, we hit a wall. Throughput didn't scale linearly. Latency spiked. Preemption warnings flooded logs.

The cliff wasn't a single bottleneck. It was three bottlenecks that compounded:

KV cache fragmentation accelerated: Higher concurrency = more varied sequence lengths = faster fragmentation. Memory pressure triggered preemptions, which worsened latency variance.
CPU tokenization overhead: vLLM V0's single-process API server became CPU-bound. Tokenization (especially for long prompts) blocked GPU from receiving new work. V1's multi-process architecture fixed this, but we hit the cliff on V0.
Network saturation: Regional hubs aggregate traffic from edge locations. At 200+ concurrent, network I/O between load balancer and vLLM instances started saturating. This added 10-20ms per request, which compounded with latency variance from preemption.

How we diagnosed this:

Correlated three metrics: - GPU utilization dropped from 85% to 60% (GPU was starved, not saturated) - CPU utilization spiked to 95%+ (tokenization bottleneck) - Network throughput hit the instance's bandwidth limit

Solution:

Upgrade to vLLM V1: Multi-process API server eliminated CPU tokenization bottleneck. GPU utilization returned to 80-90%.
Horizontal scaling with client-side load balancing: Instead of funneling all traffic through one load balancer, clients (edge locations) maintained connections to multiple vLLM instances and load-balanced locally. This distributed network I/O.
Aggressive KV cache management: Enabled FP8 KV cache quantization + 12-hour rolling restarts to prevent fragmentation from accumulating.

After these changes, the cliff moved from 200-300 concurrent to 500+. We never hit it again because our peak load topped out at ~400 concurrent across all regions.

The lesson here is throughput cliffs in production are rarely single bottlenecks. It's usually 2-3 subsystems failing simultaneously. Diagnose with metrics from multiple layers (GPU, CPU, network, memory). Don't assume GPU saturation is your limiting factor.

Part 3: System Design Decisions¶

The optimizations in Part 2 matter, but they sit on top of architectural choices. Here's how we structured the system.

Regional Hub Architecture¶

We deployed vLLM instances regionally, not centrally. Each geographic region (think Northeast, Midwest, South, West) had its own set of GPU instances running vLLM. Traffic from edge locations (the actual stores) routed to the nearest regional hub.

Why regional, not central: - Latency budget. Sub-second TTFT means every hop matters. Round-trip from California to Virginia adds 60-80ms minimum. And if you think that isn't noticeable, you'd be surprised - Network reliability. Regional failures don't take down the entire system. If the West hub goes down, stores fail over to Midwest (higher latency, but still operational)(AVAILABILITY!!! CAP!!!!) - Traffic patterns. Lunch and dinner rushes are timezone-dependent. West coast peak is 3 hours after East coast peak. Regional deployment smooths out global load.

Trade-offs we accepted: - More infrastructure to manage (4 regional hubs instead of 1 central cluster) - Model updates need to roll out regionally (can't update everything at once) - Monitoring complexity (per-region dashboards, cross-region alerting)

At this scale (~100 stores per region, 20-30 concurrent per hub during peak), regional deployment was the right call. If you're serving fewer locations or latency isn't as tight, central deployment is simpler.

Model Serving: One Model, Multiple Instances¶

We ran one model (Qwen2.5-7B) across all instances. No A/B testing different models in production. No ensemble of models for different query types.

Why single-model: - Operational simplicity. One model to optimize, one set of configs to tune, one deployment pipeline. - Prefix caching efficiency. V1's automatic prefix caching works best when the system prompt is identical across requests. Multiple models = multiple system prompts = lower cache hit rates. - Resource predictability. GPU memory requirements are consistent. No surprise OOMs from larger model variants.

We did evaluate smaller models (Qwen2.5-3B) and larger models (Qwen2.5-14B). The 3B had accuracy issues on multi-turn conversations (lost context after turn 3-4). The 14B didn't fit our latency budget (TTFT was 150-200ms, we needed sub-100ms).

Qwen2.5-7B hit the sweet spot: good enough accuracy, fast enough latency, fits on A100 with room for 25-30 concurrent.

Monitoring and Observability¶

Production inference systems can often fail silently because it may not crash but it just starts giving worse answers. You need metrics that catch degradation before customers complain.

What we tracked:

Latency percentiles (p50, p90, p99, p99.9): Averages hide preemption spikes. p99 drift is your early warning for KV cache fragmentation or CPU bottlenecks
Preemption rate: This SHOULD be zero under normal load. If preemptions are happening, you're oversubscribed. Either reduce max_num_seqs or add capacity.
GPU utilization + memory usage: These should move together.

High GPU util + low memory = compute-bound (good) Low GPU util + high memory = memory-bound (consider offloading or quantization) Low GPU util + low memory = CPU-bound or network-bound (check tokenization overhead, network throughput)

Token throughput (tokens/sec): Normalized metric across batch sizes. If throughput drops but GPU util stays high, you're hitting memory bandwidth limits.
Prefix cache hit rate: V1 makes this a default metric. Should be >80% for conversational systems with repeated prompts. If it's low, your prompts aren't consistent or your cache is thrashing.
Model accuracy proxies: You can't measure "accuracy" directly in production (no ground truth). We tracked user corrections ("that's not right", "no I said X"), entity hallucinations (product names not in catalog), and numeric discrepancies (parsed amounts vs. expected ranges).

What we learned (after several incidents):

Percentile-based alerting is non-negotiable. We initially alerted on p50 latency, which was a big mistake. p50 stayed under 500ms even when 5% of requests were hitting 2-3 seconds due to preemption. Customers complained before we saw the alert.

Switched to p99 alerts with a 10% drift threshold over 1 hour. Caught fragmentation issues 6-8 hours before they became critical.

Load Balancing Strategy¶

Client-side load balancing worked better than centralized load balancers for our use case.

How it worked: - Each edge location (store) maintained persistent HTTP/2 connections to 2-3 vLLM instances in its region - Client library (running at the store) round-robins requests across available connections - Health checks every 30 seconds; failed instances removed from rotation

Why client-side: - Eliminates single point of failure. No load balancer to become a bottleneck or fail. - Lower latency. Direct connection from store to vLLM instance, no extra hop. - Better failover. Client detects instance failure within 30 seconds and reroutes. Centralized LB would need health checks + DNS updates + TTL expiry (90+ seconds).

Trade-off: - Client library has more complexity (connection pooling, health checks, retry logic) - Harder to centrally observe traffic patterns (no single LB to pull metrics from)

For high-concurrency, latency-sensitive systems, client-side load balancing is worth the complexity. For lower-scale deployments, stick with a centralized LB (simpler ops, easier debugging).

Graceful Degradation and Fallbacks¶

Real-time conversational systems can't just return 503s when vLLM is overloaded. Customers are literally sitting there waiting. Here come our fallbacks.

What we built:

Request queue with timeout (5 seconds): If all vLLM instances are at max_num_seqs, queue the request. If it's not admitted within 5 seconds, fall back to rule-based responses.
Rule-based fallback: Hand-written responses for common queries ("What's the total?", "Add fries", "Cancel that"). Not great, but better than silence. Customers got an answer even if it wasn't AI-generated.
Graceful degradation for tool calls: Some queries required tool calls (database lookups, inventory checks). If the tool call timed out, the model generated a response without the tool result. Better to say "I couldn't check inventory, but typically that item is available" than to hang.

Impact: - Fallback rate under normal load: <1% - Fallback rate during incidents (regional outage, vLLM restart): 15-20% - Customer satisfaction didn't crater during incidents bc they still got responses

The key here is fallbacks should be built in from day 1. Retrofitting graceful degradation after you're in production is way harder.

Key Takeaways¶

If you're deploying vLLM at production scale, here's what actually matters:

V1 architecture is a step-change improvement. Prefix caching is default-on, multi-process API server eliminates CPU bottlenecks, 1.7x throughput improvement. Upgrade if you're still on V0.
KV cache fragmentation is real. p99 drift over 12-24 hours. Solution: FP8 quantization + offloading (if high cache hit rate) or rolling restarts (if low concurrency).
Preemption is a production failure mode. If you see preemption warnings regularly, you're oversubscribed. Reduce max_num_seqs or add capacity. Don't let it slide.
Chunked prefill prevents head-of-line blocking. Default is 512 tokens per chunk. Tune max_num_batched_tokens for your workload. Critical for multi-turn conversations.
Speculative decoding doesn't always help. Acceptance rate needs to be >70% to break even. Conversational AI with unpredictable responses = 50-60% acceptance = net latency increase. Test with YOUR traffic.
Quantization accuracy degrades non-uniformly. Rare entities, numeric precision, long-context conversations degrade first. Continuous log analysis catches these before A/B tests.
Little's Law governs your capacity. Throughput = Concurrent / Latency. To double throughput, either double concurrency (more GPU memory) or halve latency (faster model, better hardware).
Throughput cliffs are multi-subsystem failures. CPU tokenization + KV fragmentation + network saturation all hit simultaneously. Diagnose with metrics from GPU, CPU, network, memory layers.
Monitor p99 latency, not p50. Averages hide preemption spikes. Alert on p99 drift (10% over 1 hour) to catch fragmentation early.
Build fallbacks from day 1. Request queues with timeouts + rule-based responses. Graceful degradation is easier to build upfront than retrofit later.

What Actually Mattered¶

vLLM gave us the primitives (PagedAttention, continuous batching, prefix caching) V1 made them production-ready (automatic caching, multi-process server, unified scheduler).

The production lessons came from running it at scale for months. Memory fragmentation creeping up over days. Throughput cliffs at 200-300 concurrent. Quantization degrading rare entities. Speculative decoding backfiring on conversational traffic.

The optimizations that mattered most: - Percentile-based alerting (caught fragmentation 6-8 hours early) - Client-side load balancing (eliminated LB bottleneck, 30-second failover) - Graceful degradation (fallback rate <1% normally, 15-20% during incidents, kept customers served)

Honestly, the biggest lesson is production inference is a distributed systems problem, not just a model optimization problem. You need monitoring, load balancing, failover, graceful degradation. The model is one component in a system.

If you're building this, start with the system design (regional vs central, single vs multi-model, monitoring, fallbacks). Then optimize (quantization, offloading, chunked prefill, speculative decoding).

The system design decisions constrain what optimizations are even possible.

Multi-Mode AI Assistants with Telegram Forums

2026-02-02T16:57:00-06:00

I built something this morning that's been working surprisingly well.

I've been using openclaw for weeks now. It's an open-source AI assistant you run locally. Connects to Telegram, WhatsApp, Slack. Supports custom personas, local file access, skills. It's solid.

But I had one problem: everything routed to one agent. One context window.

Journaling, content writing, task management, accountability—all mixed together. I'd ask for help with an article, and the agent would get confused because journal entries were still in the conversation history. Every time I switched modes, I had to re-explain what I was doing. That friction added up.

The root cause: one session meant one context window. Everything mixed. Reflective journaling needs a different persona than blunt accountability. A general-purpose assistant is a jack-of-all-trades, master of none.

Openclaw already supports multi-agent routing. Multiple agents, isolated workspaces, separate session histories. I just needed a way to segment conversations. Telegram forum topics were the answer.

Caveat: This is day 1. I set this up this morning. I'm sharing what's working right now, not prescribing a universal solution. The real test is whether it holds up in 30 days.

The Setup: Using Openclaw's Multi-Agent Architecture¶

Openclaw has built-in support for multiple agents. You define agents in the config, each with its own workspace, persona files, and tool access. Then you create bindings that route messages to specific agents based on where they come from.

For Telegram, that means routing by group and forum topic.

Here's what I built:

Telegram Bot (@sohails_ghost_bot)
|
+-- Journal group (forum topics)
|   +-- Morning Journal → Journal agent
|   +-- Evening Reflection → Journal agent
|   +-- Free Write → Journal agent
|   +-- Weekly Review → Journal agent
|
+-- Content Forge group (forum topics)
    +-- Content Work → Content agent (Forge)
    +-- Accountability → Content agent (Forge)
    +-- Ideas → Content agent (Forge)

One bot. Three agents. Each forum topic maps to a specific agent and session.

Why Telegram Forums?¶

I looked at alternatives:

Separate Telegram chats?
Still pollutes the same agent workspace. Same context window. Same problem.

ChatGPT or Claude.ai?
No local files. No custom personas. No integration control. I need agents that can read my file system, run scripts, and follow specific instructions in persona files.

Slack or Discord?
Slack feels too corporate. Discord is less familiar. I wanted to stay in Telegram: clean, simple, fast.

The technical requirement: each agent needed its own persona files (SOUL.md, IDENTITY.md), its own workspace (file access, memory), and its own session history. Telegram forum topics give me built-in threading. One app, multiple agents, isolated conversations.

Design Decisions¶

One bot vs. multiple bots: - Multiple bots: Simpler config, but more tokens to manage, more processes - One bot: Single point of management, routing via bindings

I chose one bot. The routing complexity is worth the operational simplicity.

Per-topic system prompts:
Openclaw lets you configure per-topic system prompts on top of each agent's base persona. The Journal agent's "Morning Journal" topic gets: "Ask one forward-looking, energizing question. Keep it to 1-2 sentences." The Content agent's "Accountability" topic gets: "Be proactive. Check progress. No cheese, no platitudes."

Isolated workspaces:
Each agent has its own directory: - Journal agent: ~/clawd-journal/ - Content agent: ~/clawd-content/ - Main agent (Ghost): ~/clawd/

No shared files. No shared context. Session history lives in ~/.clawdbot/agents/<id>/sessions/, completely isolated.

Cron for proactive behavior:
Agents can't initiate conversations. To make the accountability agent proactive, I added a daily cron job (openclaw supports this) that fires at 9 AM, runs the content agent in an isolated session, checks blog-ideas.md for recent activity, and delivers a check-in to the Accountability topic.

The Three Agents¶

Journal Agent (📔)
Warm, Socratic, asks questions. Not a therapist. A thinking partner.

Workspace: ~/clawd-journal/
Tools: Restricted (read/write/exec only, no browser)
Topics: Morning Journal, Evening Reflection, Free Write, Weekly Review

Content Agent / Forge (🔥)
Blunt, high-standards, no-cheese accountability. Knows my writing voice. Tracks competitive pressure.

Workspace: ~/clawd-content/
Tools: Full access except canvas/discord/slack
Topics: Content Work, Accountability, Ideas

Main Agent / Ghost (default)
Sharp, analytical, task-oriented. Handles DMs, general groups, everything else.

Workspace: ~/clawd/
Tools: Full access

Session Isolation¶

Each topic gets its own session key:

agent:content:telegram:group:-1003787144697:topic:2

No cross-contamination. If I'm drafting in Content Work (topic 2) and switch to Ideas (topic 4), the Ideas topic starts fresh. It doesn't see my drafting conversation. By design.

Routing Mechanics¶

Telegram forum topics have a specific peer ID format: {chatId}:topic:{threadId}.
Example: -1003787144697:topic:3

Bindings in openclaw's config match this exact format. Most-specific match wins. If no binding matches, the default agent (Ghost) handles it.

Each binding:

{
  "agentId": "content",
  "match": {
    "channel": "telegram",
    "peer": { "kind": "group", "id": "-1003787144697:topic:2" }
  }
}

One binding per topic. No wildcards. Exact string comparison.

Configuration Steps¶

High-level overview. Full architecture doc with exhaustive details is here.

1. Create Telegram Forum Groups¶

Two groups: "Journal" and "Content Forge."

In each group: - Enable forum topics (group settings) - Add bot - Create topics (Morning Journal, Evening Reflection, Content Work, etc.)

2. Create Agent Workspaces¶

mkdir -p ~/clawd-journal
mkdir -p ~/clawd-content
mkdir -p ~/.clawdbot/agents/journal/agent
mkdir -p ~/.clawdbot/agents/journal/sessions
mkdir -p ~/.clawdbot/agents/content/agent
mkdir -p ~/.clawdbot/agents/content/sessions

The state directories must exist before routing works. Adding an agent to config isn't enough.

3. Write Persona Files¶

Each agent needs: - IDENTITY.md (name, emoji, vibe) - SOUL.md (philosophy, tone, anti-patterns) - AGENTS.md (tools, process, operating rules)

For the content agent, I copied my writing voice files (voice-writing.md, voice-speaking.md) so it knows how I write.

4. Configure Openclaw Bindings¶

Add agents to ~/.clawdbot/clawdbot.json under agents.list:

{
  "id": "journal",
  "workspace": "/Users/sohailmohammad/clawd-journal",
  "identity": { "name": "Journal", "emoji": "📔" },
  "tools": { "allow": ["read", "write", "exec"] }
}

Add bindings (one per topic). Add per-topic system prompts under channels.telegram.groups.{chatId}.topics.{threadId}.systemPrompt.

5. Discover Topic IDs¶

When I sent test messages to topics, they routed to Ghost (default agent) initially.

To find topic IDs: check ~/.clawdbot/agents/main/sessions/ for files ending in -topic-N.jsonl. Read those files to see which topic number maps to which topic name.

6. Add Cron Job (for Accountability)¶

Openclaw supports cron jobs for scheduled agent turns. I wanted the accountability agent to be proactive, so I added this to ~/.clawdbot/cron/jobs.json:

{
  "id": "content-accountability-daily",
  "agentId": "content",
  "schedule": { "kind": "cron", "expr": "0 15 * * *", "tz": "America/Chicago" },
  "sessionTarget": "isolated",
  "payload": {
    "kind": "agentTurn",
    "message": "Run accountability check. Read blog-ideas.md. Check if Sohail has been writing. Deliver to Accountability topic.",
    "deliver": true,
    "channel": "telegram",
    "to": "-1003787144697:topic:3"
  }
}

Runs daily at 9 AM CST. Executes in an isolated session (no history pollution). Delivers to the Accountability topic.

7. Restart Gateway¶

pkill -9 clawdbot-gateway

The macOS app auto-relaunches it.

What Worked¶

I used OpenCode to generate the config. It read my architecture notes and wrote the bindings. Setup was smooth. Routing worked on first try. No debugging.

Only manual step: creating the agent state directories.

Setup time: ~2 hours (most of it writing persona files).

How I Actually Use It¶

Morning:
Open Journal > Morning Journal topic. Brain dump in Neovim (or directly in Telegram). Neovim plugin auto-posts to the topic when I save the file. The journal agent reads it and asks one follow-up question. Feels like a Socratic session. Kickstarts the day.

Throughout the day:
When I have an idea, drop it in the Ideas topic. Wispr Flow makes this easy: dictate via voice, transcribes into Telegram. Agent logs it to blog-ideas.md. No mental overhead.

For content work (like drafting this article), I use the Content Work topic. Agent remembers where we left off. No need to re-explain context.

The accountability check-in:
At 9 AM, the cron job fires. Today it said: "You wrote yesterday and added some good ideas. Keep going today." That got me to start working on this article.

Context switching:
Switching topics feels like switching meetings with different departments. No cognitive cost. Each topic knows its own history.

Voice integration:
Wispr Flow for voice input. ElevenLabs for voice output (Telegram audio responses). Makes brainstorming effortless.

Mistakes so far:
Posted to the wrong topic a few times (Ideas instead of Content Work). Just muscle memory. Still building the mental map.

Early Impressions (Day 1)¶

What's Working¶

Better context retention.
Old setup: had to re-explain mode after switching. "No, I'm writing an article now, not journaling."
New setup: each topic picks up exactly where it left off. Zero clarification friction.

Easier idea capture.
I've always struggled with tracking ideas. This is simple: open Ideas topic, brain dump, it gets logged. Done.

Journaling feels fluid.
Morning journal this morning felt smooth. Follow-up questions were intuitive. Helped me dive deeper into things I wouldn't have explored on my own.

Content output.
This article exists because of the system. The accountability ping got me to start. Content Work topic kept extraction and outlining organized. +1 article directly attributable to this setup.

Tradeoffs¶

Telegram Lock-In vs. Flexibility: - Telegram-specific: Works great for personal use, but replicating this for teams means Slack or Discord - Pattern is portable: Same multi-agent architecture, different surface

I chose Telegram. For now, that works. If I need to extend this to Slack for work, openclaw supports it—same architecture, different channel.

Proactive Agents vs. Manual Triggers: - Cron jobs: Reliable, scheduled, no manual intervention - Manual triggers: More control, but requires remembering to check in

I chose cron jobs. The accountability ping this morning proved it works.

Custom Personas vs. General AI: - Custom SOUL.md: More setup, but precise tone control - General AI: Faster to start, less control

I chose custom personas. The journal agent's Socratic mode and content agent's blunt accountability are exactly what I need.

Key Takeaways¶

Context isolation improves response quality. Mixing modes in one session degrades relevance. Separate topics = better responses.
Openclaw's multi-agent architecture is powerful. It already supports everything you need: isolated workspaces, custom personas, per-topic routing. You just have to configure it.
Telegram forums are underrated for workflows. One app, one bot, multiple agents. Built-in threading. Clean UX.
Per-topic system prompts add precision. Base persona + topic-specific instructions = fine-grained behavior control.
Proactive agents need cron jobs. Agents can't initiate conversations. Openclaw's cron support + isolated sessions + delivery targeting = proactive check-ins.
Setup was smoother than expected. OpenCode helped generate config. Routing worked first try. ~2 hours total setup time.
Voice integration matters. Wispr Flow + ElevenLabs makes brain dumps effortless.
The pattern scales. Same architecture works for Slack, Discord, etc. Openclaw already supports those channels.
The real test is consistency. Day 1 is easy. Day 30 is the test. I'll revisit this in a month.

What's Next¶

Near-term:
Refine per-topic system prompts based on real usage. Maybe add another journal mode (weekly planning). Experiment with cross-agent context (accountability agent reads content work sessions for more informed check-ins).

Longer-term:
Replicate this for work. Slack channels for different teams or projects. Same architecture, different surface. Also planning to write up the enterprise version: CVS case study on multi-agent orchestration for pharmacy workflows.

If you want to try this:
Openclaw is open source: https://github.com/openclaw/openclaw

I'll publish a cleaned-up version of my full architecture doc (ARCHITECTURE-multi-agent.md) with step-by-step setup instructions.

Reach out if you build something similar.

Final Thoughts¶

I think the thing that surprised me most was how fast this clicked. I expected more debugging, more gotchas, more "why isn't this working?" moments.

It just worked. Routing worked. Context isolation worked. The personas felt right immediately.

Openclaw's architecture is well-designed. Multi-agent support is baked in. I just configured it cleverly.

Day 30 will tell me if this holds up. For now, it's working. That's enough to share.

Lessons from Scaling Enterprise RAG: Data Residency, Multi-Tenancy, and Production Reliability

2026-01-31T12:00:00-06:00

Disclaimer: This post reflects general patterns and lessons learned from building enterprise RAG systems. Technical details have been generalized, and no proprietary information from any specific organization is disclosed.

The Problem¶

The thing that surprised me most when working on enterprise data discovery was how much time analysts spent just finding data. Not analyzing it. Finding it.

The workflow is predictable in large organizations: an analyst needs specific data, knows it exists somewhere in the data warehouse, but which table? What do the columns mean? TX_AMT_CD doesn't exactly scream "transaction amount code."

So they search Confluence, Slack, internal wikis. Book office hours with data engineering. Wait days. Get pointed to someone else. Repeat.

By the time they find the right table and understand the schema, they've burned significant time. Across large analyst populations, that adds up fast. In environments I've worked in, estimates suggested 30-40% of analyst time was going to data discovery, not actual analysis.

The root cause is structural: hundreds of tables across multiple databases, inconsistent documentation, tribal knowledge trapped in people's heads, and compliance layers that make self-service nearly impossible.

The solution? RAG-based data discovery platforms.

Constraints That Shape Everything¶

I want to walk through common enterprise requirements first because they shape essentially every architectural decision.

Scale & Multi-Tenancy: Enterprise deployments often serve thousands of analysts across multiple business units. Users from one business unit typically can't see another unit's data, not just for privacy reasons, but regulatory compliance. This is non-negotiable in regulated industries.

Latency: A reasonable starting target is 10-15 seconds end-to-end, with room to optimize down to 5-10 seconds for most queries. Pure vector retrieval can hit sub-100ms. Complex agentic queries with multi-step reasoning and tool use take longer, these benefit from async orchestration so users can keep working while queries run in the background.

Strong Consistency Over Availability: This is the classic CAP theorem tradeoff. For financial or compliance-sensitive data, consistency wins. A compliance violation from stale embeddings is worse than temporary unavailability.

Data Residency: This is often the killer constraint in regulated industries. All data must stay within the organization's network. No third-party vendors, no data leaving your VPCs. SOC2 compliance, audit trails, CISO sign-off. Many teams underestimate how much this constraint eliminates from your solution space.

Audit Trails: SCD Type 2 (Slowly Changing Dimension) for everything is common in regulated environments. When metadata changes, you need to track who made the change, when, and what the previous value was. This serves compliance but also eases debugging, if retrieval quality suddenly drops, you need to know whether someone changed the underlying data.

Aggressive Timelines: Enterprise projects often face pressure to deliver POCs in weeks and production systems in months, while still navigating security reviews and compliance approvals.

Vector Database Evaluation¶

The Candidates¶

ChromaDB is fast to spin up and great for local development. But it's designed for local storage: cloud-backed infrastructure that scales across availability zones requires something else.

Pinecone is often the most tempting option. Managed infrastructure, excellent indexing performance, auto-scaling out of the box. But data residency requirements often kill it. You can't send financial metadata to a third-party vendor outside your network. Same issue with other managed services, even enterprise deployments may face CISO approval timelines that blow past deadlines.

Self-hosted options (Weaviate, Milvus, etc.) add complexity: new infrastructure to provision, new failure modes to understand, new security reviews to navigate.

That often leaves PGVector on managed Postgres (e.g., AWS RDS Aurora, Cloud SQL).

Why PGVector Often Wins in Enterprise¶

Once you lay out regulated-industry constraints, the decision often becomes obvious:

Data residency: Everything stays in your VPC
Team expertise: Most teams already have deep Postgres knowledge
Cloud-native: No new infrastructure to provision
Compliance path: Easier approval since security teams likely have existing Postgres runbooks
Migration flexibility: If you need a specialized vector DB later, you're not locked in

The real question is whether PGVector can meet latency requirements.

Indexing Strategy: HNSW vs. IVFFlat¶

HNSW (Hierarchical Navigable Small World) offers high recall with relatively low latency and supports incremental updates. But at scale:

Write amplification: Insertions trigger cascading graph modifications
Memory overhead: Graph traversal requires random access patterns, meaning the dataset essentially needs to be in memory

Quantization helps: - Product Quantization (PQ) can give 4-64x compression - Scalar Quantization (SQ) converts float32 to int8 for another 4x reduction

IVFFlat (Inverted File with Flat compression) often works better for production:

Simpler writes: Only updates the relevant posting list
Disk-based: Scales without prohibitive memory costs
Quantization-compatible: Works with scalar and binary quantization

The tradeoff is slightly lower recall compared to HNSW, but for metadata retrieval (not semantic similarity at extreme precision), the difference is often negligible.

Mitigations¶

Hybrid search complexity: PGVector handles vector similarity well, but keyword search (BM25) requires manual integration. Consider pg_search or similar extensions to add BM25 scoring alongside vector similarity.

Scaling past millions of vectors: PGVector's performance can degrade past ~5-10M vectors on a single instance. Sharding by business unit maps naturally to multi-tenancy requirements anyway.

Memory pressure: Even with quantization, large-scale vector search pressures memory. Aggressive index tuning (ef_search and lists parameters optimized per corpus size) helps.

Architecture Patterns¶

Typical Stack¶

Data Layer: - PGVector (IVFFlat + scalar quantization) on managed Postgres - Sharded by business unit - Standard embedding models (e.g., text-embedding-3-large)

Infrastructure: - Container orchestration (EKS, ECS, or similar) - Separated embedding service for background processing - Standard load balancing and API gateway patterns

Observability: - APM tooling (Datadog, Splunk, Dynatrace, etc.) - Distributed tracing (Honeycomb, Logfire, etc.) - Infrastructure as code, standard CI/CD pipelines

Multi-Tenancy Implementation¶

Each business unit can get its own database instance. When users authenticate, route queries to their unit's database based on auth metadata.

Implement defense in depth: application-level filtering AND row-level security in Postgres as a backstop.

For metadata editing, data owners can update descriptions through a separate UI. Changes get versioned using SCD Type 2. Only users in the appropriate business unit can edit their metadata.

Moving Beyond LangChain¶

LangChain is fast to prototype with, but teams often hit walls at scale:

Query inflexibility: similarity_search() is fine for simple queries. Complex filters, joins, and CTEs become painful.
Connection management: LangChain manages its own connections, which can conflict with existing pooling infrastructure.
Rigid table structure: LangChain expects a specific schema that may not fit audit trail and multi-tenancy requirements.
Naive inserts: add_documents() does row-by-row inserts. Bulk operations need COPY, upserts, and async processing.

The solution is often pure SQLAlchemy + Psycopg:

from sqlalchemy import select

def similarity_search(
    session: Session,
    query_embedding: list[float],
    user_id: UUID,
    min_similarity: float = 0.7,
    limit: int = 10
) -> list[tuple[Document, float]]:
    distance = Document.embedding.cosine_distance(query_embedding)

    stmt = (
        select(Document, (1 - distance).label('similarity'))
        .where(
            and_(
                1 - distance >= min_similarity,
                Document.user_id == user_id,
                Document.invalid == 0
            )
        )
        .order_by(distance)
        .limit(limit)
    )

    return session.execute(stmt).all()

For bulk inserts, Postgres COPY can deliver 10-50x throughput improvements over row-by-row inserts.

Zero-Downtime Re-Embedding¶

The problem: New data appears regularly. Re-embedding large corpora takes hours. You can't take the system down.

The solution: Atomic collection swaps via temporary tables.

Trigger: Scheduled jobs + event-driven triggers for priority updates
Process: Pull new metadata → Batch embed on separate service → Insert into temporary collection
Atomic swap: Single transaction: rename production → old, rename temp → production, drop old
Zero downtime: Users query the old table during the entire embedding process

@contextmanager
def atomic_collection_update(vector_store, collection_type):
    production_name = vector_store.collection_name
    temp_name = f"{production_name}_temp_{uuid.uuid4().hex[:8]}"

    temp_store = create_vector_store(temp_name)
    temp_store.create_collection()

    try:
        yield temp_store

        with temp_store._engine.begin() as conn:
            conn.execute(f"ALTER TABLE {production_name} RENAME TO {production_name}_old")
            conn.execute(f"ALTER TABLE {temp_name} RENAME TO {production_name}")
            conn.execute(f"DROP TABLE {production_name}_old")
    except Exception as e:
        temp_store.delete_collection()
        raise e

Production Lessons¶

Security¶

Jailbreak protection: DSPy-style iterative prompt hardening works well. In practice, prompt engineering is often sufficient: fine-tuning requires quality data pairs that most teams don't have.

Auth hardening: Defense in depth matters. JWT claim verification on every request, row-level security as a backstop, circuit breakers that fail closed on auth service timeouts.

Timeout Tuning¶

Each layer needs different tuning:

API Gateway: Default timeouts are often too short for complex queries
Load balancer idle timeout: Too short for WebSocket connections streaming LLM tokens
Database connection timeout: Balance keeping connections alive vs. pool exhaustion

There's no universal configuration. Tune based on production observations.

Scaling Patterns¶

Usage spikes by time of day and business unit are common. Memory is typically the critical bottleneck, which pushes toward IVFFlat + quantization.

Auto-scaling helps, but over-provisioning during known high-traffic windows is often worth the cost when analyst productivity is the metric.

Observability¶

Metrics that matter: - Query latency (p95 and p99—p50 hides real problems) - Token count and rate limiting - DB connection pool utilization - Error rates by tenant - Cache hit rate

Alerts that matter: - Consistent timeouts (not one-offs) - Latency exceeding thresholds for sustained periods - Connection pool exhaustion

Everything else is often noise.

What I'd Do Differently¶

Skip Fine-Tuning (Usually)¶

Fine-tuning sounds appealing. Custom model, theoretically better performance. In practice, effective prompting is often sufficient. Most teams don't have enough quality data pairs to make LoRA or full fine-tuning effective, and the marginal improvement rarely justifies the complexity cost.

Use LangGraph from the Start¶

Building custom agent orchestration works, but LangGraph provides better abstractions for stateful agent workflows without rigid class hierarchies.

Implement Tracing on Day 1¶

Adding observability reactively (after things break) is painful. LangSmith or Logfire from day one helps with: - Audit logs for understanding agent decisions - Debugging multi-step reasoning failures - Pattern recognition to reduce support load

Context Graphs as First-Class Citizens¶

SCD Type 2 audit trails are a primitive version of what the industry now calls context graphs. If building today, I'd make agent traces first-class: - Track every tool call - Log reasoning steps - Store confidence scores - Make it all queryable

Key Takeaways¶

Data residency kills most vendor solutions in regulated industries. Self-hosted is often the only realistic option.
Strong consistency over availability when wrong data is worse than no data.
Boring tech wins at scale. PGVector on managed Postgres isn't exciting, but it works within constraints.
Multi-tenancy is hard. Shard early. Defense in depth matters.
Prompt engineering over fine-tuning for most use cases.
Trust beats features. Ship something simple and reliable. Prove yourself. Then iterate.
Observability from day 1. You can't fix what you can't see.
Zero-downtime updates are non-negotiable. Atomic swaps prevent maintenance windows.

Final Thoughts¶

Enterprise RAG isn't about building the most cutting-edge system or using the fanciest vector database. It's about building reliable infrastructure that works within strict constraints: data residency, compliance, multi-tenancy, strong consistency.

The users don't care what vector database you used. They care whether they can find the right data in seconds instead of hours.

That's the real metric.

Sohail Mohammad is a Senior AI Engineer specializing in production ML infrastructure and agent systems. You can find him on LinkedIn and GitHub.