The open-source layer that meters every model call, routes the cheap/local model first, and escalates only when a real check requires it.
Sardor Razikov · Apache-2.0 · runs offline
Every token has a price, and agentic workloads send the bill soaring. Most calls don't need the biggest model — but nothing decides that for you, and nothing proves the cheap answer was good enough.
Know the exact $/token of every call. Thread-safe hard budget cap.
Cheap/local model first; escalate only when confidence is low.
Offline eval asserts + an LLM judge gate quality in CI.
A repeat or near-repeat prompt costs $0 — the second lever.
Confidence is pluggable — a near-free logprob signal, a one-call verifier, or self-consistency. And Frugal warns you when a pair is too close in price to bother.
LLM-judged, on a rented node. Reproduce with one command.
| Cheap → frontier | Confidence | Saved |
|---|---|---|
| GPT-4o-mini → GPT-4o (16×) | logprob (free) | 90.6% |
| Local ($0) → GPT-4o | any | 88–97% |
| Claude Haiku → Sonnet (3×) | self-consistency | −7% |
Live on a rented node: vs a plain proxy (all→strong), Frugal made half the strong-model calls — −53.9% cost, −32.5% latency on the same prompts.
~75–97% typical — the −7% row is honest: it can lose money.
| Who's burning it | The 2026 fact |
|---|---|
| Meta | AI capex $125–145B; ~$25B/qtr on inference silicon |
| Amazon | ~$200B AI capex; Bedrock hosts Anthropic + Grok |
| Gemini serves 1.3 quadrillion tokens / month | |
| Microsoft | GitHub Copilot → token-metered billing ("meter shock") |
| xAI | Anthropic pays $1.25B/mo for Colossus; >1M calls/day |
| Palantir | CEO: token pricing "broken" → air-gapped, no hosted API |
| Salesforce | Agentforce moved to resolution-based pricing |
| Uber | Burned its whole 2026 AI-coding budget by April |
| Klarna | AI assistant: 2.3M conversations / month |
| Fortune-500 firm | $500M on Claude in a single month, no caps |
| Industry | Inference = 85% of the AI budget; spend still doubling |
Token prices fell ~280× in two years, yet total spend rose ~320% — agentic volume exploded. Cheaper tokens don't save you; routing what doesn't need the big model does.
| Monthly inference spend | Frugal saves / year |
|---|---|
| $100,000 | $0.6–0.9M |
| $1,000,000 | $6–9M |
| $10,000,000 | $60–90M |
| $50,000,000 (F-500 tier) | $300–450M |
Hard budget cap + reserve() → budget can't be blown silently; routing stretches it 2–4×.
Local-first routing keeps private work on your GPUs (0 leaks); escalate only when truly needed.
2.3M chats/mo → cheap tier for simple turns; ~$210k/yr modeled saving on that volume alone.
Modeled at published 2026 API prices and a conservative routable share; your mix determines the result. Reproduce: python benchmarks/cost_model.py. Not affiliated with any company named.
Cascade + a re-sampling check loses money when the cheap tier isn't much cheaper. The rule: cascade only at a ≥ ~10× price gap, or use a free logprob signal, or a local tier. Frugal computes this and warns you — we publish where it doesn't work, not just where it does.
Cut 75–97% of inference spend; hard budget caps per tenant.
Private prompts stay on-prem (0 leaks); audit every $/token.
Agents that self-throttle: trivial steps cheap, hard ones escalate.
Reproducible model-vs-model evals with an LLM judge on your cluster.
OpenAI-compatible budget gateway + MCP cost telemetry.
One command, no keys, cheap by default.
cost, budget, zero-overshoot reserve
cascade + logprob/verifier confidence
$0 on repeat prompts
local↔cloud, 0 private leaks
asserts, drift, LLM judge
retrieval / faithfulness checks
agent reads its own spend
OpenAI-compatible, streaming
won't-save warning
logprob confidence · response cache · streaming gateway · economics guard · zero-overshoot budget.
broader human-graded evals · live LiteLLM/Portkey bake-off · more provider adapters · a persistent metrics store.
Frugal and REPOMIND are one on-prem, cost-efficient-inference thesis, built in the open. I'm open to the backing that scales it:
Fund compute, evals, and full-time work on the stack.
Research / OSS / hardware grants — no equity, no repayment.
GPU / accelerator access (AMD ROCm-first) for public benchmarks.
Team + IP — the Wang / Suleyman / Shazeer pattern. I'd bring Frugal + REPOMIND.