Your AI agents are
overpaying. Route around it.
Frugal meters every model call, sends the cheap or local model first, and escalates to a strong model only when a real confidence check requires it — then proves the quality in CI. Fully offline, no API keys.
A 3B model matched a 14B on most tasks.
Graded by an LLM judge on a rented node. Small model, big model, honest verdict.
What it saves, by scenario.
Savings vs sending everything to the frontier model. 500 in / 300 out tokens; 17% of hard prompts escalate (measured). Edit the inputs and re-run cost_model.py.
Live on a rented node — vs a plain proxy that sends every call to the strong model, Frugal made half the strong-model calls: −53.9% cost and −32.5% latency on the same 12 prompts. (Cloud-price run needs your keys.)
The tool tells you when to say no.
Cascade plus a re-sampling confidence signal loses money when the cheap tier isn't much cheaper than the frontier — the probing eats the margin (the −7% row above).
The rule from the math: cascade only when the cheap tier is ≥ ~10× cheaper,
or use a near-free logprob signal, or a local ($0/token) tier. Frugal computes this and
warns instead of quietly overcharging.
The token-tax is real money now.
Inference is now ~85% of the enterprise AI budget. Token prices fell ~280× in two years, yet total spend rose ~320% — agentic volume exploded (10–20 calls per task, always-on agents). Cheaper tokens don't save you; routing what doesn't need the big model does. Frugal is that control layer.
AI capex $125–145B; ~$25B/qtr on short-lived inference silicon.
~$200B AI capex 2026; Bedrock hosts Anthropic + Grok inference.
Gemini serving 1.3 quadrillion tokens / month.
GitHub Copilot moved to token-metered billing — the June "meter shock".
Anthropic pays xAI $1.25B/month for Colossus compute; >1M API calls/day.
CEO: token pricing "completely broken" → air-gapped, no-hosted-API AI.
OCI Enterprise AI; hosts Palantir Foundry / AIP inference.
Token costs forced Agentforce onto resolution-based pricing.
Burned its entire 2026 AI-coding budget by April (usage 32% → 84%).
AI assistant: 2.3M conversations/month (~work of 850 agents).
$500M on Claude in a single month — no spending caps.
Enterprise LLM spend doubling; inference = 85% of the AI budget.
"Uber burned its budget by April"
Thread-safe hard budget cap + reserve() (refuse before spending) → the budget can't be blown silently; routing stretches it 2–4×.
"$500M on Claude, no caps"
Exactly what the budget gateway prevents — every key/tenant gets a cap and a live $/token view via the MCP server.
Palantir's air-gapped push
Local-first routing keeps private/simple work on your own GPUs (0 leaks, tested); escalate to a hosted frontier only when truly needed.
Klarna-scale support
2.3M chats/mo ≈ 6.9M calls — most turns simple → cheap tier. Modeled saving ≈ $210k/yr on that volume alone.
Savings are modeled at published 2026 API prices and a conservative routable-share assumption (we measure 75–91% on the cheap-tier fraction); your actual mix determines the result — reproduce with python benchmarks/cost_model.py. Company facts are cited public reports; Frugal is not affiliated with, or endorsed by, any company named.
If you pay for tokens, this is for you.
Indie devs & vibecoders
Ship AI side-projects without a scary bill. Cheap/local by default; one command, no keys.
Startups
Extend runway — cut 75–97% of inference spend and put a hard budget cap on every tenant.
Enterprises
Keep private prompts on-prem (0 leaks, tested), gate quality in CI, audit every $/token.
AI-agent builders
Agents that self-throttle: route the trivial steps cheap, escalate only the hard ones.
Researchers & labs
Reproducible, offline, model-vs-model benchmarking with an LLM judge on your own cluster.
Platform & DevOps teams
An OpenAI-compatible budget gateway + MCP cost telemetry, thread-safe and memory-bounded.
Everything reads one shared cost meter.
Cost, tokens, latency; thread-safe hard budget cap; bounded memory; zero-overshoot reserve.
Cascade: cheap first, escalate only when a confidence check fails.
The second cost lever — a repeat or near-repeat prompt costs $0.
Route local ↔ cloud by cost, privacy, complexity. 0 private leaks.
Offline semantic asserts, drift, and an LLM judge for CI.
Retrieval hit-rate, faithfulness, citation coverage.
An MCP server exposing an agent's own $/token spend.
OpenAI-compatible proxy: meter, budget, route.
Tells you up front whether a pair will actually save.
Reproduce every claim in ten seconds.
No keys. It runs on a deterministic mock provider; swap in your own models any time.
# prove it yourself pip install -e . && frugal demo # end-to-end, offline python benchmarks/run_all.py # the benchmark table python benchmarks/stress_test.py # thread-safety, ReDoS, fuzz, memory python benchmarks/cost_model.py # your real-price savings → 56 tests · 7 bugs found & fixed by our own pressure, fuzz & concurrency suites
Use it, back it, build on it.
Frugal is open-source and independently built. Every star, fork and issue moves it forward.
◆ Developers
Star it, use it, fork it, share it. Open an issue, send a PR, tell a teammate. Contributions and feedback are the fuel.
◆ Companies & investors
Open to angel investment, non-refundable grants, partnership, and acquisition / acquihire — the Wang / Suleyman / Shazeer pattern (team + IP). Frugal and REPOMIND are one on-prem, cost-efficient-inference thesis — I'd bring both.
◆ Hardware & grants
Built and benchmarked solo on rented, modest hardware — that's the ceiling. GPU / accelerator sponsorship (AMD ROCm-first) or a non-refundable grant takes the benchmarks broad, human-graded and public.
Sardor Razikov
Sardor Razikov — independent AI/ML engineer
Builder from Tashkent, Uzbekistan. Works on on-prem LLM infrastructure, cost-efficient inference, and compression. Frugal productizes the cost-routing layer from his REPOMIND project. Open to acquisition, investment, partnership, and hardware sponsorship.