[{"data":1,"prerenderedAt":568},["ShallowReactive",2],{"navigation":3,"\u002Fblog\u002Flocal-ai-true-cost":46,"posts":507},[4],{"title":5,"path":6,"stem":7,"children":8,"page":45},"Blog","\u002Fblog","blog",[9,13,17,21,25,29,33,37,41],{"title":10,"path":11,"stem":12},"On an EPC Job, 'Sure, No Problem' Is the Most Expensive Sentence Onsite. An Agent Would Have Checked the Contract First.","\u002Fblog\u002Fconstruction-agents","blog\u002Fconstruction-agents",{"title":14,"path":15,"stem":16},"I Run AI Agents Around the Clock and My Best Productivity Tool This Year Has No Battery","\u002Fblog\u002Fde-digitizing","blog\u002Fde-digitizing",{"title":18,"path":19,"stem":20},"I Chased Four Benchmark Points and Almost Missed the Model That's Twice as Fast","\u002Fblog\u002Flocal-ai-moe-vs-dense","blog\u002Flocal-ai-moe-vs-dense",{"title":22,"path":23,"stem":24},"I Spent $5K on GPUs Just to Learn That One GPU Was Enough","\u002Fblog\u002Flocal-ai-rig","blog\u002Flocal-ai-rig",{"title":26,"path":27,"stem":28},"I Billed Myself for 6.7 Billion Tokens and the Total Came Out to $35.85","\u002Fblog\u002Flocal-ai-true-cost","blog\u002Flocal-ai-true-cost",{"title":30,"path":31,"stem":32},"I Spent a Saturday Trying to Replace My Local AI to Save on Tokens. The Server Voted No.","\u002Fblog\u002Flocal-ai-upgrade","blog\u002Flocal-ai-upgrade",{"title":34,"path":35,"stem":36},"Building a Roguelike Game with Amazon Q","\u002Fblog\u002Fq-roguelike","blog\u002Fq-roguelike",{"title":38,"path":39,"stem":40},"I Spent 22 Years Programming Just to Fail at Making a Skeleton Swing a Sword","\u002Fblog\u002Fthundoria-architecture","blog\u002Fthundoria-architecture",{"title":42,"path":43,"stem":44},"Build First - Learn Later","\u002Fblog\u002Fvectly-scaling","blog\u002Fvectly-scaling",false,{"id":47,"title":26,"author":48,"body":49,"date":492,"description":493,"draft":45,"extension":494,"image":495,"meta":496,"navigation":497,"path":27,"seo":498,"seoTitle":499,"sitemap":500,"stem":28,"tags":502,"updated":505,"__hash__":506},"blog\u002Fblog\u002Flocal-ai-true-cost.md","Tony Costanzo",{"type":50,"value":51,"toc":479},"minimark",[52,57,94,98,102,109,115,118,136,143,147,150,158,174,177,180,200,203,207,210,232,239,243,246,249,272,275,314,317,324,328,334,341,348,355,358,362,369,375,378,404,410,414,417,460,463,467,470,473,476],[53,54,56],"h2",{"id":55},"links","Links",[58,59,60,67,72,80,87],"ul",{},[61,62,63],"li",{},[64,65,66],"a",{"href":23},"The original rig writeup",[61,68,69],{},[64,70,71],{"href":19},"The MoE vs dense showdown",[61,73,74],{},[64,75,79],{"href":76,"rel":77},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm",[78],"nofollow","vLLM",[61,81,82],{},[64,83,86],{"href":84,"rel":85},"https:\u002F\u002Fgithub.com\u002FBerriAI\u002Flitellm",[78],"LiteLLM",[61,88,89],{},[64,90,93],{"href":91,"rel":92},"https:\u002F\u002Fopenrouter.ai\u002Fqwen\u002Fqwen3.6-35b-a3b",[78],"OpenRouter's Qwen3.6-35B-A3B pricing",[53,95,97],{"id":96},"my-model-got-called-illiterate","My Model Got Called Illiterate",[99,100,101],"p",{},"My friends spent the weekend trash-talking the new coding model on my rig. By the time I finished defending its honor, I had priced every token my GPUs have ever produced and drafted a passive-aggressive invoice to the cloud.",[99,103,104,105,108],{},"Quick context for the new folks: I run a four-GPU homelab rig that serves coding models to a handful of people over ",[64,106,86],{"href":84,"rel":107},[78],". Last week I swapped the coder from Qwen3.6-35B-A3B to Ornith-1.0-35B, a DeepReinforce fine-tune built for agentic coding that benches a couple points higher on SWE-Bench (75.6 vs ~73.4). Getting it to boot took a vLLM upgrade and five sequential startup failures, which is a war story for another post.",[110,111],"content-image",{"alt":112,"caption":113,"src":114},"Open-frame GPU rig on a wire shelf: four RTX 3090s mounted on an aaawave mining frame with an EPYC server board and dual power supplies below","The defendant's residence. Four 3090s on a mining frame, and yes, the RGB is load-bearing.","\u002Fimages\u002Fblog\u002Flocal-ai-true-cost\u002Frig.webp",[99,116,117],{},"The point is, the community was raving about this model. Then my users started filing complaints:",[58,119,120,126,131],{},[61,121,122],{},[123,124,125],"em",{},"\"this model misspells shit a lot\"",[61,127,128],{},[123,129,130],{},"\"spent about 10 minutes deleting a file and recreating it, forgot what its original task was\"",[61,132,133],{},[123,134,135],{},"\"big habit of deleting the whole file it's working on and trying to rewrite it all even though it's just missing imports\"",[99,137,138,139,142],{},"So either every reviewer on the internet was wrong, or something about ",[123,140,141],{},"my"," serving setup was making a good model act like it had a concussion. I had a suspect. It was me.",[53,144,146],{"id":145},"subpoenaing-my-own-request-logs","Subpoenaing My Own Request Logs",[99,148,149],{},"Here's a thing I didn't fully appreciate until this week: LiteLLM stores the raw request body of every call in its spend logs. Every parameter every client ever sent. It's a full forensic record, sitting in Postgres, waiting for someone to have a bad enough week to read it.",[99,151,152,153,157],{},"I pulled every Ornith-era request for the heaviest user's key. 2,706 requests. 2,703 of them sent no ",[154,155,156],"code",{},"temperature"," at all.",[99,159,160,161,165,166,169,170,173],{},"When the client doesn't send a temperature, vLLM falls back to whatever the model's generation config says. This quant's default: ",[162,163,164],"strong",{},"1.0",". Combined with the ",[154,167,168],{},"top_p"," of 1.0 the client ",[123,171,172],{},"did"," send, my users were generating production code at near-maximum randomness. The misspellings weren't a model problem.. they were a dice-rolling problem.",[99,175,176],{},"Yikes.",[99,178,179],{},"And that was just the first charge. The logs kept talking:",[58,181,182,188,194],{},[61,183,184,187],{},[162,185,186],{},"The contexts were enormous."," Average prompt: 105,394 tokens. The biggest: 230,071, damn near the model's 262K ceiling. Roughly a third of all requests came in over 140K.",[61,189,190,193],{},[162,191,192],{},"My fp8 KV cache was quietly taxing those deep contexts."," Squeezing the cache to 8-bit is how I bought headroom, but at 100K+ tokens the precision loss is real. That's your \"forgot what its original task was.\"",[61,195,196,199],{},[162,197,198],{},"Thinking was disabled globally."," I'd turned it off for my other workloads. For a model whose entire party trick is reasoning through a plan before acting, that explains the \"rewrite the whole file to fix one import\" behavior.",[99,201,202],{},"The control group sealed it: completions in the 44K to 56K range were clean, articulate, multi-step debugging. The model was fine. The config was drunk.",[53,204,206],{"id":205},"the-fix-cost-nothing-my-favorite-price","The Fix Cost Nothing (My Favorite Price)",[99,208,209],{},"Two things I learned fixing this, and both were free:",[58,211,212,226],{},[61,213,214,217,218,221,222,225],{},[162,215,216],{},"Sampling params ride along per request."," One loaded model can behave completely differently per consumer. I made a second LiteLLM model entry, ",[154,219,220],{},"ornith-thinking",", pointing at the same backend with ",[154,223,224],{},"temperature: 0.2"," and thinking enabled baked in. The coding folks switch one config line. My other consumers (a security camera captioner, my knowledge base, a chat UI) keep their fast no-thinking behavior and never notice.",[61,227,228,231],{},[162,229,230],{},"This model's KV cache is comically small."," Ornith uses aggressive GQA with only 2 KV heads, so I checked whether I even needed fp8 anymore. Turns out fp16 still fits the entire 262K context with room for ~3 fully-maxed concurrent requests, and about 8 at my users' average context size. Peak concurrency ever observed on this box: 4. I was paying a precision tax to buy concurrency nobody was using.",[99,233,234,235,238],{},"So the deep-context degradation fix was literally ",[123,236,237],{},"removing"," a flag. I love it when the answer is deleting something.",[53,240,242],{"id":241},"the-rabbit-hole-under-the-spend-tables","The Rabbit Hole Under the Spend Tables",[99,244,245],{},"Now, while I was elbow-deep in LiteLLM's spend tables gathering evidence, something kept bugging me. Every local request had a spend of $0.00. Which is adorable, but wrong. These tokens aren't free.. I pay for them in electricity, and I had all the data to compute exactly how much.",[99,247,248],{},"My Prometheus setup already scrapes per-GPU power draw and vLLM throughput. The math is one line:",[58,250,251,257,260,266],{},[61,252,253,256],{},[162,254,255],{},"Energy per token"," = watts \u002F (tokens per second)",[61,258,259],{},"The two coder GPUs pull ~700W under load (power-capped 3090s)",[61,261,262,263],{},"Decode runs ~130 tok\u002Fs, so each output token costs ",[162,264,265],{},"5.4 joules",[61,267,268,269],{},"Prefill chews ~6,000 tok\u002Fs, so each input token costs ",[162,270,271],{},"0.12 joules",[99,273,274],{},"At my rate of $0.1395\u002FkWh, that lands here:",[276,277,278,290],"table",{},[279,280,281],"thead",{},[282,283,284,287],"tr",{},[285,286],"th",{},[285,288,289],{},"$ per 1M tokens",[291,292,293,304],"tbody",{},[282,294,295,299],{},[296,297,298],"td",{},"Input (prefill)",[296,300,301],{},[162,302,303],{},"$0.0045",[282,305,306,309],{},[296,307,308],{},"Output (decode)",[296,310,311],{},[162,312,313],{},"$0.21",[99,315,316],{},"Prefill is ~46x cheaper per token than decode because it batches, which matters enormously when your workload is agents stuffing entire repos into every prompt.",[99,318,319,320,323],{},"Honesty corner, because I hate cost posts that hide the asterisks: this is ",[123,321,322],{},"marginal"," cost. It excludes idle power, cooling, and hardware amortization. And the prefill throughput is estimated, not measured, so the input rate is the softest number in this post. Sanity check though: it prices my last 7 days of usage at about $6.90, call it $30\u002Fmonth of coder electricity, which matches what the rig actually pulls. The math holds.",[53,325,327],{"id":326},"the-part-where-the-cloud-should-blush","The Part Where the Cloud Should Blush",[99,329,330,331],{},"I set those per-token prices on the local models and backfilled every request LiteLLM has ever logged. All-time damage across every local coding model since April: ",[162,332,333],{},"6.67 billion input tokens, 27.8 million output tokens, $35.85 in electricity.",[99,335,336,337,340],{},"Then I priced the identical token volume at OpenRouter's rate for Qwen3.6-35B-A3B, the exact architecture Ornith is fine-tuned from, at $0.14\u002FM in and $1.00\u002FM out. Same tokens, their bill: ",[162,338,339],{},"~$961",".",[99,342,343],{},[344,345],"img",{"alt":346,"src":347},"Bar chart comparing the all-time cost of 6.7 billion tokens: $35.85 of local electricity vs an estimated $961 at OpenRouter rates, roughly 27 times more","\u002Fimages\u002Fblog\u002Flocal-ai-true-cost\u002Fbill-vs-cloud.svg",[99,349,350,351,354],{},"That's ~27x cheaper on marginal cost, and OpenRouter's open-model pricing is ",[123,352,353],{},"already"," the cheap option. Input tokens are the absurd part: 31x cheaper locally, on the exact token type my agentic workload mainlines.",[99,356,357],{},"Before anyone yells at me (again), yes, the comparison is electricity vs fully-loaded cloud. So let's load it: the rig cost about $6k, which amortizes to ~$167\u002Fmonth over three years. The coder workload alone has been producing $320 to $480\u002Fmonth of cloud-equivalent value. It pays the rig's mortgage two or three times over before counting anything else the box does (image gen, voice, embeddings.. you name it).",[53,359,361],{"id":360},"where-67-billion-tokens-actually-went","Where 6.7 Billion Tokens Actually Went",[99,363,364,365,368],{},"Backfilling spend meant I could finally answer a question I'd never thought to ask: what is actually ",[123,366,367],{},"eating"," my rig? I mapped every API key to its workload and grouped all-time usage.",[99,370,371],{},[344,372],{"alt":373,"src":374},"Bar chart of all-time input tokens by workload: the always-on assistant used 4.75 billion (71.4%), agentic coding 1.88 billion (28.2%), vision and security 15.6 million (0.23%), and the second brain 13.6 million (0.20%)","\u002Fimages\u002Fblog\u002Flocal-ai-true-cost\u002Ftoken-attribution.svg",[99,376,377],{},"The results broke my mental model of my own homelab:",[58,379,380,386,392,398],{},[61,381,382,385],{},[162,383,384],{},"The always-on assistant dominates everything."," Hermes, my 24\u002F7 messaging agent, has quietly pushed 4.75 billion tokens.. 71% of all local inference ever. Total electricity: about $26. A billion tokens a month of ambient assistant costs less than a pizza.",[61,387,388,391],{},[162,389,390],{},"Coding is the heavyweight per request, not per volume."," 28% of the tokens from just 12% of the requests, averaging ~97K tokens per request. Agentic coding tools shove the whole repo into every single turn.",[61,393,394,397],{},[162,395,396],{},"The \"second brain\" is a rounding error."," My knowledge base and memory agents, the thing people assume is the token hog, used 13.6 million tokens all-time. Twenty-two cents. The chat agent out-consumed it 350x, purely by never shutting up.",[61,399,400,403],{},[162,401,402],{},"Vision is cheap and bursty."," The security cameras fire thousands of tiny ~1.9K-token requests. 5% of requests, 0.23% of tokens.",[99,405,406],{},[344,407],{"alt":408,"src":409},"Grouped bar chart comparing each workload's share of requests vs share of input tokens: the assistant is 82% of requests and 71% of tokens, coding is 12% of requests but 28% of tokens, vision is 5% of requests and 0.2% of tokens, second brain is 1.5% of requests and 0.2% of tokens","\u002Fimages\u002Fblog\u002Flocal-ai-true-cost\u002Frequests-vs-tokens.svg",[53,411,413],{"id":412},"three-traps-i-sprung-so-you-dont-have-to","Three Traps I Sprung So You Don't Have To",[99,415,416],{},"If you want to do this backfill on your own LiteLLM box, learn from my bruises:",[58,418,419,440,450],{},[61,420,421,427,428,431,432,435,436,439],{},[162,422,423,426],{},[154,424,425],{},"\u002Fmodel\u002Fupdate"," silently drops custom cost fields."," ",[154,429,430],{},"input_cost_per_token"," and ",[154,433,434],{},"output_cost_per_token"," only stick if you delete the model and recreate it via ",[154,437,438],{},"\u002Fmodel\u002Fnew",". No error, no warning. The API just quietly disagrees with you.",[61,441,442,445,446,449],{},[162,443,444],{},"Spend lives in three places, and the UI reads the one you'd guess last."," Per-request spend logs, per-key running totals, and daily aggregate tables. The usage dashboard reads the ",[123,447,448],{},"daily"," tables, so backfilling only the request logs changes nothing on screen. Ask me how long that took me to figure out.",[61,451,452,455,456,459],{},[162,453,454],{},"Reconcile key totals additively, never from scratch."," Old spend logs get pruned while key counters persist. Two of my keys held $42 of real recorded spend with zero surviving log rows. A naive ",[154,457,458],{},"spend = sum(logs)"," would have erased history, and bumping totals can also trip keys with budgets, so guard for that too.",[99,461,462],{},"LiteLLM has strong opinions about its spend tables. Now I have strong opinions about its opinions.",[53,464,466],{"id":465},"case-closed","Case Closed",[99,468,469],{},"So the model was never drunk. My config was rolling dice at temperature 1.0, compressing memories at 200K context, and lobotomizing a reasoning model's ability to reason. Ornith is owed a formal apology from at least three people, and I've sent mine.",[99,471,472],{},"The real souvenir is that number though. Every token my coding stack has ever produced, 6.7 billion of them, cost me less than a family dinner. The cloud wanted a grand for the same work.",[99,474,475],{},"My friends are back to happily burning 97K tokens per request like it's nothing. Because at $0.0045 per million, it basically is.",[99,477,478],{},"Go bill yourself. It's weirdly satisfying.",{"title":480,"searchDepth":481,"depth":481,"links":482},"",2,[483,484,485,486,487,488,489,490,491],{"id":55,"depth":481,"text":56},{"id":96,"depth":481,"text":97},{"id":145,"depth":481,"text":146},{"id":205,"depth":481,"text":206},{"id":241,"depth":481,"text":242},{"id":326,"depth":481,"text":327},{"id":360,"depth":481,"text":361},{"id":412,"depth":481,"text":413},{"id":465,"depth":481,"text":466},"2026-07-07T00:00:00.000Z","My friends said the new model on my rig was drunk. Root-causing that from LiteLLM request logs turned into pricing every token my GPUs have ever produced.. and a $961 cloud comparison.","md","\u002Fimages\u002Fblog\u002Flocal-ai-true-cost\u002Fhero.webp",{},true,{"title":26,"description":493},"Local LLM Cost Analysis: Electricity per Token vs the Cloud",{"loc":27,"lastmod":501},"2026-07-07",[503,504],"AI","Self Hosting",null,"cBqjzZvNz5njcBFV5Tm2DZ6qzKPGQbxuZ1nfC6CsIFk",[508,510,518,524,534,540,546,554,562],{"title":26,"author":48,"date":492,"draft":45,"description":493,"image":495,"tags":509,"navigation":497,"path":27,"stem":28,"id":47},[503,504],{"title":14,"author":48,"date":511,"draft":45,"description":512,"image":513,"tags":514,"navigation":497,"path":15,"stem":16,"id":517},"2026-06-17T00:00:00.000Z","I built a homelab full of AI agents to run my life, then got actually organized with a pen, an A5 notebook, and a bullet journal key I bent to fit my markdown brain.","\u002Fimages\u002Fblog\u002Fde-digitizing\u002Fhero.webp",[515,516],"Productivity","Journaling","blog\u002Fblog\u002Fde-digitizing.md",{"title":18,"author":48,"date":519,"draft":45,"description":520,"image":521,"tags":522,"navigation":497,"path":19,"stem":20,"id":523},"2026-06-08T00:00:00.000Z","I finally got Qwen3.6 running locally, ran the dense flagship against the sparse MoE, and learned that four benchmark points are worth a lot less than 5x prefill.","\u002Fimages\u002Fblog\u002Flocal-ai-moe-vs-dense\u002Fhero.webp",[503,504],"blog\u002Fblog\u002Flocal-ai-moe-vs-dense.md",{"title":10,"author":48,"date":525,"draft":45,"description":526,"image":527,"tags":528,"navigation":497,"path":11,"stem":12,"id":533},"2026-06-02T00:00:00.000Z","A client asks your superintendent for a small favor. He says yes. Months later it's a claim nobody can reconstruct. The case for AI agents on EPC jobs.","\u002Fimages\u002Fblog\u002Fconstruction-agents\u002Fhero.webp",[503,529,530,531,532],"Agents","Construction","EPC","Project Controls","blog\u002Fblog\u002Fconstruction-agents.md",{"title":30,"author":48,"date":535,"draft":45,"description":536,"image":537,"tags":538,"navigation":497,"path":31,"stem":32,"id":539},"2026-04-28T00:00:00.000Z","Six hours, five models, one IPMI reboot, zero upgrades -- a field report on why the bleeding edge of local LLM hosting is mostly bleeding.","\u002Fimages\u002Fblog\u002Flocal-ai-upgrade\u002Fhero.webp",[503,504],"blog\u002Fblog\u002Flocal-ai-upgrade.md",{"title":22,"author":48,"date":541,"draft":45,"description":542,"image":543,"tags":544,"navigation":497,"path":23,"stem":24,"id":545},"2026-03-25T00:00:00.000Z","Four RTX 3090s, an EPYC server, and a 122B parameter model -- the journey from underwhelming to overkill and back to surprisingly elegant.","\u002Fimages\u002Fblog\u002Fai-server\u002Fhero.webp",[503,504],"blog\u002Fblog\u002Flocal-ai-rig.md",{"title":38,"author":48,"date":547,"draft":45,"description":548,"image":549,"tags":550,"navigation":497,"path":39,"stem":40,"id":553},"2025-08-27T00:00:00.000Z","REST APIs to roguelikes: How I'm using a smaller game to learn the fundamentals before building my dream MMORPG. A honest indie gamedev journey.","\u002Fimages\u002Fblog\u002Fthundoria\u002Fhero.webp",[551,552],"Game Development","Nuxt","blog\u002Fblog\u002Fthundoria-architecture.md",{"title":42,"author":48,"date":555,"draft":45,"description":556,"image":557,"tags":558,"navigation":497,"path":43,"stem":44,"id":561},"2025-06-25T00:00:00.000Z","Building an AI app with zero knowledge of AI, hitting deployment walls, and evolving architecture to meet real-world needs.","\u002Fimages\u002Fblog\u002Fvectly\u002Fhero.webp",[503,559,560,552],"Web Development","Side Projects","blog\u002Fblog\u002Fvectly-scaling.md",{"title":34,"author":48,"date":563,"draft":45,"description":564,"image":565,"tags":566,"navigation":497,"path":35,"stem":36,"id":567},"2025-06-17T00:00:00.000Z","Using Amazon's AI coding assistant, Q, to build a browser-based roguelike with Nuxt.js — the potential and the limits of AI-assisted development.","\u002Fimages\u002Fblog\u002Fq-roguelike\u002Fhero.webp",[503,551,552],"blog\u002Fblog\u002Fq-roguelike.md",1783457673299]