[{"data":1,"prerenderedAt":519},["ShallowReactive",2],{"navigation":3,"\u002Fblog\u002Fautocomplete-retired":62,"posts":406,"blog-list-\u002Fblog\u002Fautocomplete-retired":492},[4],{"title":5,"path":6,"stem":7,"children":8,"page":61},"Blog","\u002Fblog","blog",[9,13,17,21,25,29,33,37,41,45,49,53,57],{"title":10,"path":11,"stem":12},"My First Local AI Model Was an Autocomplete Server. Its Last Customer Was an AI Agent.","\u002Fblog\u002Fautocomplete-retired","blog\u002Fautocomplete-retired",{"title":14,"path":15,"stem":16},"I Gave an AI Agent the Keys to Navisworks, and Five Trades of Takeoff Took Four Days","\u002Fblog\u002Fautomated-bim-takeoff","blog\u002Fautomated-bim-takeoff",{"title":18,"path":19,"stem":20},"I Benchmarked Nine Models to Spec a Four-GPU Node. Every Single Number Crowned the Wrong Winner.","\u002Fblog\u002Fblackwell-bakeoff","blog\u002Fblackwell-bakeoff",{"title":22,"path":23,"stem":24},"On an EPC Job, 'Sure, No Problem' Is the Most Expensive Sentence Onsite. An Agent Would Have Checked the Contract First.","\u002Fblog\u002Fconstruction-agents","blog\u002Fconstruction-agents",{"title":26,"path":27,"stem":28},"I Run AI Agents Around the Clock and My Best Productivity Tool This Year Has No Battery","\u002Fblog\u002Fde-digitizing","blog\u002Fde-digitizing",{"title":30,"path":31,"stem":32},"I Built a Bot Army to Attack My Own Game Server, and 45 of Them Took It Down","\u002Fblog\u002Fgame-server-scaling","blog\u002Fgame-server-scaling",{"title":34,"path":35,"stem":36},"I Chased Four Benchmark Points and Almost Missed the Model That's Twice as Fast","\u002Fblog\u002Flocal-ai-moe-vs-dense","blog\u002Flocal-ai-moe-vs-dense",{"title":38,"path":39,"stem":40},"I Spent $5K on GPUs Just to Learn That One GPU Was Enough","\u002Fblog\u002Flocal-ai-rig","blog\u002Flocal-ai-rig",{"title":42,"path":43,"stem":44},"I Billed Myself for 6.7 Billion Tokens and the Total Came Out to $35.85","\u002Fblog\u002Flocal-ai-true-cost","blog\u002Flocal-ai-true-cost",{"title":46,"path":47,"stem":48},"I Spent a Saturday Trying to Replace My Local AI to Save on Tokens. The Server Voted No.","\u002Fblog\u002Flocal-ai-upgrade","blog\u002Flocal-ai-upgrade",{"title":50,"path":51,"stem":52},"Building a Roguelike Game with Amazon Q","\u002Fblog\u002Fq-roguelike","blog\u002Fq-roguelike",{"title":54,"path":55,"stem":56},"I Spent 22 Years Programming Just to Fail at Making a Skeleton Swing a Sword","\u002Fblog\u002Fthundoria-architecture","blog\u002Fthundoria-architecture",{"title":58,"path":59,"stem":60},"Build First - Learn Later","\u002Fblog\u002Fvectly-scaling","blog\u002Fvectly-scaling",false,{"id":63,"title":10,"author":64,"body":65,"date":387,"description":388,"draft":61,"extension":389,"image":390,"imageHeight":391,"imageWidth":392,"meta":393,"navigation":394,"path":11,"seo":395,"seoTitle":396,"sitemap":397,"stem":12,"tags":399,"updated":404,"__hash__":405},"blog\u002Fblog\u002Fautocomplete-retired.md","Tony Costanzo",{"type":66,"value":67,"toc":375},"minimark",[68,71,76,110,114,118,121,124,127,130,138,141,162,165,168,171,175,178,181,184,216,219,222,226,229,279,282,285,289,292,295,298,301,305,308,311,314,317,320,324,327,353,356,360,363,366,369,372],[69,70],"audio-player",{},[72,73,75],"h2",{"id":74},"links","Links",[77,78,79,89,96,103],"ul",{},[80,81,82],"li",{},[83,84,88],"a",{"href":85,"rel":86},"https:\u002F\u002Fwww.techhivelabs.net\u002Fblog\u002Flocal-ai-rig",[87],"nofollow","The original rig writeup",[80,90,91],{},[83,92,95],{"href":93,"rel":94},"https:\u002F\u002Fwww.techhivelabs.net\u002Fblog\u002Fblackwell-bakeoff",[87],"The nine-model Blackwell bake-off",[80,97,98],{},[83,99,102],{"href":100,"rel":101},"https:\u002F\u002Fcontinue.dev",[87],"Continue.dev",[80,104,105],{},[83,106,109],{"href":107,"rel":108},"https:\u002F\u002Fhuggingface.co\u002Fggml-org\u002FClef-Flash-GGUF",[87],"Clef-Flash GGUF on Hugging Face",[72,111,113],{"id":112},"eviction-notice","Eviction Notice",[115,116,117],"p",{},"I was halfway through wiring Jev into my workflows when Cloudflare went and ruined my week.",[115,119,120],{},"If you haven't run into it yet, Jev is TypeSafe's decision model. It never writes a word of text. You hand it some state and a set of questions with the answers defined up front (yes or no, pick one of these, score this from 1 to 5), and it hands back a typed answer with a probability and a confidence score, all in a single pass.",[115,122,123],{},"For AI agents that's a big deal. Half of what my agents do all day is make tiny calls like \"which project does this note belong to\" or \"is this request hard enough for the big model.\" Asking a chatty LLM to make those calls is slow, and occasionally creative in ways nobody asked for.",[115,125,126],{},"I had Jev deciding which model my LLM gateway routes each request to, and a whole list of jobs lined up for it in MyMind, my second-brain app. Then on October 1st Cloudflare open-sourced Clef. Two decision models, Apache 2.0, built on Qwen, sitting on Hugging Face.. and speaking almost exactly the same API as Jev.",[115,128,129],{},"A Jev alternative I could run at home. Private, free, no rate limits. Obviously I had to try it (have you met me?).",[115,131,132,133,137],{},"The catch was that the 9B Clef-Flash needs about 6.5GB of VRAM, and every card in my rig was already spoken for. So I started playing Tetris, scrolling the service list for a victim, and landed on ",[134,135,136],"code",{},"llama-autocomplete",". A little 7B coding model sitting on a 3090, waiting patiently for me to type something in VS Code.",[115,139,140],{},"Before I killed it I had Claude Code pull the usage numbers from Prometheus, because I wanted to make sure I wasn't about to break something I'd forgotten about. Thirty days of traffic came back:",[77,142,143,150,156],{},[80,144,145,149],{},[146,147,148],"strong",{},"Prompt tokens:"," 22",[80,151,152,155],{},[146,153,154],{},"Generated tokens:"," 33",[80,157,158,161],{},[146,159,160],{},"Peak concurrent requests:"," zero, at every single scrape",[115,163,164],{},"Twenty two tokens. That's roughly one sentence. In a month.",[115,166,167],{},"And it gets better (worse?). The lifetime counters on the container lined up exactly with the shape of a smoke test. The only traffic that server had served since its last rebuild was Claude, checking that the server worked.",[115,169,170],{},"So the model that got me into self-hosting AI spent its final days with exactly one customer, an AI agent making sure it was still alive. Then the agent shut it off.",[72,172,174],{"id":173},"the-gateway-drug","The Gateway Drug",[115,176,177],{},"Like most developers, my first taste of \"AI development\" was GitHub Copilot. Grey ghost text shows up, you hit Tab, and suddenly the boilerplate writes itself. It felt like cheating in the best possible way.",[115,179,180],{},"It also meant shipping every keystroke of client code to someone else's server, and paying a monthly bill for the privilege. Neither of those sat well with me. So at the start of this year I decided to host my own (how hard could it be?).",[115,182,183],{},"The first attempt was Tabby, which promptly refused to run because my CPU didn't support AVX-512. Strong opinions for an autocomplete engine. I ended up on Continue.dev in VS Code talking to my own model, and after a few weeks of shuffling it around, it found a home that I'm still weirdly proud of:",[77,185,186,192,198,204,210],{},[80,187,188,191],{},[146,189,190],{},"The box:"," a ZimaBoard, which is a single-board server about the size of a paperback",[80,193,194,197],{},[146,195,196],{},"The GPU:"," a Quadro P2000 hanging off it, 5GB of VRAM",[80,199,200,203],{},[146,201,202],{},"The power budget:"," a 60W power supply, so I had to underclock the card just to keep the board from browning out",[80,205,206,209],{},[146,207,208],{},"The model:"," a tiny Qwen2.5-Coder, served by llama.cpp",[80,211,212,215],{},[146,213,214],{},"The speed:"," around 45 tokens per second",[115,217,218],{},"A computer smaller than a hardcover book, running an underclocked workstation GPU, finishing my lines of code without a single packet leaving my house.",[115,220,221],{},"It felt like magic. Genuinely. I remember typing a function signature and watching the body fill in from a box on my shelf and just grinning at my monitor like an idiot.",[72,223,225],{"id":224},"then-everything-escalated","Then Everything Escalated",[115,227,228],{},"That little Zima setup was the first domino. Here's what the rig is serving today, about nine months later:",[77,230,231,237,243,249,255,261,267,273],{},[80,232,233,236],{},[146,234,235],{},"Main coder:"," Qwen3.8-Flash-Next on an RTX PRO 6000 Blackwell, 262K context, the brain behind most of my agents",[80,238,239,242],{},[146,240,241],{},"Vision:"," Qwen3-VL watching my security cameras and describing what it sees",[80,244,245,248],{},[146,246,247],{},"Decision model:"," Cloudflare's Clef-Flash, the local Jev replacement that kicked the autocomplete off its card",[80,250,251,254],{},[146,252,253],{},"Embeddings and reranking:"," the search layer behind MyMind, my second-brain app",[80,256,257,260],{},[146,258,259],{},"Speech to text:"," Whisper and Parakeet for anything I say out loud",[80,262,263,266],{},[146,264,265],{},"Text to speech:"," Breeze TTS 2 with a clone of my voice, which narrates these posts",[80,268,269,272],{},[146,270,271],{},"Image generation:"," Qwen-Image in ComfyUI, which makes most of my hero images",[80,274,275,278],{},[146,276,277],{},"Music generation:"," yes, really. MiniMax Music 3. That one gets its own post",[115,280,281],{},"It all runs on a rack of four 3090s plus the Blackwell, north of $20k in hardware, heating my office whether I like it or not.",[115,283,284],{},"I went from \"can I get grey text in VS Code without paying GitHub\" to running a small, deeply unprofitable AI lab in my house. In under a year. My wife has questions. So do I.",[72,286,288],{"id":287},"i-dont-really-write-code-anymore-sort-of","I Don't Really Write Code Anymore (Sort Of)",[115,290,291],{},"Here's the part that actually got me thinking.",[115,293,294],{},"The autocomplete server died because I stopped being in the editor.",[115,296,297],{},"These days I live in a terminal. Herdr splits my screen into panes, and almost every pane has a Claude Code session in it, each one chewing on a different task. One is fixing a bug in MyMind, one is poking at the homelab, one is writing.. well, this. I open VS Code for a quick tweak or to eyeball a diff, and then I close it again.",[115,299,300],{},"Autocomplete is built for a person typing code one line at a time. It guesses the next twenty characters and waits for you to hit Tab. I don't sit in that loop anymore. I describe the change, the agent makes it, I review it, and we argue about it (it has opinions, I have worse ones).",[72,302,304],{"id":303},"the-quiet-death-nobody-noticed","The Quiet Death Nobody Noticed",[115,306,307],{},"The best evidence I have for how hard this shift happened is that the autocomplete server already died once, and I didn't notice.",[115,309,310],{},"Back in September I pulled the P2000 out of the rig during a hardware shuffle and deleted its model the next day. For two days, Continue.dev was firing requests at a port with nothing listening on it. Every completion failed.",[115,312,313],{},"Not one complaint from me. I wasn't even in VS Code long enough to see the ghost text was missing.",[115,315,316],{},"Claude eventually tripped over it during a documentation audit. It rebuilt the whole service from scratch on an idle 3090, upgraded it to the 7B model, wired the monitoring back up, and tested that it actually completed code. Thorough work. Lovely work.",[115,318,319],{},"Then nobody used it for a month. Except Claude. Checking that it worked.",[72,321,323],{"id":322},"what-actually-changed","What Actually Changed",[115,325,326],{},"I've been programming for 22 years, and the last two moved more than the twenty before them. A few things stand out when I look back at that Zima board:",[77,328,329,335,341,347],{},[80,330,331,334],{},[146,332,333],{},"The unit of work got bigger."," Autocomplete helped me finish a line. Now I hand off a whole task and come back to a diff.",[80,336,337,340],{},[146,338,339],{},"My job shifted to reviewing."," I read far more code than I type, and catching a wrong assumption early matters more than typing fast ever did.",[80,342,343,346],{},[146,344,345],{},"Self-hosting followed the workflow."," The rig grew because each new way of working wanted a new model behind it. Voice, vision, search, all of it showed up because an agent needed it.",[80,348,349,352],{},[146,350,351],{},"The tool that started it all didn't scale with me."," It did exactly what it was built for, and I moved somewhere it couldn't follow.",[115,354,355],{},"I don't feel sad about it, honestly. The Zima autocomplete is the reason I learned what a GGUF is, what quantization does, and why you never trust a model server that only returns a 200. Every one of those lessons got used on the bigger rig.",[72,357,359],{"id":358},"rest-easy-little-guy","Rest Easy, Little Guy",[115,361,362],{},"The service is stopped and disabled. I kept the unit file and the model weights on disk, because I'm a hoarder and it's one command to bring it back.",[115,364,365],{},"I won't run that command, but it's nice to know I could.",[115,367,368],{},"For the record, the eviction paid off. I ran Clef-Flash head to head against Jev on the full banking77 intent benchmark, same requests byte for byte, and Clef-Flash scored 95.9% accuracy to Jev's 79.7%. Its confidence scores were more trustworthy too, which matters when an agent decides whether to act on its own or kick the call up to me. It's now running MyMind's decisions from the same 3090 the autocomplete used to live on (that bake-off deserves its own post).",[115,370,371],{},"Somewhere in a Prometheus database there's a flat line where my first local AI model used to be, broken up by 22 tokens of an AI agent asking it if it was okay.",[115,373,374],{},"It was. It just didn't have anyone left to talk to. (Next up: the music model, which has a lot more to say.)",{"title":376,"searchDepth":377,"depth":377,"links":378},"",2,[379,380,381,382,383,384,385,386],{"id":74,"depth":377,"text":75},{"id":112,"depth":377,"text":113},{"id":173,"depth":377,"text":174},{"id":224,"depth":377,"text":225},{"id":287,"depth":377,"text":288},{"id":303,"depth":377,"text":304},{"id":322,"depth":377,"text":323},{"id":358,"depth":377,"text":359},"2026-10-15T00:00:00.000Z","I evicted my self-hosted autocomplete model to make room for Cloudflare's Clef, a local Jev alternative, and found its only remaining user was an AI agent.","md","\u002Fimages\u002Fblog\u002Fautocomplete-retired\u002Fhero.webp",1075,1920,{},true,{"title":10,"description":388},"Retiring My Self-Hosted Copilot for AI Agents and Clef",{"loc":11,"lastmod":398},"2026-10-15",[400,401,402,403],"AI","Self Hosting","Developer Workflow","AI Agents",null,"wk4Ef3KC3FTYf8ZL7gLoaAiuLXWG585XNMUnzu5UCuA",[407,409,415,424,431,437,445,451,459,465,471,478,486],{"title":10,"author":64,"date":387,"draft":61,"description":388,"image":390,"tags":408,"navigation":394,"path":11,"stem":12,"id":63},[400,401,402,403],{"title":18,"author":64,"date":410,"draft":61,"description":411,"image":412,"tags":413,"navigation":394,"path":19,"stem":20,"id":414},"2026-09-14T00:00:00.000Z","I have to spec a two-to-four card inference node, so one Blackwell went on the bench first. Every throughput test picked one winner. Then a benchmark that actually compiles the code scored it 28.9% against 64.4%.","\u002Fimages\u002Fblog\u002Fblackwell-bakeoff\u002Fhero.webp",[400,401],"blog\u002Fblog\u002Fblackwell-bakeoff.md",{"title":14,"author":64,"date":416,"draft":61,"description":417,"image":418,"tags":419,"navigation":394,"path":15,"stem":16,"id":423},"2026-08-19T00:00:00.000Z","Automated BIM takeoff platforms assume your model followed the rules. Real models never do. So an AI agent drove Navisworks headless.. and five trades of takeoff took four days.","\u002Fimages\u002Fblog\u002Fautomated-bim-takeoff\u002Fhero.webp",[400,420,421,422],"Agents","Construction","BIM","blog\u002Fblog\u002Fautomated-bim-takeoff.md",{"title":30,"author":64,"date":425,"draft":61,"description":426,"image":427,"tags":428,"navigation":394,"path":31,"stem":32,"id":430},"2026-07-25T00:00:00.000Z","My movement bots said the server could hold hundreds. Then I taught them to swing a sword and the whole thing fell over at 45.. which is how I ended up shopping for CPUs instead of RAM.","\u002Fimages\u002Fblog\u002Fgame-server-scaling\u002Fhero.webp",[429,401],"Game Development","blog\u002Fblog\u002Fgame-server-scaling.md",{"title":42,"author":64,"date":432,"draft":61,"description":433,"image":434,"tags":435,"navigation":394,"path":43,"stem":44,"id":436},"2026-07-07T00:00:00.000Z","My friends said the new model on my rig was drunk. Root-causing that from LiteLLM request logs turned into pricing every token my GPUs have ever produced.. and a $961 cloud comparison.","\u002Fimages\u002Fblog\u002Flocal-ai-true-cost\u002Fhero.webp",[400,401],"blog\u002Fblog\u002Flocal-ai-true-cost.md",{"title":26,"author":64,"date":438,"draft":61,"description":439,"image":440,"tags":441,"navigation":394,"path":27,"stem":28,"id":444},"2026-06-17T00:00:00.000Z","I built a homelab full of AI agents to run my life, then got actually organized with a pen, an A5 notebook, and a bullet journal key I bent to fit my markdown brain.","\u002Fimages\u002Fblog\u002Fde-digitizing\u002Fhero.webp",[442,443],"Productivity","Journaling","blog\u002Fblog\u002Fde-digitizing.md",{"title":34,"author":64,"date":446,"draft":61,"description":447,"image":448,"tags":449,"navigation":394,"path":35,"stem":36,"id":450},"2026-06-08T00:00:00.000Z","I finally got Qwen3.6 running locally, ran the dense flagship against the sparse MoE, and learned that four benchmark points are worth a lot less than 5x prefill.","\u002Fimages\u002Fblog\u002Flocal-ai-moe-vs-dense\u002Fhero.webp",[400,401],"blog\u002Fblog\u002Flocal-ai-moe-vs-dense.md",{"title":22,"author":64,"date":452,"draft":61,"description":453,"image":454,"tags":455,"navigation":394,"path":23,"stem":24,"id":458},"2026-06-02T00:00:00.000Z","A client asks your superintendent for a small favor. He says yes. Months later it's a claim nobody can reconstruct. The case for AI agents on EPC jobs.","\u002Fimages\u002Fblog\u002Fconstruction-agents\u002Fhero.webp",[400,420,421,456,457],"EPC","Project Controls","blog\u002Fblog\u002Fconstruction-agents.md",{"title":46,"author":64,"date":460,"draft":61,"description":461,"image":462,"tags":463,"navigation":394,"path":47,"stem":48,"id":464},"2026-04-28T00:00:00.000Z","Six hours, five models, one IPMI reboot, zero upgrades -- a field report on why the bleeding edge of local LLM hosting is mostly bleeding.","\u002Fimages\u002Fblog\u002Flocal-ai-upgrade\u002Fhero.webp",[400,401],"blog\u002Fblog\u002Flocal-ai-upgrade.md",{"title":38,"author":64,"date":466,"draft":61,"description":467,"image":468,"tags":469,"navigation":394,"path":39,"stem":40,"id":470},"2026-03-25T00:00:00.000Z","Four RTX 3090s, an EPYC server, and a 122B parameter model -- the journey from underwhelming to overkill and back to surprisingly elegant.","\u002Fimages\u002Fblog\u002Fai-server\u002Fhero.webp",[400,401],"blog\u002Fblog\u002Flocal-ai-rig.md",{"title":54,"author":64,"date":472,"draft":61,"description":473,"image":474,"tags":475,"navigation":394,"path":55,"stem":56,"id":477},"2025-08-27T00:00:00.000Z","REST APIs to roguelikes: How I'm using a smaller game to learn the fundamentals before building my dream MMORPG. A honest indie gamedev journey.","\u002Fimages\u002Fblog\u002Fthundoria\u002Fhero.webp",[429,476],"Nuxt","blog\u002Fblog\u002Fthundoria-architecture.md",{"title":58,"author":64,"date":479,"draft":61,"description":480,"image":481,"tags":482,"navigation":394,"path":59,"stem":60,"id":485},"2025-06-25T00:00:00.000Z","Building an AI app with zero knowledge of AI, hitting deployment walls, and evolving architecture to meet real-world needs.","\u002Fimages\u002Fblog\u002Fvectly\u002Fhero.webp",[400,483,484,476],"Web Development","Side Projects","blog\u002Fblog\u002Fvectly-scaling.md",{"title":50,"author":64,"date":487,"draft":61,"description":488,"image":489,"tags":490,"navigation":394,"path":51,"stem":52,"id":491},"2025-06-17T00:00:00.000Z","Using Amazon's AI coding assistant, Q, to build a browser-based roguelike with Nuxt.js — the potential and the limits of AI-assisted development.","\u002Fimages\u002Fblog\u002Fq-roguelike\u002Fhero.webp",[400,429,476],"blog\u002Fblog\u002Fq-roguelike.md",[493,495,497,499,501,503,505,507,509,511,513,515,517],{"title":10,"path":11,"date":387,"tags":494,"image":390,"draft":61},[400,401,402,403],{"title":18,"path":19,"date":410,"tags":496,"image":412,"draft":61},[400,401],{"title":14,"path":15,"date":416,"tags":498,"image":418,"draft":61},[400,420,421,422],{"title":30,"path":31,"date":425,"tags":500,"image":427,"draft":61},[429,401],{"title":42,"path":43,"date":432,"tags":502,"image":434,"draft":61},[400,401],{"title":26,"path":27,"date":438,"tags":504,"image":440,"draft":61},[442,443],{"title":34,"path":35,"date":446,"tags":506,"image":448,"draft":61},[400,401],{"title":22,"path":23,"date":452,"tags":508,"image":454,"draft":61},[400,420,421,456,457],{"title":46,"path":47,"date":460,"tags":510,"image":462,"draft":61},[400,401],{"title":38,"path":39,"date":466,"tags":512,"image":468,"draft":61},[400,401],{"title":54,"path":55,"date":472,"tags":514,"image":474,"draft":61},[429,476],{"title":58,"path":59,"date":479,"tags":516,"image":481,"draft":61},[400,483,484,476],{"title":50,"path":51,"date":487,"tags":518,"image":489,"draft":61},[400,429,476],1792076493367]