Jackbox: A 'budget' 84GB Inference Server | exoad
Building a local inference server in the DRAM Shortage
# Jackbox: A 'budget' 84GB Inference Server
> Building a local inference server in the DRAM Shortage
Published 2026-05-13 · LLM, Hardware, bibo, Local AI
Jackbox is a $1,700 local inference server I built to run large language models without cloud APIs or per-token pricing. It pairs an AMD Ryzen 5 7500X3D with an RTX 3080 20GB and a Tesla V100 32GB for 84GB of combined memory (52GB VRAM + 32GB RAM), and currently serves a 35B-parameter Qwen3.6 MoE model at 80-100 tokens/second with context windows up to 180K tokens — all for about $1.40/day in power.
The build prioritized VRAM per dollar over raw speed: a blower-cooled 3080 and a modded SXM2-to-PCIe V100, both picked up secondhand, paired with a motherboard that leaves room for a third GPU down the line. It runs Ubuntu 24.04 LTS headless, with llama.cpp splitting model layers across both GPUs by VRAM capacity.
Full text as markdown: /blog/jackbox-server-setup.md