vLLM 0.28 — 3.5x faster on ARM, cheaper to run

The vLLM team dropped version 0.28.0, and the headline is speed. On ARM chips — M-series MacBooks, Graviton instances, Raspberry Pis — inference is now 3.5 times faster than the previous release. For OpenAI's Codex CLI, that translates to noticeably shorter waits for completions, especially on machines that don't have a big GPU sitting on the desk.

The release also bundles better support for the newer 4-bit quantization formats (AWQ and GPTQ). Quantization is how you fit a large model into less memory by rounding weights down to 4 bits. The trade-off is a small hit to quality, but the savings are real: models that previously needed 80 GB of VRAM can now run on cards with 16 or even 8 GB. That's the difference between a model you can't afford and one that runs on the laptop you already own.

This is a community-maintained project — not an OpenAI product — and it's the engine behind a lot of the self-hosted LLM stacks running in garages and small shops. The ARM optimization is the first major win for non-x86 silicon in this space, which matters because a lot of Brown folks are building on ARM hardware for reasons of price, availability, or just what's sitting in the drawer. The quantization support means even older GPUs can serve models that used to require enterprise cards. If you're running a small business, a side hustle, or just a private model at home, 0.28 is the release to grab.

Why this matters for us: the models are finally getting fast enough to run on the cheap ARM hardware most of us actually use, so we don't have to rent expensive cloud GPUs to get work done.