The latest in local: Strata and TensorFold

What you'll see in this lesson

1. The problem: the model doesn't fit on the GPU

   GPU memory is fast but scarce; RAM and SSD are plentiful but much slower. The new tricks play with that.

2. Trick 1: the most-used experts on the GPU

   In an MoE model each token uses a few experts; if the most requested ones live on the GPU, the rest can wait in RAM.

3. Trick 2: speculative decoding

   A small helper guesses several tokens and the big model checks them all at once: if it's right, you move several steps for one.

4. Strata: a 125 billion MoE on a gaming PC

   Open project (MIT) that runs Qwen3.8-Flash-Next with a 12 GB GPU, 32–64 GB of RAM and about 80 GB of free SSD.

5. Strata: what's proven and what isn't

   It works and there are independent tests, but it's a 0.1 version from a single author, with one model and figures he measured.

6. TensorFold: faster without changing the answer

   Open engine (Apache 2.0) for Mac and NVIDIA based on speculative decoding that only accepts what the model would write anyway.

7. TensorFold: the fine print

   It shines with a single person; with many requests at once, a classic server like vLLM is far ahead.

8. Proven alternatives and how to test safely

   Before a weeks-long project, look at what already works in mature engines and test new things separately, with your own questions.

9. Summary

   What's new in local AI is keeping what each token reads at hand and guessing several tokens at once; test it before trusting it.