Clusters: link several machines for bigger models

What you'll see in this lesson

1. Why connect several machines

   A model only runs well if it fits entirely in memory; connecting machines adds up that memory to run models that don't fit in one.

2. Splitting layers: adds memory, not speed

   In layer parallelism, or pipeline, each machine holds one section of the model and they work in turns, one after another.

3. Splitting tensors: also adds speed

   In tensor parallelism, each layer is split across all the machines, which compute their part at the same time and combine the result.

4. The connection decides if you gain speed

   For splitting layers almost any network will do; for splitting tensors you need lots of bandwidth and, above all, low latency.

5. exo: a Mac cluster with almost no setup

   exo finds the machines on your network, decides how to split the model and offers a dashboard and an API on port 52415.

6. llama.cpp RPC: mix the machines you already have

   With llama.cpp's RPC mode, a PC with an NVIDIA card, a Mac and another PC can share the layers of the same model.

7. vLLM: tensors in the node, layers across nodes

   vLLM splits a model across several GPUs with --tensor-parallel-size and across several servers with --pipeline-parallel-size.

8. When it's worth it and when it isn't

   A cluster pays off when the model doesn't fit in any machine you can buy; if it fits in one, a single machine is almost always better.

9. Summary: clusters for local AI

   Connecting machines always adds memory; it only adds speed if you split tensors and the connection is very fast.