Georgi Gerganov
@ggerganov
llama.cpp can distribute inference on heterogeneous devices through the ggml RPC backend
It's an advanced setting but I think with time we'll make it more accessible to regular users.
It's an advanced setting but I think with time we'll make it more accessible to regular users.
Pedro Cuenca@pcuenq · Oct 7It's crazy that I can run MiMo 2.6 Flash across my RTX 6000 GPU and my M5 laptop at 40 tokens/sec over 10 GbE 🤯
These are the native mxfp4 weights of a state-of-the-art model, on heterogeneous hardware.
Supported out of the box in llama.cpp.
These are the native mxfp4 weights of a state-of-the-art model, on heterogeneous hardware.
Supported out of the box in llama.cpp.
0:12 28 495