AI Cluster networking

Framework Desktop
1

I assume some of you have already seen Jeff Geerling’s Framework Desktop review:

Is this just a weirdo thing ? Why should we care?

Well, you need a lot of VRAM to perform inference efficiently on large models, and the 128GB of shared memory available at this price point at least appears to be “a lot”, in fact, when compared to a ‘nuh-vidia’-based system, it’s around 10x cheaper for the same amount of RAM.

It’s not exactly clear whether the disappointing performance Jeff observed was due to potentially solvable software issues or simply due to network bandwidth limits.

I have heard that GPU RAM and FLOPS have grown a lot faster than networking has, which introduces problems for anyone who thinks that they can simply connect more computers together to do training or inference on larger models:

image
image1243×734 160 KB

Source: https://www.youtube.com/watch?v=IOtqqHAofmA (5 minutes 30 seconds)

It makes me wonder how much better Jeff’s menagerie would perform if the nodes could somehow talk to each-other a lot faster.

There are a lot of different ways to network computers together in a cluster, such as:

Hub and Spoke

image
image474×227 9.89 KB

  • This is what Jeff used
  • only requires 1 network interface per machine
  • the bandwidth of that interface is shared by all connections
  • the performance of the router or switch matters a lot

A “ring” network

image
image272×281 7.46 KB

  • only requires 2 network interfaces per machine
  • requires more “hops” and sharing of bandwidth to reach nodes on the other side of the ring

A “Full Mesh” Network

image
image245×259 8.34 KB

  • requires n-1 network interfaces per machine for an n-node cluster
  • always 1 hop
  • maximum bandwidth always available, no “sharing” at the network level.

The top comment on Jeff’s post suggested:

If those Pcie slots on the framework were open ended I’d suggest you add an Intel E810-CQDA1 Network Adapter to each node.

That’s a 100-gigabit fiber network adapter which has a PCI-e 4.0 16x interface.

But the framework Desktop mainboard only has a 4x PCI-e slot. So what gives? That’s what he meant by “open ended”, I think. The idea being that some peripherals may function when connected to a 4x PCI-e slot, with the rest of the pins sticking out of the “open end” of the slot, just hanging out in the open air.

Another thread about this: https://www.reddit.com/r/framework/comments/1izucdw/framework_desktop_why_no_openend_pcie_slot/

It sounds like this is a potential way to increase the Schpeeed of the hub and spoke architechture.

Of course, it wouldn’t be able to reach 100 gigabit: the PCI Express 4.0 interface is limited to 2GB/s (16 gigabit) per lane. And there are 4 lanes on the PCI-e slot. So that would total 8GB/s or 64 gigabit.

But wait-- there’s more! A reddit comment on that thread pointed out

The CPU only has 16 PCIe Lanes…
…
See the Block Diagram here:
AMD Ryzen AI Max 385 Specs | TechPowerUp CPU Database

image
image1500×493 62.9 KB

The Framework Desktop has 2 m.2 slots for NVME SSDs – according to that block diagram they should be usable ans PCI-e 4.0 slots via an adapter:

image
image640×661 60.8 KB

So in theory each framework desktop mainboard can have… 3 separate PCI-e 4.0 slots with 4 lanes each, each one capable of 64 gigabit/s theoretical bandwidth.

That would be enough for a cluster of 4. Another Geerlingguy blog commenter stated:

I’ve read that used 40G Infiniband gear can be found for cheap, …

Unfortunately that doesn’t seem to work; all of those older cheaper fiber network interfaces appear to use PCI-e 2.0 or 3.0 with 8 or 16 lanes.

Unfortunately the 100 gigabit fiber interfaces are kinda pricey, is there a cheaper alternative?

I have heard of setting up a cluster using USB4 cables and some careful configuration on the linux kernel to utilize them as network interfaces:

USB appears to be an attractive option; as a newer technology, it appears to “fit” in the PCI-e 4.0 4-lane slot.

In fact, per the block diagram, it looks like the chipset already has 2 40-gigabit usb 4 ports built in. That could already enable a cluster of 3 boards.

But can it be made any faster? What about the new, emerging 80-gigabit USB 4 spec? It sounds like the PCI-e USB4 add-on cards that exist today have limited motherboard compatibility and no 80gbps ones have been released yet. Also, per https://www.reddit.com/r/framework/comments/1f2tre5/feasibility_of_a_usb_4_v2_expansion_bay_card/ it sounds like this is just not gonna happen on the framework.

What are your thoughts on all of this? Where does it rank on the crackpot index? Have any of yall tried the USB-4 networking thing on these boards, and what was your experience with it ?

7 Likes
2

All depand on what software you want to use…

For now llama.cpp can only use a “round-robin” dispache, only 1 node can make compute, the LLM is dispache with 1/N layer on a node. only data between layers is to be transmitted with it a ring network is perfect. so tha 2x 40Gb USB-C may be one of the best possible (and sur the lower price…)
If it make simple to use large model it can’t accelerate the compute speed.

Now llama.cpp did not have the “best” clustering “model” there is much more advance. There are other techniques that allow you to speed up the calculation speed (including token generation) while limiting the network throughput. However, I have trouble seeing if it is really possible to code this in llama.cpp (and gguf…) I tend to think not.

I think pytorch (vllm?) have experiment with that. But I don’t know if it is usable for now in our case. but it is more focus on batch processing.

Tensor Parallelism (Breaking Down Parallelism Techniques in Modern LLM Inference | by Hao C. | Medium) can reduce network usage, with accelerate compute speed for exemple. In that case we need network for all reduce after 2 matmul. in that case we need to figure what is the best network topoligie (to late this night to thing of that :wink: )

3

From where I sit, based on Strix architecture, bandwidth limitations don’t look to be flexible. I think the short-term pathway for clustering is not networking but orchestration

A fantastic post @forest_johnson !