Ep. 605: Building a Unified Supercomputer: From SSI to CXL

Episode summary: Ever wondered if you could merge your old home lab servers into one giant, powerful machine? In this episode, Herman and Corn dive deep into the "Holy Grail" of distributed systems: the Single System Image (SSI). They break down why true CPU and RAM aggregation is a challenge of physics and explore the modern alternatives used in high-performance data centers today. From the low-latency magic of InfiniBand and RDMA to the cutting-edge promise of CXL and resource disaggregation, the duo explains how to move beyond simple Proxmox clusters. Whether you're a seasoned homelabber or just curious about how supercomputers actually talk to themselves, this episode provides a technical yet accessible roadmap to scaling your hardware through the power of high-speed interconnects and specialized protocols.

Show Notes

In the world of "homelabbing"—the hobby of running enterprise-grade hardware and software in a home environment—there is a recurring dream that haunts every enthusiast. It usually starts with a stack of aging office PCs and a simple question: Why can't we just make all these separate boxes act like one giant, unified supercomputer? In this episode, hosts Herman Poppleberry and Corn dive into the complexities of distributed systems, exploring the history, the physical limitations, and the modern technologies that attempt to turn a collection of "silicon islands" into a single, cohesive ocean of compute power.

### The Dream of the Single System Image Herman opens the discussion by defining the "Holy Grail" of this field: the Single System Image (SSI). The goal of SSI is to aggregate raw compute power—CPUs and RAM—so that the operating system perceives them as a single pool. Herman uses a vivid analogy to distinguish this from storage: while sharing a library (storage) among many people is relatively straightforward, getting ten people to share a single brain to solve a math problem (compute aggregation) is a much more difficult task.

Historically, this was attempted through software projects like Kerrighed or Mosix for Linux in the late 1990s. These systems aimed to allow a process to start on one node and transparently migrate to another as resources shifted. However, Herman explains that these projects largely faded into niche research because of two insurmountable enemies: latency and cache coherency. As processors became faster, the "gap" between internal chip speeds and network speeds widened. When a CPU has to wait milliseconds for data from a remote machine's RAM—rather than nanoseconds from its own—the system stalls, rendering the "unified brain" approach inefficient for general computing.

### Breaking the Bottleneck: InfiniBand and RDMA If software-based SSI is limited by physics, the solution must lie in the hardware. Corn and Herman shift the conversation toward the high-speed interlinks used in massive data centers. For a home labber like Daniel, whose question prompted the episode, a standard one-gigabit Ethernet cable is the first "straw" that will cause a bottleneck.

Herman suggests that enthusiasts look toward the used enterprise market for InfiniBand hardware. Unlike Ethernet, which was designed for moving packets over long distances with significant overhead, InfiniBand was built as a system bus for data centers. When paired with Remote Direct Memory Access (RDMA), the game changes. RDMA allows one computer to reach directly into the memory of another without involving the CPU of either machine. This "direct straw" into remote RAM significantly reduces latency and CPU overhead, making distributed systems feel like local hardware.

### Storage Aggregation: Beyond Ceph While the episode focuses on compute, storage is the area where aggregation has seen the most success. Corn brings up Ceph, a popular distributed storage system, but asks for alternatives. Herman highlights GlusterFS and MooseFS as two viable contenders for the home lab.

GlusterFS is described as a distributed file system that "bricks" local disks together into a single volume. While easier to set up than Ceph for small clusters, it remains heavily dependent on the network interconnect. On the other hand, MooseFS is praised for its ability to handle heterogeneous hardware—allowing users to mix and match disks of different sizes and speeds. However, Herman warns of the "master server" vulnerability in MooseFS; if the central node keeping track of data locations fails, the entire cluster goes blind, necessitating a high-availability setup.

### The Modern Shift: Resource Disaggregation and CXL The conversation then moves to the cutting edge of data center architecture: resource disaggregation. Instead of trying to make multiple computers act like one, modern engineers are taking the components out of the boxes entirely. In this model, you have separate chassis for processors, memory, and storage, all connected by an ultra-high-speed fabric.

A key player in this shift is Compute Express Link (CXL). Herman explains that CXL 3.0, built on PCIe 5.0 and 6.0, allows for true memory pooling. While it doesn't necessarily "combine" two small RAM sticks into one across a network, it allows a central pool of memory to be dynamically mapped to whatever server needs it most. This ensures that resources are never sitting idle, even if they aren't physically located on the server's motherboard.

### Orchestration vs. Execution Finally, the duo clarifies the difference between "orchestration-based aggregation" and true distributed computing. Corn asks if Kubernetes counts as aggregating resources. Herman explains that while Kubernetes makes it *feel* like you are deploying to one giant computer, the execution is still bound by the physical limits of a single node. You cannot run a process requiring 64GB of RAM on a 16GB node, even if your cluster has a total of 1TB of RAM.

For tasks that truly need to span multiple nodes—like weather simulations or training Large Language Models—the industry uses the Message Passing Interface (MPI). This requires the software to be specifically written to coordinate its work across a cluster, manually sending messages between nodes.

### Conclusion The episode concludes with a realistic takeaway for home labbers. While the dream of a single, unified "brain" remains elusive due to the laws of physics, technologies like RDMA, InfiniBand, and the emerging CXL standard are bringing us closer than ever to a world of seamless resource pooling. For the average enthusiast, the path forward isn't necessarily about making ten computers act like one, but about using high-speed fabrics to ensure that every scrap of silicon in the rack is working at its highest potential.

Listen online: https://myweirdprompts.com/episode/unified-supercomputer-resource-pooling