What makes PCIe cards unsuitable for dual-CPU setups?
What makes PCIe cards unsuitable for dual-CPU setups?
I’ve been pondering why dual-CPU setups using PCIe cards aren’t common. Imagine a card that runs a second processor—complete with its own memory and storage—alongside the main CPU, like a dual-socket board. This would let tasks be split between the two, offering flexibility similar to modern motherboards. Since PCIe already handles high speeds and supports GPUs and accelerators, the main barriers seem to be hardware design choices and software compatibility rather than raw limitations.
For optimal performance, CPU-to-CPU connections must offer much lower latency than PCIe can deliver. A full redesign of the CPU and chipset interfaces would likely be necessary, which is costly. Developing compatible software and drivers would also be challenging, particularly when trying to integrate both AMD and Intel components.
Both processors support x86_64 architecture, which means software shouldn't really have to worry about it. It might introduce extra challenges for the task scheduler, such as relocating AVX512 tasks to the better-suited processor. I believe a scenario that leverages unique strengths of both AMD and Intel chips would be extremely rare. I’m skeptical either company would back such a setup officially. Unless you require more cores than one CPU can offer, modern systems usually don’t justify multi-socket configurations. Adding more CPUs via an add-in card only makes sense if you later find you need additional cores after the system is installed, and you already have the maximum core count for that socket.
To leverage multiple CPUs together as a single unit, latency might become a more significant issue than bandwidth. PCIe 5.0 x16 offers greater bandwidth compared to an unclocked Zen 3 chiplet for I/O operations. If the CPU operates independently like an accelerator card, that should be feasible. However, the trade-off in cost remains minimal unless implemented at scale, and you must determine the optimal way to utilize it.
Essentially, we already handle this. These components are known as Graphics Cards. CUDA and GPGPU technology exist, and it seems Nvidia even integrates an ARM core in their GPUs for efficiency. Imagine FPGA boards too, if you can develop them and afford the cost—like a relatively inexpensive car. In any case, delays will become an issue over long distances (latency is already a challenge with modern DDR5, let alone further expansion). Distributing resources across expansion cards probably won’t solve latency problems for applications needing quick responses. External cards seem ideal for processing large volumes of work where speed of output matters more than quick response times.
Intel early video codec cards combine an internet LAN chip with four mobile CPUs each having their own RAM. You can connect via SSH and utilize the built-in GPU for decoding. It’s not a direct link—it works through the LAN chip, yet it aligns with your idea.