Evaluating PCI-E switch efficiency
Evaluating PCI-E switch efficiency
I'm setting up a workstation using the TR 5965wx and planning my storage configuration. Currently, I have one FireCuda 530 SSD, which is a solid write-optimized option that fits my requirements. I’m thinking about adding RAID1 for improved data protection (I’m using btrfs or ZFS RAID, which helps with checksum verification and corrects errors automatically). My main concern is the motherboard (ASRock WRX80 Creator) has one M.2 slot connected to the CPU and another to the chipset, plus a U.2 drive. The schematic from ASRock shows all these ports are on the chipset. It looks like the chipset shares 8 lanes compared to the 4 lanes typical in consumer CPUs, which still provides a lot of extra space. I’m wondering if PCI-E switches would work without any data loss—would that be fine? If they do, should it deliver near full speed for M.2 and U.2 devices? Also, does the chipset use DMA to move data between USB 3.2 Gen 2 and NVMe at a higher rate, avoiding bottlenecks? Given the speeds you mentioned (8x PCI-E vs 15GB/s x8, 10GB/s USB 3.2, 7GB/s NVMe), it doesn’t seem like the setup will be limited for simple tasks like copying files.
You're referring to a system that doesn't fit the typical PCIe switch model. These devices rearrange the PCIe connections from the CPU to various sections of the motherboard based on what's being used, so a setup with two PCIe x16 ports and 16 lanes would allocate all lanes if only one is active, but both would share half each if both are in use. This behavior resembles a PLX controller, which can expand the number of available lanes—often turning a standard 32 or more into a higher capacity solution. Most motherboard designs include extra provisioning to handle this flexibility without overloading the CPU's direct lanes.
The chipsets are built to manage this by reserving space for high-speed components like SSDs, GPUs, and capture cards, while leaving room for slower peripherals such as USB drives or SATA ports. These devices typically operate at lower bandwidth (around 8 lanes) compared to the CPU's maximum, which is usually 10Gb/s or more depending on the port. Networking and storage interfaces also cap at around 10Gb/s per port, with M.2 slots reaching up to 7Gb/s.
If you're looking for maximum performance, it's common to use dedicated add-in cards instead of relying solely on chipset-provided lanes. For tasks demanding more than the chipset can handle—like transferring large files between Gen 4 NVMe drives—consider using expansion cards or other solutions. While overprovisioning is standard, it’s important not to strain all lanes simultaneously, especially when managing multiple high-speed devices.
Sounds logical. I placed an order for the M.2 card and plan to use it for RAID1. With that slot now available, I think I’ll stick it to the boot drive as recommended and remove the OS files from that RAID. It should boost performance on the main system and VM storage because OS-related I/O won’t interfere with the RAID1 setup. Also, it’s cool to confirm PLX chip and PCIe switch are essentially the same thing.