The delay on the X540-AT2 (10g) is twice the delay of the I210 (1g).
The delay on the X540-AT2 (10g) is twice the delay of the I210 (1g).
Hello, I am seeing an interesting result when testing my new hardware purchases and would like to ask if these results are normal / expected. I have identical custom servers that have onboard the motherboard (supermicro H13SAE-MF) an Intel I210 Gigabit chip, and an Inspur PCI-E add on network card powered by the Intel X540-AT2 chip (8x card plugged into a slot that is at least 8x via an 8x riser card). (These results are from debian/proxmox. All network cables are three feet or less.) 1gb I210 on motherboard: I try pinging from server a -> gigabit switch -> server b, and I get average ping of: rtt min/avg/max/mdev = 0.224/ 0.579 /0.761/0.148 ms I also tried via netperf: Minimum Latency Microseconds Maximum Latency Microseconds Mean Latency Microseconds 80 971 98.10 Throughput is as expected, iperf3 says 943 Mbits/sec 10gig X540 card (using linux ixgbe module): I also connected the 10gig X540 cards together using a crossover cable (no switch). I get a ping (ICMP) of: rtt min/avg/max/mdev = 1.043/ 1.342 /1.806/0.222 ms On netperf: Minimum Latency Microseconds Maximum Latency Microseconds Mean Latency Microseconds 72 1968 458.05 Throughput is as expected, iperf3 says 9.41 Gbits/sec Summary / Questions: These results vary from run to run, but it's always the runs of the X540 have significantly more latency than the I210. I would like to ask if it is normal for latency to be this high on other X540 cards? Or does anyone else have experience with the Inspur cards and seeing similar results? Thank you
You do understand how little difference one millisecond makes, right? This is a trivial issue, sorry. (Unless you're running a high-frequency trading automation system or something...) X540s are also over a decade old (not knocking them, currently have one in my Proxmox testbed), so having slightly higher packet processing time isn't that hard to imagine. Also, X540 is a different driver than the i210, so there may be something being done differently inside the OS. Also, because this is apparently a hill I need to die on: crossovers are not necessary since gigabit Ethernet, let alone 10G. Ethernet now automatically detects Rx/Tx pairs.
I was thinking about using Ceph and wanted to reduce latency. Those good points are worth mentioning—they’re significantly more affordable compared to newer 10G cards. It’s interesting how my understanding is a bit outdated.