10G Ethernet problems – performance lags and uneven data rates
10G Ethernet problems – performance lags and uneven data rates
I have a home server setup running proxmox and the server's mother board is a z790 pro art Wi-Fi creator with a built in 10Gbe NIC. For my desktop, I am using a TP-Link TX-401 10Gbe NIC using the latest drivers from here (I have reinstalled and repaired them several times) I have them connected via a 5 port 10Gbe Trendnet switch and the link indicators on the ports are green suggesting they are at full speed. I have everything connected with Cat6 / Cat 7 cables with less than 15 meters of cable. The issue I am running into is, either through iPerf3 or just real world data transfer of a single file that is 10 gigabytes, I'm getting nowhere near 10Gbe speeds. To make things more confusing, it seems the speeds are slower in one direction than the other. I built the server and computer in the last 2 months, I did update the bios of the server when I set it up, but I did not for my desktop, although I did update the NIC's drivers as previously mentioned. When transferring the 10Gb file, it seems to peak around 120-140mbps. I'm not really sure where to begin testing because I dont have any other 10gbe capable devices to see what's working or not working. Im a proxmox noob, do I need to update the drivers of the built in NIC somehow for the server? Here are the iPerf logs Server connecting to desktop redacted@proxmox:/# iperf3 -c 192.168.0.54 Connecting to host 192.168.0.54, port 5201 [ 5] local 192.168.0.250 port 37628 connected to 192.168.0.54 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 480 MBytes 4.03 Gbits/sec 0 301 KBytes [ 5] 1.00-2.00 sec 514 MBytes 4.31 Gbits/sec 0 301 KBytes [ 5] 2.00-3.00 sec 516 MBytes 4.33 Gbits/sec 0 301 KBytes [ 5] 3.00-4.00 sec 510 MBytes 4.27 Gbits/sec 0 301 KBytes [ 5] 4.00-5.00 sec 509 MBytes 4.27 Gbits/sec 0 301 KBytes [ 5] 5.00-6.00 sec 512 MBytes 4.29 Gbits/sec 0 301 KBytes [ 5] 6.00-7.00 sec 509 MBytes 4.27 Gbits/sec 0 301 KBytes [ 5] 7.00-8.00 sec 517 MBytes 4.33 Gbits/sec 0 301 KBytes [ 5] 8.00-9.00 sec 510 MBytes 4.28 Gbits/sec 0 301 KBytes [ 5] 9.00-10.00 sec 517 MBytes 4.34 Gbits/sec 0 301 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.97 GBytes 4.27 Gbits /sec 0 sender [ 5] 0.00-10.00 sec 4.97 GBytes 4.27 Gbits /sec receiver iperf Done. Desktop connecting to server redacted@proxmox:/# iperf3 -s ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.0.54, port 53956 [ 5] local 192.168.0.250 port 5201 connected to 192.168.0.54 port 53957 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 14.1 MBytes 118 Mbits /sec [ 5] 1.00-2.00 sec 72.9 MBytes 611 Mbits /sec [ 5] 2.00-3.00 sec 284 MBytes 2.38 Gbits/sec [ 5] 3.00-4.00 sec 312 MBytes 2.62 Gbits/sec [ 5] 4.00-5.00 sec 284 MBytes 2.38 Gbits/sec [ 5] 5.00-6.00 sec 264 MBytes 2.21 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.00 sec 179 MBytes 1.50 Gbits /sec [ 5] 8.00-9.00 sec 228 MBytes 1.91 Gbits/sec [ 5] 9.00-10.00 sec 211 MBytes 1.77 Gbits/sec [ 5] 10.00-10.01 sec 1.13 MBytes 1.36 Gbits /sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 2.08 GBytes 1.79 Gbits /sec receiver Desktop connecting to server again becasue wasnt sure if it was a fluke ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- Accepted connection from 192.168.0.54, port 53971 [ 5] local 192.168.0.250 port 5201 connected to 192.168.0.54 port 53972 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 18.0 MBytes 151 Mbits /sec [ 5] 1.00-2.00 sec 21.0 MBytes 176 Mbits /sec [ 5] 2.00-3.00 sec 156 MBytes 1.31 Gbits /sec [ 5] 3.00-4.00 sec 241 MBytes 2.02 Gbits/sec [ 5] 4.00-5.00 sec 122 MBytes 1.02 Gbits /sec [ 5] 5.00-6.00 sec 212 MBytes 1.78 Gbits/sec [ 5] 6.00-7.00 sec 320 MBytes 2.68 Gbits/sec [ 5] 7.00-8.00 sec 354 MBytes 2.97 Gbits/sec [ 5] 8.00-9.00 sec 346 MBytes 2.91 Gbits/sec [ 5] 9.00-10.00 sec 414 MBytes 3.47 Gbits/sec [ 5] 10.00-10.00 sec 1.13 MBytes 1.96 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 2.15 GBytes 1.85 Gbits /sec receiver As you can see, best case scenario is less than 1/2 of the expected bandwidth, and at worst its ~1/5th the expected bandwidth... Point of note which *may* help narrow down the culprit - I have a 3gbe fiber network, so I can test internet speeds from the server and from the desktop since that also runs through the switch. Heres those results - Speedtest from a TrueNAS VM on the server redacted@truenas[~]# speedtest Speedtest by Ookla Server: redacted - redacted, redacted (id: redacted) ISP: redacted Idle Latency: 4.36 ms (jitter: 0.13ms, low: 4.22ms, high: 4.56ms) Download: 3133.23 Mbps (data used: 3.7 GB) 30.49 ms (jitter: 1.24ms, low: 2.41ms, high: 37.87ms) Upload: 3209.73 Mbps (data used: 4.3 GB) Speedtest from desktop The first day I ran a speedtest after building the computer, I got a 3Gbps up/down so Im leaning towards the issues being with the NIC in my desktop, but Im really not sure becasue clearly the desktop is capable of around 4gbe when the server connects to it, and I have no idea why it would be different the other way around... Any and all help would be greatly appreciated
Hi @osint. The iperf test you ran appears to be using TCP mode. The speed of TCP transfers is strongly influenced by the receiving device and its TCP window size or buffer capacity. You have two main approaches: 1. Execute UDF-based iperf (-u on client) and specify a target bandwidth (e.g., -b 10000M on client). You may need to reduce the target to around 90% (9000M) to compensate for IP/ethernet headers not included in iperf's calculations. 2. Perform TCP-based iperf with parallel connections (-P n on client), starting with two and increasing as needed. Keep in mind that the server-side window size might still limit performance. For more details, refer to the official iperf documentation: https://iperf.fr/iperf-doc.php#3doc
What tools are you transferring data to and from? Often your performance with 10gb connections depends more on your storage solutions than the network itself. Consider creating a ramdisk on each machine and measuring speeds across them. Are you using trueNAS core or scale? Scale is still emerging and tends to offer slower speeds than core. The Tp- links might not be reliable. Intel appears to be the only brand I’d choose.