Network performance issues? Check your setup.
Network performance issues? Check your setup.
Welcome to the forums! Here are some questions: How does your per-core CPU usage look on each device during testing? Have you used -P 8 or even 12? Are the Ubuntu systems connected in the same network? Are they somehow isolated by the firewall? What bandwidth do you require through the firewall? Do you have a 10G connection available? Please note, I have no prior experience with proxmox and opn, but I’m familiar with network troubleshooting.
Hi @OddOod, thanks for your response. The CPU usage doesn’t seem important in either scenario (~5% under peak network load). I tested with -P 1, -P 4, -P 8 and -P 12, and it consistently stays near 6Gbps total. We’ll actually have about 2Gbps WAN connectivity, but the datacenter in Milan is quite large and can go up to 100Gbps. I don’t think we’ll ever need that high, but it would be helpful to understand what’s currently limiting performance so we know our scaling limits. By the way, all Ubuntu machines are running on the same Proxmox server, connected via the identical virtual adapter and configurations.
Review firewall settings on Proxmox. Disabling them could resolve the issue. It seems the net0/vmbr0 is just attached to vmbr0, allowing the router to obtain its own IP. I might not have noticed queues=8 or missed it, but my instinct is it’s missing in this setup. If you’ve done PCI passthrough, make sure to turn on the PCI Express option. Let me know if this helps.
Unfortunately, I attempted without using the queue parameter and got the same outcome. Right now I’m not doing a passthrough, but the problem mainly appears in machine-to-machine communication via virtio network. Do you have a comparable configuration and would you like me to test it with iperf3? Thank you!
However, I'm currently away; I'll share Thursday's update once back. My configuration uses Proxmox with PFense instead of ONsense. It's mainly Vmbr0 on net0 (WAN) with Vmbr1 static IP, and Vmbr2 is isolated as a LAN without any setup. The main variation is that my VM runs on a QEMU CPU model—uncertain if it's kvm64 or qemu64. You might try connecting the VM to different network devices to test performance.
I experimented with various CPU emulation settings and network devices. Even though I thought the host was the fastest, switching CPUs didn’t show much improvement. Changing the network hardware actually reduced performance. It’s still unclear what to do next.
Have you attempted to modify the network type in OPNsense and compare it with a Virtio network? In the past, I faced issues because PFSense didn’t recognize it, so I had to switch it and it resolved. Perhaps this is your situation too. If it doesn’t help, try using PFSense or OpenWrt OS and check if that makes a difference. Sorry, I couldn’t offer further solutions.
I tried with the Intel e1000 and another one that I don't remember, the virtio was the faster anyway.. it seems like it's a well known issue but it's absolutely frustrating. Over 5gbps (single socket) or 10gbps (multi socket) I get packet loss.