Creating a LAN setup is incredibly challenging.
Creating a LAN setup is incredibly challenging.
Dear forum, I'm back in search of wisdom. I'm encountering multiple problems in my current LAN setup, and while I have a couple of suspects, everything tells me I should be getting better results or worse results, but not the exact results I get. It makes no sense to me at this point. Let me preface with two things: 1) I know there are many ways to connect two computers, so having "a" connection could be achieved by trying a different configuration and see how it goes. However, the goal here is to get this layout to work, or alternatively, understanding why it can't (as in, shouldn't) work, and then move to something else. If I just quit I'll learn nothing. 2) This is a graph of my current setup. Each line represents a 1Gig Ethernet connection. Ellipses represent bonds of ethernet connections working as one (LACP a.k.a. 802.3ad). Wireless connections go directly to the ISP modem, and are in the same space as the modem and the PFSense WAN (say, 192.168.xx.zz). PFsense LAN and all the PCs are in 192.168.yy.zz, xx=/=yy. Dashed lines delimit rooms, connections between rooms go through the walls. Spoiler The problems 1) The old: It all started after installing the PFSense router (although I remember some previous oddities preparing this post, more on that later). I made a thread about it here: Spoiler In a nutshell, the connection between PFSense and Switch breaks down, triggering a "possible flapping" message on the PFSense logs. It goes back up with no user intervention. The current state of the problem is reproducible: it happens every time I turn on or turn off either of PC1, PC2 or, since Installing that, PC3 (I've checked ans it's not exactly at power up, but when the OS starts loading). If you are using any of the PC1 or PC2, you're suddenly limited to the Switch, then after waiting a bit (varies between seconds and minutes) you can go all the way to the internet again. 2) The new: I installed Ubuntu Server on another PC last week, configuring a bond as main network interface, but without connecting to the internet during the installation. Once done, I connected the cables and try connect to the outside world to no avail. Now here's the puzzle: I can reach all the way from PC3 to the PFSense box and SSH into it. That is, the path between PC3 and PFSense seems no different than the path between PC1 and PC2, and PFSense. But PC3 gets no internet. The obvious candidate was DNS, but I checked with various built-in tools (route, netplan --debug, looking at resolv.conf and whatever the netplan config files are, etc), and in all cases it is correctly set to 1.1.1.1, 8.8.8.8 (it's the same in PC1 and PC2, PFSense is set to DNS Relay). But no internet. I did some ping and web GUI / SSH tests, as follows: PC1: can ping Switch, can access Switch GUI, can ping PFSense, can access PFSense GUI, can ping PC3, can ping 1.1.1. has internet PC3: can not ping Switch, can ping PFSense, can SSH into PFSense, can ping PC1, can not ping 1.1.1., has no internet I checked the PFSense logs and there's no trace of PC3 trying to reach the internet, or anywhere, and being blocked. The only signs of any blocking are stuff coming from my laptop (connected through WiFI to ISP modem) towards PFSense LAN and inwards. No clue what that is, but glad to have PFSense Anyway, laptop on or off does not affect my results. So that's my puzzle: I would understand having internet on PC3, and I would understand if PC3 couldn't reach PFSense due to some misconfiguration somewhere. I would understand less if PFSense was blocking PC3 while not the others, but at least it would be a lead to follow. But right now it's as if PC3 was simply not sending the requests to, say, 1.1.1.1 correctly, or at all 3) The unresolved oddities from the past Before even having the PFSense box, everything seem to work fine (PC1 and PC2 to Switch same as now, Switch to ISP), but I remember now two issues that I overlooked: i) PC1 dual-boots Windows and Linux Mint. Windows configured the team (a.k.a. bond) in two clicks, and worked since then. In Linux Mint I never got any Ethernet connectivity whatsoever as far as I remember, and 100% sure the team does not work at all in Linux to this day. I'm limited to WiFi on PC1 when on Mint. ii) I have two available modes for the "Etherchannels" (how Cisco calls the teams/bonds) in my Switch: "Static" and "LACP". THe description of "Static" was not very informative to me, but my understanding is that the correct option for my setup is "LACP". That's how all the Linux / PFSense bonds are set, and they will not work if I change that to "Static" in the switch (while keeping the settings unchanged in the computers). This is the expected behavior, I think. However, the link between Switch and PC1 is set as "Static" in the Switch. Looking back, when first setting this up, there were instances in which it would work in Static but not LACP, it would work in LACP but not Static, neither, both, etc, until I finally got it all set. My understanding is that it shouldn't work because the team is set as LACP in PC1, but OK, I guess what does work is not my main concern So, at this point: what could be preventing PC3 to reach the internet through PFSense like the other computers? What's causing the link between PFSense and the Switch yo break down when a computer connected to the Switch initializes its NICs / powers them down? I'd blame the Switch, but then, why does a direct link between ISP and Switch not break in the same situation? I do hope it's the Switch: While it would suck not being able to use it this way, I think I could setup a similar layout with PC3 acting as both server and switch. Still, I'd want to understand first what's wrong with the damned CE500G, and salvage it if possible. Thanks in advance to anyone with 2 hours to read this post and able to provide any hint
Bad news for me. Skipping the Switch and linking PC3 straight to PFSense doesn’t work well: PC3 can’t reach the PFSense at all, no ping, no SSH, nothing—previously it worked via the Switch. Only the internet connection is missing now. The "possible flapping" alert isn’t showing up in PFSense logs, which suggests the issue might be related to the Switch, while the lack of internet is PFSense-specific. It’s unclear what to do next, but I’ll need to get the switch back in its proper position so PC1 and PC2 can connect.
No specific reason was mentioned for enabling LACP on the pfSense router. The suggestion focuses on improving performance by using a dedicated access point and bridge mode instead of relying on double-NAT and WAN routing through the router.
This would enable PC3 to support several devices in room 2 and possibly link the connections to that space. I could test with a basic PFSense connection from a laptop in room 2, which appears to remain stable despite the changes. For the other issues, Problem 1 seems broader, while Problem 2 appears tied to PC3. I’m considering switching to a different operating system for that test, given the dual-boot situation on PC1. Yep, that aligns with the roadmap. Right now, direct WiFi links are helping me during my Ethernet work. Otherwise I’d lack a way to search for solutions while trying changes. Plus, I might prefer keeping WiFi devices separate from wired ones. I could manage that inside PFSense, but since I’m not treating it as a firewall, I’d check if the Switch could fit in the cabinet. Removing PFSense would still leave LAGGs to both rooms, but PC3 wouldn’t work well—after several attempts I’m seeing no SSH access, no pings, and it’s not feasible to keep it on Windows.
Are you relying on fixed ARP records from any network device for those computers? There was a problem with one of my prebuilt NAS that randomly changed MAC addresses, and pfSense prevented it because the IP was linked to a different MAC.
I believe it isn't a fully functional setup despite being a double-NAT. The real advantage usually comes from pfSense's speed in routing and firewall tasks compared to regular consumer routers. Keeping the WiFi separate would be simpler by using a second LAN on a different port and connecting a dedicated AP there. A standalone WiFi AP generally works better than bundled routers, and you can position it more effectively for strong coverage without complicated wiring.
Thanks for your responses, @Alex Atkin UK. I'm still unfamiliar with NAT beyond its basic existence and the manual configuration I had to do a decade ago. However, your last point seems significant: my ISP offers a "modem-router" that connects coaxial to Ethernet and WiFi. So, in practice, PFSense must route through that device to reach the outside network. The ISP itself isn't very clear about whether it's a proper router or just a smartphone handling requests, so its default routing and security settings are applied automatically. This means any performance issues I face on the WAN side are largely fixed, though I might find some tweaks in PFSense's interface. Regarding the update notes: I switched from Ubuntu Server to CentOS, which seemed to handle the bond setup more smoothly and consistently. All devices appeared operational except for connectivity—no ping, no internet. I attempted to disable certain features in PFSense's GUI but haven't tried it yet until the whole process goes through. Last night, I also tried disconnecting PC3's bond and connecting via a single Ethernet link to the switch on different ports. Despite changing IPs and configurations, I still didn't get a response. I attempted to set the switch as gateway, but it didn't work as expected. When I connected a Linux Mint laptop to the switch, it failed to connect via DHCP and triggered PFSense's "possible flapping" alert. I tried assigning a manual IP, but it didn't help either. Restarting the laptop in Windows gave me pings and internet access, suggesting a possible issue with LAGG in PFSense or the Linux+Switch pairing. I suspect a specific problem with the switch or the PFSense+Switch combo, and possibly a broader network configuration challenge. I'm considering bypassing PFSense to test if a direct Linux-to-switch-to-ISP path works, or if the issue lies deeper with Linux's networking stack.
Two scenarios exist: 1) Several cable providers offer the ability to switch modems/routers to modem-only mode, though they might need remote assistance if it isn't visible in the router interface. 2) Certain companies allow purchasing your own modem. It seems unnecessary to rely on pfSense if it isn't handling routing itself. Double-NAT routes traffic twice, but the heavy processing stays on the first router, making the second redundant.
I’d still be able to function as a LACP router/switch and offer more control over firewall settings, site blocking at the house level, a potential pi-hole alternative, and connecting to a VPN through one central point rather than each PC individually. Honestly, I hadn’t considered performance improvements before—I came through PFSense for flexibility. If turning the ISP modem-only mode on helps, I’d appreciate the performance gains too. But first, I need to set up the wireless AP. This thread has a lot of work ahead for someone working alone.
Linux demands a distinct setup for the Switch compared to Windows. I configured the laptop to connect by adjusting the Switch’s "smartport role" to "Desktop." I also manually configured the link speed to "1000 full duplex" on both Linux and the Switch, avoiding the automatic negotiation setting. The second adjustment was straightforward enough. Regarding the first point, Cisco historically offered role-based "optimized" configurations for ports—often labeled "Desktop" or "Server." While I set all ports to "Switch," this aligns with the role expectations. The reason for assigning every port as "switch" rather than a specific role is practical; it matches the mandatory requirement. For Etherchannels, it’s clear Cisco didn’t plan bonding for workstations or servers, focusing instead on core network segments. If I need Linux to replicate this behavior, I’ll need to understand Windows’ approach and adjust accordingly. Otherwise, I’d consider switching to a PFSense setup on PC3 or using a direct bond configuration to simplify the network.