Infiniband network setup: Part 2
Infiniband network setup: Part 2
So since the last time I posted, I've had gigabit fiber from Verizon set up, and it's glorious. I've also finished running 12-fiber MTP/MPO OM3 between floors for 40G Qdr IB and the switch just arrived. Still waiting on the actual rack cabinet itself, so everything is just piled up right now. Now, I don't know a damn thing about network topologies involving IPoIB. Right now, here's what I have in no particular order: Ubiquity Security Gateway (1u) Cisco 24-port Gigabit Ethernet Switch Intel/Qlogic TrueScale 12200-18 QDR IB Switch 4x Mellanox ConnectX-3 QDR/10GigE HBA 40GBase-SR4 Transceivers One spare 2U Xeon E3-1240v6 system Now, my plan was to run two subnets, one ethernet for internet access (desktop, smart TV, etc to Wan) and one IPoIB storage between my desktop, a server, and a storage archive. I would have to run another Cat6 cable through about three walls to get ethernet to my daily desktop though.... Don't ask why I didn't do this at the same time as the Om3 fiber, but I didn't. As such, I have a spare 2u system that is a mini compute node right now I was hoping to put a ConnectX-3 into to bridge Eth to IB. I know IPoIB is also much slower, but I've never worked with IB at this level and so don't know a better way to easily host a network drive in WinServer 2016. I know I can get everything to work, but probably not efficiently. Anyone have any opinions on the best way to go about setting this network up?
Is this the USG mini router or the USG Pro-4 server unit? If it’s the Pro-4, be cautious about the restart loop problem caused by faulty RAM and your attempts to swap it out didn’t help. Overall, a strong initial configuration. Interesting how everything fits together. I suspect the infiniband switch likely added significant cost.
It's a Pro-4, the compact 1U form factor (I’m keeping an eye on the Middle Atlantic Slim-5). The price is $189 for new old stock. I was hoping to find either this or a Mellanox 3600q-1unc, which also works as a Qdr switch. Eventually, something came up on eBay for a lower price. It’s taken a while, but it seems companies might sell these at a discount rather than paying disposal fees when they upgrade their racks and no longer need backup stock.
It seems your Pro-4 keeps restarting in a loop. Consider sending it in for replacement or replacing the RAM inside. You can find more details in this discussion: https://community.ubnt.com/t5/UniFi-Rout...095#M84571. Initially, I planned to upgrade the RAM, but the issue persisted. The EdgeRouter 4 came up, which was fine—actually quite impressive. I’m used to seeing higher-end options, but this is still a solid performance.
You're thinking differently about networking options. It seems like you might be looking for a more efficient setup instead of juggling multiple management interfaces. I've been considering consolidating the connections using 100GbE and might even switch to a different device altogether. My experience with Microcenter switches has been positive, and I'm leaning toward a TPLink model after some research. The Pro4 I own hasn't had any issues so far, and I'm more confident now that it's stable.
Hey, sorry if the USG Pro-4 took your joy away. I was just trying to explain why it kept restarting after two weeks of troubleshooting—and even thought about buying the recommended RAM upgrade. But now I’m confused about why I’m paying extra just to fix it. Then came the Edgerouter 4, and I told myself it was a waste of money, so I got a refund. If you have a model that doesn’t have this problem, it works perfectly. The layout chart is actually pretty good too.
Moved into the new house and set up Win Server 2016 on the desktop PC. Still waiting on video drivers. Focus now. It’s 2:00 AM, but we finally got a full 40Gb/s connection over 33 feet using $250 worth of gear. That counts as a win for now. The rack is in a rough heap, but I’ll wait until the wood/thermolaminate panels arrive and the storage server is ready. I was worried about ordering from FS.com (a Chinese fiber networking supplier), but the two 40G QSFP+ transceivers plus 10M of OM3 MTP/MPO cable shipped straight from California with free 2-day air, arrived quickly and everything works perfectly. As someone in a defense contractor, I know paying over $1000 for an OEM transceiver isn’t cheap, but these parts are solid and affordable—under $50 each at FS. If anyone needs a rack cabinet, Middle Atlantic’s Slim5 series is available from BH Photo, sized like a regular PC case (minus the rails in a big box). It’s just stamped with powder coating on steel. This unit looks sturdy enough to last. I’ll keep sharing updates as work progresses.
I've purchased 1000ft of CAT6, a few DACs, and some Single Node fiber from FS.com without any problems. I really like their CAT6 cable more than Monoprice—it's stiffer and smoother even though it costs a bit more. I intend to continue buying networking gear from FS.com moving forward.
I would keep everything on the same network, their is no point to waste routing power unless something is public faceing or for security reasons.