Network Upgrades - 10G Fiber, 5G WAN Failover, new switches

spookyghost -

Its been a long time since I have updated my actual network setup. I've had posts that details parts of it, but never the full config. Here it is

I've recently made a lot of changes. I've added more 10G, more Fiber, and even WAN Failover.

Hardware

First I'll get my garage out the way, there have been no real changes since this post

Multimode fiber (MMF) to Singlemode Fiber (SMF) Detached Garage Fiber Backhaul Upgrade
While cleaning up cables in my garage, I swapped the Multimode fiber run to my detached garage from MMF to SMF Details on my setup can be found here, as you can see, it used to be a mess. Garage Network Rack with 10G FiberThis details my 12u rack in
NetworkProfile.orgSpookyGhost

In my main rack however, there have been a lot of changes

Below the patch panels you can see my PFSENSE Firewall. This is a Pentium G5500 based system, with a Mellanox ConnectX-3 for 10G LAN connectivity

As you can see, I am using a few of the port for AT&T, one for T-Mobile 5G internet and one 10G SFP+ port for LAN connections

I'm using pfatt to bypass the AT&T gateway and connect directly the fiber ONT, which gets me great performance

Then, I have my 5G T-Mobile Home Internet connection

The T-Mobile gateway gets the best signal on my desk

It then goes into a wall jack, and back to my server closet (It gets TERRIBLE signal in my server closet)

Since I have 10 x Cat6a runs to my desk, its no problem taking one up

It comes out here, and I have a pink patch cable for it (Since T-Mobile...)

I get excellent speeds through this connection, especially since its just a secondary

The downside is that its all CGNAT, so no port forwarding. But for a backup connection, thats fine.

Here is the setup in PFSENSE

As soon as the AT&T Connection goes down, the gateway switches to T-Mobile pretty much instantly

You might ask why I have this connection if I have a reliable AT&T Fiber Connection. A few weeks ago the utility came and replaced a power cable, and broke my neighbors fiber. That made me realize I need to get a backup. They hap-haphazardly ran a temporary cable, which they have not been back to replace yet. I knew one day this might happen to me, and I'll need a backup.

So that covers the firewall and the backup internet.

The firewall connects directly via 10G to my Cisco SX350X-24F which is a 24 port SFP+ 10G Switch. This switch I got brand new from Amazon for just $1000. For the feature set and performance, I think its a great value

On this switch is my Firewall, My Desktop, My wifes desktop, 2 x Connections for ESXi, 2 x Connections for TrueNAS and a single connection to my garage switch. Then there is 2 x 10G DAC connections to my Dell X1052P

Here is a picture of my rack again so you can see the Dell switch. The Dell switch handles all high bandwidth 1G connections, and all PoE connections (IP Cam's, AP's etc)

As you may notice, I also have a Cisco SG300-28 switch there. This switch handles all management and low performance 1G connections. This is IPMI, printers, UPS's etc.

This switch setup means I could easily lose one switch and with minimal config get back up and running. This time last year I just had the Dell X1050P. If that went down, I'd be completely down.

That pretty much sums up my network setup upgrades. Let me know if there is any questions, and I will update this post with them