My Overkill Home Network - Complete Details 2023

spookyghost

Aug 10, 2023 • 22 min read

In this post I will hopefully detail my entire home network. Some of this has been in separate posts explaining single items, but nowhere do I have all of the network in one post with all the changes since last year.

Here is a full shot of the rack in my house. Its in a centrally located closet which happens to have a 2ft x 2ft chase into the attic, which is very handy for running network cables. The rack itself is the 25u adjustable StarTech Rack which I've been quite pleased with.

Starting at the top we have all of the networking gear and patch panels.

At the very top are 2 x Monoprice keystone patch panels. They are mostly filled with Cat6 keystone jacks, but I also have a few LC-LC Keystone jacks for long fiber runs. All of these connections end up somewhere else in the house, either to cameras, AP's or to network jacks etc. Keystone patch panels are clearly the best, a you can easily add and remove jacks without moving the entire patch panel and possibly breaking other connections.

This is a generic 1u cable management device off of Amazon, which does a pretty good job along with these Monoprice SlimRun Cat6a cables I am using.

Below the patch panels is a 1u Supermicro Server running PFSENSE

It has an Intel Pentium Gold G5500, 8GB of ECC RAM, Quad onboard Intel NIC's and a Mellonox Connect-X3 10Gb NIC for LAN + VLANS's. More details in this post

Around the back you can see the connections for my redundant internet. Ignore where it says T-Mobile, that should say Verizon!

I am using a 10Gb connection for the LAN and VLAN's because I do inter-VLAN routing on the firewall, so having that extra bandwidth helps. Also because my Internet is 1Gb symmetrical, it was very easy to saturate the 1Gb Link back to the switch.

Here is the Verizon Gateway sitting on top of the rack. I get pretty good signal here and get the rated speed, of around 300Mb/s down and 20Mb/s up. This costs $50/mo with no contract and no data cap, and serves as a backup to my main AT&T Fiber connection.

Here is the AT&T Fiber ONT, on the other side of the room. This is 1Gb symmetrical with no cap.

Both connections are in a Gateway Group in PFSENSE, so I always use the AT&T Fiber, until there is an outage

The AT&T connection is in passthrough mode so I can do inbound connections, however the Verizon connection is not. I found that the IP changes very often, and PFSENSE can sometimes get hung up on that, and cause the connection to not work. So I left it in its default configuration which gives PFSENSE an internal IP. Inbound connections are not important on the Verizon connection, however I do still need inbound remote access VPN to work, as well as a connection for some security devices.

For that to work, I have a site-to-site Wireguard VPN setup to a VPS in Linode. Because my local PFSENSE box is the client, and the VPS is the server, I essentially bypass any NAT. This is very useful if you are stuck with an internet connection with CGNAT. On the VPS, I just have forwarding enabled so I can access my local network, and then I have NGINX Proxy Manager forwarding specific ports all the way through to my home network. This means I use the same IP or DNS record to get into my network no matter what goes on at home, as the tunnel just uses whatever gateway PFSENSE tells it to.

This is also how you are probably getting to this blog, which is hosted at home. More details here

You can read more about the 5G setup here

Below the firewall is my main 1Gb Switch. This is a Dell X1052P which handles the majority of the 1Gb connections in my house, and also all of the PoE in the house. This switch Does PoE and PoE+. I am using that to power AP's, some small network devices and a lot of IP cams.

All of the blue connections are the IP Cameras on the house, connected in their own VLAN (I have more connected to the garage, coming up later)

This witch has a 20Gb LACP trunk that connects back to my main 10Gb Switch. I am using Cisco DAC's for this for no other reason than I ran out of spare transceivers when I setup the connection.

Overall the Dell X1052P is a nice switch in that it has 4 x 10Gb connections, PoE+, sliding rails, good quality fans etc, however the software is TERRIBLE. There is no way to configure via CLI, and the Web interface is the worst I have ever used.

Under that switch is a Cisco SG300-28 switch. This switch is here simply because I started running out of 1Gb connections on the Dell X1052P. I have a bunch of low bandwidth things connected here, such as IPMI for servers, Printers, etc. This is connected to the Dell switch with a single 1Gb connection. On this switch I also have the second port of the Verizon Gateway connected on a separate VLAN. This lets me connect a device DIRECTLY to the Verizon internet, bypassing PFSENSE. This is useful for a Ripe Atlas network probe that we will get to later, and makes troubleshooting very easy. Its also good for testing external connections, as it completely takes my internal DNS and firewall rules out of the question.

Below the SG300 is my favorite switch, the Cisco SX350-24F. This is a 24 Port 10Gb SFP+ switch, with 4 of the ports being combo ports for copper. This is my "Main" switch.

This switch connects everything that needs 10Gb. There is Dual 10GB connections to my NAS, one of my ESXi server, single 10Gb connections to mine and my wife's desktop PC, the Firewall uplink, the 10Gb connection back to my garage rack and the 20GB LACP trunk to the Dell switch. I am also using some WiiTek Tranceivers to get 2.5Gb connections to my HTPC. Even though the switch only support 1Gb and 10GB, the Transceiver figures out the rest to let you have Multigig support. You can see that connection here on the far right.

Below the networking are 2 shelves, one with lower power devices, and below that my three VMware ESXi hosts.

Starting on the top row we have the following

Hubitat Hub
Raspberry Pi 3B+ NTP server with GPS/PPS time source
A new Ripe Atlas Probe connected directly to the Verizon Connection
Another Raspberry Pi, this time a Pi W Zero with adapter board, also running NTP with GPS
An older Ripe Atlas Probe connected to the AT&T Connection

The Hubitat hub has now almost entirely been replaced by a HomeAssistant virtual machine. The only reason it is here is to connect the Z-Wave and Zigbee devices I have. However, I replace those devices with WiFi versions wherever possible. In general, I'm not a huge fan of the Hubitat Hub anymore, and I've had quite a few problems with it. Its configured to just pass the devices into HomeAssitant, so its just a radio at this point. The devices that I still need it for are important though, like Smoke Detectors and leak detectors.

The Ripe Atlas Probes are very cool, and generally just help the internet in general. Here is their website with more information

RIPE Atlas docs | What is RIPE Atlas? | Docs

RIPE Atlas Docs

Docs

The NTP servers I have made an entire guide on, which you can read here. They give all the devices on my network super accurate time, even if there is no internet. I have one connected to the Dell Switch, and one to the Cisco switch.

And if you were wondering how I have a Pi Zero with ethernet, you can read about that also

I have however put them in new cases, the HighPi Pro. These have plenty of space to fit the GPS modules.

Here you can see the antenna connection on the side

I just drilled a hole the same size as the connector and connected it. I put some hot glue on the back to stop it rotating when tightening also

For the Pi Zero and the other Pi's, I just used some generic standoffs to fit the GPS modules to a mounting hole where I could

The Pi Zero build has the wires soldered directly to the module saving space

But even the ones with the large connector cables fits well

The GPS antennas I am using are magnetic, so they just stick on top of the rack.

Almost everything on that top shelf including the Pi's are powered by this Anker USB power supply that I have had for years.

https://www.amazon.com/gp/product/B00P936188/

There is one Pi on the shelf below, and that is an early TinyPilot device to give my NVR Machine remote access. Its a Raspberry Pi 4 with a custom USB interface and HDMI capture, so you can get BIOS level control of the machine. This is needed as my NVR Server (We'll get to that) does not have any remote management.

Moving over is the start of the VMware ESXi hosts. All are running ESXi 7.0 U3, as I have not yet made the jump to 8.0. The Move from 6.7 to 7.0 was full of bugs, so I am going to stick on 7.0 U3 for a while. I will detail all of the VM's I am running in another post as its quite extensive, however here is a quick list of services they host

Veeam Backup, Borg Backup, Arq Backup, HomeAssistant, HomeBridge, NGINX Proxy Manager, vCenter, Rekor Scout/OpenALPR, Grafana, InfluxDB, LibreNMS, General use VDI VM's, Test VM's, Windows Domain Lab, Portainer, Kiwix, NextCloud, Mealie, Navidrone, Solarr, PLEX, public SFTP Server, Netbox, qBitTorrent, Prowlarr, This blog, and a few other services.

The first is an ASUS PN50 with 64GB of DDR4, 2TB NVMe storage and an AMD Ryzen 4800U which has 8 cores and 16 threads. This is really a beast of a CPU for such a small box, and it handles all of the compute heavy VM's on my network that don't need fast networking. The main issue with this host is that the onboard NIC is not supported in ESXi, so I have a 1Gb Intel M.2 NIC sticking out the back. I tried to get a 2.5Gb NIC working, however I never could.

To the left are my 2 Lenovo ESXi hosts. The bottom is an old M73 Tiny with just 16GB of RAM (The max it will take) and an old i5-4750T and a 1TB SATA SSD. This host handles some light VM's, however it needs to be replaced, as the NIC will no longer be supported by VMware in 8.0

Above is the Lenovo M720q Tiny, which has an i7-8700T, 64GB RAM, 2TB NVMe Disk, and best of all, an Intel X520 Dual 10GB NIC in the PCI-E Slot. This host handles all of the network heavy VM's. This is what I should have got instead of the Asus PN50.

Here is a shot of the NIC. Its a standard PCI-E NIC, with the IO cover removed. I put some Kapton tape on the NIC to make sure it doesn't touch anything it shouldn't. As you can expect a full size NIC in a system this small is quite a squeeze. Overall this small system has 21Gb/s of total bandwidth available to the network once you factor in the built in 1Gb NIC.

Below this we have 3 x 2u Supermicro chassis. The top used to my ESXi server, however now is just a disk shelf. This disk shelf contains 12 x 8TB SAS disks that are my main media storage and connects back to my NAS below it. Nothing important is stored on here, and its in a big RAIDZ2 array.

You can see here that the control lights are not lit, and the power button doesn't even do anything.

The IO shield is just a blank

And other than power cables, the only things connected are these Mini SAS SFF-8088 cables that run down to my TrueNAS box.

To get the system to power on, I just jumpered the PSU connector with a paperclip and taped it in place. Power control is not important, as this will all stay on 24/7/365. There is a better way to do it, which is to get a Supermicro JBOD control board like the SuperMicro CSE-PTJBOD-CB2. However at the time of deploying this system, they were out of stock EVERYWHERE. Even now, they seem to only be available from China. So I just skipped it.

Below the JBOD is an almost identical Supermicro Chassis, but this one is my NAS. It runs TrueNAS on a Xeon E3-1270 V5, 64GB ECC RAM, a dual port 25Gb NIC and a few HBA's, here are the full specs

For storage, the LSI 9207-8e (e for external) is connected to the 12 x 8TB SAS disks. The LSI 9300 is connected to 6 x 4TB SAS disks in striped mirrors for my important data. There is also 2 x Intel S3700 DC 800GB Enterprise SSD's for Metadata storage (Also called a Fusion Pool)

Since building it, its been completely stable, and I wouldn't change much about the build. Sadly I don't have many pictures of the inside, as its so important to my network I can't power it off to get some nice pictures!

You can read more about my storage config in this post. Nothing much has changed other than the replication destination. It now replicates to another NAS in the garage.

Below that server is my Blue Iris NVR running in an 8-bay Supermicro chassis. This has an i7 8700K, 16GB of DDR4, an NVIDIA Tesla P4 (For AI) and an LSI 9207-8i for the storage disks. I have around 48TB of storage for recording, and it does local AI to alert me to people walking on my property etc. Networking is handled by an Intel i350-T4 quad port gigabit NIC, with one port on the LAN, and one on the CCTV VLAN. This lets the server access the cameras directly, as the CCTV VLAN is completely firewalled off from everything and the internet.

This is the server that I use the TinyPilot for, as its a consumer board, it has no remote management.

Below that is my PDU's and my UPS. The PDU's are CyberPower PDU15M2F10R Metered PDU's, and I've been very happy with them. The front outlets are handy for quickly plugging something in that needs UPS power.

The UPS is an APC SRT3000RMXLA double conversion/online UPS powered from a 30a 120v receptacle.

Here you can see everything plugged in to the PDU's

The large orange cords actually go go back into the wall, where they power other parts of my house, like my desk, my wifes desk and the 2 locations we have TV's. This lets me use a single UPS for all of the critical devices in my house, and not have to worry about changing batteries in a bunch of different UPS's

I made a detailed post about that here

For a while I was able to run everything in here with no cooling, however adding more devices and more disks made the room way too hot. So I added this AC Infinity Fan which sucks the air out into a spare closet behind this one. The room its attached to we just use as storage, so the extra heat doesn't matter too much

Here is a closeup of the fan. I still need to neaten up the hole

The temp probe the controller are mounted in front of the rack

Along with that is my Vertiv/Giest Watchdog 15-P, which is a PoE Powered enviromental moniotor with the ability to send alerts if the room gets above a certain temp or humidity level. I have it to alert on high temp, and low humidity

Here are some of the charts it can give you directly on the device. However, it also support SNMP so you can get the data into whatever monitoring solution you are using.

Next we can move into the garage. In the patch panel you can see this run of SMF (Single Mode Fiber) This runs in conduit to the garage to provide networking there

It runs though this breezeway, along with the AT&T Fiber to the garage on the right of the picture

Here it exits the breezeway in the conduit. The AT&T Fiber goes left, and the SMF keeps on going to the rack

The AT&T Fiber goes down into the wall here on the other side

And into their box

And onto the pole

You can see here why I have the backup Verizon internet. As having that thin fiber run below 80ft oak trees, in a place that gets a lot of thunderstorms, isn't great. Knock on wood, I've never had a problem though (My neighbor did though!)

The SMF runs along the back of the garage to the rack, along with other ethernet runs that terminate in this rack for things like AP's and IP Cameras.

I located the rack in front of my truck, so use some of the dead space

The rack is a small 12u rack I got for free, mounted to the studs. Sadly the bottom U isn't actually usable though for anything with depth, so its really 11u.

In this rack I have another patch panel, a Cisco WS-C2960S-48LPD-L switch which has 48 x PoE+ 1Gb ports, and 2 x 10Gb SFP+ Ports.

Below that is an APC Surge Protector PDU, and then my backup TrueNAS server built from scrap parts, and using 4 x 8TB SAS disk in RAIDZ2. The only thing this NAS does is receive replicated snapshots from my main NAS in the house. So if something were to happen to the NAS, I could instantly get the data from here instead.

There is more details on that NAS here

Since the NAS is using an old consumer board, it has no remote management. So this new TinyPilot device comes in very handy here

I have another Raspberry Pi with GPS running here too, and the antenna is on top of the rack.

The UPS is an old SMT1000RM2U which has plenty of capacity for this small rack and devices. It sits at about 20% load at most.

I am using the same tactic of giving UPS power to other devices through a power inlet here too. It powers my IoTaWatt Energy monitor, and a Raspberry Pi located inside my standby generator (Also where the run of MMF goes you can see in the above images)

More details on those here

Up in the rafters there is also an antenna mounted for ADS-B. This connects back to a Pi with a SDR (Sofware Defined Radio) to feed ADS-B data from planes back to FlightAware, FlightRadar24, ADSB-X and a few others

Here is some live data. It would be better if I mounted the antenna outside, which is a project I will be taking on soon. The location of these planes is being reported entirely over ADS-B from the antenna, which is think is pretty neat.

There is also a WeatherFlow Tempest Receiver in the garage rack. This connects back to my Weather Station which is in the back yard on a pole up high

Using MQTT I am able to get the data locally into HomeAssistant without using the internet, so I can have automations based on the sensors

As for WiFi, I have access points spread around the house and garage. I am using 4 x Ruckus AP's. Here is an R510 on the ceiling in the house as an example.

And here is an R320 mounted outside on the side of the garage (Note, this is not an outdoor AP, but its been there for over 3 years now working fine)

The Ruckus AP's don't look nearly as nice as the Ubiquiti AP's, but I've found they work MUCH better. Since I've installed these over 3 years ago, I've not had to tough the config at all apart from new SSID's etc. They just work. I am using them in Unleashed mode, which lets an AP act as the controller for all of the AP's, and doesn't require a controlled of any kind.

For Power, on the outside of the garage is the ATS for the generator which gives me backup power, and my main electrical panel(s)

The generator is a 27kw Generator which powers everything in the house. This means the UPS's only need at most 10 seconds of runtime, as that's how long it takes for the generator to start and switch

There are some future projects I am working on getting setup, like permanently installing one of these Meshtastic LoRa T-Beam's to get encrypted mesh communication even with no internet.

With one at home and one on me, I was able to communicate around 2 miles just with the stock antennas, and no height advantage. If I get a better antenna and mount it up high, I will be able to communicate throughout the entire neighborhood with no internet.

I currently have one setup as a repeater sitting on my desk. Configuration is much easier now they have made the WebUI 100x better. The device connects to my network over WiFi

I think that's all!