My overkill 1u pfSense build

spookyghost -

Ever since my Ubiquiti EdgeRouter 3 died a few years ago, I have been running pfSense. I used to run it in a virtual machine under ESXi, however I wanted a physical box. This is what I have ended up with over a few years of changes

This is its current configuration:

  • Supermicro 1U SC510-203B Chassis
  • 1u Supermicro 200w PSU 80+
  • 2 x Supermicro 40mm 8500rpm FAN-0106L4 (4 Pin)
  • Supermicro MBD-X9SCM-F-O Motherboard (Dual LAN, IPMI, all the good stuff)
  • Intel Xeon E3-1220 V2 3.1GHz CPU (Quad core, no hyperthreading)
  • 2 x Crucial 4GB DDR3 1600MHz Unbuffered ECC (8GB Total)
  • Kingston KC300 60GB SATA-III SSD 2.5"
  • Intel i350-T4 Quad Port Gigabit NIC (Currently not utilized)
  • Supermicro RSC-RR1U-E8 Riser Card (For the i350)

Here it is in my rack (Which needs to be re-organized)

Here is the front and back:

And some inside shots:

You may notice the case has some very weird corrosion marks on it. Thats because the chassis was salvaged from a system which looks like it had been flooded. I will make a post on this soon.

This build is extremely quiet and uses almost no power. You could easily place this near humans without annoying them.

I am the Comcast "Gigabit" package which gives you 1Gig down, but only a pathetic 35-40Mb/s up. It handles this with no problems of course, even with ntop packet inspection and Suricata IPS

I have it connecting to Private Internet Access VPN so I can route traffic to there, I also have a Site-to-site OpenVPN link to my colocated server

Here you can see my dashboard

VLAN's

I have my main network on the default VLAN 1, but I have some additional VLAN's

Guest
This VLAN has a 30Mb download limit, and a 3Mb upload limit, and pushes all traffic over the PIA VPN. It has no access to any other network. It has DHCP.

IoT
This VLAN has a 30Mb download limit, and a 3Mb upload limit, and pushes all traffic over the normal internet gateway. It has no access to any of the other network, but LAN does have access to it, and I have Avahi mDNS running here. It has DHCP.

Lab
This VLAN has no internet restrictions, and has full access to the normal LAN network. It does not have DHCP, as I often throw seperate DHCP servers in here

VPN

I have two OpenVPN server running in pfSense, one for TCP/443 and one for UDP/1194

I also have two clients, one to Private Internet Access (Guide), and one to my colocated server (More Info)

Storage

I am using a single SSD with the zfs filesystem. I have tried to use the GEOM Mirror feature in pfSense before, but I found it to be quite unreliable (Perhaps it was just my SSD choice though?) Read more here

As you may have noticed, I am using a RAM disk. I enabled this around 2 months ago after noticing a very high amount of disk traffic (Graph from LibreNMS)

As you can see, its just under 1TB of disk writes per month. While most SSD's including consumer SSD's SHOULD handle this fine, I don't see why its needed when you can completely mitigate it using RAMDisks.

I changed the settings to 200MB for /var and /tmp, and I set it to backup every 2 hours. These settings can be found in System/Advanced/Miscellaneous

Once I changed that and rebooted, the traffic stopped almost completely.

After a few hours...

And after a full month, less than 4GB of writes!

A word of warning about this though, I went ahead and enabled the same option on my pfSense VM on my colo server which only had 1GB of RAM total. It went down for a reboot and never came back... I had to drive down to the datacenter and increase the RAM size to 2GB, and then it booted fine. So it seems like I hit the limit on how much RAM is required for this setup

Hopefully this article was interesting! Let me know in the comments