My All-in-one colocated ESXi server
I have a 1u Supermicro server colocated in a datacenter. I mainly use it for DR, Backups and hosting services which need to be on 24/7/365. Those services include this very blog, and my Nextcloud instance among other things.
The datacenter is only a 10 minute drive away from me, so going down there to upgrade or troubleshoot is very easy. Its a very secure datacenter, and this is on the second floor, so it's unlikely that the flooding that happens in my city would ever effect it.
Overview
For the hypervisor I use ESXi 6.7, and it has the following hardware:
Supermicro SYS-5017C-M Chassis
Supermicro 350w 80+ Gold PSU
Supermicro X9SCL-F Motherboard (Dual LAN, IPMI etc)
Intel Xeon E3-1270 V2 3.5GHz (4C/8T)
4 x 8GB DDR3 1600/1333 Unbuffered-ECC (32GB Total)
8GB Kingston USB 2.0 for booting ESXi
128GB Samsung SM951 NVMe M.2 SSD in a Holchips 1U M.2 NVMe Riser
1TB HP EX920 NVMe M.2 SSD in Generic PCIe to M.2 NVMe
Supermicro RSC-RR1U-E8 Riser Card
A mix of 4TB 3.5" disks
So far the CPU and RAM seems fine for what I am doing, but I do wish I had redundant power supplies to fully utilize the power setup at the datacenter.
Current VM List (We will explain these as we go):
BK02 - Windows Server 2016 Standard
Docker03 - Debian 9.6
Docker04 - Debian 9.6
FW02 - pfSense 2.4.4
Jump01 - Windows Server 2016 Standard
NAS02 - Synology DSM 6.1.2
Myname-VM_DR - Windows 10 Pro
wifename-VM_DR - Windows 10 Pro
Here is the folder layout in vCenter
Here are some pictures of the setup:
The pictures I have of the inside are a little dated, but mostly still good
I only have 1 port to use, so I plugged the IPMI into the LAN vSwitch so I can collect SNMP stats
Networking
You may be wondering how all of this works and how I connect to and from the server, here is how:
The server has 2 NIC's, and I have each of those in a separate vSwitch, one called LAN and one called WAN. I have a pfSense virtual machine that connects to both the WAN and then the LAN to act as the firewall for the entire system
As you can see, all the VM's connect to the virtual switch, but I also have the second port bound to this LAN switch. This is not required, but it lets me plug in the IPMI port to get SNMP stats, and it's also very useful if I need to visit the server physically and need to access IPMI
The pfSense VM serves many functions, including connecting back to my homelab at home via OpenVPN site to site tunnel. This allows me to access the entire ESXi box as if it were local. the pfSense box also connects to PIA VPN so I can route traffic over there if needed, here is a screenshot of how pfSense looks on this box
I give it 2GB of RAM, 30GB of disk and 2v CPU's and it seems to perform just fine. I can route at full 1G with no issue during tests
I also have an OpenVPN server instance setup here so I can connect my phone/laptop etc to here and get secure internet, or to access services not exposed over the internet. Because this box also has a VPN link back home, I mainly just connect here to access anything on my entire network. It's a much better quality connection than I have at home, so it generally works better.
Storage
For virtual machine storage I have a 128GB NMVe Samsung 950 Pro (OEM Model) as well as an HP 1TB NVMe SSD. No redundancy here, but everything is all backed up nightly, and the SSD's don't get pushed even remotely close to their limits
For file storage, I have a Synology DSM Virtual machine setup with SHR with all the 3.5" disks. This lets me have all the nice features of a Synology NAS, without overpaying for the hardware. Since I have a Synology NAS at home I can do seamless syncing of the two with Cloud Station
Here are some screenshots and explanation of the setup. You can see that it thinks its an i3, which is the CPU that is really in a DS3615xs
I gave the system a VMXNET3 as well as an E1000E for compatibility reasons. The E1000E will always work, however the VMXNET3 could stop working with an update (Since this is far from a supported config...) so I added both together in a bond with failover. This gives me 10G+ speed, paired with the compatibility of the E1000E if something fails
I don't have a ton of storage in here right now, but it's enough. As soon as I am done with a disk at home, I will move it out here. Being SHR, I can add any size disk I want
So far this setup has worked VERY well, and I can't imagine any other virtual solution being quite as good. I am able to get full access to the disks from passing through the SATA controller on the board, to the DSM virtual machine in ESXi
Now that you know hows it setup, lets get on to how I use it.
I mainly use this storage for DR and Backup. I use Synology Cloud Station ShareSync to sync (in real time) all the important data from my NAS at home, to this NAS. This means if I can't access the NAS at home due to a connectivity issue or any kind of disaster, I still have full access to my data. I also replicate the clips from my Blue Iris NVR. This means if someone steals my NAS, I still have the footage
The sync is very fast, it's pretty much up to date instantly on the other side
Some folders have a two way sync, and some are just one-way. I try to do only a one way sync unless I really need it, just to be safe
This setup gives me an extra layer of protection, as I also have BTRFS snapshots setup on these folders, as well as all being encrypted on the source and destination side. Of course all the replication traffic is encrypted, as well as going over an encrypted VPN tunnel anyway.
Backups
We have covered the replication, but now to to backups. I use a Windows Server 2016 Standard VM (BK02) which acts as a Veeam Proxy as well as running StableBit Cloud Drive and Arq Backup.
StableBit CloudDrive stores an encrypted volume on my O365 SharePoint storage, and I sync the important data up there every now and then. This just gives me another offsite location for the data. I keep the drive unmounted so nothing can delete the data even if it tried
Arq backs up the replicated data back to a secured, separated folder on the NAS. This means my off-site backups happen almost instantly. I have it set to thin backups, but never delete.
For Veeam, I have ESXi1 (Home) VM's being backed up locally, and replicated over here to NAS02 at the datacenter. I have ESXi2 (Colo) VM's being backed up locally to NAS02 and then replicated over to NAS01 (Home).
In addition to this, me and my wife both have Windows 10 Virtual Machines we use for remote access and just general stuff. These are both replicated to the datacenter every night, so if for some reason we can't access home, we can pick up where we left off. Since all of our important data is also replicated on the Synology VM, we can literally access everything even if our home burnt down.
I am using the 1 year trial of Veeam which can be renewed every year, this gives me all the good features!
Backup performance even to the virtual NAS is very good. Despite having 4 random old disks, I get around 600MB/s write speeds
Docker
The only other systems not mentioned are the Docker03 and Docker04 virtual machines. These both run Debian 9.6 with Docker installed, and Portainer so I can have a nice UI.
Docker03 is my main Docker host on the colocated server, and hosts the following
PLEX - For streaming music remotely
Ghost - This blog
Nextcloud - For cloud file storage, syncing, sharing etc
Portainer - The nice WebUI you see below
Speedtest server - Good for testing VPN throughput
Traefik - A very nice reverse proxy with automatic LetsEncrypt generation
SFTP Server - Used for Camera Upload (Ill touch on this in another post)
The other docker host, Docker04 currently doesn't do anything. However I have firewall rules in place to push all traffic from Docker04 to the Private Internet Access VPN. This means any containers I spin on on that host end up using the VPN
The Traefik reverse proxy has ports 80 and 443 forwarded to it, and it handles all of the routing and automatic Lets Encrypt certificate generation. You simply add a tag in the Docker container if you want it to be proxied, and Traefik automatically see's it and handles the rest.
This means I can access all of my services with a nice valid SSL Cert.
The End
I THINK that covers it, let me know if you have any questions or comments, I will post any updates if I think I have forgotten something!