Moving my Blog off of Linode and back Home (Sort of)
At the end of 2021, I moved my blog into Linode. The main goal for this was to keep it up in case of disaster or outage at home, and utilize the much better networking of Linode. Here is the post:
spookyghost
Its been almost 2 years, and I'm already running into problems. The networking I have no complaints with, the latency is great and generally there has been no issues. But, there has been some other major issues which I will detail below.
Cost / Akamai Merger
Back in February 2022, Linode announced it was being bought by a much larger public company, Akamai. There was understandably a lot of concern about this, and people were worried they would come in and raise the price. Well, for me that is exactly what happened. The VPS I rent increased in price by 20% right away, with no additional features. This was frustrating, but I could understand it as everything gets more expensive, but 20% does seem like a lot. What I couldn't understand is that the lower tier VPS, the Nanonode, didn't increase. So I decided to slim down what I was storing on the VPS, and move to that. I ended up reducing my bill to Linode by 58%. If they were looking for extra money, they didn't do very well. But then again, I'm just a very, very, very small drop in the bucket so I'm sure they don't care. Either way, it was a frustration. I can now rent a VPS from Vultr for less money for the same features.
Backups
I pay Linode $2 per month for backups. Since I pay a total of $5 for the entire Nanonode VPS, its quite a lot of money for a very basic backup service. But, it always worked. it keeps 3 daily backups and a manual snapshot that you can take at any time, which overrides the last. The few times I've restored a backup, its always worked.
But, my Linode VPS was running Debian 10 which is getting quite old now. And I already upgraded all my home systems to Debian 11, so I wanted to update the VPS too. I switched the repos and upgraded, but I ran into some problems with my site to site tunnel, and I didn't have time to troubleshoot it, so I restored a backup.
The first problem is that the backup takes a good 30-40 mins to actually restore. Given the entire disk is 25GB, and not even full, this is pretty poor. If I had the VM at home in ESXi, I could restore the snapshot in less than 2 seconds.
The second problem, is that the backup reports that it fails! It never used to, but now does. It looks like I'm not the only one with this issue
temabu
So I wasted about 4 hours retrying restores, trying older versions etc, before realizing the backup did restore, and everything seems okay. Of course I have zero way of knowing if everything is fine, since the backup reported it failed. Could I be missing data? Maybe.
I contacted support, and got probably the worst responses possible.
After this interaction, I voiced my displeasure and the ticket was transferred to someone else and the support did improve, but this left me with a very bad feeling. I clearly cannot trust the backup service, or support.
Performance
While all the shenanigans were going on above, I decided to just ditch the Debian 10 VPS, and deploy a new Debian 11 or 12 VPS, and slowly migrate workloads and configuration. This would also give me a chance to fully document the setup, which I didn't do first time around!
The load on the current VPS sits around 2-4% CPU and has no problems, so the new VPS I just sized the same. Here is the CPU load on the old VPS, including the move of all the data to the new VPS which is the spike you see.
If you are using docker, you can pretty much just copy the data directly to the new VPS with SCP, and then copy over the docker compose config, and start the container like nothing happened.
scp -r -p /media/ghost [email protected]:/media/ghost
I moved just my Blog over, and not any of the other containers, and right away the CPU was spiking to over 120% and rending the entire VPS unusable, and the site wouldn't load. Keep in mind this VPS is sized identical.
Looking at HTOP on the server, shows a very dire situation.
I contacted support and they gave me some documents to read over and confirm if the issue was on my end, or theirs, and if their they can move the VPS to a quieter host.
But, this paired with the backup issues got me wondering if its really worth the hassle. I could spin up a VM at home and easily handle this workload. all I'm missing is the good networking from Linode.
The Plan
The plan I came up with is to make a new VM at home for running the blog, but keep NGINX Proxy Manager in the VPS, and keep the site-to-site Wireguard tunnel in place, and just have it served right across the VPN. This would let me use the great networking of Linode, while using the great compute and storage at home. And backups can be handled by Veeam and I can then utilize VMware Snapshots again.
Another big reason for keeping the hosting via a VPS is that I have redundant internet at home, and I don't want to have to have another set of NAT rules for the secondary internet. The secondary Verizon internet also doesn't work too well in Passthrough mode as the IP changes ALL THE TIME. The site to site Wireguard tunnel uses the VPS as the server, and my PFSENSE firewall as the client, so it really doesn't matter what goes on at home, it will still get to Linode as the tunnel will just move over to the secondary internet if the primary goes down. This same tactic can be used if your home internet is behind CGNAT. Because the Linode VPS is quite geographically close to me, the latency is under 10ms, and the routing to Linode from readers is always much better.
To backup the point about the better routing. If I stand down the street from my house using Cell data on my phone, its faster to connect using Wireguard to my VPS, and have the data then cross the site-to-site tunnel back to home, then directly connect to home via Wireguard. I guess the Linode DC just has better links than whatever AT&T is doing internally on their residential services.
I made the VM on one of my new lower power ESXi hosts, an ASUS PN50, and as you'd expect, the load is minimal. I gave it 4 CPU cores expecting some more load, but that really wasn't even needed. The storage is all NVMe backed with minimal load, so the storage latency is minimal also.
I deleted the new VPS, and went back to the old Debian 10 VPS which performed much better. But now with the blog at home, I can ALSO serve it via NGINX Proxy Manager at home, and just round robin the DNS. So I added a second A record in CloudFlare for the blog to my home address, and again utilized the CloudFlare Proxy Services. Now when you do an nslookup, you get 2 addresses. They both get proxied through CloudFlare, but one ends up at the Linode VPS, and one at home.
This lets me turn off the VPS, and you can still access the site by just coming in directly on my home internet connection.
The fact that I could now access the site completely from home without the VPS let me spend some real time troubleshooting the upgrade, and in the end I was able to get the upgrade to Debian 11 done.
Now the VPS just runs NGINX PM and Wireguard, it sits almost idle all the time.
Since my blog is at home, I'm in total control over the data and backups. Of course since I now upgraded the "good" VPS, I could move it back and be done with it, but I am much happier with this new solution I think, and if Linode moves this VPS to a busy host like the other VPS I rented, I shouldn't have the CPU load issue again. I can also add as much storage as I need to, instead of having to pay substantially more for a larger VPS.
Reliability
One of the main reasons I switched to Linode is for better reliability. At the time I switched I had 1 internet connection, only 1 large ESXi host, and 1 source of power (The grid). That led to the blog being down when we got hit with a crazy winter storm.
But now, I have redundant internet, a standby generator and three ESXi hosts so I can move around workloads if a host fails.
If I think I'll need to take down the VM at home, I can easily just copy the files back to the VPS and spin up the container.
My next plan is to spin in a Vultr VPS, and start testing. Their backup solution looks much more robust, and they are now cheaper.
Hopefully this was an interesting read. If you are are planning on hosting a website, maybe give this idea a go.