Blue Iris NVR - Off-site clip syncing
Blue Iris is a great NVR solution, but its extremely lacking as FTP is its only native off-site clip exporting protocol. Here is how I made that work for me
I have been using Blue Iris for my NVR for over 6 years, and despite FTP being a legacy protocol the entire time, Blue Iris has not listened to the overwhelming cry for a modern, secure and native way to get clips off-site. At the very least SFTP would be great, but even better would be S3 compatibility. This would let you use a huge selection of cloud object storage platforms to get your clips off-site in a fast, secure and cheap way.
If you just want the guide, skip to the end!
Why do you need clips off-site?
Its pretty simple, what if someone kicks down your door and steals all your stuff, along with your NVR? Then you will not have the footage of them arriving and breaking in which could be used to help identify and catch them
Many people try and combat this by having motion emails with screenshots included, however I feel like this is a pretty poor solution as then you only get a few snapshots of what is going on. I want ALL of the clips!
Why FTP Sucks
Currently almost every other camera system supports some kind of "instant" off-site storage, usually at a cost, but at least its possible and at the very least its secured with SSL in transit. With Blue Iris, you are stuck using un-encrypted FTP
The main issue here is clearly that all of your data is transferred across the internet completely un-encrypted, since we are transferring video footage of your house, and potentially inside your house, you should be able to see why this is a problem.
The other issue is that almost no cloud storage providers support FTP. If they do, they are usually very expensive. If you want to get clips off-site in FTP, you almost certainly need to spin up some kind of virtual server in the cloud, which is expensive
Using non-native file syncing
You could just use an application outside of Blue Iris to sync the clips off-site, you can point the syncing application to the folder that Blue Iris uses for clip storage, and have it sync either on a schedule or just as changes are made. In theory this should work great, but I found that due to how Blue Iris uses the .bvr files, it would often end up with incomplete clips being synced off-site, resulting in a mismatch between what was on-site and what was off-site. Every week I'd do a comparison on the number of files synced, and the total size, and they were always different. I tried SyncBackPro, Syncovery, robocopy and a few others, all of them were a pain to setup, and just not very nice solutions to the problems. Some of them were also paid applications, which I just couldn't justify for how well they worked in this application
The other way to do this is to have your clips stored on a file server on a Linux box and use rclone to sync the clips off-site, I tried this and had the same issues as above, it just can't tolorate syncing a .brv file while its being written to, because Blue Iris doesn't lock the file (Which results in you being able to play a currently recording clip) none of the sync applications seem to figure out when to start syncing
The way I found this to actually work is to have the clips on a Synology NAS and use Synology Cloud Sync. This worked VERY well and seemed to just be able to partially update the file on the other end when the clip finished recording. The downsides to this are that you need to purchase a very expensive Synology NAS, and that you now need to store your clips off of the Blue Iris box. I personally like having the clips directly on the box so that my cameras are recording all the time, even when I am doing maintenance and updates on my other network devices. There doesn't seem to be a very good way of getting Cloud Sync to see external volumes or file shares, you could get it to work, but it never synced as well as having the files directly on the device
That then leaves you with using FTP...
A quick overview
The general idea with my setup is that my home is secure, and my NVR is in a location my in home which is also secure. This means that the clips will 100% have a chance to leave their location before someone could get to the NVR
Next, I have an FTP server on my local network, but in a different (also secure) physical location where the clips are synced. From there, the clips are stored locally and uploaded to cloud storage. Because of this setup, even if you have slow internet, this should work. And, if the internet is completely down in some kind of natural disaster, you will still have a second secure copy of your clips (Making it far superior to cloud NVR solutions!)
Prerequisites
There are a few things that come into play here, here they are before I show off my solution. If you don't get these right, then someone could still steal your NVR before the clips get off-site
Good home security
First is that you really should secure the location your NVR is in. First your house should be as secure as possible, and then make another layer securing the room or location the NVR is in. For me this is quite simple as I have a dedicated room for all my networking equipment. This is especially important if you are not syncing the clips off-site, but if you are you just should try and aim to keep someone at bay for as long as it would take to upload the clips to the internet, with some cheap security measures this is easily possible, and hopefully you are doing this anyway...
Good Internet/Networking
The second thing you need is good internet. If you want to sync all these clips off-site, it needs to get up there very fast, because you want the sync to finish before someone can get to the NVR. You will almost certainly need better internet than what some of the IoT based NVR solutions require, this is because while they might have 1080p cameras, or even 4K, they completely turn down the bit-rate of the footage resulting in much smaller file size and worse video quality. Its worth noting here that a high bitrate 1080p video will almost always look better than a low bitrate 4K video filled with artifacts and other compression noise.
In the title I said good networking, this could be an alternative to good internet. If you can find another secure place on your property you could store the clips (Maybe a hidden compartment, attic, shed, garage, friends house) then you really only need good networking to that point, preferable a hardwired Ethernet connection, but even modern WiFi should work fine. I am actually utilizing this approach as well as syncing to the internet. (I won't tell you where, because it could be used against me!)
A Linux compatible system
For the solution I have put together, it requires a Linux OS. You can probably use Windows though, it would require more resources and may be a little more complicated though
Luckily just about everything support some flavor of Linux, and pretty much anything running Windows will support Linux. There is really no limit to what you could use, the only requirement is that it has semi-decent networking (Hopefully supporting at least 50Mb/s write speed) and enough storage space for at least some clips. It could be a VM, it could be an old PC, it could be a Raspberry Pi, it could be a VPS in DigitalOcean, it could be an AWS EC2 instance, whatever you want.
My Solution
Here is how my solution works. I am going to omit some information as it could help someone either break into my house, or circumvent this solution! I personally think this is a great solution, and if you follow along this should even work for you if you don't have great internet. Because the clips get copied to a secure intermediate zone, they can take longer to get to the internet if they want even if the NVR is gone (Assuming you still have internet, though)
Because of a script I am running on the machine, the FTP server can have any amount of storage you like, and it will not be a problem. It will automatically scale back the retention based on the amount of storage available. This is critical as from day to day the storage can change a lot. With some days having just 20GB of motion clips, to other days having over 140GB
NVR Location
The NVR is located in a secured room in my house where my network rack is located. The door is solid wood, and its locked. The hinges and strike plates are secured with 3-1/4" Deck screws which go directly into the frame of the wall. This means just kicking it open would be hard.
FTP Server Location
Next, I have a computer in a secondary location which is directly on my network. For you this could be a shed, garage, attic, hidden room, inside a wall, wherever you like. This runs Debian and hosts an FTP server which Blue Iris connects to.
FTP Server Configuration
I won't go into how to configure an FTP server here, but if that's of interest to you let me know and ill make the guide. Here are the tools I am using
- Debian 10 as the OS
- ncdu to monitor file sizes and usage
- vnstat to monitor live network traffic, and get daily totals and averages
- rclone to upload directly to BackBlaze B2
- tmpreaper to manage retention of the clips since Blue Iris only uploads, and does not do anything else once they are sent over FTP
I have a script that runs every 5 minutes via cron which runs tmpreaper to clean up the clips. Here it is. As you can see, it checks the used space on the nvr_encrypted mount, and acts accordingly. Feel free to take this code and modify it to your needs
If you are going to use rclone, be sure to run the rclone config first!
#!/bin/bash
## Functions
check_space () {
used=$(df -Ph | grep '/dev/mapper/nvr_encrypted' | awk {'print $5'})
}
## Prune Clips so only the last 24 hours remains
tmpreaper -m 24h /mnt/nvr/synced
## If used disk space is 90% or greater, prune back to 23 hours
check_space
max=90%
if [ ${used%?} -ge ${max%?} ]; then
tmpreaper -m 23h /mnt/nvr/synced
sleep 5
fi
## Doing the above should have netted you at least an extra percent, if it didn't we need to be more agressive and do 20 hours
check_space
max=89%
if [ ${used%?} -ge ${max%?} ]; then
tmpreaper -m 20h /mnt/nvr/synced
sleep 10
fi
## No change? Lets go even further to 18 hours
check_space
max=89%
if [ ${used%?} -ge ${max%?} ]; then
tmpreaper -m 18h /mnt/nvr/synced
sleep 10
fi
## Still no good? Lets go to 15 hours
check_space
max=89%
if [ ${used%?} -ge ${max%?} ]; then
tmpreaper -m 15h /mnt/nvr/synced
sleep 10
fi
## Things are getting bad now, the above didn't fix the problem and we are all the way at 98%. Trim to 2 hours of clips and alert
check_space
max=98%
if [ ${used%?} -ge ${max%?} ]; then
tmpreaper -m 2h /mnt/nvr/synced
sleep 10
fi
## Really?????? Lets delete everything in the NVR folder
check_space
max=99%
if [ ${used%?} -ge ${max%?} ]; then
rm /mnt/nvr/synced/*
sleep 10
fi
## Upload clips to BackBlaze
rclone sync --fast-list --transfers 32 /mnt/nvr/synced backblaze:rclone-nvr
As you can see at the bottom, it then uploads the folder to Backblaze
I have been testing this out for a few weeks, and everything just works. I can never run out of disk space, everything just keeps on chugging along
I get around 600Mb/s average on each upload to B2, which I think is pretty good since its just small files. I often see 800+ on the upload
Blue Iris Configuration
In Blue Iris I have configured 2 things, first is that each camera will take a jpg snapshot every X amount of seconds, and upload it to FTP, and then ALL motion clips are sent over to FTP as well. The reason for the JPG clips is that if someone steals all of my stuff, it may be hard to get Blue Iris installed on something fast to view the clips, so the JPG files are a nice addition
First I configured the FTP Server in Blue Iris
Then I added a new temp storage location for FTP uploads, this is only used for the JPG screenshots
As you can see, the Queue for backup is checked, and configured
Now that you have done that, you can set your motion clips to also be backed up to FTP using the exact same setting as above, but for the storage location that you have your Motion clips saving to
For the JPG screenshots, edit each camera and add the screenshots. See my yellow highlight
Thats all
Conclusion
I think this is the best way to get your clips off site, hopefully this helps someone out. If you find a better way, let me know!
If you wanted to make this even better, give the FTP box a secondary LTE backup connection, and then as long as that box is in place, it will upload no matter what the rest of the network is doing