Archiving data to LTO Tape for long term storage and backups
This is the second part to my data archiving efforts, the first part was using M-DISC's which you can read about HERE
The Tape Hardware
I have been looking for a tape drive or library for a long time. Eventually I saw an IBM TS3100 with 2 x LTO5 SAS Drives listed on eBay from a seller who clearly didn't sell technology items usually. It was listed as untested, so I took a chance. I made a best offer for $200 and won.
The tape library arrived in great condition, and it was full of tapes!
This tape drive has a dedicated network port so you can access the WebUI and view and configure the library, this is a much better alternative to using the small screen on the front of the unit
If you are looking to purchase a tape library, unless you get a crazy unrealistic deal on something newer, go for LTO6 or LTO5. LTO7 is just too expensive on the used market right now, and both LTO5 and LTO6 tapes are very cheap, and have reasonable capacity. LTO4 lacks LTFS support, so I wouldn't recomend it. Generally LTO5 SAS tape drives are new enough to have an SF-8088 MiniSAS connector which is very easy to work with. You could also go FC, assuming you don't mind tracking down cables and NIC's
The System
I didn't want to build a whole new server just to mount the tape drive, so I decided to make a VM on my ESXi server which would be dedicated to archiving. In the ESXi server I have an LSI 9207-8e which is passed through to the VM, this lets me access the tape drive directly in the VM. Through the mess of cables you can see the two SF-8088 MiniSAS cables plugged into the HBA
I also wanted some completely seperate storage to use as scratch disks. For this I passed through an LSI 9207-8i which is connected to the main backplane in the server, as well as the two rear 2.5" SAS bays, as well as a dual eSATA card I installed (More on this in another post, but I use it to connect my BluRay optical drives)
In those two bays I installed 2 x SATA 800GB Intel Datacenter SSD's and configured them in RAID0 in Windows. Since they are just for temporary storage, I am not worried about data loss, instead I want max performance so I can write to two tapes at once, while still manipulating data. At over 1GB/s read and write, this is no problem (GigaBYTE not gigabit!)
The VM itself is running Windows Server 2016 Standard. Windows Server 2019 is not supported with the IBM software I need to use. I gave it 16GB of RAM and 16 cores so it has plenty of power to encrypt and compress file. I have dual E5-2680 V4's, so I have no shortage of CPU threads
The Software
As above the OS is Windows Server 2016 Standard, for my use case this was the best choice
To get data on the tapes I am using plain old LTFS, this simply requires me to install IBM Spectrum Archive Library Edition which will let me mount the entire library in Windows so I can copy data. As you can see, the tapes just show up to copy data to once you have set it up
To install the software you must install Python 2.7 and add it to the command line path (https://geek-university.com/python/add-python-to-the-windows-path/)
From there install C++ 2010, and finally the IBM software. If you have a problem installing it, make sure your OS is on the supported list!
Once you have it installed, go to the LTFS directory in command prompt
cd C:\Program Files\IBM\LTFS
Then run ltfslsdev to see the device name of the library which will show up in number form.
Now mount it as a letter, I am using L in this case
ltfs -o changer_devname=2.0.0.1 L
Once it mounts, you can scan for your tapes
python leadm tape list
Now format your tapes to LTFS, you can list as many tapes as you like
python leadm tape format XXX036L5 XXX037L6
Once they are formatted, you can assign them to the LTFS volume if they are not already
python leadm tape assign XXX036L5
Now you are ready to copy data to the tapes
I decided to archive my data using WinRAR and split it into 100GB chunks. This makes it managable, but still uses large files which is much better for writing to tape. I used the fastest compression option, and encrypted with AES-256
The files copied over quite fast
To actually copy the files over apart from this test, I used robocopy which is much more stable than Explorer. I have had explorer crash in the middle of a 400GB file copy, which was annoying to say the least
To copy to the tape I used the following: (The D: is my scratch SSD array)
robocopy D:\Tape L:\XXX036L5 /Copy:D /MIR /W:10 /R:10
And to copy from the tape I used the following
robocopy L:\XXX036L5 D:\Tape /Copy:DAT /MIR /W:10 /R:10
The end result is all of the RAR part files on the tape, along with the WinRAR install file which I thought could come in useful in the distant future if I couldn't find a copy
To remove the tapes you just run leadm tape unassign XXX036L6 to unassign it from LTFS and unmount it
Then run the following to move it to the I/E Slot so you can remove it from the library
python leadm tape move -L ieslot XXX036L6
Now you can walk up to the library, unlock the I/E slot from the screen and remove the tape
If you want to clear a tape so all of the contents are remove, you format but with the -f option which if force
python leadm tape format XXX021L5 -f
Tape Identification
I wanted to make some custom barcodes for my tapes, as well as some nice covers for the LTO cases. Forgive me for blocking out personal information on these tapes
If you are going to rotate the tapes out, you may not want to print something so permanent on the labels, as this tape is now forever "Yearly Archive 2019" unless I peel the label off and make a new one, but I plan to store this for a really, really long time
For the covers, I just made my own as I could not find any. You can download it here
For my custom labels, I used the below website to generate them
And then I got some Avery 6577 Label sheets for my printer, they work great. For the above website I did have to do a +1.4 vertical offset for them to line up using the 6577 option in the dropdown. I emailed the dev, so this may not always be true! Always print on paper first and try and line them up. I use a flashlight underneath to see if the lines are in the correct plate
Conclusion
Hopefully this was an interesting read, and will help you get into LTO Tape. I am 100% sure people will criticize the use of RAR files as WinRAR is not free and open source, however it just works better than anything else I have used, and it has a very good compatibility track record. I have no doubts I will have a trouble free time reading the data even in 20 years, so I am happy
This is not for ordinary backups, so not being able to quickly grab one file is not a concern. I have many other backups for that purpose
Thanks for reading!