Self Contained Encrypted iSCSI Datastore in VMware ESXi

spookyghost

• 10 min read
Self Contained Encrypted iSCSI Datastore in VMware ESXi

Hopefully the title made sense. Alternate title could be

  • LUKS Encrypted iSCSI SAN in Debian
  • LUKS Encrypted iSCSI Target in Debian
  • How to share a volume via iSCSI in Debian

You get the idea.

This article is about creating an encrypted ESXi datastore out of existing storage you have in your ESXi server. But the first half is really just how to create an LUKS encrypted LUN in Debian and share it out via iSCSI, so if you just need that, ignore the rest.

The way we configure it will mean you need to enter a passphrase on boot, and from there the system will boot and automatically mount a LUN to be shared out via iSCSI

The reason for me wanting to do this is that I have three virtual machines which contain sensitive data, and really need to be encrypted. If I do full disk encryption on the virtual machines, it blows my Veeam backup sizes up a LOT. Veeam can no longer de-dupe or do changed block tracking. Because I replicate these virtual machines over the WAN to another system in a datacenter with my pathetic 40Mb/s upload speed, this really wasn't something I could give up. The other option is to setup vSphere encryption, but it's a big setup and requires KMS servers, a bunch of messing around with certificates.

I decided to use Debian for this as its what I am used to. I used to use Ubuntu but it took a turn for the worst with version 18, so I made the switch to Debian 9 which has been great. So far its almost 100% the same, but without the crap. So if you are using Ubuntu, these steps should work fine.

How to do it

First, create a virtual machine in ESXi for Debian 9. I gave it 6 vCPU's, 512MB of RAM, an 8GB Thin disk along with a VMXNET3 NIC. The RAM could be even lower, I suspect you would have no issues with 256MB. I added a few more CPU cores because when I was hitting the VM hard with some vMotion the CPU load did ramp up a little, and since I have dual E5-2680 V2's, it means I have 40 threads laying around. I suspect you could get away with 4 vCores with no issue

We will add a second drive to share out via iSCSI later on. So skip that for now

Once you have created the VM, attach the Debian 9 netinstall ISO and boot to it. Select the regular Install option, and not the Graphical install

Now just run through and select all the usual stuff like keyboard layout, timezone, hostname etc. Eventually you will get to the storage options. You want to select the option to use encrypted LVM

Move along selecting the default options and write the changes to LVM. When you reach the point of entering a passphrase, make sure its a good one!

Move along with the setup and write the changes to disk. When you get to the software selection page, unselect everything but standard system utilities

From there finish the setup using the default options, or whatever options you want.

As soon as it reboots, it's going to prompt for your encryption key which it will do from now on

Enter your passphrase, and then log in as root

You'll want to update the system first

apt update
apt dist-upgrade -y
apt autoremove
apt clean

From there we want to install tgt, open-vm-tools, htop, ufw, sudo and the openssh server

apt install openssh-server open-vm-tools htop sudo ufw -y

Then go ahead and add your user to the sudoers group

usermod -aG sudo usernameyouchoseoninstall

Now go ahead and reboot, and then connect to the VM via SSH.

Once you connect, we should render the root account useless since we won't need it anymore. Run the following to set the hash to an impossible value, meaning you can't login. If you do need to login, just reset it or access it via su

sudo usermod -p ! root

Now its all updated and you have the base configured, we can go ahead and add the additional virtual hard drive which will end up being shared out. Shut down the VM

sudo shutdown -h now

Now edit the VM, and add a thin disk of whatever size you want. Then power it back on, unlock it with your passphrase and SSH back in

Now run lsblk to see the location of your new drive you added. As you can see, mine is sdb. which makes the full location /dev/sdb

lsblk

NAME                             MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                                8:0    0    8G  0 disk
├─sda1                             8:1    0  243M  0 part  /boot
├─sda2                             8:2    0    1K  0 part
└─sda5                             8:5    0  7.8G  0 part
  └─sda5_crypt                   254:1    0  7.8G  0 crypt
    ├─iscsi--secure1--vg-root    254:2    0  5.8G  0 lvm   /
    └─iscsi--secure1--vg-swap_1  254:3    0    2G  0 lvm   [SWAP]
sdb                                8:16   0  400G  0 disk
sr0                               11:0    1 1024M  0 rom

Now go ahead and create a volume group, you should see the confirmation output below. I name my volume group "fs-secure_iscsi"

sudo vgcreate fs-secure_iscsi /dev/sdb
  Volume group "fs-secure_iscsi" successfully created

And now create a logical volume. I named mine "fs-secure_lun1" and told it to use 100% of the free space

sudo lvcreate -l 100%FREE -n fs-secure_lun1 fs-secure_iscsi
   Logical volume "fs-secure_lun1" created.

If you run lvs, you should see it

sudo lvs

  LV             VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root           fs-secure-vg    -wi-ao----   6.76g
  swap_1         fs-secure-vg    -wi-ao----   1.00g
  fs-secure_lun1 fs-secure_iscsi -wi-a----- 400.00g

Now we want to encrypt that volume we just made, run the following. You can see the naming convention you need to use.

sudo cryptsetup -v -y luksFormat /dev/mapper/fs--secure_iscsi-fs--secure_lun1

If you are unsure, you can go to /dev/mapper and list the contents and you should see the volume name in full.

cd /dev/mapper

ls -al


total 0
drwxr-xr-x  2 root root     160 Apr 11 19:47 .
drwxr-xr-x 18 root root    3200 Apr 11 19:47 ..
crw-------  1 root root 10, 236 Apr 11 19:47 control
lrwxrwxrwx  1 root root       7 Apr 11 19:47 fs--secure_iscsi-fs--secure_lun1 -> ../dm-0
lrwxrwxrwx  1 root root       7 Apr 11 19:47 iscsi--secure1--vg-root -> ../dm-2
lrwxrwxrwx  1 root root       7 Apr 11 19:47 iscsi--secure1--vg-swap_1 -> ../dm-3
lrwxrwxrwx  1 root root       7 Apr 11 19:47 sda5_crypt -> ../dm-1
sean@iscsi-secure1:/dev/mapper$

After you run the above cryptsetup command it should give you a warning to type YES in caps, and enter the passphrase 

  WARNING!
========
This will overwrite data on /dev/mapper/fs--secure_iscsi-fs--secure_lun1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
Command successful.

Now the volume is encrypted, but you can't actually see the encrypted volume anywhere. You now need to mount the volume which will give you a NEW volume which is encrypted. This will be what you share out via iSCSI

We won't mount it just yet, we will dive directly into auto-mounting the second volume on boot. This is perfectly secure as we still need to unlock the OS drive before this will ever take place

We will now generate a key for the auto-mount process to use. First create a folder somewhere

sudo mkdir /etc/luks-keys

Now generate random characters for the key file

sudo dd if=/dev/urandom of=/etc/luks-keys/disk_secret_key bs=512 count=8

Now its created, we need to add that keyfile to the encrypted volume. After entering the first line, it will prompt for your passphrase you set when creating it, and then add the keyfile

sudo cryptsetup -v luksAddKey /dev/mapper/fs--secure_iscsi-fs--secure_lun1 /etc/luks-keys/disk_secret_key
Enter any passphrase: passphrase
Key slot 0 unlocked.
Command successful.

Now we can verify the key works by actually mouting the volume

sudo cryptsetup -v luksOpen /dev/mapper/fs--secure_iscsi-fs--secure_lun1 iscsi_crypt --key-file=/etc/luks-keys/disk_secret_key

Key slot 1 unlocked.
Command successful.

After this, you will have a new device at /dev/mapper/iscsi_crypt. This is just temporary, we will now close it

sudo cryptsetup -v luksClose iscsi_crypt

Now we know the key works, we need to configure the volume to mount on boot. First we need to find the UUID of the volume, so run the following

sudo cryptsetup luksDump /dev/mapper/fs--secure_iscsi-fs--secure_lun1 | grep "UUID"
UUID:          	2a2335bf-5662-343c-a468-fbe3454659c85

Make a note of the UUID. and open the file /etc/crypttab

sudo nano /etc/crypttab

Make a new line under the existing entry, and enter the following. Remember to change the UUID, and if you want the encrypted volume named something else, change the iscsi_crypt text.

iscsi_crypt UUID=2a2335bf-5662-343c-a468-fbe3454659c85 /etc/luks-keys/disk_secret_key luks

Save the file and exit nano

Now go ahead and reboot the system. When it comes back up go to /dev/mapper and see the contents. You should see your mounted volume listed.

cd /dev/mapper
ls -al

total 0
drwxr-xr-x  2 root root     160 Apr 11 19:47 .
drwxr-xr-x 18 root root    3200 Apr 11 19:47 ..
crw-------  1 root root 10, 236 Apr 11 19:47 control
lrwxrwxrwx  1 root root       7 Apr 11 19:47 fs--secure_iscsi-fs--secure_lun1 -> ../dm-0
lrwxrwxrwx  1 root root       7 Apr 11 20:16 iscsi_crypt -> ../dm-4
lrwxrwxrwx  1 root root       7 Apr 11 19:47 iscsi--secure1--vg-root -> ../dm-2
lrwxrwxrwx  1 root root       7 Apr 11 19:47 iscsi--secure1--vg-swap_1 -> ../dm-3
lrwxrwxrwx  1 root root       7 Apr 11 19:47 sda5_crypt -> ../dm-1

Now we can move on to

iSCSI Target Configuration

This part is actually very easy. We already installed tgt in the first few steps, so we just need to make the configuration.

Enter the following to make the config file. You can name it whatever you want

sudo nano /etc/tgt/conf.d/secure_iscsi.conf

In the file, we need to enter the configuration 

<target iqn.2018-02.iscsi-secure1:lun1>
     backing-store /dev/mapper/iscsi_crypt
     initiator-address 10.0.50.14
     incominguser username password12345
</target>
  • The first line contains the target name. This is just the name of the iSCSI target, just enter an arbitrary date and name in the same format as I have done
  • The next line is the backing for the iSCSI target, this should be the encrypted LUN we mounted.
  • Next is a line for security, we can tell it to only accept connections from this address. If you want, you can remove the line. We can always come back and change this configuration
  • The next line is for CHAP authentication. Enter a username and password here. But watch out, it must be a LEAST 12 characters for the password, and no more than 16! It also must not contain special characters

Now save the file and exit nano. Restart tgt to make the configuration take effect

sudo service tgt restart

Now its restarted, we can verify the LUN shows up

sudo tgtadm --mode target --op show

~~~snip~~~~

        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 429490 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/mapper/iscsi_crypt
            Backing store flags:
    Account information:
        user
    ACL information:
        10.0.50.14

~~~snip~~~~

We are now completely finished configuring the Debian VM. I went ahead and set key based authentication for SSH and a few other security pieces, but those are up to you

Next we need to mount it in ESXi, if you are mounting it somewhere else, we are done!

Mounting the iSCSI LUN in ESXi

This will be done from ESXi 6.7, so if you are using a different version it may be a little different.

If your ESXi box has a single NIC, then you can just bind the software iSCSI initiator to your usual VMK port group. If you have multiple, you probably want to create a seperate iSCSI network.

I split off a single 1G NIC from my usual setup, and made a new vSwitch called "iSCSI vSwitch"

I then added a new VMK NIC for this traffic. You do not need to add any services to it. Make sure its using your new iSCSI vSwitch.

If this new iSCSI VMK will be on the same subnet or VLAN as your usual ESXi network, then select the default TCP/IP Stack when creating it. If not, you need to create a new TCP/IP stack via CLI for it, and configure the correct gateway

Now move over to port groups and create a port group for virtual machines on the same vSwitch. We are going to move the Debian VM to this port group so that iSCSI traffic is all on the same vSwitch, and does not have to go onto the physical network

Now just edit the VM and add it to this port group. Even though this vSwitch is backed by a 1G NIC in my case, I get well over 20Gb/s within ESXi because its all on the same vSwitch and never leaves the host.

Go over to storage, adapters and click on Configure iSCSI

Now make sure its enabled, enter the chap configuration (Make sure to enter credentials on the drop down arrow), make sure its bound the correct port, and then at the bottom enter the IP address of your Debian VM.

Once you have done that and clicked Save, it will automatically populate the target

Now just go into devices, and create a new datastore! It will show as degraded because it only has one path, you can ignore this.

Just be sure to make the Debian VM boot first, and set the VM's that reside on it to not automatically start. When you reboot your ESXi host they will show as orphaned because the storage is not available. After a few moments, it will show up in ESXi and you can start them as normal

Hopefully this was helpful!