Bitlocker has its benefits and its issues. This write-up covers the specific problems I encountered when attempting to encrypt a virtual hard drive of a Windows 10 guest on XenServer.
What is Bitlocker?
Bitlocker is a built-in full drive encryption utility that is part of the Microsoft Windows operating systems. It affords you the ability to completely encrypt your hard drive so that NONE of the data on it is accessible unless it has been correctly ‘unlocked’. If someone were to come into possession of your computer / hard drive, they would need to know how to unlock the drive in order to read any of the information stored on it and this means that your personal data is more secure if the hard drive leaves your direct control.
How Do I Use It?
For this purposes of this write-up, I am focusing on PC-based installations of Windows 10 only. I want to also point out that, in the context of this write-up, “PC-based” means any x86 32- or 64-bit hardware with a very specific focus on a 64-bit version of Windows 10 installed into XenServer 7.2.
To use Bitlocker, there are a variety of minimum requirements that may need to be met. Microsoft maintains a page with those requirements here: Bitlocker Requirements.
So, What’s the Problem?
For reference, here are some details about the environment I am working with to create this write-up:
- XenServer 7.2
- Older Workstation-Class hardware with dual Intel Xeon CPU’s and a large amount of ECC RAM
- Various guest operating systems on the host (linux, Windows)
- Hardware does not provide UEFI boot
- Local storage repository formatted with EXT3 filesystem (Important! – more on this later)
- NFS-accessible storage repository
- Windows 10 Pro guest that has been in use for at least 7-8 years, had previous been Windows 7 Pro and had been upgraded
- Guest had been provisioned with a 250G virtual drive when it was first virtualized, approximately 100G of space used
I cleaned up a number of files that I didn’t need, emptied the recycle bin, enabled the “extra security” options for my OS drive via the registry (no TPM available), shut the machine down and took a snapshot. I started the VM back up, logged in, and shut down all of the programs that auto-launch. I went into the Control Panel, opened the Bitlocker applet, and Turned On Bitlocker for Drive C. After going through the initial setup steps and doing a system check, I received a prompt that the system needed to restart to begin the process.
Initial Problem – Non-Completion of Encryption
After a restart and initialization of the drive at boot, I logged back in and was able to see the encryption process progressing. My first problem occurred when the drive made it to a little over 99% complete and then basically just hung there. I would have been happy to let the process continue to run in the background, but the system had slowed WAY down at this point and was essentially becoming non-responsive.
I forced the machine to power off, reverted to my snapshot, and brought the machine back up. Before trying everything again, I removed the paging file (which I was surprised to find was 16G!), restarted, and then used the Disk Management tool to shrink the OS partition by a small amount as I had read elsewhere that this was a viable remedy.
Second Problem – System Becomes Non-Responsive
The good news is that shrinking the OS partition did, in fact, allow the encryption process to complete. Upon completion, however, the machine would still go into a non-responsive state where it would take tens of minutes to respond to mouse clicks or similar.
No matter what I tried to remedy this, the machine would become non-responsive because of disk activity. What was interesting is that XenCenter never showed high disk reads, writes, latency, etc. – everything continued to look as if the machine were running ‘normal.’ I went through a number of iterations of trying to encrypt the drive by reverting back to my snapshot and making different changes. I removed all kinds of software that had been installed, shut off auto-start programs, shrunk the partition down to about 110G… Nothing worked.
The Solution
Remember when I mentioned that my local storage was formatted as EXT3? I found some discussion online that seems to indicate that an EXT3 volume in XenServer effectively makes the virtual drive only as big as it needs to be based on data written inside of it. This is known in other worlds as “lazy zeroed”. My virtual drive was intended to be 250G in size but was much smaller due to the actual amount of data stored in it. And it was sitting on a drive with OTHER data on it, and the drive was actually over-subscribed. For day-to-day use of these various machines, this is not a problem at all. But…
In my situation, I believe that what was happening was that the drive would show that it was fully encrypted but then went into a state where it was doing “something else” that was causing a massive amount of disk activity. And, since the drive that the virtual hard drive was on was essentially full, there was lot of swap-type activity happening and the machine was pretty much useless.
So, I added another SSD to the server, formatted it as a standard LVM, and moved the virtual hard drive over to the new local storage. I was able to complete the encryption process and everything is running again as it was before.
Other Details
There is actually a LOT more that I did here, including creating a new virtual hard drive for the machine at 110G in size and then cloning the first two partitions on the original virtual HD over to the new one (don’t clone the restore partition). This entire process took me about a week worth of calendar time to complete because I kept running into issues with cloning a 250G partition (even though the filesystem had been shrunk, cloning software still sees the full partition) to a 110G partition. But, it’s back up and running and serving me well again for its intended use. Phew!
Comment and Subscribe!
Comments are welcome – tell me about your own struggles with Bitlocker! And, be sure to subscribe to get updates of new articles.
Comments
Hi, Mark! The Lazy Zero problem with Bitlocker is interesting and something I had never thought of… It sounds like you lose all the benefits over-provisioning with Bitlocker. With that in mind, do you think it would make more sense to provide encryption at the storage level (e.g. some motherboards/hard drives support encryption at the drive level or FreeNAS/TrueNAS can encrypt the drives at the device layer) rather than at the VM level?
Author
Encryption at the drive level is a great solution for drives that support it (and operating systems that can interact with it), Ben. In this particular scenario, drive-level encryption probably isn’t the best option (for me) because I have multiple local storage devices and network-attached storage. Any of them could ultimately hold the virtual disk, and it definitely puts the onus on the admin (me) of the overall solution to ensure that all storage, now and in the future, have the encryption turned on.
This was a nightmare for sure, but I have been back up and running now for a number of days with zero impacts being felt and I did the encryption within the OS.
Side note… There have been reports of issues where a drive will report that it can do hardware encryption and Windows 10 (not sure of older versions) will ‘see’ that an not do the encryption itself. And then, the drive doesn’t actually do the encryption either and you end up with zero protection.