I thought it would be fun to write up a summary of the problems I encountered in my first week of using Qubes, and how I realigned my way to thinking to come up with a workable solution.
Quick Architecture Summary:
For those of you not super familiar with QubesOS, it's essentially a desktop Linux distro that makes heavy use of the Xen hypervisor to compartmentalize the ever-loving crap out of every activity that you would normally want to do on a computer. There exist Template VMs which contain the base operating systems, and there are App VMs which are made to contain your apps. In your App VMs, anything you touch outside of your home directory gets wiped at reboot, so if you want to install stuff, you need to install it in your Template VMs. Template VMs are also firewalled such that they can't touch anything on the internet except for update servers. There are also Service VMs that manage your network and firewall configurations, but you typically don't need to touch them.
Problem #1 Decoration time!
One thing that I actually *really really* like about Qubes is that every compartmentalized VM uses colorized window decorations to give you an instant, visceral understanding of the privilege level you're in the window you're typing into. For example, all if your work windows could be blue, and all your personal windows could be green. The mappings of which colors go to which VMs is always configurable at any time.
I also need to figure out how I want to compartmentalize my data, which is the key feature of Qubes. This machine is my personal desktop system at home which I use for things such as:
- Shitposting on reddit
- Browsing random onion sites
- Playing with interesting new crypto-currencies
- Playing around with weird machine learning stuff
- Managing random servers via SSH
- Downloading and serving up TV shows to my various devices
Why have I been doing this all on one machine for so many years you ask? Shut up, that's why. There are obviously some clear security wins to be had by breaking some of this out. I started by making a VM called "browsing", which I colored green. Then I made a VM called "media" which I made purple. Then I decided to make my cryptocoin VMs yellow (It's a nice, golden yellow), and figured I'd make my SSH VM and weird code VMs all blue. This is where all my SSH keys go. It's nice to know that if my browser gets popped, I don't lose all my keys too.
An awesome thing about the latest release is that QubesOS 3.2 now comes by default with a VM called "anon-whonix" for Tor browsing, which is colored red. It uses the same workstation/gateway model that Whonix uses, and it all works just beautifully out of the box.
Problem #2 Redhat Sucks!
I dunno, for one reason or another, I've never been a RedHat fan. I know Joanna loves it, but it's just not my thing. This has killed me in my previous attempts at running Qubes, but this time, there was a simple, one line solution.
[user@dom0 ~]$ sudo qubes-dom0-update qubes-template-debian-8
If you want something else, check out the template documentation on the qubes documentation page : https://www.qubes-os.org/doc/ You can actually install Ubuntu, Kali, Arch, even Windows as a template VM. Debian works find for me now though, so moving on. Another neat thing here is that I didn't need to rebuild my browsing VM. Since my AppVM is really just a home directory, I easily swaped the underlying template from Fedora to Ubuntu, and everything was good to go.
Problem #3 USB is hard.
The last time I tried QubesOS was the previous release, 3.1. Back then, if you wanted to use a USB stick, it was open heart surgery time. USB sticks had to be wired to AppVMs manually on the command line, and if you forgot to detach any USB devices, and rebooted, you'd get some crazy cascading failures that would prevent even the Service VMs from coming up. I'm happy to say, that's all changed now in 3.2. Now you just plug in a USB device, right click on the App VM you want to attach it to, and you're golden. It even works nicely with my LUKS encrypted USB sticks. I just open the file browser from the drop-down menu, and I can see my encrypted device. When I click on it, it prompts for my LUKS passphrase in a nice, graphical, password prompt. How handy!
Problem #4 FIghting with Plex
I use Plex as my primary media server, which led to some complications. First of all, there isn't really a Debian version of Plex available, so I ended up just using a Fedora template. Remember that things you install in your AppVM don't stick around on reboot. Also, if you install Plex in your Fedora template, then any time you boot any AppVM that uses Fedora as a base, you be running a happy little plex server in your AppVM too, which is undesirable.
I ended up creating my own Template VM for Plex. Sure, it seems a bit silly to have a Template VM in use with only one App VM, but if it's stupid, and it works, it's not stupid.
This worked pretty well, but I had another problem. Plex likes to store all its data in /var/lib/plexmediaserver, so it gets wiped at each reboot, requiring me to reconfigure the server each time I restart the media VM. I originally solved this by just configuring Plex in the Template VM. One problem is that I needed internet access to set up Plex with my online account (remember Template VMs typically only access update servers). There's actually a button in the Firewall Configuration for the VM to allow full internet access for five minutes.
Still though, my viewing data was not being saved across reboots, so everything would need to re-thumbnail, re-encode, and show up as new. The solution that finally hit me was simple. From the Template VM:
cp -avr /var/lib/plexmediaserver /home/user/plexmediaserver
mv /var/lib/plexmediaserver /var/lib/plexmediaserver-bak
ln -s /home/user/plexmediaserver /var/lib/plexmediaserver
Since the plex data is now in my home directory, when I configure it from my App VM, now the data will persist. Everything works exactly as expected now.
Problem #5 It's getting crowded in here!
I like to install my base operating systems on SSDs, but unfortunately, the SSD in this system is only a few hundred gigs. The bitcoin blockchain alone is over 100 gigs these days, and media servers tend to fill up quick. I've got this nice 6tb drive sitting right here too, but none of my AppVMs can access it, except one at a time.
I'll admit that I spent way too long scheming on fancy bindmount setups, or some shared filesystem situation. I also almost broke down and re-partitioned my 6tb drive so I could share out mount points individually, but I didn't want to risk data loss.
My eventual obvious breakthrough was simlinks. I ran this from Dom0 after shutting down all running AppVMs.
cp -avr /var/lib/qubes/appvms /mnt/6t/appvms
mv /var/lib/qubes/appvms /var/lib/qubes/appvms-bak
ln -s /mnt/6t/appvms /var/lib/qubes/appvms
I actually came up with this solution before the final step of my Plex configuration, and you'll note how similar they look, but I figured it would be weird to revisit the Plex thing later.
I also edited my crypttab and fstab to make sure the 6tb drive gets attached at boot. After this, I was able to go into the AppVM settings for each AppVM and set the disk storage to be as large as the AppVM would need to get. There seems to be a 1tb maximum unfortunately. I don't know if that's a Xen limitation, or just an arbitrary value that some Qubes developer thought would be more data than anyone would need, but I do have a VM that wants 4tb, though I can make due without it for a while.
Problem #6 OMG the CIA is hacking everyone!
Conveniently, the Vault 7 Wikileaks leak happened this week. A few clicks later, and I had my shiny new red Wikileaks AppVM. This was pretty neat, because in the AppVM, I was able to apt-get install qbittorrent, download the torrent file in my browsing VM, send it over to the wikileaks VM, and download it there.
Once the passphrase was announced, I could dig through it, without fear of any browser exploits or PDF exploits. Not that Wikileaks would ever release malicious files, but it felt really good to be able to dig through them in a completely isolated environment.
This is also when I got really used to how copy and paste works in Qubes. This is actually super neat. From my green browsing VM, I could go to twitter, pull up the wikileaks tweet with the crazy long 7zip password, and then "ctrl+c" like normal. Now, this put the password into my "browse" copy buffer. Then, with the browse window up front, I pressed "ctrl+shift+c" which indicated to qubes that I wanted to pass around my copy buffer to another VM. Once I alt-tabbed over to my wikileaks VM, I pressed "ctrl-shift-v" to load the password into the copy buffer on the wikileaks VM. Then I right clicked and said "paste" in the context menu. This was so much slicker than how VMWare does copy and paste when it works, and much less frustrating than trying to hand type data across VMs. It's also pretty damn secure. A shared copy buffer across all AppVMs would be a disaster, but this method seems very precise and simple enough that after doing it a few times you start to do it without thinking.
Problem #7 Oh crap, I probably just said too much
#YOLO. I'm a firm believer in Kerckhoffs's principle. That is to say, knowledge of how I deploy my security should only serve to discourage any potential attackers. Even a potential exposure on my end in the name of education is a net win for everyone. If you understood most of what the hell I was taking about in this post, you're probabally ready to try out Qubes.
Download here: https://www.qubes-os.org/downloads/