Review: Copperhead OS

A few weeks ago I found an article in my news feed about a Tor phone, and it introduced me to Copperhead OS. This is an extremely hardened version of the Android Open Source Project (AOSP) designed for both security and privacy. It has become my default mobile OS so I thought I’d write about my experiences with it.

TL;DR: Copperhead OS is not for everyone. Due to its focus on security is it not easy to install any software that relies on Google Services, which is quite a bit. But if you are concerned with security and privacy, it offers a very stable and up to date operating system. The downside is that I am not able to totally divorce myself from Google, so I’ve taken to carrying two phones: one with Copperhead and one with stock Android for my “Googly” things. What we really need is a way to run a hypervisor on mobile device hardware. That way I could put all of my personal stuff on a Copperhead and the stuff I want to share with Google in a VM.

I pride myself to the point of being somewhat smug about the fact that I use free software for most of my technology needs, or so I thought. My desktops, laptop, servers, router, DVR and even my weather station all use free and open source software, and I run OmniROM (an AOSP implementation) on my phone. I also “sandbox” my Google stuff – I only use Chrome for accessing Google web apps and I keep everything else separate (no sharing of my contacts and calendar, for example). So, I was unpleasantly surprised at how much I relied on proprietary software for my handy (short for “hand terminal” or what most people call a “mobile phone”, but I rarely use the “phone” features of it so it seems like a misnomer).

But first a little back story. I was sitting on the toilet playing on my mobile device (“playing on my handy” seemed a little rude here) when I came across a page that showed me all of the stuff Google was tracking about my mobile usage. It was a lot, and let’s just say any bathroom issues I was having were promptly solved. They were tracking every call and text I made, which apps I opened, as well as my location. I knew about the last one since I do play games like Ingress and Pokémon Go that track you, but the others surprised me. I was able to turn those off (supposedly) but it was still a bit shocking.

Of course, I had “opted in” to all of that when I signed in to my handy for the first time. When you allow Google to backup your device data, you allow them to record your passwords and call history.

Google Backup Terms

If you opt in to help “improve your Android experience”, you allow them to track your app usage.

Google App Terms

And most importantly, by using your Google account you allow them to install software automatically (i.e. without your explicit permission).

Google Upgrade Terms

Note that this was on a phone running OmniROM, and not stock Google, but it still looks like you have to give Google a lot of control over your handy if you want to use a Google account.

Copperhead OS is extremely focused on security, which implies the ability to audit as much software on the device as possible, as well as to control when and what gets updated. This lead them to remove Google Play Services from the ROM entirely. Instead, they set up F-Droid as the trusted repository. All the software in F-Droid is open source, and in fact all of the binaries are built by the F-Droid team and not the developer. Now, of course, someone on that team could be compromised and put malicious software into the repo, but you’ve got to trust somebody or you will spend your entire life doing code reviews and compiling.

Copperhead only runs on a small subset of devices: the Nexus 6P, the Nexus 5X and the Nexus 9 WiFi edition. This is because they support secure boot which prevents malicious code from modifying the operating system. Now, I happened to have a 6P, so I figured I would try it out.

The first hurdle was understanding their terminology. On the download page they refer to a “factory” image, which I initially took to mean the original stock image from Google. What they mean is an image that you can use for a base install. If you flash your handy as often as I do, you have probably come across the process for restoring it to stock. You install the Android SDK and then download a “factory” image from Google. You then expand it (after checking the hash, of course) and run a “flash-all” script. This will replace all the data on your device, including a custom recovery like TWRP, and you’ll be ready to run Copperhead. Note that I left off some steps such as unlocking and then re-locking the bootloader, but their instructions are easy to follow.

The first thing you notice is that there isn’t the usual “set up your Google account” steps, because, of course, you can’t use your Google account on Copperhead. Outside of missing Google Apps, the device has a very stock Android feel, including the immovable search bar and the default desktop background.

This is when reality began to set in as I started to realize exactly how much proprietary software I used to make my handy useful.

The first app I needed to install was the Nova Launcher. This is a great Launcher replacement that gives you a tremendous amount of control over the desktop. I looked around F-Droid for replacement launchers, and they either didn’t do what I wanted them to do, or they haven’t been updated in a couple of years.

Then it dawned on me – why don’t I just copy over the apk?

When you install a package from Google Play, it usually gets copied into the /data/apps directory. Using the adb shell and the adb pull commands from the SDK, I was able to grab the Nova Launcher software off of my Nexus 6 (which was running OmniROM) and copy it over to the 6P. Using the very awesome Amaze file explorer, you just navigate to the apk and open it. Now, of course, since this file didn’t come from a trusted repository you have to go under Security and turn off the “trusted sources” option (and be sure to turn it back on when you are done). I was very happy to see that it runs just fine without Google Services, and I was able to get rid of the search bar and make other tweaks.

Then I focused on installing the open source apps I do use, such as K-9 Mail and Wikipedia, both of which exist in F-Droid. I had been using the MX Player app for watching videos, pretty much out of habit, but it was easy to replace with the VLC app from F-Droid.

I really like the Poweramp music player, with the exception that it periodically checks in with the Play store to make sure your license is valid. Unfortunately, this has happened to me twice when I was in an airplane over the ocean, and the lack of network access meant I couldn’t listen to music. I was eager to replace it, but the default Music app that ships with Copperhead is kind of lame. It does a good job playing music, but the interface is hard to navigate. The “black on gray” color scheme is very hard to read.

Default Music Player Screenshot

So I replaced it with the entirely capable Timber app from F-Droid.

Timber Music Player Screenshot

Another thing I needed to replace was Feedly. I’m old, so I still get most of my news directly from websites via RSS feeds and not social media. I used to use Google Reader, and when that went away I switched to Feedly. It worked fine, but I bristled at the fact that it tracked my reading habits. Next to each article would be a number representing the number of people who clicked on it to read it, so at a minimum they were tracking that. I investigated a couple of open source replacements when I was pleasantly surprised to discover that Nextcloud has a built in News service. We have had a really good experience with Nextcloud over the last couple of months, and it was pretty easy to add the news service to our instance. Using OPML I was able to export my numerous feeds from Feedly into Nextcloud, and that was probably the easiest part of this transition. On the handy I used an F-Droid app called OCReader which works well.

There were still some things I was missing. For example, when I travel overseas I keep in touch with my bride using Skype (which is way cheaper than using the phone) so I wanted to have Skype on this device. It turns out that it is in the Amazon App Store, so I installed that and was able to get things like Skype and the eBay and IMDB apps (as well as Bridge Baron, which I like a lot). Note that you still have to allow unknown sources since the Amazon repository is not trusted, and remember to set it back when you are done.

This still left a handful of apps I wanted, and based on my success with the Nova Launcher I just tried to install them from apks. Surprisingly, most of them worked, although a couple would complain about Google Services being missing. I think background notifications is the main reason they use Google Services, so if you can live without that you can get by just fine.

One app that wouldn’t work was Signal, which was very surprising since they seem to be focused on privacy and security. Instead, the default messenger is an app called Silence, which is a Signal fork. It works well, but it isn’t in the Play store (at least in the US due to a silly trademark issue that hasn’t been fixed) and no one I know uses it so it kind of defeats the purpose of secure messaging. Luckily, I discovered that the Copperhead gang has published their own fork called Noise, which removes the Googly bits but still works with the rest of the Signal infrastructure, so I have been using it as my default client with no issues. Note that it is in the F-Droid app but doesn’t show up on the F-Droid website for some reason.

For other apps such as Google+ and Yelp, I rediscovered the world wide web. Yes, browsers still work, and the web pages for these sites are pretty close to matching the functionality of the native app.

There are still some things for which there is no open source replacement: Google Maps, for example. Yes, I know, by using Google Maps I am sharing my location with Google, but the traffic data is just so good that it has saved literally hours of my life by directing me around accidents and other traffic jams. OpenStreetMap is okay and works great offline, but it doesn’t know where the OpenNMS office is located (I need to fix that) and without traffic it is a lot less useful. There is also the fact that I do like to play games like Ingress and Pokémon Go, and I have some movies and other content on Google servers.

I also lost Android Wear. I really enjoy my LG Urbane but it won’t work without Google Services. I have been playing with AsteroidOS which shows a lot of promise, but it isn’t quite there yet.

Note that Compass by OpenNMS is not yet available in F-Droid. We use Apache Cordova to build it and that is not (yet) supported by the F-Droid team. We do post the apks on Github.

To deal with my desire for privacy and my desire to use some Google software, I decided to carry two phones.

On the Nexus 6P I run Copperhead and it has all of my personal stuff on it: calendar, contacts, e-mail, etc. On the Nexus 6 I am running stock Google with all my Googly bits, including maps. I still lock down what I share with Google, but I feel a lot more confident that I won’t accidentally sync the rest of my life with them.

It sucks carrying two phones. With the processors and memory in modern devices I’m surprised that no one has come up with a hypervisor technology that would let me run Copperhead as my base OS and stock Google in a VM. Well, not really surprised since there isn’t a commercial motivation for it. Apple doesn’t have a reason to let other software on its products, and Google would be shooting itself in the foot since its business model involves collecting data on everything. I do think it will happen, however. The use case involves corporations, especially those involved in privacy sensitive fields such as health care. Wouldn’t it be cool to have a locked down “business” VM that is separate from a “personal” VM with your Facebook, games and private stuff on it.

As for the Copperhead experience itself, it is pretty solid. I had a couple of issues where DNS would stop working, but those seem to have been resolved, and lately it has been rock solid except for one instance when I lost cellular data. I tried reseting the APN but that didn’t help, but after a reboot it started working again. Odd. Overall is it probably the most stable ROM I’ve run, but part of that could be due to how vanilla it is.

Copperhead is mainly concerned with security and not extending the Android experience. For example, one feature I love about the OmniROM version of the Alarm app is the ability to set an action on “shake”. For example, I set it to “shake to dismiss” so when my alarm goes off I can just reach over, shake the phone, and go back to bed. That is missing from the stock ROM (but included in AOSP) and thus it is missing from Copperhead. The upside is that Copperhead is extremely fast with updates, especially security updates.

The biggest shortcoming is the keyboard. I’ve grown used to “gesture” typing using the Google keyboard, but that is missing from the AOSP keyboard and no free third party keyboards have it either. I asked the Copperhead guys about it and got this reply:

If the open-source community makes a better keyboard than AOSP Keyboard, we’ll switch to it. Right now it’s still the best option. There’s no choice available with gesture typing, let alone parity with the usability of the built-in keyboard. Copperhead isn’t going to be developing a keyboard. It’s totally out of scope for the project.

So, not a show stopper, but if anyone is looking to make a name for themselves in the AOSP world, a new keyboard would be welcome.

To further increase security, there is a suggestion to create a strong two-factor authentication mechanism. The 6P has a fingerprint sensor, but I don’t use it because I don’t believe that your fingerprint is a good way to secure your device (it is pretty easy to coerce you to unlock your handy if all someone has to do is hold you down and force your finger on to a sensor). However, having a fingerprint and a PIN would be really secure, as the best security is combining something you have (a fingerprint) with something you know (a PIN).

So here was my desktop on OmniROM:

Old Phone Desktop

and here is my current desktop:

New Phone Desktop

Not much different, and while I’ve given up a few things I’ve also discovered OCReader and Nextcloud News, plus the Amaze file manager.

But the biggest thing I’ve gained is peace of mind. I want to point out that it is possible to run other ROMs, such as OmniROM, without Google Services, but they aren’t quite as focused on security as Copperhead. Many thanks to the Copperhead team for doing this, and if you don’t want to go through all the work I did, you can buy a supported device directly from them.

Review: X-Arcade Gaming Cabinet

Last year I wanted to do something special for the team to commemorate the release of OpenNMS Meridian.

Since all the cool kids in Silicon Valley have access to a classic arcade machine, I decided to buy one for the office. I did a lot of research and I settled on one from X-Arcade.

X-Arcade Machine

The main reasons were that it looked well-made but it also included all of my favorites, including Pac-Man, Galaga and Tempest.

X-Arcade Games

The final piece that sold me on it was the ability to add your own graphics. I went to Jessica, our Graphic Designer, and she put together this wonderful graphic done in the classic eight-bit “platformer” style and featuring all the employees.

X-Arcade Graphic

Ulf took the role of Donkey Kong, and here is the picture meant to represent me:

X-Arcade Tarus

The “Tank Stick” controls are solid and responsive, although I did end up adding a spinner since none of the controls really worked for Tempest.

When you order one of these things, they stress that you need to make sure it arrives safely. Seriously, like four times, in big bold letters, they state you should check the machine on delivery.

I was going to be out of town when it arrived, so I made sure to tell the person checking in the delivery to make sure it was okay (i.e. take it out of the box).

They didn’t (the box looked “fine”) and so we ended up with this:

X-Arcade Cracked Top


Outside of that, everything arrived in working order. You get a small Dell desktop running Windows with the software pre-installed, but you also get CDs with all the games that are included with the system. It’s a little bit of a pain to set up since the instructions are a little vague, but after about an hour or so I had it up and running.

Anyway, it is real fun to play. It supports MAME games, Sega games, Atari 2600 games and even that short lived laserdisc franchise “Dragon’s Lair”. You can copy other games to the system if you have them, although scrolling through the menu can get a bit tiring if you have a long list of titles.

We had an issue with the CRT about 11 months after buying the system. I came back from a business trip to find the thing dark (it never goes dark, if the computer is hung for some reason you’ll still see a “no signal” graphic on the monitor”). Turns out the CRT had died, but they sent us a replacement under warranty and hassle free. It took about an hour to replace (those instructions were pretty detailed) and it worked better than ever afterward.

This motivated me to consider fixing the top. When we had the system apart to replace the monitor, I noticed that the top was a) the only thing broken and b) held on with eight screws. I contacted them about a replacement piece and to my surprise it arrived two days later – no charge.

The only issue I have remaining with the system is the fact that it is Windows-based. This seems to be the perfect application for a small solid-state Linux box, but I haven’t had the time to investigate a migration. Instead I just turned off or removed as much software as I could (all the Dell Update stuff kept popping up in the middle of playing a game) and so far so good.

I am very happy with the product and extremely happy with the company behind it. If you are in the market for such a cabinet, please check them out.

First Thoughts on Linux Mint 18 “Sarah”

I am a big fan of Linux Mint and I look forward to every release. This week Mint 18 “Sarah” was released. I decided to try it out on my Dell XPS 13 laptop since it is the easiest machine of mine to base and they really haven’t suggested an upgrade path. The one article I was able to find suggested a clean install, which is what I did.

First, I backed up my home directory, which is where most of my stuff lives, and I backed up the system /etc directory since I’m always making a change there and forgetting that I need it (usually concerning setting up the network interface as a bridge).

I then installed a fresh copy of Mint 18. Now they brag that the HiDPI support has improved (as I will grouse later, so does everyone else) but it hasn’t. So the first thing I did was to go to Preferences -> General and set “User interface scaling” to “Double”. This worked pretty well in Mint 17 and it seems to be fine in Mint 18 too.

I then did a basic install (I used a USB dongle to connect to a wired network since I didn’t want to mess with the Broadcom drivers at this point) and chose to encrypt the entire hard drive, which is something I usually do on laptops.

I hit my first snag when I rebooted. The boot cycle would hang at the password screen to decrypt the drive. In Mint 17 the password prompt would be on top of the “LM” logo. I would type in the password and it would boot. Now the “LM” logo has five little dots under it, like the Ubuntu boot screen, and the password prompt is below that. It’s just that it won’t accept input. If I boot in recovery mode, the password prompt is from the command line and works fine.


This seems to be a problem introduced with Ubuntu 16.04. Well, before I dropped back down to Mint 17 I decided to try out that distro as well as Kubuntu. My laptop was based in any case.

I ran into the usual HiDPI problems with both of those. I really, really want to like Kubuntu but with my dense screen I can’t make out anything and thus I can’t find the option to scale it. Ubuntu’s Unity was easier as it has a little sliding scaler, but when I got it to a resolution I liked many of the icon labels were clipped, just like last time I looked at it.


Then it dawned on my that I could just install Mint 18 but see if encrypting just my home directory would work. It did, so for now I’m using Mint 18 without full disk encryption. The next step was to install the proprietary Broadcom driver and then wireless worked.

Next, I edited /etc/fstab and added my backup NFS mount entry, mounted the drive and started restoring my home directory. That went smoothly, until I decided to reboot.

The laptop just hung at the boot screen.

Now there is a bug in Dell BIOS that if I try to boot with a USB network adapter plugged in, it erases the EFI entry for “ubuntu” and I have to go into setup and manually re-add it. Thus I was disconnecting the dongle for every reboot. On a whim I plugged it back in and the system booted. This led me to believe that there was an issue with the NFS mount in /etc/fstab, and that’s what the problem turned out to be.

The problem is that systemd likes to get its little hands into everything, so it tries to mount the volume before the wireless network is initialized. The solution is to add a special option that will cause systemd to automount the volume when it is first requested. Here is what worked: /media/backups nfs noauto,x-systemd.automount,nouser,rsize=8192,wsize=8192,atime,rw,dev,exec,suid 0

The key bits are “noauto,x-systemd.automount”.

With that out of the way, I added mounts for my music and my video collection. That’s when I noticed a new weirdness in Cinnamon: dual icons on the desktop. I have set the desktop option to display icons for mounted file systems and now I get two of them for each remote mount point.

Double Desktop Icons

Annoying and I haven’t found a solution, so I just turned that option back off.

Now I was ready to play with the laptop. I’m often criticized for buying brand new hardware and expecting solid Linux support (yeah, you, Eric) but this laptop has been out for over a year. Still the trackpad is a little wonky – the cursor tends to jump to the lower right hand corner. Mint 18 ships with a 4.4 kernel but I had been using Mint 17 with a 4.6 kernel. One of the features of 4.6 is “Dell laptop improvements” so while I was hoping 4.4 would work for me (and that the features I needed would have been backported) it isn’t so. I installed 4.6 and my trackpad problems went away.

The final issue I needed to fix concerned ssh. I use ssh-agent and keys to access a lot of my remote servers, and it wasn’t working on Mint 18. Usually this is a permissions issue, but I compared the laptop to a working configuration on my desktop and the permissions were identical.

The error I got was:

debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tarus/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0

It turns out that OpenSSH 7.0 seems to require that an “IdentityFile” parameter be expressly defined. I might be able to do this in ssh_config but instead I just created a ~/.ssh/config file with the line:

IdentityFile ~/.ssh/id_dsa_main

That got me farther. Now the error changed to:

debug1: Skipping ssh-dss key /home/tarus/.ssh/id_dsa_main - not in PubkeyAcceptedKeyTypes
debug1: Skipping ssh-dss key - not in PubkeyAcceptedKeyTypes

It seems the key I created back in 2001 is no longer considered secure. Since I didn’t want to go through the process of creating a new key just right now, I added another line to my ~/.ssh/config file:

IdentityFile ~/.ssh/id_dsa_main

and now it works as expected. The weird part is that you would think this would be controlled on the server side, but the failure was coming from the client and thus I had to fix it on the laptop.

Now that it is installed and seems to be working, I haven’t really played around with Mint 18 much, so I may have to write another post soon. I do give them props for finally updating the default desktop wallpaper. I know the old wallpaper was traditional, but man was it dated.

This was a more complex upgrade than usual, and I don’t agree that you must base your system to do it, even from major release to major release. This isn’t Fedora. It’s based on Ubuntu which is based on Debian and I have rarely had issues with those upgrades. Usually you just change you repositories and then do “apt-get dist-upgrade”.

But … I might wait a week or two once they approve an upgrade procedure and let other people hit the bugs first, just in case. My desktops are more important to me than my laptop.

Hats off to the Mint team. I’m pretty tied to this operating system so I’m encouraged that it keeps moving forward as quickly as it does.

Review: System 76 Wild Dog Pro

Recently, I was trying to work on the desktop at my office and it kept rebooting. Nothing in the logs nor anything to indicate that there might be a software issue, so I just assumed that my now 5 year old machine was probably at its end of life.

Without hesitation I decided to order a new desktop from System 76. I really liked the Sables we bought from them last year, so I figured it would be simple to order a Linux-compatible machine from them.

I went to their “Desktops” page and without much thought decided on the “Wild Dog Pro“. I don’t have huge requirements, so the big monster with wheels the “Silverback” (probably named after the gorilla) was right out. I picked more of a middle of the road machine with the following specs:

  • Case: Black brushed aluminum
  • CPU: 4.2 GHz i7-6700K (4.0 up to 4.2 GHz – 8MB Cache – 4 Cores – 8 Threads) + Liquid Cooling
  • Memory: 32 GB Dual-channel DDR4 at 2133 MHz (4× 8 GB)
  • Graphics: 2 GB GTX 960 Superclocked with 1024 CUDA Cores
  • Storage: 1 TB 2.5″ Solid State Drive
  • Dual Layer CD-RW / DVD-RW
  • WiFi up to 867 Mbps + Bluetooth
  • 3 Year Limited Parts and Labor Warranty

I also ordered two day shipping, since I thought I would need it fast.

I got an order confirmation almost immediately with an estimate of 2 to 6 days to ship. Soon after that I got a note stating that the Wild Dog was running toward the latter end of that range. I figured I could just use my laptop until the new machine arrived if necessary, and I waited.

While I was waiting, I still continued to use my old desktop. I noticed the rebooting issue happened toward the end of the day. It finally dawned on me (I’m a little thick) that it might be heat related. I crawled under the desk to find that the power supply fan wasn’t working. I ordered a new one of those to see if it would help.

Since the new power supply arrived before the Wild Dog shipped, and it fixed my issue, I contacted System 76 to see if I could change the shipping from “speedy” to something more like “camel”. They were happy to do it and refunded the difference in price.

Anyway, the new machine finally arrived (I ordered this on 29 January and got it on 16 February – a little slow but faster than Lenovo and Dell have been for me in the past). It showed up in a standard brown box:

Wild Dog Pro Box

The unit was minimally packaged inside (which I like):

Wild Dog Pro Box Open

Pretty case with a minimalist look:

Wild Dog Pro Front

with all of the “business” being on the back:

Wild Dog Pro Back

This has USB 2.0, USB 3.0 and USB 3.1 ports, including a USB-C connector should you be into such things.

I like the case, but they tape a letter on the top that, when removed, you can still see the marks left by the tape. I haven’t hit this with goo cleaner since it is going under my desk, but it did detract from the overall look of the unit. The letter contained a welcome note and some stickers, as well as a little cut-out dude called the “Desktop Sentinel” and named “M3lvin”. Not quite sure what that is, and a quick Google search turned up nothing.

Wild Dog Pro Letter

Of course, the first thing I did was open it up. The case is nice, although I’ve grown used to captive screws to remove side panels and was surprised when the two I took off came completely off. The system is well laid out inside with room for expansion (I wanted to put in a backup SATA drive to go with the SSD).

Wild Dog Pro Right

Wild Dog Pro Left

I can’t tell you much about the performance. It seems plenty fast, and I downloaded the test suites from Phoronix but just didn’t have the hours to run them for benchmarks. While it ships with Ubuntu 15.10, I’m a Linuxmint guy so I immediately went to install Mint on the machine.

This was harder than I thought it would be. I could not get the BIOS to boot off of the USB stick no matter what I tried (it saw the stick in the boot menu but wouldn’t boot to it). I ended up burning the image to DVD and, while slower, worked fine.

Then it dawned on me that they probably shipped with Ubuntu 15.10 because it has one of those fancy “Skylake” processors which benefits from later kernels. Luckily I had run into this with my Dell laptop, so I installed gcc-4.9 and the 4.4 kernel and everything worked but the wireless card. Turns out you need to install the latest ndiswrapper and you’ll be good to go.

Needless to say I’m eager for Mint 18 to come out with support for the later kernels.

Overall, I’m happy with my purchase. There is room for improvement on the speed of producing it and shipping it out, but my decision to use Mint was totally on me. I look forward to getting many hours of use out of this machine.

Review: Angel Sensor Fitness Tracker

Angel Sensor represents everything that’s wrong with the technology industry today.

TL;DR; Two years ago, Angel Sensor ran an Indiegogo campaign to create an “open sensor for health and fitness”. They implied that the software would be open source. I finally got mine this week and it is total bollocks. Not only is the software not open source, the app that goes with it is barely an app. There is little communication from the vendor to the community, and while the hardware is solid, it is too expensive to manufacture so the “classic” model is obsolete on delivery. Don’t deal with this company.

Okay, so I like metrics. I work on an open source project to monitor anything you reach over the network. I have a weather station at my house and a temperature sensor in my workshop. I am very eager to gather information about what’s going on in my body, and while companies like Fitbit make great products for that purpose, I distrust sending this most personal data to a third party.

So a couple of years ago I did a search on “open source fitness tracking” and came across Angel Sensor. This company claimed that they were going to create an open health platform where the software would be open source, so I bought an Angel Sensor wristband and eagerly awaited its arrival.

And waited. And waited.

Two years later, it finally arrived and it is a total disappointment.

First, the good.

The packaging is nice.

Angel Sensor Box

The band itself is in its own compartment, and on the side of the box is a little drawer that you can pull out containing the accessories:

Angel Sensor What's In the Box

You get the band, a small instruction booklet, a charging cradle and seven flexible clasps (of various lengths) that help hold the band to your wrist.

I picked out a clasp and pretty soon had it on my wrist:

Angel Sensor On Wrist

Although heavier than I would have expected, it felt comfortable, something I could wear 24/7. My LG Urbane watch is slightly thicker but overall a bit lighter:

Angel Sensor vs. LG Urbane

The instruction booklet says to charge the band fully before using, and this is where the problems started. The “classic” uses a charging cradle. At both ends of the band (where the clasp connects) are metal studs. You insert one set of studs (marked on the band) into the cradle to charge it. The problem is that there is nothing in the charger to really grab on to the studs, so in my case it kept losing the connection and charging would stop. I had to prop it up at an angle in order to keep a connection, and even then I wasn’t sure about it staying in place.

Which is a shame, since the band itself is rather stylish. While there is no screen, there are two white LEDs on each side of the band that can glow and pulse to let you know something is going on. I kept having to keep an eye on the LEDs to make sure the thing was still charging.

Note that all this is moot since the classic proved too hard to produce. The new unit is called the M1. The M1 is thicker and you lose water resistance, which I think is an important feature. While I don’t plan to dive with a fitness tracker, I might wash dishes while wearing it, so the ability to be submerged in liquid for a small amount of time is a requirement. The M1 does use a standard microUSB charging connector so that is a plus in its favor.

Summary to this point: solid hardware design, although now obsolete, with a major flaw in the charger.

My real disappointment set in with the software.

I knew something was wrong when they announced on their blog that the first app released would be iOS only. Now I don’t have a problem with people leading with the iOS version, it is a huge market, but when your market differentiation is based on being “open” one would assume that an Android version would be first to encourage more contribution. Alas, the Android version seems to be more of an afterthought. You want to see it? Here it is:

Angel Sensor Android App Screen

Yup. You’re looking at it. No menu, no explanation, just four values.

To get to this point, I downloaded the app from Google Play, launched it, and then paired it with the band. You do this by tapping on the band’s button once, which will cause it to vibrate. The sensor will then show up on the app’s screen and you can connect to it. Note that you have to do this every time you launch the app, or at least I did. The sensor will be identified by a number and a MAC address.

On the main screen you get what I assume to be heart rate, body temperature, number of steps and some unknown value represented in units of “g”. No history, no way to, say, gather and export collected data, no way to even change the temperature units from Celsius to Fahrenheit. The title bar does show connection strength and battery life, but the only other thing is a tab for firmware updates. From what I can tell, it doesn’t actually check anywhere for firmware updates, but it would give you the ability to install one should it be released.

Of course, the app doesn’t tell you the current firmware version, and there doesn’t seem to be a download section anywhere on the Angel website (the support section is just a duplicate of the FAQ section).

Oh, and if you turn it sideways, you get some graphs.

Angel Sensor Android App graphs

No explanation of what is actually being graphed, but it does wiggle around a bit. I did find a Youtube video that suggests the top two graphs are heart related and the bottom one is motion, and the IOS app shown in the video has more features, but since I don’t have a modern iPhone I couldn’t check it out.

Since I couldn’t believe this was it, I kept searching for software related to the Angel Sensor. I did manage to find some code on Github (which appears to be an SDK for accessing the API and perhaps the sad Android app). Of course, the License file is incomplete and doesn’t really say under what license the software is published. Once again people, access to source code doesn’t make it open source.

And this is the biggest flaw with Angel Sensor and one reason I have such an issue with the current technology environment. You have people like Paul Graham crowing about creating income inequality and that has resulted in a new start up life cycle. You come up with an idea (that hasn’t changed) but now you wrap it a bunch of buzzwords, like “open”, “IoT” and “mobile”, even if you don’t really understand what they mean.

Next, you raise some money through crowdfunding, often underestimating the amount needed so your “funding” can be a success. Now, remember, your audience isn’t the poor saps who decided to give you money – you want to show viability to VCs who can deliver the money you’ll actually need. You use the initial cash to build just enough product to either get a lot of investment or get acquired by someone wanting to cut a year or so off developing their own product. This is considered “success” in some circles.


This whole thing is a shame since there is a market for a truly open mobile health platform. I’m not insisting that the hardware be open (I’m not going to build my own silicon molds) but all the software, including the firmware, should be available under an OSI-approved license. Then you need to focus on building a community, and that really requires good communication.

Angel Sensor sucks at communication.

In addition to the total lack of documentation, even their blog fails miserably. In the last half of 2015 there were a total of three posts, the last one from 3 November. I used to be able to get a rapid response from a person named Io Salant, but when I wrote to them a few months ago asking for a status I got:

Io is on a well-deserved leave. Your email has been forwarded to
The team is quite busy, but will make every attempt to get to your email as quickly as possible.

Of course, never heard back. So my now “classic” Angel Sensor is destined for eBay.

In summary, this is another case of a crowdfunded effort done by people in over their heads with no real desire to create something that lasts but to make money at the expense of their customers. I wouldn’t trust Angel Sensor to feed my dog, much less monitor my health, and I can only hope someone with actual ability will come out with a truly open personal medical platform.

Mint 17.3 (Rosa) on the Dell XPS 13 (9343)

I’m a big fan of the Dell XPS 13. It is the first laptop I’ve felt an emotional attachment to since my first Powerbook. The only issue is that I have not been able to run my distro of choice, Linux Mint, due to severe issues with the trackpad.

Mint on XPS

With the release of Mint 17.3 (Rosa) I decided to give it another shot. I burned the image to a USB stick and booted to it, and the trackpad issues were gone.


So I based my system and installed Mint. I did have to use a wired network connection since the Broadcom drivers don’t seem to work on install (there is probably a way around that) but once installed they were easy to enable.

One thing I liked about Mint when I had installed it previously was that it recognized the HiDPI screen of the XPS right away. Even though the “What’s New” page says that HiDPI detection has been improved in 17.3, I found that it had regressed and I needed to squint to get the O/S installed. Once I did, however, I was able to go to Settings -> General and switch to HiDPI mode and everything was fine.

Mint HiDPI Setting

Now, the XPS hardware is so new that it really requires a 4.2 kernel. I decided to install it. No biggie, since I had to do it with Ubuntu 15.04, but I’ll be happy when Mint 18 comes out and it is supported natively (you have to do some apt magic to ignore kernel updates). Once installed, my wireless connection failed to work, and that’s where the fun began.

Usually, all I had to do was reinstall the bcmwl-kernel-source package, but this kept failing with an error. I even built the package from source but while it built just fine, DKMS would fail when installing it, complaining about “-fstack-protector-strong”. Turns out this was added in gcc 4.9 and Mint 17.3 ships with gcc 4.8.


Anyway, not hard to fix. I ran the following commands:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get dist-upgrade
sudo apt-get install gcc-4.9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 70

and now gcc 4.9 was my default compiler. I then rebuilt and installed the bcmwl-kernel-source pacakge and things were golden.

$ modinfo wl
filename:       /lib/modules/4.2.6-040206-generic/updates/wl.ko
license:        MIXED/Proprietary
srcversion:     D46E6565F844EFBD46CE0FC
alias:          pci:v*d*sv*sd*bc02sc80i*
depends:        cfg80211
vermagic:       4.2.6-040206-generic SMP mod_unload modversions 
parm:           passivemode:int
parm:           wl_txq_thresh:int
parm:           oneonly:int
parm:           piomode:int
parm:           instance_base:int
parm:           nompc:int
parm:           intf_name:string

Just like with Ubuntu Gnome, I did have to manually install the bluetooth driver, but at the moment everything seems to work: wireless, bluetooth, the touchscreen, the clickpad, sleep, backlit keyboard, etc.

Now I use a desktop as my primary machine, so I haven’t really taken the XPS through its paces, but I’m scheduled to travel soon and I’ll be sure to post if I have any issues. I did enable the screensaver and once when I came back to the machine my mouse pointer was gone (the mouse still worked, you just couldn’t see the pointer) and I was unable to fix it without a restart (I tried the suggestions in Google but it didn’t work). For now I’ve just disabled the screensaver.

All in all, great work from the Mint team, and while I actually enjoyed my time with Ubuntu Gnome I’m happy to be back. Looking forward to Mint 18 in the Spring which should require less effort to run on the XPS with built-in support for the 4 series kernel.

Review: Signal by Open Whisper Systems

I like security, and one of the biggest security holes in my technology concerns text messaging and phone calls. While I can secure my data (for the most part), it is hard to secure traffic over the telephone network, especially with the proliferation of devices like the Stingray.

Awhile ago my friend Jeff introduced me to Red Phone by Open Whisper Systems, which was an app that would encrypt your phone calls. I could never get it to work very well, so I didn’t use it, plus Jeff was the only person I talked with who used it.

Flash forward more than a year, and I’m finding that I quite often don’t get texts from Jeff, while he gets mine just fine. He did some investigation and traced the issue to TextSecure, which was an encrypted text app also from Open Whisper Systems. Apparently I was registered on his phone as a TextSecure user, so it was trying to send text to me by that method. Since I no longer had Red Phone on my device (I play a lot with the software on my mobile devices and had not restored it after a clean install) I wasn’t getting the messages.

I went to install TextSecure and found that it has been replaced by Signal. My, what a difference a year makes. Not only was it easy to use, the app itself is pretty nice. It combines both TextSecure and Red Phone features, and is now the default SMS application on my handy.

Signal is 100% open source. The only way for true security is if everyone has the opportunity to examine the code and look for vulnerabilities. Plus, think about it, if you care about security chances are you want to send sensitive information using the service. Without open source you can’t be sure that information isn’t being intercepted by third parties.

This has resulted in some pretty high endorsements:

Quotes about Signal

Signal is available for both Android and iOS, Note that is uses a data connection to send encrypted SMS messages, so it will count against your data cap. I haven’t had the chance to try out the phone functionality as of yet, but it works fine as a normal SMS client as well.

It is nice to come across such a useful piece of software that is 100% open source, and if I happen to send you SMS messages, be on notice that I will be sending you an invite to Signal (grin).

UPDATE: This is so cool. Since the app uses data instead of the SMS protocol for encrypted texts, it works as long as the mobile device has data. Which means that I can get texts no matter what SIM card is currently in my handy. Cool! So I’m in Germany using my Ortel SIM and I’m able to get SMS messages from friends in the US who have no idea where I am or what network I’m using. Killer feature.

First Look at Ubuntu Gnome 15.10

Back when I was an Apple fanboy, I would eagerly await the announcement of new products by Steve Jobs, with one window open to the live blog feed and the other refreshing the Apple Store page so I could be the first to order the new shiny. Steve Jobs made me fall in love with my technology.

I’ve rarely felt that since, but when the new Dell XPS 13 came out I became once again attached to a laptop and I was determined to make it work under Linux.

While it ships with the latest stable Ubuntu release, 14.04, there are issues. Now I often say that we in the open source community suffer an embarrassment of riches when it comes to choice. Since I’ve found that Linux Mint with Cinnamon works best for me I tried it, but I just could not get it to work with the XPS. To address the shortcomings in Ubuntu 14.04, I read Barton’s Blog and decided to upgrade to 15.04. That addressed a lot of the problems, and I used Ubuntu with Unity for awhile, and although Unity was my first real Linux desktop it doesn’t work as well for me anymore. I also found that its HiDPI support was not quite there. I also tried Kubuntu but its HiDPI support (in my experience) was even worse, and since I’d based my laptop I figured I’d give Ubuntu Gnome a shot.

Now I wasn’t one of those haters who just ranted on Gnome 3.0, but when it came out I couldn’t get used to it. However, when I went to install Ubuntu Gnome on the XPS, I was encouraged that the installer recognized out of the box that I was on a HiDPI screen. There have been a lot of changes since that initial release and I found myself warming to it.

I do want to note that while I found all the desktop options I tried to be pleasantly polished, and, well, “pretty”, I decided to stick with Ubuntu Gnome.

A pesky issue with the touchpad and the touch screen required the 4.1 kernel or later. For months I’ve been running mainline kernels, so when 15.10 was announced with the 4.2 kernel standard, I was eager for the upgrade, and I ran it as soon as it became available.

So what does 15.10 offer? All I can really say at the moment is that it offers a pretty painless upgrade process. I ran “do-release-upgrade -d” and after answering a few prompts it went on its merry way.

Wireless worked out of the box (I used to have to futz with the Broadcom driver when on mainline) and overall the system seemed to be pretty smooth. During the boot process I get this error concerning lvmetad which I think is due to the fact that my entire laptop disk is encrypted, but the boot completes without any other issue and I have confidence it will soon be addressed.

Speaking of boot, Ubuntu Gnome has changed the logo on the boot screen. Instead of the familiar foot:

Old Ubuntu Gnome Logo

You get this new one:

New Ubuntu Gnome Logo

Forgive the quality as I had to produce the second image by taking a picture of the screen. While I like that the colors have been softened from black to a gray, I don’t like the new logo, which looks like two U’s mating. I think it is supposed to represent “UG” but I still don’t like it (and I tend to embrace change). I’m hoping someone puts together a splash screen replacement.

The only real issue that is driving me bonkers at the moment concerns the touchpad. One thing Apple just nailed is the touchpad and the Synaptics one on the XPS is oh so close.

The problem I’m experiencing concerns the cursor jumping when I left click. There are no “real” buttons, so you left click by depressing the lower left corner of the touchpad (or clickpad, whatever it is officially called). Sometimes when this happens, instead of registering a click the cursor will jump to the lower left corner of the screen, and *then* click. It is real annoying in Thunderbird since the icon in the lower left corner puts it in offline mode.

I’ve tried most of the suggestions I’ve found in the t00bz but nothing has helped. I just found a reference to HorizHysteresis and VertHysteresis so I’ll play with those values and see if it helps (update – doesn’t seem to). Not quite sure what they do, however. I think the issue has something to do with a finger from my right hand still grazing the touchpad surface when I make the click.

On the upside, the palm detection issues I was dealing with seem to be improved. Not sure if they have been solved but I’m not noticing it as much. Could be that I’ve just modified my typing form to avoid the touchpad better.

Overall, I’m pretty pleased with the upgrade. It should set up a nice base for the next LTS release, 16.04. I’m not quite willing to give up Linux Mint on the desktop just yet, and I’ll probably try out Mint 18 when it is released next year, but Ubuntu Gnome 15.10 has at least made switching a possibility.

One final note, I like the new shiny and I’m willing to put up with a lot in order to play with it. I give money to Dell to encourage them to supply more Linux offerings, but the downside is that Dell leads with devices designed for Windows first. If you want a true Linux experience with zero issues, check out the offerings from System 76. Our Sable all-in-one desktops Just Worked™.

Okay, so that wasn’t the final note. While I doubt any of my three readers work for major laptop vendors, I really want to see a push for physical kill switches on things like the camera and the microphone, such as on the Librem 15. I considered getting one of those but they are a little sketchy on what “PureOS” actually is, and so I’ll wait to see what others think of it first.

Review: Varidesk Standing Desk

Several years ago we did a lot of work in Sweden (Hi, Lasse!), and that is where I first saw some really nice standing desks. The first standing desk I ever saw was when I worked at Northern Telecom and it was for an employee who needed one due to health reasons, but it was fixed in place. The ones they had in Sweden (from IKEA, ‘natch) had a little switch that you could use to raise and lower it as needed, and they had places to mount a PC and run cables so they wouldn’t get snagged when it moved.

When I looked for them for the office, I was shocked by the price. A decent one with options pushed $900 and they could go north of $2000 fully loaded. While I’ve read a lot about the health benefits of standing I just couldn’t afford to get such a desk.

Recently I was on an American Airlines flight, and I just happened to see a small ad for something called a Varidesk in the in-flight magazine (and I’ve never bought something from the back of an in-flight magazine). This was something you put on an existing desk and you could use it to lift a monitor, keyboard, etc. to a standing height. It was manual, but it was considerably less than a dedicated desk.

Now, being the CEO of a profitable company it is required that I have the huge executive desk, so I do. Of course, mine was free from a business that was moving offices and all I had to do was go get it, and then repair all of the broken bits so I could put it back together. My monitor sits in one corner of this monstrosity, and I was happy to see Varidesk made a product that would fit perfectly.

Varidesk Lowered

First off, the sucker’s heavy. It cost a lot to ship due to its weight, but that translates to a lot of stability when raised. The unit I bought had a shelf for the monitor, speakers, etc., with a lower shelf for the keyboard.

In the upper shelf you will notice two holes. You place your hands through them to release levers which will allow you to raise the desk. It does take strength to get it started, but then it is balanced so that it becomes easier.

(Note: the little green light on my PC is my OpenNMS Blink notification)

Varidesk Raised

I love that everything comes up with it: the speakers, the monitor, the keyboard, my Yeti mic, etc. It will also go fairly high – I’m a little over six feet tall and I can get it high enough that I’m comfortable using it. It isn’t perfectly stable, if you are energetically pounding on the keyboard it will move slightly, but it is easy to get use to it. I did have to get some USB cable extenders to make sure things like my camera didn’t go flying off when I raised the desk, but outside of that it pretty much worked out of the box.

And, yes, when standing I like to crank the tunes and dance. You do not want to see me dance.

The Varidesk is well built and I did find myself using it, so some of the other guys in the office were interested. They don’t have fancy executive desks, so I got a slightly cheaper model that fit theirs better.

Varidesk Developers

We bought three more and everyone seems to enjoy them, although we probably don’t use them as much as we should. Because they are stylish and convertible, even in the down position they look good.

I found that after about an hour of standing my legs started to hurt. Our office, like many, is pretty much industrial carpet over concrete. There is little padding, so I bought a pad on Amazon that works well for me and I can dance longer.

It’s also cool to elevate the laptop for our daily scrum call:

Varidesk During Scrum

So, if you are thinking about getting a standing desk but already own a desk, consider the Varidesk. While it isn’t the cheapest thing out there, it is well made and will give you experience to see if you even like working standing up, which would be considerably cheaper than buying a new desk and finding you didn’t like it.

I, Robot

Today is the 11th anniversary of The OpenNMS Group. We started on September 1st, 2004 with little more than a drive to build something special, a business plan of “spend less than you earn” and a mission statement of “Help Customers, Have Fun, Make Money”.

Since I’m still working and people are using software other than OpenNMS to manage their networks, I can’t say “mission accomplished” but we’re still here, we have a great team and the best users anyone could want, so by that measure we are successful.

When it comes to the team, one thing I worry about is how to connect our remote people with the folks in North Carolina. We do a lot of Hangouts, etc, but they lack the aspect of initiative – the remote guys have to be passive and just sit there. Then I got the wild idea to investigate getting a telepresence robot. Wouldn’t it be cool if remote people could pop in and drive around the office, attend meetings, etc?

After a lot of research, I decided on a robot from Double Robotics.

Robot Tarus

The buying decision wasn’t a slam dunk. It is a very iPad/iOS centric solution which bothered me, and I had some issues concerning the overall security of the platform. So, I sent in a note and ended up having a call with Justin Beatty.

It was a great call.

Double is pretty serious about security, and assuming there are no firewall issues, the connection is encrypted peer-to-peer. While there are no plans to remove the requirement that you buy an iPad in order to use the robot, they are working on an Android native client. You can drive it on almost any platform that supports the Chrome browser (such as Linux) and you can even use it on Android via Chrome. There is a native iOS app as well.

What really sold me on the company is that they are a Y Combinator project, and rather than focus on raising more capital, they are focused on making a profit. They are small (like us) and dedicated to creating great things (like us).

Justin really understood our needs as well, as he offered us a refurbished unit at a discount (grin).

Anyway, I placed an order for a Double and (gulp) ordered an iPad.

It was delivered while I was away in England, but I was able to get it set up on Monday when I returned to the office. They have a number of easy to follow videos, and it probably took about 20 minutes to understand how everything went together.

You take the main body of the robot out of the box and place it on the floor. I had purchased an external speaker kit (otherwise, it uses the iPad speaker) which makes it look like a little Dalek, and you install that on the main post. Then you plug in the iPad holder and screw it to the post with a bolt. That’s about it for robot assembly.

The next step is to take the USB charging cable that came with your tablet and mount it inside the iPad holder. You then insert the iPad upside down and connect the cable so that the robot can power and recharge the iPad. The Double supports any iPad from version 2 onward, and they have a spacer to use for the iPad Air (which is thinner). Finally, you connect a directional microphone into the audio slot on the bottom of the iPad (or top, depending on how you look at it) and the unit is assembled.

Then I had to set up the iPad, which was a bit of a pain since I’m no longer an Apple person and needed a new Apple account (and then I had to update iOS), but once it was configured I could then pair the iPad to the robot via bluetooth. Next, I had to download the Double app from the App Store and create a Double account. Once that process was complete, I could login to the application on the tablet and our robot was ready to go.

To “drive” the robot, you log in to a website via Chrome. There are controls in the webapp for changing the height of the unit, controlling audio and video, and you move the thing around with the arrow keys.

It’s a lot of fun.

When moving you want to have the robot in its lowest height setting. Not only will it go faster, it will be more stable. This isn’t an off road, four wheeling type of robot – it likes smooth services. There is a little bump at the threshold to my office and once the robot has gone over that you want to wait a second or two because it will wobble back and forth a little bit. Otherwise, it does pretty well, and because the rubber wheels are the part of the robot that stick out the most in the front and the back; if you run into a wall it won’t damage the iPad.

I did have to mess with a couple of things. First of all, it needed a firmware upgrade before the external audio speaker would work. Second, sometimes it would keep turning in one direction (in my case, to the right), but restarting the browser seem to fix that.

You do need to be careful driving it, however. One of my guys accidentally drove it into a table, so it hit the table along the “neck” of the robot and not on the wheels. This caused the unit to shoot backward, recover and then try to move forward. It fell flat on its face.

Which, I am thankful, did no damage. The iPad is mounted in a fairly thick case, and while I wouldn’t want to test it you are probably safe with the occasional face plant.

I bought an external wireless charger which allows you to drive the robot into a little “dock” for charging instead of plugging it in. To help park it, there is a mirror mounted in the iPad holder that directs the rear camera downward so you can see where you are going (i.e. look at the robot’s “feet”). Pretty low tech but they get points for both thinking about it and engineering such a simple solution.

Everyone who has driven it seems to like it, although I’m thinking about putting a bell on the thing. This morning I was jammin’ to some tunes in my office when I heard a noise and found Jeff, piloting the robot, directly behind me. It was a little creepy (grin).

I bought it with a nice (i.e. expensive) Pelican case since the plan is to take it on road trips. I bought the iPad that supports 4G SIM cards so I should be able to use it in areas without WiFi. It’s first outing will be to the OpenNMS Users Conference, which is less than a month away. If you haven’t registered yet, you should do so now, and you’ll get to see the robot in action.

Robot Bryan

Bad Voltage will also be there, with Bryan Lunduke piloting the robot from his home in Portland. I had him try it out today and he commented “So rad. So very, very rad”.

At the moment I’m very pleased with the Double from Double Robotics. It’s a little spendy but loads of fun, and I can’t wait to use it for team meetings, etc, when people can’t make it in person. You can also share the output from the unit with other people with the beta website, although you could always just do a Google Hangout and share the screen.

Double Logo

I even like the Double Robotics logo, which is a silhouette of the robot against a square background to form a “D”. I am eager to see what they do in the future.