2017 Europe: Three SIM card

Just a short post to praise the Three SIM card I bought in the UK several years ago.

I tend to buy unlocked phones and so when I travel I like to get a local SIM card, mainly for data. For this trip this was going to prove difficult, as I’m visiting five countries in nine days.

One thing I like about my Three SIM is that it never gets disabled. As long as I have a balance I have never had a problem, although I do travel enough that I end up using it at least once every six months or so. I am not able to top up the card on the Three website since I don’t have a UK credit card, so I simply use Mobiletopup.co.uk to get a £20 voucher from Paypal. Using that I just buy an “All in One 20” add-on which gives me 12GB of network access, 300 minutes and 3000 SMS messages – way more than I need. I turn that on before I leave the US and my phone works when I land.

What’s wonderful about it is that the plan is valid in any EU country. So far this trip I’ve used it in London, Helsinki and Tallinn, and I expect it to work in Riga and Brussels. I have yet to experience any network issues, although I have not moved far outside of major metropolitan areas.

I have no idea if Brexit will change this plan, but I sincerely hope not. So much of the technology I use in my life comes with headaches that I am grateful when things just work. Thanks Three.

2017 Europe: Helsinki

I am spending a week touring the eastern side of Europe, with the first stop being two nights in Helsinki. I should end up in Brussels next weekend for FOSDEM, and I am looking forward to my first time at that conference.

I’m here because I was invited to speak at an open tech conference in Riga, Latvia, and I couldn’t resist the invitation. Riga is home to Zabbix, a company very much like OpenNMS in that we both do network monitoring and we are both 100% open source. One might think this would make us enemies – quite the contrary. For some reason we really get along and also, for some reason, we rarely compete.

In trying to find a route from North Carolina to Latvia, I noticed a number of choices went through Helsinki. I had been to Helsinki once and really enjoyed it (despite it being winter). I also remembered from that trip that Finland is very close to both Russia and Estonia. You can be in St. Petersburg in three hours by train or Tallinn in two hours by ferry.

It was my goal to visit 50 countries by the time I turned 50 years old. I didn’t make that goal (I got to 37), but I figured I could use this trip to both visit Estonia and Latvia, adding two to the list.

My first flight out of RDU was canceled, so they routed me through JFK. I arrived in Helsinki three hours later than planned, but my bag made it with me so it worked out. It was dark and sleeting, but it wasn’t too difficult to take the new train into the city center and find my hotel.

HSL Train Helsinki Airport

I always like coming to Finland because it was the home of Linus Torvalds. Now I know he has lived in the US for many years and I also know he didn’t invent the idea of free software, but I still feel some sort of homecoming when I arrive since I doubt OpenNMS would be here if it weren’t for Linus.

There is an awesome company in Helsinki that is also an OpenNMS customer, so I was able to spend Monday visiting with them. Due to an NDA I can’t name them, but they are doing some amazing work in this part of the world. I got to learn more about their business as well as to share where we are going with OpenNMS.

Like many of our larger clients, they have an inventory system that they have integrated with OpenNMS in order to manage their monitoring needs. Since that system also contains customer relationships (which equipment is used to provide network services for particular clients of theirs) we played around with the Business Service Monitor (BSM). They should be able to export their network information into OpenNMS to create a customer impact topology, so that when there is an issue they can quickly determine the root cause. It is exactly why we created the feature and I’m eager to see how they use it.

They are also interested in using the Minion feature due in Horizon 19. This should allow them to easily deal with overlapping address space and any scalability concerns, plus they should be able to get rid of their current “manager of managers” solution. Exciting times.

They are looking to hire, so if you are in the area and have OpenNMS experience, send me your CV and I’ll be happy to forward it on to them.

Ulf and Hacienda Napoles at Liberty or Death

That evening, Ulf and I managed to indulge our taste for vintage and craft cocktails with a visit to Liberty or Death. This is a bar near my hotel that serves amazing cocktails in a very relaxed atmosphere. It was a nice ending to a very good day.

Ferry Terminal Statue

The next day will find me on a ferry boat to Tallinn. I don’t know of any OpenNMS users in Estonia, but I am still eager to see the city.

“OpenNMS WHO” at OSMC 2016

There is a really cool monitoring conference held each year in Germany. Called the “Open Source Monitoring Conference” (OSMC) it is put on by Netways, one of the maintainers of the Icinga project, but they welcome other projects such as OpenNMS and Zabbix.

It is a lot of fun, and usually Jeff and I fight over who gets to go. This year David won (he was in Germany for other reasons) and they now have his talk available for viewing:

It’s an overview of what we have been up to and where we are going with the Project. Check it out.

Speaking of conferences and travel, next week I plan to be in Helsinki, Tallinn, Riga and Brussels. I’ll be in Riga for the Open Tech conference and hope to spent some time with my Zabbix friends, and I’ll be in Brussels for FOSDEM where OpenNMS will have a booth. It’s my first time at either conference, and if you happen to be in the area drop me a note and perhaps we can meet up.

OpenNMS 101

One of my favorite things to do is to teach people about OpenNMS. I am one of the main trainers, and I usually run the courses we hold here at OpenNMS HQ. I often teach these classes on-site as well (if you have three or more people who want to attend, it can be cheaper to bring someone like me in for a week than to send them here), and the feedback I got from a recent course at a defense contractor was “that was the best class I’ve ever attended, except for the ones I got to blow stuff up.”.

Unfortunately, a lot of people can’t spare a week away from the office nor do they have the training or travel budget to come to our classes. And teaching them can be draining. While I can easily talk about OpenNMS for hours on end, it is much harder to do for days on end.

To help with that I’ve decided to record the lessons in a series of videos. I am not a video editing wizard, but I’ve found a setup using OBS that works well for me and I do post production with OpenShot.

The first class is called “OpenNMS 101” and we set it up as a video playlist on Youtube. The lessons are built on one another so beginners will want to start with Module 0, the Introduction, although you can choose a particular single episode if you need a refresher on that part of OpenNMS.

My goal is to put up two or three videos a week until the course material is exhausted. That will not begin to cover all aspects of OpenNMS, so the roadmap includes a follow up course called “OpenNMS 102” which will consist of standalone episodes focused on a particular aspect of the platform. Finally, I have an idea for an “OpenNMS 201” to cover advanced features, such as the Drools integration.

I’ve kept the videos as informal as the training – when I make a mistake I tend to own it and explain how to fix it. It also appears that I use “ummmmmmm” a lot as a place holder, although I’m working to overcome that. I just posted the first part of “Module 4: Notifications” and I apologize for the long running time and the next lessons will be shorter. I had to redo this one (the longest, of course) as during the first take I forgot to turn on the microphone (sigh).

We have also posted the slides, videos and supporting configuration files on the OpenNMS project website.

I’d appreciate any feedback since the goal is to improve the adoption of OpenNMS by making it easier to learn. Any typos in the slides will be fixed on the website but I am not sure I’ll be able to redo any of the videos any time soon. I think it is more important to get these out than to get them perfect.

Perfection is the enemy of done.

OpenNMS Is Once Again on FLOSS Weekly

Way back in 2006 I was invited to be on one of the first FLOSS Weekly shows. That was when it was hosted by Chris Dibona and Leo Laporte. Now it is run by the very capable Randal Swartz, and I was excited to be invited back, ten years later. It was also fun to meet Jonathan Bennett, his co-host, for the first time.

Jeff Gehlbach joined me to chat about OpenNMS and all things FLOSS, and I even thought he got a word or two in edgewise. Like FLOSS Weekly, I think our major achievement is that we are still here and still going strong (grin). The only complaint I could have is that this was episode 418 and I was originally on episode 15 so it would have been cooler to be on three shows ago to make it an even 400, but I’m OCD like that.

FLOSS Weekly

One thing I love about free (libre) and open source software is that it is self-selecting. People choose to use it, and thus there tend to be certain things we all hold in common that makes meeting others involved in FLOSS like immediately making a new friend. Chatting with Randal and Jonathan was more like catching up with old friends, although I’d never talked with them before. I look forward to this as the beginning of a beautiful friendship.

Anyone who has had the misfortune of listening to me drone on about OpenNMS in the past will here a number of “bingo” stories in this show, but we do touch on some new ideas and I think it went really well. Please check it out and let me know what you think.

Strong Encryption and Death

I try to use strong encryption wherever I can. While I doubt it will keep my thoughts from prying eyes forever, at least it should make peeking a little harder.

But it dawned on me: what happens when I die? I want to let my business partners see what is on my encrypted desktop and I know my wife will need access to the files on my systems at home. I could share them with her now, but my passphrases are complex and she isn’t very familiar with the operating systems I use.

Now I’m not planning on dying any time soon, in fact I want to live until I am at least 95 and a half. Why that age? Because that is when Halley’s Comet will return. I saw the comet when I was living in California in 1986 and I could care less about seeing it again, but I do want to be the old guy they interview:

“Back in ’86, now that’s 1986 for you young folks, I was livin’ in Los Angeles. The comet was too dim to see in the city, so we drove out to Joshua Tree …”

Halley's Comet 1986

So, how do I safely pass on my important passphrases? This is the solution I chose.

I created a file called “deathnote.txt” which I then encrypted using GPG:

gpg --encrypt --recipient tarus@opennms.com \
    --recipient alice@example.com \
    --recipient bob@example.com deathnote.txt

This will encrypt the file so that both Bob and Alice can read it (and I can too). I then sent it to several friends unrelated to them with instructions that, upon my death (but not before), please send this file to Bob and Alice. I also remembered to include a copy of my GPG private key:

gpg --export-secret-keys -a tarus@opennms.com

Just in case they can’t find it on my systems.

This does require a certain level of trust in my friends, but I am blessed with having several I can count on. As long as I remember to keep it updated this should provide a secure way to pass on this important information, although I hope no one has to use it any time soon.

Review: Copperhead OS

A few weeks ago I found an article in my news feed about a Tor phone, and it introduced me to Copperhead OS. This is an extremely hardened version of the Android Open Source Project (AOSP) designed for both security and privacy. It has become my default mobile OS so I thought I’d write about my experiences with it.

TL;DR: Copperhead OS is not for everyone. Due to its focus on security is it not easy to install any software that relies on Google Services, which is quite a bit. But if you are concerned with security and privacy, it offers a very stable and up to date operating system. The downside is that I am not able to totally divorce myself from Google, so I’ve taken to carrying two phones: one with Copperhead and one with stock Android for my “Googly” things. What we really need is a way to run a hypervisor on mobile device hardware. That way I could put all of my personal stuff on a Copperhead and the stuff I want to share with Google in a VM.

I pride myself to the point of being somewhat smug about the fact that I use free software for most of my technology needs, or so I thought. My desktops, laptop, servers, router, DVR and even my weather station all use free and open source software, and I run OmniROM (an AOSP implementation) on my phone. I also “sandbox” my Google stuff – I only use Chrome for accessing Google web apps and I keep everything else separate (no sharing of my contacts and calendar, for example). So, I was unpleasantly surprised at how much I relied on proprietary software for my handy (short for “hand terminal” or what most people call a “mobile phone”, but I rarely use the “phone” features of it so it seems like a misnomer).

But first a little back story. I was sitting on the toilet playing on my mobile device (“playing on my handy” seemed a little rude here) when I came across a page that showed me all of the stuff Google was tracking about my mobile usage. It was a lot, and let’s just say any bathroom issues I was having were promptly solved. They were tracking every call and text I made, which apps I opened, as well as my location. I knew about the last one since I do play games like Ingress and Pokémon Go that track you, but the others surprised me. I was able to turn those off (supposedly) but it was still a bit shocking.

Of course, I had “opted in” to all of that when I signed in to my handy for the first time. When you allow Google to backup your device data, you allow them to record your passwords and call history.

Google Backup Terms

If you opt in to help “improve your Android experience”, you allow them to track your app usage.

Google App Terms

And most importantly, by using your Google account you allow them to install software automatically (i.e. without your explicit permission).

Google Upgrade Terms

Note that this was on a phone running OmniROM, and not stock Google, but it still looks like you have to give Google a lot of control over your handy if you want to use a Google account.

Copperhead OS is extremely focused on security, which implies the ability to audit as much software on the device as possible, as well as to control when and what gets updated. This lead them to remove Google Play Services from the ROM entirely. Instead, they set up F-Droid as the trusted repository. All the software in F-Droid is open source, and in fact all of the binaries are built by the F-Droid team and not the developer. Now, of course, someone on that team could be compromised and put malicious software into the repo, but you’ve got to trust somebody or you will spend your entire life doing code reviews and compiling.

Copperhead only runs on a small subset of devices: the Nexus 6P, the Nexus 5X and the Nexus 9 WiFi edition. This is because they support secure boot which prevents malicious code from modifying the operating system. Now, I happened to have a 6P, so I figured I would try it out.

The first hurdle was understanding their terminology. On the download page they refer to a “factory” image, which I initially took to mean the original stock image from Google. What they mean is an image that you can use for a base install. If you flash your handy as often as I do, you have probably come across the process for restoring it to stock. You install the Android SDK and then download a “factory” image from Google. You then expand it (after checking the hash, of course) and run a “flash-all” script. This will replace all the data on your device, including a custom recovery like TWRP, and you’ll be ready to run Copperhead. Note that I left off some steps such as unlocking and then re-locking the bootloader, but their instructions are easy to follow.

The first thing you notice is that there isn’t the usual “set up your Google account” steps, because, of course, you can’t use your Google account on Copperhead. Outside of missing Google Apps, the device has a very stock Android feel, including the immovable search bar and the default desktop background.

This is when reality began to set in as I started to realize exactly how much proprietary software I used to make my handy useful.

The first app I needed to install was the Nova Launcher. This is a great Launcher replacement that gives you a tremendous amount of control over the desktop. I looked around F-Droid for replacement launchers, and they either didn’t do what I wanted them to do, or they haven’t been updated in a couple of years.

Then it dawned on me – why don’t I just copy over the apk?

When you install a package from Google Play, it usually gets copied into the /data/apps directory. Using the adb shell and the adb pull commands from the SDK, I was able to grab the Nova Launcher software off of my Nexus 6 (which was running OmniROM) and copy it over to the 6P. Using the very awesome Amaze file explorer, you just navigate to the apk and open it. Now, of course, since this file didn’t come from a trusted repository you have to go under Security and turn off the “trusted sources” option (and be sure to turn it back on when you are done). I was very happy to see that it runs just fine without Google Services, and I was able to get rid of the search bar and make other tweaks.

Then I focused on installing the open source apps I do use, such as K-9 Mail and Wikipedia, both of which exist in F-Droid. I had been using the MX Player app for watching videos, pretty much out of habit, but it was easy to replace with the VLC app from F-Droid.

I really like the Poweramp music player, with the exception that it periodically checks in with the Play store to make sure your license is valid. Unfortunately, this has happened to me twice when I was in an airplane over the ocean, and the lack of network access meant I couldn’t listen to music. I was eager to replace it, but the default Music app that ships with Copperhead is kind of lame. It does a good job playing music, but the interface is hard to navigate. The “black on gray” color scheme is very hard to read.

Default Music Player Screenshot

So I replaced it with the entirely capable Timber app from F-Droid.

Timber Music Player Screenshot

Another thing I needed to replace was Feedly. I’m old, so I still get most of my news directly from websites via RSS feeds and not social media. I used to use Google Reader, and when that went away I switched to Feedly. It worked fine, but I bristled at the fact that it tracked my reading habits. Next to each article would be a number representing the number of people who clicked on it to read it, so at a minimum they were tracking that. I investigated a couple of open source replacements when I was pleasantly surprised to discover that Nextcloud has a built in News service. We have had a really good experience with Nextcloud over the last couple of months, and it was pretty easy to add the news service to our instance. Using OPML I was able to export my numerous feeds from Feedly into Nextcloud, and that was probably the easiest part of this transition. On the handy I used an F-Droid app called OCReader which works well.

There were still some things I was missing. For example, when I travel overseas I keep in touch with my bride using Skype (which is way cheaper than using the phone) so I wanted to have Skype on this device. It turns out that it is in the Amazon App Store, so I installed that and was able to get things like Skype and the eBay and IMDB apps (as well as Bridge Baron, which I like a lot). Note that you still have to allow unknown sources since the Amazon repository is not trusted, and remember to set it back when you are done.

This still left a handful of apps I wanted, and based on my success with the Nova Launcher I just tried to install them from apks. Surprisingly, most of them worked, although a couple would complain about Google Services being missing. I think background notifications is the main reason they use Google Services, so if you can live without that you can get by just fine.

One app that wouldn’t work was Signal, which was very surprising since they seem to be focused on privacy and security. Instead, the default messenger is an app called Silence, which is a Signal fork. It works well, but it isn’t in the Play store (at least in the US due to a silly trademark issue that hasn’t been fixed) and no one I know uses it so it kind of defeats the purpose of secure messaging. Luckily, I discovered that the Copperhead gang has published their own fork called Noise, which removes the Googly bits but still works with the rest of the Signal infrastructure, so I have been using it as my default client with no issues. Note that it is in the F-Droid app but doesn’t show up on the F-Droid website for some reason.

For other apps such as Google+ and Yelp, I rediscovered the world wide web. Yes, browsers still work, and the web pages for these sites are pretty close to matching the functionality of the native app.

There are still some things for which there is no open source replacement: Google Maps, for example. Yes, I know, by using Google Maps I am sharing my location with Google, but the traffic data is just so good that it has saved literally hours of my life by directing me around accidents and other traffic jams. OpenStreetMap is okay and works great offline, but it doesn’t know where the OpenNMS office is located (I need to fix that) and without traffic it is a lot less useful. There is also the fact that I do like to play games like Ingress and Pokémon Go, and I have some movies and other content on Google servers.

I also lost Android Wear. I really enjoy my LG Urbane but it won’t work without Google Services. I have been playing with AsteroidOS which shows a lot of promise, but it isn’t quite there yet.

Note that Compass by OpenNMS is not yet available in F-Droid. We use Apache Cordova to build it and that is not (yet) supported by the F-Droid team. We do post the apks on Github.

To deal with my desire for privacy and my desire to use some Google software, I decided to carry two phones.

On the Nexus 6P I run Copperhead and it has all of my personal stuff on it: calendar, contacts, e-mail, etc. On the Nexus 6 I am running stock Google with all my Googly bits, including maps. I still lock down what I share with Google, but I feel a lot more confident that I won’t accidentally sync the rest of my life with them.

It sucks carrying two phones. With the processors and memory in modern devices I’m surprised that no one has come up with a hypervisor technology that would let me run Copperhead as my base OS and stock Google in a VM. Well, not really surprised since there isn’t a commercial motivation for it. Apple doesn’t have a reason to let other software on its products, and Google would be shooting itself in the foot since its business model involves collecting data on everything. I do think it will happen, however. The use case involves corporations, especially those involved in privacy sensitive fields such as health care. Wouldn’t it be cool to have a locked down “business” VM that is separate from a “personal” VM with your Facebook, games and private stuff on it.

As for the Copperhead experience itself, it is pretty solid. I had a couple of issues where DNS would stop working, but those seem to have been resolved, and lately it has been rock solid except for one instance when I lost cellular data. I tried reseting the APN but that didn’t help, but after a reboot it started working again. Odd. Overall is it probably the most stable ROM I’ve run, but part of that could be due to how vanilla it is.

Copperhead is mainly concerned with security and not extending the Android experience. For example, one feature I love about the OmniROM version of the Alarm app is the ability to set an action on “shake”. For example, I set it to “shake to dismiss” so when my alarm goes off I can just reach over, shake the phone, and go back to bed. That is missing from the stock ROM (but included in AOSP) and thus it is missing from Copperhead. The upside is that Copperhead is extremely fast with updates, especially security updates.

The biggest shortcoming is the keyboard. I’ve grown used to “gesture” typing using the Google keyboard, but that is missing from the AOSP keyboard and no free third party keyboards have it either. I asked the Copperhead guys about it and got this reply:

If the open-source community makes a better keyboard than AOSP Keyboard, we’ll switch to it. Right now it’s still the best option. There’s no choice available with gesture typing, let alone parity with the usability of the built-in keyboard. Copperhead isn’t going to be developing a keyboard. It’s totally out of scope for the project.

So, not a show stopper, but if anyone is looking to make a name for themselves in the AOSP world, a new keyboard would be welcome.

To further increase security, there is a suggestion to create a strong two-factor authentication mechanism. The 6P has a fingerprint sensor, but I don’t use it because I don’t believe that your fingerprint is a good way to secure your device (it is pretty easy to coerce you to unlock your handy if all someone has to do is hold you down and force your finger on to a sensor). However, having a fingerprint and a PIN would be really secure, as the best security is combining something you have (a fingerprint) with something you know (a PIN).

So here was my desktop on OmniROM:

Old Phone Desktop

and here is my current desktop:

New Phone Desktop

Not much different, and while I’ve given up a few things I’ve also discovered OCReader and Nextcloud News, plus the Amaze file manager.

But the biggest thing I’ve gained is peace of mind. I want to point out that it is possible to run other ROMs, such as OmniROM, without Google Services, but they aren’t quite as focused on security as Copperhead. Many thanks to the Copperhead team for doing this, and if you don’t want to go through all the work I did, you can buy a supported device directly from them.

Monitoring Certificates with OpenNMS

Awhile ago I posted about how easy it was to implement SSL certificates using Let’s Encrypt.

The main issue that people encounter is that the certificates do expire, and while you can set up a cron job to automatically update them, sometimes it doesn’t work. This is why I like to use OpenNMS to check the expiration date of all the certificates I use on the network.

The documentation for the SSLCertMonitor is pretty detailed, and it can be used for almost any cert, not just the one for HTTPS. The example shows configuration for SMTPS and IMAPS as well.

SSLCertMonitor Example

What it doesn’t show is how to discover these services. You could, of course, just provision them directly via a requisition, but I’m lazy so I set up the TCP detector to look for those services on their well known ports.

SSLCertMonitor Detectors

This may result in a false positive if, for some reason, the port was in use by another application, but in practice I haven’t seen it yet.

So now I can rest assured that all my important SSL-based services have valid certificates and there shouldn’t be any interruption in service due to one expiring.

SSLCertMonitor Services Displayed