Strong Encryption and Death

I try to use strong encryption wherever I can. While I doubt it will keep my thoughts from prying eyes forever, at least it should make peeking a little harder.

But it dawned on me: what happens when I die? I want to let my business partners see what is on my encrypted desktop and I know my wife will need access to the files on my systems at home. I could share them with her now, but my passphrases are complex and she isn’t very familiar with the operating systems I use.

Now I’m not planning on dying any time soon, in fact I want to live until I am at least 95 and a half. Why that age? Because that is when Halley’s Comet will return. I saw the comet when I was living in California in 1986 and I could care less about seeing it again, but I do want to be the old guy they interview:

“Back in ’86, now that’s 1986 for you young folks, I was livin’ in Los Angeles. The comet was too dim to see in the city, so we drove out to Joshua Tree …”

Halley's Comet 1986

So, how do I safely pass on my important passphrases? This is the solution I chose.

I created a file called “deathnote.txt” which I then encrypted using GPG:

gpg --encrypt --recipient tarus@opennms.com \
    --recipient alice@example.com \
    --recipient bob@example.com deathnote.txt

This will encrypt the file so that both Bob and Alice can read it (and I can too). I then sent it to several friends unrelated to them with instructions that, upon my death (but not before), please send this file to Bob and Alice. I also remembered to include a copy of my GPG private key:

gpg --export-secret-keys -a tarus@opennms.com

Just in case they can’t find it on my systems.

This does require a certain level of trust in my friends, but I am blessed with having several I can count on. As long as I remember to keep it updated this should provide a secure way to pass on this important information, although I hope no one has to use it any time soon.

Review: Copperhead OS

A few weeks ago I found an article in my news feed about a Tor phone, and it introduced me to Copperhead OS. This is an extremely hardened version of the Android Open Source Project (AOSP) designed for both security and privacy. It has become my default mobile OS so I thought I’d write about my experiences with it.

TL;DR: Copperhead OS is not for everyone. Due to its focus on security is it not easy to install any software that relies on Google Services, which is quite a bit. But if you are concerned with security and privacy, it offers a very stable and up to date operating system. The downside is that I am not able to totally divorce myself from Google, so I’ve taken to carrying two phones: one with Copperhead and one with stock Android for my “Googly” things. What we really need is a way to run a hypervisor on mobile device hardware. That way I could put all of my personal stuff on a Copperhead and the stuff I want to share with Google in a VM.

I pride myself to the point of being somewhat smug about the fact that I use free software for most of my technology needs, or so I thought. My desktops, laptop, servers, router, DVR and even my weather station all use free and open source software, and I run OmniROM (an AOSP implementation) on my phone. I also “sandbox” my Google stuff – I only use Chrome for accessing Google web apps and I keep everything else separate (no sharing of my contacts and calendar, for example). So, I was unpleasantly surprised at how much I relied on proprietary software for my handy (short for “hand terminal” or what most people call a “mobile phone”, but I rarely use the “phone” features of it so it seems like a misnomer).

But first a little back story. I was sitting on the toilet playing on my mobile device (“playing on my handy” seemed a little rude here) when I came across a page that showed me all of the stuff Google was tracking about my mobile usage. It was a lot, and let’s just say any bathroom issues I was having were promptly solved. They were tracking every call and text I made, which apps I opened, as well as my location. I knew about the last one since I do play games like Ingress and Pokémon Go that track you, but the others surprised me. I was able to turn those off (supposedly) but it was still a bit shocking.

Of course, I had “opted in” to all of that when I signed in to my handy for the first time. When you allow Google to backup your device data, you allow them to record your passwords and call history.

Google Backup Terms

If you opt in to help “improve your Android experience”, you allow them to track your app usage.

Google App Terms

And most importantly, by using your Google account you allow them to install software automatically (i.e. without your explicit permission).

Google Upgrade Terms

Note that this was on a phone running OmniROM, and not stock Google, but it still looks like you have to give Google a lot of control over your handy if you want to use a Google account.

Copperhead OS is extremely focused on security, which implies the ability to audit as much software on the device as possible, as well as to control when and what gets updated. This lead them to remove Google Play Services from the ROM entirely. Instead, they set up F-Droid as the trusted repository. All the software in F-Droid is open source, and in fact all of the binaries are built by the F-Droid team and not the developer. Now, of course, someone on that team could be compromised and put malicious software into the repo, but you’ve got to trust somebody or you will spend your entire life doing code reviews and compiling.

Copperhead only runs on a small subset of devices: the Nexus 6P, the Nexus 5X and the Nexus 9 WiFi edition. This is because they support secure boot which prevents malicious code from modifying the operating system. Now, I happened to have a 6P, so I figured I would try it out.

The first hurdle was understanding their terminology. On the download page they refer to a “factory” image, which I initially took to mean the original stock image from Google. What they mean is an image that you can use for a base install. If you flash your handy as often as I do, you have probably come across the process for restoring it to stock. You install the Android SDK and then download a “factory” image from Google. You then expand it (after checking the hash, of course) and run a “flash-all” script. This will replace all the data on your device, including a custom recovery like TWRP, and you’ll be ready to run Copperhead. Note that I left off some steps such as unlocking and then re-locking the bootloader, but their instructions are easy to follow.

The first thing you notice is that there isn’t the usual “set up your Google account” steps, because, of course, you can’t use your Google account on Copperhead. Outside of missing Google Apps, the device has a very stock Android feel, including the immovable search bar and the default desktop background.

This is when reality began to set in as I started to realize exactly how much proprietary software I used to make my handy useful.

The first app I needed to install was the Nova Launcher. This is a great Launcher replacement that gives you a tremendous amount of control over the desktop. I looked around F-Droid for replacement launchers, and they either didn’t do what I wanted them to do, or they haven’t been updated in a couple of years.

Then it dawned on me – why don’t I just copy over the apk?

When you install a package from Google Play, it usually gets copied into the /data/apps directory. Using the adb shell and the adb pull commands from the SDK, I was able to grab the Nova Launcher software off of my Nexus 6 (which was running OmniROM) and copy it over to the 6P. Using the very awesome Amaze file explorer, you just navigate to the apk and open it. Now, of course, since this file didn’t come from a trusted repository you have to go under Security and turn off the “trusted sources” option (and be sure to turn it back on when you are done). I was very happy to see that it runs just fine without Google Services, and I was able to get rid of the search bar and make other tweaks.

Then I focused on installing the open source apps I do use, such as K-9 Mail and Wikipedia, both of which exist in F-Droid. I had been using the MX Player app for watching videos, pretty much out of habit, but it was easy to replace with the VLC app from F-Droid.

I really like the Poweramp music player, with the exception that it periodically checks in with the Play store to make sure your license is valid. Unfortunately, this has happened to me twice when I was in an airplane over the ocean, and the lack of network access meant I couldn’t listen to music. I was eager to replace it, but the default Music app that ships with Copperhead is kind of lame. It does a good job playing music, but the interface is hard to navigate. The “black on gray” color scheme is very hard to read.

Default Music Player Screenshot

So I replaced it with the entirely capable Timber app from F-Droid.

Timber Music Player Screenshot

Another thing I needed to replace was Feedly. I’m old, so I still get most of my news directly from websites via RSS feeds and not social media. I used to use Google Reader, and when that went away I switched to Feedly. It worked fine, but I bristled at the fact that it tracked my reading habits. Next to each article would be a number representing the number of people who clicked on it to read it, so at a minimum they were tracking that. I investigated a couple of open source replacements when I was pleasantly surprised to discover that Nextcloud has a built in News service. We have had a really good experience with Nextcloud over the last couple of months, and it was pretty easy to add the news service to our instance. Using OPML I was able to export my numerous feeds from Feedly into Nextcloud, and that was probably the easiest part of this transition. On the handy I used an F-Droid app called OCReader which works well.

There were still some things I was missing. For example, when I travel overseas I keep in touch with my bride using Skype (which is way cheaper than using the phone) so I wanted to have Skype on this device. It turns out that it is in the Amazon App Store, so I installed that and was able to get things like Skype and the eBay and IMDB apps (as well as Bridge Baron, which I like a lot). Note that you still have to allow unknown sources since the Amazon repository is not trusted, and remember to set it back when you are done.

This still left a handful of apps I wanted, and based on my success with the Nova Launcher I just tried to install them from apks. Surprisingly, most of them worked, although a couple would complain about Google Services being missing. I think background notifications is the main reason they use Google Services, so if you can live without that you can get by just fine.

One app that wouldn’t work was Signal, which was very surprising since they seem to be focused on privacy and security. Instead, the default messenger is an app called Silence, which is a Signal fork. It works well, but it isn’t in the Play store (at least in the US due to a silly trademark issue that hasn’t been fixed) and no one I know uses it so it kind of defeats the purpose of secure messaging. Luckily, I discovered that the Copperhead gang has published their own fork called Noise, which removes the Googly bits but still works with the rest of the Signal infrastructure, so I have been using it as my default client with no issues. Note that it is in the F-Droid app but doesn’t show up on the F-Droid website for some reason.

For other apps such as Google+ and Yelp, I rediscovered the world wide web. Yes, browsers still work, and the web pages for these sites are pretty close to matching the functionality of the native app.

There are still some things for which there is no open source replacement: Google Maps, for example. Yes, I know, by using Google Maps I am sharing my location with Google, but the traffic data is just so good that it has saved literally hours of my life by directing me around accidents and other traffic jams. OpenStreetMap is okay and works great offline, but it doesn’t know where the OpenNMS office is located (I need to fix that) and without traffic it is a lot less useful. There is also the fact that I do like to play games like Ingress and Pokémon Go, and I have some movies and other content on Google servers.

I also lost Android Wear. I really enjoy my LG Urbane but it won’t work without Google Services. I have been playing with AsteroidOS which shows a lot of promise, but it isn’t quite there yet.

Note that Compass by OpenNMS is not yet available in F-Droid. We use Apache Cordova to build it and that is not (yet) supported by the F-Droid team. We do post the apks on Github.

To deal with my desire for privacy and my desire to use some Google software, I decided to carry two phones.

On the Nexus 6P I run Copperhead and it has all of my personal stuff on it: calendar, contacts, e-mail, etc. On the Nexus 6 I am running stock Google with all my Googly bits, including maps. I still lock down what I share with Google, but I feel a lot more confident that I won’t accidentally sync the rest of my life with them.

It sucks carrying two phones. With the processors and memory in modern devices I’m surprised that no one has come up with a hypervisor technology that would let me run Copperhead as my base OS and stock Google in a VM. Well, not really surprised since there isn’t a commercial motivation for it. Apple doesn’t have a reason to let other software on its products, and Google would be shooting itself in the foot since its business model involves collecting data on everything. I do think it will happen, however. The use case involves corporations, especially those involved in privacy sensitive fields such as health care. Wouldn’t it be cool to have a locked down “business” VM that is separate from a “personal” VM with your Facebook, games and private stuff on it.

As for the Copperhead experience itself, it is pretty solid. I had a couple of issues where DNS would stop working, but those seem to have been resolved, and lately it has been rock solid except for one instance when I lost cellular data. I tried reseting the APN but that didn’t help, but after a reboot it started working again. Odd. Overall is it probably the most stable ROM I’ve run, but part of that could be due to how vanilla it is.

Copperhead is mainly concerned with security and not extending the Android experience. For example, one feature I love about the OmniROM version of the Alarm app is the ability to set an action on “shake”. For example, I set it to “shake to dismiss” so when my alarm goes off I can just reach over, shake the phone, and go back to bed. That is missing from the stock ROM (but included in AOSP) and thus it is missing from Copperhead. The upside is that Copperhead is extremely fast with updates, especially security updates.

The biggest shortcoming is the keyboard. I’ve grown used to “gesture” typing using the Google keyboard, but that is missing from the AOSP keyboard and no free third party keyboards have it either. I asked the Copperhead guys about it and got this reply:

If the open-source community makes a better keyboard than AOSP Keyboard, we’ll switch to it. Right now it’s still the best option. There’s no choice available with gesture typing, let alone parity with the usability of the built-in keyboard. Copperhead isn’t going to be developing a keyboard. It’s totally out of scope for the project.

So, not a show stopper, but if anyone is looking to make a name for themselves in the AOSP world, a new keyboard would be welcome.

To further increase security, there is a suggestion to create a strong two-factor authentication mechanism. The 6P has a fingerprint sensor, but I don’t use it because I don’t believe that your fingerprint is a good way to secure your device (it is pretty easy to coerce you to unlock your handy if all someone has to do is hold you down and force your finger on to a sensor). However, having a fingerprint and a PIN would be really secure, as the best security is combining something you have (a fingerprint) with something you know (a PIN).

So here was my desktop on OmniROM:

Old Phone Desktop

and here is my current desktop:

New Phone Desktop

Not much different, and while I’ve given up a few things I’ve also discovered OCReader and Nextcloud News, plus the Amaze file manager.

But the biggest thing I’ve gained is peace of mind. I want to point out that it is possible to run other ROMs, such as OmniROM, without Google Services, but they aren’t quite as focused on security as Copperhead. Many thanks to the Copperhead team for doing this, and if you don’t want to go through all the work I did, you can buy a supported device directly from them.

Monitoring Certificates with OpenNMS

Awhile ago I posted about how easy it was to implement SSL certificates using Let’s Encrypt.

The main issue that people encounter is that the certificates do expire, and while you can set up a cron job to automatically update them, sometimes it doesn’t work. This is why I like to use OpenNMS to check the expiration date of all the certificates I use on the network.

The documentation for the SSLCertMonitor is pretty detailed, and it can be used for almost any cert, not just the one for HTTPS. The example shows configuration for SMTPS and IMAPS as well.

SSLCertMonitor Example

What it doesn’t show is how to discover these services. You could, of course, just provision them directly via a requisition, but I’m lazy so I set up the TCP detector to look for those services on their well known ports.

SSLCertMonitor Detectors

This may result in a false positive if, for some reason, the port was in use by another application, but in practice I haven’t seen it yet.

So now I can rest assured that all my important SSL-based services have valid certificates and there shouldn’t be any interruption in service due to one expiring.

SSLCertMonitor Services Displayed

Network World Reviews OpenNMS

Today Network World published the results of a comparison among open source network monitoring applications. OpenNMS did not win but I was pretty happy with the article.

The main criticism I have is that the winner, Pandora FMS, seems to be the only one of the four reviewed that is more “open core” than “open source”. They have a large number of versions, each with different features, and you have to pay for those features based on the number of monitored devices. It seems to be difficult to have open source software that is limited in this fashion, as anyone should be able to easily remove that limit. Thus I have to assume that their revenue model is firmly based on selling software licenses, which is antithetical to open source. That said, it looks like the review was based on the “community” version of Pandora which does appear to be free software, just don’t expect any of the “enterprise” features to be available in that version any time soon.

I don’t know why I have such a visceral dislike of the “per managed node” pricing model, outside of having to deal with it back in the 1990s and 2000s. It seems like an unnecessary tax on your growth, “hey, customer, for every new device you add you have to pay for another monitoring license.” Plus, in these days of virtualization and microservices it seems silly. Our customers might spin up between 10 and 100 virtual servers as needed and tear them down just as quickly, and I can’t imagine the complexity that would get added to have to manage a license of each one of them.

Network World Comparison

Of the other applications reviewed, I’m not familiar with NetXMS but I do know Zabbix. They, like OpenNMS, are 100% open source and they are great people. It was awesome to finally meet Alexei Vladishev in person at this year’s All Things Open conference.

Alexei Vladishev and Tarus Balog

The only other thing that immediately pushed a button was the sentence “All four products were surprisingly good.” At first I took it to express surprise that free software could also be good, but then I calmed down a bit and figured they meant it was surprising that all four applications were strong.

For the article they installed OpenNMS on Windows. When I read that my heart just sank, because while it does run on Windows our support of that operating system grew out of a bet. We were talking many years ago about Java’s “write once, run anywhere” slogan and I mentioned that if that were true, why don’t we run on Windows? The team took up the challenge and it took two weeks to port. The first week was spent getting the few bits of code written in C to compile on Windows, and the second week on soft-coding the file separator character so that it would use a back-slash instead of a forward-slash. Even on Windows, the comments in the article were really positive, which make me think this whole Java thing isn’t such a bad idea after all (grin).

They used Windows because apparently was an issue with getting OpenNMS installed on CentOS 7, which was a surprise to me, but then Ronny pointed out that there can be some weird conflicts with Java and packages like LibreOffice that I don’t experience since I always do a minimal install. There is a cool installer for CentOS 7 which may help with that. We also maintain Docker images that make installation easy if you are used to that environment.

Fortunately, or unfortunately, not much has been done for OpenNMS on Windows since we got it working. It is fortunate because not much is required to keep OpenNMS running on Windows due to Java, but it is unfortunate because we really don’t have the Windows expertise that would be required to get it to run as a service, create an MSI installer, etc. Susan Perschke, the author of the article, seems to be a Windows-guru so I plan to reach out to her about improving the OpenNMS experience for Windows users.

One thing that is both common and valid is criticism of the web user interface. At the moment we spend most of our time focused on making OpenNMS even more scalable, and thus we don’t have the resources to make the user interface easier to use. That is changing, and most of the current effort goes into Compass™, the OpenNMS mobile app. The article didn’t mention it which means they probably didn’t try it out, which is more a failure on our part to market it versus an oversight on theirs.

They also didn’t talk directly about scalability, although it was listed in the comparison chart (see above). OpenNMS is designed to monitor tens of thousands to hundreds of thousands of devices with our goal to be virtually unlimited in order to address scale on the order of the Internet of Things. That is why we wrote Newts for storing performance data and are working on both the Minion and Underling to easily distribute OpenNMS functionality.

Another reason we haven’t spent much time on the user interface is that our larger customers tend not to use it much. They rely on the ReST interface to integrate their own systems with OpenNMS and on things like the Business Service Monitoring.

But still, it was nice to be included. We don’t do much direct marketing and even though typing “open source network monitoring” into Google returns OpenNMS as the first hit we are often overlooked. Let’s hope they revisit this in a year and we can impress them even more.

Android Open Source Frustrations

I used to be a huge fan of Apple products, but as they started to lock down their ecosystem the limitations they created started to bother me, so I switched to running as much open source as possible.

It wasn’t, and isn’t always now, easy. One of the gripes I still have against Apple is that their commercial success has spawned a ton of imitators who have decided to lock down their products, quite often without the Apple savvy to back it up. Unfortunately, Google seems to be joining these ranks.

I’m a fan of Google, they do a lot to support open source, and I use a Nexus 6 as my primary “hand terminal” (handy). However, I run alternative software on it, namely OmniROM, which gives me more control over my experience and security.

I pretty much run open source software on all my technology with few exceptions, one being my Android Wear watch. I noticed that there was a new update to Android Wear (version 2.0) so I went to play with it. When I launched the app I got this screen:

Android Wear App Error

(sigh)

So I went off to search for a solution to the error message “This phone has been flashed with an unsupported configuration for companion. you must re-flash it as either signed/user or unsigned/userdebug”. I found a couple of answers that suggested I edit the build.prop file and change

ro.build.type=userdebug

to

ro.build.type=user

In order to do this, you have to have root access to your phone.

(sigh)

I do root my phone, but I haven’t done it in awhile because Google has introduced this thing called “SafetyNet“. The stated purpose is to prevent malware but in practice what it does is torpedo people like me who actually want to control the software on the devices they own. If you install a custom ROM or have root access, certain applications will not run.

Now I have to choose between running the Android Wear app or, say, Pokémon Go. I chose Android Wear (I pretty much finished Pokémon Go).

The process: Boot into recovery, install SuperSU, boot into system, use a file editor to edit /system/build.prop and change ro.build.type from “userdebug” to “user”, reboot.

Android Wear Mute

So Android Wear will start now, but to add to the frustration the one feature I hoped they would fix is still broken for me. It used to be that if my watch was actively paired with the phone, it would mute ringing and other audio notifications. It doesn’t (and none of the fixes I’ve found work for me) so now I just remember to decrease the volume on the phone down to “vibrate”.

Pokemon Go Blocks root

And, I verified that Pokémon Go will not start now – it hangs on the login screen and then reports an error. This is whether or not SuperSU is enabled, and I think I would have to remove it entirely to get it to work.

Now I know that I can install other apps that will hide the fact that my phone is rooted, but if I do that the terrorists win. I would just rather use apps that don’t force me to give up my rights.

Which brings me to the last frustration. I purchased a bunch of content from Google, but now I can’t access it on this phone. I get “couldn’t fetch license”. This started recently so I believe it has something to do with SafetyNet, but repeated calls to Google Play support yielded no answers.

Google License Error - Deadpool

I have a Google 6P that is stock and doesn’t suffer from the download issue, so it stands to reason that there is some “protection” in place that is preventing me from accessing the content I purchased. I solved the problem by not buying content from Google Play anymore.

I’m pretty certain that it is only going to get worse. Google used to be much better about such things but I think they want to emulate Apple in more ways than one (see the new Pixel phone if you don’t believe me) and that is a shame for all of us.

UPDATE: I found a better way to do this that doesn’t require root. Assuming you have a custom recovery like TWRP, you can simply boot into recovery and then connect the handy to a computer. Using “adb shell” you can then access the system directory and edit the build.prop file directly.

2016 All Things Open

I made the decision to stop going to conferences for 2016, but I made an exception for All Things Open (ATO). Not only is it an amazing show, it’s also in my back yard, and the combination is not something I can pass up.

I love conferences. My favorite track is always the “hallway” track and I really enjoy spending time with people that I tend only to see these events. The problem is that I started to do the math.

In 2015, due to work travel, I was gone part or all of 26 weekends (I travel about 50% of the time, and often that means I head out on Sunday and back on Saturday). That leaves 26 weekends free. Of those, at least 10 are taken up with vacations, holidays, birthdays and other social engagements, leaving me just 16 or so weekends to myself. If I do 5 to 10 conferences, most of which are held over a weekend, I’m left with less than a weekend a month.

Plus, OpenNMS is going like gang-busters, so I really need to focus on that business. While I love open source conferences, we don’t get many customers out of them (one exception is the Ohio Linuxfest which seems to attract a large number of OpenNMS users) so it can be hard to justify the time (although they are a whole lot of fun).

Anyway, since ATO was the main show I was going to be involved with this year, we decided to host a party that first night. I also submitted some papers, and to my surprise two of them were accepted.

I headed out on Tuesday afternoon, as the wonderful team at opensource.com was hosting a gathering for contributors that night. That was a lot of fun and a number of us ended up at Foundation afterward. As a cocktail enthusiast I had always wanted to visit, but it is about an hour from my house I don’t want to drink and drive. Since I was staying downtown for the event, that issue went away and I had a great time.

The conference was held in the Raleigh Convention Center, and you could see the registration desk from my hotel room.

ATO - View from Marriott

Wednesday was start of the conference. ATO is organized by Todd Lewis, the nicest guy in open source, and he kicked off the keynotes.

ATO - Todd Lewis

Todd’s superpower is organization, and not only did the conference run smoothly, he got some great speakers. Jim Whitehurst, the CEO of Red Hat, did a talk on the social benefits of open source.

ATO - Jim Whitehurst

We also got a talk from Mark Hinkle, the VP of Marketing of the Linux Foundation. He was recruited at the last minute due to a cancellation, and I thought he did a good job especially considering his time to prepare (unlike normal, I actually had my presentations done at least a week before the conference).

ATO - Mark Hinkle

He started off with some “separated at birth” pictures between punk rockers and open source personalities, which reminded me of something that hit me when it was announced that the DB Cooper investigation was being closed.

ATO - DB Cooper and Jim Whitehurst

I think Jim was about four years old when DB Cooper hijacked that plane, but the similarity is striking.

Another keynote speaker was Jono Bacon.

ATO - Jono Bacon

Always (well, usually) interesting, I love how he has been working the relatively new field of behavioral economics into his talks of late. It is the study of how human psychology can impact economic decision-making and I think it has a lot of relevance in a field where businesses often tout the word “free”. By understanding how we behave we can better align our communities to meet the needs and desires of their participants.

After the keynotes were the individual sessions. I had two back-to back.

ATO - Tarus Balog

Thanks to Ben for the picture, which captures me in my full “Fred Flintstone” glory. Click on the pic below if you want to see the slides, and I did a interview for DZone on my talks. I did embed some video which won’t show up on the PDF, though.

My first talk was on the challenges facing us with the Internet of Things, especially when it comes to monitoring.

ATO - Silos Presentation

It was lightly attended but everyone who came seemed to get a lot out of it.

Right after that I did a new, updated version of my open source business talk.

ATO - Business Presentation

That one was standing room only, and I was really pleased with the feedback. One guy was telling me that he has seen a number of presentations about running an open source business but mine was the only one with concrete examples. I’m glad folks liked it.

Once my talks were done it was time for lunch and I was pretty much done with my obligations. The main one left was to help prepare for the OpenNMS Group sponsored concert at King’s Raleigh. We had hired MC Frontalot and his band to play a show in Portland, Oregon for OSCON, and the Doubleclicks opened. It was so much fun we decided it would be cool to bring it closer to home.

ATO - Doubleclicks

If you haven’t heard of the Doubleclicks you should check out their music. Even if you have, you might want to familiarize yourself with their catalog, especially if, like I did, you think it would be funny to shout out “Freebird!” in the middle of their show (ouch).

ATO - Mc Frontalot

The MC Frontalot set was really tight as well. I love working for professionals. We when got there and there was no keyboard and half the drum kit was missing, I was a mess. They calmly got it all sorted and then really kicked it during the show. They premiered “Freedom Feud” – a song we commissioned about free software. Front is still working on the final master and we have a video in production, so look for it to be posted soon, and thanks to Ben for the concert pics.

Even though I didn’t get to bed until about 04:30 (we eventually ended up in the hotel listening to some tracks Front is writing for the next album that’s all about the Internets) I was back up at 08:00 for Day Two of ATO. With my responsibilities out of the way it was nice to listen to the talks and visit with all the cool people in attendance.

Many thanks to everyone who came to my talks, to Todd and Company for a great show, and to OpenNMS for hosting a party for all my friends. See you next year.

Move to Let’s Encrypt – it’s soooo easy!

This weekend I wanted to play around with setting up Nextcloud on my home network (we already use it at work and it is awesome). Since I am planning on putting personal information into that app, I wanted to make sure that access to it was encrypted end-to-end.

This meant setting up SSL on my home web server. Now, it used to be that you either had to use a self-signed certificate (which could cause problems) or you had to spend a bunch of money on a certificate from a recognized Certificate Authority (CA).

Enter Let’s Encrypt. Launched in April of this year, Let’s Encrypt provides free certificates that are recognized by most of the things you need to recognize them.

I had been putting it off since dealing with certs is, quite frankly, a pain. You have to fill out a request, send it to the CA, get back a key file, install it in the write place, etc. Even with a free one I didn’t have time for the hassle.

I shouldn’t have worried – with Certbot it is dead simple. Seriously.

Certbot Screen

I went to their site (as directed from the Let’s Encrypt site) and just followed the instructions. I downloaded a script which downloaded all the required dependencies via apt, answered a few questions, and, bam, I had a functioning web server running SSL. They even prompted me if I wanted all requests to port 80 (http) to be redirected to port 443 (https) and when I said “yes” it did it for me.

The whole process took a couple of minutes.

Amazing stuff. The certificates are only good for 90 days, but they even include an automated way to update them.

Certbot Certificate Renewal

As more and more of our personal information becomes digitized, it is extremely important to use strong encryption. In the past this could be inconvenient if not outright difficult, but you really don’t have an excuse with Let’s Encrypt. Use it.

Open Core Returns from the Dead (sigh)

The last 18 months of my life have been delightfully free of “open core” companies. These were companies who pretended to be “open source”, at least in their marketing materials, yet their business model was based on selling “enterprise extensions” which consisted of proprietary software that actually had the features you wanted. Basically, the open source piece was a loss leader to get you to buy the commercial edition, and as Brian Prentice pointed out so eloquently there was no real difference between “open core” and traditional closed source software. We like to call these businesses “fauxpen source“.

Customers realized this as well, which lead most open core companies to switch their tactics. While many still maintain an open source project, they have removed the term “open source” from their websites and most of their marketing (often replacing it with “open architecture”). I’m happy with this, as it allows true open source companies like OpenNMS and Nextcloud to differentiate ourselves while allowing these other companies to still produce open source software without misleading the market.

But lately I’ve been introduced to two new licenses that offer access to the source code without meeting the ten requirements of the Open Source Definition. These licenses further muddy the waters due to giving access to the code without including the freedoms of truly open software.

The first case was from Monty Widenius, who announced a proprietary Business Source License (BSL) for some of the MariaDB products. Monty was the guy who earned €16.6 million by selling MySQL to Sun and then got upset when Sun got bought by Oracle. Apparently, he seems to be unhappy that he isn’t earning enough money from his fork of MySQL products so he wants to create commercial software but not call it that.

The BSL, or as I call it, the “Rape of Large Companies License” allows the developer to offer the code up for use for free unless you cross some sort of arbitrary threshold, also set by the developer. In three years that code will revert to an OSI approved license, in this case the GPLv2, and if you are above the usage threshold then you don’t have to pay anymore.

I’m not sure what his goals are here, outside of running a commercial software business while paying lip service to open source software. Perhaps he hopes to get people to contribute to BSL licensed projects as long as their use case is small enough not to cross the “pay me” threshold, but more likely he just wants to ride on the coattails of the success of open source software without committing to it.

I learned of another such license called the Fair Source License (FSL) from a post by Ben Boyter who writes the Searchcode Server. Ben, at least, is a lot more up front about his reasons for adding a “GPL Timebomb” to his code. Initially, the code is published under the FSL but with a switch to the GPLv3 in three years. He isn’t expecting contributions and instead has offered up the code simply so it can be audited for back doors. As this is one of the more powerful features of open source software I applaud him for doing it, but I really wish he hadn’t used the term “bomb”. I have to deal with terms like “GPL poisoning” enough in my business that negative words like that just tend to scare people. He should have called it “Happy Fun Lucky GPL Gift Giving Time!”

Look, I’m all for anything that gets more code out there under an OSI-approved license but, c’mon, three years is a lifetime in this industry. Enterprise customers, who would be most affected by this license, will still have to approach the buying decision as if a BSL or FSL licensed application were commercial software. Even with the three year revenue window, it is unclear what happens if, say, a huge security bug is discovered three years out. Does the code to fix that bug restart the clock?

The whole process is confusing and doesn’t help the cause of open source software. I think open source is awesome and extremely powerful, and when I see things like this I’m almost insulted, as if the developer is saying “when I’m done you are welcome to my leftovers”. Instead of announcing a future switch to an open source license years in advance, they should just open it when they are ready, like id Software does with the Doom engine.

I’m giving a talk at All Things Open about running a truly open source business, the core point of which is that you can’t have an open source business with a business model based around selling software. No matter how you dress it up by either calling it “open core” or “business source” it is still proprietary software.

OpenNMS Group Turns Twelve

Heh, it almost slipped my mind completely but The OpenNMS Group turned 12 years old today.

I did have to go give our co-founder, David Hustace, a hug and if we weren’t so slammed it would have been time for a beer. Raincheck.

I did spend a second reflecting on our wonderful customers who make all this possible, as well as all the people who contribute to and use OpenNMS. There are a lot of people who don’t believe a company can survive with a 100% open source model, but the funny thing is that we’ve outlived quite a few proprietary software companies in the last decade or so, thus we must be doing something right.

Our business plan of “Spend Less Money Than You Earn” and our mission statement of “Help Customers, Have Fun, Make Money” are as true today as they were in 2004. I look forward to getting ever better on delivering on both of them.

Kippis!

Nextcloud and OpenNMS

Last weekend, OpenNMS-er extraordinare Ronny Trommer was at a conference where he met Jos Poortvliet from Nextcloud. I’ve been following Nextcloud pretty intently since I recognized kindred souls in their desire to create a business that was successful and still 100% open source (and not, for example, fauxpensource). Jos mentioned that Nextcloud was getting a new monitoring API and thought it would be cool if OpenNMS could use it.

Since their API returns the monitoring information as XML, Ronny used the XML Collector to gather the data. Once the data is in OpenNMS, you can graph it, set thresholds, configure notifications, etc.

Available metrics include:

  • CPU load and memory usage
  • Number of active users over time
  • Number of shares in various categories
  • Storage statistics
  • Server settings like PHP version, database type and size, memory limits and more

Here’s an example of the number of files from a small demo system:

Files in Nextcloud

Of course, since OpenNMS is a platform, once the data is in the system you can leverage its integrations with applications such as Grafana:

Nextcloud Metrics in Grafana

Some applications will go on and on about how many “plugins” they have. Often, these are little more than scripts that do something simple, like an SNMP GET, but with all the overhead of having to run a shell. To add something like Nextcloud to OpenNMS, it is just a simple matter of configuring a couple of files, but to make that easier a lot of configurations have been added to a git repository. If you want to try out the Nextcloud integration, follow these instructions.

True open source solutions can offer the best feature, performance and value for most companies, but unfortunately there are so few pure open source companies providing them. I applaud Nextcloud and look forward to working with them for years to come.