2015 Open Source Monitoring Conference

Once again I got to visit the wonderful town of Nürnberg, Germany, for the Open Source Monitoring Conference.

OSMC - Badge

Hosted by Netways, the conference started out ten years ago as a Nagios conference. The name was changed due to an issue with the Nagios trademark, but it still focused heavily on Nagios. However, the organizers are pretty open to all things monitoring, so they started inviting projects like Zabbix and OpenNMS to come. When the Nagios fork Icinga was created, the amount of Nagios content dropped considerably, and out of 24 talks over 2 days there were only two that had Nagios in the title. Part of this has to do with Icinga 2 being a total rewrite and thus has started to move past its Nagios roots.

This year it was a cornucopia of monitoring choices. In addition to Icinga, Zabbix and OpenNMS, there was Alyvix, Assimilation, Heroic, and Prometheus. Grafana was popular and most tools are adding support for that data visualization tool, and it was nice to see talks on NSClient++ and MQTT. A little less than half the talks were in German, so there is a large German focus to the conference, but there was always an English-language talk available as well.

Nürnberg is a cool town. There is a big castle and lots of walls are left over from the original fortifications for the city. It is also home to SuSE Linux, and I made sure to swing by if just to get a picture for Bryan Lunduke:

OSMC - SuSE Office

Ronny and I got there on Monday. While the main conference is held over two days, this year there were workshops on Monday and a “hack-a-thon” on Thursday. The conference pretty much takes over the Holiday Inn, City Center, hotel. While the facilities are nice, it is right next to the city’s “eros center” which seems to creep closer and closer to the hotel each year I attend. It doesn’t impact the conference in any way, and those who might be sensitive to such things can easily avoid it.

There is always lavish catering and this year we had a nice, small crowd of OpenNMS enthusiasts in attendance, and we met up for the hosted dinner on Monday night. I had not seen some of the people since the OUCE, so it was nice to catch up.

My talk was on Tuesday, the first day of the main conference. The event was sold out, with about 250 people, and at times the rooms could get quite full.

OSMC - Crowd

The talks were all rather good. Torkel Ödegaard talked about Grafana:

OSMC - Grafana

which was a big hit with crowd, and as I mentioned before a lot of projects are leveraging his work to provide better data visualization, including OpenNMS. My talk went well (I think) as I went over all of the amazing things we’ve done since last year at the OSMC, which included four major releases of our application. I was stumped with the question “How do I get started with OpenNMS?” when I realized that I didn’t have an easy answer. I can tell you how to install it, but that doesn’t get you started. I need to work on that.

That evening we returned to Terminal 90, which is an odd place to hold a dinner but it seems to work. Terminal 90 is a restaurant located at the Nürmberg airport, and it does a good job of holding everyone. We have to take the U-bahn to get there, and at least this year there were no incidents (last year someone tried to hold open the doors, which caused the autonomous train to shut down and wait for human intervention).

OSMC - Terminal 90

The food and drinks were good, and toward the end of the evening they had woman impersonating German pop star Helene Fischer, which was lost on me but the crowd seemed to enjoy it.

I called it a night fairly early, but this is a group that tends to hang out until the wee hours of the morning. Although my room was on the first floor, I didn’t hear much noise from “Checkpoint Jenny” across the street, so maybe everyone is getting more mellow in their old age. (grin)

The second day featured a number of talks from different projects. Usually the Zabbix talk is done by Rihards Olups, but he was unable to make it this year so Wolfgang Alper did the honors.

OSMC - Zabbix

After that was a really good talk by Martin Parm on how Spotify monitors its music service.

OSMC - Spotify

It started out with all of the tools they tried that failed, and I kept thinking to myself “don’t let it be OpenNMS, don’t let it be OpenNMS” (it wasn’t) and ended with a tool they wrote in-house called Heroic. It is a time-series data store built on top of Cassandra, and it looks a lot like the Newts tool we built. Both are open source and Apache-licensed so I’m hoping to find some synergy between the two projects. There is another large music streaming service that uses OpenNMS, but maybe we can get all of them (grin).

OSMC - Prometheus

Then there was a talk by Fabian Reinhartz on a monitoring system called Prometheus. I had to joke that the name refers to the daily experience of most network managers of having their liver eaten out, but it seems like an interesting tool. Written in Go, it may find resistance from users due to the configuration being more like writing code, but that also makes it powerful. Sounds familiar to me.

I had to leave right after lunch in order to be ready to catch my flight home, but I really enjoyed my time there, even more than usual. Many thanks to Bernd Erk and the Netways gang for holding it, and they should be posting the videos soon. If you are interested in next year be sure to register early as it is likely to sell out again.

Open Source Software and Corporations

An interesting post caught my eye this week entitled “Corporations and OSS Do Not Mix” by Ian Cordasco. It was kind of depressing – here was a person who had spent a lot of free time contributing to open source code, but the actions of some users of that code had taken the fun out of it.

My only issue with it was the targeting of “corporations” in the title. At OpenNMS we have a large number of corporate customers and we get along with them just fine. I want to talk about that in a bit, but first I want to address some of the other experiences Ian had that were similar to mine.

When I became the maintainer of OpenNMS back in 2002, I would often get e-mails from people that would start out with “OpenNMS is good, but what you need to do is …”. I used to spend a lot of time responding to them, pointing out that it was open source and anyone can help contribute to it, so they didn’t have to wait on me to do anything, but it never really helped and it turned into a huge time suck. I started to send back a generic e-mail that went along the lines of “OpenNMS is an enterprise product and if you won’t take the time to understand it then you should try something easier like Nagios” which would usually result in a reply calling me an asshole, but it took little of my time and then conversation was over. Now I pretty much just ignore them.

When you create something and share it, you are putting a bit of yourself out there and there are bound to be critics. For the most part they can be ignored, and you have to develop a thick skin to be in this environment. I’ve found that overall the good far outweighs the bad, and if you can learn to brush off the bad you can be very happy working in open source.

People tend to forget that open source “business” is still “business”. People exchange money in return for services. If I had Ian’s talent I would simply set up various custom development options, so when someone complained about a bug he could just return an e-mail with a price list. If you don’t have time to do it, make the prices really, really large – large enough that you would make time to do it. It’s your life – you are in the driver’s seat. I used to give a talk on running an open source business and I always stressed that you should never compete on price, or at least you shouldn’t lead with “my solution is cheaper”. Sure, open source software can provide tremendous savings over the life of the solution, but that doesn’t mean the solution itself is inexpensive to get set up. Done right, it will be better than any proprietary solution, but that doesn’t mean it comes without cost.

Always remember: free software does not mean free solution.

Getting back to dealing with corporations, like any interaction between two parties is it extremely important to set up expectations. You need to clearly outline what the product the client is buying covers (response time, 24/7 support, etc.). If they aren’t buying anything, then you don’t need to worry about them. I chuckled when I read “Well if you’re not going to take this seriously, we’ll have to start using another project.” We often get the “use another project” line and my response is “knock yourself out”. If you want to take this seriously, then pay me for my work. It’s like going into a free kitchen and complaining the soup is too salty.

A more difficult issue comes when someone wants to submit substandard code. This does require a little effort, since you can’t be sure that this isn’t just an eager but inexperienced coder versus someone lazy. Again, expectations are important. If you publish what the base level of quality should be, such as “must include unit tests”, then you can point to that when you don’t accept a submission. Plus, git makes it very easy to track a master branch and just apply your changes, so some sort of reply about how to do that could deflect criticism about the speed in accepting pull requests.

Ian makes a lot of really good points in his post, but I think he misses a point that if you run your open source project like a business then corporations (i.e. other businesses) will respect you and treat you like a business. We have one amazing company that just hired four (!) OpenNMS developers to work on code that they need. While some of it, if not most of it, will address their particular needs, all of it will be put into OpenNMS and they are paying us (gasp) to help project manage that team. That relationship did not happen overnight, but was built on a series of successful projects where we delivered particular value in exchange for money.

Look, I love, by and large, the open source community and I like being a part of it, but that doesn’t mean that open source and business are mutually exclusive. Learning to deal with open source as a business not only insures more open source gets created, but it also keeps it fun.

♫ The Lunatic is on My Web ♫

The TL;DR of it is that I needed to create a new forum called OpenNMS Connect. This will be a place for Luna. So far I’ve been happy.

When I first started my quest for forum software a couple of month ago, I did what most geeks do and did a search for it. I found a very helpful Wikipedia page (‘natch).

After dismissing the non-open source options, I started looking at the programming language. Now I know I really shouldn’t be a PHP snob (this blog is presented using PHP software) but having been burned in the past with security issues my first inclination is to avoid it.

Now the guys in the office are trying to get me to think all “agile-ly” and so I need a “user story”. For any forum we use it has to support LDAP, for which the story could be “User must be able to access forum using directory services” or better yet “Admin needs a central way of controlling forum access”. We implement LDAP via the FreeIPA project, and it will just be so much easier if we can add and remove people from a particular group and just have it work.

The first project I looked at was Discourse. I was especially interested in a hosted version if I could tie it into our IPA instance. Discourse is kind of the “new hotness” at the moment, but I didn’t see an easy way to implement LDAP. There is a Single Sign On (SSO) option but it would require writing our own authentication page, and it wouldn’t work if we hosted it with them anyway.

The next project that caught my eye was the eXo Platform. It’s written in Java (as is OpenNMS) and it seems to have a ton of features. Perhaps too many. In any case I put the team on it and asked them to get it working with LDAP.

They succeeded in getting LDAP authentication to work, but then hit a ton of other snags. The authenticated users couldn’t access the default /portal/intranet site no matter how often we tweaked the permissions. They could reach the /portal/meridian site but we couldn’t figure out how to change the default portal. And in all cases we couldn’t get the top menu bar to load with an LDAP user which meant you couldn’t log out, etc.

On Friday I decided to see what I could do about it. Friday was a long day.

eXo is one of those companies that produces an open source version of their software as well as a paid version. My three readers know how I feel about that business model, and it made it kind of frustrating to figure out things since I couldn’t tell if the documentation would actually work on the “community” version. Also, to access the forums you need to register, which gets you a couple of spam-y e-mails trying to sell you on their paid version. Not too obnoxious and I can understand why they do it, but it was a little annoying.

It can also be hard to administer. A lot of the configuration is buried in .war files. For example, in order to set the default portal above, you have to unpack portal.war, change it and repack it. In playing around with the system, I decided that while the LDAP authentication is nice, the platform itself is way overkill for what we need. It is huge and on our system took several minutes to start up and would often spike the load with limited users.

So I spent a lot of time looking for alternatives. Unfortunately, the only option I found that had easy to understand LDAP integration was phpBB. When I mentioned that to the team, Jeff threw up in his mouth a little and I wasn’t too happy about that choice either. I don’t have the same prejudices as some, but I felt that its style was a little dated and there have been some serious security issues in the past associated with it.

But for grins I installed phpBB anyway. It was rather easy to do, which made me happy, but then I noticed that it was not easy to make the forum itself private. Another user story is that “Admin requires that only authorized users see the forum”. You can make certain parts of phpBB private, but I kind of wanted the same thing as eXo – an initial log in screen you have to use before accessing the site.

Then it dawned on me that we could just put it in a directory by itself in the web root, say /forum, and then make a pretty splash page on on the site with a link to it. Apache LDAP authentication is something we already figured out and knew worked and I could just require a valid login to access /forum.

This caused another lightbulb to go off. If we are going to do it that way, then why not just put any forum we like behind an LDAP authenticated directory?

The downside would be that users would need to create a forum-specific user if they wanted to add content, but on the upside they could choose their own usernames, thus obfuscating their identities for people who work at sensitive organizations. Thus we could have an LDAP user tied to, say, obama@whitehouse.gov and their forum name could be something totally different, like “Hot Cocoa”.

Yes, I know it is dressing up a bug as a feature, but to me it did seem useful.

Then I thought, hey, let’s revisit Discourse. That turned out to be harder than it would seem

Well, the only way to install Discourse on CentOS is as a Docker container, and at the moment it doesn’t seem to work.

The first time I tried to install it, it died complaining about lack of access to an SMTP server. No where in the instructions did it say you had to modify the app.yml and put in a valid mail server. In any case, I did that and restarted the install.

At one point during the install process I get this:

-- 0:  unicorn (4.8.3) from
Bundle complete! 92 Gemfile dependencies, 189 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into ./vendor/bundle.

I, [2015-04-04T04:49:47.161747 #38]  INFO -- : > cd /var/www/discourse
&& su discourse -c 'bundle exec rake db:migrate'
2015-04-04 04:49:55 UTC [339-1] discourse@discourse ERROR:  relation "users" does not exist at character 323
2015-04-04 04:49:55 UTC [339-2] discourse@discourse STATEMENT:      SELECT a.attname, format_type(a.atttypid, a.atttypmod),	                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
	                FROM pg_attribute a LEFT JOIN pg_attrdef d
	                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
	               WHERE a.attrelid = '"users"'::regclass
	                 AND a.attnum > 0 AND NOT a.attisdropped
	               ORDER BY a.attnum

which a Google search says to ignore, but then a little while later the install fails with:

RuntimeError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #
Location of failure: /pups/lib/pups/exec_command.rb:105:in `spawn' exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}

on which Google is much less helpful. No matter what I did I couldn’t get past it.

This kind of brings up an issue I have with Docker. Now let’s get this out of the way: I am jealous of the Docker project. We’ve been around for 15 years and gotten little notice whereas they have become huge in a short time. It would be nice if, say, I could get up to four readers on my blog.

But I really, really, really hated how hidden this whole process was. You install software on your system and then load “magic bits” from the Internet and hope it works. I think this is great on a intranet when you need to deploy lots of the same things, but without developing it internally first it was a little scary. When it doesn’t work it is incredibly hard to diagnose. Because the app wouldn’t build I couldn’t play with the database or really do anything, so I just uninstalled and reinstalled numerous times to try to fix this.

Plus, by running in a container, we would then need to modify nginx to use our LDAP configuration and that seems to be much harder than with Apache. I didn’t think it would be easy to just forward requests to the Docker instance, but since I couldn’t get it to work I’ll never know.

By this time I said, screw it, reinstalled phpBB and went home. It’s now about 8pm and I’ve been at it 11 hours.

Well, I have a mild form of OCD, or maybe it’s just being a geek, but I couldn’t let it rest. So early this morning (as in soon after midnight) I discovered a project called Luna (an active project from the aforementioned Wikimedia page).

Luna is the next iteration of the ModernBB project which is in turn is a fork of FluxBB. It’s simple, does almost everything I could want, and was incredibly easy to install. No Docker containers, no large Java app, just some PHP that you drop in your web root. Plus the webUI is built on bootstrap just like OpenNMS.

In about an hour I had it running, had changed the style to match our color palette, and fixed an issue where jquery wasn’t getting loaded by copying it down as a local file.

OpenNMS Luna Website

The downside is that it isn’t production yet. I installed 0.7 and earlier this morning they released 0.8. Jesse fixed an issue with the internal mail system and I have a couple of more issues that I’d like to see fixed, but overall I’m very happy with it. They are aiming to release 1.0 on 13 April.

And I really like their attitude and philosophy. They are self-funded and I love Yannick’s tag line of “You Can Do Anything.”

To help that I sent them 100€. (grin)

Anyway, sorry for the long post. I’ll let you know how it goes.

2014 Open Source Monitoring Conference

This year I got to return to the Open Source Monitoring Conference hosted by Netways in Nürnberg, Germany.

Netways is one of the sponsors of the Icinga project, and for many years this conference was dedicated to Nagios. It is still pretty Nagios-centric, but now it is focused more on the forks of that project than the project itself. There were presentations on Naemon and Sensu as well as Icinga, and then there are the weirdos (non-check script oriented applications) such as Zabbix and OpenNMS.

I like this conference for a number of reasons. Mainly there really isn’t any other conference dedicated to monitoring, much less one focused on open source. This one brings together pretty much the whole gang. Plus, Netways has a lot of experience in hosting conferences, so it is a nice time: well organized, good food and lots of discussion.

My trip started off with an ominous text from American Airlines telling me that my flight from RDU to DFW was delayed. While flying through DFW is out of the way, it enables me to avoid Heathrow, which is worth the extra time and effort. On the way to the airport I was told my outbound flight was delayed to the point that I wouldn’t be able to make my connection, so I called the airline to ask about options.

With the acquisition by US Airways, I had the option to fly through CLT. That would cut off several hours of the trip and let me ride on an Airbus 330. American flies mainly Boeing equipment, so I was curious to see if the Airbus was any better.

As usual with flights to Europe, you leave late in the evening and arrive early in the morning. Ulf and I settled in for the flight and I was looking forward to meeting up with Ronny when we landed.

The trip was uneventful and we met up with Ronny and took the ICE train from the airport to Nürnberg. The conference is at the Holiday Inn hotel, and with nearly 300 of us there we kind of take over the place. I did think it was funny that on my first trip there the instructions on how to get to the hotel from the train station were not very direct. I found out the reason was that the most direct route takes you by the red light district and I guess they wanted us to avoid that, although I never felt unsafe wandering around the city.

We arrived mid-afternoon and checked in with Daniela to get our badges and other information. She is one of the people who work hard to make sure all attendees have a great time.

I managed to take a short nap and get settled in, and then we met up for dinner. The food at these events is really nice, and I’m always a fan of German beer.

I excused myself after the meal due in part to jet lag and in part due to the fact that I needed to finish my presentation, and I wanted to be ready for the first real day of the conference.

The conference was started by Bernd Erk, who is sort of the master of ceremonies.

He welcomed us and covered some housekeeping issues. The party that night was to be held at a place called Terminal 90, which is actually at the airport. Last time they tried to use buses, but it became pretty hard to organize, so this time they arranged for us to take public transportation via the U-Bahn. After the introduction we then broke into two tracks and I decided to stay to hear Kris Buytaert.

I’ve known Kris through his blog for years now, but this was the first time I got to see him in person. He is probably most famous in my circles for introducing the hashtag #monitoringsucks. Since I use OpenNMS I don’t really agree, but he does raise a number of issues that make monitoring difficult and some of the methods he uses to address them.

The rest of the day saw a number of good presentations. As this conference has a large number of Germans in attendance, a little less than half of the tracks are given in German, but there was also always an English language track at the same time.

One of my favorite talks from the first day was on MQTT, a protocol for monitoring the Internet of Things. It addresses how to deal with devices that might not always be on-line, and was demonstrated via software running on a Raspberry Pi. I especially liked the idea of a “last will and testament” which describes how the device should be treated if it goes offline. I’m certain we’ll be incorporating MQTT into OpenNMS in the future.

Ronny and I missed the subway trip to the restaurant because I discovered a bug in my presentation configuration and it took me a little while to correct it, but I managed to get it done and we just grabbed a taxi. Even though it was in the airport, it was a nice venue and we caught up with Kris and my friend Rihards Olups from Zabbix. I first met Rihards at this conference several years ago and he brought me a couple of presents from Lativa (he lives near Riga). I still have the magnet on my office door.

Ulf, however, wasn’t as pleased to meet them.

We had a lot of fun eating, drinking and talking. The food was good and the staff was attentive. Ulf was much happier with our waitress (so was Ronny):

Since I had to call it an early night because my presentation was the first one on Thursday, a lot of people didn’t. After the restaurant closed they moved to “Checkpoint Jenny” which was right across the street (and under my window) from the hotel. Some were up until 6am.

Needless to say, the crowds were a little lighter for my talk. I think it went well, but next year I might focus more on why you might want to move away from check scripts to something a little more scalable. I did a really cool demo (well, in my mind) about sending events into OpenNMS to monitor the status of scripts running on remote servers, but it probably was hard to understand from a Nagios point of view.

Both Rihards and Kris made it to my talk, and Rihards once again brought gifts. I got a lot of tasty Latvian candy (which is now in the office, my wife ordering me to get it out of the house so it won’t get eaten) as well as a bottle of Black Balsam, a liqueur local to the region.

Rihards spoke after lunch, and most people were mobile by then. I enjoyed his talk and was very impressed to learn that every version of the remote proxy ever written for Zabbix is still supported.

I had to head back to Frankfurt that evening so I could fly home on Friday (my father celebrated his 75th birthday and I didn’t want to miss it) but we did find time to get together for a beer before I left. It was cool to have people from so many different monitoring projects brought together through a love of open source.

Next year the conference is from 16-18 November. I plan to attend and I hope to spend more time in Germany that trip than I had available to me this one.

Test Driven Development

One of the things that bothers me a lot about the software industry is this idea that proprietary software is somehow safer and better written than open source software. Perhaps it is because a lot of people still view software as “magic” and since you can’t see the code, is must be more “magical”. Or perhaps is it because people assume that something you have to pay for must be better than something that is free.

I’ve worked for and with a number of proprietary software companies, so I’ve seen how the sausage is made, and in some cases you don’t want to know. Don’t get me wrong, I’ve seen well managed commercial software companies that produce solid code because in the long run solid code is better and costs less, but I’ve also seen the opposite done simply to get a product to market quickly.

With open source, at least if you expect contribution, you have to produce code that is readable. It also helps if it is well written since good programmers respect and like working with other good programmers. It’s out there for everyone to see, and that puts extra demands on its quality.

In the interest of making great code, many years ago we switched to the Spring framework which had the benefit that we could start writing software tests. This test driven development is one reason OpenNMS is able to stay so stable with lots of code changes and a small test team.

What’s funny is that we’ve talked to at least two other companies who started implementing test driven development but then dropped it because it was too hard. It wasn’t easy for us, either, but as of this writing we run 5496 tests every time something changes in the main OpenNMS application, and that doesn’t include all of the other branches and projects such as Newts. We use the Bamboo product from Atlassian to manage the tests so I want to take this opportunity to thank them for supporting us.

OpenNMS 14 contained some of the biggest code changes in the platform’s history but so far it has been one of the smoothest releases yet. While most of that was due to to the great team of developers we have, part of it was due to the transparency that the open source process encourages.

Commercial software could learn a thing or two from it.

Announcing OpenNMS 14 and Newts 1.0

It is with great pleasure that I can announce the release of OpenNMS 14. Yup, you heard right, OpenNMS *fourteen*.

It’s been more than 12 years since OpenNMS 1.0 so we’ve decided to pull a Java and drop the “1.” from the version numbers. Also, we are doing away with stable and development branches. The Master branch has been replaced with the develop branch, which will be much more stable than development releases have been in the past, and we’ll name the next major stable release 15, followed by 16, etc. Do expect bug fix point releases as the in past, but the plan is to release more major releases per year than just one.

A good overview of all the new features in 14 can be found here:


The development team has been working almost non-stop over the last two months to make OpenNMS 14 the best and most tested version yet. A lot of things has been added, such as new topology and geographic maps, and some big things have been made better, such as linkd. Plus, oodles of little bugs have finally been closed making the whole release seem more polished and easier to use.

Today we also released Newts 1.0, the first release in a new time series data storage library. Published under the Apache License, this technology is built on Cassandra and is aimed at meeting Big Data and Internet of Things needs by providing fast, hugely scalable and redundant data storage. You can find out more about this technology here:


While not yet integrated with OpenNMS, the 1.0 release is the first step in the process. Users will have the option to replace the JRobin/RRDtool storage strategies with Newts. Since Newts stores raw data, there will be a number of options for post-processing and graphing that data that I know a number of you will find useful. Whether your data needs are simple or complex, Newts represents a way to meet them.

Feel free to check out both projects. OpenNMS 14 should be in both the yum and apt repos, and as usual I welcome feedback as to what you think about it.

OpenNMS Newts at ApacheCon Europe

Being Hungarian, I am very jealous and yet still proud that our very own Eric Evans will be presenting at ApacheCon Europe in Budapest, Hungary.

He will be talking about Newts which is a new time series data store built on top of Apache Cassandra. It will be a key part of positioning OpenNMS for the Internet of Things as well as being very useful on its own.

Eric is a dynamic and interesting speaker, so if you are attending the conference be sure to check out his talk.

And while you are there, eat a Túró Rudi or three for me.

STUIv2: Focus on the Network

One of the things that really makes me angry is when critics of open source claim that open source doesn’t innovate, despite the fact that the very business model is incredibly innovative and probably the most disruptive thing to happen to the software industry since its inception.

Another example of innovation is in the new network visualization (mapping) software coming in the next release of OpenNMS.

I have been a vocal critic of maps for years. It stems from a time when I was working at a client during the first Internet bubble and my job was pretty much to spend several hours a day moving icons into container objects on the OpenView map. It was mind-numbingly dull work that returned little value. Most experienced network and systems managers move away from maps early on, but often the bosses who tend to make the buying decisions demand it as part of any solution.

Now, I’ve seen “cool” maps so it’s not that maps aren’t cool, it’s just that they tend to require more work to make cool than they save by being useful.

That is about to change with the new OpenNMS Semantic Topology User Interface (STUI).

Before I talk about that, I should mention that OpenNMS has a map. In fact it has a number of them. The first one was built for the Carabinieri in Italy who liked OpenNMS but wanted it to have something like OpenView’s map. Now called the “SVG” map, and it does its job well, as well as any map of that type can.

Then when we built the remote poller we needed a way to represent the pollers’ location geographically, and thus the “distributed” map was born. People liked the geographical representation, so we made it available to all nodes and not just remote pollers with the “geographical” map.

None of this work was really innovative, map-wise. But we started to depart from the norm with the topology map introduced in 1.12.

The topology map was novel in that it lets the user determine the topology to view. By default OpenNMS ships two different topology APIS. One is based on level 2 connections discovered by the “linkd” process, and the other is based on VMWare data showing the relationship between a host machine and its guest operating systems, as well as any network attached storage.

But it doesn’t have to stop there. In JunOS Space, Juniper is able to show connection data through all of its devices by using the API. Any other source of topology data and business intelligence can be added to the OpenNMS system.

However, me, the map hater, still wasn’t sold. While it is fine for smaller networks, what happens when you enter into the realm of tens of thousands of devices? We eventually see OpenNMS as being the platform for managing the Internet of Things, and any type of map we create will have to scale to huge numbers of devices.

Thus the team created the new topology map (STUIv2), available in 1.13 and coming in the next stable OpenNMS release. The key to this implementation is that you can add and remove “focus” from the map. This lets you quickly zoom in to the area of the map that is actually of interest, and then you can navigate around it quickly to both understand network outages as well as to see their impact.

While I like words, it’s probably better if you just check out the video that David created. It’s 20 minutes long and the first ten minutes cover “what has gone before” so if you are pressed for time, jump to the ten minute mark and follow it from there.

I like the fact that the video shows you the workflow from the main UI to the map, but then shows you how you can manage things from the map back to the main UI.

Note that I had nothing to do with this map. I often say that my only true talent is attracting amazing people to work with me, and this just drives that point home.

While I’m still not sold on maps, I am warming up to this one. I got goose bumps around minute 16:45 and then again at 17:30.

It’s great, innovative work and I’m excited to see what the community will do with this new tool.

Nagios News

My friend Alex in Norway sent me a link to a Slashdot story about the Nagios plugin site being taken over by Ethan Galstad’s company Nagios Enterprises. From what I’ve read about the incident, it definitely sounds like it could have been handled better, and it points out one of the main flaws with the “fauxpensource” business strategy.

I assume that at least two of my three readers are familiar with Nagios, but for the one who isn’t, Nagios is one of the most popular tools for monitoring servers, and it has been around just as long as OpenNMS (the NetSaint project, the original name of Nagios, was registered on Sourceforge in January of 2000 while OpenNMS joined in March of that year). Its popularity is mainly based around how easy it is to extend its functionality through the use of user written scripts, or “plugins”, plus it is written in C which made it much easier for it to be included in Linux distributions. A quick Google search on “nagios” just returned 1.7 million hits to 400K for “opennms”
A few years ago Ethan decided to adopt a business model where he would hold back some of the code from the “core” project and he would charge people a commercial licensing fee to access that code (Nagios XI). If your business plan is based on a commercial software model, then your motivations toward your open source community change. In fact, that community can change, with projects such as Icinga deciding to fork Nagios rather than to continue to work within the Nagios project framework.

Enter the Nagios Plugins site.

To say that Nagios Plugins was instrumental to the success of the Nagios application would be an understatement. Back when I tracked such things, there were way more contributions of both new plugins and updates to existing plugins on that project than were given to the Nagios code itself. The plugins community is why Nagios is so popular in the first place, and it seems like they deserve some recognition for that effort.

Trademark issues within open source projects are always tricky. Over a decade ago a company in California started producing “OpenNMS for Mac”. Even though we had OpenNMS on OS X available through the fink project, it required a version of Java that wasn’t generally available to Mac users (just those in the developer program). However, that version was was required to allow OpenNMS to actually work at scale, but this company decided to remove all of the code that depended on it and to release their own version. Unfortunately, they called it “OpenNMS” which could cause a lot of market confusion. Suppose a reviewer tested that program, found it didn’t scale, and decided to pan the whole application. It would have a negative impact on the OpenNMS brand. After numerous attempts to explain this to the man responsible for the fork, I had to hire a lawyer to send a cease and desist order to get him to stop. It was not a happy experience for me. When you give your software away for free, the brand is your intellectual property and you need to protect it.

So I can understand Ethan’s desire to control the Nagios name (which is actually a little ironic since the switch from NetSaint to Nagios was done for similar reasons). He has a commercial software company to run and this site might lead people to check out the open source alternatives available. Since they are based on his product, the learning curve is not very steep and thus the cost to switch is low, and that could have a dramatic impact on his business plan and revenues.

At OpenNMS we treat our community differently. We license the OpenNMS trademark to the OpenNMS Foundation, an independent organization based in Europe that is both responsible for the annual Users Conference (coming to the UK in April) as well as creating the “Ask OpenNMS” site to provide a forum for the community to provide support to each other. They own their own domains and their own servers, and outside of an small initial contribution from the OpenNMS Group, they are self funded. Last year’s conference was awesome – the weather notwithstanding (it was cold and it snowed).

OpenNMS is different from Nagios in other key ways. At its heart, Nagios is a script management tool. The user plugins are great, but they don’t really scale. Almost all of the OpenNMS “checks” are integrated into the OpenNMS code and controlled via configuration files which gives users the flexibility of a plugin but much greater performance. For those functions that can’t be handled within OpenNMS, we teach in our training classes how to use the Net-SNMP “extend” function to provide secure, remote program execution that can scale. OpenNMS is a management application platform that allows enterprises and carriers to develop their own, highly custom, management solution, but at the cost of a higher learning curve than products such as Nagios.

Now that doesn’t make OpenNMS “better” than Nagios – it just makes it better for certain users and not for others (usually based on size). The best management solution is the one that works for you, and luckily for Nagios users there are a plethora of similar products to choose from which use the same plugins – which I believe is at the heart of this whole kerfuffle.

The part of the story that bothers me the most is the line “large parts of our web site were copied“. If this is true, that is unfortunate, and could result in a copyright claim from the plugins site.

To me, open source is a meritocracy – the person who does the work gets the recognition. It seems like the Nagios Plugins community has done a lot of work and now some of that recognition is being taken from them. That is the main injustice here.

It looks like they have it in hand with the new “Monitoring Plugins” site. Be sure to update your bookmarks and mailing list subscriptions, and lend your support to the projects that support you the most.

Goodbye Cyanogenmod, I'll miss you

It is with some disappointment that I read of Cyanogenmod’s descent into fauxpensource. Not only does it appear that they are doing everything they can to ruin any credibility with their community, it also means that I need to find a new operating system for my android devices.

For those who don’t know, Cyanogenmod is was a very popular implementation of the Android Open Source Project (AOSP). Basically, it is a recompiled version of the software Google and others distribute with their phones but the aim of AOSP is to be as open source as possible (i.e. without a lot of proprietary add-ons). If you were to buy, say, a Google Nexus 4 and a Samsung S4, both android phones, you would find that the user interface on both is radically different.

The reason is that it is rare for a company to want to sell commodity products. If the software on android devices were the same across them all, price becomes the main differentiator. If you are a device maker aiming to get the same margins that Apple is able to demand for its products, then you want to add something unique that isn’t available elsewhere, and it is hard to do that under the open source model. Also, the traditional way to offset costs is through deals to bundle other products into your offering. Does anyone here remember buying a retail computer with Windows installed on it? Usually the desktop would arrive full of pre-installed software, or “crapware”, that the vendors paid to have ship with the product. This happened when I bought my Galaxy S3. I tried to remove all of the kruft, such as the Yellowpages app, only to have the operating system tell me that it was a “critical” system app and couldn’t be removed.

So, within two hours of getting my phone I had root access and installed Cyanogenmod.

Now, I have struggled for over a decade to balance the desire to create free and open source software with the need to make money. I can understand the pressures that the Cyanogenmod team must have felt watching their buddies at commercial software companies making large salaries with a decent amount of job security while they toiled along with no real business model. I, too, have heard the siren song of Venture Capitalists who believe that all you need to make a lot of money is to offer some sort of “enterprise” or commercial version of your open source project.

Most of them are wrong.

I was in a meeting with a VC a few weeks ago when this came up. Now you have to realize that there has only been one “Valley-grade” success story with open source (well, that still exists as a private company), and that is Red Hat. However, most in the Valley don’t view it as a success, and I think that is mainly because it wasn’t a Valley deal. The first thing the VCs will say is that Red Hat is too small – it’s not a “real” success – when the fact is that they have a market capitalization similar to Juniper Networks (about US$10 billion). The second thing is that they’ll point out that Red Hat has “an enterprise version”. This is also not true. Red Hat sells time, just like we at OpenNMS do, through support and ease of use. If I want to, I can buy that access, take the product, remove all of the trademarked information and create an open source, feature for feature copy. This is exactly what CentOS does and why I call the measure of whether or not a company’s products are truly open source the “CentOS Test“. The main reason that the Valley has been unable to duplicate Red Hat’s success is that they always undermine it with some sort of commercial software component that removes the reason people would use it in the first place.

Take Eucalyptus for example. They tout themselves as an “open source” cloud solution, but the barriers they erected with their commercial offerings caused the creation of OpenStack – a truly open source solution that in just a few years has easily eclipsed their product. In that same VC meeting the guy asked “yeah, you’re open source, but what is the ‘secret sauce’?”. Well, the “secret sauce” is the fact that OpenNMS is open source. If I were to screw with that we’d stop being a market leader and just become one of many hundreds of commercial offerings, despite any features that make us better than them.

“But,” the open core people will exclaim, “we need to make money.”

One way to make money is to dual-license an open source project. In order to do that, one must own 100% of the copyright. This brings us to the contentious topic of copyright assignment, and Cyanogenmod seems embroiled in this issue at the moment.

I think it was MySQL that pioneered this idea. Their argument was “Sure, you can contribute to the project, but we need you to assign the copyright to the code you wrote to us. Thus, we can offer it under a license like the GPL, but if you want to pay us you can use it under another license.”

In theory this is a great idea, but there are two flaws. The first is that, as a programmer, if I were to create some code and then give away my copyright, then I no longer own what I wrote. Imagine that you wrote some code for MySQL, and, I don’t know, the company gets acquired by, say, Oracle, and you decide you’d like to work on that code for MariaDB. You can’t. You gave it away. You no longer own it.

The second flaw is that when a company makes a commercial offering, the pressure is on to add more stuff to it and leave it out of the “free” version. MySQL started down this path with offering new versions to commercial customers six months or so before releasing them under an open source license, then six months became a year, and then became never. This is exactly how Cyanogenmod hopes to pay back that $7 million investment by requiring device manufacturers to pay for features that they plan to keep out of the open source version.

OpenNMS, I think, has avoided these two traps. First, we do require copyright assignment. One main reason is that we need to be able to defend OpenNMS from people who would try to steal it. This happened a few years ago when a company was using our code in violation of the GPL. When we started legal action to make them stop, their defense was that “if” they were stealing the code, they were stealing from OpenNMS 1.0 (which at the time we didn’t own the copyright) and thus we couldn’t defend it. Myself and David Hustace mortgaged our houses to acquire that copyright and were able to bring the existing OpenNMS code under one copyright holder.

The next problem to solve was future contributions. Instead of unilaterally declaring that we get sole copyright to all contributions, we actually bothered to ask our community for suggestions. DJ Gregor pointed out the Sun Contributors Agreement (now the Oracle Contributors Agreement) which introduced “dual copyright” to the software industry. In much the same way two authors can share copyright on a book, it is possible for a code author to contribute the copyright to their code to a project while retaining the rights as well. We adopted this for OpenNMS and everyone seems to be pretty happy with it.

Now the second issue, that of a dual license, is harder to address. In the case of OpenNMS it comes down to trust. Trust is very important in the open source world. When I install a pre-compiled binary I am trusting that the person who compiled it didn’t do anything evil. Mark Shuttleworth came under fire for implying that Canonical “had root” in response to some questions about Ubuntu and privacy. While the statement was a little harsh in light of the valid concerns of the community, it was also true. We, as Ubuntu users, trust Canonical not to put in any sort of backdoor into their binaries. The difference between that and commercial software, however, is that it can be verified and I have the option of compiling the code myself.

At OpenNMS we promised the community that 100% of the OpenNMS application would always be available under an open source license, and we have kept that promise. In fact, when Juniper (one of our “Powered by OpenNMS” customers) licensed the code, all the additional work they contract from us ended up in OpenNMS as well (you can actually see the code we are working on in feature branches in our git repository). This is a great way to make money and advance the project as it can be used to pay for some of the development.

This is not a plan that Cyanogenmod plans to follow, if the experience of Guillaume Lesniak is any indication.

The only reason I was interested in Cyanogenmod was the fact that it was open source. Now, the beauty of it is that open source almost always offers options. Bradley Kuhn, a person I consider a friend and whose blog post pushed my button to write this in the first place, offers up Replicant as an alternative. I hadn’t looked at that project in awhile and it seems to be coming along nicely, with a lot of newly supported devices. Unfortunately my AT&T S3 isn’t one of them (they only support the international version), so I’m looking to switch to AOKP as soon as I can find the time.

It will be interesting to revisit Cyanogenmod in a year. My guess is that anyone not employed by Cyanogenmod, Inc. will flee to other projects, and Cyanogenmod, instead of being the go-to AOSP alternative, will fade into just another commercial offering. It is doubtful that Samsung will license it, since they pride themselves on in-house expertise, and Google is, well, Google. With the exception of HTC, no one else has any market share.

But, what do I know, right?