Why There Will Never Be Another Red Hat

October 13th, 2014

My friend Nick sent me a link to a post called “Why There Will Never Be Another Red Hat: The Economics Of Open Source“. It immediately pushed a bunch of my buttons before I read the first word of the article.

First, it was from TechCrunch. I have nothing against TechCrunch and I respect a lot of their work, but they are Silicon Valley-centric, if not the main mouthpiece, and thus I have to take that bias into account when reading their articles. What works in the Valley doesn’t necessarily translate to the world as a whole, which is why a lot of Valley companies seem to quickly plateau after an initial success.

Second, continuing the theme of Valley biases, I strongly believe Red Hat doesn’t get the respect it deserves because it is headquartered down the road from us in North Carolina and not California. There is a strong sense of “you can’t make it if you aren’t here” in the Valley and that extends to a somewhat dismissive view of Red Hat. Plus, I have a big ol’ man-crush on Jim Whitehurst as he is the most successful tech CEO I know that really, really “gets” open source and thus anyone trying to tell me about the “economics of open source” without respecting Red Hat starts off on my bad side.

Finally, the author is currently at the VC firm Andreessen Horowitz, or A16Z as those of us in the know refer to it. In last year’s investment tour I met with a large number of people in the Valley, and the guy I met from A16Z was easily the worst of the bunch. He made the caricatures of the Silicon Valley TV show look mild. He had no interest in us since we weren’t in California and he was more concerned with who we knew than what we did. I left that popularity crap behind in high school. Granted, we only rated an audience with the lower levels of the company, which overall does have a pretty solid reputation on Sand Hill Road, but still it was almost insulting.

Let’s just say my bullshit meter was halfway to pegged before I started the first sentence.

However, I found most of the article to be spot on. In my “Is Open Source Dead?” post I talked about how open source is both greatly increasing while the classic ideals of open source (i.e. free software for everyone) seem to be going away.

My own philosophy is that, at least for certain large and complex (i.e. expensive) software, the proprietary software model is doomed. Customer needs are changing so fast that no closed system can really keep up, and we’ve seen that in the biggest OpenNMS customers. I spent the last week in Ireland and the client was complaining that before they started using OpenNMS, whenever they needed some new functionality in their management solution the proprietary vendor took too long, cost too much and delivered too little to be worth it. Using a free platform like OpenNMS made it much easier to adapt the tool to their business workflow, instead of having to change it to meet the workflow of the management software. There is value in that – value that can be monetized.

Peter Levine started to win me over with “Red Hat is a fantastic company” (grin) and I as I read on I found my head nodding in agreement. He states

Unless a company employs a majority of the inventors of a particular open source project, there is a high likelihood that the project never gains traction or another company decides to create a fork of the technology.

While I estimate OpenNMS has around 40 to 50 active contributors, at least 15 of those are on my payroll, either directly or indirectly as contractors. While I definitely would like to increase the overall number, we are growing fast enough that we can usually hire someone who contributes a lot to the project, and then, since they can spend their full time on it, we as a group continue to contribute greater and greater amounts of the overall code. When I started out on my own back in 2002, I think at least half of the code came from outside of the .com side of things. Now it is probably closer to 5% or less. It has managed to let us focus on our direction for the application.

Then Levine continues:

To make matters worse, the more successful an open source project, the more large companies want to co-opt the code base. I experienced this first-hand as CEO at XenSource, where every major software and hardware company leveraged our code base with nearly zero revenue coming back to us. We had made the product so easy to use and so important, that we had out-engineered ourselves. Great for the open source community, not so great for us.

I’ve heard this tale from a number of people. Become successful and someone like IBM could dump 100 developers on your project and take it over. While we haven’t experienced it directly (rarely do people tell me OpenNMS is “easy to use”), we are constantly finding out about companies who have either based products on OpenNMS or used OpenNMS to provide services for profit. I think this is great, but it would be nice to capture some of that effort back into the project, either in the form of contributions or cash. It is one reason that the next major release of OpenNMS will be published under the Affero GPL.

So, with all this doom and gloom, what is the solution? Levine’s answer is “sell open source as a service”. I couldn’t agree more. This is exactly what we pitched to A16N. It’s something of a “win/win”.

This recipe – combining open source with a service or appliance model – is producing staggering results across the software landscape. Cloud and SaaS adoption is accelerating at an order of magnitude faster than on-premise deployments, and open source has been the enabler of this transformation.

If you have the in house development staff to leverage open source, it makes sense to become active in the community. David just finished a five stop roadshow in both the US and Europe describing the OpenNMS roadmap over the next year as we position the product for the Internet of Things. He met with our largest customers and all of them are eager to get involved, many pledging, OpenDaylight style, to provide the project with developers. They get input into the direction of the product and we get great open source code.

But what if you aren’t a large medical information company or a worldwide financial institution? You may need what OpenNMS can provide but don’t have the time to build in the workflow or customize it. We will have a solution for you.

The only issue I ended up having with the article was when he compared Red Hat to Microsoft, Oracle, Amazon, etc. Sure there might not be another Red Hat, but I don’t expect to see another Microsoft (operating systems and office suites), Oracle (enterprise software), or Amazon (online product distribution) either. This doesn’t mean that there won’t be new mega-companies. In ten years I expect there will be some huge companies that no one today knows about, probably in the areas of 3D printing and biotech (specifically integrating tech into the human body). Enterprise software companies will be represented by a number of large-ish but nimble companies leveraging open source, and I wouldn’t count out Red Hat just yet as they too are pivoting to follow this new business model.

I want to close with one little story. I spent last week in Ireland where I just happened to be in Dublin where the band Wheatus was touring with my friend Damian (MC Frontalot) opening. They had a grueling schedule (shows almost every night) but I managed to see their concert and talk with them afterward.

Damian has just released a new album (it’s excellent, buy it) and he closed his set with “Charity Case” which includes the lyric:

It’s true:
Frontalot’s destitute.
I need you
to buy my CD so I can buy food.

He often pokes fun on the changing nature of the music business (his song “Captains of Industry” suggests that he is not a musician but instead is in the T-shirt business), and we talked about various business models. I pointed out that acts like Elton John are living the high life from work that basically peaked in the 1970s, and that sort of “royalties for life” model is gone. In its place are artists who sell directly to their fans, often including personalized premiums for a higher price, and touring. The band Phish tours extensively and they make millions, all the while encouraging their fans to bootleg their music, something old school musicians wouldn’t think to do.

At this point Brendan, the founder and front man of Wheatus, joined us and stated “there is nothing an old musician can teach a young one about the music business”. That quote really resonated with me.

In summary, I really enjoyed this article. It mirrored a lot of our thoughts over the last year as we seek to make OpenNMS successful. Remember, our plan is for OpenNMS to be *the* de facto management solution of choice and for that to happen will take a lot of work as well as money. But one thing that we will continue to do to emulate Red Hat is to keep publishing as much software as possible under an OSI-approved open source license.

That is still a key to our success: OpenNMS will always be free and OpenNMS will never suck.

Using an XML Parser

October 9th, 2014

You know when the XML nerds say not to use regular expressions to parse XML? They’re right.

As part of a less is more project, we wanted to remove the tags from all of the OpenNMS event files. We spent much of the morning playing with a number of methods to find and replace with empty space those tags, and we failed. We came close a couple of times, but then some weird aspect of formatting (tags that spanned multiple lines, some with spaces and some without, etc.) would foil it.

Then I found out about xmlstarlet. We installed it and ran:

xml ed -L -d "/events/event/alarm-data" [filename]

and it just worked. Pipe that bad boy through find and you are good to go.

While I don’t think the option exists, it would be cool if instead of deleting the tag we could just comment it out, but that doesn’t seem to be currently possible.

When Less is More

October 7th, 2014

One of the things I’ve noticed in my years of deploying network management solutions is that people can get real excited when they go from having no visibility into their network to being able to see in great detail what’s happening, as when they deploy OpenNMS. The problem then changes from having no information to having too much.

Network geeks like myself tend to be loathe to turn off certain alerts, but sometimes that can be the best thing for an organization.

When OpenNMS was started, workflow was based on events. Events appeared in the browser, events triggered notices, you could acknowledge events – pretty much everything was events. But events can be noisy, especially if you leverage the SNMP trap capability of many devices. This is why we implemented the alarms subsystem. Alarms can take many events and reduce them into a single alarm. Alarm processing can be automated to insure that issues that are important are escalated, and issues that have been cleared can be removed. The alarms list is supposed to be a “to do” list for the NOC staff.

In order to make that happen, it is a good idea to consider each alarm in your system and insure that it is “actionable”. Each alarm comes with two fields for tracking the resolution progress, and these can be used to document the actions taken to fix the alarm.

The “Sticky” memo field is used to annotate a particular instance of an alarm. For example, suppose there was a “link down” alarm due to a circuit being cut by a backhoe. The NOC engineer would be able to note that the repair was in process and maybe even include a case number. Once the issue is resolved the sticky memo goes away.

The “Journal” memo field is permanently associated with the alarm. This is for notes that could be useful the next time the alarm happens, such as “Contact Jim – he knows how to fix this”, etc.

Alarms can be acknowledged, which will remove them from the list of current issues. It is pretty easy to create an automation that can unacknowledge an alarm if it hasn’t been cleared in a particular amount of time. Thus you can automate “reminders” that the issue is still outstanding.

This doesn’t discount the value of events. In OpenNMS, events have become more like log messages. When an alarm happens on a particular node, that node’s page will reflect the events associated with it, which may shed some light onto the problem. But having too many events appear as alarms can overwhelm the NOC staff to the point that they stop using the system.

Unfortunately, often the best way of dealing with network issues involves trial and error. By limiting alarms it is possible to miss something important. But once that happens, alarms can be created to insure it doesn’t happen again. But the opposite, dumping too much information into alarms, will guarantee that alarms will be ignored, greatly increasing the chance that something important will be missed.

I developed my alarm philosophy during my first network management deployment in the early 1990s. I was consulting for a cellular provider and installing HP OpenView Network Node Manager (version 2.2 I believe) and they had me working in the server room. Besides being a bit cold, in the corner was a large UPS that was constantly beeping.

Beep … Beep … Beep

I asked Avery, the guy I was working with, what was wrong and he replied “Oh, it always does that”. At that very moment I decided that if there is an alarm and you don’t do anything to resolve it, just turn it off.

Just remember that OpenNMS is a platform and thus you get to make a lot of the decisions on how best to get it to work with your organization. Consider that when deciding which events to turn into alarms, and then focus on using automations to insure that the most important issues are treated as such.

OpenNMS-based App Wins Digital Jersey Hackathon

October 3rd, 2014

I was delighted to find out that an Android app using OpenNMS as the backend won the “Best App” prize at the first ever Jersey Hackathon.

Note: This is Jersey as in the island and not Jersey as in New.

The Open Alert “Man on Site” app is a small Android application that is designed to track the activities of people working alone at a remote site. From the wiki:

When activated this reports the location of the phone on a regular basis back to a central OpenNMS server. OpenNMS is configured to plot the current location and status of the device on a geographical map (Open Streetmap).

The App has four buttons;

Start Job – This is pressed by the worker when they start lone working on site. This starts a timer in the local App and on OpenNMS. The local timer will generate an alarm on the local device if the user forgets to report in after a set time.

Report In – This must be pressed when prompted by the local timer. If it is pressed both the timer in OpenNMS and the local device will be reset. If it isn’t pressed then OpenNMS will escalate the ‘Man on site’ event to the next level of severity and notify the OpenNMS operator that there is a problem. (Obviously the local timer should be set to 5-10 minutes less than the OpenNMS time out.) OpenNMS will keep escalating the alarm until it is signalled as critical. If the alarm is escalated, then there should be manual processes in place to contact the worker by other means or send someone else to site to make sure they are OK.

Finish Job – This should be pressed when the worker leaves site. The man on site alarm is cleared in OpenNMS and no further escalation takes place.

Panic – If the panic button is pressed, an immediate critical alarm is created in OpenNMS indicating that the worker on site is in trouble and needs help.

OpenNMS maintains a log of all of the movements of the user and also of the time of starting work / stopping work / panic events which could be important for triage if an incident happens.

Congratulations to the authors, Craig Gallen and Mark Wharton, who created this during the 48 hours of the Hackathon. We built OpenNMS to be a platform and not just an application and this is one example of what can be created leveraging it.

More information can be found on the UK OpenNMS Site and the code is available on Github.

OpenNMS in Dublin

October 2nd, 2014

I’ve been to 34 countries so far, and my goal is to hit 50 by the time I’m 50 (which is closer than I’d like). In all that time I’ve managed to miss Ireland, but that is about to change.

Airspeed Telecom is hosting a workshop next Wednesday, October 8th, at the Morrison Hotel in Dublin, described as “probably the hippest & coolest luxury hotel in Dublin city centre”.

That’s just how we roll.

The workshop will feature a case study by Airspeed, as well as a futures roadmap presentation by Dr. Craig Gallen and David Hustace.

Oh, and I’ll be there, too.

If you can make it, please email Liz at lhand@airspeed.ie.