Be Well, Honduras

This morning a magnitude 7.1 earthquake struck off the coast of Honduras.

Just recently we at OpenNMS welcomed a new client from Honduras, so Jeff sent them a note to make sure they were okay. This is the response we got back:

Yes as a matter of fact we felt the earthquake at 2am, but thankfully we are all fine and dry. Here in Tegucigalpa the quake was not so strong and there was very little damage if any.

But on the other hand our network suffered some damage and many of our sites which are closer to the epicenter are running on generators since 2am and are now very low on fuel, and other sites lost their microwave alignments. So we have a bit of a mess right now.

I’m glad everyone is okay and my thoughts are with you. Let’s hope that a minimal loss of mobile phone service is the worst that happens.

Interop 2009

In my commercial software days I used to go to the Interop show in Las Vegas, back when it was held at the main convention center. It was a huge show and pretty much the premiere event for networking gear. I think the last time I went was 2000.

I had the opportunity to return this year. The show has changed, it is now in the Mandalay Bay Convention Center and it is smaller than I remember. The NOC staff, however, is still pretty much the same.

As you can imagine, running a NOC at a show like this is no minor undertaking, but believe it or not the entire NOC is staffed by volunteers. Getting through an ordeal like an Interop show seems to bring people together, as many volunteers have been coming for years (I met one guy who had been coming here since 1996). The only downside was that this Interop marked the first since the passing of Jim “Haggis” Brown, a longtime NOC member. They had a place set out for him, along with a bottle of scotch.

Speaking of bringing people together, this trip has been pretty serendipitous. For example, my plane from RDU to DFW had mechanical problems, so they routed me through Miami. As I was leaving the Admirals Club to walk to my gate, I ended up sharing an elevator with Chris McGugan. Chris is something of a superstar in networking circles. He was at Cisco for many years (based out of North Carolina), and now he is working at Avaya out in California. We used to share a townhouse about 20 years ago, and it had been about that long since I’d seen him. The odds of us running into each other the way we did were pretty long.

Even stranger, Chris used to work in the NOC at Interop, and he knew many of the people I had come to meet.

Another example of serendipity: on our first day at the show, Jeff and I were at a table utilizing some wireless bandwidth when John Willis walked by. He didn’t know we were going to be there, so it was nice to see he had decided to wear his OpenNMS shirt anyway.

Jeff Gehlbach, High Mobley and John Willis

Things have changed a bit in Las Vegas since I was last here. There is no smoking near food (which pretty much leaves the casinos) and coins no longer work in the slot machines. Payouts are given on little slips of paper, and the machines will only accept bills or those little slips. I really miss the sound of the coins clanking around, and it makes the casinos seem quieter.

According to the cab driver, 40% of the usual conventions have cancelled this year, so the area is surviving on tourism. We stayed at the Luxor for $69 a night, and although it was a tower room, it was a deal.

The Luxor is my favorite hotel on the strip. It is not the nicest or the most luxurious, but think about it – it had to have been built by a geek. If I was given a boatload of money and told to build something impressive in the desert, it would be a pyramid. Plus at night its blackness contrasts well with the brightness of the other hotels, even with the sides having been given over to advertising.

However, one of the Luxor’s main acts is Carrot Top, and the dude is just scary looking. His face is everywhere you go in the hotel, even on the keys and the “do not disturb” signs, and it gets creepy after awhile.

Back to Interop: the show had most of the people you would expect. We stopped by the HP booth to look at the latest OpenView. HP must be doing well, because they had some seriously thick padding under the booth carpet, which was awesome (if you have ever worked a show on a concrete floor for a couple of days, you know what I am talking about). I decided to talk a little smack to their folks in the booth. I thanked them for raising their prices so drastically since it helped us out, which caused them to asked about OpenNMS. When I told them it was an open source network management platform, the reply was “yes, but OpenView is for the enterprise.”.

I took that as my cue to bring up that we have customers monitoring over 55,000 devices with OpenNMS (them: “with a single instance?”, me: “yup”) and that we were replacing OpenView at a client in Italy because their devices, which have more than 32,000 interfaces each, break OpenView but work with us. Things got quiet and a little awkward after that, so we left (but the lady kept my card).

Microsoft was a no-show (or at least I didn’t see their booth), but I did get introduced to a company called Xirrus. Xirrus builds wireless arrays that have a high level of built in switching, and their marketing pitch was a face-off between their wireless “switches” and wired ones. They had a boxing ring in the middle of the booth and several times a day held actual bouts. When it wasn’t being used by humans, one corner held your traditional network switch (with lots cables of course), and the other corner held a Xirrus array.

The arrays looked like big roombas with RJ-45 connections, and they had really cool lights (Jeff took a video).

All in all it was a fun time, mainly because we got hang “backstage” with people who really seemed to both love networking as well as knowing a lot about it. What did surprise me were the number of people that were using OpenNMS. When we’d get introduced we were often met with “Oh, we use OpenNMS. It’s great.”

It’s nice to hear. While we have things like the Order of the Blue Polo and the Wall of Cards, we rarely hear from people who use the tool outside of our clients. And while we love our clients, usually when we hear from them it is to ask a question or report a problem. We work hard to make OpenNMS great while remaining 100% open source so it definitely motivates us to meet people who find it useful.

It was a little sad when the show ended and the equipment started coming down. Perhaps we can return next year.

OpenNMS on MSN (sort of)

One of our oldest clients is Papa Johns International, better known as Papa Johns Pizza. They have sold over a billion dollars in pizza over the Internet, and they use OpenNMS to monitor that product line.

Chris Rodman sent me a link today to an MSN site called “Business on Main” that featured John Schnatter (the “John” in “Papa Johns”) mentoring a yogurt vendor from New York City. It’s an interesting clip, with quotes like “technology is a big part of our success” and his conscious decision to “make the internet a big part of our business”.

The fun part comes about 3:38 minutes in (the web site will show 4:15 remaining) when they move into the PJI NOC. There you can see OpenNMS front and center as their management solution.

We constantly get asked, sometimes in a condescending manner, about who would trust their important revenue streams to an open source solution. Papa Johns is a shining example of a technology-savvy enterprise that has become successful by not only embracing technology but by focusing on technology that works. Over the years OpenNMS has been molded to fit their business model (not the other way around), and we are happy to play at least a small part in their success.

UPDATE: There’s a shot of the OpenNMS maps as well:

OpenNMS 1.6.5 Released

I am happy to announce the availability of OpenNMS 1.6.5, the next stable release of OpenNMS.

New Features and Enhancements

  • Capsd will now use ifHighSpeed instead of ifSpeed when available (Bug #1822)
  • The SNMP configuration shipped with OpenNMS now uses SNMP v2c, fewer retries, and a longer timeout by default, for more efficient scanning. (Bug #3050)
  • Data collection and graphing has been added or updated for Alcatel-Lucent, Allot, Alteon, Aruba, Ascend, Avocent, Bluecat, BlueCoat, Cisco Airespace, Lucent, NetApp, Overture, Packeteer, and Powerware devices (Bugs #3096, #3099, #3109, and #3138)
  • Support has been added for using raw instance identifiers in thresholds when no data source label can be inferred (Bug #3108)
  • Support was added for a number of new Cisco devices in Linkd (Bug #3110)
  • Support has been added for extracting the bgpPeerRemoteAddr instance identifier from the varbinds of the bgpEstablished and bgpBackwardTransition traps defined in the BGP4-MIB. This allows us to make meaningful alarm annotations for these events so that the resulting alarms can self-clear. (Bug #3156)
  • Some tools have been added to OpenNMS “contrib” for creating SNMP walk dumps suitable for debugging with OpenNMS’s mock SNMP server. (Bug #3173)
  • The default “Database” category now includes MSSQL and Oracle in addition to MySQL and PostgreSQL. (Bug #3175)

Bug Fixes

  • The Capsd HTTP plugin now properly honors response-text parameters. (Bug #2774)
  • The HTTP collector now properly honors the “retry” parameter. (Bug #3133)
  • The Hyperic HQ service monitor definition now detects Hyperic HQ 4.x properly. (Bug #3106)
  • Multiple nodes with the same name/label can now be distinguished from each other in category views. (Bug #3112)
  • Vacuumd cleans out old (invalid) nodes from topology map tables. (Bug #3121)
  • The JMX collector now works when store-by-group persisting is enabled. (Bug #3160)
  • Configuration file filtering has been fixed so that tags aren’t properly expanded in (Bug #3174)
  • The remote poller build now ensures it is compiled with the correct version of Groovy. (Bug #3176)

The full list of changes is available in bugzilla.


I received an interesting call last week from Gary Read, the CEO of Nimsoft. He informed me that his company was buying the intellectual property assets of Cittio, a company with a rather storied history with respect to OpenNMS. With this acquisition, parts of the Cittio Watchtower product involving Layer 2 discovery, topology and root cause will be integrated with Nimsoft solutions and the Watchtower product will cease to be produced or supported.

Now, I’m a firm believer in karma. Not the formal concept as defined by many Indian religions (and I mean no disrespect by using the term casually) but the idea that if you do good things, good things will happen to you, and if you do bad things, well, bad things will happen. In my mind the failure of Cittio is karma in action.

If you search for “cittio” on my blog, you’ll see that since 2005 we have been rather curious as to how OpenNMS and Watchtower were related. The early releases of Watchtower seemed, feature for feature, to be remarkably similar to OpenNMS. I had a couple of conversations with Jamie Lerner, the CEO and founder of Cittio, where he assured me that any use of OpenNMS code was within the license requirements of the GPL. Information surfaced in 2008 that seemed to indicate that claim was inaccurate, so we hired the law firm of Moglen Ravicher to help us get to the bottom of this.

When our lawyers contacted Mr. Lerner, he reiterated his claim that Watchtower was not in violation of the GPL, and even if it was, any OpenNMS code being used by his application would be based on OpenNMS version 1.0.2. Since that code was copyright the Oculan corporation and not The OpenNMS Group, we had no right to enforce the copyright.

While we had received information that OpenNMS code post-version 1.0.2 was being used in Watchtower, we had no proof, and so without a long and expensive discovery process we were effectively stymied from getting to the bottom of this matter.

Thus it was with some surprise that we watched Cittio release a fork of OpenNMS in November of 2008 called Rooftop. If, as Mr. Lerner claimed, Cittio was always in compliance with the GPL, the fork should have been unnecessary, and I found it amusing that the press release went out of its way to specify it was a fork of OpenNMS 1.0.2.

As I write this, it appears that Rooftop has been downloaded around 60 times in the last 6 months or so. Compared to the 5000 downloads a month of OpenNMS from our repositories I would say that the fork hasn’t hurt our project at all. And for those who wish to test the claim that no OpenNMS code post version 1.0.2 was being used, I invite you to download the 1.0.2 source as well as the source of, say, 1.2.0, and draw your own conclusions. A search for the word “interfaceresolve” might provide a good starting point.

With the end of Cittio as a management company, I guess a lot of these points have become moot. Whatever the actual history of how Mr. Lerner came to use the OpenNMS code in Watchtower, the future should be much clearer. Mr. Read has stressed that he wants to make sure the OpenNMS project is satisfied that none of the OpenNMS code will be a part Nimsoft’s offerings, and he has offered, at his expense, to fly one of us out to inspect the code.

While I can’t (currently) see the Watchtower code, in looking through Rooftop there is no mention of linkd (our Layer-2 topology discovery process), alarmd (our event correlation engine) or our “drools-based” event correlator, so I assume that Nimsoft has done enough due diligence to be sure of what they are buying. While their emphasis on transparency is obviously driven by a wish to avoid any taint of license violations in Nimsoft products, it is refreshing that they took this seriously enough to contact me well before the press release and to make the blanket offer of showing us the code.

One might think this is strange behavior considering the press Nimsoft was getting in open source circles after the announcement that Hyperic was being acquired by SpringSource. Part of it was stupidity. A person at Nimsoft’s PR company apparently did a search on articles referencing the sale and sent a generic anti-open source article to Matt Asay. While Matt and I have had our differences, he was right to skewer the guy. I don’t mind the marketing effort, heck I spread my own FUD about commercial software with Cittio being a prime example (as in “what do I do now that this license I bought is to code no longer supported”), but just as with a lot of the near spam e-mails I get from potential “partners,” the PR guy should have done his homework before sending it to Matt. I do object to laziness.

Gary Read got his own bunch of replies on his blog post about the acquisition. Some people saw it as an attack on open source, but if you actually read his post it was an attack on the hybrid (what I call “fauxpen source“) model. When he called me, we had a nice long conversation and were pretty much in agreement that while pure open source has advantages, and commercial software has advantages, the hybrid model is not able to capitalize on either, except perhaps for a short-lived bounce in marketing exposure.

Moving forward, Cittio customers will obviously have the choice of migrating to Nimsoft’s solutions, but I also want to encourage them to check out OpenNMS. They might find it strangely familiar.

Welcome Honduras!

Last week we received a purchase order from Honduras, and we’d like to welcome that country to our little family. While the OpenNMS Project has users all over the world (such as in Vietnam), we’re happy that the OpenNMS Group also has commercial clients all of over world as well. I look forward to the day that every country who has downloaded the software also includes a commercial client who wants a Greenlight, since there is a huge list of places I haven’t been able to visit yet, including anywhere in South America or Africa.

The current list of client countries includes the USA, Canada, Mexico, Singapore, Japan, Australia, Israel, Denmark, France, Germany, Switzerland, The Netherlands, the UK, Italy, Trinidad, Malta, India, and now Honduras. We also just finished work for a company in the Virgin Islands, but I’m not sure I can count it as a new country, although I’d love to visit.

Queremos dar la bienvenida a nuestros nuevos amigos en Honduras, y a esperanza de estar con ellos durante muchos años a venir.

Order of the Blue Polo Profile: Alex Hoogerhuis

I first came into contact with Alex in 2003 when he started buying things off of our Wishlist (which is looking a little bare at the moment). It was nice to know that someone appreciated our work enough to actually spend money, and Alex was one of the first people to regularly appear on our IRC channel (#opennms on

Alex travels a lot, and when I started doing a large amount of international travel he would often promise to meet me. After a number of “missed connections” we started to think that he was simply an IRC robot. However, at one of the LinuxWorld UK shows in London he did manage to show up, but that didn’t prevent us from naming the robot in our channel “_sndbot” after Alex’s “_snd” nick.

He lives in Norway, and he was kind enough to host me at his house last year. It was a fun, if hurried, trip, and I hope to return sometime soon.

Alex has posted a nice testimonial for the Order of the Blue Polo. We target OpenNMS for people just like Alex who make their living as open source experts, and I for one am happy that he is able to make his clients happy with our project. He is also the first and only person to have a running installation of OpenNMS in Africa (at least that I know about).

Please check out his note to see how well OpenNMS fits as a tool for solutions providers such as Alex. He also lists a number of his clients (with their permission, of course) who are able to benefit from the combination of our software and Alex’s talent.

Finally, I got the last shipment of Blue Polos out today. Those of you in the US should see yours by the end of the week. For those of you overseas, please give it a week or two to get there by mail. Let me know if you don’t get your shirt after that time.

If you haven’t already posted on the OBP, it is still open. We’d love to hear from you.

Twitter Outage

There is currently a Twitter outage going on:

However, Jeff thought it would be cool to monitor Twitter, so we all got notified.

Cool, huh? And we’ll know pretty soon after it comes back up.

NOTE: It actually came back up as I was typing this and I got the RESOLVED message. So much fun with network management.

Wanted: A Reasoned Debate, Mood: Resigned

When I presented my “Fauxpen Source” post, I had several goals. One was to introduce the term “fauxpen source“, which I think is clever. The second was to firmly state my position on the term “open source” as it is applied to “open source companies”. Finally, I wanted to show that open source qualifies as free software per the Free Software Definition.

I figured it would stir up some debate, but I was actually quite disappointed that while my post was attacked as being “tired” and of limited usefulness, not a single person stepped up to offer a different definition or to debate the points I raised.

I’m a pretty open minded guy, and I would love to see someone take a stab at a definition that takes into account the open core folks in a way that differentiates what they offer from commercial vendors, as well as what companies like OpenNMS offer. Instead the debate wandered all around the subject and never addressed it directly.

Matt Aslett questioned whether or not the “industry” needed a definition at all. I would claim it does, since many governments, including that of the US, are considering mandating or at least exploring requiring the use of more open source software. Isn’t that important? Shouldn’t we define terms? Aslett doesn’t like the OSD to be applied to business, but he didn’t go as far as to offer an alternative.

Some dude over at Information Week expressed the fact that he was bored with the whole thing, and asked “at what point does it cease to even matter?” (I would have commented on his post but it appears that they are unable to get comments working over there). I would claim that if it doesn’t matter, then why do all of these fauxpen source firms emphasize that they are “open source”? Just this week I saw a press release from one of them that started out “Open source network management” – if it isn’t important why lead with it?

The best comment I saw on the whole thing was from David Dennis of Groundwork:

So if a vendor isn’t primarily making money from license sales, but only partially, would they be misusing the term open source?

If a vendor gets 40% of its revenue from license sales, and 60% from support and services, they’re not ‘primarily’ getting money from non-open-source software.

Yay! An actual question that is relevant and is hard to answer. It is easy for me to claim a company is fauxpen source if their business plan is based on software license sales (i.e. the hockey stick revenue graphs they presented to their VCs ). But what if only 1% of their revenue comes from such licenses? What about 10% or 20%? At what point does it cross the line?

I don’t know, but I’d love to discuss it.

Ben recently posted about intent. I think intent does have a lot to do with it. If your intent is to basically force people toward your “enterprise extensions” by purposely crippling your “community edition” then that obviously isn’t in the spirit of open source. However, can I imagine a situation where a complementary add-on in no way blocks the functionality of the main project, so would it be okay to license that as commercial software and still call oneself an open source company?

Unfortunately, as it is not possible to objectively measure “intent” so we can’t use that as part of a definition. Again, the question is not an easy one.

Furthermore, it doesn’t seem like anyone is really interested in talking about this in a rational manner. People seem to prefer making blanket statements and not backing them up. Not to purposely pick on Matt Asay, but in one of his posts today he stated:

Red Hat is an example of “free done right,” following analysis from TechDirt. We’ve moved beyond the business models that insist that every line of software be open source: they couldn’t scale and tended to treat openness as an end in and of itself, rather than as a means to an end.

Today, if you look at the most successful open-source businesses, none of them pass the ideologues’ unrealistic and counterproductive “100-percent freedom” litmus test. Not a single one of them.

He uses Red Hat as an example quite a bit, but when Andres Garcia (and myself) asked “which software made by Red Hat has lines of code that aren’t open source?” there was no reply. This is because, to my knowledge, all of the software that Red Hat actually distributes (versus uses in house) comes with the source. So does OpenNMS, but my guess is that Matt wouldn’t categorize us as a successful open-source business. However, it is hard to argue that Red Hat is not (plus they pass the CentOS Test). But my guess is that Matt will continue to insist that Red Hat is a hybrid vendor since many of his arguments come crashing down if they are not.

I also assume am I one of those “ideologues” he talks about, but that is not the case. I just insist that software called “open source” meets the Open Source Definition – not that all software is open or free. I’m even open to an alternative definition, but so far no one has come up with one. Thus Matt hopes to sway people by using rhetoric, informal fallacies and irrelevant examples – not to enter into a real dialog.

Unfortunately, this is the state of the debate. Heck, it is the internet after all, so I guess I was a bit naive to expect otherwise. I’ll probably take a break from it for awhile, since it’s like the old joke about the pig – you always lose and end up dirty while the pig enjoys it.

10) Almost all panel discussions stink.

Chris Dibona wrote a post with points on becoming a better speaker, and number 10 on that list was to point out that “Almost all panel discussions stink.”

I have been invited by John Mark Walker to participate on a panel discussion at the LinuxWorld reboot (called OpenSourceWorld) on 12 August in San Francisco. It’s billed as “DM4: The Open Source Management Stack”. Moderated by John Mark, the panel will also include Michael Coté and Luke Kanies.

I’ve known John Mark and Coté for some time, and Luke and I have shared a number of e-mails, so at the very least it will be interesting just to have us together. Being a bit shy and withdrawn, I am not sure how much I will be able to add to the discussion, but I doubt it will fail to be … different … than your usual panel.

I’m hoping that this will be one of those panels that caused Chris to use the word “almost”.