Be Well, Honduras

This morning a magnitude 7.1 earthquake struck off the coast of Honduras.

Just recently we at OpenNMS welcomed a new client from Honduras, so Jeff sent them a note to make sure they were okay. This is the response we got back:

Yes as a matter of fact we felt the earthquake at 2am, but thankfully we are all fine and dry. Here in Tegucigalpa the quake was not so strong and there was very little damage if any.

But on the other hand our network suffered some damage and many of our sites which are closer to the epicenter are running on generators since 2am and are now very low on fuel, and other sites lost their microwave alignments. So we have a bit of a mess right now.

I’m glad everyone is okay and my thoughts are with you. Let’s hope that a minimal loss of mobile phone service is the worst that happens.

Interop 2009

In my commercial software days I used to go to the Interop show in Las Vegas, back when it was held at the main convention center. It was a huge show and pretty much the premiere event for networking gear. I think the last time I went was 2000.

I had the opportunity to return this year. The show has changed, it is now in the Mandalay Bay Convention Center and it is smaller than I remember. The NOC staff, however, is still pretty much the same.

As you can imagine, running a NOC at a show like this is no minor undertaking, but believe it or not the entire NOC is staffed by volunteers. Getting through an ordeal like an Interop show seems to bring people together, as many volunteers have been coming for years (I met one guy who had been coming here since 1996). The only downside was that this Interop marked the first since the passing of Jim “Haggis” Brown, a longtime NOC member. They had a place set out for him, along with a bottle of scotch.

Speaking of bringing people together, this trip has been pretty serendipitous. For example, my plane from RDU to DFW had mechanical problems, so they routed me through Miami. As I was leaving the Admirals Club to walk to my gate, I ended up sharing an elevator with Chris McGugan. Chris is something of a superstar in networking circles. He was at Cisco for many years (based out of North Carolina), and now he is working at Avaya out in California. We used to share a townhouse about 20 years ago, and it had been about that long since I’d seen him. The odds of us running into each other the way we did were pretty long.

Even stranger, Chris used to work in the NOC at Interop, and he knew many of the people I had come to meet.

Another example of serendipity: on our first day at the show, Jeff and I were at a table utilizing some wireless bandwidth when John Willis walked by. He didn’t know we were going to be there, so it was nice to see he had decided to wear his OpenNMS shirt anyway.

Jeff Gehlbach, High Mobley and John Willis

Things have changed a bit in Las Vegas since I was last here. There is no smoking near food (which pretty much leaves the casinos) and coins no longer work in the slot machines. Payouts are given on little slips of paper, and the machines will only accept bills or those little slips. I really miss the sound of the coins clanking around, and it makes the casinos seem quieter.

According to the cab driver, 40% of the usual conventions have cancelled this year, so the area is surviving on tourism. We stayed at the Luxor for $69 a night, and although it was a tower room, it was a deal.

The Luxor is my favorite hotel on the strip. It is not the nicest or the most luxurious, but think about it – it had to have been built by a geek. If I was given a boatload of money and told to build something impressive in the desert, it would be a pyramid. Plus at night its blackness contrasts well with the brightness of the other hotels, even with the sides having been given over to advertising.

However, one of the Luxor’s main acts is Carrot Top, and the dude is just scary looking. His face is everywhere you go in the hotel, even on the keys and the “do not disturb” signs, and it gets creepy after awhile.

Back to Interop: the show had most of the people you would expect. We stopped by the HP booth to look at the latest OpenView. HP must be doing well, because they had some seriously thick padding under the booth carpet, which was awesome (if you have ever worked a show on a concrete floor for a couple of days, you know what I am talking about). I decided to talk a little smack to their folks in the booth. I thanked them for raising their prices so drastically since it helped us out, which caused them to asked about OpenNMS. When I told them it was an open source network management platform, the reply was “yes, but OpenView is for the enterprise.”.

I took that as my cue to bring up that we have customers monitoring over 55,000 devices with OpenNMS (them: “with a single instance?”, me: “yup”) and that we were replacing OpenView at a client in Italy because their devices, which have more than 32,000 interfaces each, break OpenView but work with us. Things got quiet and a little awkward after that, so we left (but the lady kept my card).

Microsoft was a no-show (or at least I didn’t see their booth), but I did get introduced to a company called Xirrus. Xirrus builds wireless arrays that have a high level of built in switching, and their marketing pitch was a face-off between their wireless “switches” and wired ones. They had a boxing ring in the middle of the booth and several times a day held actual bouts. When it wasn’t being used by humans, one corner held your traditional network switch (with lots cables of course), and the other corner held a Xirrus array.

The arrays looked like big roombas with RJ-45 connections, and they had really cool lights (Jeff took a video).

All in all it was a fun time, mainly because we got hang “backstage” with people who really seemed to both love networking as well as knowing a lot about it. What did surprise me were the number of people that were using OpenNMS. When we’d get introduced we were often met with “Oh, we use OpenNMS. It’s great.”

It’s nice to hear. While we have things like the Order of the Blue Polo and the Wall of Cards, we rarely hear from people who use the tool outside of our clients. And while we love our clients, usually when we hear from them it is to ask a question or report a problem. We work hard to make OpenNMS great while remaining 100% open source so it definitely motivates us to meet people who find it useful.

It was a little sad when the show ended and the equipment started coming down. Perhaps we can return next year.

OpenNMS on MSN (sort of)

One of our oldest clients is Papa Johns International, better known as Papa Johns Pizza. They have sold over a billion dollars in pizza over the Internet, and they use OpenNMS to monitor that product line.

Chris Rodman sent me a link today to an MSN site called “Business on Main” that featured John Schnatter (the “John” in “Papa Johns”) mentoring a yogurt vendor from New York City. It’s an interesting clip, with quotes like “technology is a big part of our success” and his conscious decision to “make the internet a big part of our business”.

The fun part comes about 3:38 minutes in (the web site will show 4:15 remaining) when they move into the PJI NOC. There you can see OpenNMS front and center as their management solution.

We constantly get asked, sometimes in a condescending manner, about who would trust their important revenue streams to an open source solution. Papa Johns is a shining example of a technology-savvy enterprise that has become successful by not only embracing technology but by focusing on technology that works. Over the years OpenNMS has been molded to fit their business model (not the other way around), and we are happy to play at least a small part in their success.

UPDATE: There’s a shot of the OpenNMS maps as well:

OpenNMS 1.6.5 Released

I am happy to announce the availability of OpenNMS 1.6.5, the next stable release of OpenNMS.

New Features and Enhancements

  • Capsd will now use ifHighSpeed instead of ifSpeed when available (Bug #1822)
  • The SNMP configuration shipped with OpenNMS now uses SNMP v2c, fewer retries, and a longer timeout by default, for more efficient scanning. (Bug #3050)
  • Data collection and graphing has been added or updated for Alcatel-Lucent, Allot, Alteon, Aruba, Ascend, Avocent, Bluecat, BlueCoat, Cisco Airespace, Lucent, NetApp, Overture, Packeteer, and Powerware devices (Bugs #3096, #3099, #3109, and #3138)
  • Support has been added for using raw instance identifiers in thresholds when no data source label can be inferred (Bug #3108)
  • Support was added for a number of new Cisco devices in Linkd (Bug #3110)
  • Support has been added for extracting the bgpPeerRemoteAddr instance identifier from the varbinds of the bgpEstablished and bgpBackwardTransition traps defined in the BGP4-MIB. This allows us to make meaningful alarm annotations for these events so that the resulting alarms can self-clear. (Bug #3156)
  • Some tools have been added to OpenNMS “contrib” for creating SNMP walk dumps suitable for debugging with OpenNMS’s mock SNMP server. (Bug #3173)
  • The default “Database” category now includes MSSQL and Oracle in addition to MySQL and PostgreSQL. (Bug #3175)

Bug Fixes

  • The Capsd HTTP plugin now properly honors response-text parameters. (Bug #2774)
  • The HTTP collector now properly honors the “retry” parameter. (Bug #3133)
  • The Hyperic HQ service monitor definition now detects Hyperic HQ 4.x properly. (Bug #3106)
  • Multiple nodes with the same name/label can now be distinguished from each other in category views. (Bug #3112)
  • Vacuumd cleans out old (invalid) nodes from topology map tables. (Bug #3121)
  • The JMX collector now works when store-by-group persisting is enabled. (Bug #3160)
  • Configuration file filtering has been fixed so that tags aren’t properly expanded in (Bug #3174)
  • The remote poller build now ensures it is compiled with the correct version of Groovy. (Bug #3176)

The full list of changes is available in bugzilla.


I received an interesting call last week from Gary Read, the CEO of Nimsoft. He informed me that his company was buying the intellectual property assets of Cittio, a company with a rather storied history with respect to OpenNMS. With this acquisition, parts of the Cittio Watchtower product involving Layer 2 discovery, topology and root cause will be integrated with Nimsoft solutions and the Watchtower product will cease to be produced or supported.

Now, I’m a firm believer in karma. Not the formal concept as defined by many Indian religions (and I mean no disrespect by using the term casually) but the idea that if you do good things, good things will happen to you, and if you do bad things, well, bad things will happen. In my mind the failure of Cittio is karma in action.

If you search for “cittio” on my blog, you’ll see that since 2005 we have been rather curious as to how OpenNMS and Watchtower were related. The early releases of Watchtower seemed, feature for feature, to be remarkably similar to OpenNMS. I had a couple of conversations with Jamie Lerner, the CEO and founder of Cittio, where he assured me that any use of OpenNMS code was within the license requirements of the GPL. Information surfaced in 2008 that seemed to indicate that claim was inaccurate, so we hired the law firm of Moglen Ravicher to help us get to the bottom of this.

When our lawyers contacted Mr. Lerner, he reiterated his claim that Watchtower was not in violation of the GPL, and even if it was, any OpenNMS code being used by his application would be based on OpenNMS version 1.0.2. Since that code was copyright the Oculan corporation and not The OpenNMS Group, we had no right to enforce the copyright.

While we had received information that OpenNMS code post-version 1.0.2 was being used in Watchtower, we had no proof, and so without a long and expensive discovery process we were effectively stymied from getting to the bottom of this matter.

Thus it was with some surprise that we watched Cittio release a fork of OpenNMS in November of 2008 called Rooftop. If, as Mr. Lerner claimed, Cittio was always in compliance with the GPL, the fork should have been unnecessary, and I found it amusing that the press release went out of its way to specify it was a fork of OpenNMS 1.0.2.

As I write this, it appears that Rooftop has been downloaded around 60 times in the last 6 months or so. Compared to the 5000 downloads a month of OpenNMS from our repositories I would say that the fork hasn’t hurt our project at all. And for those who wish to test the claim that no OpenNMS code post version 1.0.2 was being used, I invite you to download the 1.0.2 source as well as the source of, say, 1.2.0, and draw your own conclusions. A search for the word “interfaceresolve” might provide a good starting point.

With the end of Cittio as a management company, I guess a lot of these points have become moot. Whatever the actual history of how Mr. Lerner came to use the OpenNMS code in Watchtower, the future should be much clearer. Mr. Read has stressed that he wants to make sure the OpenNMS project is satisfied that none of the OpenNMS code will be a part Nimsoft’s offerings, and he has offered, at his expense, to fly one of us out to inspect the code.

While I can’t (currently) see the Watchtower code, in looking through Rooftop there is no mention of linkd (our Layer-2 topology discovery process), alarmd (our event correlation engine) or our “drools-based” event correlator, so I assume that Nimsoft has done enough due diligence to be sure of what they are buying. While their emphasis on transparency is obviously driven by a wish to avoid any taint of license violations in Nimsoft products, it is refreshing that they took this seriously enough to contact me well before the press release and to make the blanket offer of showing us the code.

One might think this is strange behavior considering the press Nimsoft was getting in open source circles after the announcement that Hyperic was being acquired by SpringSource. Part of it was stupidity. A person at Nimsoft’s PR company apparently did a search on articles referencing the sale and sent a generic anti-open source article to Matt Asay. While Matt and I have had our differences, he was right to skewer the guy. I don’t mind the marketing effort, heck I spread my own FUD about commercial software with Cittio being a prime example (as in “what do I do now that this license I bought is to code no longer supported”), but just as with a lot of the near spam e-mails I get from potential “partners,” the PR guy should have done his homework before sending it to Matt. I do object to laziness.

Gary Read got his own bunch of replies on his blog post about the acquisition. Some people saw it as an attack on open source, but if you actually read his post it was an attack on the hybrid (what I call “fauxpen source“) model. When he called me, we had a nice long conversation and were pretty much in agreement that while pure open source has advantages, and commercial software has advantages, the hybrid model is not able to capitalize on either, except perhaps for a short-lived bounce in marketing exposure.

Moving forward, Cittio customers will obviously have the choice of migrating to Nimsoft’s solutions, but I also want to encourage them to check out OpenNMS. They might find it strangely familiar.