Adventures in Open Source

One of the things I hate is the buzzword du jour, be it virtualization, “devops” or “the cloud“. It’s not that there isn’t some nugget of truth in all of the press surrounding such things, but one of the reasons I got into open source in the first place was its focus on results and not fluff.

With a commercial software product it is very difficult to determine if it is the right solution to a particular problem without buying it. With open source software, there is no licensing cost and thus it is possible to easily try it out before making a commitment to use it. Thus the focus is on usefulness and not a flyer saying “we’re the best”.

This isn’t to say that the open source world is completely free of fluff and posturing. With the prevalence of venture-backed open core companies, their ultimate goal is not the proliferation of robust open source code but to be purchased for a large multiplier. The best way for them to create perceived value is to latch on to the latest buzzword, as if to say “hey – you need a piece of this – better hurry up and buy us,” and it is a strategy that has worked well in a number of cases. I just don’t like calling it open source.

So I have been pretty quiet on the use of OpenNMS in “the cloud”. This isn’t to say that we don’t manage cloud resources, but the management challenges of cloud-based services aren’t much different than “normal” ones. The power and flexibility of OpenNMS make it as useful in the cloud as elsewhere.

In fact, one of the major players in cloud computing, Rackspace, uses OpenNMS to manage its Cloud Files system.

We are happy to announce that we are working with another major company BT (British Telecom Group) in developing a trusted cloud management platform called the Cloud Service Broker. In the words of John Gillam, Programme Director, BT Global Services:

The Cloud Service Broker TM Forum Catalyst provides an excellent opportunity to address the barriers to cloud adoption for enterprise customers. Whilst enterprises wish to lever value from the cloud, they are apprehensive over losing control, citing areas of concern such as IT Governance, application performance, runaway costs, inadequate security and technology lock-in. The CSB addresses this by matching cloud services to each enterprise’s needs, enforcing the right policies, and then showing how this can be backed up by an ongoing service level agreement. We believe developments of this nature will be of primary importance in future cloud services.

We will be presenting our work at the TMForum’s Management World conference in Nice, France, this May. In addition to BT’s offering, we will be demonstrating integration with products from Comptel, Square Hoop and Infonova in order to deliver a complete cloud services platform.

In my commercial software days I used to go to the Interop show in Las Vegas, back when it was held at the main convention center. It was a huge show and pretty much the premiere event for networking gear. I think the last time I went was 2000.

I had the opportunity to return this year. The show has changed, it is now in the Mandalay Bay Convention Center and it is smaller than I remember. The NOC staff, however, is still pretty much the same.

As you can imagine, running a NOC at a show like this is no minor undertaking, but believe it or not the entire NOC is staffed by volunteers. Getting through an ordeal like an Interop show seems to bring people together, as many volunteers have been coming for years (I met one guy who had been coming here since 1996). The only downside was that this Interop marked the first since the passing of Jim “Haggis” Brown, a longtime NOC member. They had a place set out for him, along with a bottle of scotch.

Speaking of bringing people together, this trip has been pretty serendipitous. For example, my plane from RDU to DFW had mechanical problems, so they routed me through Miami. As I was leaving the Admirals Club to walk to my gate, I ended up sharing an elevator with Chris McGugan. Chris is something of a superstar in networking circles. He was at Cisco for many years (based out of North Carolina), and now he is working at Avaya out in California. We used to share a townhouse about 20 years ago, and it had been about that long since I’d seen him. The odds of us running into each other the way we did were pretty long.

Even stranger, Chris used to work in the NOC at Interop, and he knew many of the people I had come to meet.

Another example of serendipity: on our first day at the show, Jeff and I were at a table utilizing some wireless bandwidth when John Willis walked by. He didn’t know we were going to be there, so it was nice to see he had decided to wear his OpenNMS shirt anyway.

Jeff Gehlbach, High Mobley and John Willis

Things have changed a bit in Las Vegas since I was last here. There is no smoking near food (which pretty much leaves the casinos) and coins no longer work in the slot machines. Payouts are given on little slips of paper, and the machines will only accept bills or those little slips. I really miss the sound of the coins clanking around, and it makes the casinos seem quieter.

According to the cab driver, 40% of the usual conventions have cancelled this year, so the area is surviving on tourism. We stayed at the Luxor for $69 a night, and although it was a tower room, it was a deal.

The Luxor is my favorite hotel on the strip. It is not the nicest or the most luxurious, but think about it – it had to have been built by a geek. If I was given a boatload of money and told to build something impressive in the desert, it would be a pyramid. Plus at night its blackness contrasts well with the brightness of the other hotels, even with the sides having been given over to advertising.

However, one of the Luxor’s main acts is Carrot Top, and the dude is just scary looking. His face is everywhere you go in the hotel, even on the keys and the “do not disturb” signs, and it gets creepy after awhile.

Back to Interop: the show had most of the people you would expect. We stopped by the HP booth to look at the latest OpenView. HP must be doing well, because they had some seriously thick padding under the booth carpet, which was awesome (if you have ever worked a show on a concrete floor for a couple of days, you know what I am talking about). I decided to talk a little smack to their folks in the booth. I thanked them for raising their prices so drastically since it helped us out, which caused them to asked about OpenNMS. When I told them it was an open source network management platform, the reply was “yes, but OpenView is for the enterprise.”.

I took that as my cue to bring up that we have customers monitoring over 55,000 devices with OpenNMS (them: “with a single instance?”, me: “yup”) and that we were replacing OpenView at a client in Italy because their devices, which have more than 32,000 interfaces each, break OpenView but work with us. Things got quiet and a little awkward after that, so we left (but the lady kept my card).

Microsoft was a no-show (or at least I didn’t see their booth), but I did get introduced to a company called Xirrus. Xirrus builds wireless arrays that have a high level of built in switching, and their marketing pitch was a face-off between their wireless “switches” and wired ones. They had a boxing ring in the middle of the booth and several times a day held actual bouts. When it wasn’t being used by humans, one corner held your traditional network switch (with lots cables of course), and the other corner held a Xirrus array.

The arrays looked like big roombas with RJ-45 connections, and they had really cool lights (Jeff took a video).

All in all it was a fun time, mainly because we got hang “backstage” with people who really seemed to both love networking as well as knowing a lot about it. What did surprise me were the number of people that were using OpenNMS. When we’d get introduced we were often met with “Oh, we use OpenNMS. It’s great.”

It’s nice to hear. While we have things like the Order of the Blue Polo and the Wall of Cards, we rarely hear from people who use the tool outside of our clients. And while we love our clients, usually when we hear from them it is to ask a question or report a problem. We work hard to make OpenNMS great while remaining 100% open source so it definitely motivates us to meet people who find it useful.

It was a little sad when the show ended and the equipment started coming down. Perhaps we can return next year.

There is currently a Twitter outage going on:

However, Jeff thought it would be cool to monitor Twitter, so we all got notified.

Cool, huh? And we’ll know pretty soon after it comes back up.

NOTE: It actually came back up as I was typing this and I got the RESOLVED message. So much fun with network management.

I like to think that the people who use our services get value for their money, but I sure many more ask the question “why do I even need support?”

At OpenNMS, we don’t sell software (all our software is free). I like to say we sell time. At the moment, anyone who has found out about OpenNMS, installed it and decided to use it obviously possesses well above average intelligence, impeccable taste and is most likely devilishly attractive. They are capable of figuring out issues without a support contract, either by experimentation, using the free resources such as the mailing lists, or both. But do they have the time?

Normally, most of the trouble tickets we get concern configuration, a few involve actual bugs with OpenNMS itself, and more than you would think are the result of vendors not honoring standards. We spend a lot of time figuring out issues with things like poorly written SNMP agents and even operating system problems.

And then there are the bad MIBs.

Recently I got an e-mail from a person who uses the Anevia Flamingo product. They wanted some help using mib2opennms to convert Flamingo SNMP traps into a format they could use.

Usually I have to politely decline helping people who contact me privately about OpenNMS issues. It wouldn’t be fair to our paying clients if I spent time helping people one-on-one for free, so I point them to free resources like the mailing lists. When I have time I try to help out there, as that gets archived publicly and might help others. The catch is that you may or may not get a timely answer to your question on the list, whereas you can always pester us about support tickets.

But this question involved mib2opennms. I’ve been using that tool for six years and my mib2opennms-fu is strong, so I took the Anevia MIB I was sent, cranked it through the tool and sent back the output.

I received a reply that it wasn’t working and the user was still getting unformatted trap errors like:

Received unformatted enterprise event (enterprise:.
1.3.6.1.4.1.20967.1.12.1.30 generic:6 specific:2). 3 args: .
1.3.6.1.4.1.20967.1.12.1.30="" .1.3.6.1.4.1.20967.1.12.1.30.1="1" .
1.3.6.1.4.1.20967.1.12.1.30.2="10.180.1.232"

I went into the file I had created and noticed that the enterprise id was missing the last “.30″, which is why it wasn’t matching, so it was off to look at the MIB.

It started off normally enough, with some object definitions:

anevia OBJECT IDENTIFIER ::= { enterprises 20967 }
anevia1 OBJECT IDENTIFIER ::= { anevia 1 }
tsnmp OBJECT IDENTIFIER ::= { anevia1 1 }
manager OBJECT IDENTIFIER ::= { anevia1 12 }
aneviaManager1 OBJECT IDENTIFIER ::= { manager 1 }
aneviaManagerTraps1 OBJECT IDENTIFIER ::= { aneviaManager1 30 }

and then later in the MIB came the trap:

inputDownTrap TRAP-TYPE
  ENTERPRISE aneviaManager1
  VARIABLES { streamerInputIndex, streamerAddress }
  DESCRIPTION
    "This trap is sent when an input on a streamer becomes unavailable,
     and can no longer provide any useful data, the provided index is the
     index of this input."
  ::= 2

At least the mystery of the missing “.30″ was solved. The “ENTERPRISE” value for this trap should be “aneviaManagerTraps1″ instead of “aneviaManager1″. Easy enough to fix. But then I noticed that instead of the two varbinds listed in the MIB, the agent was sending three (see above) where the first one was blank (as well as being just the enterprise OID).

Grrrr.

The second varbind value of “1″ could easily be the streamerInputIndex and “10.180.1.232″ could be the streamerAddress but these won’t be correctly reflected in the events file since they’re off by one due to the mystery blank initial varbind.

This is the case of a poorly written MIB and a poorly implemented agent, and there is little we can do about it but work around it in configuration. I asked the user to make sure we had the latest Anevia MIB and was told we did. I wrote Anevia support but since I don’t have a relationship with them I never got a reply.

This happens way more than you might imagine, and we’ve gained a lot of experience in diagnosing and either correcting or working around such issues. Because we’ve seen stuff like this before, we can do this quickly, which is why I like to say I sell time. It only takes a few issues like this to have a support subscription pay for itself.

[Note: This post isn't meant to be a pitch for services but a rant about the time I wasted playing with the Anevia MIB, but if it helps sell a support contract, that's cool too (grin)]

Nice is nice.

Okay, got that out of my system.

The trip to Nice was uneventful. Our hotel is very comfortable considering the price, and it’s in a great location.

David and I are in Nice for the TeleManagement Forum conference. This is the premier worldwide telecom event, and we are slowly introducing the concept of free and open software to this market. Craig Gallen (OGP) got us involved a couple of years ago, but this is the first time we’ve been able to attend the conference.

One of the dominating management concepts of the TMForum is Next Generation Operational Systems and Software (NGOSS). This defines a large number of interfaces for various management functions to interact. Through Craig’s work OpenNMS includes support for the “quality of service” (QoS) interface, and we’ve completed a proof of concept implementation using it.

It’s also cool to be in a place where I am the customer vs. the vendor.

If any of the three people who read this blog are also here, please drop me a note so we can meet up. I’m here until Thursday afternoon when we head to Paris.