Lucky Number Seven

It was seven years ago today that David Hustace, Matt Brozowski and myself started The OpenNMS Group.

I had been working on OpenNMS for several years under three different companies by that point in time, but this seems to be the one that stuck.

Those of us who were in town went over to the Brewery and ordered every appetizer and every dessert they had on the menu.

That’s supposed to be lucky, too.

Nagios Spreads FUD About OpenNMS

It was brought to my attention that Nagios Enterprises has decided to go after OpenNMS by publishing a document called “How Nagios Compares to OpenNMS“. I am always flattered when companies feel the need to compare themselves to our project, especially when Nagios is considerably better known than OpenNMS, at least according to a quick look at Google hits (5.3 million to 453 thousand just now). I assume they feel that positioning themselves against our product is useful.

And I guess it could be, if they weren’t so wrong.

I would not pretend to make such a list from the OpenNMS viewpoint, since I don’t know all that much about Nagios, and what I do know comes from replacing it at a number of our customers’ sites. But let me correct some misunderstandings presented by this document.

Almost every missing check mark on the OpenNMS side is wrong. OpenNMS can’t monitor “Web Transactions”? What about the Page Sequence Monitor, used to insure the performance of such websites as papajohns.com? It doesn’t have “Google Maps Integration”? Well, not just Google Maps, but Mapquest and Open Street Map as well. It can’t do Active Directory Authentication? Please, I set it up all the time.

I really chuckled when I saw that the product can’t do Data Export, Advanced or Scheduled reporting, since I just spent a week doing just that at a large bank based in Chicago (one that caters to institutions and wealthy individuals).

With the JasperReports integration, OpenNMS can generate amazing reports, and since we’ve extended it to be able to mine JRobin/RRDtool data directly, we were able to get OpenNMS to report on a huge Virtual Machine server farm at the bank on things such as CPU utilization, process memory utilization and load average in a format that could be exported to Excel without the need to even glance at a graph. It was fully automated, including the report generation, distribution and even provisioning the devices to be included in it.

The kicker is that OpenNMS can run any Nagios checkscript, even using NRPE (although I strongly recommend using the Net-SNMP extend function versus that protocol for reliability), so I fail to understand where OpenNMS fails in the “Custom Plugins” department.

They even get our pricing wrong, even though we publish it online. The package they refer to as “1 year of standard support and one 1 week of consulting” maps to our Greenlight Project, which is $23,000 not $30,000 (that is for the Greenlight Plus Project, which includes two weeks of on-site services).

But if you are choosing to use Nagios XI based on price, I think you should go with with it. OpenNMS is designed to be a network management application platform, and as such has a much wider scope than Nagios, which, let’s face it, is at its heart a script management interface. I’m not sure how one provisions devices in Nagios, but since that was left off of their chart I must assume it isn’t a key feature, whereas it plays a huge role when you are trying to manage a network of any size, such as the one at Towerstream. Considering the scale at which OpenNMS is useful, our prices are a bargain when truly compared to competitors’ products.

Two years ago at the Netways Monitoring Conference I saw a presentation from Audi where they implemented Nagios. It took them over a year. We have done the same at similar sites in less than three weeks. If you have more than one thousand devices, you are going to be very unhappy with the performance of Nagios, whereas OpenNMS has a track record of monitoring tens of thousands of devices on a single server for numerous companies over several years.

And finally, every bit of OpenNMS code is published under an open source license. Nagios XI is not.

On Thursday, The OpenNMS Group turns seven. We don’t have a million customers, but considering how awesome our customers are, I know that I couldn’t find a million of their caliber.

And never once have we resorted to this kind of FUD to promote our products.

OpenNMS Training in September

The biggest complaint I hear about open source in general is the lack of documentation. Writing good documentation is hard, and with complex tools like OpenNMS, sometimes a more hand-on, interactive approach is better.

That is why we periodically offer training. We have two courses scheduled for September.

The first is the week of 12-16 September and will be held at the OpenNMS HQ in Pittsboro, North Carolina.

The second is the week of 26-30 September and will be held in Fulda, Germany, the site of our Users Conference back in May. We are happy to be able to offer this in conjunction with our partner, NETHINKS.

Both courses will be taught by me and David Hustace, so if the thought of listening to me talk for hours on end frightens you, take solace in the fact that David teaches half of it.

Hope to see you there.

Chicago Board Options Exchange

I’ve spent this week in Chicago. I love Chicago. I think if I didn’t love living out in the middle of nowhere in North Carolina I’d seriously consider moving here.

It also helps that we seem to have a cluster of OpenNMS customers in the area, mostly in finance. The trip this week was to work for the Chicago Board Options Exchange.

One of the schools I got kicked out of was Harvey Mudd College. A friend of mine from those years was a math major, and after graduation he went to work for a private equity firm as an options trader. It was fascinating to me, and once I spent an hour and a half just staring at the action on the floor of the Pacific Exchange. All the traders wear special jackets identifying who they work for, and all those colors moving around was quite mesmerizing.

But that was over 20 years ago, and options trading has changed a lot since then. The physical trading floor has shrunk and about 95% of trades are completed electronically. The CBOE is the largest options exchange in the US by volume, so you can imagine that information technology plays a key role.

I came up here as part of a Greenlight project to replace a Netview installation, and so far I’ve had a blast. The CBOE team are sharp guys. They had managed to do quite a bit with OpenNMS before bringing me up, and I’ve gotten to focus on cool integration stuff. After several installs, I think I finally have a handle on the black magic that is LDAP authentication, and we did some notification work to create pop-ups on operator screens as well as an integration with BMC Event and Impact Management (BEM).

But despite the fact that Chicago is pretty pleasant in July, I’m looking forward to heading home. I’ve been on the road a lot lately, and it doesn’t show signs of letting up anytime soon. After a week on the farm I’m off to Portland for OSCON. Hope to see some of you there.

Hershey Medical Center

One of the best ways to market your company is through customer stories. The downside: getting approval to post something publicly usually takes an extremely long time. We have one customer, a private equity firm, that won’t even let us recognize they exist (we use a code name, even internally).

But since we have great customers it is worth the effort and the wait. Today I was able to post a story from Hershey Medical Center that has been in the works for a year. I just did some consulting for them last week, and we were able to get it finished then.

Dale Meyerhoffer, Senior Network Analyst at HMC, has been a client for over three years now. I must say, that if I were tasked to make up the most flattering customer story I could think of, I’d have a hard time matching this.

We are humbled that he took the time to talk about his OpenNMS experience, and I hope it goes a long way toward removing any doubts people might have about switching.

Fighting Religious Battles

George Carlin once said:

Religion … is like a lift in your shoe. If you need it for a while, and it makes you walk straight and feel better – fine … I say just don’t ask me to wear your shoes. And let’s not go down and nail lifts onto the natives’ feet.

A newbie to the free and open source software world might think it is weird to talk about religion, but most of us who have been around awhile realize that the equivalent of religious fanaticism is pretty common with FOSS. People feel very emotional about certain technologies. And while this kind of fervor has its place, it doesn’t really work within open source business.

I was on a call today where a friend of mine (and OpenNMS proponent) who was talking about a meeting he was in where a team developing a new management application was presenting their technology assessment. To almost every requirement they stated, he was able to reply “OpenNMS does that”, yet when they presented their final scorecard OpenNMS was rated pretty low.

On further examination, it turns out that while most requirements were rated on a scale of 1 to 5, there was a category that basically said “fun to work with” that had some obscene weighting like 1 to 200. Of course, that meant the development team could basically fix their report anyway they wanted (and in this case it was for a technology based on Javascript).

My response was “oh well”.

I don’t fight religious battles.

Our job at the OpenNMS Group is to help people who want to work with OpenNMS get the most out of it. We don’t spend much time trying to get people to switch. For the most part, if there isn’t an immediate, pending reason why a switch is necessary (such as a large renewal bill for a commercial product coming up or they have outgrown their existing solution) there is little use for us to spend the time.

For example, another big OpenNMS fan got a new job, and he was trying to get OpenNMS in to replace their Spectrum/Netcool solution. He writes:

What the real problem here is that the two main Architects go way back 17-20 years, They fought to get Spectrum here when the shop had Cabletron, then Netcool about 5-6 years ago. Both are very good Perl programmers and have written custom scripts to tightly couple the two systems. Real good work.

But the Spectrum guy won’t go down with out a fight, chewed my tail more then once for even bringing up open source. The Netcool guy doesn’t want OpenNMS in here because it will cut into his turf and he worked hard to get it in here in the first place.

I replied:

I don’t know, man. Sounds like a losing fight from my perspective.

OpenNMS is not a free or low cost solution. It’s a powerful replacement for tools like Spectrum, OpenView and Tivoli. Sure, there are no licensing costs, but it does require just as much expertise as Spectrum to customize in order to get the best value.

Of course, I’d love to see another OpenNMS user, but I’m not sure how much help I can be selling it. Think about it, the best I could reasonably hope for would be a Greenlight Plus contract out of the deal. This would be two weeks on-site and a year of support.

So I send one of my guys out there and they meet your two main Architects. They are hostile and refuse to cooperate (kind of like Congress) and so my guy has a miserable time of it, but still manages to get some stuff done. Then comes support – these guys open ten tickets a day complaining about what OpenNMS can’t do. It costs me a guy just to deal with them.

It ain’t worth it.

When it comes to our mission statement of “Help customers, Have fun, Make money” it is much better to focus on people who really want your help and want to succeed than to try to usurp the incumbent solution. While it may mean a lost opportunity or even losing a client, it’s a win in the long run.

I Like Traffic Lights

I am constantly amazed at the uses our users find for OpenNMS.

We just posted an interview with John Jennery, IT Manager of the City of Grapevine, Texas. Grapevine is the city just outside of Dallas/Fort Worth which is where the airport is located, so if you have ever flown through DFW you’ve been there.

In addition to normal IT management tasks, John uses OpenNMS to monitor the traffic light system for the town. You might not think that is such a big deal, but its importance becomes clear when you realize that this year a rare snowstorm hit the city just as 150,000 people arrived for the Super Bowl.

Check it out, and many thanks to John for letting us post his story.

20 Questions

I got an e-mail asking for information about OpenNMS. Since we really don’t have a great source of information for things like this, I thought I’d post my reply publicly so everyone can share.

OS support

Q: Which platforms do you have support for?
A: Anything with a 1.5 or later Java JDK. These include Windows, Solaris, Linux and Mac OS X. OpenNMS has been known to run on an IBM OS/390.

Q: Which platforms has native install packages?
A: All of the above, except for the OS/390.

Q: Is it possible to repackage your product?
A: Yes. It is 100% open source software so you can repackage it and redistribute it based on the GPL.

Q: Can we install the product on our hosts?
A: You don’t need to. OpenNMS doesn’t require software to be installed on all hosts, just the OpenNMS server.

Q: Can we run the product in a virtual host?
A: Yes, although for performance reasons it is not recommended.

User interface

Q: Which are the interfaces to your application? Web, java, flash client?
A: Web and iPhone.

Q: Are there any requirements on browsers or operating system to access your user interface?
A: No.

Manage data

Q: List all the supported ways of collecting client side data? (Example: Install agents, use ssh, snmp, Json over http or other)
A: SNMP, WMI, XMP, HTTP, JDBC, JMX and NSClient.

Q: Is it possible to extend your product with our own scripts? If yes which languages are supported?
A: Yes, using any language that exists on the OpenNMS server.

Q: In which ways can we get access to the collected data?
A: Tell me how you want to access it, and I bet you can.

Q: How can we work with the data? Is the collected data stored in a open/accessible format, so that we can access it from other tools? If yes which tools can we use (scripts, tools…)?
A: Collected data is stored in round robin database files. You can access it by querying the application which will return it in XML, dump the data directly from a file into XML or use the JasperReports integration to extract the data as a PDF or CSV file.

Q: How is the graphing and configuration data stored?
A: In XML and properties files.

Q: Which levels of data granularity is supported?
A: For collected data? one second, although five minute polls are configured by default.

Q: For how long can we store the data? Will/can the granularity change over time?
A: For as long as you want. However, the more data you save, the more disk resources will be required. By default OpenNMS polls every five minutes and stores five minute samples for one week. Then the data is rolled up into an hourly average for 62 days. Daily minimum, maximum and average values are stored for one year.

Integration

Q: Is it possible to interface to your product via api or scripts so we can change configuration or retrieve data from other tools?
A: Yes. OpenNMS includes a ReST interface to accomplish most of these functions.

Q: Is it possible to integrate your product with Splunk?
A: Yes.

Q: Is it possible to integrate your product with user access control systems? If yes which (ldap, kerberos, active directory, other)
A: Yes – all of the above.

Q: Is it possible to add host/checks/graphs at runtime with out restart?
A: Sometimes. Some changes, however, still require a restart. This should be corrected by OpenNMS version 1.10.

Notification options

Q: In which ways can we get notifications?
A: How would you like to get notifications? Out of the box you can get e-mails, SMS (via e-mail), XMPP instant messages, microblog (Twitter), IRC as well as an integration with Asterisk to have the system call you. Custom methods can be easily added.

Q: Can the system create tickets in a ticket system? If yes, which?
A: Yes. Out of the box support for Request Tracker, Jira, OTRS, Quickbase and Concursive (CentricCRM). An API exists for others.

Q: Is it possible to set up a escalation chain of who receive the alarms?
A: Yes.

Q: Is it possible set up user patterns for who should be notified? (Timing shifts, escalation)
A: Yes.

Q: Is it possible to set up dependencies to prevent flooding of alarms?
A: Yes.

Redundancy

Q: How do we set up redundancy across different data centers and in one data center?

Read the white paper or the wiki.

Q: Can we setup the redundancy in master-master mode?
A: Yes.

Q: How can we scale your product?
A: Get a big machine. OpenNMS currently has one customer monitoring 66K discreet devices from a single instance, another collecting 1.2M data points every five minutes, one handling 120K syslog messages each minute, another monitoring devices with over 30K virtual interfaces per device, and one with 3000 remote pollers measuring performance back to the central NOC.

Q: What is the number of checks/hosts we can query from one server?
A: We haven’t hit a limit if hardware is not an issue, although I am sure there is one.

SLA (service level agreement)

Q: What are the options for us to measure SLAs?
A: OpenNMS generates availability reports based on outages. The data is customizable – so it would depend on what type of SLA you wanted to measure.

Licenses

Q: Describe the license model.
A: OpenNMS is 100% free and open source software licensed under the GPL.

Q: Is there an extra cost to monitor test environments?
A: No.

Support

Q: What types of support can you offer?

A: Standard Support, 24×7 Emergency Support, and Ultra Support (for those clients who have a need to present data from OpenNMS to their customers).

See: https://www.adventuresinoss.com/pricing

Q: Where are you located?
A: The east coast of the United States, with headquarters in North Carolina.

Q: In which time zone do you have office hours support?
A: Standard support is 7am to 7pm Eastern US time, business days. We do offer 24×7 support for emergencies that occur outside of that time.

OpenNMS Seminars

There was an article on opensource.com today that talked about whether or not a user community can drive innovation.

We rely on feedback from our users to help make OpenNMS better, and we specifically listen to the suggestions from our commercial (i.e. paying) users. While we welcome all input, on the commercial side of the business we want to make money. The best way to do that is attract more commercial clients, so we tend to pay a lot of attention to what our current clients want.

I was in Chicago back in December visiting customers, and while talking to one of them he suggested that we should focus on providing more services, in addition to support, that were just available to paying clients. One idea he suggested was some sort of training seminar that could be held to introduce customers to new ways of using OpenNMS and/or new features.

I thought it was an awesome idea, so today we held the first event of our monthly seminar series. I did a presentation on “Stupid Net-SNMP Tricks” that I took mainly from some blog posts I’ve done on the topic. The webinar lasted about 80 minutes with questions and I think it was well received. We held one at 10am and one at 2pm to accommodate clients as far east as Dubai and as far west as Hawaii.

Next month we’ll talk about the JasperReports integration.

What I enjoyed about it was that I got to interact with clients that I usually only see when they have an issue or it is time to renew. I can’t wait for the next great suggestion from our amazing customers.