Adventures in Open Source

The new year has been no less hectic than the old one, which is actually a good thing, I guess.

Yesterday was my 44th birthday. It was pretty much just another birthday, with the only somewhat interesting thing is that now my age starts and ends with the same numeral. It is also the cost in US cents for a first class postage stamp (which has been the case with my age since 1998).

Anyway, I did get a delightful and unexpected present. The gang at the Southern California Linux Expo (SCaLE) issued a press release announcing that I have been named one of the two keynote speakers for the conference in February.

This is pretty cool, and I am both honored and humbled by their decision. I will be presenting a talk about what it is like to start and run a business based on open source software. One reason I write this blog is to both inform, and perhaps inspire, others to take the risk and start their own business, and to be able to talk about this in front of the SCaLE attendees is pretty exciting. It has been one of my favorite shows since we first attended. OpenNMS will also have a booth where you can see the latest stuff we are working on, as well as to meet some of the folks from the community.

This will be my first ever keynote, if you don’t count last year’s inaugural OpenNMS Users Conference in Frankfurt, and speaking of that, we had so much success last year that we are doing again. The 2010 Users Conference will be back in Frankfurt, and we’ve extended it to two days to include a day of workshops. Nethinks is once again a major sponsor along with the OpenNMS Group, and we will hold it on 6-7 May. If you are interested in being a sponsor, please drop me a note.

We’ve had a great response to our Call for Papers, but if you are interested in presenting, there is still time as it doesn’t close until 31 January. Accepted speakers will receive two nights in the hotel and all meals, but you will be responsible for getting there. We are looking for both presentations on any aspect of OpenNMS as well as workshops.

I’ll post more information on attending once we’ve settled on the agenda.

Finally, we’ve added new training dates in April (the January training is booked). If you are finding OpenNMS a little daunting, consider coming to metropolitan Pittsboro for one of our training courses. They’re useful and a whole lot of fun.

The latest stable and unstable releases for OpenNMS landed this week. Please use your install method of choice to upgrade. We are now on a two month cycle for releases, the last one being in October and the next one being in February. It is my sincere hope that the February release includes 1.7.90, the 1.8 release candidate, but with the holidays approaching I’m not 100% sure we’ll make it. Once 1.7.90 is out I expect two week release cycles until 1.8 is ready.

Version 1.6.8 includes a number of bug fixes and small features. One of the coolest things, in my opinion, is an easy installer. While I find OpenNMS simple to install, there are a couple of little things one has to do in order to get it up and running. We decided it would be cool to help automate or at least assist people with that process.

Version 1.7.8 is the next release in our march toward 1.8. More code was added and cleaned up, as well as a number of bugs fixed. I would definitely not use 1.7 in production unless you are very experienced with OpenNMS, but feel free to use it on a development system to see what were working on for 1.8.

After last year’s inaugural OpenNMS Users Conference – Europe, we want to do it again. We’re going to extend it to two days and hope to model it more on the Nagios Users Conference I attended in Nürnberg back in October. Ours will once again be held in Frankfurt, Germany.

If you are interested in presenting, the Call for Papers is now open. The author of any accepted paper will have their hotel costs covered.

There are two types of papers being sought. The first are presentations, which can be either of a technical nature (how to use the OpenNMS application) or a business nature (how to use OpenNMS to improve business processes). These should be about 45 minutes long with 15 minutes for discussion.

The second are workshops. We are seeking two hour workshops to cover hands-on examples with OpenNMS. This is a new feature in this year’s conference.

The deadline for submitting an abstract is 31 December 2009 and speakers will be notified on 31 January 2010.

Hope to see you there.

One of our clients in Australia was asking me about the best way to monitor applications on his Linux servers. He wanted to be alerted when they died (or if a particular number of them were not running, etc.)

OpenNMS has a monitor based on the host resources MIB which can do this, but the downside is that there is no corresponding capsd plugin to do the discovery portion of it, so it can be a real pain to set up. I thought that Net-SNMP should be able to do this quite simply, but I found out that it isn’t nearly as easy as I thought it would be.

There is a directive within the Net-SNMP configuration file (snmpd.conf) called “proc”. For example:

proc testing 3 1

Configuring this line will cause the agent to look through the running processes for the string “testing”. If there are at least 1 and no more than 3 matches, the test is considered to pass.

This is reflected in the process table:

$ snmpwalk -v1 -c public localhost .1.3.6.1.4.1.2021.2.1
UCD-SNMP-MIB::prIndex.1 = INTEGER: 1
UCD-SNMP-MIB::prNames.1 = STRING: testing
UCD-SNMP-MIB::prMin.1 = INTEGER: 1
UCD-SNMP-MIB::prMax.1 = INTEGER: 3
UCD-SNMP-MIB::prCount.1 = INTEGER: 1
UCD-SNMP-MIB::prErrorFlag.1 = INTEGER: 0
UCD-SNMP-MIB::prErrMessage.1 = STRING:
UCD-SNMP-MIB::prErrFix.1 = INTEGER: 0
UCD-SNMP-MIB::prErrFixCmd.1 = STRING:

As you can see, the prErrorFlag is set to “0″ which means there is no error. This makes sense since the prCount is 1 and that is within the min/max range.

[Note: Leaving off both max and min results in a max value of infinity and a min value of 1. Just listing a max value results in a min value of 0]

If the “testing” process is stopped, this table changes:

$ snmpwalk -v1 -c public localhost .1.3.6.1.4.1.2021.2.1
UCD-SNMP-MIB::prIndex.1 = INTEGER: 1
UCD-SNMP-MIB::prNames.1 = STRING: testing
UCD-SNMP-MIB::prMin.1 = INTEGER: 1
UCD-SNMP-MIB::prMax.1 = INTEGER: 3
UCD-SNMP-MIB::prCount.1 = INTEGER: 0
UCD-SNMP-MIB::prErrorFlag.1 = INTEGER: 1
UCD-SNMP-MIB::prErrMessage.1 = STRING: Too few testing running (# = 0)
UCD-SNMP-MIB::prErrFix.1 = INTEGER: 0
UCD-SNMP-MIB::prErrFixCmd.1 = STRING:

This is all well and good, but the problem is how do we get the state change to generate a trap?

That was the hard part.

On reading the documentation, something like this should work. First, set the trap destination by adding a “trapsink” entry:

trapsink 172.20.1.11 public

and then add a “monitor” line:

monitor -r 15 "procTable" prErrorFlag 0 1

This should monitor the value of “prErrorFlag” and generate an error if it is “1″ or greater. You can also use expressions, so something like

monitor -r 15 "procTable" prErrorFlag > 1

should do the same thing. The “-r 15″ says to check for errors every 15 seconds.

Unfortunately, I was unable to get this to work. It took a lot of digging, but I found out that Net-SNMP requires a valid SNMPv3 username and password in order to access the tables so that “monitor” can check these values. Adding:

createUser snmpdInternalUser
rouser snmpdInternalUser noauth .1
iquerySecName snmpdInternalUser

enabled me to start getting traps.

This was a good start, but it still wasn’t perfect. Net-SNMP uses the Distributed Management events MIB to send the traps, so I was getting something like this:

It was a little generic and didn’t really tell me what I needed to know (namely, what was the process and what was the error). Also, I found the DISMAN events were hidden in the Cisco2.events.xml file, so I broke them out into their own file (which will be included in the releases in December).

A little more research uncovered that I could add other varbinds to the generic trap with just a few options to the “monitor” directive in the configuration file:

monitor -r 15 -o prNames -o prErrMessage "procTable" prErrorFlag 0 1

This would add the name and the error message to the trap. Finally, I noticed that while I got the rising or “down” traps, I wasn’t getting the falling or “up” traps. It turns out that I needed the “-t” option:

monitor -t -r 15 -o prNames -o prErrMessage "procTable" prErrorFlag 0 1

With a little added configuration to the new DISMAN.events.xml file, I was getting the proper rising events:

and the proper falling events:

So, with a little configuration it becomes quite easy to set up Net-SNMP to send traps into OpenNMS.

Here’s a summary:

Add the following lines to your Net-SNMP configuration file (usually /etc/snmp/snmpd.conf or /etc/snmpd.conf):

# Set up a V3 security name for internal queries
createUser snmpdInternalUser
rouser snmpdInternalUser noauth .1
iquerySecName snmpdInternalUser

trapsink 172.20.1.11 public

proc testing 3 1
monitor -t -r 15 -o prNames -o prErrMessage "procTable" prErrorFlag 0 1

You can have multiple “proc” entries while you only need one “monitor” entry.

Next, you’ll need to get the latest DISMAN.events.xml file from Sourceforge. Place it in your OpenNMS “etc/events” directory and be sure to add it to the list of include files at the bottom of the eventconf.xml file (add it before the Cisco2.events.xml file).

I saw this morning that RRDtool 1.4 was released. While there are a lot of changes, the biggest one seems to be the addition of rrdcached. From the release notes:

The RRD Caching Daemon can dramatically improve the ‘update’ performance of your system. Due to file handling overheads, the time it takes todo one update is virtually the same as to doing two updates in a row.

The Cache Daemon intercepts rrdtool update calls, assembling multiple updates before writing them to the actual rrd file. When calling rrdtool graph in such a setup, the command will tell the daemon to flush out all pending updates for the rrd files, required to draw the graph.

What I think is funny about this is that Matt Brozowski added this same functionality to OpenNMS years ago. It has been the only way we have been able to keep up with data collection at our larger installations.

The next time I get access to a large environment, I’d love to measure the performance of OpenNMS running with JRobin and queuing versus OpenNMS running with RRDtool with rrdcached and no queuing. If anyone has time to play with this, please let me know the results. You will need to compile jrrd against 1.4, of course. Also, it doesn’t look like Dag Wieers has 1.4 RPMs yet, but my guess is that he will soon.