In one of my past lives, I was actually training to be a chemist. I remember pretty much only one thing from those days, and that was an exchange between a professor and myself.
The professor asked me “What does a thermometer measure?”. I thought the answer was obvious: temperature. “The temperature of what?” he replied. Hmmm, whatever it is in? He answered, “A thermometer measures one thing – the temperature of the thermometer.”
The point he was trying to make involved techniques to insure that the temperature of the thermometer and the temperature of what you wanted to measure were close if not the same. But I learned that any measuring, or in our case monitoring, system used is constrained to measure only that which is known to it.
Take ICMP response time in OpenNMS, for example. I was real happy my “pings” were on the order of microseconds, and OpenNMS reported round trip times on the order of microseconds. Way to go OpenNMS.
However, it was pointed out to me that the times reported by OpenNMS didn’t necessarily match up with reality. I’d never really looked at it before. I was expecting a number on the order of a millisecond or less, and I got numbers on the order of milliseconds.
But take a command line tool like ping and use it on the OpenNMS system. Local RTT for me is around 50 microseconds. OpenNMS reports an average of 300 microseconds, with peaks of 4-6 milliseconds. And notice that when OpenNMS is first started or under load, the times get longer.
I spent two days trying to figure this out. The problem is that no Java API exists for ICMP, so you have to use an outside program. We use code written in C and accessed via JNI to perform “ping-like” functions.
But unlike other pollers, where we make a request and wait for the answer, with pings we send an ECHO_REQUEST packet, and then another process examines the ECHO_REPLY packets that get returned (as well as all other ICMP traffic, which the program discards). This ReplyReceiver process calls the C code to check for any new ECHO_REPLY packets.
Here’s where the problem lies. We send the system time out with the ECHO_REQUEST packet, and the ECHO_REPLY packet returns that time. When that packet is received, we take another sample of the system time, and the difference is the RTT. But because we don’t check immediately when the reply packet is received, there is a “lag” between when the packet is really received and when OpenNMS marks it as received. This difference can vary under system load, and is why the RTT for ICMP is so off.
Note that this doesn’t affect other pollers, just ICMP.
I am not sure that this can be fixed. The way to fix it is to send the ECHO_REQUEST packet and then wait for the reply, all within the same process. But you can probably see how this would cause problems on large networks. Packets get lost, retries have to be attempted, etc. We would have to write pretty much the whole ICMP management piece in C and then loosely tie it back to Java, instead of doing most of it in Java and using a bare minimum of C. Think this is trivial? Check out all of the O/S specific code in IcmpSocket.c and you’ll see what I am talking about.
Anyway, native ICMP support is not even an option until Java 1.5 (maybe). So I think for now I may remove ICMP response times from the default OpenNMS install. They can still be useful as a number that goes down when things are good and up when things are bad, but since it really isn’t measureing temperature, we shouldn’t call it a thermometer.
You aren’t throwing the baby out with the bath water, are you?
Seems like if your margin of error is wide and variable, that’s a problem. However, if it’s wide and variable but still in an acceptable range (as compared to the values which would cause consternation amongst network admins), then it’s not a big deal.
Do what you like, but I don’t know if getting rid of “features” is a good idea, especially if this dismissal is because of inaccuracies that are still within the tolerances of most.
My 2 cents…
I agree with Shane. Especially since OpenNMS focuses on *interactive* response time, a small amount of application-induced delay is reasonable, since no regular user app is going to be as accurate as command-line ping. I say keep it but make a note that the current architecture and delays introduced by java mean that the measurements are only useful as an off-the-cuff “rule of thumb” measurement. OpenNMS’s *real* goal is service polling anyways.
OpenNMS running on a fast machine monitoring around 125 interfaces frequently reports latencies on the order of 300ms on a network where the average latency is closer to 2ms. This generates numerous threshold alarms (over 1000/week) even with the trigger set to 3.
I cannot, in good faith, state that OpenNMS measures latency when the “range of error” is so large. But when I said I was going to remove ICMP response times from the default OpenNMS install, I meant the default installation configuration. Anyone who finds them valuable can turn them on at will.
Finally, we may have a code fix in the works. A process written in C that grabs the packets as soon as they come in and time stamps them, and then queues them up for the JNI IcmpSocket receive request. If we can get close to what a command line ping reports, then I will be happy turning ICMP reponse time back on by default.
Agreed. My comment was conditional on the inaccuracies being within an acceptable range. 300ms is not an acceptable range.
That said, if it’s consistently at 300ms, then perhaps the system-induced latency could be measured and removed (if consistent). Or perhaps, if the whole measurement process is consistent, then the important factor is extreme changes–not pure measurement.
Either way – I see your point.
Sorry, but that is still not the best option.
I too noticed that the PING times measured the OpenNMS box as well as latency.
But they doesn’t invalidate the whole thing. We still use a thermometer, even though it does nothing more than measure its own temperature.
I might make a note of how PING response times may very with heavy system utilization, and perhaps do some terminology adaptation.
But the concept and measurements are not invalid completely.
I guess it’s time to write an ICMP poller in Java, eh? (what was that document I read that said ONMS was all about writing an NMS in ALL Java 🙂
Unfortunately, it is not possible to write it all in Java since Java has no API for ICMP (I hear rumours about 1.5).
Basically, if someone off the street installs OpenNMS to evaluate, and they see ping times that are way off (perhaps they are running on a slow system, etc.) they may begin to doubt the entire product.
While we hope to fix this soon, I am planning on removing the default entries from the poller configuration that create these graphs out of the box. An experience OpenNMS user will still be able to configure and use them.
Actually, I hope this whole point will be moot soon (grin).