Archive for the 'OpenNMS – general' Category

Important Security Issue with OpenNMS

Tuesday, January 13th, 2015

It is said that “given enough eyeballs, all bugs are shallow”, which is true, but the tricky part is finding enough eyeballs, especially useful ones and not the ones in that jar in Blade Runner.

Recently, an end user reported a rather severe security issue with OpenNMS.

The process that serves up the “Categories” section on the front page of the web interface is called RTC (for Real Time Console). The database queries that create the availability numbers on that page can be expensive in terms of resources, so the RTC daemon was created to periodically query the database and then cache the results so that lots of users wouldn’t create an undo load on the system.

We use a tool called Castor to process XML data within OpenNMS. Due to a bug in Castor, if Castor discovers an error when processing an XML file, it can throw an exception that includes the contents of the file.

This is very useful when the files relate to OpenNMS and you are trying to debug them, but you don’t exactly want the contents of /etc/shadow or /etc/passwd displayed indiscriminately. That’s exactly what this exploit allows.

Since the default username and password for the RTC user is “rtc” and exists on every system, a malicious person could use that information to obtain the contents of any file on the system. Note that as far as the OpenNMS application is concerned, the RTC user has very limited permissions, but this is caused by an issue with Castor and it has just
enough permissions to trigger it.

This has been reported as our first ever CVE: CVE-2015-0975

The best fix is to upgrade to OpenNMS 14.0.3. If, however, you are unable to upgrade soon, you can edit the Spring security file to limit requests from RTC to just the localhost, which should mitigate most of the issue. Full instructions and files can be found on the wiki.

To summarize, all versions of OpenNMS prior to 14.0.3 contain a bug where *anyone* with access to the webUI (port 8980 on the OpenNMS server) can retrieve any file that is on the system. While this isn’t the end of the world, it definitely could be considered bad and should be addressed.

OpenNMS 14.0.2 Released

Thursday, December 11th, 2014

Today we released version 14.0.2 of OpenNMS. It is a recommended upgrade for all OpenNMS 14 users because is addresses a memory leak caused by the version of Vaadin we were using.

Here is a list of all the changes.

Sub-task

  • [NMS-7238] – Citrix Netscaler trap events

Bug

  • [NMS-6551] – Syslog Northbounder throws exceptions on certain alarms
  • [NMS-7073] – ICMP availability with custom packet size doesn't work with JNI
  • [NMS-7092] – Node page for a switch or router is unusable with Enhanced Linkd enabled
  • [NMS-7130] – Vaadin applications show Page Not Found error
  • [NMS-7186] – The XML Collector is not storing the proper data for node-level resources
  • [NMS-7187] – The XML Collection Handler is caching the resourceTypes
  • [NMS-7190] – Edit an existing scheduled outage from node's page doesn't work
  • [NMS-7193] – The report "Total Bytes Transferred By Interface" is not working with RRDtool
  • [NMS-7195] – When the DNS name of a discovered node changes, Provisiond doesn't update the node label.
  • [NMS-7218] – Null pointer exception removing services from node
  • [NMS-7227] – Some GWT pages are not working on IE
  • [NMS-7231] – The downtime model never removes the nodes when it is instructed to do it
  • [NMS-7243] – XML collector in JSON mode assumes all element content is String
  • [NMS-7245] – NPE on "manage and unmanage services and interfaces"
  • [NMS-7250] – Clicking On View Node Link Detailed Info Give java.lang.IllegalArgumentException

Enhancement

  • [NMS-7194] – Move the "Add new outage" to the top of the page.
  • [NMS-7230] – The Wallboard app makes OpenNMS unusable after a few days even if it is not used.
  • [NMS-7237] – Mikrotik RouterOS trap definitions

Test Driven Development

Friday, November 7th, 2014

One of the things that bothers me a lot about the software industry is this idea that proprietary software is somehow safer and better written than open source software. Perhaps it is because a lot of people still view software as “magic” and since you can’t see the code, is must be more “magical”. Or perhaps is it because people assume that something you have to pay for must be better than something that is free.

I’ve worked for and with a number of proprietary software companies, so I’ve seen how the sausage is made, and in some cases you don’t want to know. Don’t get me wrong, I’ve seen well managed commercial software companies that produce solid code because in the long run solid code is better and costs less, but I’ve also seen the opposite done simply to get a product to market quickly.

With open source, at least if you expect contribution, you have to produce code that is readable. It also helps if it is well written since good programmers respect and like working with other good programmers. It’s out there for everyone to see, and that puts extra demands on its quality.

In the interest of making great code, many years ago we switched to the Spring framework which had the benefit that we could start writing software tests. This test driven development is one reason OpenNMS is able to stay so stable with lots of code changes and a small test team.

What’s funny is that we’ve talked to at least two other companies who started implementing test driven development but then dropped it because it was too hard. It wasn’t easy for us, either, but as of this writing we run 5496 tests every time something changes in the main OpenNMS application, and that doesn’t include all of the other branches and projects such as Newts. We use the Bamboo product from Atlassian to manage the tests so I want to take this opportunity to thank them for supporting us.

OpenNMS 14 contained some of the biggest code changes in the platform’s history but so far it has been one of the smoothest releases yet. While most of that was due to to the great team of developers we have, part of it was due to the transparency that the open source process encourages.

Commercial software could learn a thing or two from it.

OpenNMS 14 Timelines

Wednesday, November 5th, 2014

I often talk about how OpenNMS is a platform and not just an application, and with the release of OpenNMS 14 there is a lovely way to demonstrate the difference.

There is a cool little GUI improvement that I believe was started at last year’s Dev Jam which provides graphical timeline for outages. So now instead of having to look at the outage table on a node’s page, you can just look at the service availability section.

Cool, huh? What you may not realize is that instead of hardcoding the feature the timelines are rendered through ReST. The GUI sends a ReST request to the server which returns the graphic information. Let’s examine the “Update” service above.

The query

/opennms/rest/timeline/image/46/172.20.1.38/Update/1415119622/1415206023/480

results in:

with a format of:

/opennms/rest/timeline/image/{nodeId}/{ipAddress}/{serviceName}/{start}/{end}/{width}

Even the header graphic is done the same way

/opennms/rest/timeline/header/1415119622/1415206023/480

results in:

with a format of:

/opennms/rest/timeline/header/{start}/{end}/{width}

Of course, assembling all of that can be tedious, so this query:

/opennms/rest/timeline/html/46/172.20.1.38/Update/1415119622/1415206023/480

with a format of:

/opennms/rest/timeline/html/{nodeId}/{ipAddress}/{serviceName}/{start}/{end}/{width}

will create the whole HTML code needed to render the timeline:

document.write('<img src="/opennms/rest/timeline/image/46/172.20.1.38/Update/1415119622/1415206023/480" usemap="#46-172.20.1.38-Update">
<map name="46-172.20.1.38-Update"><area shape="rect" coords="128,2,412,18" href="/opennms/outage/detail.htm?id=153740" alt="Id 153740"
title="2014-11-04 18:13:24.628"><area shape="rect" coords="-111,2,-26,18" href="/opennms/outage/detail.htm?id=153724" alt="Id 153724" 
title="2014-11-04 06:12:56.322"><area shape="rect" coords="-2051,2,-1925,18" href="/opennms/outage/detail.htm?id=153348" alt="Id 153348" 
title="2014-10-31 06:13:11.421"><area shape="rect" coords="-2291,2,-2291,18" href="/opennms/outage/detail.htm?id=153289" alt="Id 153289" 
title="2014-10-30 18:11:33.006"><area shape="rect" coords="-2691,2,-2397,18" href="/opennms/outage/detail.htm?id=153258" alt="Id 153258" 
title="2014-10-29 22:13:27.086"><area shape="rect" coords="-2871,2,-2871,18" href="/opennms/outage/detail.htm?id=153235" alt="Id 153235" 
title="2014-10-29 13:12:29.747"><area shape="rect" coords="-3071,2,-2884,18" href="/opennms/outage/detail.htm?id=153137" alt="Id 153137" 
title="2014-10-29 03:12:13.887"><area shape="rect" coords="-3232,2,-3231,18" href="/opennms/outage/detail.htm?id=153132" alt="Id 153132" 
title="2014-10-28 19:11:02.873"><area shape="rect" coords="-3690,2,-3670,18" href="/opennms/outage/detail.htm?id=153086" alt="Id 153086" 
title="2014-10-27 20:14:11.949"><area shape="rect" coords="-6431,2,-6431,18" href="/opennms/outage/detail.htm?id=152786" alt="Id 152786" 
title="2014-10-22 03:11:05.149"></map>');

If a service isn’t monitored, such as the StrafePing service in the above example, that empty timeline is also available:

/opennms/rest/timeline/empty/1415119622/1415206023/480

with a format of:

/opennms/rest/timeline/empty/{start}/{end}/{width}

Pretty cool, huh? A lot of OpenNMS is accessible by ReST and the wiki page covers most of the options. Thus you can use the data via the OpenNMS GUI or integrate it with one of your own.

Announcing OpenNMS 14 and Newts 1.0

Tuesday, November 4th, 2014

It is with great pleasure that I can announce the release of OpenNMS 14. Yup, you heard right, OpenNMS *fourteen*.

It’s been more than 12 years since OpenNMS 1.0 so we’ve decided to pull a Java and drop the “1.” from the version numbers. Also, we are doing away with stable and development branches. The Master branch has been replaced with the develop branch, which will be much more stable than development releases have been in the past, and we’ll name the next major stable release 15, followed by 16, etc. Do expect bug fix point releases as the in past, but the plan is to release more major releases per year than just one.

A good overview of all the new features in 14 can be found here:

https://github.com/OpenNMS/opennms/blob/release-14.0.0/WHATSNEW.md

The development team has been working almost non-stop over the last two months to make OpenNMS 14 the best and most tested version yet. A lot of things has been added, such as new topology and geographic maps, and some big things have been made better, such as linkd. Plus, oodles of little bugs have finally been closed making the whole release seem more polished and easier to use.

Today we also released Newts 1.0, the first release in a new time series data storage library. Published under the Apache License, this technology is built on Cassandra and is aimed at meeting Big Data and Internet of Things needs by providing fast, hugely scalable and redundant data storage. You can find out more about this technology here:

http://newts.io

While not yet integrated with OpenNMS, the 1.0 release is the first step in the process. Users will have the option to replace the JRobin/RRDtool storage strategies with Newts. Since Newts stores raw data, there will be a number of options for post-processing and graphing that data that I know a number of you will find useful. Whether your data needs are simple or complex, Newts represents a way to meet them.

Feel free to check out both projects. OpenNMS 14 should be in both the yum and apt repos, and as usual I welcome feedback as to what you think about it.