SCaLE 13x – February 2015

January 30th, 2015

We are three weeks away from the Southern California Linux Expo and I am getting really excited about it.

For those of you who are in to OpenNMS then tune in that day because we are making a pretty significant announcement at the show. Be sure to come buy the booth on the expo floor and say “hi” to the team, and both Jeff and I will be speaking (although at least during my talk you probably have better things to go see. For example, have you met our Lord and Savior, Docker?)

We are also incredibly excited that MC Frontalot will be performing. I’m not sure of the exact details but I believe it will be Saturday night.

(Note: I stole that picture from here since I like the fact that he has hair in it, well for certain values of “hair”, and note that link may not be safe for work [nudity])

If you are unfamiliar with his work, be sure to check out his YooToob Channel, and if you are so inclined I strongly recommend reading this well written bit (on Jezebel no less) concerning an issue surrounding a Penny Arcade comic a few years ago that really showcases the type of guy he is. Again, might not be safe for work (language). Be sure to click on the link to the original post for more detail.

If you are still on the fence about SCaLE, perhaps this little nugget will sway you: use Promo Code “ONMS” and get 40% off show registration. It’s cheap at twice the price and one of my favorite events of any year, but we want it to be extra special for 2015.

Twitter

January 29th, 2015

After a long absence, I thought I’d let my three readers know I’m back on Twitter (as @sortova). Don’t expect much from me since I can’t even say “I’m back on Twitter” in less than 140 characters, and I tend to echo the sentiments of John Cleese on the subject, but it should allow those of you with nothing better to do more things with which to do nothing.

Welcome to OpenNMS 15

January 28th, 2015

Today OpenNMS 15 was released. It was a year and a half between the release of OpenNMS 1.12 and OpenNMS 14, but only three months between OpenNMS 14 and OpenNMS 15.

As we move forward this year we are trying to adhere more to the open source mantra of “release early, release often”, and thus the new major release. There have been 1177 new commits since 14.0.3

You’ll also notice that this version of OpenNMS has a new name – Horizon. We’ve always thought that OpenNMS represents the best network management platform available and the name is meant to reflect that. We hope to make as many improvements we can, as fast as we can, without sacrificing quality, thus keeping OpenNMS out on the “horizon” from the competition.

The main improvement for the 15 release is in the webUI. Although you might not notice it at first, we’ve spent months migrating the whole interface to a technology called Bootstrap. The Bootstrap framework allows us to create a responsive UI that should look fine on a computer, a tablet or a phone. This should allow us a lot more freedom to modify the style sheet and we hope to be able to add “skinable” theme options soon.

A cool feature that can be found in this new UI is the ability to automatically resize resource graphs. If you have a particular set of resource graphs displayed:

and then you shrink the window, you’ll note that the menu turns into a dropdown and the graphs themselves now fit the more narrow window:

There are a number of bug fixes and other new features, and a complete list can be found at the bottom of this post or in our Jira instance (but for some reason you have to be logged in to see it). I am happy to say that there was no need for major security fixes in this release. (grin)

Sub-task

  • [NMS-6642] – CiscoPingMibMonitor
  • [NMS-6674] – NetScalerGroupHealthMonitor
  • [NMS-7060] – merge DocuMerge branch into develop branch
  • [NMS-7086] – alter documentation deploy step in bamboo to match the new structure
  • [NMS-7164] – Fix fortinet event typos (fortinet vs fortimail)
  • [NMS-7238] – Fix UEI names for CitrixNetScaler trap events
  • [NMS-7264] – Document CORS Support

Bug

  • [NMS-1956] – Missing localised time in web pages
  • [NMS-2358] – Time to load Path Outages page grows with each entry added
  • [NMS-2580] – Null/blank sysName value causes null/blank node label
  • [NMS-3033] – Create a HibernateEventWriter to replace JdbcEventWriter
  • [NMS-3207] – Able to get to non authorised devices via path outages link.
  • [NMS-3615] – Custom Resource Performance Reports not available
  • [NMS-3847] – jdbcEventWriter: Failed to convert time to Timestamp
  • [NMS-4009] – wrong content type in rss.jsp
  • [NMS-4246] – Paging arrows invisible with firefox on mac
  • [NMS-4493] – Notification WebUI has issues
  • [NMS-4528] – Time format on Event webpage is different that on Notices webpage
  • [NMS-5057] – Installer database upgrade script (install -d) scans every RRD directory, bombs with "too many open files"
  • [NMS-5427] – RSS feeds are not valid
  • [NMS-5618] – notifications list breadcrumbs differs from notifications index page
  • [NMS-5858] – Resource Graphs No Longer Centered
  • [NMS-6022] – Vaadin Header not consistent with JSP Header
  • [NMS-6042] – Empty Notification search bug
  • [NMS-6472] – Map Menu is not listing all maps
  • [NMS-6529] – Web UI shows not the correct Java version
  • [NMS-6613] – Problems installing "Testing" on Ubuntu 14.04
  • [NMS-6826] – Queued Ops Pending default graph needs rename
  • [NMS-6827] – Many graph definitions in snmp-graph.properties have line continuation slashes
  • [NMS-6894] – New Focal Point Topology UI (STUI-2) very slow
  • [NMS-6917] – Node page availability graph isn't "(last 24 hours)"
  • [NMS-6924] – WMI collector does not support persistence selectors
  • [NMS-6956] – test failure: org.opennms.mock.snmp.LLDPMibTest
  • [NMS-6958] – Requisition list very slow to display
  • [NMS-6967] – GeoMap polygons activation doesn't accurately reflect cursor location
  • [NMS-7015] – Navbar in Distributed Map is missing
  • [NMS-7059] – Local interface not displayed correctly in "Cdp Cache Table Links"
  • [NMS-7075] – xss in device snmp settings
  • [NMS-7112] – provision.pl just works if the admin user credentials are used
  • [NMS-7115] – Message Error in DnsMonitor
  • [NMS-7120] – Unable to add graph to KSC report
  • [NMS-7126] – ReST call for outages ends up with 500 status
  • [NMS-7144] – OpenNMS logo doesn't point to the same file
  • [NMS-7149] – footer rendering is weird in opennms docs
  • [NMS-7170] – Add a unit test for NodeLabel.computeLabel()
  • [NMS-7176] – ie9 does not display any 'interfaces' on a switch node – the tabs are blank
  • [NMS-7185] – NullPointerException When Querying offset in ReST Events Endpoint
  • [NMS-7246] – OpenNMS does not eat yellow runts
  • [NMS-7270] – HTTP 500 errors in WebUI after upgrade to 14.0.2
  • [NMS-7277] – WMI changed naming format for wmiLogicalDisk and wmiPhysicalDisk device
  • [NMS-7279] – Enable WMI Opennms Cent OS box
  • [NMS-7287] – Non provisioned switches with multiple VLANs generate an error
  • [NMS-7322] – SNMP configuration shows v1 as default and v2c is set.
  • [NMS-7330] – Include parts of a configuration doesn't work
  • [NMS-7331] – Outage timeline does not show all outages in timeframe
  • [NMS-7332] – Unnecessary and confusing DEBUG entry on poller.log
  • [NMS-7333] – Switches values retrieved incorrectly in the BSF notification strategy
  • [NMS-7335] – QueryManagerDaoImpl crashes in getNodeServices()
  • [NMS-7359] – Acknowledging alarms from the geo-map is not working
  • [NMS-7360] – Add/Edit notifications takes too much time
  • [NMS-7363] – Update Java in OpenNMS yum repos
  • [NMS-7367] – Octectstring not well stored in strings.properties file
  • [NMS-7368] – RrdDao.getLastFetchValue() throws an exception when using RRDtool
  • [NMS-7381] – Authentication defined in XML collector URLs cannot contain some reserved characters, even if escaped.
  • [NMS-7387] – The hardware inventory scanner doesn't recognize PhysicalClass::cpu(12) for entPhysicalClass
  • [NMS-7391] – Crash on path outage JSP after DAO upgrade

Enhancement

  • [NMS-1595] – header should always contain links for all sections
  • [NMS-2233] – No link back to node after manually unmanaging services
  • [NMS-2359] – Group path outages by critical node
  • [NMS-2582] – Search for nodes by sysObjectID in web UI
  • [NMS-2694] – Modify results JSP to render multiple columns
  • [NMS-5079] – Sort the Path Outages by Critical Path Node
  • [NMS-5085] – Default hrStorageUsed disk space relativeChange threshold only alerts on a sudden _increase of free space_, not a decrease of free space
  • [NMS-5133] – Add ability to search for nodes by SNMP values like Location and Contact
  • [NMS-5182] – Upgrade JasperReports 3.7.6 to most recent version
  • [NMS-5448] – Add link to a node's upstream critical path node in the dependent node's web page
  • [NMS-6508] – Event definitions: Fortinet
  • [NMS-6736] – ImapMonitor does not work with nginx
  • [NMS-7123] – Expose SNMP4J 2.x noGetBulk and allowSnmpV2cInV1 capabilities
  • [NMS-7157] – showNodes.jsp should show nodes in alphabetical order
  • [NMS-7166] – Backup Exec UEI contain "http://" in uei
  • [NMS-7205] – Rename link to configure the Ops Board in the Admin section.
  • [NMS-7206] – Remove "JMX Config Generator Web UI ALPHA" from stable
  • [NMS-7228] – Document that user must be in 'rest', 'provision' or 'admin' role for provision.pl to work
  • [NMS-7247] – Add collection of SNMP MIB2 UDP scalar stats
  • [NMS-7261] – CORS Support
  • [NMS-7278] – Improve the speed of the ReST API and Service Layer for the requisitions' repositories.
  • [NMS-7308] – Enforce selecting a single resource for Custom Resource Performance Reports
  • [NMS-7317] – Rearrange Node/Event/Alarm/Outage links on bootstrap UI
  • [NMS-7384] – Add configuration property for protobuf queue size
  • [NMS-7388] – IpInterfaceScan shouldDetect() method should check for empty string in addition to null string

OUCE 2014 Videos Now Available

January 28th, 2015

The dates are now set for the 2015 OpenNMS Users Conference, but if you can’t wait until September you can now relive the 2014 conference through the magic of YouTube.

You can visit the 2014 conference events calendar and if a video is available it will show up under the “Links” section.

Markus Neumann has been working through the videos and doing his best to improve them, but apologies in advance for the quality of some of them. We’ll attempt to record things better in Fulda.

Building an Open Source PVR: Step Three – Electronic Program Guide

January 26th, 2015

The most frustrating thing about this project was getting the Electronic Program Guide (EPG) to work. Unfortunately, it isn’t easy.

This was one of the things that TiVo excels at doing. You are basically paying for a very up to date program guide. They also offered something called a “Season Pass” which would cause all of the episodes of a particular program to be recorded without having to explicitly select them.

When I got my EyeTV system, this part was a snap. They partner with TV Guide to provide the service, and unlike TiVo’s $14.95/month fee it is a yearly fee of $19.95 (with the first year being included with the unit).

Even my Sony Bravia is able to get over the air EPG information, but I wasn’t able to get that to work with OpenELEC.

The actual EPG configuration occurs in the Tvheadend software. You get a screen like this:

There are three main areas of configuration: the “Internal Grabber”, “Over-the-air Grabbers” and “External Interfaces”.

The internal section displayed a cron job but no module options. None of the OTA grabbers seemed to work, and there wasn’t a module for North America. That left the external grabbers.

I started digging around and found that it really isn’t easy to get this running.

One tool that kept coming up was XMLTV. On the frontend configuration for the Tvheadend client in OpenELEC they even have a section on it:

XMLTV is a number of things, such as a format for representing TV listings, but it is mainly a set of tools “to obtain, manipulate, and search TV Listings”. It contains programs that will connect to an external source to gather EPG data.

Unfortunately, OpenELEC doesn’t ship with it. There is a script called “tv_grab_file” which is used to manage the XMLTV data, but not to actually acquire that data.

For me the easiest solution was to install XMLTV via apt on my home Debian server. It comes with a script called tv_grab_na_dd that can be used to fetch the data.

But I still wasn’t done. I needed a data source. It looks like all the cool kids use Schedules Direct. They are a non-profit that promotes open source software and provides, for a fee, access to EPG information. Since they had a free trial I signed up, configured my tv_grab_na_dd script to access their information, and voilà, I had an XML file with what appeared to be useful information.

I placed that in the webroot of my server, and then configured OpenELEC to point to it. Nothing happened. So I copied the file to the OpenELEC server, modified the client to use the “FILE” method (see screenshot above) and nothing happened.

I finally had to uncheck the XMLTV checkbox under “External Interfaces”. When I did that I finally had something under the “Internal Grabbers” section.

The last chore was to associate the channels I had discovered with the program guide.

Prior to getting all of that to work, the drop down for “EPG Source” had been blank.

So, to summarize my steps:

  1. Get an account at schedulesdirect.org
  2. Install the XMLTV tools somewhere (I used a Debian box)
  3. Configure XMLTV to access your Schedules Direct account
  4. Set up a cron job to periodically grab the updated EPG information and store it in a web root:
     0 1,13 * * * /usr/bin/tv_grab_na_dd --config-file ~/xmltv.conf --days 7 > /secure/html/xmltv.xml
    
  5. On the OpenELEC box, set up a cron to fetch the data:
    0 2,14 * * * wget http://172.20.10.12/xmltv.xml -O /storage/xmltv.xml
    

Whew. So far everything has been working well. You want to be sure not to fetch the data too often as that can overwhelm the Schedules Direct servers. My current seven day XML file is about 10MB.

I went ahead and signed up for a year account for $25, bringing my total to $705.92 (the hardware was $680.92 and the software was, yup, $0). It’s quite possible to shave off about $200 by going with less memory and a smaller SSD (or using an HDD) or if you already have a server to run the Tvheadend backend you could get by with a Raspberry Pi.

My next steps are to play with all the cool add-ons and to try and organize my pictures in a fashion where they would be usable with the system. More fun for me.