Last weekend, OpenNMS-er extraordinare Ronny Trommer was at a conference where he met Jos Poortvliet from Nextcloud. I’ve been following Nextcloud pretty intently since I recognized kindred souls in their desire to create a business that was successful and still 100% open source (and not, for example, fauxpensource). Jos mentioned that Nextcloud was getting a new monitoring API and thought it would be cool if OpenNMS could use it.
Since their API returns the monitoring information as XML, Ronny used the XML Collector to gather the data. Once the data is in OpenNMS, you can graph it, set thresholds, configure notifications, etc.
Available metrics include:
CPU load and memory usage
Number of active users over time
Number of shares in various categories
Server settings like PHP version, database type and size, memory limits and more
Here’s an example of the number of files from a small demo system:
Of course, since OpenNMS is a platform, once the data is in the system you can leverage its integrations with applications such as Grafana:
Some applications will go on and on about how many “plugins” they have. Often, these are little more than scripts that do something simple, like an SNMP GET, but with all the overhead of having to run a shell. To add something like Nextcloud to OpenNMS, it is just a simple matter of configuring a couple of files, but to make that easier a lot of configurations have been added to a git repository. If you want to try out the Nextcloud integration, follow these instructions.
True open source solutions can offer the best feature, performance and value for most companies, but unfortunately there are so few pure open source companies providing them. I applaud Nextcloud and look forward to working with them for years to come.
I am very happy to announce that Chris Manigan has joined the OpenNMS team.
Chris has been using OpenNMS since 2010 when he worked at Towerstream in Rhode Island. He gave us a very nice testimonial for the website, and has a lot of experience with using OpenNMS as scale.
He put that experience to use at Turbine, insuring that their infrastructure could deliver gaming content to users who demand performance. Now he’s going to use that experience to insure that OpenNMS is ready to take on the Internet of Things, for both our internal infrastructure and those of our customers.
I also want to announce that Jesse White, our CTO, and his wife Sara welcomed Charles White into the world early yesterday morning.
Weighing 7 pounds and 11 ounces, he is already writing code in Python and we hope to have him making commits in Java in the next week or so.
It seems like our friends over at Nagios have been watching a little too much election coverage this year, and they’ve updated their “Nagios vs. OpenNMS” document with even more rhetoric and misinformation.
As my three readers may recall, back in 2011 I tore apart the first version of this document. Now they have decided to update it to target our Meridian™ version.
Let’s see how they did (please look at it and follow along as it is quite amusing).
The first misleading bit is the opening paragraph with the phrase “most widely used open-source monitoring project in the world”. Now, granted, they do indicate that means “Nagios Core” but it seems a little disingenuous since what they are selling is Nagios XI, which is much different.
Nagios XI is not open source. It is published under the “Nagios Open Software License” which is about as proprietary as they get. I’m not even sure why the word “open” was added, except to further mislead people into thinking it is open source. The license contains clauses like “The Software may not be Forked” and “The Software may only be used in conjunction with products, projects, and other software distributed by the Company.” Think about it, you can’t even integrate Nagios XI with, say, a home grown trouble ticketing system without violating the license. Doesn’t sound very “open” at all. OpenNMS Meridian is published under the AGPLv3, or a similar proprietary license should your organization have an issue with the AGPL. You don’t have that choice with Nagios XI.
Next, let’s check out the price. The OpenNMS Group has always published its prices on-line. One instance of Meridian, which includes support in the form of access to our “Connect” community, is $6,000. They have it listed as $25,995, which is the price should you choose the much more intensive “Prime” support option. I’m not sure why they didn’t just choose our most expensive product, Ultra Support with the 24×7 option, to make them seem even better.
Also, note the fine print “Price based on one instance of XI with 220 nodes/devices”. There is no device limit with OpenNMS Meridian. So let’s be clear, for $6000 you get access to the Meridian software under an open source license versus $5000 to monitor 220 nodes with extreme limitations on your rights.
Our smaller customers tend to have around 2000 devices, which means to manage that with Nagios XI you would need roughly ten instances costing nearly $50,000 (using the math presented in this document). And from the experience we’ve heard with customers coming to us from Nagios, the reason it is limited to so few nodes is that you probably can’t run much more on a single instance of Nagios XI. Compare that to OpenNMS where we have customers with over 100,000 devices in a single instance (and they’ve been running it for years).
We also price OpenNMS as a platform. You get everything: trouble-ticketing integration, graphing, reporting, etc. in one application. It looks like Nagios has decided to nickel and dime you for logs, etc. and a thing called “Nagios Fusion” which you’ll need to manage your growing number of Nagios instances since it won’t natively scale. And remember, due to the license you are forbidden from using the software with your own tools.
I especially had to laugh at the “You Speak, We Listen” part. If you have a feature or change you need, if you ask nicely they might make it for you. With OpenNMS Meridian you are free to make any changes you need since it is 100% open source, and with our open issue tracker we address dozens of user requests each point release.
Finally, there is the feature comparison, which at a minimum is misleading and is often just blatantly false. Almost every feature marked as lacking in Meridian exists, and at a level far beyond what Nagios XI can provide. Seriously, is it really objective to state that OpenNMS doesn’t support Nagvis, a specific tool that even has “Nagios” in the name?
I had to laugh at the hubris. They obviously didn’t Google “opennms nagvis“, because, guess what? There has been an OpenNMS Nagvis integration for some time now, contributed by our community. Just in case you were wondering, we have an integration with Network Weathermap as well.
Nagios is just another proprietary software product that wants to lock you into its ecosystem, and this is just a shameful attempt to monetize an application that is long past its prime. Heck, it was the inability of the Nagios leadership to get along with others that resulted in the very popular Icinga fork, and with it Nagios lost a lot of contribution that helped make up its “Thousands of Free Add-Ons” (and the way Nagios took over the community lead plug-in site was also poorly handled). Plus, many of those add-ons won’t scale in an enterprise environment, which probably lead to the 220 device limit.
Compare that to OpenNMS. We not only want to encourage you integrate with other products, we do a lot of it for you. OpenNMS has great graphing, but we also created the first third party plug-in for Grafana. When it comes to mapping, OpenNMS is on the leading edge, with a focus on various topology views that can ultimately handle millions of devices in a fashion that is actually usable. Need to see a Layer 2 topology? Choose the “enhanced linkd view”. Run VMware and Vcenter? It is simple to import all of your machines and see them in a view that shows hosts, guests and network storage. Plus the unique ability to focus on just those devices of interest allows you to use a map with hundreds of thousands if not millions of nodes.
Compare that to the Nagios map screenshot where it looks like “localhost” is having some issues. Oh no, not localhost! That’s like, all of my machines.
As for “Business Process Intelligence” I’ve been told that the Nagios XI version is like our Business Service Monitor “Except BSM is more featureful, and has a significantly better UI/UX”. Need real Business Intelligence? OpenNMS has Red Hat Drools support, the open source leader, built right into the product.
We also support integration with popular Trouble Ticketing systems such as Request Tracker, Jira, OTRS and Remedy. And the kicker is that you can also run any Nagios check script natively in OpenNMS using the “System Execute Monitor“, but once you get used to the OpenNMS platform, why would you?
I’m not really sure why Nagios goes out of its way to spread fear, uncertainty and doubt about OpenNMS. We rarely compete in the same markets. I’m sure that Sunrise Community Banks get their money’s worth from Nagios, and for companies like NRS Small Business Solutions, Nagios might be a good fit. But if you have enterprise and carrier-level requirements, there is no way Nagios will work for you in the long term.
When a company does something like this to mislead, from wrong information about our product to using terms like “open” when they mean “closed”, it shows you what they think of their competition. What does it say about what they think about their customers?
I figured I’d better get this review of the latest Bad Voltage out before the next one drops this week (sigh).
The episode clocks in at a svelte 51 minutes, and mainly focuses on two segments, one on Pokémon Go and the other on streaming music.
As the guys point out, unless you’ve been living in a cave you have probably heard of Pokémon Go (and even people in caves are playing it). It is the augmented reality (AR) game from Niantic based on the characters made popular by Nintendo.
It has also had its share of controversy, with stories of people being injured while playing and in my own neck of the woods a row over people being fined for visiting the grave of a friend.
The game from Niantic that proceeded it was Ingress, which I’ve talked about before and I showed it to the team when they did their show in Fulda. Ingress can be pretty addictive, so I was set on not playing Pokémon at all. I didn’t really need another time sink in my life.
But a couple of things happened. First, I was introduced to this short from South Park that parodied Pokémon with “Chinpokomon”. I laughed since “chinpoko” is a rude Japanese word, so of course it was one of the few words of Japanese I know. I was determined to be “chinpokomon” on Pokémon Go.
I installed the app the weekend it was released and tried to register with that “trainer” name. It wasn’t to be. I tried every variation I could think of but it wouldn’t accept it. I’m not sure if it was because they were disallowing names with anything like Pokémon in them, or that, by that time, some of the 10 million people who had downloaded it had had the same idea. So I uninstalled the app and forgot about it.
Flash forward a week or so and not only did Andrea start playing, a bunch of people at the office did too, so I decided to check it out again.
I’ll post a full review soon, but I have a few thoughts to share here. Ingress suffers from three main issues: GPS “spoofing” where people fake their location, people playing multiple accounts and the in-game chat system which is often used to heap abuse on other players.
Pokémon Go is much nicer in that there is no chat system and you can’t trade items (making multiple accounts somewhat useless). That may change in the future but for now you can play in a rather friendly environment. Even in battles your Pokémon don’t die, they “faint” and you get them back. There is still an issue with spoofing, which is how many players access the game in countries that don’t have it.
The problem with Pokémon Go is the gameplay gets old, fast. The variety of game items makes it an order of magnitude more complex than Ingress, but I’m not really into collecting a 100% version of each Pokémon. I do like getting new ones (catch them all) but Niantic has made that pretty difficult. There is a part of the screen that will show you nearby Pokémon but you don’t get a clue as to where they are. There was a website called Pokévision that reverse engineered the API and would display them on a map, and I used that extensively to get uniques. I got a lot of exercise running around the UMN campus during Dev-Jam to get one I needed. I was averaging 25,000 steps a day according to my watch, but since Pokévision has been shut down I am less eager to run around in circles hoping a Pokémon will pop up.
In a couple of weeks of casual play I’ve made it to Level 24 and caught 105 Pokémon (I’ve seen 106, damn Wheezing) and my interest is starting to wane (although the Tauros is my favorite, ‘natch). I’ll probably hit Level 25 (where you get access to a new item) and then cut back drastically. Which I think is going to be the major problem with Pokémon Go.
We often eat at this one restaurant in Pittsboro every Friday night, and two weeks ago one young lady who works there was really into the game. This past Friday I asked her how she was doing with it, and she said she’d stopped playing because “it was boring”.
Don’t get me wrong, Niantic has a hit on its hands, I just don’t think they will sustain the level of interest they had a launch.
The guys made some good points about it. Jeremy noted that while it is called “AR” it is really nothing more than taking the video feed from the camera and superimposing Pokémon pictures on it. It does nothing for scale or distance, for example.
Bryan detailed some interesting history that I didn’t know concerning the origin of Niantic. It grew out of a spooky company called Keyhole with designs on tracking and influencing people’s habits (although they are more well known for being the technology behind Google Maps and Google Earth). Now, as an Ingress player I’ve already opted in to allow Google to track my location, and it came in handy when Jeremy roofied me at Oktoberfest and I wandered around Munich for a few hours. I had a record of where I had gone.
On a side note, Bryan went on to state that on Android you can’t control access to the microphone. Now, I’ll agree that the only way to be sure would be to have a hardware kill switch installed so you could disable the microphone entirely, but I run a version of AOSP called OmniROM and I seem to have the choice to limit access to the microphone on a “per app” basis.
Not sure if that isn’t available on all Android releases but it seems to work on mine. Of course, many apps use Google Play Services so there’s that.
The second segment was on streaming music services. I don’t stream music so I don’t really have much to add, but I have heard that Pandora uses OpenNMS so I’m a big fan. (grin)
I do sometimes listen to SiriusXM at my desk. We have it in our cars so I have the option to stream it as well. I was listening to AltNation when I heard a track called “Loud(y)” by Lewis Del Mar. I found it on Soundcloud with a number of other tracks by them, and after it played them all it continued with similar artists. I really liked the mix (which included songs like “Thrill” by CZAR) and ended up listening to it for a couple of days. What I liked most about it is that all of them were from artists new to me. I buy the music I like and so I tend not to get much from streaming, and I also tend to listen when being connected to a network is not feasible (such as in a car or a plane), but I am considering the service from Soundcloud that let’s you listen offline (ironically called Soundcloud Go).
Which brings me to another sore point with me. The guys brought up vinyl. As many people know, vinyl is making a comeback, but dammit, it is just some sort of hipster thing since almost all music today is digitally mastered. You probably haven’t listened to a commercial record that didn’t go through Pro Tools, so when I hear “oh, but vinyl is so much richer and warmer” I have to call bullshit. Get a FLAC version of the song and you can’t get any better. Sure, you may need to upgrade your sound card and your speakers, but when, say, I get a FLAC master track from MC Frontalot it is the one that is sitting on his computer where he created it. It contains all of the information captured, and I can’t see how that gets improved by sticking it on a vinyl record whose sound quality starts to decay the moment you play it for the first time.
The Outro for the episode was kind of cool, as the guys talked about old gadgets and things like BBS’s. I can remember being in Tokyo when the Sharp Zaurus was introduced and I scoured the city looking for one in English. It was a cool device and I also liked the name. And the show brought back memories of having flame wars on a WWIV BBS system over a 2400 baud modem. The host (a high school kid who worked as a bag boy at a grocery store to pay the phone bill) could only afford a single phone line so you had to take turns. It made flame wars kind of fun – once you got in, you’d post your rant, log off, wait 30 minutes and then log back in to see if there was a reply.
All in all a nice, light episode. Nothing too heavy, kind of a like a sorbet. Hoping they bring back the meat this week.
OpenNMS is headquartered in the idyllic small town of Pittsboro, NC, sometimes just called “PBO”. Since a number of people who come to Dev-Jam travel a fair distance, we’ve started a tradition of a “mini Dev-Jam” the week after, hosted at OpenNMS HQ.
This is much more focused on the work of The OpenNMS Group, but it is still a lot of fun. Last night as a team building exercise we decided to try an “escape” room.
This is a a relatively new thing where a group of people get put in a room and they have a certain amount of time to figure out puzzles and escape. Jessica set us up with Cipher Escape in their “Geek Room” which was the only one that could accommodate 11 of us.
It’s a lot of fun. For our experience we were lead into about a 15×15 room and given the following backstory: you are watching your neighbors cat while they are on vacation and after you feed her you realize you are locked in their house. You have 60 minutes to escape.
One thing I thought was funny was that the room was dotted with little pink stickers and we were told that these indicate things that don’t need to be manipulated (e.g. there was a picture frame that when you turned it over you would see the stickers, which meant you weren’t supposed to take it apart). I can only imagine the beta testing that went into determining where to put the stickers (our hostess specifically mentioned that you didn’t need to take the legs off the furniture).
To tell anything more would spoil it, but I was extremely proud that the team escaped with over 10 minutes to spare (we missed the best time by ten minutes, so it wasn’t close, but we did beat a team from Cisco that didn’t escape at all).
It was a ton of fun, and I’d put this team up against any challenge.
Afterward, most of us went out for sushi at Waraji. I’ve known the owner Masatoshi Tsujimura for almost 30 years, and even though they were packed they were able to set us up with a tatami room.
It’s a bit out of the way for me to visit often, so I was happy to have an excuse for a victory celebration.
Chandra has been working on adding provisioning detectors to the Minion:
Deepak and Pavan, who work for a large electronic medical records company, discuss how they are using OpenNMS at scale:
Seth has been managing a lot of that work, which is currently focused on syslog, and he did a presentation on new syslog parsing functionality:
Alejandro presented some awesome improvements to the UI:
Markus has been working on project Atlas, which includes major improvements to OpenNMS maps. Here he demonstrates the integration of the geographical map with the topology map:
More UI enhancements were offered by Christian who added trend lines to the OpenNMS home page:
Ronny talked about his ideas for making device configurations more modular and managing that with git:
And he has also been creating reusable Docker containers with OpenNMS:
One project I found extra exciting was “Underling” which is an instance of Minion written in Go. This makes it incredibly small (about 6MB) which should allow the Minion to run on very inexpensive hardware.
Dev-Jam is made up of two main groups of people: those who work on OpenNMS full time and those who don’t. For those who work on OpenNMS full time, we try to depart from the day to day running of the project to both try new things and have fun. Think of it as “special projects week”.
Since OpenNMS is aiming to be a platform for the Internet of Things, this tends to involve a lot of electronics.
I decided to take some time out to further explore the Virtual Reality provided by Google Cardboard. I played with it last Dev-Jam, but I bought a nice headset from Homido since the Cardboard experience with the actual cardboard holder, while novel, was a little bit wanting.
The downside is that it doesn’t have the little magnet thingie that acts as a mouse click. Most people using the Homido tend to pair some other controller to their Android device in order to navigate, and since I have a PS3 (that I mainly use to play Blu-ray disks) I had a Sixaxis controller I could use. I had to buy an app in order to deploy a driver on my Nexus 6P that would work with the Sixaxis, and after a bit of tinkering I got it to work (note that it disables the regular Bluetooth driver when you run it).
I configured the “X” button to act as a mouse click, and pretty soon I was able to move about the Google Cardboard demos. The Homido fits well and the image is good, but it does allow some light to bleed around the edges in so it would work best in a dim or dark room.
I then went off to find some apps. This is not a field that a lot of developers have explored, and most of them are pretty passive. While this can work (check out the creepy “Sisters”) I wanted something more along the lines of what I experienced with the Samsung Gear VR, which includes immersive games. I found one called Hardcode VR that was fun, sort of a platformer along the lines of Portal. The controller worked out of the box exactly like you would expect it to: the right joystick was used for looking around and the left one for moving. I did get a slight headache after playing it for awhile, though, so I think that for the time being mobile device-based VR is still a novelty.
My experiment did amuse some of the folks at the conference, and Ronny made this comparison:
I am always humbled by the people who give a week of their lives up to come to Dev-Jam, and even more so since DJ was away from his wife on his birthday. We did make sure he had a cake, though.
The cake was from Salty Tart and it was mighty tasty.
It’s hard to believe this year’s Dev-Jam is half over. After months of planning it seems to go by so fast.
One of the goals I had this week was to understand more about the OpenNMS Documentation Project. For years I’ve been saying that OpenNMS documentation sucks like most open source projects, but I can’t say that any more. It has actually become quite mature. There is a detailed installation guide, a users guide, and administrators guide and a guide for developers. Each release the docs are compiled right alongside the code, and it even rates its own section on the new website.
It’s written in AsciiDoc, and all of the documentation is version controlled and kept in git.
Ronny Trommer is one of the leads on the documentation project, and I asked him to spend some time with me to explain how everything is organized.
Of the four main guides, the installation guide is almost complete. Everything else is constantly improving, with the user guide aimed at people working through the GUI and the administration guide is more focused on configuration. For example, the discussion of the path outage feature is in the users guide but how to turn it on is in the admin guide.
There is even something for everyone in the developers guide (I am the first to state I am not a developer). One section details the style rules for documentation, in great detail. For example, in order to manage changes, each sentence should be on a single line. That way a small change to, say, a misspelled word, doesn’t cause a huge diff. Also, we are limited as to the types of images we can display, so people are encouraged to upload the raw “source” image as well as an exported one to save time in the future should someone want to edit it.
It is really well done and now I’m eager to start contributing.
Speaking of well done, Jonathan has figured out what is keeping OpenNMS from using the latest version of OTRS (and he’s sent a patch over to them) and Jesse showed me some amazing work he’s done on the Minion code.
We’ve been struggling to figure out how to implement the Minion code since we want to be able to run it on tiny machines like the Raspberry Pi, but since OpenNMS is written in Java there is a lot of overhead to using that language on these smaller systems. He re-wrote it in Go and then uploaded it to a device on my home network. At only 5.6MB it’s tiny, and yet it was able to do discovery as well as data collection (including NRTG). Sheer awesomeness.
Wednesday was also Twins night.
For several years now we’ve been going as a group to see the Minnesota Twins baseball team play at Target Field. It’s a lot of fun, although this year the Germans decided that they’d had enough of baseball and spent the time wandering around downtown Minneapolis.
At first I thought they had the right idea, as the Braves went up 4 to 0 in the first and by the top of the fourth were leading 7 to 0. However, the Twins rallied and made it interesting, although they did end up losing 9 to 7.
By Day Two people have settled into a rhythm. Get up, eat breakfast, start hacking on OpenNMS. I tend to start my day with these blog posts.
It’s nice to have most of the team together. Remember, OpenNMS is over 15 years old so there is a lot of different technology in the monitoring platform. I think David counted 18 different libraries and tools in the GUI alone, so there was a meeting held to discuss cleaning that up and settling on a much smaller set moving forward.
In any case ReST will play a huge role. OpenNMS Compass is built entirely on ReST, and so the next generation GUI will do the same. It makes integrating with OpenNMS simple, as Antonio demonstrated in a provisioning dashboard he wrote for one of his customers in Italy.
They needed an easier way to manage their ten thousand plus devices, so he was able to use the ReST interface to build out exactly what they wanted. And of course the source is open.
Several years ago we started a tradition of having a local restaurant, Brasa, cater dinner one night. This year it was Tuesday, and it is always the best meal of the week.
As we were getting ready to eat, Alex Hoogerhuis, a big supporter of OpenNMS who lives in Norway, decided to join us via our Double Robotics robot, Ulfbot. It worked flawlessly, and he was the best first time driver we’ve had. Ben, Jeff and Jonathan joined him for a picture.
We like using the Yudof Hall Club Room for Dev-Jam for a number of reasons, one includes the big patio overlooking the river with picnic tables. Alex was able to drive around and spend some time with the rest of the team, although we had to lift him up to see over the wall to the Mississippi (we also had to carry him in when the wind picked up – heh).
After dinner people kept working (DJ was up until nearly 2am chasing a bug) but we also took a break to watch Deadpool. It’s why “Dev-Jam” rhymes with “fun”.
Last year I wanted to do something special for the team to commemorate the release of OpenNMS Meridian.
Since all the cool kids in Silicon Valley have access to a classic arcade machine, I decided to buy one for the office. I did a lot of research and I settled on one from X-Arcade.
The main reasons were that it looked well-made but it also included all of my favorites, including Pac-Man, Galaga and Tempest.
The final piece that sold me on it was the ability to add your own graphics. I went to Jessica, our Graphic Designer, and she put together this wonderful graphic done in the classic eight-bit “platformer” style and featuring all the employees.
Ulf took the role of Donkey Kong, and here is the picture meant to represent me:
The “Tank Stick” controls are solid and responsive, although I did end up adding a spinner since none of the controls really worked for Tempest.
When you order one of these things, they stress that you need to make sure it arrives safely. Seriously, like four times, in big bold letters, they state you should check the machine on delivery.
I was going to be out of town when it arrived, so I made sure to tell the person checking in the delivery to make sure it was okay (i.e. take it out of the box).
They didn’t (the box looked “fine”) and so we ended up with this:
Outside of that, everything arrived in working order. You get a small Dell desktop running Windows with the software pre-installed, but you also get CDs with all the games that are included with the system. It’s a little bit of a pain to set up since the instructions are a little vague, but after about an hour or so I had it up and running.
Anyway, it is real fun to play. It supports MAME games, Sega games, Atari 2600 games and even that short lived laserdisc franchise “Dragon’s Lair”. You can copy other games to the system if you have them, although scrolling through the menu can get a bit tiring if you have a long list of titles.
We had an issue with the CRT about 11 months after buying the system. I came back from a business trip to find the thing dark (it never goes dark, if the computer is hung for some reason you’ll still see a “no signal” graphic on the monitor”). Turns out the CRT had died, but they sent us a replacement under warranty and hassle free. It took about an hour to replace (those instructions were pretty detailed) and it worked better than ever afterward.
This motivated me to consider fixing the top. When we had the system apart to replace the monitor, I noticed that the top was a) the only thing broken and b) held on with eight screws. I contacted them about a replacement piece and to my surprise it arrived two days later – no charge.
The only issue I have remaining with the system is the fact that it is Windows-based. This seems to be the perfect application for a small solid-state Linux box, but I haven’t had the time to investigate a migration. Instead I just turned off or removed as much software as I could (all the Dell Update stuff kept popping up in the middle of playing a game) and so far so good.
I am very happy with the product and extremely happy with the company behind it. If you are in the market for such a cabinet, please check them out.