♫ Don’t Call It a Comeback ♫

Welcome to 2016. My year started out with an invitation to join the AARP. (sigh)

As my three readers know, when it comes to this business of open source we are pretty much making things up as we go along. We are lead by our business plan of “spend less money than you earn” and our mission statement of “help customers, have fun, make money” but the rest is pretty fluid.

In 2013 we mixed things up and tried a more “traditional” start up path by seeking out investment and spending more money than we had. It didn’t work out so well.

Thus 2014 was more of a rebuilding year as we tried to move the focus back to our roots. It paid off, as 2015 was a very good year. We had record gross revenues, and although we didn’t make much money on the bottom line, it was positive once again. At the moment we are still investing in the company and the project so pretty much every extra dollar goes into growth.

And we had a lot of growth. The decision to split OpenNMS into Meridian and Horizon paid off in three major Horizon releases. Horizon 17 was an especially large and important release as it brought in the Newts integration. At the moment we are working with it on a customer site using a ScyllaDB cluster capable of supporting 75K inserts per second. The technologies introduced in 2015 will make it in to Meridian 2016, due in the spring, and it should solidify OpenNMS as a platform that can really scale.

In 2015 we also received orders from two of the Fortune 5 companies. I’ll leave it as an exercise to the reader to guess which two and you have a 1 in 16 shot at getting it right (grin). The fact that companies that can choose, literally, any technology they want yet they choose OpenNMS speaks volumes.

One of these days we’re going to have to figure out a way to talk about our customers by name, since they are all so cool. We are working on it, but it is surprisingly difficult to get permission to publicly post that information. Above all we respect our clients’ privacy.

I have high expectations for 2016 and the power of the Open Source Way. Thanks to everyone who has supported us over the last decade and more, and we just hope you find our efforts provide some value.

Happy New Year.

The Inverter: Episode 56 – Moon Pigeons

A bit more navel-gazing than normal, the latest Bad Voltage clocks in at nearly 90 minutes. Whew.

It was nice that Jeremy was back, and I found it hilarious that in the past two weeks he hadn’t bothered to listen to the show he missed. Considering the fact that that show was one of the shortest of the year, I guess we know who is doing all the talking. Or, as Jono points out, Jeremy is the one who clutters up everything with facts. I thought Aq’s audio was a bit off at the beginning (it sounded like he was down a well) but it seemed to get better as the show progressed.

The first segment concerned the failure of open source mobile projects like FireOS and Jolla. I thought this bit ran long, but there were some gems to be had. Bryan was talking about running Linux on tiny mobile devices for which he was mercilessly teased, but I had to agree with him. While I would never want to be forced to run LibreOffice exclusively on a device the size of my Nexus 6, sometimes it would be nice to be able to do quick edits on the go. I hate using ssh on my handy, but when I need it, I need it.

Jono points out that a lot of people tie their personal identity to their mobile devices. A lot of the way people interact with each other these days is through SMS, Facebook and Instagram, and the constant use of an iPhone or an Android phone can cause people to get very attached to them. Any new challenger to the iOS/Android juggernaut has to not only support those apps, they have to overcome the fact that people (to some degree myself included) have strong ties to their technology choices. Unlike how OpenStack disrupted the nascent cloud market, it seems to be hard for open source projects to do the same in the mobile arena, and I had to laugh when Aq suggested replacing “disrupt” with “f*ck up”.

It was pointed out that if companies like Microsoft who can throw tens of billions of dollars at a market can only garner a little over 2% market share, it is doubtful that a new open source project would have better success.

On a side note, I just spend a few days up on the Eastern Shore of Maryland and the client liked to use Surface Pro tablets. I got to see them in action, and they are pretty amazing – for many they could be a laptop replacement just like the ads suggest. But I doubt that Microsoft is going to dent iPad sales just because of the brand Apple has built. Often it is not the superior technology that wins.

The second segment was a review of a couple of security cameras that Jeremy was trying out: the Arlo by Netgear and the Guardian DCS-2630L by D-link.

I have a couple of cameras at my place, although I don’t have the budget of these guys. Inside buildings I have the D-Link (DCS-5010L) which is a great little camera. It does pan and tilt and works in low light conditions. Since it wouldn’t do well outside, I have the Agasio A602W which is no longer available.

Why neither of them are totally wireless (i.e. you have to plug them in) they are both supported via open source tools like Zoneminder, although with the purchase of my Synology box I just use the Surveillance Station app that comes with that. It can continuously record, record only when motion is detected, etc., and you can set how much video to store per camera. I really dislike the thought of video from my house going “to the cloud” so I love the fact that I can control where it goes, and Synology has a mobile app that lets me access the video whenever I want it (plus, my DSL upstream would suck for constantly uploading video). The Arlo does seem to be compatible with the Surveillance Station, so as Jeremy’s pick I might have to try one out.

[UPDATE: WCCFTech is full of crap amd the Arlo is not compatible with the Surveillance Station]

One last comment from Aq brings up a coming issue with the Internet of Things. All of these toys should play nice together, but often they don’t. He calls it “IoT lockout” but I like “Internet of Silos” (i.e. Z-Wave vs. ZigBee). I do like how most of these cameras have a web interfaces where the video stream can be accessed by a URL, which means third party tools can access and integrate with them, but I can expect vendors to start locking stuff like that down to force people into their own particular cloud infrastructure.

The third segment concerned the “Luna Ring” – an idea started a few years ago by a Japanese engineering firm to ring the moon with solar panels and beam the energy back to Earth via microwave and lasers. I did laugh out loud at Jono’s comment that the name sounds like a contraceptive device.

Odd names aside, I think this is both a cool idea and one that will never happen. The guys point out some of the obvious flaws, but I can’t help but think of the resistance the world would have to high powered beams of light focused on points on the Earth. Sounds like something a James Bond villain would think up.

I did get embarrassed for my home state when it was brought up that the town of Woodland, NC, recently voted down zoning for a solar farm. The click-bait reason given was that one citizen pointed out that solar farms would “suck up all the energy from the sun”.

(sigh)

The actual story is a little more involved. There are already three solar farms in the area surrounding a local substation, so the town is obviously not anti-solar. Small towns like Woodland are getting hit hard with the decline of manufacturing, so I can see the residents there being frightened and looking for a scapegoat. Still, I had to be embarrassed by some of the comments, and it is obvious our educational system needs some work (but that’s a totally different topic).

One person commented that the solar panels were killing the plants. That reminded me of a project my friend Lyle produced called “solar double-cropping“.

As I write this, it is over 72F (22C) on Christmas Eve, the hottest Christmas Eve on record. Our climate is changing and plants that used to thrive are having issues. The idea of solar double-cropping is to use shade from solar panels to help those plants while generating electricity.

And yes, they came up with it in North Carolina.

The final segment was a “year in review”. The guys lamented the lack of innovation, but there were some good things, too. As a “freetard” (someone who runs open source software almost exclusively) I had to agree with Aq that those of us who feel this way are having to compromise less and less as the open source options get better (although I still have to tease him about the compromise he made for his closed source One Plus X phone).

We saw high definitions pictures of Pluto. I’m still amazed that nine years ago, we as a civilization chucked a bunch of metal up into space and it managed to rendezvous with a planetoid without major issues. We lobbed another piece of metal at a comet, and while not as successful it was still quite a feat.

In entertainment, the amazing Mr. Robot television series offered us a portrayal of hacking that wasn’t totally made up.

Speaking of entertainment, the show closed with a reminder that Live Voltage will be happening at next month’s SCaLE conference. If you can, you should go, and they are still accepting ideas for “upSCALE” talks. From their latest e-mail:

UpSCALE Talks: There is still room for an UpSCALE Talk or two – UpSCALE Talks are held in the style of Ignite presentations offered at various O’Reilly-sponsored events where participants are given five minutes with 20 automatically-advancing slides. Those interested in submitting an UpSCALE Talk can submit through the SCALE CFP system – https://www.socallinuxexpo.org scale/14x/cfp – and mark your talk with the UpSCALE tag.

So that’s it for 2015. I’m off to put on some shorts and sunscreen. ♫ Oh the weather outside is frightful … ♫

Mint 17.3 (Rosa) on the Dell XPS 13 (9343)

I’m a big fan of the Dell XPS 13. It is the first laptop I’ve felt an emotional attachment to since my first Powerbook. The only issue is that I have not been able to run my distro of choice, Linux Mint, due to severe issues with the trackpad.

Mint on XPS

With the release of Mint 17.3 (Rosa) I decided to give it another shot. I burned the image to a USB stick and booted to it, and the trackpad issues were gone.

Yay!

So I based my system and installed Mint. I did have to use a wired network connection since the Broadcom drivers don’t seem to work on install (there is probably a way around that) but once installed they were easy to enable.

One thing I liked about Mint when I had installed it previously was that it recognized the HiDPI screen of the XPS right away. Even though the “What’s New” page says that HiDPI detection has been improved in 17.3, I found that it had regressed and I needed to squint to get the O/S installed. Once I did, however, I was able to go to Settings -> General and switch to HiDPI mode and everything was fine.

Mint HiDPI Setting

Now, the XPS hardware is so new that it really requires a 4.2 kernel. I decided to install it. No biggie, since I had to do it with Ubuntu 15.04, but I’ll be happy when Mint 18 comes out and it is supported natively (you have to do some apt magic to ignore kernel updates). Once installed, my wireless connection failed to work, and that’s where the fun began.

Usually, all I had to do was reinstall the bcmwl-kernel-source package, but this kept failing with an error. I even built the package from source but while it built just fine, DKMS would fail when installing it, complaining about “-fstack-protector-strong”. Turns out this was added in gcc 4.9 and Mint 17.3 ships with gcc 4.8.

(sigh)

Anyway, not hard to fix. I ran the following commands:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get dist-upgrade
sudo apt-get install gcc-4.9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 70

and now gcc 4.9 was my default compiler. I then rebuilt and installed the bcmwl-kernel-source pacakge and things were golden.

$ modinfo wl
filename:       /lib/modules/4.2.6-040206-generic/updates/wl.ko
license:        MIXED/Proprietary
srcversion:     D46E6565F844EFBD46CE0FC
alias:          pci:v*d*sv*sd*bc02sc80i*
depends:        cfg80211
vermagic:       4.2.6-040206-generic SMP mod_unload modversions 
parm:           passivemode:int
parm:           wl_txq_thresh:int
parm:           oneonly:int
parm:           piomode:int
parm:           instance_base:int
parm:           nompc:int
parm:           intf_name:string

Just like with Ubuntu Gnome, I did have to manually install the bluetooth driver, but at the moment everything seems to work: wireless, bluetooth, the touchscreen, the clickpad, sleep, backlit keyboard, etc.

Now I use a desktop as my primary machine, so I haven’t really taken the XPS through its paces, but I’m scheduled to travel soon and I’ll be sure to post if I have any issues. I did enable the screensaver and once when I came back to the machine my mouse pointer was gone (the mouse still worked, you just couldn’t see the pointer) and I was unable to fix it without a restart (I tried the suggestions in Google but it didn’t work). For now I’ve just disabled the screensaver.

All in all, great work from the Mint team, and while I actually enjoyed my time with Ubuntu Gnome I’m happy to be back. Looking forward to Mint 18 in the Spring which should require less effort to run on the XPS with built-in support for the 4 series kernel.

OpenNMS Horizon 17 Released

I am extremely happy to announce the availability of OpenNMS Horizon 17. This marks the fourth major release of OpenNMS in a little over a year, and I’m extremely proud of the team for moving the project so far forward so quickly.

There is a lot in this release. One of the major things is support for a new storage backend based on the Newts project. This will enable OpenNMS to basically store unlimited amounts of time-series data. The only thing missing, which should completed soon, is a way to convert all of your old RRD-based data to Newts. Since it will take people awhile to get a Newts/Cassandra instance set up, we didn’t want to hold the rest of the release until this was done. If you are installing OpenNMS from scratch and don’t have any legacy data, the Newts integration is ready to go now.

The team is also making great strides in improving the documentation. There is a better version of the Release Notes there.

Horizon 17 will form the basis for Meridian 2016, which we expect in early spring. The next Horizon release will contain the completed Minion functionality, which adds the ability to distribute OpenNMS so that, along with Newts, OpenNMS will have nearly limitless scalability.

Not bad for a free software product, eh? Remember you can always play with the latest and greatest of any OpenNMS development branch just by installing the desired repository.

Anyway, enjoy, and I’ll be sure to post when the RRD converter is available.

Bug

  • [NMS-5613] – odd index "ifservicves_ipinterfaceid_idx" in database – typo?
  • [NMS-5946] – JMX Config Tool CLI is not packaged correctly
  • [NMS-6012] – Statsd randomly looks for storeByForeignSource rrds
  • [NMS-6478] – 'Overall Service Availability' bad info in case of nodeDown / nodeUp transition
  • [NMS-6493] – Running online report "Response Time Summary for node" produces Unexpected Error
  • [NMS-6555] – Outdated Quartz URL in provisiond-configuration.xml file
  • [NMS-6803] – Not evaluating threshold for data collected by HttpCollector
  • [NMS-6927] – test failure: org.opennms.web.alarm.filter.AlarmRepositoryFilterTest
  • [NMS-6942] – test failure: org.opennms.web.svclayer.DefaultOutageServiceIntegrationTest
  • [NMS-6944] – When building the "Early Morning Report" I get a "null" dataset argument Exception.
  • [NMS-7000] – Early Morning Report will not run correctly without any nodes in OpenNMS
  • [NMS-7001] – Availability by node report needs a "No Data for Report" Section
  • [NMS-7024] – Event Translator cant translate events with update-field data present
  • [NMS-7095] – Topology Map does not show selected focus in IE
  • [NMS-7254] – MigratorTest fails on two of the 3 tests.
  • [NMS-7407] – Inconsistent naming in Admin/System Information
  • [NMS-7411] – Fonts are too small in link detail page
  • [NMS-7417] – Fix header and list layout glitches in the WebUI
  • [NMS-7459] – Dashboard node status shows wrong service count
  • [NMS-7516] – XML Collector is not working as expected for node-level resources
  • [NMS-7600] – build failure in opennms-doc/guide-doc on FreeBSD
  • [NMS-7649] – etc folder still contains references to capsd
  • [NMS-7667] – Vaadin dashboard meaning of yellow in the surveillance view
  • [NMS-7679] – Audiocodes.events.xml overrides RMON.events.xml
  • [NMS-7680] – JMX Configuration Generator admin page fails
  • [NMS-7693] – Example Drools rules imports incorrect classes
  • [NMS-7695] – Logging not initialized but used on Drools Rule files.
  • [NMS-7702] – Problems on graphs for 10 gigabit interface
  • [NMS-7703] – Database Report – Statement correction
  • [NMS-7709] – Building OpenNMS results in a NullPointerException on module "container/features"
  • [NMS-7723] – PSQLException: column "nodeid" does not exist when using manage/unmanage services
  • [NMS-7728] – Add support for jrrd2
  • [NMS-7729] – Log messages for the Correlation Engine appear in manager.log
  • [NMS-7736] – bug in EventBuilder method setParam()
  • [NMS-7739] – Unit tests fail for loading data collection
  • [NMS-7748] – SeleniumMonitor with PhantomJS driver needs gson JAR
  • [NMS-7750] – Cannot edit some Asset Info fields
  • [NMS-7755] – c.m.v.a.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@59804d53 — APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
  • [NMS-7762] – noSuchObject duplicates links on topology map
  • [NMS-7764] – Error when you drop sequence vulnnxtid
  • [NMS-7766] – Incorrect unit divisor in LM-SENSORS-MIB graph definitions
  • [NMS-7770] – HttpRemotingContextTest is an integration test and needs to be renamed as such
  • [NMS-7771] – Fix unit tests to run also on non-US locale systems.
  • [NMS-7772] – JMX Configuration Generator (webUI) is not working anymore
  • [NMS-7777] – node detail page failure
  • [NMS-7778] – Measurements ReST API broken in develop (CXF)
  • [NMS-7785] – OSGi-based Web Modules Not Accessible
  • [NMS-7791] – OSGi-based web applications are unaccesible
  • [NMS-7794] – Cannot load events page in 17
  • [NMS-7802] – JSON Serialization Broken in REST API (CXF)
  • [NMS-7814] – Queued RRD updates are no longer promoted when rendering graphs
  • [NMS-7816] – The DataCollectionConfigDao returns all resource types, even if they are not used in any data collection package.
  • [NMS-7818] – Measurements ReST API Fails on strafeping
  • [NMS-7819] – Requesting IPv6 resources on measurements rest endpoint fails
  • [NMS-7822] – Remove Access Point Monitor service from service configuration
  • [NMS-7824] – The reload config for Collectd might throws a ConcurrentModificationException
  • [NMS-7826] – Exception in Vacuumd because of location monitor changes
  • [NMS-7828] – NPE on "manage and unmanage services and interfaces"
  • [NMS-7834] – Smoke tests failing because OSGi features fail to install: "The framework has been shutdown"
  • [NMS-7835] – "No session" error during startup in EnhancedLinkdTopologyProvider
  • [NMS-7836] – KIE API JAR missing from packages
  • [NMS-7839] – Counter variables reported as strings (like Net-SNMP extent) are not stored properly when using RRDtool
  • [NMS-7844] – Some database reports are broken (ResponseTimeSummary, etc.)
  • [NMS-7845] – New Provisioning UI: 401 Error when creating a new requisition
  • [NMS-7847] – Graph results page broken when zooming
  • [NMS-7848] – Parameter descriptions are not shown anymore
  • [NMS-7852] – UnsupportedOperationException when using the JMXSecureCollector
  • [NMS-7855] – distributed details page broken
  • [NMS-7856] – Default log4j2.xml has duplicate syslogd appender, missing statsd entries
  • [NMS-7857] – Cisco Packets In/Out legend label wrong
  • [NMS-7858] – Enlinkd CDP code fails to parse hex-encoded IP address string
  • [NMS-7861] – IpNetToMedia Hibernate exception in enlinkd.log
  • [NMS-7867] – Duplicate Drools engines can be registered during Spring context refresh()
  • [NMS-7870] – PageSequenceMonitor broken in remote poller
  • [NMS-7874] – The remote poller doesn't write to the log file when running in headless mode
  • [NMS-7875] – Distributed response times are broken
  • [NMS-7877] – HttpClient ignores socket timeout
  • [NMS-7884] – RTC Ops Board category links are broken
  • [NMS-7890] – Remedy Integration: the custom code added to the Alarm Detail Page is gone.
  • [NMS-7893] – LazyInitializationException when querying the Measurements API
  • [NMS-7897] – Statsd PDF export gives class not found exception
  • [NMS-7899] – Deadlocks on Demo
  • [NMS-7900] – JMX Configgenerator Web UI throws NPE when navigating to 2nd page.
  • [NMS-7901] – Incorrect Fortinet System Disk Graph Definition
  • [NMS-7902] – Pages that contain many Backshift graphs are slow to render
  • [NMS-7907] – The default location for the JRRD2 JAR in rrd-configuration.properties is wrong.
  • [NMS-7909] – Missing dependency on the rrdtool RPM installed through yum.postgresql.org
  • [NMS-7917] – Alarm detail filters get mixed up on the ops board
  • [NMS-7921] – Startup fails with Syslogd enabled
  • [NMS-7926] – FasterFilesystemForeignSourceRepository is not working as expected
  • [NMS-7930] – Heat map ReST services just produce JSON output
  • [NMS-7935] – ClassNotFoundException JRrd2Exception
  • [NMS-7939] – HeatMap ReST Xml output fails
  • [NMS-7942] – Apache CXF brakes the ReST URLs for nodes and requisitions (because of service-list-path)
  • [NMS-7944] – Jersey 1.14 and 1.5 jars mixed in lib with Jersey 1.19
  • [NMS-7945] – Incorrect attribute types in cassandra21x data collection package
  • [NMS-7948] – Bad substitution in JMS alarm northbounder component-dao wiring
  • [NMS-7959] – Bouncycastle JARs break large-key crypto operations
  • [NMS-7962] – Missing graphs in Vaadian dashboard when storeByFs=true
  • [NMS-7963] – JSoup doesn't properly parse encoded HTML character which confuses the XML Collector
  • [NMS-7964] – MBean attribute names are restricted to a specifix max length
  • [NMS-7968] – Auto-discover is completely broken – Handling newSuspect events throws an exception
  • [NMS-7969] – JMS alarm northbounder always indicates message sent
  • [NMS-7972] – Querying the ReST API for alarms using an invalid alarmId returns HTTP 200
  • [NMS-7974] – The ICMP monitor can fail, even if valid responses are received before the timeout
  • [NMS-7977] – JMX Configuration Generation misbehavior on validation error
  • [NMS-7981] – The ReST API code throws exceptions that turns into HTTP 500 for things that should be HTTP 400 (Bad Request)
  • [NMS-7985] – New servers in install guide
  • [NMS-7997] – Background of notifications bell icon is too dark
  • [NMS-7998] – Provisiond default setting does not allow to delete monitoring entities
  • [NMS-7999] – Upgrade to commons-collections 3.2.2
  • [NMS-8001] – NPE in JMXDetector
  • [NMS-8004] – Iplike could not be installed following install guide

Enhancement

  • [NMS-1488] – Add option to the <service> element in poller-configuration.xml to specify service-specific RRD settings
  • [NMS-1910] – Additional storeByGroup capabilities
  • [NMS-2362] – Infoblox events file
  • [NMS-3479] – Adding SNMP traps for Raytheon NXU-2A
  • [NMS-4008] – Add A10 AX load balancer trap events
  • [NMS-4364] – Interactive JMX data collection configuration UI
  • [NMS-5016] – Add Force10 Event/Traps
  • [NMS-5071] – Event definition for Juniper screening SNMP traps
  • [NMS-5272] – events definiton file for DSVIEW-TRAP-MIB
  • [NMS-5397] – Trap definition files for Evertz Multiframe and Modules
  • [NMS-5398] – Trap and data collection definitions for Ceragon FibeAir 1500
  • [NMS-5791] – New (additional) event file for NetApp filer
  • [NMS-6770] – New Fortinet datacollection / graph definition
  • [NMS-7108] – DefaultResourceDao should use RRD-API to find resources
  • [NMS-7131] – MIB support for Zertico environment sensors
  • [NMS-7191] – Implement "integration with OTRS-3.1+" feature
  • [NMS-7258] – Unit tests should be able to run successfully from the start of a compile.
  • [NMS-7404] – Create a detector for XMP
  • [NMS-7520] – Remove linkd
  • [NMS-7553] – Add Juniper SRX flow performance monitoring and default thresholds
  • [NMS-7614] – Enable real SSO via Kerberos (SPNEGO) and LDAP
  • [NMS-7618] – Create opennms.properties option to make dashboard the landing page
  • [NMS-7689] – Get rid of servicemap and servermap database tables
  • [NMS-7700] – Add support for Javascript-based graphs
  • [NMS-7722] – Dell Equallogic Events
  • [NMS-7768] – Persist the CdpGlobalDeviceIdFormat
  • [NMS-7798] – Add Sonicwall Firewall Events
  • [NMS-7805] – JMS Alarm Northbounder
  • [NMS-7821] – DNS Resolution against non-local resolver
  • [NMS-7868] – Recognize Cisco ASA5580-20 for SNMP data collection
  • [NMS-7949] – Promote Compass app when mobile browser detected
  • [NMS-7986] – Document how to configure RRDtool in OpenNMS

Story

  • [NMS-7711] – nodeSource[] resource ids only work when storeByFs is enabled
  • [NMS-7894] – Flatten and improve web app style
  • [NMS-7929] – Document HeatMap ReST services
  • [NMS-7940] – Cleanup docs modules

The Inverter: Episode 55 – Faster than Lightning

I started writing these “inverter” posts because many Bad Voltage episodes would raise topics that I felt deserved commentary. By the middle segment in this episode I was screaming at the computer.

So, good show.

First, whoever decided on the cover art gets some points. It references a groaner of a pun Jono makes that gets dropped in the Intro.

Second, also in the intro, we learn that Jeremy Garcia will not be on the show due to jury duty of all things. While I’ve always considered Jeremy one of the calmer and more reasoned members of the team, since this show clocks in at scant 52 minutes maybe he’s the one who drags things out. They did stumble a bit on the whole “… and now, Bad Voltage” line so I do look forward to Jeremy’s return.

Okay, the first segment concerns the “new” economy of begging. It kind of focuses on what we would call “crowdfunding“, but as Stuart points out, crowdfunding usually means that you get something in return. However, with sites like “GoFundMe” the term has been expanded to include outright begging, as in “Dear Internet, help, can you spare a dollar for a sandwich”. A quick perusal of the site with a search in my local area brings up a number of articles ranging from a person who was defrauded by a builder, to two women who want to go to the ACC tournament, to another woman who needs help finishing her Ph.D.

I’m not saying this is a bad thing, as the sucker/minute ratio remains high, but it is a bit different from crowdfunding sites like Indiegogo and Kickstarter where the donors have a non-zero expectation of actually getting something. That is more along the lines of “new economy” than asking strangers to pay for your vacation.

So, let’s talk about those programs. I have to admit I don’t participate in them. Before you go and call me a cheapskate and a leech, I do donate a lot of money to local and free software causes, but I just don’t do it via these programs. I’ve participated in exactly two Indiegogo campaigns and one Kickstarter campaign. Let’s see how they went.

The first time was the Indiegogo campaign for the Ubuntu phone. While I am perfectly happy with my Android phone (more on that later) I support open source efforts and this seemed like a good thing. They were organized and they had realistic expectations for what it would cost. The campaign fell well short of their goal and my money was returned. All in all, I’m okay with that.

The next time was also on Indiegogo. It was for the Angel Sensor wearable health device. I have a keen interest in how my body is behaving as metrics are the key to making successful improvements. The problem is I don’t want to be sending my activity and sleep pattern information to some third party like Fitbit or Jawbone. I was very eager for an open source solution.

I’m still waiting.

Plagued by production problems and lack of communication, I have no idea if I’ll ever see the device on which I spent US$178. The one person I knew there is on “a well deserved leave”. Furthermore, I’m not sure if they are releasing the server and client code as open source, which I what I was lead to believe was the plan. Finally, the first app they wrote for it is for the iPhone of all things, which makes me think that their dedication to open source is a bit lacking. At this point in time I’ve written the whole thing off.

When the Mycroft project did the crowdfunding thing, I was sorely tempted to buy in, but my experience with Angel has made me cautious. I think a lot of technology-based projects severely underestimate what is needed to be successful. They aim low and then trumpet when their stretch goals are met, only to wake up later to the fact that it is going to be a lot harder to deliver than they thought, like the hangover after a big bender.

Please note that I’m not saying this will happen with Mycroft, I wish them all the luck in the world, it’s just that I’ll shell out a few extra ducats for the finished thing when it arrives rather than gamble.

Does anyone remember Diaspora? It was the open source, distributed Facebook. I thought the project was dead, but it is apparently still around, although the pressure of delivering on it is blamed for the suicide of one of the co-founders. Diaspora was one of the most successful Kickstarter projects at the time.

This isn’t to say that these things always fail. The “Exploding Kittens” project was phenomenal and while I haven’t played it I’ve given it as a gift and people say it is a lot of fun. This is where I think crowdfunding can shine – in creative projects where the sponsors have a huge amount of control over the product. I’ve heard of a number of successful movie, music and video projects that were crowdfunded without problems.

Which brings me to my one foray into Kickstarter. I’m a huge fan of the band De La Soul. To me they were the first nerdcore hip-hop group. When hip-hop seemed solely focused on “bitches ‘n hos,” De La Soul was delivering thoughtful, fun and energetic music. When they announced their Kickstarter for a new album, I signed up and ordered the album to be digitally delivered on a 1GB Posdnuos USB drive set for September delivery.

Well, it ain’t here. (grin)

I really don’t mind – I’d rather the album ship when it is ready (probably next Spring) than for them to release crap on time but I’m basically 0-3 on the whole crowdfunding thing.

I was thinking about this when the second segment started with Aq reviewing his new One Plus X (OPX) phone, giving it a 9 out of 10.

This is when I started yelling.

See, while I have zero experience with the OPX I bought a One Plus One (OPO) and I found One Plus to be one of the most horrible companies on the planet.

I was first introduced to the OPO by some friends in Germany. Here was a powerful phone in an attractive package at a reasonable price. It also ran open source software in the form of a version of Cyanogenmod, a packaged instance of the Android Open Source Project (AOSP). Finally, it was relatively inexpensive. Too good to be true?

It was.

They have an “invite” system in order to even buy the phone, but I managed to wrangle one. While I thought the phone was too big initially, I got used to it and soon I was telling everyone how great it was, just like Stuart does in his review.

But then things started to go sour. The upper half of the digitizer started acting up and so I opened a ticket with support. This is when One Plus started to lie and cheat, trying to wrangle out of the fact that they had a hardware problem. The problem has one topic on their forums that had 125 pages of posts before they closed it, and another that is at 305 pages as I write this. That’s 305 pages of pure horror stories.

So when I say lie, we all know that One Plus is a tiny Chinese firm, yet all of my support replies came from “different” people with traditionally English female names, like Kathy, Leah and Jessa. I think this was a tactic to make us more sympathetic to them since they knew they were going to provide crappy support.

When I say cheat, they refused to honor warranty support and kept asking me to perform a number of increasingly complex tasks culminating in disassembling my phone. When I refused, fearing I would damage it, they refused service, even when I offered to send it to them at my expense.

In my mind, One Plus is pure scum and no one should buy their products. I came extremely close to launching a class action lawsuit against them before I decided I had better things to do than to sue a company that won’t be around in five years.

Seriously, if I had to choose solely between an iPhone and a One Plus phone I’d grab the iPhone so fast I’d break my fingers. Finally, their new OxygenOS is closed source so you are up the same creek as if you had bought a Samsung or other closed Android phone.

So I’m screaming at the computer because I know Aq’s “9 out of 10” review will move people to consider buying one. Don’t! Aq has hooked up with the same skank that did me wrong, and while part of me wishes them well, I know it will end in misery.

But what are the options, you might ask. Samsung is expensive and closed, Google is getting more and more closed, and so perhaps One Plus is the least of the evils.

There are options, but Stuart’s will be pretty limited since he seems to have two huge prejudices. First, he expresses disdain for hold people who root their phones. This is odd, since I don’t think he’d have any issue with buying a laptop that shipped with Windows and putting Linux on it, and this is, after all, a podcast about things hackable. Second, he seems to dislike anyone with a “big” phone.

I love the alternative ROM crowd. These are the true AOSP disciples, and my favorite ROM is OmniROM. I love OmniROM so much that when I need a new phone I work backwards. I start with the list of officially supported OmniROM devices and make my choice from there. While I closely identify with the philosophy behind OmniROM (it was started as a fork from Cyanogenmod when they got tons of VC money and went evil), what I love are the options. You can choose just how many or how few applications you want from the Google ecosystem, which allows you to easily limit what to want to share (note that this is available with almost any alternative ROM), and they turn on a lot of things Google doesn’t, such as “shake to dismiss” in the alarm.

As for size, when I unpacked my OPO I thought the thing was huge. I was using an HTC One and it seemed tiny in comparison. It took me about two days to get used to it. When I replaced the OPO because they are huge douches (or whatever is Chinese for douche) I went with the Nexus 6. Now that is a huge phone, and I’m sure Aq will belittle it.

Know what? After about two days of using it, it felt normal. I love my Nexus 6 running OmniROM. The large screen allowed me to retire my Nexus 7 since I can comfortably watch videos on it when traveling. It has an amazing camera, is extremely fast and gets all the latest Android shiny. In fact, I was amazed that when the new Nexus phones came out I found myself asking myself why in the world would I switch? Plus the Nexus 6 still has wireless charging, which I’ve become used to.

I think Aq’s size issues stem from the fact that everyone thinks that if someone is using a phone bigger than the one they use, those people are crazy. If he spent a week with a Nexus 6, I’m sure his mind would change. Now, he’s given up freedom for a pretty face with a cheap price tag.

Now it seems like I’m picking on Stuart a lot, but I don’t mean to be mean. I love the guy and I want him to be happy, but that little tramp will only bring misery. Mark my words.

If One Plus did you wrong, let him know, but I think it is too late. As with every doomed relationship, when you are in it you can’t see it coming.

Whew.

After the first two, the last segment was pretty conflict free. It concerns the US Department of Justice wanting to force Apple to unlock a phone. I thought this case raised a couple of interesting points.

First the reason they want to force Apple to do it instead of the owner is to avoid the issues of self-incrimination. I never really thought about that before, but it is good to know.

Second, the DoJ is using the logic that since Apple still owns the software on the phone, they should be able to unlock it. Most people (well, non-software people) don’t know or realize that they don’t own most of the software they use. They have just been granted a right to use it. Now Apple (and Google) are taking steps to encrypt phones so even they can’t unlock them. This case involved an older iPhone, but it does make the case for using free software and kudos to Apple for fighting the order.

While there may be a fine line as far as “ownership” is concerned, free and open source software is much more in the hands of the user (you don’t pay for it) so you may have additional protections against self-incrimination when you use it. I am not a lawyer, but it is fun to think about.

The show ended with a reminder that the next Live Voltage show will be at SCaLE in January. I also learned why Bryan missed our little post-show gathering last year – he went to bed.

And here I thought he hated me.

2015 Open Source Monitoring Conference

Once again I got to visit the wonderful town of Nürnberg, Germany, for the Open Source Monitoring Conference.

OSMC - Badge

Hosted by Netways, the conference started out ten years ago as a Nagios conference. The name was changed due to an issue with the Nagios trademark, but it still focused heavily on Nagios. However, the organizers are pretty open to all things monitoring, so they started inviting projects like Zabbix and OpenNMS to come. When the Nagios fork Icinga was created, the amount of Nagios content dropped considerably, and out of 24 talks over 2 days there were only two that had Nagios in the title. Part of this has to do with Icinga 2 being a total rewrite and thus has started to move past its Nagios roots.

This year it was a cornucopia of monitoring choices. In addition to Icinga, Zabbix and OpenNMS, there was Alyvix, Assimilation, Heroic, and Prometheus. Grafana was popular and most tools are adding support for that data visualization tool, and it was nice to see talks on NSClient++ and MQTT. A little less than half the talks were in German, so there is a large German focus to the conference, but there was always an English-language talk available as well.

Nürnberg is a cool town. There is a big castle and lots of walls are left over from the original fortifications for the city. It is also home to SuSE Linux, and I made sure to swing by if just to get a picture for Bryan Lunduke:

OSMC - SuSE Office

Ronny and I got there on Monday. While the main conference is held over two days, this year there were workshops on Monday and a “hack-a-thon” on Thursday. The conference pretty much takes over the Holiday Inn, City Center, hotel. While the facilities are nice, it is right next to the city’s “eros center” which seems to creep closer and closer to the hotel each year I attend. It doesn’t impact the conference in any way, and those who might be sensitive to such things can easily avoid it.

There is always lavish catering and this year we had a nice, small crowd of OpenNMS enthusiasts in attendance, and we met up for the hosted dinner on Monday night. I had not seen some of the people since the OUCE, so it was nice to catch up.

My talk was on Tuesday, the first day of the main conference. The event was sold out, with about 250 people, and at times the rooms could get quite full.

OSMC - Crowd

The talks were all rather good. Torkel Ödegaard talked about Grafana:

OSMC - Grafana

which was a big hit with crowd, and as I mentioned before a lot of projects are leveraging his work to provide better data visualization, including OpenNMS. My talk went well (I think) as I went over all of the amazing things we’ve done since last year at the OSMC, which included four major releases of our application. I was stumped with the question “How do I get started with OpenNMS?” when I realized that I didn’t have an easy answer. I can tell you how to install it, but that doesn’t get you started. I need to work on that.

That evening we returned to Terminal 90, which is an odd place to hold a dinner but it seems to work. Terminal 90 is a restaurant located at the Nürmberg airport, and it does a good job of holding everyone. We have to take the U-bahn to get there, and at least this year there were no incidents (last year someone tried to hold open the doors, which caused the autonomous train to shut down and wait for human intervention).

OSMC - Terminal 90

The food and drinks were good, and toward the end of the evening they had woman impersonating German pop star Helene Fischer, which was lost on me but the crowd seemed to enjoy it.

I called it a night fairly early, but this is a group that tends to hang out until the wee hours of the morning. Although my room was on the first floor, I didn’t hear much noise from “Checkpoint Jenny” across the street, so maybe everyone is getting more mellow in their old age. (grin)

The second day featured a number of talks from different projects. Usually the Zabbix talk is done by Rihards Olups, but he was unable to make it this year so Wolfgang Alper did the honors.

OSMC - Zabbix

After that was a really good talk by Martin Parm on how Spotify monitors its music service.

OSMC - Spotify

It started out with all of the tools they tried that failed, and I kept thinking to myself “don’t let it be OpenNMS, don’t let it be OpenNMS” (it wasn’t) and ended with a tool they wrote in-house called Heroic. It is a time-series data store built on top of Cassandra, and it looks a lot like the Newts tool we built. Both are open source and Apache-licensed so I’m hoping to find some synergy between the two projects. There is another large music streaming service that uses OpenNMS, but maybe we can get all of them (grin).

OSMC - Prometheus

Then there was a talk by Fabian Reinhartz on a monitoring system called Prometheus. I had to joke that the name refers to the daily experience of most network managers of having their liver eaten out, but it seems like an interesting tool. Written in Go, it may find resistance from users due to the configuration being more like writing code, but that also makes it powerful. Sounds familiar to me.

I had to leave right after lunch in order to be ready to catch my flight home, but I really enjoyed my time there, even more than usual. Many thanks to Bernd Erk and the Netways gang for holding it, and they should be posting the videos soon. If you are interested in next year be sure to register early as it is likely to sell out again.

The Inverter: Episode 54 – The Trolley Problem

Throw out the first segment, and this is one of the best Bad Voltage episodes yet.

It’s not that the first segment sucks (well, for certain values of “suck”), but it pales in comparison to the rest of the show.

That first bit concerns a rant, introduced by Aq, about a trend in programming to rely on “frameworks” instead of actually learning how to code in a particular language. It was set off, as I understand it, by someone wanting to know how to add together two numbers using JQuery, and the response was, uh, why don’t you just add the numbers together using Javascript?

I can understand the frustration. There was a recent rant by Linus Torvalds about a pull request submitted against the kernel that was unnecessarily obtuse. As the pressure mounts to get more and more code out faster and faster, not only are novice programmers being asked to do more complex tasks, they are relying more and more on frameworks and libraries to do them.

While I am not a coder, I do view the writing of code as an art form, and I like code that is artistic: beautiful, clever and functional. I can remember many years ago visiting an especially ugly page on a government website, and when I looked at the source I found it had been generated by Microsoft Frontpage. Yes, that tool would create a web page, but in no way will the code be beautiful or clever, or in this case, functional.

I was not sure if this rant applied to IDEs. Almost all OpenNMS code is done in Eclipse. I think I’m the only one who uses vi, along with healthy amounts of recursive grep. We also use a lot of libraries. Why reinvent the wheel? Of course, this has caused the size of the OpenNMS application to balloon, currently pushing more than half a gigabyte. But space is relatively cheap and time matters, so why not?

I thought it very telling when Aq decided he disliked code that involved any level of abstraction above what he was using. It reminded me of the old George Carlin joke that anyone who drives faster than you is a maniac, while anyone who drives slower than you is an idiot. I did like it when they reminisced about classic code that was very compact and just plain fast. These days we trade speed of completion for speed of execution. My own memory is of running Mac OS 6 on one double sided (800K) floppy. I could put the O/S, MacPaint, MacDraw and MacWrite all on one disk will about 100K left for my files. I couldn’t afford a Mac back then (they ran about US$5K) but the school had ones you could use and all I needed to carry was that disk.

The next segment talked about the Blue Yeti microphone. I bought one of these specifically for the time I was on Bad Voltage, so there must be something in the water about this show and owning one. I was a little confused, however, when the segment starts and Jono states he bought his as a travel mic. This sucker is huge, and as I like to travel as light as possible I can’t imagine dragging it around. However, as the segment continues, it is obvious we are talking about the same mic.

It is a great device. While I like getting input from the gang on which toys to buy, my go-to source for tech advice is The Wirecutter, and the Yeti is their microphone of choice as well. If you plan on recording for the Internet, you should seriously consider getting one of these.

It is the third segment that I thought was brilliant. I’m not sure who came up with the idea, but the discussion centered around ethics programming in self-driving cars. While I disagree with Jeremy that this is something that will need to be figured out before these vehicles become mainstream, it will be a question in need of an answer as they mature.

The scenario offered is this: You are in your self-driving car going along a mountain road. Suddenly, you turn a corner and there are five people in the way. Assuming the car can detect this, should it continue on, protecting the passenger but possibly killing the five people, or should it drive over the side of the cliff, killing the passenger but saving the people in the road?

Wow – what a neat question.

I have no idea of the correct answer. It did dawn on me (as it did the gang) that if the solution was to sacrifice the passenger that pranksters would be more than happy to jump in front of these cars just to see what happens, and I think in at least those models aimed at higher end consumers, they may tout that passenger safety has been programmed into the system to be paramount.

It was a real “grown up” question and I think spawned one of the better discussions ever done on the show.

I was surprised no one brought up Spock’s death speech, “The needs of the many, outweigh … (the needs of the few) … or the one” but Aq did reference the I, Robot movie so he gets points for that.

The final segment concerned the UK government’s decision to put pressure on technology providers to eschew strong encryption in favor of either weak encryption or some sort of back door. Apple has stood up and stated that, if enforced, they would stop selling their products in the UK. It was scary to think about this, since no elected official in any company would want to be labeled as the guy who stood in the way of someone getting an iPhone. Bryan pointed out that the market capitalization of Apple is roughly US$700B, putting it at about 25% of the UK’s GDP (with its fifth highest GDP in the world), and so that threat carries a lot of weight.

This was another “big boy question” and I liked the discussion. Should anyone announce that a back door exists in a popular technology, you can bet the bad guys will throw everything at exploiting it. It’s just not a good idea, although it isn’t surprising that it comes from the UK, a country known for the ubiquitous use of CCTV (on a side note, they have also started using traffic cameras that track you between points and if you exceed the posted speed between them, you get ticketed.)

Of course, there is the thought that a private company like Apple has the ability to sway governments, but no one minds the 800 pound gorilla when it is on your side.

During the outro the guys announced they are returning to SCaLE next year to do a Live Voltage show. These are awesome and shouldn’t be missed, and they have room for nearly 1000 people in the venue so expect it to be crazy. Plus, if you visit the site you’ll see Bryan Lunduke right on the front page next to Cory Doctorow – which I think is pretty cool. Outside of Live Voltage, he’ll be doing a presentation on why he hates freedom, I mean, why Linux sucks.

While we aren’t sponsoring that show, OpenNMS is a gold sponsor at the conference, so be sure to go and stop by our booth.

Anyway, the lads did a great job this week. If you have never listened to Bad Voltage, this would be a great one with which to start.

Review: Signal by Open Whisper Systems

I like security, and one of the biggest security holes in my technology concerns text messaging and phone calls. While I can secure my data (for the most part), it is hard to secure traffic over the telephone network, especially with the proliferation of devices like the Stingray.

Awhile ago my friend Jeff introduced me to Red Phone by Open Whisper Systems, which was an app that would encrypt your phone calls. I could never get it to work very well, so I didn’t use it, plus Jeff was the only person I talked with who used it.

Flash forward more than a year, and I’m finding that I quite often don’t get texts from Jeff, while he gets mine just fine. He did some investigation and traced the issue to TextSecure, which was an encrypted text app also from Open Whisper Systems. Apparently I was registered on his phone as a TextSecure user, so it was trying to send text to me by that method. Since I no longer had Red Phone on my device (I play a lot with the software on my mobile devices and had not restored it after a clean install) I wasn’t getting the messages.

I went to install TextSecure and found that it has been replaced by Signal. My, what a difference a year makes. Not only was it easy to use, the app itself is pretty nice. It combines both TextSecure and Red Phone features, and is now the default SMS application on my handy.

Signal is 100% open source. The only way for true security is if everyone has the opportunity to examine the code and look for vulnerabilities. Plus, think about it, if you care about security chances are you want to send sensitive information using the service. Without open source you can’t be sure that information isn’t being intercepted by third parties.

This has resulted in some pretty high endorsements:

Quotes about Signal

Signal is available for both Android and iOS, Note that is uses a data connection to send encrypted SMS messages, so it will count against your data cap. I haven’t had the chance to try out the phone functionality as of yet, but it works fine as a normal SMS client as well.

It is nice to come across such a useful piece of software that is 100% open source, and if I happen to send you SMS messages, be on notice that I will be sending you an invite to Signal (grin).

UPDATE: This is so cool. Since the app uses data instead of the SMS protocol for encrypted texts, it works as long as the mobile device has data. Which means that I can get texts no matter what SIM card is currently in my handy. Cool! So I’m in Germany using my Ortel SIM and I’m able to get SMS messages from friends in the US who have no idea where I am or what network I’m using. Killer feature.

Reflections on Paris and My Cowardice

I was on a bus in Ireland when I heard the news about the Paris attacks. I had gotten up early to head to the opposite coast as I wanted to see an Ireland that wasn’t Dublin, and I don’t think I could have picked a better spot than Doolin, in County Clare.

Today was to be a particularly gray day and it was dark when I started out. It didn’t get much lighter as we rode to Galway, and when I changed buses the driver was playing the news from the radio. Of course the only story was about the more than one hundred people killed in senseless violence overnight.

Peace Symbol by @jean_jullien

I have some friends in Paris and so I immediately reached out to them. As I waited for a response, I pretty much sat, stunned, as the Irish countryside passed by outside my window.

Once I got to my B&B, I dropped my bag and took a long walk, looking for lunch. The day reflected my mood perfectly. It was like nature itself was in mourning. At high noon the sky wasn’t much lighter than at dusk. A roaring wind came off the sea, churning up angry whitecaps. The clouds drizzled rain like tears.

By the time I was getting cold, I found the recommended pub and went in. It was packed, as this is a popular tourist location and they drop people off by the bus load. Since I was alone, I offered to sit at the bar to make room for the next coach, which arrived about five minutes after I did.

A boisterous crowd of mainly young people came in and crowded around the bar where I sat. They were laughing and joking, blissfully unaware of how quickly that can change. I took a little comfort in the normalcy of that moment: people ordered food, the Indian guy asked about vegetarian options, and drinks were poured (including an inexplicable request for a bottle of Miller beer).

As I ate my meal, a nice smoked salmon salad and a wonderful seafood chowder stuffed with mussels, I was reminded of the last time I had mussels this good, which just happened to be in a Belgian restaurant in Paris called La Gueuze.

And I struggled with a dilemma. The Paris Open Source Summit is next week and I am supposed to be there. Heck, I lobbied hard for the opportunity to participate. But while the chance of anything happening is very slim, I can’t say I’m eager to be in Paris at the moment, especially as part of a large crowd.

So I decided not to go.

There were a number of factors. Part of it was concern for my wellbeing. Part of it was concern for my family. I travel a lot and I know they worry no matter where I’m going, and they have been very understanding when I’ve gone to places that don’t exactly have a reputation for safety. I refuse to put my decision on them, but it did play a role.

But I think the deciding factor was actually how much I enjoyed Paris on my last trip. It is an amazing city, and I didn’t want that memory ruined by seeing soldiers on every corner or having to go through intrusive screening at every point of entry.

It makes me feel like a coward. The terrorists have won.

And I can’t understand it. Of all the countries in Europe, the French bend over backwards to be accommodating to different views and ways of thinking. The French motto “Liberté, égalité, fraternité” leads with the word for freedom, and they go to great lengths to explore all the weird corner cases to insure their society is as free as possible.

And that’s what makes me the most angry. I’m certain these acts are going to change that. Not only will it move France to be more restrictive, it will give the more aggressive countries reason to step up military action in the Middle East. A lot more people will die, and most of them will have darker skin. This will create more terrorists, and the cycle will continue.

I hope France and the rest of the world shows some restraint. I’m not, in any way, shape or form, suggesting justice not be sought out, but I’m reminded of something I saw many years ago.

I was living at my parents’ house and my two-year-old nephew was staying with us. It was a beautiful day and so the windows were open, and there was a gentle breeze throughout the house. One strong breeze caught the door behind the boy and slammed it shut. It scared him, so he reached out and smacked the door, as if to punish it. It struck me as a perfect example of a childish reaction – I’m scared and angry so I need to strike out at the nearest thing, whether is makes sense or not.

I hope the world remembers that we are not children.

I don’t have any answers on how to make things better. The best I can do is to promote free and open source software. I know it sounds silly, using FOSS to cure the world’s problems, but in every place I’ve visited (and I’ve been to 37 different countries) I’ve found like-minded people in that community with a strong desire to create new things through cooperation. It creates an environment where anything is possible. In a small way, it creates hope.

I am writing this sitting on my bed at the B&B. It’s cold, and the wind is whipping around the house, but I feel cozy and safe. Here’s a wish that everyone can find a place to be cozy and safe, as well as the hope that tomorrow will be a better day.

Horizon 16.0.4 Security Release

In response to the Apache Commons library that OpenNMS uses, version 16.0.4 has been released to help secure against a remote exploit.

The exploit involves Java Remote Method Invocation (RMI) which listens on port 1099 by default. In my previous post I pointed out that if that port is inaccessible, then the exploit can’t happen.

What 16.0.4 does is limit RMI to only listen on localhost. While that will prevent remote exploits even in the event port 1099 is blocked via the firewall, it doesn’t completely solve the problem. To fix the root cause of the issue will require changes to Apache Commons, and we are ready to upgrade to the fixed version as soon as it is available.

We tend to be very internally critical of security issues within OpenNMS, and some people complained that my last post wasn’t technical enough. So I’m hoping to correct that with this one, but if you don’t care about such things you should probably skip it (grin). I have started updating the Security Considerations page on the wiki with details about securing OpenNMS in general, and that will have better information for people interested in security and OpenNMS than this blog post.

While blocking external access to port 1099 will secure OpenNMS against this attack for most people, it doesn’t prevent people who have access to the machine from exploiting the vulnerability. This is called a “privilege escalation” attack vs. a “remote exploit”, as a “normal” user can now have rights (i.e. root access) if they are locally on the machine. Most of our users tend to limit shell access to the server, so this shouldn’t be a problem, but in environments that rely heavily on directory services such as LDAP, the default may be to allow non-privileged access to certain users (say, the “IT Group”) that aren’t involved in maintaining OpenNMS.

And there is also the slim chance that there is a vulnerability in our webUI that could allow a user access to the system. We, of course, don’t know of any and we take great care to prevent it, but simply hoping to limit access to the server as a way to prevent this exploit is insufficient.

So, to prevent it entirely, we are removing RMI. It was introduced in the first iteration of the OpenNMS Remote Poller, but real world installation found that getting the proper ports open was a real pain. So instead the remote poller now talks over HTTP/HTTPS (with the latter being the most secure). Most networks have ports 80 and 443 open, so that made things a lot easier.

Until that is introduced (most likely with Horizon 17), it is still a good idea to limit access to the OpenNMS server to only essential people.

Note that Java Management Extensions (JMX) also use serialized objects and thus could be vulnerable. OpenNMS has a JMX port (18980) but it is bound to localhost by default. In fact, all ports are bound to localhost by default in 16.0.4 except for the webUI, port 8980.

There are a number of other steps you can take to harden your OpenNMS server. I’m planning on detailing them on the wiki, but start with only doing a minimal operating system install. The less software on the system, the smaller the chance one will have a vulnerability.

Also, OpenNMS currently runs as the “root” user. This is due to the fact that it needs access to ICMP traffic as well as port 162 for SNMP traps. Both of these require root by default. With some “stupid kernel tricks” you can run OpenNMS as a non-root user, but it has not been heavily tested. We have a detailed list of issues for running as non-root on our Jira instance.

Sorry to drone on about this, but we take security extremely seriously at OpenNMS. We also have to labor under the misconception that Java is inherently unsafe. It is not true, although people still have nightmares from the early issues with client-side Java applets. The Java in OpenNMS is server-side and we don’t use applets, and the language is used securely in a tremendous amount of software.

For comparison, WordPress, an application I love, is currently estimated to run 25% of the world’s websites. It is written in PHP, a language that has a huge track record of security exploits, and many of the spam e-mails I get link to compromised WordPress sites.

It is possible to secure WordPress (we use it for all of our websites as well) but it takes some diligence. We will remain as diligent as we can concerning the security of OpenNMS, and we will continue to take steps to make it even more secure.