2018 New Zealand Network Operators Group (NZNOG)

One thing that all open source projects struggle with is getting users. Most people in IT and software are overwhelmed with a plethora of information and options, and matching the right material to the right audience is a non-trivial problem.

Last year my friend Chris suggested that I speak at a Network Operators Group (NOG) meeting, specifically AusNOG. It was a lot of fun. I felt very comfortable among this crowd. so I decided to reach out to more NOGs to see if they would be interested in learning about OpenNMS.

The thing I like the most about NOGs is that they value getting things done above all else. While “getting things done” is still important with the free and open source crowd, there seems to be more philosophy and tribalism at those shows. “Oh, that’s written in PHP, it must suck” etc. As a “freetard” I live for the philosophical and social justice aspects of the community, but from a business standpoint it doesn’t translate well into paying customers.

At NOGs the questions are way more business-focused. Does it work? Is it supported? What does it cost? While I’m admittedly biased toward OpenNMS and its open source nature, the main reason I keep promoting it is that it just makes solid business sense for many companies to use it instead of their current solution.

Plus, these folks are pretty smart and entertaining while dispensing solid advice and knowledge.

Anyway, with that preamble, at AusNOG I learned about the New Zeland NOG (NZNOG) and submitted a talk. It got accepted and I found myself in Queenstown.

NZNOG Scenary

The main conference was spread out over two days, and like AusNOG it consisted of 30 to 45 minute talks in one track.

While I know it won’t work for a lot of conferences, I really like the “one track” format. It exposes me to things I wouldn’t have gone to otherwise, and if there is something I am simply not interested in learning about I can use that time to catch up on work or participate in the hallway track.

NZNOG Clare Curran

The conference started with a presentation by the Honorable Clare Curran, a newly minted Member of Parliament (they recently held elections in New Zealand). I’m slowly seeing politicians getting more involved in information technology conferences, which I think is a good thing, and I can only hope it continues. She spoke about a number of issues the government is facing with respect to communications technology.

Several things bother me about the US government, but one big one is the lack of understanding of the importance of access to the Internet at broadband speeds. Curran stated that “lack of reliable high-speed network access is a new measure of poverty”. Later in the day John Greenhough spoke on New Zealand’s Ultra-Fast Broadband (UFB) project, where on one slide broadband was defined as 20Mbps download speed.

NZNOG John Greenhough

Where I live in the US I am lucky to get 10Mbps and many of my neighbors are worse off, yet the government is ceding more of the decision making process about where to build out new infrastructure to the telecommunications companies which have zero incentive to improve my service. It’s wonderful to see a government realize the benefits of a connected populace and to take steps to make it happen.

Because we all need Netflix, right? (grin)

There was a cool talk about how Netflix works, and I didn’t realize that they are working with communications providers to provide low-latency solutions distributed geographically. This is done by supplying providers with caching content servers so that customers can access Netflix content while minimizing the need for lots of traffic over expensive backhaul links.

NZNOG Netflix RRD

I did find it cool that one of the bandwidth graphs presented was obviously done using RRDtool. I don’t know if they collected the data themselves or used something like OpenNMS, but I hope it was the latter.

With this push for ubiquitous network access comes other concerns. New Zealand has a law called TICSA that requires network providers to intercept and store network traffic data for use by law enforcement.

NZNOG Lawful Intercept

I thought the requirements were pretty onerous, but I was told that the NZ government did set aside some funds to help providers with deploying solutions for collecting and storing this data (but I doubt it can cover the whole cost, especially over time). The new OpenNMS Drift telemetry project might be able to help with this.

NZNOG Aftab Siddiqui

There were a couple of talks I had seen in some form at AusNOG. The ever entertaining Aftab Siddiqui talked about MANRS (Mutually Agreed Norms for Routing Security) but unlike in Australia he was hard pressed to find good examples of violations. Part of that could be that New Zealand is much smaller than Australia, but I’m giving the NZ operators the credit for just doing a good job.

NZNOG NetNORAD

The Facebook folks were back to talk about their NetNORAD project. While I have a personal reluctance to deploy agents, there really isn’t a way to measure latency at the detail they want without them. I think it would be cool to be able to gather and manage the data created by this project under OpenNMS.

NZNOG Geoff Huston

What I like most about these NOG meetings is that I always learn something cool, and this one was no different. Geoff Huston gave a humorous talk on DNSSEC and handling DNS-based DDoS attacks. While I was somewhat familiar with DNSSEC, I was unaware of the NSEC part of it.

Most DNS DDoS attacks work by asking for non-existent domains, and the overhead in processing them is what causes the denial of service. The domain name is usually randomly generated, such as jeff123@example.com, jeff234@example.com, etc. If the DNS server doesn’t have the domain in its cache, it will have to ask another DNS server, which in turn won’t have the domain as it doesn’t exist.

The NSEC part of DNSSEC, when responding to a non-existent domain request, will return the next valid domain. In the example above, if I ask for jeff123@example.com, the example.com DNS server can reply that the domain is invalid and, in addition, the next valid domain is www.example.com. If implemented correctly, the original DNS server should then never query for jeff234@example.com since it knows it, too, doesn’t exist.

Pretty nifty.

NZNOG Rata Stanic

One talk I was eagerly awaiting was from Rada Stanic at Cisco. She also spoke at AusNOG but I had to leave early and missed it. While she disrespected SNMP a little more than I liked (grin), her talk was on implementing new telemetry-based monitoring protocols such as gRPC. OpenNMS Drift will add this functionality to the platform. Our experience so far is that the device vendor implementation of the telemetry protocols leaves something to be desired, but it does show promise.

NZNOG Ulf

It was nice being in New Zealand again, and our mascot Ulf seemed to be popular with the locals. Can’t imagine why.

2018 Linuxconf Australia Sysadmin Miniconf

I just wanted to put up a quick post on my trip to Linuxconf Australia (LCA) being held this week in Sydney.

First, a little background. I’ve been curtailing my participation in free and open source software conferences for the last couple of years. It’s not that I don’t like them, quite the opposite, but my travel is funded by The OpenNMS Group and we just don’t get many customers from those shows. A lot of people are into FOSS for the “free” (as in gratis) aspect.

Contrast that with telcos and network operators who tend to have the opposite viewpoint, if they aren’t spending a ton of money then they must be doing it wrong, and you can see why I’ve been spending more of my time focusing on that market.

Anyway, we have recently signed up a new partner in Australia to help us work with clients in the Pacific Rim countries called R-Group International, and I wanted to come out to Perth and do some training with their team. Chris Markovic, their Technical Director as well as being “mobius” on the OpenNMS chat server, suggested I come out the week after LCA, so I asked the LCA team if they had room on their program for me to talk about OpenNMS. They offered me a spot on their Sysadmin Miniconf day.

Linuxconf Australia Sign

The conference is being held at the University of Technology, Sydney (UTS) and I have to say the conference hall for the Sysadmin track was one of the coolest, ever.

Linuxconf Australia - UTS Lecture Hall

The organizers grouped three presentations together dealing with monitoring: one on Icinga 2, one from Nagios and mine on OpenNMS. While I don’t know much about Icinga, I do know the people who maintain it and they are awesome. One might think OpenNMS would have an antagonistic relationship with other FOSS monitoring projects, but as long as they are pure FOSS (like Icinga and Zabbix) we tend to get along rather well. Plus I’m jealous that Icinga is used on the ISS.

Linuxconf Australia Icinga2 Talk

I think my talk went well. I only had 15 minutes and for once I think I was a few seconds under that limit. While it wasn’t live-streamed it was up on YouTube very quicky, and you can watch it if you want.

I had to leave LCA to head to the New Zealand Network Operator’s Group (NZNOG) meeting, so I missed the main conference, but I am grateful the organizers gave me the opportunity to speak and I hope to return in the future.

Linuxconf Australia During a Break

Conferences: Australia, New Zealand and Senegal

Just a quick note to mention some conferences I will be attending. If you happen to be there as well, I would love the opportunity to meet face to face.

Next week I’ll be in Sydney, Australia, for linux.conf.au. I’ll only be able to attend for the first two “miniconf” days, and I’ll be doing a short introduction to OpenNMS on Tuesday as part of the Systems Administration Miniconf.

Then I’m off to Queenstown, New Zealand for the New Zealand Network Operators Group (NZNOG) conference. I will be the first presenter on Friday at 09:00, talking about, you guessed it, OpenNMS.

The week after that I will be back in Australia, this time on the other side in Perth, working with our new Asia-Pacific OpenNMS partner R-Group International. We are excited to have such a great partner bringing services and support for OpenNMS to organizations in that hemisphere. Being roughly 12 hours out from our home office in North Carolina, USA, can make communication a little difficult, so it will be nice to be able to help users in (roughly) their own timezone.

Plus, I hope to learn about Cricket.

Finally, I’m excited that I’ve been asked to do a one day tutorial at this year’s African Network Operators Group (AfNOG) in Dakar, Senegal, this spring. The schedule is still being decided but I’m eager to visit Africa (I’ve never been) and to meet up with OpenNMS users (and make some new ones) in that part of the world.

I’ll be posting a lot more about all of these trips in the near future, and hope to see you at at least one of these events.

Welcome to 2018

I love New Year’s. Not exactly the party on New Year’s Eve, as I tend to spend it as a quiet evening with friends, but the idea of starting over and starting fresh.

It is also a good time to reflect on the year past. While 2017 was pretty tumultuous for the world at large, for OpenNMS it was a pretty good year.

Our decision to split OpenNMS into two versions is still paying off. We did three major releases of Horizon (19, 20, and 21) as well as point releases every month there wasn’t a major release, and Meridian 2017 finally came out, although later than I would have liked. Horizon users get to experience rapid advancements in power and features while Meridian users can relax knowing their system is very stable and secure.

While it is hard to pick out the best features added in 2017, I’d have to go with OpenNMS Helm and the Minion.

Helm allows you to combine and manage multiple instances of OpenNMS from a Grafana dashboard.

OpenNMS Helm

The Minion is our foray into the whole “Internet of Things” space with an application that can be installed on a small device and used to send remotely collected data to a central OpenNMS instance. Minions have minimal configuration and can be configured redundantly, yet they have the ability to collect massive amounts of monitoring data. We’re very eager to see what novel uses our users come up with for the technology (we have one customer that is “Minion-only”, i.e. they do no monitoring or collection from the central OpenNMS instance at all and instead just put two Minions at each location).

As for the OpenNMS Group, the company behind OpenNMS, we experienced modest growth but still had a record year for gross revenue. What is more exciting is that net income was also a record and several hundred percent above last year, so we are going into 2018 well positioned in our Business Plan of “Spend less than you earn”.

2018 should be exciting. The OpenNMS Drift project brings telemetry (flow) data into OpenNMS, and we are working on some exciting features regarding correlation which will probably involve new machine learning technology.

As always, these features will be available as 100% free and open source software.

Personally, I added three new countries to my list, bringing the total number of countries I’ve been in to forty. I had a great time in Estonia and Latvia, and I really enjoyed my trip to Cuba.

One last thing. If you are reading this you are probably a user of OpenNMS. If so, thank you. We are a small but dedicated group of people creating this platform and often we don’t get much feedback on who uses it and what they like about it. The fact that people do find it useful makes it worthwhile, and we wouldn’t exist without our users and clients.

So, Happy New Year, and may 2018 exceed your wildest expectations.