UKNOF41

I love tech conferences, especially when I get to be a speaker. Nothing makes me happier than to be given a platform to run my mouth.

For the last year or so I’ve been attending various Network Operators Group (NOG) meetings, and I recently got the opportunity to speak at the UK version, which they refer to as a Network Operators Forum (UKNOF). It was a lot of fun, so I thought I’d share what I learned.

UKNOF41 was held in Edinburgh, Scotland. I’d never been to Scotland before and I was looking forward to the visit, but Hurricane Florence required me to return home early. I ended up spending more time in planes and airports than I did in that city, and totally missed out on both haggis and whisky (although I did drink an Irn-Bru). I arrived Monday afternoon and met up with Dr. Craig Gallen, the OpenNMS Project representative in the UK. We had a nice dinner and then got ready for the meeting on Tuesday.

Like most NOG/NOF events, the day consisted of one track and a series of presentations of interest to network operators. I really like this format. The presentations tend to be relatively short and focused, and this exposes you to concepts you might have missed if there were multiple tracks.

UKNOF is extremely well organized, particularly from a speaker’s point of view. There was a ton of information on what to expect and how to present your slides, and everything was run from a single laptop. While this did mean your slides were due early (instead of, say, being written on the plane or train to the conference) it did make the day flow smoothly. The sessions were recorded, and I’ll include links to the presentations and the videos in the descriptions below.

UKNOF41 - Keith Mitchell

The 41st UKNOF was held at the Edinburgh International Conference Centre, located in a newer section of the city and was a pretty comfortable facility in which to hold a conference. Keith Mitchell kicked off the the day with the usual overview of the schedule and events (slides), and then we got right into the talks.

UKNOF41 - Kurtis Lindqvist

The first talk was from Kurtis Lindqvist who works for a service provider called LINX (video|slides). LINX deployed a fairly new technology called EVPN (Ethernet VPN). EVPN is “a multi-tenant BGP-based control plane for layer-2 (bridging) and layer-3 (routing) VPNs. It’s the unifying L2+L3 equivalent of the traditional L3-only MPLS/VPN control plane.” I can’t say that I understood 100% of this talk, but the gist is that EVPN allows for better use of available network resources which allowed LINX to lower its prices, considerably.

UKNOF41 - Neil McRae

The next talk was from Neil McRae from BT (video|slides). While this was my first UKNOF I quickly identified Mr. McRae as someone who is probably very involved with the organization as people seemed to know him. I’m not sure if this was in a good way or a bad way (grin), probably a mixture of both, because being a representative from such a large incumbent as BT is bound to attract attention and commentary.

I found this talk pretty interesting. It was about securing future networks using quantum key distribution. Current encryption, such as TLS, is based on public-key cryptography. The security of public-key cryptography is predicated on the idea that it is difficult to factor large numbers. However, quantum computing promises several orders of magnitude more performance than traditional binary systems, and the fear is that at some point in the future the mathematically complex operations that make things like TLS work will become trivial. This presentation talked about some of the experiments that BT has been undertaking with quantum cryptography. While I don’t think this is going to be an issue in the next year or even the next decade, assuming I stay healthy I expect it to be an issue in my lifetime. It is good to know that people are working on solving it.

At this point in time I would like to offer one minor criticism. Both of the presenters thus far were obviously using a slide deck created for a purpose other than UKNOF. I don’t have a huge problem with that, but it did bother me a little. As a speaker I always consider the opportunity to speak to be a privilege. While I joke about writing the slides on the way to the conference, I do put a lot of time into my presentations, and even if I am using some material from other decks I make sure to customize it for that particular conference. Ultimately what is important is the content and not the deck itself and perhaps UKNOF is a little more casual than other such meetings, but it still struck me as, well, rude, to skim through a whole bunch of slides to fit the time slot and the audience.

UKNOF41 - Julian Palmer

After a break the next presentation was from Julian Palmer of Corero (video|slides). Corero is a DDOS protection and mitigation company, which I assume means they compete with companies such as Cloudflare. I am always fascinated by the actions of those trying to break into networks and those trying to defend them, so I really enjoyed this presentation. It was interesting to see how much larger the DDOS attacks have grown over time and even more surprising to see how network providers can deal with them.

UKNOF41 - Stuart Clark

This was followed by Stuart Clark from Cisco Devnet giving a talk on using “DevOps” technologies with respect to network configurations (video|slides). This is a theme I’ve seen at a number of NOG conferences: let’s leverage configuration management tools designed for servers and apply them to networking gear. It makes sense, and it is interesting to note that the underlying technologies between both have become so similar that using these tools actually works. I can remember a time when accessing network gear required proprietary software running on Solaris or HP-UX. Now with Linux (and Linux-like) operating systems underpinning almost everything, it has become easier to migrate, say, Ansible to work on routers as well as servers.

It was my turn after Mr. Clark spoke. My presentation covered some of the new stuff we have released in OpenNMS, specifically things like the Minion and Drift, as well as a few of the newer things on which we are actively working (video|slides). I’m not sure how well it was received, but number of people came up to me afterward and say they enjoyed it. During the question and answer session Mr. McRae did state something that bothered me. He said, basically, that the goal of network monitoring should be to get rid of people. I keep hearing that, especially from large companies, but I have to disagree. Technology is moving too fast to ever get rid of people. In just half a day I was introduced to technologies such as EVPN and quantum key distribution, not to mention dealing with the ever-morphing realm of DDOS attacks, and there is just no way monitoring software will ever evolve fast enough to cover everything new just to get rid of people.

Instead, we should be focusing on enabling those people in monitoring to be able to do a great job. Eliminate the drudgery and give them the tools they need to deal with the constant changes in the networking space. I think it is a reasonable goal to use tools to reduce the need to hire more and more people for monitoring, but getting rid of them altogether does not seems likely, nor should we focus on it.

I was the last presentation before lunch (so I finished on time, ‘natch).

UKNOF41 - Chris Russell

The second half of the conference began with a presentation by Chris Russell (video|slides). The title was “Deploying an Atlas Probe (the Hard Way)”, which is kind of funny. RIPE NCC is the Internet Registry for Europe, and they have a program for deploying hardware probes to measure network performance. What’s funny is that you just plug them in. Done. While this presentation did include discussion of deploying an Atlas probe, it was more about splitting out a network and converting it to IPv6. IPv6 is the future (it is supported by OpenNMS) but in my experience organizations are very slowly migrating from IPv4 (the word “glacier” comes to mind). Sometimes it takes a strong use case to justify the trouble and this presentation was an excellent case study for why to do it and the pitfalls.

UKNOF41 - Andrew Ingram

Speaking of splitting out networks, the next presentation dealt with a similar situation. Presented by Andrew Ingram from High Tide Consulting, his session dealt with a company that acquired another company, then almost immediately spun it back out (video|slides). He was brought in to deal with the challenges of dealing with a partially combined network that needed to be separated in a very short amount of time with minimal downtime.

I sat next to Mr. Ingram for most of the conference and learned this was his first time presenting. I thought he did a great job. He sent me a note after the conference that he has “managed to get OpenNMS up and running in Azure with an NSG (Network Security Gateway) running in front for security and a Minion running on site. It all seams to be working very nicely”

Cool.

UKNOF41 - Sara Dickinson

The following presentation would have to be my favorite of the day. Given by Sara Dickinson of Sinodun IT, it talked about ways to secure DNS traffic (video|slides).

The Internet wouldn’t work without DNS. It translates domain names into addresses, yet in most cases that traffic is sent in the clear. It’s metadata that can be an issue with respect to privacy. Do you think Google runs two of the most popular DNS servers out of the goodness of their heart? Nope, they can use that data to track what people are doing on the network. What’s worse is that every network provider on the path between you and your DNS server can see what you are doing. It is also an attack vector as well as a tool for censorship. DNS traffic can be “spoofed” to send users to the wrong server, and it can be blocked to prevent users from accessing specific sites.

To solve this, one answer is to encrypt that traffic, and Ms. Dickinson talked about a couple of options: DoT (DNS over TLS) and DoH (DNS over HTTPS).

The first one seems like such a no-brainer that I’m surprised it took me so long to deploy it. DoT encrypts the traffic between you and your DNS server. Now, you still have to trust your DNS provider, but this prevents passive surveillance of DNS traffic. I use a pfSense router at home and decided to set up DoT to the Quad9 servers. It was pretty simple. Of all of the major free DNS providers, Quad9 seems to have the strongest privacy policy.

The second protocol, DoH, is DNS straight from the browser. Instead of using a specific port, it can use an existing HTTPS connection. You can’t block it because if you do you’ll block all HTTPS traffic, and you can’t see the traffic separately from normal browsing. You still have to deal with privacy issues since that domain name has to be resolved somewhere and they will get header information, such as User-Agent, from the query, so there are tradeoffs.

While I learned a lot at UKNOF this has been the only thing I’ve actually implemented.

After a break we entered into the all too common “regulatory” section of the conference. Governments are adding more and more restrictions and requirements for network operators and these NOG meetings are often a good forum for talking about them.

UKNOF41 - Jonathan Langley

Jonathan Langley from the Information Commissioner’s Office (ICO) gave a talk on the Network and Information Systems Directive (NIS) (video|slides). NIS includes a number of requirements including things such as incident reporting. I thought it was interesting that NIS is an EU directive and the UK is leaving the EU, although it was stressed that NIS will apply post-Brexit. While there were a lot of regulations and procedures, it wasn’t as onerous as, say, TICSA in New Zealand.

UKNOF41 - Huw Saunders

This was followed by another regulatory presentation by Huw Saunders from The Office of Communications (Ofcom) (video|slides). This was fairly short and dealt primarily with Ofcom’s role in NIS.

UKNOF41 - Askar Sheibani

Askar Sheibani presented an introduction to the UK Fibre Connectivity Forum (video|slides). This is a trade organization that wants to deploy fiber connectivity to every commercial and residential building in the country. My understanding is that it will help facilitate such deployments among the various stakeholders.

UKNOF41 - David Johnston

The next to the last presentation struck a cord with me. Given by David Johnston, it talked about the progress the community of Balquhidder in rural Scotland is making in deploying its own Internet infrastructure (video|slides). I live in rural North Carolina, USA, and even though the golf course community one mile from my house has 300 Mbps service from Spectrum, I’m stuck with an unreliable DSL connection from CenturyLink, which, when it works, is a little over 10 Mbps. Laws in North Carolina currently make it illegal for a municipality to provide broadband service to its citizens, but should that law get overturned I’ve thought about trying to spearhead some sort of grassroots service here. It was interesting to learn how they are doing it in rural Scotland.

UKNOF41 - Charlie Boisseau

The final presentation was funny. Given by Charlie Boisseau, it was about “Layer 0” or “The Dirty Layer” (video|slides). It covered how cable and fiber are deployed in the UK. The access chambers for conduit have covers that state the names of the organizations that own them, and with mergers, acquisitions and bankruptcies those change (but the covers do not). While I was completely lost, the rest of the crowd had fun guessing the progression of one company to another. Anyone in the UK can deploy their own network infrastructure, but it isn’t exactly cheap, and the requirements were covered in the talk.

After the conference they served beer and snacks, and then I headed back to the hotel to get ready for my early morning flight home.

I had a lot of fun at UKNOF and look forward to returning some day. If you are a network provider in the UK it is worth it to attend. They hold two meetings a year, with one always being in London, so there is a good chance one will come near you at some point in time.

Open Source is Still Dead

Last week I attended the 20th O’Reilly Open Source Conference (OSCON), held this year in Portland, Oregon.

OSCON 20th Anniversary Sign

This is the premiere open source conference in the US, if not the world, and it is rather well run. It is equal to if not better than a lot of proprietary technology conferences I’ve attended, perhaps because it is pretty much a proprietary software conference in itself. I found it a little ironic that the Wednesday morning keynotes started off with a short, grainy video clip where an open source geek shouts out “We’re starting a revolution!”.

I tried to find the source of that quote, and I thought it came from the documentary “Revolution OS“. That movie chronicles the early days of open source software in which the stated goal was to take back software from large companies like Microsoft. There is a famous quote by Eric S. Raymond where he replies to a person from Microsoft with the words “I’m your worst nightmare.” Microsoft is now a major sponsor of OSCON.

When I attended OSCON in 2014 I asked the question “Is Open Source Dead?” Obviously the open source development model has never been more alive, but I was thinking back to my early involvement with open source where the idea was to move control of software out of the hands of big companies like IBM and Microsoft and into the hands of the users. Back then the terms “open source” and “free software” were synonymous. It was obvious that open source operating systems, mainly Linux, would rule the world of servers, so the focus was on the desktop. No one in open source predicted the impact of mobile, and by extension, the “cloud”. Open source today is no more than a development model used mostly to help create proprietary software, usually provided as a subscription or a service over the network. I mean, it makes sense. Companies like Google, Facebook and Amazon wouldn’t exist today if it wasn’t for Linux. If they had to pay a license to Microsoft or Sun (now Oracle) for every server they deployed their business models simply wouldn’t work, and the use of open source for building the infrastructure for applications simply makes sense.

Please note that I am not trying to make any sort of value judgement. I am still a big proponent of free software, and there are companies like Red Hat, OpenNMS and Nextcloud that try to honor the original intention of open source. All of us, open and proprietary, benefit from the large amount of quality open source software being created these days. But I do mourn the end of open source as I knew it. It used to be that open source software was published with “restrictive” licenses like the GPL, whereas now the trend is to move to “permissive” licenses like the MIT or Apache licenses. This allows for the commercialization of open source software, which in turn creates an incentive for large software companies to get involved.

This trend was seen throughout OSCON. The “diamond” sponsors were companies like IBM, Microsoft, Amazon and Google. The main buzzword was “Kubernetes” (or “K8s” if you’re one of the cool kids) which is an open source orchestration layer for managing containers. Almost all of the expo companies were cloud companies that either used open source software to provide a platform for their applications or to create open source agents that would feed back to their proprietary cloud back-end.

I attended my first OSCON in 2009 as a speaker, and I was a speaker for several years after that. My talks were always well-attended, but then for several years none of my paper submissions were accepted. I thought I had pissed off one or more of the organizers (it happens) but perhaps my thoughts on open source software had just become outdated.

I still like going to the conference, even though I no longer attempt to submit a talk. When I used to speak I found I spent most of my time on the Expo floor so now I just try to schedule other business during the week of OSCON and I get a free “Expo only” pass. You also get access to the keynotes, so I was sure to be in attendance as the conference officially started.

OSCON Badge

My favorite keynote was the first one, by Suz Hinton from Microsoft. She is known for doing live coding on the streaming platform Twitch, and she did a live demonstration for her keynote. She used an Arduino to control a light sensor and a servo. When she covered the sensor, the servo would move and “wave” at the OSCON audience. It was a little hard to fight the cognitive dissonance of a Microsoft employee using a Mac to program an open hardware device, but it was definitely entertaining.

OSCON Suz Hinton

My second favorite talk was by Camille Eddy. As interactions between computers and humans become more automated, a number of biases are starting to appear. Google image search had a problem where it would label pictures of black people as “gorillas”. An African-American researcher at MIT named Joy Buolamwini found that a robot recognized her better if she wore a white mask. Microsoft had an infamous experiment where it created a Twitter bot named “Tay” that within 24 hours was making racist posts. While not directly related to open source, a focus on an issue that affects the user community is very much in the vein of classic open source philosophy.

OSCON Camille Eddy

The other keynotes were from Huawei, IBM and Amazon (when you are a diamond sponsor you get a keynote) and they focused more on how those large software companies were using the open source development model to, well, offset the cost of development.

OSCON Tim O'Reilly

The Wednesday keynotes closed with Tim O’Reilly who talked about “Open Source and Open Standards in the Age of Cloud AI”. It kind of cemented the theme for me that open source had changed, and the idea is now much more about tools development and open APIs than in creating user-owned software.

OSCON Expo Floor

The rest of my time was spent wandering the Expo floor. OSCON offers space to traditional open source projects which I usually refer to as the “Geek Ghetto”. This year it was split to be on either side of the main area, and I got to spend some time chatting with people from the Software Freedom Conservancy and the Document Foundation, among others.

OSCON Geek Ghetto

I enjoyed the conference, even if it was a little bittersweet. Portland is a cool town and the people around OSCON are cool as well. If I can combine the trip with other business, expect to find me there next year, wandering the Expo floor.

2018 New Zealand Network Operators Group (NZNOG)

One thing that all open source projects struggle with is getting users. Most people in IT and software are overwhelmed with a plethora of information and options, and matching the right material to the right audience is a non-trivial problem.

Last year my friend Chris suggested that I speak at a Network Operators Group (NOG) meeting, specifically AusNOG. It was a lot of fun. I felt very comfortable among this crowd. so I decided to reach out to more NOGs to see if they would be interested in learning about OpenNMS.

The thing I like the most about NOGs is that they value getting things done above all else. While “getting things done” is still important with the free and open source crowd, there seems to be more philosophy and tribalism at those shows. “Oh, that’s written in PHP, it must suck” etc. As a “freetard” I live for the philosophical and social justice aspects of the community, but from a business standpoint it doesn’t translate well into paying customers.

At NOGs the questions are way more business-focused. Does it work? Is it supported? What does it cost? While I’m admittedly biased toward OpenNMS and its open source nature, the main reason I keep promoting it is that it just makes solid business sense for many companies to use it instead of their current solution.

Plus, these folks are pretty smart and entertaining while dispensing solid advice and knowledge.

Anyway, with that preamble, at AusNOG I learned about the New Zeland NOG (NZNOG) and submitted a talk. It got accepted and I found myself in Queenstown.

NZNOG Scenary

The main conference was spread out over two days, and like AusNOG it consisted of 30 to 45 minute talks in one track.

While I know it won’t work for a lot of conferences, I really like the “one track” format. It exposes me to things I wouldn’t have gone to otherwise, and if there is something I am simply not interested in learning about I can use that time to catch up on work or participate in the hallway track.

NZNOG Clare Curran

The conference started with a presentation by the Honorable Clare Curran, a newly minted Member of Parliament (they recently held elections in New Zealand). I’m slowly seeing politicians getting more involved in information technology conferences, which I think is a good thing, and I can only hope it continues. She spoke about a number of issues the government is facing with respect to communications technology.

Several things bother me about the US government, but one big one is the lack of understanding of the importance of access to the Internet at broadband speeds. Curran stated that “lack of reliable high-speed network access is a new measure of poverty”. Later in the day John Greenhough spoke on New Zealand’s Ultra-Fast Broadband (UFB) project, where on one slide broadband was defined as 20Mbps download speed.

NZNOG John Greenhough

Where I live in the US I am lucky to get 10Mbps and many of my neighbors are worse off, yet the government is ceding more of the decision making process about where to build out new infrastructure to the telecommunications companies which have zero incentive to improve my service. It’s wonderful to see a government realize the benefits of a connected populace and to take steps to make it happen.

Because we all need Netflix, right? (grin)

There was a cool talk about how Netflix works, and I didn’t realize that they are working with communications providers to provide low-latency solutions distributed geographically. This is done by supplying providers with caching content servers so that customers can access Netflix content while minimizing the need for lots of traffic over expensive backhaul links.

NZNOG Netflix RRD

I did find it cool that one of the bandwidth graphs presented was obviously done using RRDtool. I don’t know if they collected the data themselves or used something like OpenNMS, but I hope it was the latter.

With this push for ubiquitous network access comes other concerns. New Zealand has a law called TICSA that requires network providers to intercept and store network traffic data for use by law enforcement.

NZNOG Lawful Intercept

I thought the requirements were pretty onerous, but I was told that the NZ government did set aside some funds to help providers with deploying solutions for collecting and storing this data (but I doubt it can cover the whole cost, especially over time). The new OpenNMS Drift telemetry project might be able to help with this.

NZNOG Aftab Siddiqui

There were a couple of talks I had seen in some form at AusNOG. The ever entertaining Aftab Siddiqui talked about MANRS (Mutually Agreed Norms for Routing Security) but unlike in Australia he was hard pressed to find good examples of violations. Part of that could be that New Zealand is much smaller than Australia, but I’m giving the NZ operators the credit for just doing a good job.

NZNOG NetNORAD

The Facebook folks were back to talk about their NetNORAD project. While I have a personal reluctance to deploy agents, there really isn’t a way to measure latency at the detail they want without them. I think it would be cool to be able to gather and manage the data created by this project under OpenNMS.

NZNOG Geoff Huston

What I like most about these NOG meetings is that I always learn something cool, and this one was no different. Geoff Huston gave a humorous talk on DNSSEC and handling DNS-based DDoS attacks. While I was somewhat familiar with DNSSEC, I was unaware of the NSEC part of it.

Most DNS DDoS attacks work by asking for non-existent domains, and the overhead in processing them is what causes the denial of service. The domain name is usually randomly generated, such as jeff123@example.com, jeff234@example.com, etc. If the DNS server doesn’t have the domain in its cache, it will have to ask another DNS server, which in turn won’t have the domain as it doesn’t exist.

The NSEC part of DNSSEC, when responding to a non-existent domain request, will return the next valid domain. In the example above, if I ask for jeff123@example.com, the example.com DNS server can reply that the domain is invalid and, in addition, the next valid domain is www.example.com. If implemented correctly, the original DNS server should then never query for jeff234@example.com since it knows it, too, doesn’t exist.

Pretty nifty.

NZNOG Rata Stanic

One talk I was eagerly awaiting was from Rada Stanic at Cisco. She also spoke at AusNOG but I had to leave early and missed it. While she disrespected SNMP a little more than I liked (grin), her talk was on implementing new telemetry-based monitoring protocols such as gRPC. OpenNMS Drift will add this functionality to the platform. Our experience so far is that the device vendor implementation of the telemetry protocols leaves something to be desired, but it does show promise.

NZNOG Ulf

It was nice being in New Zealand again, and our mascot Ulf seemed to be popular with the locals. Can’t imagine why.

2018 Linuxconf Australia Sysadmin Miniconf

I just wanted to put up a quick post on my trip to Linuxconf Australia (LCA) being held this week in Sydney.

First, a little background. I’ve been curtailing my participation in free and open source software conferences for the last couple of years. It’s not that I don’t like them, quite the opposite, but my travel is funded by The OpenNMS Group and we just don’t get many customers from those shows. A lot of people are into FOSS for the “free” (as in gratis) aspect.

Contrast that with telcos and network operators who tend to have the opposite viewpoint, if they aren’t spending a ton of money then they must be doing it wrong, and you can see why I’ve been spending more of my time focusing on that market.

Anyway, we have recently signed up a new partner in Australia to help us work with clients in the Pacific Rim countries called R-Group International, and I wanted to come out to Perth and do some training with their team. Chris Markovic, their Technical Director as well as being “mobius” on the OpenNMS chat server, suggested I come out the week after LCA, so I asked the LCA team if they had room on their program for me to talk about OpenNMS. They offered me a spot on their Sysadmin Miniconf day.

Linuxconf Australia Sign

The conference is being held at the University of Technology, Sydney (UTS) and I have to say the conference hall for the Sysadmin track was one of the coolest, ever.

Linuxconf Australia - UTS Lecture Hall

The organizers grouped three presentations together dealing with monitoring: one on Icinga 2, one from Nagios and mine on OpenNMS. While I don’t know much about Icinga, I do know the people who maintain it and they are awesome. One might think OpenNMS would have an antagonistic relationship with other FOSS monitoring projects, but as long as they are pure FOSS (like Icinga and Zabbix) we tend to get along rather well. Plus I’m jealous that Icinga is used on the ISS.

Linuxconf Australia Icinga2 Talk

I think my talk went well. I only had 15 minutes and for once I think I was a few seconds under that limit. While it wasn’t live-streamed it was up on YouTube very quicky, and you can watch it if you want.

I had to leave LCA to head to the New Zealand Network Operator’s Group (NZNOG) meeting, so I missed the main conference, but I am grateful the organizers gave me the opportunity to speak and I hope to return in the future.

Linuxconf Australia During a Break

Conferences: Australia, New Zealand and Senegal

Just a quick note to mention some conferences I will be attending. If you happen to be there as well, I would love the opportunity to meet face to face.

Next week I’ll be in Sydney, Australia, for linux.conf.au. I’ll only be able to attend for the first two “miniconf” days, and I’ll be doing a short introduction to OpenNMS on Tuesday as part of the Systems Administration Miniconf.

Then I’m off to Queenstown, New Zealand for the New Zealand Network Operators Group (NZNOG) conference. I will be the first presenter on Friday at 09:00, talking about, you guessed it, OpenNMS.

The week after that I will be back in Australia, this time on the other side in Perth, working with our new Asia-Pacific OpenNMS partner R-Group International. We are excited to have such a great partner bringing services and support for OpenNMS to organizations in that hemisphere. Being roughly 12 hours out from our home office in North Carolina, USA, can make communication a little difficult, so it will be nice to be able to help users in (roughly) their own timezone.

Plus, I hope to learn about Cricket.

Finally, I’m excited that I’ve been asked to do a one day tutorial at this year’s African Network Operators Group (AfNOG) in Dakar, Senegal, this spring. The schedule is still being decided but I’m eager to visit Africa (I’ve never been) and to meet up with OpenNMS users (and make some new ones) in that part of the world.

I’ll be posting a lot more about all of these trips in the near future, and hope to see you at at least one of these events.

2017 Cubaconf

I’ve just returned from Cubaconf in Havana, which was also my first visit to Cuba. It was a great trip and I’ve got enough material for at least four blog posts. Most of them won’t deal with free and open source software, so I’ll put them up on my personal blog and I’ll add links here when they are done..

Cubaconf is in its second year, and while I really wish they would have called it “Cuba Libré” (grin) it was a good conference.

There is a spectrum within the Free, Libré, Open Source Software (FLOSS) community, and this is often described by trying to separate the term “open source” from “free software”. If we define “open source” as any software with a license that meets requirements of the Open Source Definition (OSD) and “free software” as any software with a license that meets the requirements of the Four Freedoms, they are the same. You can map the ten requirements of the OSD onto the four requirements of free software.

Open Source is Free Software Chart

However, it can be useful to separate those who look at FLOSS as simply a development methodology from those who view it as a social movement. When companies like Microsoft and Facebook publish open source software, they are simply looking to gain value for their business that such sharing can create. It’s a development methodology. When people talk about free software, they tend to focus on the “help your neighbor” aspect of it, and this was more the focus of Cubaconf than simply creating new code.

The main thing I discovered on my visit was that Cubans face severe limitations on many things, but I’ve never met a people more determine to do as much as they can to make their situation better, and to do it with such passion. If I had to pick a theme for the conference, that would be it: passionate continuous improvement.

Cubaconf Registration

The three day conference had the following structure: Day One was a standard conference with keynotes and five tracks of presentations, Day Two had keynotes and more of a “barcamp” organization, and Day Three was set aside for workshops, as well as the obligatory video game tournament.

They did have the best lanyard sponsor I’ve seen at a technology conference:

Cubaconf Lanyard and Badge

I was in Cuba with my friend and coworker Alejandro, who used to live in Venezuela and is a fluent Spanish speaker, and Elizabeth K. Joseph, who promotes the open source aspects of Mesosphere. We shared a three bedroom “casa particular” in Old Havana, about a ten minute walk from the conference, which was held on the second floor (third floor if you are American) of the Colegio de San Gerónimo.

Everyone was together on the main room for the first keynote.

Cubaconf People in Room

While both English and Spanish were spoken at the conference, the presentations were overwhelmingly in Spanish, which was to be expected. I can get by in Spanish, but the first speaker, Ismael Olea, spoke fast even for the native speakers. At least I could understand most of the content in his slides.

Cubaconf Ismael Olea

Olea is from Spain and did a keynote on HackLab Almería. Almería is a province in the southeastern part of Spain, and with a population of around 700,000 people it is much smaller than provinces like Madrid (6.5 million) and Barcelona (5.5 million). As such, the region doesn’t get as much attention as the larger provinces, and so they goal of Hacklab Almería is to use technology at the “hyperlocal” level. They define themselves as a “collective of technological , social and creative experimentation” and FLOSS plays a large role in their mission.

After the keynote, we broke up into individual sessions. I went to one called “How to Make Money with Free Software” presented by Valessio Brito from Brazil. While he spoke mainly in Spanish, his slides were in Portuguese, but I was able to follow along. His presentation focused on how he used his knowledge of FLOSS to get consulting engagements around the world. This was pretty topical since in Cuba, as elsewhere, having strong software skills can be lucrative, and since a lot of proprietary software is either impossible to get or too expensive, having skills in open source software is a plus.

Cubaconf Valessio Brito

Also, I liked his shirt.

Our OpenNMS presentation was in the next time slot. I asked our hosts if they would like the presentation in English or Spanish, and when they said Spanish I asked Alejandro to give it. He did a great job, even though he had only a short time to understand the slides.

Cubaconf Alejandro Galue

The lunch break came next, and we walked a couple of blocks to the Casa de Africa, a museum dedicated to the African influence in Cuban culture.

Cubaconf Lunch Break

We ate sandwiches and talked out on the patio. This would be the location for lunch for all three days.

Cubaconf Maira Sutton

After lunch I watched a presentation by Maira Sutton called “Fighting Cyber Dystopia with Tech Solidarity and the Digital Commons” which is a long way to express the idea of using free software combined with working together to take back some of the power from large corporations. Her main example talked about the city of Austin, Texas, and its fight with Uber and Lyft. Austin wanted all ride sharing drivers to have to undergo a background check that included fingerprints. Sounds reasonable, but Uber and Lyft resisted, eventually leaving the city.

However, services like Uber and Lyft can be downright useful, so a number of startups filled in the gap, offering similar services that met the City’s fingerprint requirements. Instead of acquiescing to local laws, Uber and Lyft took their fight to the State legislature, which overturned Austin’s regulation.

Even though it is a sad ending, the example did demonstrate that combining technology and social action can result in solutions that can meet or exceed those provided by large commercial companies.

Cubaconf First Night's Event

For each night of the conference there was an event, and the one for Tuesday was held at a modern art gallery on the southern side of Old Havana. There was lots of food and drink, and I got exposed to a project called cuban.engineer. This is a group to promote technology careers within Cuba, and I had seen their shirts at the conference.

Cubaconf cuban.engineer shirts

In a lot of the world we take Internet access for granted. I can remember accessing the Internet from the night market in Siem Reap, Cambodia, on an open wi-fi connection. That doesn’t exist in Cuba. Cuba is one of the most disconnected countries in the world, which can make working with technology difficult. Access is controlled by an agency called ETESCA. To access the Internet you purchase a card which offers a certain number of hours of use, and then you have to locate an area with a wi-fi hotspot (usually near a park). The card has a number of digits for a username and a number of digits as a password, and once you get connected you hope you stay connected long enough to do what you need to do.

No one is exempt from this. Even in our apartment the owner would use one of these cards to enable access for the hotspot on the ground floor. So, if you are a technology business in Havana, your first job is to located your office near a hotspot, and then buy a bunch of these cards.

Thus you can imagine that sharing in a big part of the culture. People burn and swap CDs with software such as Ubuntu on them, and they tend to use Gitlab to make local mirrors of code repositories. While wi-fi equipment can be hard to come by, people have been able to set up their own, private wi-fi networks within cities like Havana to make sharing easier. There is no Internet access (I joked that it was Cuba’s “dark web“) but they can set up tools like Rocket.Chat to communicate and share.

Despite limitations in acquiring software, Microsoft Windows is still the most common operating system running on Cuban computers. An attempt was made to create a Cuba focused Linux-based distro called Nova. I was told that they even experimented with making it look as close to Windows 7 as possible, but people were still tied to using Windows. According to Wikipedia this distro is no more, which is a shame.

Cubaconf Mixæl Laufer

The second day started with the meter pegged at full on social justice, with a presentation by Dr. Mixæl S. Laufer, Ph.D., from Four Thieves Vinegar. They are a collective aiming to share information on how to create pharmaceuticals in places where they might not be available. If you live in the US than you probably heard of Martin Shkreli who as CEO of Turing Pharmaceuticals raised prices 5600%, and EpiPen maker Mylan who raised the price of this life saving device several hundred percent just because they could. Laufer showed how you could make your own EpiPen for around $30, among other things.

Now drug companies will say that they have to charge that amount to cover the costs of creating new drugs, but the EpiPen greed demonstrated that wasn’t true. Running health care as a “for profit” enterprise has always seemed inherently wrong due to the incentives being more toward making money versus keeping people healthy, but that is commentary for another time and place.

I had to leave after that presentation for something special. I make classic cocktails as a hobby, and one of our hosts asked me to speak to a school for bartenders (cantineros) on the great tradition of Cuba cocktails. It was a blast and I’ll write that up soon.

Cubaconf Wednesday Event

Wednesday night’s event was held, appropriately, at a bar in an area called Barber’s Alley. It was a fun gathering and I got a nice picture of some of our hosts.

Cubaconf Hosts

Left to right is Jessy, Pablo, pb, and Adalberto.

There was also a guy there who made pipes, specifically replicas of native American peace pipes, and one was passed around.

Cubaconf Peace Pipe

The third and final day was a series of workshops, but was started with a keynote from Ailin Febles, from the Uniōn de Informāticas de Cuba, a non-profit organization to bring together “all technicians, professionals and people related to information and communication technologies in a space that enables mutual support of the associates in the achievement of their professional, academic, scientific, cultural and personal objectives”.

Cubaconf Ailin Febles

Of course, a lot of their organization is driven by open source software.

Cubaconf Software used by UIC

I hope they switch to Nextcloud from Owncloud soon.

There was one morning workshop in English, ironically by a German named Christian Weilbach, on machine learning. I was interested in the topic since I keep hearing about it lately, and the fact that I would probably be able to understand it was a plus. To me machine learning is magic, and I wanted to dispel some of that magic.

Cubaconf Christian Weilbach

It worked. It turns out that machine learning is, to a large extent, what we used to call linear algebra. It just is able to work on much larger and more complex data sets. I’m still eager to play more with this technology, but it was nice to learn that it really isn’t all that new.

Cubaconf Old Car Taxi

After lunch we decided to spend our last afternoon exploring Havana a bit.

Cubaconf Brewery Event

The final evening event was in a brewery, and I enjoyed the beer. What I enjoyed more was the opportunity to talk with Inaury about race in Cuba. Cubans come in all shapes and sizes, from people with light skin, blond hair and blue eyes to people so dark they are almost blue, yet they all seem to interact and socialize with each other more so than any other place I’ve been. I plan to chat more about that in a blog post as well.

Overall I had a great time in Cuba. I love the fact that working in free software means I can make new friends in almost any country, and that even a place with limited resources can put on a great conference. If you get a chance to go to Cubaconf, you should take it.

2017 Ohio Linuxfest

The Ohio Linuxfest was one of the first open source conferences I ever attended, way back in 2010. I had heard about it from the local Linux Users Group, TriLUG, and we ended up renting a van and taking a couple of other open source geeks with us, including “Mr. IPv6” Kevin Otte.

It was a blast.

Ohio Linuxfest Banner

This year looks like another great show, with one of my favorite people, Karen Sandler, giving a keynote on Friday and yours truly will be giving the last keynote on Saturday.

If you are into free and open source software and are able to make it, I strongly encourage you to check out the conference. You’ll be glad you did.

2017 Australian Network Operators Group Conference

Back in June I was chatting with “mobius” about all things OpenNMS. He lives and works in Perth, Australia, and suggested that I do a presentation at AusNOG, the Australian Network Operators Group.

One of the things we struggle with at OpenNMS is figuring out how to make people aware it exists. My rather biased opinion is that it is awesome, but a lot of people have never heard of it. To help with that I used to attend a lot of free and open source conferences, but we’ve found out over the years that our user base tends to be more along the lines of large enterprises and network operators that might not be represented at such shows.

Imagine my surprise when I found out that there were a whole slew of NOGs, network operator groups, around the world. It seems to me that people who attend these conferences have a more immediate need for OpenNMS, and with that in mind I submitted a talk to AusNOG. I was very happy it was selected, not in the least because I would get to return to Australia for a third time.

AusNOG David Hughes

I wasn’t sure what to expect, but was pleasantly surprised. The event was extremely well organized and I really liked the format. Many conferences as they become successful respond by adding multiple tracks. This can be useful if the tracks are easy to delineate, but often you can get “track bloat” where the attendees get overwhelmed with choice and as a presenter you can end up with a nearly empty room if you are scheduled against a popular speaker. At AusNOG there is only one track of somewhat short, highly curated talks that results in a very informative conference without the stress of trying to determine the best set of talks to attend.

AusNOG Program

(Note: Visit the “programme” site and click on talk titles to download the presentations)

The venue was very nice as well. Held at the Langham Hotel, the conference took place in a ballroom that held the 200+ people with a lobby out front for socializing and a few sponsor booths. The program consisted of 90 minutes of presentations separated by a break. They alternated sets of three 30 minute talks with two 45 minutes talks. I found all of the presentations interesting, but I have to admit that I spent a lot of time looking up unfamiliar acronyms. As network operators Autonomous System (AS) numbers were thrown around in much the same way SNMP Private Enterprise Numbers are shared among network management geeks. Australia is also in the process of implementing a nationwide National Broadband Network (nbn™) to provide common infrastructure across the country, so of course that was the focus of a number of talks.

In the middle of each day we broke for lunch which was pretty amazing. The Langham restaurant had a sushi section, a section for Indian food, a large buffet of your standard meat and veg, and at least three dessert sections: one with “healthy” fruit and cheese, another focused on ice cream and a chocolate fountain, and a large case full of amazing pastries and other desserts. All with a view out over the Yarra river.

AusNOG Yarra River

I really liked the format of AugNOG and suggest we adopt it for the next OpenNMS conference. For those few talks that were either over my head or not really of interest, they were over pretty quickly, but I found myself enjoying most of them. I thought it was interesting that concepts we usually equate with the managing servers are being adopted on the network side. One talk discussed topics such as running switch software in containers, while another discussed using Ansible and Salt to manage the configuration of network gear.

AusNOG Runing Switch Software in Containers

I was happy to see that my talk wasn’t the only one that focused on open source. Back fifteen years ago getting large companies to adopt an open source solution was still in the evangelical stage, but now it is pretty much standard. Even Facebook presented a talk on their open source NetNORAD project for monitoring using a distributed system to measure latency and packet loss.

I did have a few favorite talks. In “The Future Is Up in the Sky” Jon Brewer discussed satellite Internet. As someone who suffered for years with a satellite network connection, it was interesting to learn what is being done in this area. I used a system with a satellite in geosynchronous orbit which, while it worked, ended up with latency on the order of a second in real-world use. It turns out that there are a number of solutions using shorter distances with satellites in low earth or high earth orbit. While they present their own challenges, it is still the most promising way to get network access to remote areas.

Another talk by Mark Nottingham discussed issues associated with the increased use of encrypted protocols and the challenges they create for network operators. While the civil libertarian in me applauds anything that makes it harder for surveillance to track users, as a network monitoring guy this can make it more difficult to track down the cause of network issues.

And this will become even more important as the network changes with the adoption of Internet of Things (IoT) devices. Another good talk discussed the issue of IoT security. Even today the main consensus is that you protect weakly secured devices with a firewall, but a number of new exploits leverage infected systems within the firewall for DDoS attacks.

AusNOG Internet of Things Vulnerabilities

I think my own talk went well, it was hard to squeeze a good introduction to OpenNMS into 30 minutes. I did manage it – 30 minutes on the nose – but it didn’t leave time for questions. As a speaker I really liked the feedback the conference provided in the form of a rather long report showing what the attendees thought of the talk, complete with cool graphs.

AusNOG Speaker Response

I really enjoyed this conference, both as an attendee and a speaker. While I hope to speak to more NOGs I would much rather encourage OpenNMS users who are happy with the project to submit real-world talks on how they use the platform to their local tech groups. I think it tells a much stronger story to have someone a little less biased than myself talking about OpenNMS, and plus you get to visit cool conferences like AusNOG.

2017 Australian Network Operators Group (AusNOG) Conference

I am excited to be returning to Australia for the third time next week. This trip is to speak at the Australian Network Operators Group (AusNOG) annual conference in Melbourne.

AusNOG Promo Graphic

I can’t believe I’ve gone for so long and not known about Network Operator Groups (NOGs). There are quite a few of them and I think they would be a perfect audience to introduce to the OpenNMS Project. One of our users on the OpenNMS chat server is from Perth and he made me aware of the conference, and I was humbled and delighted to have my presentation accepted.

At OpenNMS we strive very hard to separate the project (.org) from the commercial entity that supports the project (.com) and this presentation will be strictly focused on the project. It’s a wonderful thing about OpenNMS: if it meets your needs, cool. If not, also cool. I just want more people to be aware of open source options, especially in the carrier and enterprise space.

And it looks like open source is definitely making inroads at AusNOG. The talk before mine is about Ansible and Salt. There is another talk on using open source to build a version of NetNorad, and another one on open source for big data analytics. The one after mine is about modern network monitoring, so I hope I tick at least a few boxes on his list.

I hope to see you there (although it looks like it is sold out) but let me know if you are in the area and perhaps I can at least say “hi”.

2017 Dev-Jam – Days 4 and 5

Apologies to my three readers for the two-plus week delay in writing this up. I know you’ve been waiting for this post with more anticipation than Season 7 of Game of Thrones, but things have been crazy busy in OpenNMS-land of late.

As you know, this year’s Dev-Jam was held at Concordia University in Montréal. For most of the week we had access to a nice, large space which included air conditioners (the Grey Nuns building does not have central air), but due to a scheduling conflict we had to move out for the day on Thursday.

We moved to the basement cafeteria, which worked out rather well.

Dev-Jam: Grey Nuns Cafeteria

There were no A/C units but with the windows open there was a good cross breeze and it was comfortable. By the fourth day of Dev-Jam people are pretty much settled into a routine, so the day was mostly spent working to finish up various projects.

On Friday we moved back upstairs. The last full day of Dev-Jam is always bittersweet for me, as I love the “show and tell” part where people share what they have been working on, but I hate the fact that it is over for another year. We also forgot to bring the equipment we usually use for video capture (it was left back in Minnesota) so while the sessions were recorded, they haven’t been posted to Youtube yet.

Dev-Jam: Jesse White presents on Minion

Jesse kicked off the presentations talking about work he was doing to assign specific monitoring tasks to particular Minions.

Dev-Jam: Markus presents on Doughnut Graphs

Markus followed that with his work on adding “doughnut” graphs to the user interface. These resemble the graphs available with Compass™, our mobile app. That is supposed to be a green doughnut and not a grey one but the projector didn’t render it very well.

Dev-Jam: Christian presents on IFTTT

As OpenNMS wants to be the monitoring platform of choice for the Internet of Everything, Christian did some work on integrating it with “If This, Then That” (IFTTT)

Dev-Jam: Ronny presents on ASCIIBinder

Ronny is our “documentation czar” who led the effort to create the most awesome docs.opennms.org site. He explored using ASCIIBinder to manage our growing collection of documentation.

Dev-Jam: Seth presents on ReST

I work with some amazing people, and years ago they saw the potential in adding ReST functionality to OpenNMS. It was a great decision and makes OpenNMS extremely flexible when integrating with other systems. Seth presented some of the work he is doing to extend that feature.

Dev-Jam: Alejandro presents on Drools

OpenNMS has a couple of ways to correlate alarms. The basic method is using the “vacuumd” configuration and SQL, but a stronger (although more complex) method is to used the Drools business intelligence engine. Alejandro presented some work he is doing to move some of the legacy vacuumd tasks to Drools.

Dev-Jam: Jeff presents on CLA Assistant

Contributor License Agreements are a controversial topic in open source, but we use one for OpenNMS. The main reason is to ask any contributor to certify that they have the right to contribute the code. It may seem trivial, but not only does signing such an agreement make the person think about it, it does give the project some cover in case of a dispute.

We currently manage our own CLAs, but the website CLA Assistant aims to make it easier. Jeff presented on what he found out about the service, and we may be migrating to it in the future.

Dev-Jam: Antonio presents on Enhanced Linkd

And last but not least, Antonio talked about the work he is doing on Enhanced Linkd. This is the process that figures out Layer 2 connections between devices. It is non-trivial as vendors seems to relate this information in different ways, and we really appreciate the time he has put into that part of the project.

It was great seeing everyone again, and it was also cool to hold the conference in a new location (well, cool once it was over, it added a lot of stress to my life). I’m eager to start planning next year’s conference.