Freedom Feud

My official title at OpenNMS is CEO, but I’ve worn several hats in the last 12+ years, including accountant, receptionist, HR manager and janitor. Now I get to add record producer to the list.

I guess it all started back in 2012. OpenNMS was doing pretty well and I wanted us to give a little something back to the community. As a fan of MC Frontalot I came across his FAQ and found out that you could actually book him for things like conferences, kids parties, bar mitzvahs and the like. We were sponsoring the Ohio Linuxfest (by the way, I’m a keynote speaker there this year along with the ever amazing Karen Sandler) and I decided to see if I could book him to play a show. Turns out he is pretty affordable (for contrast, Henry Rollins starts out at $10,000 per event, which isn’t unreasonable but doesn’t count as affordable for us just yet). I booked him to play a solo gig and finally got to meet the man. He did a great show, everyone seemed to enjoy it, and we became friends of a sort.

While Front is very much a nerd, he had not had much exposure to free software. A lot of musicians rely on Windows-based software to create their music (Front mainly uses Reaper and Professor Kliq is such an Ableton fan he has their logo tattooed on his wrists) and thus they aren’t used to using open source. The OLF event went so well I hired him a few more times, and I think it was at SCaLE when I suggested he write a free software song. His reply was, well, why don’t you commission one.

Front is talent for hire. He did a couple of tracks for New Relic, “Nerd Life” and “Small Data“, and while we don’t have anywhere near the budget of that company we felt that free software deserved to be examined under his lyrical microscope, so we started the process.

Note that this was a couple of years ago, back in 2015, so it took awhile. Front and I had a number of conversations about free software and I started him on his education. I pointed him to the works of Richard Stallman and Eric S. Raymond (notably The Cathedral and the Bazaar), as well as Lawrence Lessig and organizations such as the Free Software Foundation (FSF) and the Electronic Frontier Foundation (EFF). We also talked about the classic “free as in beer” vs. “free as in freedom” confusion that arises out of the term “free software”, which ended up forming the basis of the song.

Later in 2015 I wanted to do something special for the OSCON show in Portland. This time I decided to hire Front along with his band. In the previous shows he’d done for us it was him and “DJ CPU” providing the music, and while those were great shows I was unprepared for the “live band” experience. It took it to another level. During that show Front performed a bit of the song, but it wasn’t until last year’s All Things Open that the whole song was played for the first time (again with the band, since, awesome).

With the song almost complete we now how to figure out how to present it to the world. I wanted a video, so I decided to turn to animator Chad Essley. Chad had done the video for “Shudders” off of Question Bedtime and we had gotten to know each other through a fund raising promotion he did for the EFF where we sponsored adding OpenNMS references to that video. I felt he could do a good job with it, since he is both talented and he gets the subject matter.

Now when I said “record producer” above I basically meant I signed the checks, but it was cool watching artists such as Front and Chad work out even small details when it came to the video. Animation can take a really long time, so we debated on combining it with some live action, etc., to both speed up the process and reduce the cost. It was at this point that I was introduced to the concept of a “lyric video”.

Ed Sheeran had just come out with a new song, and in order to promote it as quickly as possible he released a video that pretty much consisted of just the song’s lyrics. While at this point in our process we had a portion of the animation completed, I thought that adding the lyrics to it would both speed things up as well as improve the experience, as Front’s rhymes on this track are some of the tightest he’s ever written. The end result is both a video that is fun to watch as well as one that gets the message across in an entertaining manner.

It seems to have been well received, and as I write this it has had over 10,000 views and mostly positive comments, and we got a shout out on Boing Boing.

Speaking of lyrics, the phrase that has gotten the most comments is the line “Pull down capitalism till it’s rubble and chain”. When asked about it, Front commented:

I thought it was funny to equate Free Software with that dirty commie yearning for collectivized well-being and shared responsibility! Those two realms of thought are not directly in line with each other as far as I can tell. But I imagined the kind of business dude who is reflexively distrustful of free software, and I pictured him saying, “you mean… LIKE COMMUNISM??” Thought I’d give him a little dig at the end there.

I can’t remember if I shared my story with Front when we were first talking about creating this song, but when I got started with OpenNMS I was called a communist a number of times on various message boards. It confused me, since it came mainly from people who made their living as network management consultants. I was thinking, hey, here is a tool that lets you provide better solutions for your customers while showcasing your unique talents via your ability to deliver them, and that’s communism? Sounds like good business to me. But there is still that element of “anti-capitalism” associated in free software (I blame the phrase “so you can help your neighbor” in the Free Software Definition, but that’s just me).

The whole process was a lot more work than I thought it would be, but I’m very pleased with the result. Check out the video as there are a lot of in-jokes and Easter eggs, and I’ve been told that “floppy head Lawrence Lessig” was met with approval by the man himself.

Oh, in the spirit of free software, Front has published the song under the Creative Commons (CC BY-SA) license, and I am hoping to see a few cool remixes come out of this. I’ve reached out to both Professor Kliq and Raccoon Fink and if they find time to play with it, I’ll be sure to let you know (and let me know if you find some out there). Front is working on a new album tentatively entitled The Internet Sucks so maybe this track will make the cut.

When Not To Start an Open Source Company

Over the weekend, Chris Aniszczyk posted a link on Twitter to a very interesting article by Matt Klein about his decision not to start an open source company around his project, Envoy. I thought it raised a number of interesting points worth a few comments.

First off, Matt works for Lyft, which, in case you haven’t heard of it, is Uber without the moral decay. I abandoned Uber some time ago, despite being an early adopter, and I’ve been very happy with Lyft. One of the main differences is that Lyft allows you to tip your driver, which I almost always do with few exceptions. The fact that Lyft is able to keep and motivate people like Matt speaks volumes for their corporate culture.

It also demonstrates a wonderful trend of commercial companies starting and maintaining open source projects. I’ve been working with open source for almost two decades and I can remember when any software developed at a company was considered confidential. To this day there are a number of vendors who consider their SNMP MIB files (which, I should point out, are really only useful to people who have purchased their products) proprietary information. Companies like Lyft, Paypal and Facebook, none of which would self-identify as open source companies, have gained a lot of value for little cost by making the tools they use open source.

When talking about open source for the enterprise, I often talk about the fact that it is the processes that a company uses to serve its customers that make it unique and define its value, not the tools used by the company. So often with commercial software you have to change those process to fit how the application thinks you should work, and in the process you lose some part of what makes you special to your customers. With open source you can fit the application to those processes. It is how you use the tools and not the tools themselves that is important, and so there is a lot to gain and little to lose by making them open source.

Getting back to Matt’s article, he is a project maintainer for Envoy, which is a “high performance C++ distributed proxy and communication bus designed for large service oriented architectures.” While I don’t consider myself a coder so I don’t claim to fully understand the its advantages, I do recognize enough buzzwords in that sentence to know that it would attract some attention from investors, and Matt was approached about leaving Lyft to start a commercial business around Envoy. He decided not to, and as I read his article about his decision I realized I’d found a kindred soul, someone who was more interested in creating something of value that would last versus making a quick buck.

He had me with this paragraph:

In my opinion, the best opportunity to commercialize OSS lies with projects that can be easily turned into SaaS products. Ultimately, even if software is completely open, many customers are happy to pay for a turnkey solution that “just works” and has a defined SLA with 24/7 monitoring and support. In some sense, customers pay for the operational expertise that comes from deeply understanding and running the software, versus the software itself.

Amen.

I’ve been making a living on open source for 15 years now working with OpenNMS, and I’ve spent a lot of time thinking about business models. We started out with the “service and support” model, which kept the doors open but limited growth. Then our clients started asking us for features, so we added custom development, which was time intensive but allowed us to finance OpenNMS features which attracted even more customers as the platform became more powerful. When we hit the problem of trying to balance the “release early, release often” philosophy of open source with the need for stability, we adopted the Red Hat model of splitting our application into a feature-rich, rapidly developed release (which we call Horizon™, similar to Fedora) and a more stable, subscription-based release that may lag in features but is better suited production environments (which we call Meridian®, similar to RHEL). But ultimately we came to the decision that what we really wanted to do was to offer OpenNMS as a service.

One company that inspired that decision was Automattic, maintainers of WordPress. I don’t think I know of a more powerful piece of software that is easier to install. They have a famous “5 Minute Install” that is quite simple. First, you drop the software into the webroot of your web server of choice. Next, you create a database account on your database of choice with certain permissions. Then you navigate to a web page and follow the prompts.

However, for a lot of people, terms like “webroot” are gibberish, and even with WordPress you still need some minimal database skills to maintain it. So Automattic offers up WordPress as a service. For a small monthly fee they’ll do everything for you, and this has generated revenues on the order of tens of millions of dollars per year.

OpenNMS is way more complicated, thus the value of a hosted version should be greater. In order to do so we needed some way to access the client’s network in a secure fashion, so with Horizon 20 we introduced the Minion. The Minion software allows for OpenNMS functionality to be distributed. It is built on the Karaf container, so once installed all of its features can be remotely managed. For smaller networks, the Minion can be sold as an appliance and talk to a hosted version of OpenNMS. It can bring a complex and powerful tool like OpenNMS into the hands of the masses.

For larger companies it solves issues of scale as Minions can be deployed to cover even the largest networks (our goal is IoT scale). We’ve had them in production at one client for months now handling over 2 million events an hour. That translates to around 555 events per second, although the system itself can handle over 10,000 events per second so they have room to grow. If they ever hit that limit, we can simply add more Minions. They have the option of hosting all of OpenNMS in their own data center, or they could choose a hybrid model where some of the functionality is outsourced.

For pretty much the first time in the history of OpenNMS, we are seriously and actively seeking investment. There are a number of companies entering this space who have raised enormous amounts of money, and we think we can be competitive for far less money and provide a better solution. Plus, also for the first time in the history of OpenNMS, we have a reason to make it easier to use versus spending all of our resources making it more powerful.

Matt talks about investment in his post (remember Matt? As usual, I’ve made this all about me. Meeee!) It was actually his stories about dealing with investors that prompted me to write this. As Envoy started to get some traction, investors wanted him to leave and start a company. He writes:

Over the last few months I’ve been told by several investors that no OSS has become ubiquitous without having explicit commercial backing. I think this is false and is situation dependent. If anything, I would argue that if I were to leave Lyft now and start a platform company around Envoy, it will decrease the chance of Envoy becoming ubiquitous, primarily because it would negate all of the reasons laid out above.

That first sentence is interesting, since “ubiquitous” and “commercial” are a little vague. I would make the claim that the Apache web server was ubiquitous until its success spawned NGINX, and it was backed by the Apache Software Foundation which is a non-profit. Is a foundation “commercial”? The idea that for a project to become successful it needs a number of people to spend a lot of time working on it seems obvious, and the best way to achieve that is to pay those people to work on it.

He goes on to write:

It took me a lot of time to ultimately understand the previous simple point. Investors are extremely persuasive. They capitalize on “fear of missing out.” However, it’s important to realize that the opportunity cost is hugely mismatched between investor and company.

When he writes “investors” above I believe he means specifically venture capitalists. We’ve talked with a few VCs in the past and I can remember the almost “strong arm” tactics they used. If I hear “a rising tide lifts all boats” one more time, I might have to hit somebody. I’m not saying that all VCs are the same, but many of them come across as gamblers and not investors. I’m risk friendly but I don’t gamble. I’m heavily invested in wanting to build something with OpenNMS that outlasts me (it is already much bigger than me as the team I work with has way more to do with its success than I do) and I don’t want to gamble with it.

I do hope that there are some investors out there that can appreciate that aspect of our company as well as the fact that we’re profitable, have mature products and wonderful customers. Perhaps private equity or perhaps another company that shares our vision and wants to advance the project through acquisition. In any case we’re looking for them.

When I was a young man, old guys like I am now would tell me “work on something you love, not just for the money”. I always dismissed it with the thought that with enough money I can buy love. When you immerse yourself in something as personal as an open source project for ten to twelve hours a day, year after year, you really do have to love it and the satisfaction you get just can’t be bought. Matt’s thoughts are similar:

Ultimately, on a personal level I’m just having too much fun solving tough computer science problems at large scale at Lyft and building a community around Envoy. The bar to do something different is therefore extremely high, and it took a long time to realize that it’s perfectly OK to accept that and keep going down the existing path that I’m on. On another level, leaving now to start a company would feel very much like not following through on my original goal of open sourcing Envoy; the industry desperately needs a high quality and community-driven solution to microservice networking. Follow-through is something I take very seriously.

With that attitude the success of Envoy is almost assured.

Why the FCC’s Title II is so Important (Spectrum Rant)

Here is a rant about Time Warner/Charter/Spectrum or whatever the heck they call themselves these days. It illustrates how this large company can have a huge negative impact on a small business, and why treating Internet providers as common carriers is so important.

Our company wouldn’t exist without the Internet. Outside of the fact that our products are mainly used to monitor Internet resources, we host a number of servers from our office and about half of the staff works remotely so we rely on the Internet to communicate and coordinate.

Back in 2012 I contracted with Time Warner to provide Internet access to our office. We had fiber to the building and while our service was considerably more expensive than coax, I liked the fact that it was symmetrical and expandable. We started of with 20 Mbps but soon increased that to 50 Mbps. Over five years we only had one outage, due to a misconfiguration of our Customer Premise Equipment (CPE), and they corrected it within 20 minutes. I love the fact that when you called in the person who answered the phone understood terms like “duplex” and they were always very helpful.

Note the scenario: happy customer who is happy paying a premium for enterprise-level service.

Now let me tell you why all that goodwill has gone away.

Earlier this year we decided to move our office from Pittsboro, NC to Apex, NC. The first thing I did was contact Time Warner (well, Charter at the time) to insure that they could provide fiber to the new location. They said they could, although it would take 45 to 60 days. As our new office space needed to be completed, we were targeting an April 1st move in date anyway, so on February 15th I placed the order for the new service. At best, it would be available on the 1st and at worst it would be ready by the 15th. We told the old landlord we’d be out by April 30th just in case and to give us more time to move.

Finally, Spectrum doubled our speed and cut the price in half. I was feeling pretty good about the whole thing.

The feeling didn’t last.

As we got closer to April, things started to go wrong, most of it due to the fact that Spectrum is now such a behemoth that they have no idea what they are doing. In order to get fiber into our new building, they needed what is called a “Right of Entry”. They sent it to our landlord who promptly completed the form and sent it back. However, that person didn’t let the project manager know the form had been received, so he did absolutely nothing. Ten days (!) later I get a note that our build out had been suspended because of the lack of the ROE form. A form, I should point out, that was sent to them, twice.

At the end of March I’m told that our new date is May 11th. I’m unhappy – due to their poor processes I now have a new office that I can’t use for six weeks (remember, we took possession and started paying rent on April 1st). We also had to be out of the old office by the end of April. Luckily I work with a great team that is able to be productive when working from home, so I decided to suck it up and live with it.

On April 12th I get an update – the new date for the end of construction is now May 15th due to processes within Spectrum taking too long to finalize the work with a contractor. Now the actual date we’ll have Internet has been pushed out to the week of May 29th.

I am livid. By this point I’m ready to switch to the other option, AT&T. Unfortunately, they also need 45 to 60 days for service installation so I realize at this point I’m stuck with Spectrum.

I ask my salesperson for options and he suggests we get coax installed for a month (for a fee, of course). Since our office is right next to a large housing development they can get coax in the following week. I sign off on it.

It didn’t happen. When May arrived some of us started working in the new office mooching off the neighbor’s Wi-Fi from AT&T (with permission of course). I ended up traveling for a couple of weeks so I completely forgot about the coax option (it’s not like Spectrum was keeping me updated on anything – I’d have to reach out to them for an update). I did get a note on May 10th that all construction had been completed for the fiber and another note on May 18th that our new install date was June 2nd.

(sigh)

So, 45 days late, we have a firm install date. Wonderful.

Imagine how I felt when on the 24th of May I received a note that more construction was needed and that it would be pushed out another 30 days at least. When I get extremely angry I refer to it as going “non-linear” as that how fast my blood pressure rises. As I was ranting to pretty much everyone I’d ever interacted with at Spectrum it dawned on me that this could be for the coax order. Turns out that was the case. Apparently our crack project manager on the coax side decided to route our service from a point several miles away instead of from the one nearly across the street. This is why it was delayed and why the construction was needed. By this time we are about a week out from having fiber so I canceled the order. I did get a very apologetic call from the coax salesperson which I appreciated (under Spectrum, fiber [Enterprise] is handled by one sales team and coax [Business] is handled by another), and I made it clear that I’d be okay with everything as long as the fiber was delivered as promised on the 2nd.

It was. Around noon on June 2nd we had our 100 Mbps service and on the 3rd we moved all of our devices from the old office in Pittsboro to the new one in Apex. I informed my salesperson that they could disconnect the old service and despite all of the problems, I was happy with the new service.

So the whole process cost me two months rent and a few years off my life, but it was finally over.

Not so fast – the other shoe fell today.

I get an e-mail that I need to confirm my disconnect request. That didn’t bother me, in fact I appreciated it, but what did bother me was an additional note that it would be done within 30 days. When I replied I asked for clarification – would I be *paying* for the service I wasn’t using until they could disconnect it? The answer was “yes”.

I experienced a new word – apoplectic.

Due to the fact that the bureaucracy behind the new merged Spectrum company is so bad, I’m out nearly ten thousand dollars. That is the real money – it’s probably cost us twice that again in lost productivity from lack of network access and dealing with them throughout this process. We’re not one of those companies that is too big to fail so this really impacts us negatively. Had it been explained to me that I’d have to pay for the service until it was disconnected, I would have put the disconnect order in a month ago, but then had I used the date I was originally promised, our servers would have been off-line for over a month. That would have been catastrophic to our company.

Finally, I’ve gone from a happy customer to an extremely pissed off one who will be actively looking for options. Based on my experience I would suggest any business looking for network access look elsewhere.

Access to the Internet has become as important as other utilities such as electricity, water and sewer and just like those utilities it needs to be regulated as one. This is why the decision by the new industry-picked head of the FCC to reverse the decision to classify Internet access under Title II as a “common carrier” is so devastating to businesses like mine. Our company is small, yet we put millions of dollars into the local economy each year. You multiply that by the number of other small businesses and it can have a great impact to any community. Barriers put up by companies like Spectrum demonstrate that they can’t self-regulate and the government needs to take a firmer hand (and this is coming from a left-leaning libertarian).

I will be protesting that final bill for Internet access and I would welcome any advice on how to deal with a company like Spectrum. Let’s hope that there is a change soon so that other businesses can focus on creating value and not have to deal with the crap we had to endure.

I’m not holding my breath.

Monitoring? Meh.

Recently, I was talking to a person in the tech industry and describing all of the cool things we are doing with OpenNMS, when he kind of cut me off and went “Oh, monitoring? Meh.”

Well, I can’t remember if there was an actual “meh” but that’s how it came across, and I’m afraid the reaction is probably more common that I would think. Monitoring isn’t sexy, but it surprises me that people can’t see how critical it will be to the future of any business.

IoT Devices Over Time

While forecasts vary, by 2020 there are expected to be over 30 billion devices on the Internet, and that figure will skyrocket to over 75 billion by 2025. Just knowing what is connected to your business network is going to become critical, as well as making sure it belongs there in the first place and, if so, is functioning properly.

Outside of the obvious security concerns, as people began to transact business more and more through devices rather than people, faults in those devices will directly impact revenue as people search for other options when faced with a bad experience.

Here are a couple of examples.

One of the greatest inventions in my lifetime is the ability to buy fuel at the pump. You just pull up, swipe your card, pump and then leave. You used to have to pay inside, and some places made you pay first which meant two trips in if you were paying by credit card. It could be cold or rainy, and not only did you have to wait in line behind people buying food or lottery tickets, you had to leave your car out by the pump possibly blocking the next customer.

The only problem I’ve experienced with this process concerns the receipt. Quite frequently I need a receipt, but it seems the pumps I choose are always out of paper. The little red indicator mark when the paper roll is almost finished isn’t visible to the cashier since there really isn’t one out by the pump. It is frustrating, but it is not like I have a choice at the moment. If there was some way to monitor the pump for a “low paper” alarm, it would improve my shopping experience.

One shopping experience that did result in my leaving the store without a purchase happened yesterday at a Lowe’s Home Improvement store. I needed some florescent lights for the new office so I went by on my way home. I picked up four bulbs (two that I needed and two spares) and went to the checkout area.

I walked past several unmanned cash registers until I got to the “Self Checkout” section, which was the only thing open. Of the four machines, two had red blinking lights on them (that are green when things are functioning normally) and the one lone, overworked cashier was doing her best to help people out. I usually don’t mind using Self Checkout and when I noticed one of the two machines was open (everyone else was waiting for the attention of the lone cashier) I went to it and started my purchase.

I scanned my “My Lowe’s” card and then the first bulb. “Eight ninety-five” piped up the voice and I placed it in a bag.

Here is where the problems started. First, I hate the fact that with these Self Checkout kiosks they don’t trust you to use a “quantity” key. I was buying four identical items but I was required to scan each one. Next, the bulb was light enough that it didn’t register as having been bagged, so the interface yelled at me and presented me with a button marked “Skip Bagging Item?”.

I sighed and, having no other option, hit the button. I then went on to scan the next three bulbs. However, as I bagged the fourth bulb, the scale must have started working since the whole unit went into some kind of alarm mode, screeching “Unidentified Object in the Bagging Area!” and the screen was locked until the cashier had time to come and fix it.

I looked around the area, and by this time all four kiosks had a flashing red light, there were at least three shoppers lined up to use them in addition to those of us already there, and our valiant cashier was busy helping a guy ring up his plumbing supply purchase which consisted of a ton of small copper fittings which most likely wouldn’t be registered by the scale.

I gave up. I picked up my bulbs and returned them to the Lighting section, passing three employees in the customer service area helping zero customers. Before I reached the car I’d ordered the same bulbs on Amazon at a fraction of the price, and they’ll be here on Friday.

Yes, I’m complaining, but how could monitoring have helped here? First, there is some sort of monitoring – those little red lights. When they all light up you would assume someone, or perhaps multiple someones, would come by to help. A monitoring system could have made sure that happened by using an additional notification system outside of the lights, and escalating it until the problem was addressed.

A more long term solution would be to collect information on the purchasing experience and the problems people encountered and to make changes to the automated kiosk software. I’m certain that Lowe’s didn’t write that software but instead bought it, and like most proprietary software solutions they now have to fit their processes to the application instead of the other way around. It probably wasn’t designed for a store that sells a lot of small, light things which is central to the issues I have using it.

With the rise of IoT devices, robotics and other forms of automation, monitoring is going to become extremely important. Lowe’s lost out on a $40 sale, but think of something like an assembly line where a problem could result in the loss of thousands of dollars a minute. Our goal at OpenNMS is to be ready for it, and to build products that make people go “Monitoring? Oh yeah!”.

Server Room Nightmares

I’m interested in any server room nightmares people would like to share.

Here’s one of mine.

We are in the process of moving offices from Pittsboro, NC down the road to Apex. Unfortunately, we are having some issues getting Spectrum Enterprise to complete the fiber installation at the new place, so while we are out of our old building the lack of network access in the new building means we have a bunch of servers in the old location.

Today while I was working in the new office and mooching of our kind neighbor’s wi-fi, I got several notices that links had failed.

linkDown event list

These were some workstations that we use for training, but when they are not in use we use them as part of our continuous improvement Bamboo farm. I immediately hopped on our Mattermost IT channel and asked if anyone was rebooting or otherwise messing with the machines, and when the answer was “no” I started to investigate.

One suggestion was that the air conditioning may have failed and those machines shut down from overheating. It has happened in the past, but it was both rather cool today and other machines that are more sensitive to such things were still running. I checked it out anyway using our AKCP probe.

temperature graph

The temperature had increased a bit, but it wasn’t anything that should have caused problems (it was caused by the server room door being left open).

Being 30 minutes away, I decided to text my friend Donnie, who is technically gifted as well as working in our old location, and he went to investigate.

For some reason, those three machines had been disconnected from the switch.

Now just for this situation we have an Arlo camera installed in the server room, so using the time stamp on the linkDown traps I found the following video.

Note the slightly balding guy in the red shirt in the lower left corner of the video. He is busy unplugging our devices.

Why? I have no idea. These people represent the IT people for the new tenant, and I assume they had legitimate reasons for being in the server room but messing with our equipment was not one of them.

Seriously, in over 30 years of working with computers, I’ve never heard of anyone going into someone’s house, office, server room or data center and just start unplugging cables. I still have not heard an explanation, but the landlord has had a discussion with the new tenant and it shouldn’t be happening again. It is one reason the important stuff is in that locked half-rack seen in the upper left corner of the video, and the really important stuff is hosted elsewhere.

I am curious – I’m certain this pales compared to other stories out there. Do you have any whoppers to share?

Fifteen Years

On Sunday my mother celebrated her 75th birthday.

Although a happy occasion, why is this relevant to an open source blog? Well, it was soon after her 60th birthday in 2002 that I started my first company around OpenNMS.

I did not start OpenNMS, it began in the summer of 1999, with the first code posted on Sourceforge in March of 2000 by a company called Oculan. I started working with Oculan in September of 2001, and in May of 2002 they decided to stop contributing to OpenNMS. I saw the potential, so I asked Steve Giles, the founder and CEO, if I could have the OpenNMS project. He looked at his watch and said if I was off his payroll by Friday, he’d give me the domain names, a couple of servers, and he would sprinkle water on me and I would be the new OpenNMS maintainer.

That was actually the easy part. Explaining to my wife that I had quit my job and started a company “selling free software” was a bit harder.

sortova.com from archive.org circa May 2002

And thus Sortova Consulting Group was born. It was named after my farm. When Andrea and I decided we wanted to have a farm, we first bought raw land. In driving out from Raleigh to work on it we would pass this little farm with a barn, some cows, etc., and on the mailbox was a sign reading “Almosta Farm”. I joked that if that was “almost a farm” then what we had was just “sort of a farm”. Later, when we bought the place where we still live, the name Sortova Farm stuck.

We pronounce it “Sore-toe-va”. Only one customer ever pulled me aside and asked if it really meant “sort of a” consulting group. He laughed when I confirmed that it did.

Considering that I didn’t have any prior business experience, Java experience, or even real Internet access at my home, it is amazing that OpenNMS survived to this day. It is a wonder what you can accomplish with pure stubbornness.

Now my one true superpower is my ability to get the most fantastic people on the planet to work with me. The first group of those came from the OpenNMS community. When I was running Sortova it was the gang that later became the Order of the Green Polo that kept me going, mainly through mailing lists and IRC. In September of 2004 my good friend and business partner David Hustace and I founded the OpenNMS Group, and that corporation is still going strong. In 2009 we mortgaged our houses to buy the copyright to the Oculan OpenNMS code and thus brought all of it back under one organization, and two of the original OpenNMS team at Oculan now work for OpenNMS.

When I visit Silicon Valley I often get to meet some brilliant people, but the joy of this can be offset by the pervasive attitude of focusing on technology simply to make money. I know of a number of personally successful people who built companies, sold them, and then those products vanished into obscurity. Remember VA Linux? Their stock rose over 700% on the first day of trading, but where are they now? Did they ever deliver on their promises to the stockholders?

I want to build with OpenNMS something that will last well beyond my involvement with the project. I’ve gotten it to the point where I am not longer expressly required to make it thrive, but I am still working on its legacy. We want it to be nothing less than the de facto standard for monitoring everything, which is a high bar.

Note that I still would like to make a lot of money, but that isn’t the core driving force of the business. Our mission statement is “Help Customers – Have Fun – Make Money” in that order. If you have happy customers and happy employees, the money will come.

Fifteen years ago I made a leap of faith, in both myself, my family and my friends. I’m extremely happy I did.

How Version 2.0 Killed Android Wear

I am the happy owner of an LG Urbane smartwatch. Unfortunately, I just upgraded to Android Wear 2.0 and now I can’t use it.

Andrea Wear 2.0 Upgrade

Luckily for me, my smartwatch is not “mission critical”. If I leave it at home by mistake, I don’t turn around to go back to get it. The main thing I use it for is notifications. I like the fact that if it is with me, it will automatically mute my phone and then vibrate when I have a notice. A quick glance at my wrist will tell me if I need to deal with it right this moment, or if it can wait.

The second thing I use it for is to do simple voice searches or to set reminders and timers. Outside of that there are a few apps I use and I like the fact that it tracks my steps, but overall I don’t use a ton of features.

When the notice popped up that I could upgrade, I blindly went ahead and did it. In retrospect, that was stupid, but I often get in trouble rushing out to install the “new shiny”. The upgrade seemed to go fine, and I didn’t think that much about it until lunch.

One of the things I do before heading out to lunch is check the temperature to see if I need a jacket. So I did the usual wrist flick to “wake” the watch and said “Ok Google” to get to the voice prompt.

Nothing happened.

Hrm, I did some research and apparently with 2.0 you have to press the button on the side of the watch to get to the Google prompt. I think this is a huge step backward, because now I have to involve both hands, and I find it ironic that with Android Wear 1.5 I I had to sit through a demo of one-handed gestures over and over again (I often have to re-pair my watch due to reloading software on my phone) and now they’ve thrown “do everything with one hand” out the window.

Anyway, I pressed the button which then brought up the Google Assistant setup screen on my phone. With 2.0 if you want to use voice searches, etc., you must use Google Assistant and you have to give Google access to all of your contacts, calendars etc.

(sigh)

I work hard to “sandbox” my Google activity from the rest of my digital life. It’s not that I think they are evil, it’s just that I don’t want anyone to have that much information on me, well, other than me. I kind of despair for free and open source software solutions in the consumer space. Everyone seems to be rushing to adopt these “always on” digital assistants with absolutely no regard to privacy, and this is causing vendors to lock down their ecosystems more and more. While open source is definitely winning on the server side, I don’t think the outlook has ever been so grim on the consumer side.

There were some upsides with 2.0, such as improvements to the look and feel, but I also found that I didn’t care for the new notification system (I seemed to miss a lot of them – perhaps I needed to change a configuration). But the requirement for Google Assistant was a deal breaker.

I thought about going back to 1.5, which I liked, but I can’t seem to find a factory image. In trying to locate one, I discovered that TWRP does have a version for bass (the codename for the LG Urbane) and I should have installed that and made a backup before upgrading. I contacted LG and they told me it was impossible to downgrade. That’s a load of crap because I could easily sideload the old version if they made it available, but then I’d have to deal with constant upgrade reminders and the few apps I do use would probably stop support for 1.5 to focus on 2.0.

It just isn’t worth it.

I know at least one of my three readers is thinking I should just cave and learn to embrace the Google, but I can’t bring myself to do it. I am eagerly awaiting open source alternatives like Asteriod OS (which just isn’t ready for daily use) and Mycroft (which is supposed to be shipping units this month) but I really don’t think I’ll miss my Urbane enough to spend the time on it.

I plan to sell my Urbane on eBay and I’ve gone back to my previous “dumb” watch (a nice little Frederique Constant I bought on a flight from Dubai to London). It’s kind of a shame since I enjoyed using it, but to be honest I’m not going to miss it all that much.

The Importance of Contributor Agreements

One thing that puzzles me is the resistance within the open source community to contributor agreements. This was brought into focus today when I read that the OpenSSL Project wants to migrate to the Apache 2.0 license from the current project specific OpenSSL license.

In order to do that they need permission from all of the nearly 400 contributors of the project over the last 20+ years, and contacting them will be a huge undertaking. If one person refuses to agree, then they will either have to abandon the effort, or locate that person’s contribution and either remove or replace it.

Many years ago we found out that a company was using OpenNMS in violation of our license. When our lawyer approached them about it, they claimed that they were only using those parts of the code for which we didn’t hold copyright. At that time, early versions of OpenNMS were still copyright Oculan, the company that started the project, and not OpenNMS. Since Oculan wasn’t around anymore it took us awhile to track down the intellectual property, but in the end David and I were able to mortgage our houses to purchase that copyright so that now the project can control all of the code and defend it from license abuse in the future.

But the question arose about what to do moving forward, specifically how should we deal with community contributions? In the past companies like MySQL required all contributors to sign a document with phrases like “You hereby irrevocably assign, transfer, and convey to MySQL all right, title and interest in and to the Contribution” which seemed a little harsh.

I posed this question to the Order of the Green Polo, the de facto project administrators, and DJ Gregor suggested we adopt the Sun Contributor Agreement that we now call the OpenNMS Contributor Agreement, or OCA. This was a straightforward document that asked two things.

First, you attest that you have the right to contribute the code. This is more important than you know, because it helps remove liability from the project should the contribution turn out to be encumbered in some way, such at the author writing it as part of their job and thus it is actually the property of the employer. We allow both individuals and companies to sign the OCA.

Second, you assign copyright to OpenNMS while retaining copyright yourself. This introduces the concept of “dual copyright”. Now some critics will say that this concept hasn’t been tested in court, but there is a long history of authors sharing copyright. Considering that Oracle maintained the agreement in the form of the Oracle Contributor Agreement, it appears that their lawyers were satisfied.

I claim responsibility for the license under which these Contributor Agreements are published: the Creative Commons Attribution-Share Alike License. When DJ suggested the Sun Contributor Agreement I noticed that there wasn’t any license on the agreement itself. I didn’t want to just copy it and change “Sun” to “OpenNMS”, so I contacted Brian Aker who had just moved to Sun with the MySQL acquisition and asked him about it. Soon thereafter the Agreement was updated with the license and we adopted our version of it.

Once we adopted the OCA, I was tasked with tracking down anyone who had ever contributed to OpenNMS outside of the company or Oculan and asking them to sign it. They all did, but I can tell you that I had a hard time tracking down a number of them (people move, e-mails change). I don’t envy OpenSSL at all.

I hope this story illustrates the importance of some sort of Contributor Agreement for open source projects. They don’t have to be evil, and in the end getting your copyright and licensing issues completely sorted out will make managing them in the future so much easier.

Electronic Devices and CPB

With the change in administration in the United States, Customs and Border Protection (CBP) have modified their behavior to include actions with which I don’t agree. These include forcing a US citizen to unlock his mobile device, even though it was a work device and contained sensitive information. I set out to come up with how I will deal with this situation should it arise in the future.

TL;DR My plan is as follows: before I enter the United States, I will generate a long, random password and set that as the encryption password for my laptop and my handy. I will then ssh into an old iMac I have on my desk, store the password in a file, and then shut the computer down. At that point I will not be able to access the information on my device until I return to the office and power on the system.

UPDATE: The EFF has published a detailed guide to help understand your rights at the border.

First off, let me say that until recently I’ve always respected CPB. They have a tough job and everyone I’ve ever met while returning from my travels has been efficient, competent and friendly.

But after the recent “Muslim Ban” fiasco I’ve come to realize that my experience is not universal. I think one of the main problems is this idea that the Constitution stops at the CBP desk, and until you are past it you really aren’t “in America” and thus the Constitution doesn’t apply.

I don’t agree with this interpretation, but it can probably be traced to the actions taken by the US government after 9/11 and the creation of the prison at Guantanamo Bay.

Prior to that, when “bad hombres” were captured by the US government, they fell into one of two categories: criminals or prisoners of war. How each class was treated was fairly well defined. Criminals were processed according to the rule of law, and the treatment of POW’s was covered under the various Geneva Conventions.

The US government decided that those two classifications were inconvenient, and so they ventured into the murky waters of “enemy combatant” and Guantanamo. Their logic goes that since Guantanamo isn’t in the US, US law doesn’t apply, and since these people aren’t members of a foreign country’s military force with which we are at war, then they aren’t POWs. So, the US gets to make up its own rules about how these people are treated.

This is dangerous for a number of reasons. Since nothing is really codified about the treatment and rights of the detainees at Guantanamo, the rules are arbitrary. Also, this opens the door for other countries such as Russia to do similar things without fear of international repercussions. The US has survived for so long because things like this are not supposed to happen, yet here we are.

This thought now extends to the border. Even though a US citizen is being questioned by another US citizen, in the role of a representative of the US government on US soil, somehow the rules of the Constitution are suspended. It’s arbitrary and I don’t buy it. The Constitution codifies a right to privacy in the Fourth Amendment, and it doesn’t go away when entering the country. And it definitely extends to mobile devices, which in today’s world are probably the most personal item people own.

So how can people like me, with almost no political power, resist this threat to our freedom?

I’ve always done little things, like opting out of millimeter wave scans at airports and getting a pat down instead (I’m not shy). If everyone did this the whole system would collapse, and they would find better ways of dealing with security than the security theater we have now. Seriously, if the Israelis don’t use it, it ain’t worth using.

When I turned to the problem of dealing with CBP, my main thoughts went to two devices that I use when traveling: my handy (mobile “phone”) and my laptop. I figured the easiest thing to do would be to just wipe them before coming into the country, but that presents some logistics problems.

For example, I could make a backup of my handy, copy it to a server at home, and then wipe it. The problem is that I have 64GB of storage on the device and I doubt I could transfer a backup in time over, say, a hotel Wi-Fi connection. One of my coworkers uses an iPhone and they thought about wiping their phone and just restoring it from iCloud when they were in the country, but then CBP could require that he turn over his iCloud password.

On my laptop I use whole disk encryption, but I thought about just rsync’ing my home directory and then deleting it before leaving, then again there is the WiFi issue and I really don’t want to have to deal with copying everything back when I’m home.

Then it dawned on me that if I didn’t know the encryption password, then I couldn’t reveal it. The problem became how to create a secure password that I couldn’t remember yet get it back when I needed it.

While my main desktop computer runs Linux Mint, I keep an old iMac on my desk mainly to run WebEx sessions and for those rare times I am forced to use a piece of software not available for Linux. It’s connected to the network, so I can access it remotely. But, if I can access it, I would be lying if CBP asked me for my password and I said I couldn’t retrieve it. Unlike the US Attorney General, I refuse to perjure myself.

Then it dawned on me that I could shut the iMac down remotely and have no way to turn it back on. Thus I could store a passphrase on it, retrieve it when I was back in the country, but until then I would be unable to unlock my devices.

That became the plan. So, the next time I’m returning from overseas, I’ll generate a new, random password. I’ll set that as the whole disk encryption password on my laptop and the encryption password on my handy (note that this is different from the screen-lock password). This will also tie up all of my social network passwords since I use complex ones and store them on those devices. Well, with the exception of my Google account, but since I use two-factor authentication I should be safe as my handy is the device that generates the codes (and I won’t carry any of the backup codes). As long as both of those devices stay powered on, I’ll be able to use them, but once I power them off they will be useless until I get to the office, power on the iMac, and retrieve the passphrase. Note that in order to do that, I’ll be firmly in the US and anyone who wants me to unlock my devices will need a court order.

Which I would respect, unlike CBP. I think the scariest part of the whole “Muslim Ban” incident was when CBP refused to honor court orders. America is built on three branches of government, and when the Executive branch ignores the orders of the Judicial branch we are all in trouble.

I had a two other problems to address, one of which is done. If I’m in the US but my handy is locked, how would I make calls? I might need to call my ride home, etc. To that end I bought a cheap “feature” phone and I’ll just move the SIM card to it when we land.

ZTE Feature Phone

The second issue is that while I should be on solid legal ground concerning my electronic devices, there is nothing preventing CBP from holding me for a long time. Thus the final step is to find an attorney and execute a G-28 form allowing them to represent me. I’m not sure if I need a civil rights lawyer or an immigration lawyer but I’m looking into it. My goal is to be able to notify my attorney when I am coming back into the country, and then send an SMS to them when I am through immigration. If that doesn’t arrive within two hours of my scheduled arrival, they need to come and get me.

I think the thing that bothers me the most about this whole process is the need for it. I’m not a tinfoil-hat conspiracy guy but the actions of the new government have me worried. As I use open source software almost exclusively I know I’m safer than most when it comes to surveillance, and I also don’t expect to run into any problems being an older, white male. But I’d rather be safe than sorry, and the only thing necessary for the triumph of evil is that good men do nothing.

Fourteen Years

I just wanted to take a second to thank my three readers for fourteen years of support.

My first post on this blog happened on this date in 2003, and when I wrote it I had little idea I’d still be doing it almost a decade and a half later.

It does seem weird that I still consider OpenNMS a start-up. We took a much different path than a lot of other companies, focusing on our customers instead of fundraising. With our mission statement of “Help Customers, Have Fun, Make Money” and our business plan of “Spend Less Than You Earn” we’ve not only managed to survive but thrive, and both the company and the project have never been stronger. While we are always looking for good investors, this allows us to pick just the right partner.

I’d like to end this with a quote from Michael Seibel of Ycombinator. Actually, it is almost his entire blog post but it really resonated with me.

I’d like to make the point that success isn’t the same as raising a round of financing. Quite the opposite: raising a round should be a byproduct of success. Using fundraising itself as a benchmark is dangerous for the entire community because it encourages a culture of optimizing for short term showmanship instead of making something people want and creating lasting value.

I believe founders, investors, and the tech press should fundamentally change how they think about fundraising. By deemphasizing investment rounds we would have more opportunity to celebrate companies who develop measurable milestones of value creation, focus on serving a customer with a real need, and generate sustainable businesses with good margins.

Optimizing for funding rounds is just as unproductive as optimizing for headcount, press mentions, conference invites, fancy offices, speaking gigs or top line revenue growth with massively negative unit economics.