The Brotherhood of the Traveling Kiwi

One of the favorite parts of my job is OpenNMS training. OpenNMS is a large, complex and powerful system (which is why I refer to it as a “network management application platform” instead of just an application) and it is difficult for someone new to easily get involved with it.

Perhaps you can imagine how much fun it is to watch interested people start out on Day One of our training course finally understanding the basics and end up on Day Five going “wow – OpenNMS can do a crazy amount of things”.

It’s a rite of passage and it can be a bit intense. So much so that at the end of the week I sometimes feel like I’m dealing less with a student and more with a brother.

This is stronger during our remote training courses. Uwe Bergmann, CEO of our German partner Nethinks, set up training this week at his office in Fulda. This allowed me to go out with the class in the evenings. Usually when I teach I have others things demanding my attention, but when I am overseas it is too much fun to pass up.

Last night we all went out to a local brewery in Fulda called the Wiesenmühle. They had an amazing “Swartz Bier” called “Dunkel” and after a few of those it is hard to say no to a shot of schnapps.

OpenNMS has turned a number of strangers into my friends, but I am more excited about seeing friendships evolve between members of the community.

At this year’s Dev Jam, Craig Miskell brought a stuffed Kiwi toy from New Zealand. He gave it to Ronny and Markus from Germany, and now “Ulf” is extremely well traveled – going wherever OpenNMS takes him.

I’m not sure what Ulf was up to last night, but on the left is a card from the Toro Negro Rodizio restaurant where we ate on Monday night, and the one on the right is a card for Stefan Junger, one of our students, featuring his photography website (may not be safe for work).

I can’t explain all the cash.

While not every student comes up to us afterward and says “this is the best training I have ever had”, a surprisingly large number do. It is rare that we do better than break even with training, but the long term relationships (and often future support customers) more than make up for it.

Our next training is scheduled for December. Perhaps I’ll see you there.

On the Nature of MySQL Closed Extensions

Matt Raykowski pointed out a couple of notes about the recent announcement by Oracle that it would begin supplying closed source extensions to MySQL, which it acquired when it bought Sun Microsystems (such as this one from Simon Phipps and this one from Monty Widenius)

I can’t say that the move surprised me, but I’ve been busy teaching this week and am newly obsessed with De Rerum Natura by Lucretius, which was rescued from obscurity by Poggio the Florentine when he found a copy at a Benedictine Abbey in Fulda, where I happen to be.

I believe I have tracked down that abbey, so I am now free to once again bore my three readers by discussing the open core business model.

First, when it comes to Oracle, one must surely realize that they are a commercial software company and their customers consist of people both used to and comfortable with buying proprietary code. I don’t really see this move as “open core” since Oracle has never, to my knowledge, described itself as an open source company. Instead, they join the ranks of Microsoft, Google and Apple who sell proprietary products based on open source code. There is no market confusion here, as the vast majority of companies who buy these products realize that they are buying commercial software.

I think it is more interesting to examine the role of commercially licensed software within the realm of an open source business.

To use MySQL AB as an example, I had no problem with the company both providing MySQL code under an open license as well as selling it under a proprietary license so it could be imbedded into commercial products. As long as all the bits are the same, I think it is a great way to fund open source development. We offer the same model with our “Powered by OpenNMS” program.

The problem arises when features are withheld from the “open” version. One of the powerful aspects of open source software is that the community of users and developers around it allow it to develop rapidly. OpenNMS is able to compete with products that have yearly R&D budgets in the tens of millions, mainly because features can advance rapidly due to a large, disparate development community.

This “withholding of features” is a key part of the open core model. It doesn’t work, and it is why open core companies are running away from it as fast as they can. Take Hyperic for example. Just recently on their messaging boards came a post asking about the next community release, since the “enterprise” release has been out for awhile now. Because the software has this separation, someone has to make the decision on how much of it to release as open. Thus, the decision on how to manage, test, distribute and improve the code has been removed from the community and placed in the hands of the vendor. The code is hidden until the vendor decides to release it, which all but removes the advantage of having it as open source in the first place.

Again, to contrast that to OpenNMS, anyone with access to the Internet can get to our git repository and see the latest and greatest code whenever they want. If they want to run our next stable release before its time, they can download testing (the next point release), unstable (the next major release) or snapshot (the raw, unrefined, latest code) which are built nightly.

The second major issue when talking about the role of commercial software within an open source business is the topic of copyright assignment. MySQL required that developers contributing code assign the copyright to them. Only the copyright holder can set the terms of the license, so Oracle would not be able to create a proprietary version of the MySQL database without it.

At OpenNMS, the discussion was lengthy when we were deciding on a copyright assignment strategy. We had been stung by the fact that a company had taken our code and turned it into a commercial product in violation of the license, and when we pursued legal action, that company’s response was that if they were using the OpenNMS code, it was from an earlier version to which we did not own the copyright. Now that The OpenNMS Group holds the entire copyright to OpenNMS, it makes such issues easier to defend.

We adopted, at the suggestion of DJ Gregor, the Sun contributors agreement, in which the copyright is assigned to the project but the contributor also maintains copyright.

There is one huge positive to this and one minor negative.

The positive is that the contributor retains ownership of their work. Think about it – supposed you wrote some part of MySQL and assigned the copyright to MySQL AB. Now you want to improve that code and submit it to another project. Since MySQL owns the copyright to the original work, you are bound by whatever license under which it is published. If it was published under the GPLv2, you wouldn’t be able to submit it to a project licensed under GPLv3 without express permission from MySQL. You don’t own your own work.

In the case of “dual copyright”, you still maintain ownership and you can license it any way you want.

The negative is that dual copyright may not be enforceable around the world, and it has not, to my knowledge, been tested in court. But this was the same FUD spread about the GPL in the beginning, and numerous court challenges have upheld the license’s validity. I am confident that should this ever be tested in court, it too will be found valid (just as if, say, William Gibson and Bruce Sterling decided to write a book together, they could each share in the ownership of the final product).

Publishing all the bits under an open source license (even if they are also sold commercially) and having a permissive contributors agreement that favors the developer are key to being successful as an open source business.

I’m afraid I have little sympathy for the developers of MySQL. They took their 40 pieces of silver when they sold themselves to Sun for US$1 billion. At that valuation, they had a ton of options, including an IPO, but somewhere along the way they made decisions that eventually led to Oracle owning the project.

I think this is more a lesson in foresight versus some strange and unexpected strategy by Oracle.

Testing, testing

Recently, a “Powered by OpenNMS” customer asked Matt Brozowski, our CTO, about the OpenNMS test plan.

I thought his reply warranted a blog post.

The way OpenNMS does testing is as follows:

We have extensive automated tests that run whenever code is changed.

The list of successful tests of the 1.10 release is here (as you can see there are 3743 tests)

Our automated build system does not create RPMs unless all of these tests have passed, otherwise it make a new RPM each night.

In addition to these tests, as a part of each build we also have a set of tests that run against an installed system (we call these Smoke Tests)

The Smoke Tests have automated scripts that install and configure a system and then uses the Selenium GUI emulator to validate that the GUI is functioning correctly. These tests are relatively new for us but I’m sure we will be adding to these tests of the coming months.

Lastly we release a milestone every month and we have a number of community members that install OpenNMS on systems in parallel with their production system and validate that the features that they are using still work with the devices that they have and report bugs as they come up.

After features are complete we make these milestones as release candidates and label them something like 1.9.9x and from this point on only bug fixes are allowed. Next week we will release 1.9.91 as a release candidate in preparation for 1.10.

This is the list of issues that are currently considered critical for release of 1.10 as a stable release.

Not all of these issues have been validated but these are the ones that remain to be considered.

After we release 1.9.91 these will be moved to 1.9.92 and so on until we get them fixed and can release 1.10.

I hope this helps give you an idea about the quality of our testing strategy.

And Who Says Nixon Is the Only One Who Can Go to China?

Please forgive me if this post is a little spacier than usual, but I was up from 2am until 6am this morning doing an upgrade at a client site.

When I came into the office, everyone was talking about a project Ronny Trommer spotted on Sourceforge called YbmNMS.

It appears to be a Chinese fork of OpenNMS.

I am always amazed at where our code ends up, and I think it is pretty cool that someone in China is using it. While I visited China in the 1990s (Guangzhou and Chengdu), I haven’t been back since, and we don’t have any commercial support customers in China.

I always worry, however, about whether or not projects that use our code obey the license. The code for YbmNMS seems to be hosted as the “Netcools” project on Sourceforge, and it appears that all that has been done, with respect to the copyright, is a global search and replace on the string “OpenNMS” and substituting “YbmNMS”.

From one file:

// YbmNMS(R) is Copyright (C) 2002-2005 The YbmNMS Group, Inc. All rights reserved.
// YbmNMS(R) is a derivative work, containing both original code, included code and modified
// code that was published under the GNU General Public License. Copyrights for modified
// and included code are below.
// YbmNMS(R) is a registered trademark of The YbmNMS Group, Inc.

Considering that not all of our code is there, YbmNMS may be focusing on just that functionality that is similar to IBM’s Netcool product, hence the Netcools name.

I tried to send an e-mail to the “mailto” link on their website ( but it bounced, so I sent a note to Yin Bangmin to learn more about what he is doing and to get the copyright notice corrected.

“The only thing worse than being talked about is not being talked about.”

-Oscar Wilde

Some Thoughts on 9/11

Today marks the tenth anniversary of the terrorist attacks on New York City and Washington, DC (and not to forgot the one foiled attempt that ended in Pennsylvania). The world changed that day.

I didn’t lose any friends that Tuesday, even though several lived near the attacks. One was at the airport at LAX heading to a meeting at the World Trade Center later that afternoon. His plane, of course, never left. But, I am thankful, that is about as close as it got.

The 9/11 anniversary is always an important milestone in my life, coming a day after my anniversary with OpenNMS. I didn’t know it at the time, but OpenNMS would also change my world, but in a much better way.

The goal of those involved in the 9/11 attacks was to inspire terror. Terror is an extreme form of fear, and I believe they met their goal of increasing fear. The 9/11 attacks led directly to things such as Guantanamo Bay, The Patriot Act, airport scanners and the seizure of electronics from citizens at the border. Not to mention the somewhat justifiable invasion of Afghanistan and the totally inappropriate invasion of Iraq.

As I write this, they are reading the names of the dead in New York. My guess is that they won’t be reading the names of the 6207 service men and women who died in those military operations.

Fear is the direct result of feeling powerless. The people who died in the Towers and who died in the Pentagon could have done little to save themselves. It appears that those people on Flight 93 turned that fear into action, and probably prevented the deaths of hundreds more people at the expense of their own.

I wish I could say that the actions of my country’s government were as brave.

Many years ago, my two year old cousin and his mother came to spend the summer with us, as his father was on an extended deployment with the Navy. I had not spent much time around children, so it was pretty cool to try and figure out what was going on in his fresh little mind.

I can remember one incredibly nice day, we had the windows open and there was a nice breeze coming through the house. As my cousin walked by a door, an especially strong breeze slammed it shut. It startled him and he reacted by smacking his hand against the door, as if to punish it for scaring him.

I classify a lot of the policies put in place after 9/11 as similar, and about as effective.

On one hand, government officials hate to appear powerless. Since it is nearly impossible to prevent other, similar attacks with threats of incarceration or bodily harm, we engage in security theatre. If you are inconvenienced at the airport (take off your shoes, put your liquids in a baggie) you get the impression that something is being done, and it is hoped that you don’t question its effectiveness.

On the other hand, there is serious money to be made in the terrorism business. Those airport scanners aren’t cheap, and what’s even better is that they are only used about 50% of the time in my experience. Thus the manufacturer still gets paid and they don’t have to maintain anything. Anyone remember those machines that would blow air around you and sniff for the chemical signatures of explosives? Yup, they are mothballed now but someone made a chunk of change when they were the hot, new anti-terrorist thing.

So, how does one combat fear? Not through current government policies, which seem to scream “Be Afraid and Buy Shit”. I believe one combats fear by transferring as much power to the individual as possible. When one feels in control of their own destiny, they are much less likely to throw their life away in a terrorist attack.

Thus, after several meandering paragraphs, we come to my tie-in between combating terrorism and open source software.

Open source is the ultimate expression of personal power, at least within the realm of computer software. If that seems a little limited, you can read Marc Andreessen’s essay on “Why Software Is Eating the World” or just look to how social networks (based on open source) were key to this year’s Arab Spring. People who feel empowered are less afraid, and when fear is lessened (if not removed) amazing things can happen.

I’ve been to both Syria and the United Arab Emirates. In both cases I was afraid for my life. My hosts were determined to entertain me to death. True Arab hospitality can be a frightening thing.

The only saddening part of my trip was that I saw a tremendous number of extremely poor people. For example, in Syria I can remember seeing a hut made out of stacked, corrugated steel sheets sitting in a field made of rocks. Under the desert heat it must have been unbearable, but it should have provided some protection from the cold desert nights.

Prior to that, my idea of an extremely poor person came from the peasant scene in Monty Python and the Holy Grail. Even though they were knee deep in mud, they had trees and a thatched roof shack – making them kings to the poor in the desert (or at least an autonomous collective).

To truly prevent terrorism, we have to remove the sense of hopelessness that drive people to be willing to kill themselves. It seems to me that it would be hard to throw one’s life away if it was made worth living. Perhaps we can use open source to help pave the way.

With the advent of utility computing, in order to get someone online and coding should only take a small laptop, a network connection and some education. The horsepower can live elsewhere, and perhaps we could even use solar cells to power the laptop (and solar power is plentiful in the desert). In a perfect world, idle youth would spend their time building things instead of seeing destruction as their only option. Imagine how far we would have gotten toward this if we took the billion dollars we spent on airport scanners or the trillion dollars we spent on an unnecessary war and spent it on laptops and training?

I’m probably being silly and idealistic, but open source software changed my life for the better, and I can’t help but think it could change the lives of others as well.

Then maybe I won’t shed so many tears at the reading of names.

Ten Years with OpenNMS

It was ten years ago today that I first drove to the Oculan offices in Raleigh as an employee, and began working on the OpenNMS Project.

Oculan built a network management appliance on top of OpenNMS, and that was their main business. However, there was enough interest in the OpenNMS platform by itself that I was brought on board to build a service and support business around it.

I worked on that until May of 2002, when Oculan decided to focus solely on their appliance business, as well as to stop working on OpenNMS. I saw enough potential in the project to ask to take it over. Thus I became the sole maintainer of OpenNMS and started my own business called Sortova Consulting Company (named after Sortova Farm, where I live).

Somehow, I managed to keep the project alive as well as pay my mortgage. I owe no small part of that to a group of strangers that formed the inaugural class of the Order of the Green Polo (OGP). Although we had not met (yet), through IRC and the mailing lists they keep me going, if not the project itself.

In 2003 the business grew to the point where my satellite internet connection just wasn’t adequate. It was the only affordable option I had at the farm for any form of non-dialup internet, and since a T1 circuit with internet connectivity was going to run me close to $1000 a month, I decided it would be cheaper to rent an office in town and just get DSL.

I did that, but then I needed to get a new ISP (since I couldn’t afford both the office and to keep my satellite connection at home). Always wanting to “buy local”, I called up the local internet provider called Blast, and opened an account. While I was on the phone I asked about their monitoring solution, and was told I needed to “talk to Lyle”.

Lyle Estill is one of more animated and driven people I’ve ever met, and we hit it off. Since Blast was mainly a services company, he suggested that I bring OpenNMS and come work for him. So for the first time in over a year I ended up with a salary, as well as other people to help share the work around the .com side of the OpenNMS Project.

Through Blast I was able to hire David Hustace, and shortly after that, Matt Brozowski. We now had a real team of people working full time on the project, and things were looking up.

In 2004 Lyle decided to focus his time on a (then) nascent biofuels project. Before he left Blast, the three of us approached him about buying back the OpenNMS business, which we did to form The OpenNMS Group. That has been my current employer since September 1st, 2004.

We had reached the profitability tipping point, and have been organically growing ever since.

Working on OpenNMS has put me in touch with some amazing people. First there are my coworkers, who have been there through the best of times and the worst of times. Second are the people involved in the project itself, mainly through the OGP, but also the people I’ve met through the mailing lists and at conferences. Third are my customers. My job would be a lot harder without such a wonderful group of clients. They not only understand the value of OpenNMS, their input has had a direct impact on the development of the code. I’d be a poor businessman if I ignored the needs of the very people willing to pay me to build it.

Finally, I’ve gotten to travel the world. As a farmboy in a rural section of North Carolina, I’ve been to places like Singapore, Japan, Australia, Switzerland, Italy, the UK, France, Germany, Norway, Dubai, Portugal, and Mexico. I’ve been all over the US, including Alaska and Hawaii, and what I’ve found is that while the politics of our respective countries may differ greatly, people all over the world who love open source software are pretty much the same: bright, eager people who really enjoy having control over their software solutions.

Oh, I did I mention really good looking? Serious model potential here, yessiree.

This day did not come without some sad notes. I was looking forward to celebrating at the Ohio LinuxFest, with a number of those good people I mentioned above. Unfortunately, my usually illness free body betrayed me on Thursday, and with chest pains and a high fever I decided it was best not to travel. Things are better today and I should be fine, but it is still disappointing not to be there. Instead, I spent the morning watching David, my spouse and several of her coworkers in the Trooper Challenge Mud Run. It was a fun time, though not nearly as fun for me as the OLF. Oh, David finished 19th and plans to run the New York Marathon this year. He’s doing it for charity so if any of the three people reading this want to sponsor him, drop me a note.

Also today, Rackspace decommissioned Rackspace Hosting is an awesome company and they have been both a client and a supporter of OpenNMS since 2002. When I started out on my own, they donated a server to the project, and it has hosted our mailing lists, website, FTP services, etc. over the years.

It was in their old datacenter in downtown San Antonio which is being closed, so they migrated us to a much newer and powerful machine out of Dallas. Still, it was like losing an old friend.

But on net, the number of friends I’ve gained in the last decade far outweighs that one loss. I am truly blessed to be able to make a living doing something I love, surrounded by awesome people who share similar goals.

Thank you.

Lucky Number Seven

It was seven years ago today that David Hustace, Matt Brozowski and myself started The OpenNMS Group.

I had been working on OpenNMS for several years under three different companies by that point in time, but this seems to be the one that stuck.

Those of us who were in town went over to the Brewery and ordered every appetizer and every dessert they had on the menu.

That’s supposed to be lucky, too.