The Brotherhood of the Traveling Kiwi

One of the favorite parts of my job is OpenNMS training. OpenNMS is a large, complex and powerful system (which is why I refer to it as a “network management application platform” instead of just an application) and it is difficult for someone new to easily get involved with it.

Perhaps you can imagine how much fun it is to watch interested people start out on Day One of our training course finally understanding the basics and end up on Day Five going “wow – OpenNMS can do a crazy amount of things”.

It’s a rite of passage and it can be a bit intense. So much so that at the end of the week I sometimes feel like I’m dealing less with a student and more with a brother.

This is stronger during our remote training courses. Uwe Bergmann, CEO of our German partner Nethinks, set up training this week at his office in Fulda. This allowed me to go out with the class in the evenings. Usually when I teach I have others things demanding my attention, but when I am overseas it is too much fun to pass up.

Last night we all went out to a local brewery in Fulda called the Wiesenmühle. They had an amazing “Swartz Bier” called “Dunkel” and after a few of those it is hard to say no to a shot of schnapps.

OpenNMS has turned a number of strangers into my friends, but I am more excited about seeing friendships evolve between members of the community.

At this year’s Dev Jam, Craig Miskell brought a stuffed Kiwi toy from New Zealand. He gave it to Ronny and Markus from Germany, and now “Ulf” is extremely well traveled – going wherever OpenNMS takes him.

I’m not sure what Ulf was up to last night, but on the left is a card from the Toro Negro Rodizio restaurant where we ate on Monday night, and the one on the right is a card for Stefan Junger, one of our students, featuring his photography website (may not be safe for work).

I can’t explain all the cash.

While not every student comes up to us afterward and says “this is the best training I have ever had”, a surprisingly large number do. It is rare that we do better than break even with training, but the long term relationships (and often future support customers) more than make up for it.

Our next training is scheduled for December. Perhaps I’ll see you there.

On the Nature of MySQL Closed Extensions

Matt Raykowski pointed out a couple of notes about the recent announcement by Oracle that it would begin supplying closed source extensions to MySQL, which it acquired when it bought Sun Microsystems (such as this one from Simon Phipps and this one from Monty Widenius)

I can’t say that the move surprised me, but I’ve been busy teaching this week and am newly obsessed with De Rerum Natura by Lucretius, which was rescued from obscurity by Poggio the Florentine when he found a copy at a Benedictine Abbey in Fulda, where I happen to be.

I believe I have tracked down that abbey, so I am now free to once again bore my three readers by discussing the open core business model.

First, when it comes to Oracle, one must surely realize that they are a commercial software company and their customers consist of people both used to and comfortable with buying proprietary code. I don’t really see this move as “open core” since Oracle has never, to my knowledge, described itself as an open source company. Instead, they join the ranks of Microsoft, Google and Apple who sell proprietary products based on open source code. There is no market confusion here, as the vast majority of companies who buy these products realize that they are buying commercial software.

I think it is more interesting to examine the role of commercially licensed software within the realm of an open source business.

To use MySQL AB as an example, I had no problem with the company both providing MySQL code under an open license as well as selling it under a proprietary license so it could be imbedded into commercial products. As long as all the bits are the same, I think it is a great way to fund open source development. We offer the same model with our “Powered by OpenNMS” program.

The problem arises when features are withheld from the “open” version. One of the powerful aspects of open source software is that the community of users and developers around it allow it to develop rapidly. OpenNMS is able to compete with products that have yearly R&D budgets in the tens of millions, mainly because features can advance rapidly due to a large, disparate development community.

This “withholding of features” is a key part of the open core model. It doesn’t work, and it is why open core companies are running away from it as fast as they can. Take Hyperic for example. Just recently on their messaging boards came a post asking about the next community release, since the “enterprise” release has been out for awhile now. Because the software has this separation, someone has to make the decision on how much of it to release as open. Thus, the decision on how to manage, test, distribute and improve the code has been removed from the community and placed in the hands of the vendor. The code is hidden until the vendor decides to release it, which all but removes the advantage of having it as open source in the first place.

Again, to contrast that to OpenNMS, anyone with access to the Internet can get to our git repository and see the latest and greatest code whenever they want. If they want to run our next stable release before its time, they can download testing (the next point release), unstable (the next major release) or snapshot (the raw, unrefined, latest code) which are built nightly.

The second major issue when talking about the role of commercial software within an open source business is the topic of copyright assignment. MySQL required that developers contributing code assign the copyright to them. Only the copyright holder can set the terms of the license, so Oracle would not be able to create a proprietary version of the MySQL database without it.

At OpenNMS, the discussion was lengthy when we were deciding on a copyright assignment strategy. We had been stung by the fact that a company had taken our code and turned it into a commercial product in violation of the license, and when we pursued legal action, that company’s response was that if they were using the OpenNMS code, it was from an earlier version to which we did not own the copyright. Now that The OpenNMS Group holds the entire copyright to OpenNMS, it makes such issues easier to defend.

We adopted, at the suggestion of DJ Gregor, the Sun contributors agreement, in which the copyright is assigned to the project but the contributor also maintains copyright.

There is one huge positive to this and one minor negative.

The positive is that the contributor retains ownership of their work. Think about it – supposed you wrote some part of MySQL and assigned the copyright to MySQL AB. Now you want to improve that code and submit it to another project. Since MySQL owns the copyright to the original work, you are bound by whatever license under which it is published. If it was published under the GPLv2, you wouldn’t be able to submit it to a project licensed under GPLv3 without express permission from MySQL. You don’t own your own work.

In the case of “dual copyright”, you still maintain ownership and you can license it any way you want.

The negative is that dual copyright may not be enforceable around the world, and it has not, to my knowledge, been tested in court. But this was the same FUD spread about the GPL in the beginning, and numerous court challenges have upheld the license’s validity. I am confident that should this ever be tested in court, it too will be found valid (just as if, say, William Gibson and Bruce Sterling decided to write a book together, they could each share in the ownership of the final product).

Publishing all the bits under an open source license (even if they are also sold commercially) and having a permissive contributors agreement that favors the developer are key to being successful as an open source business.

I’m afraid I have little sympathy for the developers of MySQL. They took their 40 pieces of silver when they sold themselves to Sun for US$1 billion. At that valuation, they had a ton of options, including an IPO, but somewhere along the way they made decisions that eventually led to Oracle owning the project.

I think this is more a lesson in foresight versus some strange and unexpected strategy by Oracle.

Testing, testing

Recently, a “Powered by OpenNMS” customer asked Matt Brozowski, our CTO, about the OpenNMS test plan.

I thought his reply warranted a blog post.

The way OpenNMS does testing is as follows:

We have extensive automated tests that run whenever code is changed.

The list of successful tests of the 1.10 release is here (as you can see there are 3743 tests)

Our automated build system does not create RPMs unless all of these tests have passed, otherwise it make a new RPM each night.

In addition to these tests, as a part of each build we also have a set of tests that run against an installed system (we call these Smoke Tests)

The Smoke Tests have automated scripts that install and configure a system and then uses the Selenium GUI emulator to validate that the GUI is functioning correctly. These tests are relatively new for us but I’m sure we will be adding to these tests of the coming months.

Lastly we release a milestone every month and we have a number of community members that install OpenNMS on systems in parallel with their production system and validate that the features that they are using still work with the devices that they have and report bugs as they come up.

After features are complete we make these milestones as release candidates and label them something like 1.9.9x and from this point on only bug fixes are allowed. Next week we will release 1.9.91 as a release candidate in preparation for 1.10.

This is the list of issues that are currently considered critical for release of 1.10 as a stable release.

Not all of these issues have been validated but these are the ones that remain to be considered.

After we release 1.9.91 these will be moved to 1.9.92 and so on until we get them fixed and can release 1.10.

I hope this helps give you an idea about the quality of our testing strategy.

And Who Says Nixon Is the Only One Who Can Go to China?

Please forgive me if this post is a little spacier than usual, but I was up from 2am until 6am this morning doing an upgrade at a client site.

When I came into the office, everyone was talking about a project Ronny Trommer spotted on Sourceforge called YbmNMS.

It appears to be a Chinese fork of OpenNMS.

I am always amazed at where our code ends up, and I think it is pretty cool that someone in China is using it. While I visited China in the 1990s (Guangzhou and Chengdu), I haven’t been back since, and we don’t have any commercial support customers in China.

I always worry, however, about whether or not projects that use our code obey the license. The code for YbmNMS seems to be hosted as the “Netcools” project on Sourceforge, and it appears that all that has been done, with respect to the copyright, is a global search and replace on the string “OpenNMS” and substituting “YbmNMS”.

From one file:

// YbmNMS(R) is Copyright (C) 2002-2005 The YbmNMS Group, Inc. All rights reserved.
// YbmNMS(R) is a derivative work, containing both original code, included code and modified
// code that was published under the GNU General Public License. Copyrights for modified
// and included code are below.
// YbmNMS(R) is a registered trademark of The YbmNMS Group, Inc.

Considering that not all of our code is there, YbmNMS may be focusing on just that functionality that is similar to IBM’s Netcool product, hence the Netcools name.

I tried to send an e-mail to the “mailto” link on their website ( but it bounced, so I sent a note to Yin Bangmin to learn more about what he is doing and to get the copyright notice corrected.

“The only thing worse than being talked about is not being talked about.”

-Oscar Wilde