Archive for January, 2010

Why We Do What We Do

Friday, January 15th, 2010

“There are two goddesses in your heart,” he told them. “The Goddess of Wisdom and the Goddess of Wealth. Everyone thinks they need to get wealth first, and wisdom will come. So they concern themselves with chasing money. But they have it backwards. You have to give your heart to the Goddess of Wisdom, give her all of your love and attention, and the Goddess of Wealth will become jealous, and follow you.”

- Dr. Joe Vigil, quoted in Born to Run by Christopher McDougall

I am reading a book David loaned me called Born to Run. It discusses the Tarahumara people who live in the Sierra Madre mountains of Mexico. The book focuses on the fact that these people can run hundreds of miles without stopping or getting hurt (I also found it interesting that their economy is based, in part, on the trading of favors).

In trying to determine why these people can run so far so fast, it appears that the main reason is that, quite simply, they like to do it.

When I was younger I kept hearing from older people “choose something you like to do for a career” and I kind of ignored them. I wanted money, so I was going to choose the career that provided the best path to financial security.

It was only later that I lucked into doing something that I loved. When I became the OpenNMS admin in 2002, I went from a solid six figure salary to sometimes earning $300 a month. That was way less than minimum wage (and was offset by better months), but at least I enjoyed the work. We made adjustments to our expenses and I was surprised to find that at the end of the first year I had spend only $5000 out of savings.

Things took off from there, and I managed to attract amazing people who also loved what they were doing. We decided on a mission statement of “Help Customers – Have Fun – Make Money” with the emphasis on the first two. The money showed up. While all of us could make higher salaries in a more traditional job, we are content for now to use our profits to build a better company, since that increases our ability to both help customers and have fun doing it.

I can remember bringing up open source software on the OpenView Forum mailing list many years ago and being called a communist. I am far from it, although communist and community do share the same root (and I am very interested in the latter). I just want to create an environment where people are rewarded for doing good work, and the best way to get good work is to find people who enjoy and even look forward to doing it.

Ultimately, I think this is the best way to make money. Provide value and you will become valuable.

I end every employment offer letter I make with the same sentence. It has nothing to do with money or even network management, but it distills into five words why I get out of bed in the morning: Let’s go do great things.

“Let us live so that when we come to die, even the undertaker will be sorry” – Mark Twain

Upcoming OpenNMS Events

Wednesday, January 13th, 2010

The new year has been no less hectic than the old one, which is actually a good thing, I guess.

Yesterday was my 44th birthday. It was pretty much just another birthday, with the only somewhat interesting thing is that now my age starts and ends with the same numeral. It is also the cost in US cents for a first class postage stamp (which has been the case with my age since 1998).

Anyway, I did get a delightful and unexpected present. The gang at the Southern California Linux Expo (SCaLE) issued a press release announcing that I have been named one of the two keynote speakers for the conference in February.

This is pretty cool, and I am both honored and humbled by their decision. I will be presenting a talk about what it is like to start and run a business based on open source software. One reason I write this blog is to both inform, and perhaps inspire, others to take the risk and start their own business, and to be able to talk about this in front of the SCaLE attendees is pretty exciting. It has been one of my favorite shows since we first attended. OpenNMS will also have a booth where you can see the latest stuff we are working on, as well as to meet some of the folks from the community.

This will be my first ever keynote, if you don’t count last year’s inaugural OpenNMS Users Conference in Frankfurt, and speaking of that, we had so much success last year that we are doing again. The 2010 Users Conference will be back in Frankfurt, and we’ve extended it to two days to include a day of workshops. Nethinks is once again a major sponsor along with the OpenNMS Group, and we will hold it on 6-7 May. If you are interested in being a sponsor, please drop me a note.

We’ve had a great response to our Call for Papers, but if you are interested in presenting, there is still time as it doesn’t close until 31 January. Accepted speakers will receive two nights in the hotel and all meals, but you will be responsible for getting there. We are looking for both presentations on any aspect of OpenNMS as well as workshops.

I’ll post more information on attending once we’ve settled on the agenda.

Finally, we’ve added new training dates in April (the January training is booked). If you are finding OpenNMS a little daunting, consider coming to metropolitan Pittsboro for one of our training courses. They’re useful and a whole lot of fun.

Health Savings Accounts

Tuesday, January 12th, 2010

Note: My one reader outside of the US might have issues understanding the “purchase” of health care insurance and should probably skip this post.

I can remember how happy we were in March of 2005 to finally be able to offer health care as a benefit.

The OpenNMS Group is a company focused on the happiness of its employees since they are responsible for all of our revenue, and we don’t want anyone worrying about being sick or whether or not they should go to the doctor. We cover 100% of the employee’s insurance premium: medical, dental, disability, etc.

There is also the benefit of having a group plan so that the premiums for family plans (which we don’t cover) can come out of pre-tax dollars.

This has worked fine for over five years, and I’ve been happy with our provider, Blue Cross Blue Shield of North Carolina. Well, as happy as one can expect to be when dealing with an insurance company (I’m also happy with our lawyer, if you can believe that).

However, this year our premiums went up nearly 27%. This is more than twice what we usually see and it was strange since no one hit an age barrier (30-34, 35-39, 40-44, etc.) or had (as far as I know) a catastrophic illness. Think about it – the premiums would double every four years if this was a constant rate.

That’s about as bad as some of those open core license fees (grin).

Anyway, our options were to either pay more, consider having the employee shoulder some of the burden, or change plans. All of which are a bit of a pain.

But then our broker suggested a Health Savings Account (HSA) combined with a High Deductible Health Plan (HDHP).

An HSA is basically a savings plan, similar to a 401(k) retirement plan, that can only be used for health expenses. The HDHP is similar to a standard health plan with one main catch – no co-payments. This means if you go to the doctor, you play the insurance company’s negotiated rate in full. The same thing goes for prescription drugs. You pay everything up to the deductible, and then after that the insurance company covers 100% (or possibly less, but we’re looking at a 100% plan).

The HDHP premiums can be significantly less than the “normal” premiums.

The way this is supposed to work is the higher deductible means lower premiums, for both the company’s individual coverage and the employee’s family coverage. Both the company and the employee can fund the HSA, and the goal would be to put enough in the HSA to cover the deductible.

For example: supposed you have an HDHP with a $2500 deductible for an individual, and the company puts in $200 a month toward that account for $2400 a year. Any expenses up to $2400 are paid for, and the employee would only be responsible for the last $100 up to the deductible. Any expenses after that are fully covered.

Better yet, say you use only $500 of it, then there’d be $1900 left over for the future. And, while it isn’t much these days, these accounts can earn interest.

With families it is a little different, as the deductible tends to be doubled. In this case, the added savings of having a reduced premium can be applied to the HSA to cover that difference.

My favorite part is that there is nothing preventing you from contributing just the amount of your expenses to the HSA at the end of the year and immediately withdrawing it, tax free.

In the short term, an HSA that is funded by the company does not save much, if any, money for the company. But over time, as premiums increase, it could be possible to raise the deductible so that eventually it does, or is at least at parity.

The (small “L”) libertarian in me loves the idea. I like the fact that people can use as much or as little healthcare as they choose. Plus, any funds that have been accrued can be taken out at retirement age similar to an IRA (and also, hardship withdrawals with tax penalties are also available).

We’re still investigating whether this option is for us, but my current take on it is that most of the guys are for it.

Having a decent healthcare plan is important for any business in the United States, so I thought I’d share.

Rackwatch and OpenNMS

Sunday, January 10th, 2010

One of my favorite clients is Rackspace Managed Hosting. They have been using OpenNMS since 2002, and they were either our second or third official customer (I can’t remember if they came before or after NASA), and I’m happy to say that they are still a client.

We like to describe OpenNMS as a network management application platform, and what that means is that it is easy to integrate OpenNMS into other systems to build a custom and unique management solution. In the case of Rackwatch, OpenNMS is integrated into an internal Rackspace system called CORE (CORE Objects Reused Everywhere).

We have a server hosted at Rackspace which we, of course, monitor with OpenNMS. This morning I got an e-mail from our OpenNMS system:

Subject: Notice #19845: HTTP down on 10.1.1.1 (10.1.1.1) on node server.opennms.org.
Date: January 10, 2010 7:46:07 AM EST
To: Tarus Balog

The HTTP service poll on interface 10.1.1.1 (10.1.1.1) on node server.opennms.org failed at Sunday, January 10, 2010 7:38:56 AM EST.

What I though was cool was that at approximately the same time I got an e-mail from Rackspace:

From: support@rackspace.com
Subject: Created Rackspace Ticket #100110-01016: Service Down: Webport (Computer #40906)
Date: January 10, 2010 7:39:59 AM EST
To: Tarus Balog

Dear Tarus Balog,

The following support ticket has been created:

Ticket #: 100110-01016
Subject: Service Down: Webport (Computer #40906)
Status: Confirm Solved
Account #: 14290 (Sortova Consulting Group)
Date: 01/10/2010 6:39am CDT
Comment:
————————————————————————

The Rackwatch monitoring system was unable to reach the
Webport service on computer #40906.
It may be down.
————————————————————————

Note the times of the notices were nearly identical, although this is more of a coincidence since our polling rate it set at 5 minutes (I believe Rackwatch is more frequent).

I just thought it was cool to actually experience OpenNMS in action at such a large company (we have instances of OpenNMS running in all nine data centers as part of the Rackwatch application).

Anyway, this notice allowed me to fix an issue with the Apache config on that server, and I dutifully got “resolved” messages from both Rackwatch and our own internal instance of OpenNMS when it was detected as being back up.

Sometimes you get so caught up in the internal issues with running a project that you forget that people actually use it, and it is nice to think that we play some small role in helping Rackspace provide fanatical service to their tens of thousands of customers.

More on Copyright Assignment

Friday, January 8th, 2010

A couple of days ago I saw a post by Dana Blankenhorn continuing the discussion of copyright assignment in open source. He pointed to a piece by Michael Meeks that he describes as:

This may be one of the most important papers on open source since The Cathedral and the Bazaar. It is well worth your time to read it in full.

With a comment like that, how could I not read it?

The main driver for a lot of this discussion is the impending acquisition of Sun, and thus MySQL, by Oracle. MySQL had a policy that any code contributed to the project required that the author assign the copyright to to the MySQL corporation. This gave them full control over the application, including the ability to publish it under different licenses.

The problem I foresaw with this was that some contributors would be reluctant assign copyright, and thus community contribution to MySQL would be lessened. This really didn’t seem to affect MySQL at all, and the fact that they “owned” 100% of the code definitely helped them get bought for US$1 billion by Sun. Their ability to generate revenue from that code was also responsible for their rapid growth and in a large part helped make the MySQL database what it is.

But now that MySQL looks destined to be owned by Oracle, people are worried that Oracle won’t put much energy into the project. MySQL was designed to be a replacement for Oracle’s bread and butter products, so it is obvious that as the new owner, Oracle won’t be working too hard to put itself out of business, thus the only real question is how much less effort will be put into MySQL. Since MySQL was published under the GPL, it would be very hard for another company to commercialize it, which will limit the chances that a well funded fork could be created. MySQL’s future growth looks pretty small, or at least much smaller than it could have been under a different owner.

Because of this, one of the MySQL founders, Monty Widenius, has even started a petition to prevent the sale by appealing to the EU.

Now I’ve had a couple of e-mail exchanges with Monty and I found him to be a very friendly and nice guy, but I’m a little puzzled by his actions. One can only assume that as a founder of MySQL he made quite a bit of money on the sale of the company, and that he was also instrumental in creating the company that eventually was sold. Thus his decisions led directly to this predicament. It seems to me, well, “wrong” to accept VC money, have a record breaking buy-out and then want things back the way they were, so I have little sympathy for this effort. Perhaps instead of trying to block the sale via legal channels he would be better off forming a foundation to try and purchase MySQL back from Oracle and then he could make it as free as he wanted.

Anyway, had MySQL been licensed under a more permissive license, or had contributors not assigned copyright, it would make it much easier for a third party to step in and commercialize a fork. I responded to some of Brian Aker’s comments on the subject a few weeks ago, and now I thought it’d be fun to examine those of Michael Meeks.

The two main points I took away from Michael’s paper were that copyright assignment should be avoided, and that one should use a permissive license. This is in order to build “a social environment built for the common good”.

I can’t really disagree with his conclusions. I believe that, yes, if one wants “a diverse, and thriving developer community” creating software with the most freedom is important. But it is based on one of the biggest flaws and misconceptions about open source, which is that simply by being open source thousands of qualified people will give up nights and weekends to work on your project.

I would never base any business plan on altruism. In fact, I don’t think it exists, at least on a large scale. People are selfish, and they do things in order to bring themselves personal gratification. The trick is to align those things that benefit individuals with those that benefit the group.

In many cases the ideas that Michael describes work. If you take a look at some of the successful open source projects, the end users are also developers (Spring, JBoss and to a lesser extent MySQL). The product that is sold is usually built using the open source tools, but it is not the tool itself. Thus having permissive licenses can make this very beneficial for all involved, since they are then free to commercialize the final product as they see fit.

But what happens when the project is aimed at end users and not developers? Take OpenNMS for example: our end users are network and system administrators, not Java coders. The project is the product. In order to develop this software, someone has to write it, and the most qualified coders tend to have things like mortgages, car payments and other needs that require money. It’s fine to preach altruism when you work for a large company like Novell or Sun, but what about small companies that are dedicated to open source? How can they make money and protect their work, while still remaining true to open source ideals?

In my own experience with OpenNMS we had a company that took our project, made some changes to it and distributed it in violation of the license. They had raised several million dollars in VC money and thus were able to hire the resources necessary to rapidly advance the application, and they claim to have made millions more selling, ultimately, the work of our community.

Had OpenNMS been published under a permissive license, this would have been perfectly legal. Thus the work of a small but dedicated group of people could have easily been commercialized by a larger company with more money. But since OpenNMS is published under the GPL this was not permitted, so we decided to pursue legal action.

The first thing you learn is that you are on your own. No one really cares that someone is abusing an open source license, especially if the code being stolen is maintained by a commercial institution. Luckily we were in a position to afford to hire a legal team.

Then we hit the second hurdle. At the time no single entity held copyright to the OpenNMS code. All code up until version 1.0 was held by Raritan (which had bought the assets of Oculan after they went out of business) and most of the remaining code was held by the OpenNMS Group. The company in question claimed that if it was using the code in violation of the license, it was only the code for which Raritan owned the copyright, and thus we had no recourse, as only the copyright holder can enforce the license.

It took us a year working with Raritan before they could join us in pursuing this company, and in that time the company ripping off our community’s work tried to clean up their act by releasing a fork of OpenNMS. While I can’t see how that fork would absolve them of their licensing issues (OpenNMS is a Java program published without the classpath exception, so simply importing OpenNMS classes is the creation of a derivative work under the GPL and there is no way this forked code could have been used without importing those classes) it did muddy the water quite a bit. I also found out that the legal system in the United States is reluctant to award damages based on software that used to violate a license, even if that software was sold for large amounts of money.

Before we could pursue it much farther, that company closed its doors. Whether our actions had anything to do with it, I don’t know, but part of me likes to think that there were some consequences for the theft of our code. But we did get a benefit: Raritan was willing to sell us the copyright to the code we didn’t own. It wasn’t cheap (two houses had to be mortgaged to cover the cost of the loan) but it was fair.

Once we purchased the copyright to the 1.0 code, all that remained for us to be able to defend OpenNMS from cases like this in the future was to reach some sort of agreement concerning copyright with the 40 or so contributors to OpenNMS since 1.0. Copyright assignment seemed to be the best way to go, but it didn’t seem fair to me. For example, suppose a member of our community comes up with a cool algorithm for doing some task and they integrate that into OpenNMS. Copyright assignment would mean that they were giving away that work, and if they wanted to reuse it in the future they would have to license it back from us. While it is important for all of the OpenNMS code to have a single owner, that was not fair to, and definitely discourages contribution from, the original programmer.

Our solution came from an OGP member named DJ Gregor who suggested that we adopt the Sun Contributors Agreement (SCA). This introduces the concept of dual-ownership: the copyright is assigned to a third party yet the author also maintains copyright. While this has never been tested in court, I trust the Sun legal team that it will hold up. I was happy to see that this meets, somewhat, with Michael’s approval, as he mentions the SCA a number of times in his article.

Thus, based on my experiences with OpenNMS, for a small company trying to make it with a business based on open source software, I think restrictive licenses like the GPL are crucial, as is copyright assignment.

The key part of any community is trust, and open source communities are no different. We don’t have huge numbers of people outside of the company contributing code (heck, we tend to hire the most prolific coders) but we do have an active core of people that help keep the project moving forward. The way we’ve been able to maintain that is by promising that no matter how much OpenNMS grows or is packaged in the future, the source will always be 100% available under an open source license. This is a promise we maintain by doing all of our development publicly – even custom development projects have their own branches in our git repository.

And we truly do listen to our team (DJ’s suggestion of the SCA as a case in point) even if they aren’t employed by the commercial side of the project. They are empowered to help determine the direction of the OpenNMS even though they don’t work for the company.

I think it is easy to describe a utopian world where all software is free, especially when your paycheck doesn’t directly depend on revenue from that software, but for a company that wants to both generate revenue and remain 100% open source, some sense of ownership and control is necessary.

I agree with Dana that the article is definitely worth reading, although I’d stop short of comparing it to The Cathedral and the Bazaar. Also check out the links at the bottom of Michael’s post. It includes the aforementioned article by Brian Aker as well as a great one by Bradley Kuhn called “‘Open Core’ is the New Shareware“.

At OpenNMS we try to avoid the pitfalls of open core commercial software by publishing 100% of our work publicly, but that requires such things as a restrictive license and copyright assignment. As with any situation involving trust it takes time to build, but we hope to continue to earn it.