Worst BBQ in Texas

I’m sitting here in my hotel room listening to the thunder outside. I’m in San Antonio this week, doing some work for Rackspace. Every time I’m in SAT I try to make it out to Rudy’s, home of the worst BBQ in Texas.

Five of us went at lunch and ordered a pound of brisket, a pound of turkey, a pound of chopped pork, five links and a quart of creamed corn. We ate it all.


Coté picked up on my post on support for a blog entry he was working on concerning commercial open source support business models. As usual he raises a number of good points, but as someone who has been running just such a business for over five years I always find the discussion of this here new business model slightly amusing, as our business model is actually quite old.

At the OpenNMS Group we sell services, of which support is one. We also sell consulting, training and custom development. “Software Support” is supposed to mean the same thing as the support you buy for any proprietary software product (but better). In our support Statement of Work we list the three types of issues that are covered by support: configuration, bugs, and enhancements.

A configuration issue is along the lines of “how do I get OpenNMS to do something”. The resolution is that we help the client with their configuration. We do not, however, do it for them (usually). For example, if there is a MIB file with a number of traps in it that the client would like to import into OpenNMS, we will help them use mib2opennms to generate the events, and show them how to modify one or two. If there are 100+ events we will not configure all of them. That would be out of scope.

A bug is “OpenNMS should behave this way, and it don’t”. Luckily these are much more rare, and we respond (after verifying it is indeed a bug) with a workaround and/or a code change.

Finally, there is the “how do I get OpenNMS to do something” where the answer is “it don’t”. In this case we are more than happy to work up a custom development proposal.

That’s how we define “support”. The different levels we offer differ in what’s covered, how many people can open support tickets, response time and hours of coverage.

Everything else on Coté’s list tends to fall under “consulting” or “custom development” in our company. Configuration, scaling, the big questions of “how do I get this application to provide value in my environment” are the real meat of our business.

This is really nothing new. When I used to work as a VAR, the software vendor would tell us that for every dollar in software we sold, we could expect eight dollars in services. Thus the real money for VARs was in “value added” services, not selling software. We just decided to change the model a bit and say, hey, keep your dollar, you can have the software for free. And, oh, since our tool is much more flexible than the stuff you’re used to, we can do the work for four dollars instead of eight.

What is new is the transition of traditional software companies to the services model. We’ve never been a software company, so it doesn’t really impact us. No one assumes a plumber can’t make a living if he doesn’t manufacture pipes and fittings, and no one assume a doctor can’t make a living if he doesn’t create the medicines he prescribes. So it’s amusing when I’m asked how do we make money selling free software. The answer is we don’t. Of course in the eyes of the traditional software company investor that is the wrong answer. There is no way a services company can be successful.

I, of course, don’t believe that. Almost half of IBM’s revenue comes from services, and that amount keeps growing. IBM is turning itself into a services company, but I doubt they will ever consider themselves a “support” company.

While I usually don’t name our clients, I bring up Rackspace because they more than any other company I work with seem to really understand support as a business. As I was wandering around the office today there was a lot of positive energy. Lunch was supplied by the company for the support staff, and it included hot dogs and hamburgers grilled outside along with a large number of other dishes.

“Fanatical Support” is their tagline, and they tend to deliver. I was really confused for the first couple of days of this trip because everywhere you go there are these McCarthy-era signs saying “Report Them!” and “Finger Pointing Encouraged” and I thought they’d had a rash of equipment thefts. I looked a little closer and in much smaller letters it says “Report Fanatical Behavior”. It’s a program to recognize those who go the extra mile.

This is nothing new at Rackspace. There is a wall of photos of outstanding “Rackers” wearing a “Fanatical Support” straight jacket that has been growing in number at least for the five years I’ve been working with them.

But the hard part is getting people to realize that while the software is free, the expertise to use it is not. Our happiest customers are those who buy a lot of services from us.

I really get the recurring joke about the doctor at party who is asked for free medical advice. I travel a lot, and sometimes I’m asked about what I do for a living. When I say “I work with computers” it is rare that it isn’t followed by a question about why their three year old computer is slow, or why do they get so much spam. This has happened, I swear, when I was wearing my “No I will not fix your computer” shirt. I could lie, but on occasion I end up meeting someone in the business who I really do want to talk to. But I do have a sure fire way to respond when meeting the other kind. I reply:

“Sorry, I wasn’t clear. When I say I work with computers I meant I recycle them.”

"Help! I Need Somebody"

Something has been kind of nagging at me lately. Why it is that in the IT world, the task of “support” is held in such low esteem?

You know what I’m talking about. I spend time with a lot of organizations, and rarely do I see support people treated the same way as, say, developers. In fact, support people tend to be the most overworked and the least appreciated technical people in most companies, and I’m not sure why.

Of course, I have some ideas.

Prior to open source, software support was a cost center. The company made money on selling software licenses and to a large extent support contracts, but the best they could hope for was that the clients would buy the software and never use it. When they actually had to provide the support they sold, it would eat away at the bottom line.

With the rise in services-based business models, services and support can represent the majority of a company’s revenue. But even in those companies it is rare that a top support person will be paid as much or have as much chance for advancement as a top coder.

The OpenNMS Group makes most of its money through support, and everyone in the company can be considered part of the support team. We’re small enough that each and every support ticket is sent to the whole company. As we are growing, I’m trying to find people who get into support as much as I do, and it’s hard.

I love support. Well, to be honest, I love support most days. It’s a huge challenge, and I think its fun to try and figure out why systems aren’t behaving the way they should. Is it a bug? A misconfiguration? A problem with the operating system? Loose nut on the keyboard?

My first real support job was with a division of Harris called Harris Digital Telephone Systems (HDTS – no longer around). When you joined up, they put you through several weeks of support training. I really took to their product, so the veterans who were training me took special joy in trying to make the classes challenging. At one point we had to configure a telephone switch, and then they would go in and muck with it. We had to correct the problems they introduced. But for me they added a twist: I had to do it over the phone from the next room.

It’s one thing to be able to fix a problem on a system you have direct access to, but it is a lot harder to figure it out strictly from what the person is telling you on the phone. This is one reason we got a WebEx account to enable an easy way to get connected to a client’s OpenNMS install. The hardest thing to deal with is a client who knows just enough to be dangerous. In my years of support I have learned that there are times with the client will lie to you. They don’t do it maliciously, but they know enough to make some assumptions that aren’t always safe. To many it can be frustrating, but to me it just adds to the challenge.

Yes, I’m weird. But as I go looking to hire some more people for support roles I’m finding that it is hard to find someone who feels the same way as I do about the job. I have had a number of people approach me about a position as a developer. I need to hire one of those in the next couple of months, but to be honest we have a pretty solid development community as it is. I’ve had a couple of people who want to come on and do “business development”. While we could do a better job at sales, quite frankly we have about as much on our plate at the moment as we can handle.

So why am I not overwhelmed with people wanting support jobs?

1) Support is Hard

Not to knock developers, but writing code is much less stressful than handling support. There are pressures to be sure, but the number of unknown variables to be managed in a coding project is less than those in the average support issue. I constantly face support issues that could be due to configuration issues, the customer’s network, the server, the operating system, or bugs. Last week alone I dealt with one client who was having issues because they used Solaris tar versus GNU tar on a build (Solaris tar tends to truncate long file names) and another client who had bad memory in the server (which resulted in a number of crazy symptoms). Those aren’t easy to catch.

Add to that the fact that you are interacting directly with the client, and there is a lot of pressure to perform.

2) Support Gets No Respect

Before you jump on me for this one, think of all of the stereotypes of bad support. The overseas help desk. The recent story about Apple Geniuses not being able to figure out that the ringer on an iPhone was turned off. Everyone who has owned a computer can provide at least one support horror story, or they haven’t really used a computer.

Part of the reason is that employers don’t treat the support role the way they should. They think that grabbing someone off the street and sending them to a class is sufficient. I was talking with the CEO of a small software company a few months ago, and I asked him about his support team, and he pointed to the two interns in the corner. This is a guy who spent hundreds of thousands of dollars on his developers, but thought nothing of hiring the cheapest labor out there for support.

One sign of respect in this business is salary, and support positions tend to pay the least in a number of IT fields. Many people only accept the job in order to get their foot in the door to move up to a consulting or coding position, which brings me to …

3) No Advancement

How does one advance within a support organization, yet still do support? There’s always management, I guess. Within development you can always aspire to increase your responsibilities for given projects, but within support the only real upward movement is from first level to second, and from second to third.

In thinking about it, this isn’t really a fair observation. When I was at Northern Telecom there were about 13 “Salary Bands” yet it was extremely hard to get above a Band 7 unless you wanted to manage people. But I still think that there is more room for growth in areas such as programming versus support.

I don’t have any answers to these problems, but I’m working on ’em. The first step is to find a core group of people who really get into support, and then to refine the process for taking inexperienced yet motivated people and turning them into support gurus. I can see the ads now: It’s Like Sudoku on Steriods!

Second is to pay good support people well.

As far as advancement is concerned, I like to take support folks and make sure they do some on-site consulting with clients as well. It is real important to get face time with the people using your product, as well as to step back from the ticket queue and focus on one client and their problems for a change. Plus we get to travel to some neat places and work on some of the world’s coolest networks.

But none of this works without respect. At OpenNMS the support role is held in the highest regard, but I am as guilty as the next person when using the support services of other vendors. I’m going to make a point of giving some “props” to the next good support person I deal with. They deserve it.

Pittsboro, Jewel of the South, Gets A Brewery

People often ask me, “Tarus, even though you are over 40, how do you keep looking so darn good?” The answer is a steady diet of fine fermented beverages, usually involving hops, barley, malt and pure water. In fact, about once a year I take a trip to England (which in the Saxon tongue means “Land where everything costs double”) simply for health reasons, and since beer is the only thing affordable over there I pretty much live on a liquid diet (again, it’s for health reasons).

Recently, the Carolina Brewery opened its second location in my small town. A harbinger of development to come, it is at least locally owned. I was looking for an excuse to visit when I found out that John Willis was going to be in nearby Durham, NC.

Me and John hoisting pints of Copperline Amber Ale

I first met John at this year’s OSCON, and again at LinuxWorld. John reminds me a lot of Doug Stevenson. Doug has forgotten more about network management then I will ever know, and back in the OVForum days he was pretty much the “Elvis” of the OpenView world. John seems to be his opposite number on the Tivoli side of things.

Having worked in the network management world for a number of years on the proprietary side of things, John is getting real excited about the benefits of open source solutions. It was fun chatting with someone who obviously knows his stuff and sees the potential of what we are doing. Whether they admit it or not, the so called Big Four have stagnant product lines that are slow to react to changes in the market, and since they have been reliant on professional services for so long it isn’t too much of a stretch for those consultants now using their solutions to find it faster and easier to use open source.

John wants to get the Enterprise Systems Management (ESM) world together to address some of these issues, and he really wants me to come to BarCampESM. I am a big fan of the BarCamp format (I missed this year’s BarCampRDU ’cause I was on the road), but I am not certain that it will provide the format he is looking for. But it is in Austin (I’ll be in San Antonio next week) which I like, and I think I’ll have to show up to keep just to keep everyone honest (grin).

whurley, one of the organizers, now works for BMC (one of the Big Four) and I have my own little history with PATROL and the Boole and Babbage stuff, so while I like him I still have to remain skeptical of the motivations of BMC. There has been a lot of talk lately of successful open source projects being acquired by commercial companies (forgive me for not being able to find a relevant link, it seems they have slipped out of my RSS reader). One has to wonder if it is because these commercial companies have had a change of heart or if they just want to get a handle on, and perhaps stifle, projects that threaten their bottom line. But I like whurley, as does Ethan Galstad, and so I’ll probably go with my gut and at least listen to what he has to say. Plus it looks like Coté will be there, and I want to meet him in person.

Oh, speaking of Ethan Galstad, check out this Splunk ad that came across the screen when I was working on Sourceforge. Studly. Too bad he’s too skinny. Perhaps he can join me on a trip to England.

For health reasons, of course.

Back in the office

Yes, we’re open.

Neon Sign on Ben Reed’s Desk at The Office

I’m trying my damnedest to get caught up after my three week absence. I hope to get my final thoughts on Dev-Jam and some notes from LinuxWorld down before I forget them all.

I did manage to get the new training schedule up. If you really want to get the most out of OpenNMS, this is a great opportunity to learn about it from some of the people who make it, and to visit that rare jewel of the South, Pittsboro, NC.


Dev-Jam 007: Day 4

While a lot of positive things happened today, the tragedy that hit Minneapolis pretty much overshadowed them.

At 6:05pm local time, the Interstate 35W/Broadway bridge collapsed into the Mississippi river. This was within a short walk of Dev-Jam, and the area behind our building became a staging area for the rescue effort.

Rescue workers gather downriver.

Cell phone service was non-existant as the locals called to insure that their loved ones were okay, and we listened to the sounds of sirens as the city mobilized to help the injured.

Injured transported on trucks.

So rather than focus on Dev-Jam, our thoughts go out to the people of our host city and we hope that the recovery is swift.

Dev-Jam 007: Days 2 and 3

Sorry for the lack of updates. It’s been pretty crazy around here.

Monday Dev-Jam kicked off in earnest. Robert Hanson passed around the GWT kool-aid, and several coding efforts got underway to use GWT for charts and maps. fastjay and vwdude started implementing a RESTful architecture into OpenNMS so that data could be more easily shared between instances of the application.

On Tuesday I finally got to meet in person someone that I’ve known for several years via e-mail: Ethan Galstad. For those who don’t know, Ethan is the creator of Nagios, one of the first open source management tools and one of the most popular.

Me and Ethan Galstad

Note the shirts.

I was happy to discover that Ethan is as cool in person as he is in e-mail, which is not always the case. In the last couple of years the management space has really taken off, and it was nice to get his take on the emergence of a number of so-called “open source” management options. Both Nagios and OpenNMS have been around for over seven years, so we’ve experienced some of the same issues, although the fame of Nagios far eclipses that of OpenNMS. We’ve both had to deal with trademark issues, as well as how to maintain and encourage a growing community without succumbing to help vampires. We got to chat for about six hours, and the gang probably would have kept him there longer.

Ethan will be coming out to LinuxWorld and I think it would be cool if he happens to be visiting our booth in the .org Pavilion when someone asks the invariable question of “How is OpenNMS better than Nagios?” in a tone that implies we are somehow enemies. They are different tools focusing on different things: Nagios on monitoring in depth while OpenNMS has a broader agenda. It’s all about what works for you.

True open source is a meritocracy, not a marketing exercise. Respect in open source can’t be bought and must be earned, and at OpenNMS we hope to one day have earned the respect that Ethan enjoys. We plan to do that by staying true to the ideals of free software, and we hope by example to show that people can work on free software and make money at it, and not at the expense of the community.