The Centurylink Amateurs Are At It Again

Hey, #centurylink, if you want to play with the big boys, you are going to have to invest in qualified people. Not people who take out business networks for hours at a time.

I was looking forward to starting my Labor Day vacation a little early today, but that’s not going to happen. About 2am this morning Centurylink made some changes to their network (both my home DSL circuit and my father’s, who lives an hour away, got new DHCP address) and the network at the office went down completely.

When I called, the automated voice told me that they were aware of the outage and that it would be fixed by 4pm.


a) in what universe is a business class circuit allowed to have a 14 hour outage?

b) who does a major network change on a Friday?

c) who does a major network change on a Friday before a major holiday?

d) If you can’t plan an outage, including disaster recovery, to last less than an hour or two, get out of the business.

This was on top of a routing loop last week that almost cost me a major customer (since we had a demo planned and couldn’t reach the server we needed).

I waited a couple of hours and then decided I wanted to vent. So I called Centurylink back. I didn’t expect it to get anything fixed faster but I figured I’d at least get a “mea culpa” and a little “we screwed up – sorry”.

The monkey in first level who answered the call not only did not apologize, but seemed to act if 14 hour outages were normal. When I explained to him that my business depends on that circuit and in fact we’d flown a guy up from Atlanta in order to work today (but that couldn’t happen from the office, now could it) he dismissed my concerns with “well, it’s an area outage, not just your circuit”.

I then pointed out that my home DSL line was fine, and since I live 10 miles from the office it obviously wasn’t an “area outage”. His suggestion? We needed to upgrade to a T1.

A T1? Was is this, 1993?

When I pointed out that a T1 was only 1.544 Mbps he told me I was wrong, that it was much faster than the 10 Mbps I was getting now. I suggested that there might be some sort of technology one could run over a T1 that would result in higher realized speeds (i.e. DSL over a T1 versus a POTS line) and that must be what he was referring to, he continued to insist that a T1 was much faster.


I then asked to speak to a manager, and was told I couldn’t but that the monkey would be happy to relay any concerns I had.

I’m going to do it myself after the Time Warner guy gets back to me with my new office circuit. I recommend that anyone running a business that depends on Internet access stay as far as possible away from Centurylink, and that probably goes for voice service as well considering the level of customer support they find acceptable.

Another Story About Broken Healthcare

I had a wonderful weekend away on the North Carolina coast, but when I returned home I found that I had a letter from Blue Cross Blue Shield of North Carolina:

Dear Group Administrator:

According to our records, your group insurance premium has not been received. As a result, claims for services rendered after 02/29/2012, are now suspended.

This was followed by:

If you are planning to terminate your group coverage, please be advised that under North Carolina General Statute 58-50-40 employers are required to give employees 45 days prior notice of insurance cancellations. Violation of this law is a felony.

What a way to harsh the mood. First I’m informed that claims will be denied and then I’m threatened with a felony.

The part that pisses me off the most is that I pay all premiums on time. I pay my bills. On time. I have no idea what our business credit score is, but my personal score is over 800 at all three major reporting agencies. I take debts seriously.

This letter is a result of a bug in their software. It turns out that our annual renewal starts on 1 March. This is when our premiums increase, on average 30-35%, and for some reason this causes their billing system to invoice us late.

Usually, the insurance premium invoice arrives at the beginning of the prior month – i.e. I already have an invoice for April that’s due on the 22nd of March. But because of their billing system issues, my March bill did arrive until March 1st, with a due date of the 12th.

I paid it on the 8th, and according to my bank, BCBSNC cashed the check on the 12th – the same day they sent me a letter for non-payment.

What’s funny is that I have a collection of six of these letters. Yes, every single year they do this to me, and I’m not taking it anymore. In the spirit of Karen Sandler, I’d decided to take them to task for their crappy software.

So I called 877-237-6275, the number on the letter I received from “Sonya Walker, Director of Membership Operations” where I sat in queue for 20 minutes. I was then greeted by “Chris” who told me, oh, yeah, your account is paid in full, and that my payment and the letter must have passed in the mail.

I informed him that this wasn’t the case – I was never in arrears and that I was about to get very angry. If he wanted to, I suggested he transfer me to a supervisor, because I was getting ready to yell.

He decided to transfer me. Wise man.

After another 20 minutes of waiting I found myself talking to Misha Newman. I explained the situation again that I was tired of being told I was delinquent in my payments when it was untrue, and she replied that their software couldn’t handle renewals, which is why the letters get sent.

I asked for a bug report number.

She was a bit confused, so I told her that I’ve been getting these for six years now and that it needed to be fixed if they want to keep me as a customer. You can’t just send out threatening letters and then brush it off. If BCBSNC can’t manage the simplest accounting issue, doesn’t that cast serious doubt on their ability to manage things more complex things like medical claims?

I want someone in their IT department to at least document the issue and give me a way to know when it has been fixed.

I also demanded an apology from “Sonya Walker” if she even exists. It is quite common for large companies to make up names and positions for letters like the one I received. When someone calls to complain, it gives them a code to route the call. Ms. Newman informed me that Ms. Walker was no longer with their group, but that she did exist, and she would be happy to get her director to send me a letter. That should be interesting.

The sad part is that I doubt the alternatives to BCBSNC are any better (suggestions welcome). I think it is incredibly callous of them to value my time so poorly. On just this call alone I spent nearly an hour. You might say that I brought it on myself, which is partially true, but I also have over a dozen people that rely on me to make those insurance payments and I can’t be sure that there wasn’t some other mistake – so I’m forced to make that call.

I can’t help but think if health care was more transparent and used more open source ideas, if not software, that problems like this would be less common and easier to fix.

MonitoringForge, RIP

The following post contains language that some may find offensive. If that describes you, please skip it. I don’t aim to be crude for the sake of being crude, but sometimes certain words can’t be replaced.


At times, I am an asshole. I know this. Being human and full of faults I sometimes let my emotions get the better of me. But sometimes I believe I am unfairly portrayed as an asshole. It’s just that my bullshit meter pegs a little quicker than most.

Back in 2006, the Open Management Consortium was announced to a lot of fanfare. We were asked to join at the 11th hour, but something didn’t seem quite right to me. While the idea was strong, in order to get a bunch of separate and often competing companies to play nice together involves a lot more than a mission statement. It requires a tremendous amount of work and time.

Needless to say, the Consortium didn’t get much traction outside of a lot of press, and even a reboot in 2008 didn’t help. The domain name is now parked (and possibly up for sale it seems).

Jump forward to 2009 and the formation of MonitoringForge. A similar organization as the Consortium, I was just as wary about it and I said so. This got me labeled as an asshole, as in “why can’t you just be nice for once and stop being so negative.” It’s just that it seemed to be to be nothing more than a marketing ploy, and I felt the need to tell people about it.

As I was looking for my Sourceforge password today (I had to change it earlier this year due to a security issue and couldn’t remember what I had changed it to), I came across an exchange I’d had with Ross Turk about MonitoringForge and decided to see whatever happened to it. The website just says “The Monitoringforge Service was stopped as of June 30.” and lists a contact e-mail.

Now I’m not sure if it was this June 30th or last year’s June 30th (their twitter feed stopped in May of last year), but I think it is funny that the site was launched with such hoopla but died with nary a whimper.

The natural asshole part of me brings this up with a touch of schadenfreude. But the reason I’m writing about it has a direct bearing on open source and running an open source business.

When you are just starting out, the temptation is to do anything, simply anything, to get noticed. If you are a VC-backed company, marketing and perceived value is more than half of the business plan. But in the long run, your customers and your potential customers will put more value into your words and actions than your press releases.

I work hard not to involve OpenNMS or lend my name to any endeavor in which I am not 100% confident. It’s hard enough to get a large company to trust in a small company without having a string of failures to cast doubt. When I tell our clients that I plan for OpenNMS to be around in 10, 20, or even 50 years I mean it, and while they may not fully believe it, at least I haven’t given them any reason to worry by trading that trust for some short-lived notoriety.

Another thing I do that earns me the “asshole” or “zealot” title is rail against the improper use of the term “open source” when it comes to business models. Matt Aslett wrote recently about a number of companies dropping the term “open source” from their marketing, and he listed several of them. He wrote “The list below represents a small and unscientific sample, but these are among the highest profile open source-related vendors” which included companies like Zenoss and Groundwork.

We didn’t make the cut. Perhaps if we had joined in the Open Management Consortium (like Zenoss) or MonitoringForge (like Groundwork, who built it) we would have been “high profile” enough to make his list. But since the article was basically about a retreat from open source I can see at least one reason why we were omitted. It could also be why Zenoss is facing a user revolt and possible fork and we, thank goodness, are not.

I present this as a cautionary tale. In business, focus above all else on your customers. Be completely truthful and never get involved in anything you do not believe in 100%. Remember the sage Tony Montana who said “All I have in this world is my balls and my word and I don’t break them for no one.”

How Not to Get Help from an Open Source Project

Other people have posted about this before, but it is a lesson that bears repeating. Getting help from an open source project is not like getting help from a commercial software company. In the latter case, one has exchanged money for software and so can expect a certain amount of assistance.

Open source is different. In most cases the majority of the people who work on a project are volunteers. Complaining about the timeliness of free support from such a community is like getting a free Mercedes and complaining about the color.

First off, anyone who deals with forums on-line, open source or otherwise, should read Eric Raymond’s seminal “How to Ask Questions“.

Second, here’s how to get someone like me to *not* help you.

On the OpenNMS discuss list, we’ve had a user post 53 e-mails since Thanksgiving. The last several have concerned monitoring MySQL with OpenNMS. There are a number of ways to monitor MySQL using the platform, and people have been trying to help him out.

Unfortunately, we get replies like this:


As most people know, posting in all caps is the equivalent of yelling. Yelling that you wanted a silver Mercedes when you were given a black one is rather rude and probably has a negligible effect on getting a different color. And don’t get me started on top posting.

But I guess the mailing list wasn’t good enough for this user. He decided to call our office.

Now, we are mainly located on the east coast of the US, so calling me numerous times, starting at 2:28am, is also not going to win you any friends.

But hey, there is always Facebook, right? Posting something like:

you all are looking good but i am still not satisfied with opeenms because opennms help is not good like microsoft. i want to get the license of it bus my initial requirment is to monitor mysql databases in it which is not yet complete

will get results, right? After all, companies like Microsoft are renowned for their high level of support, and I’m sure posting a comment on Microsoft’s Facebook page would cause hundreds of people to drop what they are doing to help you.

When all else fails, you can post a message on the OpenNMS Group contact page (which, of course, specifically mentions not to do this for support):

I am very dishard about the opennms help. I have submit my problem a lot of time but no proper solution were recived yet. why? My problem is i want to monitor mysql table spaces etc in opennms using jetty is it possible or not just tell me yes or no

So I replied: “Yes”.


Look, literally tens of millions of dollars *that I can document* have gone into making OpenNMS, and that doesn’t include the tens of millions of dollars worth of donated time and effort. Throwing an online hissy fit won’t get you help any faster.

And I hate the “well, if you just get OpenNMS running for me I’m sure I’ll buy a support contract later” line. It’s like going to the doctor and asking him to treat you for free on the off chance that if you feel better you might pay him. I can count the number of times someone has led with that line and actually bought a contract on each one of my rippling, six-pack abs.

When I first started out providing services for OpenNMS, I got a call from Motorola. They were considering OpenNMS, and they wanted me to come out and show it to them (i.e. fly to Texas). I pointed them to our “Getting to Know You” package where I would fly out and spend a couple of days showing them how it works on their network. They were aghast. How could I possibly ask them to pay for something like that? Even pointing out the fact that OpenNMS was free software and that once installed they could both own their solution totally and not have to pay license fees couldn’t get them past the fact that I was asking them to pay for “presales”.

Trust me, you don’t need customers like that. Customers that “get it” will have a competitive advantage. This will eventually allow them to provide better service to their customers (either through a better solution or cost savings put to other use) and thus distance themselves from their commercial software-using competitors.

While I doubt OpenNMS had anything do to with it, ask yourself what was the last model of Motorola mobile phone you owned? Did it come in a bag?

A Little Microsoft and VMWare Rant

I’m out at a customer site this week, and while the customer is awesome, a couple of things have made me very frustrated.

The first concerns Windows Management Instrumentation (WMI). OpenNMS now supports native WMI (thanks mainly to Matt Raykowski) and this is the first time I got to play with it. Works like a charm and how you would expect with OpenNMS – simply. I edited wmi-config.xml, put in a valid username and password, edited capsd-configuration.xml to discover WMI, and turned it on in collectd-configuration.xml. Restart, and now I’m collecting a ton of WMI stats out of the box.

So far, so good.

One of their concerns is monitoring Exchange 2007. So I think, great, I’ll just configure some WMI classes and objects dealing with Exchange, make some graphs, and we’re done.

Not so fast.

First, there doesn’t seem to be a good place to get a list of all the available WMI classes easily. I did find some rather thick Technet docs, but for the most part it is a lot of digging. It would be nice if there was a MIB-like document that described them.

Second, it turns out that Exchange 2007 doesn’t support WMI. You have to use Powershell “cmdlets” and script it from there.


Okay, so Microsoft decides that SNMP isn’t good enough to use for exchanging data between a manager and an agent, so they invent their own management protocol called WMI, and a few years later decide it isn’t worth supporting.


My second source of frustration deals with VMWare. The client currently uses ESX, so I’m like – hey, just go in, enable the Net-SNMP agent, enable the “dlmod” for the ESX MIB and we’re set.

That is all well and good, but they are migrating everything to ESXi which, wait for it, doesn’t support SNMP. Well, at least GETs.

From the VMWare documentation (PDF), you first get:

… hardware monitoring through SNMP continues to be supported by ESXi, and any third-party management application that supports SNMP can be used to monitor it. For example, Dell OpenManage IT Assistant (version 8.1 or later) has ESXi MIBs pre-compiled and integrated, allowing basic inventory of the server and making it possible to monitor hardware alerts such as a failed power supply. SNMP also lets you monitor aspects of the state of the VMkernel, such as resource usage, as well as the state of virtual machines.

Okay, good, but the next paragraph reads

ESXi ships with an SNMP management agent different from the one that runs in the service console of ESX 3. Currently, the ESXi SNMP agent supports only SNMP traps, not gets.

Again, what?

I mean, okay, traps are great, but how am I supposed to monitor “resource usage” if I can’t do a GET?

In both cases there does exist a non-standard, proprietary API that can be used to mine this data, and if the demand is high enough we’ll definitely put it into OpenNMS. Thank goodness the architecture is abstracted so that it is easy to add such plugins without having to re-write everything.

But, c’mon people, we have standards for a reason. Can’t we all just get along?