OpenNMS, F5 and Bogus DMCA Notices

I don’t know much about the network infrastructure company F5. I know some of our users have their gear, and I’ve heard positive things about it, but one place where F5 fails is in its legal department.

It was pointed out to me today that four pages hosted on the opennms.org website were named in a DMCA Copyright Complaint to Google. The complaint states:

These URL pages submitted contain questions taken without authorisation from their owners and holders (F5.com) which are Examination Questions from tests by which Trainees of F5 are able to become qualifed support technicians for F5 products. F5 itself writes on these Training questions:

F5 offers instructor-led courses that provide a hands-on learning environment, real-world problem-solving activities, and immediate constructive feedback. Our courses follow an aggressive schedule of accelerated lessons covering many of our application delivery networking products.

It further states:

SWORN STATEMENTS

I have a good faith belief that use of the copyrighted materials described above as allegedly infringing is not authorized by the copyright owner, its agent, or the law.

The information in this notification is accurate, and I swear, under penalty of perjury, that I am the copyright owner or am authorized to act on behalf of the owner of an exclusive right that is allegedly infringed.

This, to use a complicated legal term, is complete crap, at least with respect to the referenced links on our website, and it makes me angry that this kind of sloppy research is allowed to pass just because F5 has a lot more money than our little project (and apparently uses it to hire lazy legal help).

I don’t really have an opinion on the original complaint. It does appear that most of the links do point to web-based study sites which may have lifted questions from F5 tests. It is a long tradition to provide study questions for exams, but if they did actually source those questions from F5 copywritten material without permission I think F5 has a valid issue.

What I strongly disagree with is that our project is grouped into that list when it is quite obvious that none of those links references F5 material. Heck, one link is to an IRC log that only contains the characters “F5” as part of a hex string.

I’m not too worried about it. The notice was not aimed at us, and Google would be stupid to de-index those pages (at the moment it appears they haven’t). What worries me is this trend where large companies use vague laws like the DMCA coupled with lazy legal work to bully others.

UPDATE: It does appear that Google has de-indexed those pages. Luckily, they have an easy form you can fill out to file a counter-notice, which I have done. I got an e-mail verifying receipt, but with a notice that it may take some time, but I’ll keep updating this post with the status.

Lands' End, You Are Dead to Me.

[UPDATE: Lands’ End contacted me and went to great links to rectify the situation, so be sure to click through to the link if you decide to read this.]

I can’t stress how important computer systems are to today’s business world. Here is an example of a company who is doing it wrong and one who is doing it right.

For years I’ve used LandsEnd Business Outfitters for our OpenNMS polos. The shirts are high quality, the ordering process is simple and on the occasion that I do need help, a human picks up in three rings and solves my problem with that one call.

Then, they “upgraded their system”. Now, my orders don’t get processed, when I call I sit in queue for a half hour (thank goodness for speaker phones) and when I do get to speak to a human, they don’t have any visibility into the order process so they can’t tell me a valid status. Then they make stuff up, like “well, your order was placed a month ago so it should be shipping by the end of the week”. When the end of the week comes and goes and it doesn’t ship, I have to repeat the process.

To add insult to injury, they are offering people who place orders now free logos. The order I canceled this morning was for nearly US$3000, over US$500 of which was logo fees. If I restarted the process now it probably wouldn’t take a month and I’d save a ton of money, but if I had remained “loyal” it would cost me. Did they offer some sort of concession due to the delay? No, so I’m looking to switch companies (perhaps Brand Fuel – anyone have a suggestion?).

Contrast this with Amazon. Amazon consistently under promises and over delivers, which two day orders arriving in one, and offering one day Saturday delivery in some places.

Recently I had to place orders on amazon.co.uk and amazon.de. Not only did my US login work on those sites, all of my account information was also available. Heck, even their “Prime” shipping service worked for me in Germany. The order I placed on the German site was done totally in German, which I don’t speak, yet the process was so similar to what I experience on the US site that it took me the same amount of time to process the order.

Guess who gets to keep me as a customer?

The Centurylink Amateurs Are At It Again

Hey, #centurylink, if you want to play with the big boys, you are going to have to invest in qualified people. Not people who take out business networks for hours at a time.

I was looking forward to starting my Labor Day vacation a little early today, but that’s not going to happen. About 2am this morning Centurylink made some changes to their network (both my home DSL circuit and my father’s, who lives an hour away, got new DHCP address) and the network at the office went down completely.

When I called, the automated voice told me that they were aware of the outage and that it would be fixed by 4pm.

Now:

a) in what universe is a business class circuit allowed to have a 14 hour outage?

b) who does a major network change on a Friday?

c) who does a major network change on a Friday before a major holiday?

d) If you can’t plan an outage, including disaster recovery, to last less than an hour or two, get out of the business.

This was on top of a routing loop last week that almost cost me a major customer (since we had a demo planned and couldn’t reach the server we needed).

I waited a couple of hours and then decided I wanted to vent. So I called Centurylink back. I didn’t expect it to get anything fixed faster but I figured I’d at least get a “mea culpa” and a little “we screwed up – sorry”.

The monkey in first level who answered the call not only did not apologize, but seemed to act if 14 hour outages were normal. When I explained to him that my business depends on that circuit and in fact we’d flown a guy up from Atlanta in order to work today (but that couldn’t happen from the office, now could it) he dismissed my concerns with “well, it’s an area outage, not just your circuit”.

I then pointed out that my home DSL line was fine, and since I live 10 miles from the office it obviously wasn’t an “area outage”. His suggestion? We needed to upgrade to a T1.

A T1? Was is this, 1993?

When I pointed out that a T1 was only 1.544 Mbps he told me I was wrong, that it was much faster than the 10 Mbps I was getting now. I suggested that there might be some sort of technology one could run over a T1 that would result in higher realized speeds (i.e. DSL over a T1 versus a POTS line) and that must be what he was referring to, he continued to insist that a T1 was much faster.

(sigh)

I then asked to speak to a manager, and was told I couldn’t but that the monkey would be happy to relay any concerns I had.

I’m going to do it myself after the Time Warner guy gets back to me with my new office circuit. I recommend that anyone running a business that depends on Internet access stay as far as possible away from Centurylink, and that probably goes for voice service as well considering the level of customer support they find acceptable.

Another Story About Broken Healthcare

I had a wonderful weekend away on the North Carolina coast, but when I returned home I found that I had a letter from Blue Cross Blue Shield of North Carolina:

Dear Group Administrator:

According to our records, your group insurance premium has not been received. As a result, claims for services rendered after 02/29/2012, are now suspended.

This was followed by:

If you are planning to terminate your group coverage, please be advised that under North Carolina General Statute 58-50-40 employers are required to give employees 45 days prior notice of insurance cancellations. Violation of this law is a felony.

What a way to harsh the mood. First I’m informed that claims will be denied and then I’m threatened with a felony.

The part that pisses me off the most is that I pay all premiums on time. I pay my bills. On time. I have no idea what our business credit score is, but my personal score is over 800 at all three major reporting agencies. I take debts seriously.

This letter is a result of a bug in their software. It turns out that our annual renewal starts on 1 March. This is when our premiums increase, on average 30-35%, and for some reason this causes their billing system to invoice us late.

Usually, the insurance premium invoice arrives at the beginning of the prior month – i.e. I already have an invoice for April that’s due on the 22nd of March. But because of their billing system issues, my March bill did arrive until March 1st, with a due date of the 12th.

I paid it on the 8th, and according to my bank, BCBSNC cashed the check on the 12th – the same day they sent me a letter for non-payment.

What’s funny is that I have a collection of six of these letters. Yes, every single year they do this to me, and I’m not taking it anymore. In the spirit of Karen Sandler, I’d decided to take them to task for their crappy software.

So I called 877-237-6275, the number on the letter I received from “Sonya Walker, Director of Membership Operations” where I sat in queue for 20 minutes. I was then greeted by “Chris” who told me, oh, yeah, your account is paid in full, and that my payment and the letter must have passed in the mail.

I informed him that this wasn’t the case – I was never in arrears and that I was about to get very angry. If he wanted to, I suggested he transfer me to a supervisor, because I was getting ready to yell.

He decided to transfer me. Wise man.

After another 20 minutes of waiting I found myself talking to Misha Newman. I explained the situation again that I was tired of being told I was delinquent in my payments when it was untrue, and she replied that their software couldn’t handle renewals, which is why the letters get sent.

I asked for a bug report number.

She was a bit confused, so I told her that I’ve been getting these for six years now and that it needed to be fixed if they want to keep me as a customer. You can’t just send out threatening letters and then brush it off. If BCBSNC can’t manage the simplest accounting issue, doesn’t that cast serious doubt on their ability to manage things more complex things like medical claims?

I want someone in their IT department to at least document the issue and give me a way to know when it has been fixed.

I also demanded an apology from “Sonya Walker” if she even exists. It is quite common for large companies to make up names and positions for letters like the one I received. When someone calls to complain, it gives them a code to route the call. Ms. Newman informed me that Ms. Walker was no longer with their group, but that she did exist, and she would be happy to get her director to send me a letter. That should be interesting.

The sad part is that I doubt the alternatives to BCBSNC are any better (suggestions welcome). I think it is incredibly callous of them to value my time so poorly. On just this call alone I spent nearly an hour. You might say that I brought it on myself, which is partially true, but I also have over a dozen people that rely on me to make those insurance payments and I can’t be sure that there wasn’t some other mistake – so I’m forced to make that call.

I can’t help but think if health care was more transparent and used more open source ideas, if not software, that problems like this would be less common and easier to fix.

MonitoringForge, RIP

The following post contains language that some may find offensive. If that describes you, please skip it. I don’t aim to be crude for the sake of being crude, but sometimes certain words can’t be replaced.

 
 
 
 
 
 
 
 

At times, I am an asshole. I know this. Being human and full of faults I sometimes let my emotions get the better of me. But sometimes I believe I am unfairly portrayed as an asshole. It’s just that my bullshit meter pegs a little quicker than most.

Back in 2006, the Open Management Consortium was announced to a lot of fanfare. We were asked to join at the 11th hour, but something didn’t seem quite right to me. While the idea was strong, in order to get a bunch of separate and often competing companies to play nice together involves a lot more than a mission statement. It requires a tremendous amount of work and time.

Needless to say, the Consortium didn’t get much traction outside of a lot of press, and even a reboot in 2008 didn’t help. The domain name is now parked (and possibly up for sale it seems).

Jump forward to 2009 and the formation of MonitoringForge. A similar organization as the Consortium, I was just as wary about it and I said so. This got me labeled as an asshole, as in “why can’t you just be nice for once and stop being so negative.” It’s just that it seemed to be to be nothing more than a marketing ploy, and I felt the need to tell people about it.

As I was looking for my Sourceforge password today (I had to change it earlier this year due to a security issue and couldn’t remember what I had changed it to), I came across an exchange I’d had with Ross Turk about MonitoringForge and decided to see whatever happened to it. The website just says “The Monitoringforge Service was stopped as of June 30.” and lists a contact e-mail.

Now I’m not sure if it was this June 30th or last year’s June 30th (their twitter feed stopped in May of last year), but I think it is funny that the site was launched with such hoopla but died with nary a whimper.

The natural asshole part of me brings this up with a touch of schadenfreude. But the reason I’m writing about it has a direct bearing on open source and running an open source business.

When you are just starting out, the temptation is to do anything, simply anything, to get noticed. If you are a VC-backed company, marketing and perceived value is more than half of the business plan. But in the long run, your customers and your potential customers will put more value into your words and actions than your press releases.

I work hard not to involve OpenNMS or lend my name to any endeavor in which I am not 100% confident. It’s hard enough to get a large company to trust in a small company without having a string of failures to cast doubt. When I tell our clients that I plan for OpenNMS to be around in 10, 20, or even 50 years I mean it, and while they may not fully believe it, at least I haven’t given them any reason to worry by trading that trust for some short-lived notoriety.

Another thing I do that earns me the “asshole” or “zealot” title is rail against the improper use of the term “open source” when it comes to business models. Matt Aslett wrote recently about a number of companies dropping the term “open source” from their marketing, and he listed several of them. He wrote “The list below represents a small and unscientific sample, but these are among the highest profile open source-related vendors” which included companies like Zenoss and Groundwork.

We didn’t make the cut. Perhaps if we had joined in the Open Management Consortium (like Zenoss) or MonitoringForge (like Groundwork, who built it) we would have been “high profile” enough to make his list. But since the article was basically about a retreat from open source I can see at least one reason why we were omitted. It could also be why Zenoss is facing a user revolt and possible fork and we, thank goodness, are not.

I present this as a cautionary tale. In business, focus above all else on your customers. Be completely truthful and never get involved in anything you do not believe in 100%. Remember the sage Tony Montana who said “All I have in this world is my balls and my word and I don’t break them for no one.”

How Not to Get Help from an Open Source Project

Other people have posted about this before, but it is a lesson that bears repeating. Getting help from an open source project is not like getting help from a commercial software company. In the latter case, one has exchanged money for software and so can expect a certain amount of assistance.

Open source is different. In most cases the majority of the people who work on a project are volunteers. Complaining about the timeliness of free support from such a community is like getting a free Mercedes and complaining about the color.

First off, anyone who deals with forums on-line, open source or otherwise, should read Eric Raymond’s seminal “How to Ask Questions“.

Second, here’s how to get someone like me to *not* help you.

On the OpenNMS discuss list, we’ve had a user post 53 e-mails since Thanksgiving. The last several have concerned monitoring MySQL with OpenNMS. There are a number of ways to monitor MySQL using the platform, and people have been trying to help him out.

Unfortunately, we get replies like this:

STILL I AM UNABLE TO SEE THE DATABASES GRAPH. I HAVE EXACTLY FOLLOW YOUR DOCUMENTS AS IT IS I DONT KNOW WHERE IS THE PROBLEM COULD U ATTACHED THE /ETC FILES TO ME AND ALSO THE GRAPH PICTURE WHICH U ARE GETTING RIGHT NOW.

As most people know, posting in all caps is the equivalent of yelling. Yelling that you wanted a silver Mercedes when you were given a black one is rather rude and probably has a negligible effect on getting a different color. And don’t get me started on top posting.

But I guess the mailing list wasn’t good enough for this user. He decided to call our office.

Now, we are mainly located on the east coast of the US, so calling me numerous times, starting at 2:28am, is also not going to win you any friends.

But hey, there is always Facebook, right? Posting something like:

you all are looking good but i am still not satisfied with opeenms because opennms help is not good like microsoft. i want to get the license of it bus my initial requirment is to monitor mysql databases in it which is not yet complete

will get results, right? After all, companies like Microsoft are renowned for their high level of support, and I’m sure posting a comment on Microsoft’s Facebook page would cause hundreds of people to drop what they are doing to help you.

When all else fails, you can post a message on the OpenNMS Group contact page (which, of course, specifically mentions not to do this for support):

I am very dishard about the opennms help. I have submit my problem a lot of time but no proper solution were recived yet. why? My problem is i want to monitor mysql table spaces etc in opennms using jetty is it possible or not just tell me yes or no

So I replied: “Yes”.

(sigh)

Look, literally tens of millions of dollars *that I can document* have gone into making OpenNMS, and that doesn’t include the tens of millions of dollars worth of donated time and effort. Throwing an online hissy fit won’t get you help any faster.

And I hate the “well, if you just get OpenNMS running for me I’m sure I’ll buy a support contract later” line. It’s like going to the doctor and asking him to treat you for free on the off chance that if you feel better you might pay him. I can count the number of times someone has led with that line and actually bought a contract on each one of my rippling, six-pack abs.

When I first started out providing services for OpenNMS, I got a call from Motorola. They were considering OpenNMS, and they wanted me to come out and show it to them (i.e. fly to Texas). I pointed them to our “Getting to Know You” package where I would fly out and spend a couple of days showing them how it works on their network. They were aghast. How could I possibly ask them to pay for something like that? Even pointing out the fact that OpenNMS was free software and that once installed they could both own their solution totally and not have to pay license fees couldn’t get them past the fact that I was asking them to pay for “presales”.

Trust me, you don’t need customers like that. Customers that “get it” will have a competitive advantage. This will eventually allow them to provide better service to their customers (either through a better solution or cost savings put to other use) and thus distance themselves from their commercial software-using competitors.

While I doubt OpenNMS had anything do to with it, ask yourself what was the last model of Motorola mobile phone you owned? Did it come in a bag?

A Little Microsoft and VMWare Rant

I’m out at a customer site this week, and while the customer is awesome, a couple of things have made me very frustrated.

The first concerns Windows Management Instrumentation (WMI). OpenNMS now supports native WMI (thanks mainly to Matt Raykowski) and this is the first time I got to play with it. Works like a charm and how you would expect with OpenNMS – simply. I edited wmi-config.xml, put in a valid username and password, edited capsd-configuration.xml to discover WMI, and turned it on in collectd-configuration.xml. Restart, and now I’m collecting a ton of WMI stats out of the box.

So far, so good.

One of their concerns is monitoring Exchange 2007. So I think, great, I’ll just configure some WMI classes and objects dealing with Exchange, make some graphs, and we’re done.

Not so fast.

First, there doesn’t seem to be a good place to get a list of all the available WMI classes easily. I did find some rather thick Technet docs, but for the most part it is a lot of digging. It would be nice if there was a MIB-like document that described them.

Second, it turns out that Exchange 2007 doesn’t support WMI. You have to use Powershell “cmdlets” and script it from there.

What?

Okay, so Microsoft decides that SNMP isn’t good enough to use for exchanging data between a manager and an agent, so they invent their own management protocol called WMI, and a few years later decide it isn’t worth supporting.

(sigh)

My second source of frustration deals with VMWare. The client currently uses ESX, so I’m like – hey, just go in, enable the Net-SNMP agent, enable the “dlmod” for the ESX MIB and we’re set.

That is all well and good, but they are migrating everything to ESXi which, wait for it, doesn’t support SNMP. Well, at least GETs.

From the VMWare documentation (PDF), you first get:

… hardware monitoring through SNMP continues to be supported by ESXi, and any third-party management application that supports SNMP can be used to monitor it. For example, Dell OpenManage IT Assistant (version 8.1 or later) has ESXi MIBs pre-compiled and integrated, allowing basic inventory of the server and making it possible to monitor hardware alerts such as a failed power supply. SNMP also lets you monitor aspects of the state of the VMkernel, such as resource usage, as well as the state of virtual machines.

Okay, good, but the next paragraph reads

ESXi ships with an SNMP management agent different from the one that runs in the service console of ESX 3. Currently, the ESXi SNMP agent supports only SNMP traps, not gets.

Again, what?

I mean, okay, traps are great, but how am I supposed to monitor “resource usage” if I can’t do a GET?

In both cases there does exist a non-standard, proprietary API that can be used to mine this data, and if the demand is high enough we’ll definitely put it into OpenNMS. Thank goodness the architecture is abstracted so that it is easy to add such plugins without having to re-write everything.

But, c’mon people, we have standards for a reason. Can’t we all just get along?

Here We Go Again

I guess I should be flattered, but it appears that not one but two companies have decided to build commercial products on OpenNMS. Both of them were brought to my attention by members of our community.

The first product is called “RuggedNMS” by a company out of Canada called RuggedCom.

If you look at the screenshots, it is quite obviously OpenNMS with a slightly different skin. If you zoom in on their “Extensive Reports” screenshot you can see “uei.opennms.org” in the unique event identifiers.

How they can sell this as a commercial product without violating the GPL is a mystery, especially when you read their terms which state:

RuggedCom provides a trial copy RuggedNMS™ solely for the direct use of the person who is identified in the trial software request form. This is only a 30-day trial.

Redistribution of RuggedNMS™ software files by any means is prohibited.

The second case is a product called OpenGate by Encodex TeleSystems. They at least mention OpenNMS which is a good sign, but I am also confused by their use of the phrase “Encodex TeleSystems developed a proprietary Network Management product using OpenNMS platform and framework”. While technically possible, I am at a loss to understand how OpenNMS can easily be used as the basis for a proprietary product under the GPL.

While it is unfortunate, we do have experience in dealing with this in the past. One major difference over last time is that all of the OpenNMS copyright is now held by one entity, which makes enforcement of the license much, much easier.

I’ve contacted our team at Moglen Ravicher and asked them to look into this. My hope is that it can easily be resolved. We chose to make OpenNMS open source for a reason, and I have to wonder if it is too much to ask for others to respect that.

UPDATE: Okay, now I’m starting to wax sorely pissed. A friend of mine pointed me to this link on the RuggedCom site about discovery. Now compare that to the How-To I wrote years ago. Seem familiar?

UPDATE 2: Got a reply about OpenGate

Hi Tarus,

Thanks for your email. OpenNMS base product that you see on the website will be deleted, as this was never a product and was not built.
This was a conceptual design activity that never progressed.

We will remove it from the website right away.

Thanks

Arun Joshi, CEO
Encodex Telesystems

Fun With Billing

I bought my iPhone back in November, which was the last month I would have a Sprint bill.

So I thought.

Since December, every month like clockwork I get a multipage statement in the mail letting me know I have a credit for $1.24.

If we assume it costs, conservatively, $0.50 to send that bill, they’ve already spent almost twice what they owe me to let me know about the credit.

In other billing news, our newest insurance provider, Guardian, has informed my insurance broker that we haven’t paid our last two bills. As someone who is incredibly anal-retentive about paying bills I can explain: we haven’t gotten any.

And, in order to sign up to pay your bill online, you have to have information that is only sent with the paper bill, which I haven’t received.

(sigh)

Is it any wonder that the adoption of paperless billing has stalled? If we can’t trust our suppliers to deal with paper bills correctly, why would we trust them to do it electronically? With the huge penalties for missing, say, a credit card payment by even by one day, entering into an agreement to only get an electronic reminder can be a bit scary.

I wish they would adopt the process my eye doctor uses to verify appointments. Two days before the appointment they send out an e-mail. In the e-mail is a link to confirm or reschedule the appointment. If you don’t click on it, they call you the next day.

If Citibank or Chase would implement a system where they would send a paperless statement (or let you know that it is ready) and if you didn’t verify receipt within a certain period of time they’d send out the paper statement, I’d sign up in a heartbeat. Technologically it wouldn’t be hard to implement.

Can you tell I’m a little overwhelmed with paperwork lately? As OpenNMS has grown I’ve had to spend more and more time with administrative tasks than playing with software. I’m not sure I like it.

At least I got to help a client figure out a notification issue that was giving him trouble yesterday, and a lot of it is dealing with new business, which is always great, so I guess I shouldn’t complain.

MonitoringForge Redux

A few months ago I blogged about a new site called “MonitoringForge.org“. It seemed to me to be a thinly veiled marketing attempt with little value, but I was willing to give it the benefit of the doubt and time would tell.

Well, I was reading CotĂ©’s blog today and read a link where they created a press release to trumpet their 2,000th registered member.

This struck me as funny because, in the same sentence, they state that there are “more than 2,000 projects” registered on the site.

Wah?

So, if only one unique member of each project on their site registered, there should be more than 2,000 of them, yet they have less than that, and this is considered news? Heck, we have nearly 1300 people on the opennms discussion list and we’re just one project, but with their site running at an average of less than one person per project I guess we’re doing pretty well. And while I’m sure that 80 or so of our subscribers are directly working on OpenNMS, that still leaves about 1200 end users.

I’ll leave the similar calculation for MonitoringForge as an exercise for the reader.

(sigh)

Now I’m not one to beat a dead horse, but when the “Chief Marketing Officer” is willing to issue a press release on a site she calls “the epicenter of all open source projects that relate to IT monitoring” with such, in my humble opinion, lame numbers, I’m willing to stand by my original impression that this is just a marketing ploy.

Am I wrong? Can anyone comment who found the site valuable? Inquiring minds want to know.