Review: The Slow Regard of Silent Things by Patrick Rothfuss

I have yet to decide whether or not Patrick Rothfuss is an asshole.

I know that sounds like a mean thing to say, but I have my reasons which I’ll get to soon.

I was introduced to Rothfuss through his first novel, The Name of the Wind. Since that can get tedious to type, allow me to abbreviate it TNotW.

TNotW is the first book in a trilogy. It concerns a near mythical figure named Kvothe, and it is one of the best novels ever written in the fantasy genre, or any genre for that matter.

At least one of my three readers is asking themselves why I would write about fantasy literature on an open source blog. One reason is that open source tends to be a geeky thing and so is reading fantasy, but the other thing is that it helps me think about the future. As the third law of Arthur C. Clarke states “Any sufficiently advanced technology is indistinguishable from magic,” in my mind anyone who wants to create advanced technology must first start with “I want to do magic. How should I begin?” Reading fantasy allows me to think in different ways that I find both enjoyable and useful.

Anyway, in TNotW Rothfuss introduces his magic system. In the best fantasy there are rules that both empower and limit the characters, and I really like his. Called “sympathy”, magic in his world requires three things: a link, a source of energy and strength of will.

For example, suppose you wanted to move an iron skillet off of a fire through magic. First you would need some way to link what you wanted to move with something you could easily manipulate. In this case the best thing would be a small piece of the skillet itself. That would form a very strong link. Barring that, you could use any piece of iron, but that link would be weaker. Weaker still would be a non-ferrous metal, etc.

So let’s assume you have a small chip of the skillet in your hand. You would then need a source of energy. The bigger the magic, the more energy you need (i.e. rules). In this case you could probably use the heat from the fire itself.

Finally, you would need strength of will to connect your piece of the skillet to the whole thing. This is the hard part, as you basically have to imagine, with the full weight of reality, that the small part of the skillet you hold is the skillet itself, so when you move your piece, the skillet will move.

I’m oversimplifying but you get the gist. In TNotW Kovthe starts to learn about sympathy and is admitted to The University, a place where its principals are studied and taught. Out of them comes a form of engineering, a form of medicine, a form of chemistry, etc.

However, in addition to this sympathetic magic, there is a more primal, raw form of magic based on names. It is a common theme in fantasy that by knowing a thing’s “true name” one can control it. Names are powerful, which is why I obsess over them more than most people. In the world that Rothfuss creates, the pursuit of “Naming” is magic in its truest sense, but it is also the most dangerous. One of my favorite characters in his stories is Master Elodin, the Master Namer, who is quite bent.

In any case, Rothfuss is the rare author who inspires a certain type of rabid fandom. Anything he posts on his blog is almost always met by a chorus of fawning comments. It’s not that he isn’t talented, quite the contrary, but this type of fandom ends up rubber stamping everything he does as “great”.

For example, the second book in the series, The Wise Man’s Fear, did not resonate with me like the first. I came close to actually disliking it upon a first reading.

Now, granted, once I set it on the shelf for awhile and then took it down and re-read it, I liked it more, but still, it didn’t affect me like the first book. I look on it like Tolkien’s The Two Towers where “things happen that must happen” but it acts like a bridge between the first and last books of a trilogy. I eagerly await the third book, tentatively titled Doors of Stone to see if he can pull off the magic of TNotW.

And this is where the asshole part comes in. I have some friends who have met Rothfuss and spent some time with him and some of their comments tip the meter toward “asshole”. Some of the stuff that he writes on his blog rub me the wrong way, thus re-enforcing the thought. But I guess I am mainly upset because I just want him to work on that third book instead of all the other stuff he does. This is very selfish of me, because some of the stuff he does is very worthwhile and makes the world a better place, but at this point I am emotionally invested in the story of Kvothe and I want to know how it ends.

Which brings me to a sobering point: I know almost nothing about Patrick Rothfuss. One of the fallacies of the Internet is this illusion of intimacy. The thought that I can read a blog or a twitter feed or an interview and think that really gives me insight into who the person is is ludicrous. To paraphrase Silent Bob, what I don’t know about Patrick Rothfuss could just about squeeze into the Grand Canyon.

But I do know one thing without a doubt: he loves words.

I like words. I like my ten cent words and my five dollar words. But to me they are a means to an end. I like how a certain word can convey just the right feeling or evoke a particular response. But I don’t love words.

Rothfuss loves words almost as much as his family (which, if you read his blog, he loves a lot). He dotes on them. He caresses them. And I’m almost certain that he stays up nights obsessing over finding the right word.

Which brings me to his latest book, The Slow Regard of Silent Things.

This is a tiny book, around 150 pages. It’s even shorter than Gaiman’s The Ocean at the End of the Lane. He warns on both his blog and in the forward that many people won’t like this book. Heck, it only has one character in it, Auri, the mysterious girl befriended by Kvothe in TNotW.

I loved it.

This is a love song to words. He uses them to paint pictures and to compose symphonies. It is a three ringed circus of adjectives and adverbs, nouns and verbs all coalescing to create an experience if not exactly a narrative.

Auri is a woman of unknown age. She is very small, about the size of a child. She lives in a complex of tunnels and forgotten rooms called “The Underthing” that exists beneath the grounds of The University. While Rothfuss has never told us straight up her history, I’ve always imagined that she was a great student at The University who studied Naming and went crazy. She decided to “make herself small” and hid herself away. It is one of the characteristics of Kvothe that he was able to befriend her. He even gave her the name “Auri” which inspired Master Elodin to instruct Kvothe in Naming. But don’t expect to see those characters in The Slow Regard of Silent Things. It is all about Auri and can stand alone from the rest of the series.

If you haven’t read any of his books, then you won’t know what I’m talking about. Heck, I’m not even sure I know what I’m talking about. All I know is that I feel like a better person from having read it.

It covers several days in the life of Auri. And that’s about it. Pretty easy not to spoil. She has good days and bad days but to her they are just “days”. The narrative focuses a lot on her drive to put things in their proper places and in some cases, give them names.

One of the world philosophies that I strongly identify with is Taoism. Now I’m certain that a true scholar of the Tao will be horrified, if that is possible, over how I’m about to describe it, so my apologies in advance.

The Tao is all things and how they are connected. There is no “good” or “evil”, there is just the natural cycle of things. When one lives in tune with the Tao, this we call happiness. When one struggles against the Tao, sadness ensues. It stresses a very low impact existence and an acceptance of the way things are, but still manages to get a lot of stuff done, which sounds a little like an oxymoron.

One of the best books on the subject is The Tao of Pooh by Benjamin Hoff. In it he demonstrates the principals of Taoism through the stories about Winnie the Pooh. It works, and it is one of my favorite books. It sits next to me at my desk in case I’m having a rough day and I need a reminder.

As I was reading The Slow Regard of Silent Things I couldn’t help but think that Auri was a Taoist master. Here is a quote from toward the end of the book:

That meant you could move smoothly through the world without upsetting every applecart you came across. And if you were careful, if you were the proper part of things, then you could help. You mended what was cracked. You tended to the things you found askew. And you trusted that the world in turn would brush you up against the chance to eat. It was the only graceful way to move. All else was vanity and pride.

Seems very Taoist to me.

I once saw Kurt Vonnegut give a lecture. He was talking about “the shape of stories” and the normal Western narrative and how it has these huge swings in mood. The example he used was Cinderella. It starts off pretty bad. Her parents have died and she’s living with her evil stepmother. Then it gets really good. She gets to go to the ball and she meets the Prince. Then the clock hits midnight and things are bad again. Then the Prince finds her and all is well. He drew this on a white board in the form of a big sine wave that swung from bad to good.

He compared that to Native American stories. Usually the mood is very flat. We walked in the woods. We saw a deer. We caught some fish. We ate. We went to sleep. That sort of thing. There really isn’t this whole process that we expect from our stories. On his white board he drew a straight line, pretty much neutral between good and bad.

Then he examined Hamlet. Hamlet is not a happy story. Things start of bad and remain that way. As Vonnegut talked through the plot he drew another straight line. Granted, this was firmly on the “bad” side of the chart but it had a lot more in common with a Native American narrative than a traditional one, and Hamlet is one of the greatest stories ever told.

Heh – I just decided to take a stab at the premise that “everything is on the Internet” and I found a page talking about this very thing.

I don’t think that The Slow Regard of Silent Things is one of the greatest stories ever told, but it is a very good one. It, too, has a flat narrative arc. I will reread it a number of times. While I think a lot of his fans will be put off by it, and he knows this, but the fact that he created it and felt strong enough to see it through to publication moves the needle, at least for me, back firmly into the “not asshole” side of the meter.

OpenNMS Newts at ApacheCon Europe

Being Hungarian, I am very jealous and yet still proud that our very own Eric Evans will be presenting at ApacheCon Europe in Budapest, Hungary.

He will be talking about Newts which is a new time series data store built on top of Apache Cassandra. It will be a key part of positioning OpenNMS for the Internet of Things as well as being very useful on its own.

Eric is a dynamic and interesting speaker, so if you are attending the conference be sure to check out his talk.

And while you are there, eat a Túró Rudi or three for me.


Note: This is a travel/food blog post with no OpenNMS content. As usual, feel free to skip.

I consider myself insanely wealthy. Not in terms of money, most people could always use a little more, but because I’ve managed to meet some truly amazing people in my life, and even better, there still seems to be a nice supply of them. My life has been greatly enriched because of it.

One such person is Geoff Davis, who I met in high school (he was a year younger but in all my advanced courses) who now works at Google. The last time I was out there he and his bride took me to dinner in the Mission, and while we were talking about food he mentioned the Alinea.

The Alinea restaurant is located in Chicago, and it is one of only thirteen (when I booked it was ten) Michelin three star restaurants in the country. I happened to be in Chicago this week, and as this trip was planned months ago I was able to book a table for six. The experience was pretty amazing.

The name comes from an archaic mark similar to the “paragraph” symbol: ¶. The alinea looks more like an inverted “D” with two vertical marks, and in the name of the restaurant it represents “a new beginning”.

The booking process was unique as well. I like to cook, and occasionally (very occasionally) after one of my better efforts, someone will say “you should open a restaurant”.

Hell no. Owning and running a restaurant is incredibly hard work. People who aren’t into food (and some who are) just don’t realize it. Part of the problem is that people have few qualms about either canceling a reservation at the last minute or not showing up at all.

The Alinea addresses this by selling tickets instead of taking reservations. I’m not sure it would work for every restaurant, but when you have a place with a) high demand and b) a prix fixe menu it seems to work well. There is a long discussion of the history of the ticketing system available so I won’t repeat it here, but it did resonate with my innate fondness for unusual business models.

The restaurant itself is very non-descript. Most of our party met at Boka next door for drinks (an excellent restaurant on its own right with its own Michelin star). I probably wouldn’t have known where to enter the Alinea if I wasn’t shown the way.

You enter in via a dark corridor highlighted by red lights, and it seems to narrow a bit. When you reach the end you are greeted by what I would come to recognize as the level of service that helps the Alinea keeps its stars. For our group of six there were at least three people there to take our coats. One pair in our party had arrived with their car, so the staff took the valet parking slip to make sure the car would be ready when it was time to leave. We were served by no less than twenty people during the evening but it was perfect – they were there when you needed them and quickly gone when you didn’t.

We were shown upstairs to our table. The restaurant only accepts parties of two, four or six. Party of three? You buy an extra ticket, and since they are quite spendy my guess is you’d find an excuse to invite Aunt Edna out for the evening.

Just a quick note: I wanted to immerse myself in the experience so I took no photographs or notes. I might get some of the details wrong so I will apologize in advance since my memory isn’t what it used to be.

Our host Miguel, I would not call him a server even though he did serve us food, welcomed us and explained how the evening would play out. We were to have 15, maybe 16 courses and there was an optional wine pairing. He stressed that the wine servings were modest and that it would end up being about four glasses of wine total. The “maybe” 16th course was a special: white truffle risotto. When I visit my friend Antonio in Italy (another amazing person who has enriched my life) I always tease him about this very expensive food. When he asks what I want to eat I say “tartufo bianco”. I love truffles but as Cat, one of our dinner companions, pointed out, having such an extravagant course here would be like gilding the lily. Plus, the $150/person cost put it out of my league – I had to cash in my Christmas and Birthday presents to swing the dinner alone.

I got excited when they prepared to bring out the first course. They placed little white pillows on the table in front of each of us and on it was a caviar spoon (I love fish eggs of any type). Miguel explained that the silverware we’d need for each course would be placed on the pillow and that we were to leave the used items with the serving dish in order to keep the pillow clean for the next course.

Speaking again of the crazy level of service, if one of us got up to use the restroom, one of the staff would immediately remove the napkin and replace it with a fresh one. Cat explained that napkins are meant to be used, and no one should have to come back to a soiled one. Crazy, but cool.

The first course was Osetra caviar served with a variety of other flavors. I was not going to do the wine pairing, but I agreed to start off with a glass of champagne (Pierre Moncuit Blanc des Blancs Brut NV) as did Ron (what else do you pair with caviar?). Demetri and Cat did get the pairing, while my lovely bride Andrea ordered a glass of Riesling and Colleen stuck with water.

The caviar was delightful.

The second course arrived in the form of two large “tumbleweeds” made up of intertwined brown vines. Each was about twice the size of a basketball. We were told we had to find the second course hidden within the branches. In each ball was hidden a piece of salsify, a root vegetable that had been cooked sous vide and heavily seasoned. It matched the color of the vine exactly but had a much softer texture. It was tasty, although this is one of the few places I’ve eaten where you had to ask yourself “is this bit the food?”.

The third course was skate, cooked sous vide and seasoned with brown butter, lemon and herbs. The paired wine was an A.J. Adam “Dhroner” Riesling (Mosel 2012). I made a joke about “Dhroner kebabs” that fell flat. Demetri had me taste some of his and I liked it so much I changed my mind about skipping the wine pairing. It was the most tender skate I’ve ever eaten, and it makes me want to explore sous vide cooking even more.

Course number four was ebi, or sweet shrimp for those of you who like sushi. It was paired with Muscadet Sevre et Maine “Granite” (Domaine de l’Ecu 2010) which I found to be too mineral. I remember liking this dish but I don’t remember too much of the details, since it was eclipsed by …

Corn. The fifth course arrived served in roasted corn husks, causing me to remark “look, the world’s most expensive tamale”. It was far from a traditional tamale. The corn was whole, roasted and served over a bed of creamy grits. The whole thing was seasoned with truffle, manchego cheese and sherry. Think about how much corn you have eaten in your lifetime – this redefined it for me. We were told that this was the next to last night it would be served as the seasons were changing. When asked how they were able to get the corn off the cob yet have it hold its shape, the secret is that the whole cob is dipped in clarified butter so that the kernels can be sliced off but still stick together.

The sixth course arrived on pieces of wood that used to be barrel staves. It was trout in white pepper, coriander and broccoli. Outside of eating off a stick (we asked and were told the staves were cleaned in the dishwasher and then burnt with a torch) in the middle were pieces of bone – a small bit of spine and the plate near the cheek of the fish. We were told that, indeed, this was food. It was surprisingly tasty, although I think I cut my tongue a little on the spiny bit. This was paired with a Chablis (Laurent Tribut 2011) that I quite liked.

Flowers made up the seventh course, mainly lilies, with rambutan and a caviar lime sauce.

Now at the halfway point, the staff brought out two stone plates topped with pieces of wood. They set these on fire, and the smoke and light added a nice ambiance.

Course number eight featured matsutake mushrooms, which I adore. It was served with a number of sweet flavors, including a huckleberry foam and tapioca. In keeping with the Japanese association with this food, the pairing was sake, Tensei “Song of the Sea” Junmai Ginjo-shu (Kumazawa Shuzo, Kanagawa-ken). I loved the food but just liked the sake (I’m pretty particular about sake).

The ninth course was a bit of a surprise. The wood/fire centerpieces, which had now gone out, were dissected and out of them came our food. It was charred pork belly served with compressed parsnip. Yummy. It was served with our first red of the evening, Donkey and Goat “Testa Vineyard” Carignane (Mendocino 2013), which was very nice.

Dish number ten was also unusual. It was a warm butter-based broth served in a small, concave bowl made of wax (it reminded me of nothing less than a huge contact lens). Suspended above it on a steel pin was a piece of potato and truffle. You were told to pull the pin, causing the items on it to fall into the broth and then eat the whole thing in a shot. Tasty, but I spilled a little of mine. I believe it was on the table for about 450ms before someone had wiped it up.

The last mainly savory dish of the night was squab. Served on a piece of dark wood that looked like the top of a tree trunk, the roasted squab was served with carrot, potato, beet and an orange sauce. There was also a piece of squab liver as well. The pairing was another red, Valpolicella Classico Superiore (Tomasso Bussola 2009) which complemented the meat well.

The twelfth dish started the transition to sweet. Called “graffiti” it was a hazelnut candy, like a praline, served with balsamic vinegar on a matching piece of gray stone. This was the only misstep by the staff. Ron doesn’t like balsamic vinegar but they missed his request not to spray it on. I like balsamic but this dish was only “good” for me. The pairing, however, was a Vergano Chinato Moscato “Luli”. Lately I’ve become enamoured of sweet dessert wines like Sauternes and this Moscato was delicious. I’ll seek it out when I get home.

Probably the oddest dish (in an evening of odd dishes) was dish number thirteen. Featuring blueberries, part of the dish was a foam made from Bubble Yum™ chewing gum. I double-checked to make sure we could eat it. Believe it or not, it really worked. The pairing was another Muscat wine, Jorge Ordonez “Victoria” (Muscat, Malaga 2013) which I liked considerably less than the Vergano, but it was still good.

If you read about the Alinea, you’ll often see pictures of transparent white balloons. That was our fourteenth course of the evening. Made with a green apple candy, the balloons are served filled with helium with a taffy string. The way we would told to eat this was to try and suck the helium out of the balloon and then talk. Yes – fun with food. It is this kind of whimsy that made the evening worthwhile, and the fact that in the end you could eat pretty much everything was also a plus.

The final course was the most elaborate. We all moved to one side of the table which was then covered in a large rubber sheet. The head chef himself, Grant Achatz, came out and using about 15-20 small dishes painted our final course directly on the table. It was a combination of tropical fruit flavors. There was mango, passion fruit, compressed melon, a coconut meringue flash frozen in liquid nitrogen, a kaffir lime “candy” with edible wrapper, roasted banana, pineapple – and I forget what else. The chef was professional if not extremely outgoing. I was so busy watching him and his associate work (the table size required two people) that he was there and gone before I knew it.

The final wine pairing was a Passito di Pantelleria (Fernandes 2006) which I honestly can’t remember. It was at this time I first looked at my watch and realized that four hours had passed.

Leaving the restaurant was as efficient as our arrival. Not only were our coats and bags waiting for us, so was Ron’s car and two taxis for the rest of the group.

If you are into food and can make it to Chicago I can highly recommend the Alinea. Better yet, if you have some amazing friends like I have in Ron, Colleen, Demetri and Cat (plus my BFF Andrea) it will be an experience to remember. Yes, it is pricey, but one of the keys to happiness is to buy experiences and not things.

In that realm I am a very rich and happy man.

Internet of Silos

My friend Phil sent me a link to a Gartner article listing the “Top 10 Strategic Technology Trends for 2015” and I wasn’t surprised to see the “Internet of Things” at number two.

Now that mobile computing has become ubiquitous, there is a rush to network connect everything. Plus, there is a lot of money in it – take a look at the Nest acquisition. I’m not really certain this is a Good Thing™. My experience with low end network devices shows that corners that can be cut often are (i.e. crappy SNMP agents) and my guess is that in this rush to get things out the door we will end up with some serious issues.

One thing that can circumvent this is open source. By making the code transparent, especially for medical devices, there is a strong chance that major issues could be identified and corrected.

If, and that is a big “if”, we could get people to standardize around various open source software for the IoT there is a chance we can prevent the “Internet of Silos”. I first heard that term in a BBC article where it was announced that ARM (a British company) was creating an open source programming language for devices powered by its chips.

As you might expect, the article was short on specifics, but if the language is any good and the license is permissive, perhaps other chip manufacturers will port it. At a minimum it should encourage companies like Intel to also open source the technology used for their chips and the community might be able to build wrappers around both of them.

No matter what happens, we hope to support any management capabilities they introduce. We are actively working on making OpenNMS almost infinitely scalable to be able to handle the needs of the IoT, from insanely fast data storage (Newts) to highly distributed polling and data collection (Minion). We hope that the open nature of the platform will encourage more and more product vendors to use OpenNMS for their element management system, and then at least on the management front we can prevent the silos.

Conferences? We Got Yer Conferences

This is my “oh” face

as in “Oh how awesome is All Things Open”.

Last year was the first edition of the All Things Open conference in Raleigh, NC. I was very happy that we could be a sponsor, and the above picture was taken during my talk.

We ended up with about 50% more people than were expected, and the keynotes were standing room only. It was really cool to see such a turnout, especially since it sort of validates the Raleigh area as a center for open source excellence.

This year we will have a booth where you can come by, get some OpenNMS swag, and hear about the pending release of OpenNMS 14 (yes, fourteen) which is only a few days away.

Now, “oh” could also mean “oh-hi-oh” as in the Ohio LinuxFest. Directly after All Things Open, the Ohio LinuxFest will be held in downtown Columbus this weekend. This has been one of my favorite open source conferences, and it looks like this year is going to be no exception.

Unfortunately, I will not be able to make either of those shows due to another commitment. But if you want to see my “oh” face in person, come to the “Oh Ess Em Cee” conference in November.

Last year all three conferences were held the same week, which was very disappointing for me as it was hard to choose which to attend. This year the Open Source Monitoring Conference was pushed out a month and will be held in Nürnberg, Germany, 18-20 November. While mainly thought of as an Icinga and Nagios conference, the organizers have been very inviting of other projects. We have had a presence there for the last couple of years, but I have only personally been once and it was amazing. So many people sign up that they are able to pretty much rent out an entire hotel, so while the conference is always good it is the conversations outside of the presentations that are the most enjoyable.

I’ll be giving a talk on OpenNMS (‘natch) as well as getting up to speed on what else is going on in the monitoring world.

I hope you can make at least one of these shows. You won’t regret it.

Why There Will Never Be Another Red Hat

My friend Nick sent me a link to a post called “Why There Will Never Be Another Red Hat: The Economics Of Open Source“. It immediately pushed a bunch of my buttons before I read the first word of the article.

First, it was from TechCrunch. I have nothing against TechCrunch and I respect a lot of their work, but they are Silicon Valley-centric, if not the main mouthpiece, and thus I have to take that bias into account when reading their articles. What works in the Valley doesn’t necessarily translate to the world as a whole, which is why a lot of Valley companies seem to quickly plateau after an initial success.

Second, continuing the theme of Valley biases, I strongly believe Red Hat doesn’t get the respect it deserves because it is headquartered down the road from us in North Carolina and not California. There is a strong sense of “you can’t make it if you aren’t here” in the Valley and that extends to a somewhat dismissive view of Red Hat. Plus, I have a big ol’ man-crush on Jim Whitehurst as he is the most successful tech CEO I know that really, really “gets” open source and thus anyone trying to tell me about the “economics of open source” without respecting Red Hat starts off on my bad side.

Finally, the author is currently at the VC firm Andreessen Horowitz, or A16Z as those of us in the know refer to it. In last year’s investment tour I met with a large number of people in the Valley, and the guy I met from A16Z was easily the worst of the bunch. He made the caricatures of the Silicon Valley TV show look mild. He had no interest in us since we weren’t in California and he was more concerned with who we knew than what we did. I left that popularity crap behind in high school. Granted, we only rated an audience with the lower levels of the company, which overall does have a pretty solid reputation on Sand Hill Road, but still it was almost insulting.

Let’s just say my bullshit meter was halfway to pegged before I started the first sentence.

However, I found most of the article to be spot on. In my “Is Open Source Dead?” post I talked about how open source is both greatly increasing while the classic ideals of open source (i.e. free software for everyone) seem to be going away.

My own philosophy is that, at least for certain large and complex (i.e. expensive) software, the proprietary software model is doomed. Customer needs are changing so fast that no closed system can really keep up, and we’ve seen that in the biggest OpenNMS customers. I spent the last week in Ireland and the client was complaining that before they started using OpenNMS, whenever they needed some new functionality in their management solution the proprietary vendor took too long, cost too much and delivered too little to be worth it. Using a free platform like OpenNMS made it much easier to adapt the tool to their business workflow, instead of having to change it to meet the workflow of the management software. There is value in that – value that can be monetized.

Peter Levine started to win me over with “Red Hat is a fantastic company” (grin) and I as I read on I found my head nodding in agreement. He states

Unless a company employs a majority of the inventors of a particular open source project, there is a high likelihood that the project never gains traction or another company decides to create a fork of the technology.

While I estimate OpenNMS has around 40 to 50 active contributors, at least 15 of those are on my payroll, either directly or indirectly as contractors. While I definitely would like to increase the overall number, we are growing fast enough that we can usually hire someone who contributes a lot to the project, and then, since they can spend their full time on it, we as a group continue to contribute greater and greater amounts of the overall code. When I started out on my own back in 2002, I think at least half of the code came from outside of the .com side of things. Now it is probably closer to 5% or less. It has managed to let us focus on our direction for the application.

Then Levine continues:

To make matters worse, the more successful an open source project, the more large companies want to co-opt the code base. I experienced this first-hand as CEO at XenSource, where every major software and hardware company leveraged our code base with nearly zero revenue coming back to us. We had made the product so easy to use and so important, that we had out-engineered ourselves. Great for the open source community, not so great for us.

I’ve heard this tale from a number of people. Become successful and someone like IBM could dump 100 developers on your project and take it over. While we haven’t experienced it directly (rarely do people tell me OpenNMS is “easy to use”), we are constantly finding out about companies who have either based products on OpenNMS or used OpenNMS to provide services for profit. I think this is great, but it would be nice to capture some of that effort back into the project, either in the form of contributions or cash. It is one reason that the next major release of OpenNMS will be published under the Affero GPL.

So, with all this doom and gloom, what is the solution? Levine’s answer is “sell open source as a service”. I couldn’t agree more. This is exactly what we pitched to A16N. It’s something of a “win/win”.

This recipe – combining open source with a service or appliance model – is producing staggering results across the software landscape. Cloud and SaaS adoption is accelerating at an order of magnitude faster than on-premise deployments, and open source has been the enabler of this transformation.

If you have the in house development staff to leverage open source, it makes sense to become active in the community. David just finished a five stop roadshow in both the US and Europe describing the OpenNMS roadmap over the next year as we position the product for the Internet of Things. He met with our largest customers and all of them are eager to get involved, many pledging, OpenDaylight style, to provide the project with developers. They get input into the direction of the product and we get great open source code.

But what if you aren’t a large medical information company or a worldwide financial institution? You may need what OpenNMS can provide but don’t have the time to build in the workflow or customize it. We will have a solution for you.

The only issue I ended up having with the article was when he compared Red Hat to Microsoft, Oracle, Amazon, etc. Sure there might not be another Red Hat, but I don’t expect to see another Microsoft (operating systems and office suites), Oracle (enterprise software), or Amazon (online product distribution) either. This doesn’t mean that there won’t be new mega-companies. In ten years I expect there will be some huge companies that no one today knows about, probably in the areas of 3D printing and biotech (specifically integrating tech into the human body). Enterprise software companies will be represented by a number of large-ish but nimble companies leveraging open source, and I wouldn’t count out Red Hat just yet as they too are pivoting to follow this new business model.

I want to close with one little story. I spent last week in Ireland where I just happened to be in Dublin where the band Wheatus was touring with my friend Damian (MC Frontalot) opening. They had a grueling schedule (shows almost every night) but I managed to see their concert and talk with them afterward.

Damian has just released a new album (it’s excellent, buy it) and he closed his set with “Charity Case” which includes the lyric:

It’s true:
Frontalot’s destitute.
I need you
to buy my CD so I can buy food.

He often pokes fun on the changing nature of the music business (his song “Captains of Industry” suggests that he is not a musician but instead is in the T-shirt business), and we talked about various business models. I pointed out that acts like Elton John are living the high life from work that basically peaked in the 1970s, and that sort of “royalties for life” model is gone. In its place are artists who sell directly to their fans, often including personalized premiums for a higher price, and touring. The band Phish tours extensively and they make millions, all the while encouraging their fans to bootleg their music, something old school musicians wouldn’t think to do.

At this point Brendan, the founder and front man of Wheatus, joined us and stated “there is nothing an old musician can teach a young one about the music business”. That quote really resonated with me.

In summary, I really enjoyed this article. It mirrored a lot of our thoughts over the last year as we seek to make OpenNMS successful. Remember, our plan is for OpenNMS to be *the* de facto management solution of choice and for that to happen will take a lot of work as well as money. But one thing that we will continue to do to emulate Red Hat is to keep publishing as much software as possible under an OSI-approved open source license.

That is still a key to our success: OpenNMS will always be free and OpenNMS will never suck.

Using an XML Parser

You know when the XML nerds say not to use regular expressions to parse XML? They’re right.

As part of a less is more project, we wanted to remove the tags from all of the OpenNMS event files. We spent much of the morning playing with a number of methods to find and replace with empty space those tags, and we failed. We came close a couple of times, but then some weird aspect of formatting (tags that spanned multiple lines, some with spaces and some without, etc.) would foil it.

Then I found out about xmlstarlet. We installed it and ran:

xml ed -L -d "/events/event/alarm-data" [filename]

and it just worked. Pipe that bad boy through find and you are good to go.

While I don’t think the option exists, it would be cool if instead of deleting the tag we could just comment it out, but that doesn’t seem to be currently possible.

When Less is More

One of the things I’ve noticed in my years of deploying network management solutions is that people can get real excited when they go from having no visibility into their network to being able to see in great detail what’s happening, as when they deploy OpenNMS. The problem then changes from having no information to having too much.

Network geeks like myself tend to be loathe to turn off certain alerts, but sometimes that can be the best thing for an organization.

When OpenNMS was started, workflow was based on events. Events appeared in the browser, events triggered notices, you could acknowledge events – pretty much everything was events. But events can be noisy, especially if you leverage the SNMP trap capability of many devices. This is why we implemented the alarms subsystem. Alarms can take many events and reduce them into a single alarm. Alarm processing can be automated to insure that issues that are important are escalated, and issues that have been cleared can be removed. The alarms list is supposed to be a “to do” list for the NOC staff.

In order to make that happen, it is a good idea to consider each alarm in your system and insure that it is “actionable”. Each alarm comes with two fields for tracking the resolution progress, and these can be used to document the actions taken to fix the alarm.

The “Sticky” memo field is used to annotate a particular instance of an alarm. For example, suppose there was a “link down” alarm due to a circuit being cut by a backhoe. The NOC engineer would be able to note that the repair was in process and maybe even include a case number. Once the issue is resolved the sticky memo goes away.

The “Journal” memo field is permanently associated with the alarm. This is for notes that could be useful the next time the alarm happens, such as “Contact Jim – he knows how to fix this”, etc.

Alarms can be acknowledged, which will remove them from the list of current issues. It is pretty easy to create an automation that can unacknowledge an alarm if it hasn’t been cleared in a particular amount of time. Thus you can automate “reminders” that the issue is still outstanding.

This doesn’t discount the value of events. In OpenNMS, events have become more like log messages. When an alarm happens on a particular node, that node’s page will reflect the events associated with it, which may shed some light onto the problem. But having too many events appear as alarms can overwhelm the NOC staff to the point that they stop using the system.

Unfortunately, often the best way of dealing with network issues involves trial and error. By limiting alarms it is possible to miss something important. But once that happens, alarms can be created to insure it doesn’t happen again. But the opposite, dumping too much information into alarms, will guarantee that alarms will be ignored, greatly increasing the chance that something important will be missed.

I developed my alarm philosophy during my first network management deployment in the early 1990s. I was consulting for a cellular provider and installing HP OpenView Network Node Manager (version 2.2 I believe) and they had me working in the server room. Besides being a bit cold, in the corner was a large UPS that was constantly beeping.

Beep … Beep … Beep

I asked Avery, the guy I was working with, what was wrong and he replied “Oh, it always does that”. At that very moment I decided that if there is an alarm and you don’t do anything to resolve it, just turn it off.

Just remember that OpenNMS is a platform and thus you get to make a lot of the decisions on how best to get it to work with your organization. Consider that when deciding which events to turn into alarms, and then focus on using automations to insure that the most important issues are treated as such.

OpenNMS-based App Wins Digital Jersey Hackathon

I was delighted to find out that an Android app using OpenNMS as the backend won the “Best App” prize at the first ever Jersey Hackathon.

Note: This is Jersey as in the island and not Jersey as in New.

The Open Alert “Man on Site” app is a small Android application that is designed to track the activities of people working alone at a remote site. From the wiki:

When activated this reports the location of the phone on a regular basis back to a central OpenNMS server. OpenNMS is configured to plot the current location and status of the device on a geographical map (Open Streetmap).

The App has four buttons;

Start Job – This is pressed by the worker when they start lone working on site. This starts a timer in the local App and on OpenNMS. The local timer will generate an alarm on the local device if the user forgets to report in after a set time.

Report In – This must be pressed when prompted by the local timer. If it is pressed both the timer in OpenNMS and the local device will be reset. If it isn’t pressed then OpenNMS will escalate the ‘Man on site’ event to the next level of severity and notify the OpenNMS operator that there is a problem. (Obviously the local timer should be set to 5-10 minutes less than the OpenNMS time out.) OpenNMS will keep escalating the alarm until it is signalled as critical. If the alarm is escalated, then there should be manual processes in place to contact the worker by other means or send someone else to site to make sure they are OK.

Finish Job – This should be pressed when the worker leaves site. The man on site alarm is cleared in OpenNMS and no further escalation takes place.

Panic – If the panic button is pressed, an immediate critical alarm is created in OpenNMS indicating that the worker on site is in trouble and needs help.

OpenNMS maintains a log of all of the movements of the user and also of the time of starting work / stopping work / panic events which could be important for triage if an incident happens.

Congratulations to the authors, Craig Gallen and Mark Wharton, who created this during the 48 hours of the Hackathon. We built OpenNMS to be a platform and not just an application and this is one example of what can be created leveraging it.

More information can be found on the UK OpenNMS Site and the code is available on Github.

OpenNMS in Dublin

I’ve been to 34 countries so far, and my goal is to hit 50 by the time I’m 50 (which is closer than I’d like). In all that time I’ve managed to miss Ireland, but that is about to change.

Airspeed Telecom is hosting a workshop next Wednesday, October 8th, at the Morrison Hotel in Dublin, described as “probably the hippest & coolest luxury hotel in Dublin city centre”.

That’s just how we roll.

The workshop will feature a case study by Airspeed, as well as a futures roadmap presentation by Dr. Craig Gallen and David Hustace.

Oh, and I’ll be there, too.

If you can make it, please email Liz at