Android Open Source Frustrations

I used to be a huge fan of Apple products, but as they started to lock down their ecosystem the limitations they created started to bother me, so I switched to running as much open source as possible.

It wasn’t, and isn’t always now, easy. One of the gripes I still have against Apple is that their commercial success has spawned a ton of imitators who have decided to lock down their products, quite often without the Apple savvy to back it up. Unfortunately, Google seems to be joining these ranks.

I’m a fan of Google, they do a lot to support open source, and I use a Nexus 6 as my primary “hand terminal” (handy). However, I run alternative software on it, namely OmniROM, which gives me more control over my experience and security.

I pretty much run open source software on all my technology with few exceptions, one being my Android Wear watch. I noticed that there was a new update to Android Wear (version 2.0) so I went to play with it. When I launched the app I got this screen:

Android Wear App Error


So I went off to search for a solution to the error message “This phone has been flashed with an unsupported configuration for companion. you must re-flash it as either signed/user or unsigned/userdebug”. I found a couple of answers that suggested I edit the build.prop file and change


In order to do this, you have to have root access to your phone.


I do root my phone, but I haven’t done it in awhile because Google has introduced this thing called “SafetyNet“. The stated purpose is to prevent malware but in practice what it does is torpedo people like me who actually want to control the software on the devices they own. If you install a custom ROM or have root access, certain applications will not run.

Now I have to choose between running the Android Wear app or, say, Pokémon Go. I chose Android Wear (I pretty much finished Pokémon Go).

The process: Boot into recovery, install SuperSU, boot into system, use a file editor to edit /system/build.prop and change from “userdebug” to “user”, reboot.

Android Wear Mute

So Android Wear will start now, but to add to the frustration the one feature I hoped they would fix is still broken for me. It used to be that if my watch was actively paired with the phone, it would mute ringing and other audio notifications. It doesn’t (and none of the fixes I’ve found work for me) so now I just remember to decrease the volume on the phone down to “vibrate”.

Pokemon Go Blocks root

And, I verified that Pokémon Go will not start now – it hangs on the login screen and then reports an error. This is whether or not SuperSU is enabled, and I think I would have to remove it entirely to get it to work.

Now I know that I can install other apps that will hide the fact that my phone is rooted, but if I do that the terrorists win. I would just rather use apps that don’t force me to give up my rights.

Which brings me to the last frustration. I purchased a bunch of content from Google, but now I can’t access it on this phone. I get “couldn’t fetch license”. This started recently so I believe it has something to do with SafetyNet, but repeated calls to Google Play support yielded no answers.

Google License Error - Deadpool

I have a Google 6P that is stock and doesn’t suffer from the download issue, so it stands to reason that there is some “protection” in place that is preventing me from accessing the content I purchased. I solved the problem by not buying content from Google Play anymore.

I’m pretty certain that it is only going to get worse. Google used to be much better about such things but I think they want to emulate Apple in more ways than one (see the new Pixel phone if you don’t believe me) and that is a shame for all of us.

UPDATE: I found a better way to do this that doesn’t require root. Assuming you have a custom recovery like TWRP, you can simply boot into recovery and then connect the handy to a computer. Using “adb shell” you can then access the system directory and edit the build.prop file directly.

OmniROM 6.0

For the last few days it has been hard to remain true to my free and open source roots. I guess I’ve been spoiled lately with almost everything I try out “just working”, but it wasn’t so with my upgrade to OmniROM 6.0 on my Nexus 6 (shamu).

I’ve been a big fan of OmniROM since it came out, and I base my phone purchases on what handsets are officially supported. While I tend not to rush to upgrade to the latest and greatest, once the official nightlies switched to Android “Marshmallow” I decided to make the jump.

Now there are a couple of tools that I can’t live without when playing with my phone. They are the Team Win Recovery Project (TWRP) and Titanium Backup. The first lets you create easy to restore complete backups and the latter allows you to restore application status even if you factory reset your device, which I had to do.

[NOTE: I should also mention that I rely on Chainfire’s SuperSU for root. It took me awhile to find a link for it I trust.]

When I tried the first 6.0 nightlies, all I did was sideload the ROM, wipe the caches, and reboot. I liked the new “OMNI” splash screen but once the phone booted, the error “Unfortunately process has stopped” popped up and couldn’t be cleared. Some investigation suggested a factory reset would fix the issue, but since I didn’t want to go through the hassle of restoring all of my applications I decided to just restore OmniROM 5.1 and wait to see if a later build would fix it.

Well, this weekend we got a dose of winter weather and I ended up home bound for several days, so I decided to give it another shot. I sideloaded the latest 6.0 nightly and sure enough, the same error occurred. So I did a factory reset and, voilà, the problem went away.

Now all I had to do was reload all 100+ apps. (sigh)

I started by installing the “pico” GApps package from Open GApps and in case you were wondering, the Nexus 6 uses a 32-bit ARM processor.

I guess I really shouldn’t complain, as doing a fresh install once in awhile can clean out a bunch of kruft that I’ve installed over the past year or so, but I’ve come expect OmniROM upgrades to be pretty easy.

One of the first things I installed from the Play store was the “K-9 Mail” application. Unfortunately, it kept having problems connecting to my personal mail server (the work one was fine). The sync would error with “SocketTimeoutException: fai”. So I rebooted back to Omni 5.1 and things seemed to work okay (although I did see that error when trying to sync some of the folders). Back I went to 6.0 (see where TWRP would come in handy here?) and I noticed that when I disabled Wi-Fi, it worked fine.

As I was trying to sleep last night it hit me – I bet it has something to do with IPv6. We use true IPv6 at the office, but not to our external corporate mail server, which would explain why a server in the office would fail but the other one work. At home I’m on Centurylink DSL and they don’t offer it (well, they offer 6rd which is IPv6 encapsulated over IPv4 but not only is it not “true” IPv6 you have to pay extra for a static IP to get it to work). I use a Hurricane Electric tunnel and apparently Marshmallow utilizes a different IPv6 stack and thus has issues trying to retrieve data from my mail server when using that protocol.


I tried turning off IPv6 on Android. It’s not easy and I couldn’t get any of the suggestions to work. Then I found a post that suggested it was the MTU, so I reduced the MTU to 1280 and still no love.

So I turned off the HE tunnel. Bam! K-9 started working fine.

For now I’ve just decided to leave IPv6 off. While I think we need to migrate there sooner rather than later, there is nothing I absolutely have to have IPv6 for at the moment and I think as bandwidth increases, having to tunnel will start to cause performance issues. Normal traffic, such as using rsync, seems to be faster without IPv6.

That experience cost me about two days, but at the moment I’m running the latest OmniROM and I’m pretty happy with it. The one open issue I have is that the AOSP keyboard crashes if you try to swipe (gesture type) but I just installed the Google Keyboard and now it works without issue.

I have to say that there were some moments when I was very close to installing the Google factory image back on my Nexus 6. It’s funny, but the ability to shake the phone to dismiss an alarm is kind of a critical app with me. Since the last time I checked it wasn’t an available option on the Google ROM, I was willing to stick it out a little longer and figure out my issues with OmniROM.

Heh, freedom.

First Look at Ubuntu Gnome 15.10

Back when I was an Apple fanboy, I would eagerly await the announcement of new products by Steve Jobs, with one window open to the live blog feed and the other refreshing the Apple Store page so I could be the first to order the new shiny. Steve Jobs made me fall in love with my technology.

I’ve rarely felt that since, but when the new Dell XPS 13 came out I became once again attached to a laptop and I was determined to make it work under Linux.

While it ships with the latest stable Ubuntu release, 14.04, there are issues. Now I often say that we in the open source community suffer an embarrassment of riches when it comes to choice. Since I’ve found that Linux Mint with Cinnamon works best for me I tried it, but I just could not get it to work with the XPS. To address the shortcomings in Ubuntu 14.04, I read Barton’s Blog and decided to upgrade to 15.04. That addressed a lot of the problems, and I used Ubuntu with Unity for awhile, and although Unity was my first real Linux desktop it doesn’t work as well for me anymore. I also found that its HiDPI support was not quite there. I also tried Kubuntu but its HiDPI support (in my experience) was even worse, and since I’d based my laptop I figured I’d give Ubuntu Gnome a shot.

Now I wasn’t one of those haters who just ranted on Gnome 3.0, but when it came out I couldn’t get used to it. However, when I went to install Ubuntu Gnome on the XPS, I was encouraged that the installer recognized out of the box that I was on a HiDPI screen. There have been a lot of changes since that initial release and I found myself warming to it.

I do want to note that while I found all the desktop options I tried to be pleasantly polished, and, well, “pretty”, I decided to stick with Ubuntu Gnome.

A pesky issue with the touchpad and the touch screen required the 4.1 kernel or later. For months I’ve been running mainline kernels, so when 15.10 was announced with the 4.2 kernel standard, I was eager for the upgrade, and I ran it as soon as it became available.

So what does 15.10 offer? All I can really say at the moment is that it offers a pretty painless upgrade process. I ran “do-release-upgrade -d” and after answering a few prompts it went on its merry way.

Wireless worked out of the box (I used to have to futz with the Broadcom driver when on mainline) and overall the system seemed to be pretty smooth. During the boot process I get this error concerning lvmetad which I think is due to the fact that my entire laptop disk is encrypted, but the boot completes without any other issue and I have confidence it will soon be addressed.

Speaking of boot, Ubuntu Gnome has changed the logo on the boot screen. Instead of the familiar foot:

Old Ubuntu Gnome Logo

You get this new one:

New Ubuntu Gnome Logo

Forgive the quality as I had to produce the second image by taking a picture of the screen. While I like that the colors have been softened from black to a gray, I don’t like the new logo, which looks like two U’s mating. I think it is supposed to represent “UG” but I still don’t like it (and I tend to embrace change). I’m hoping someone puts together a splash screen replacement.

The only real issue that is driving me bonkers at the moment concerns the touchpad. One thing Apple just nailed is the touchpad and the Synaptics one on the XPS is oh so close.

The problem I’m experiencing concerns the cursor jumping when I left click. There are no “real” buttons, so you left click by depressing the lower left corner of the touchpad (or clickpad, whatever it is officially called). Sometimes when this happens, instead of registering a click the cursor will jump to the lower left corner of the screen, and *then* click. It is real annoying in Thunderbird since the icon in the lower left corner puts it in offline mode.

I’ve tried most of the suggestions I’ve found in the t00bz but nothing has helped. I just found a reference to HorizHysteresis and VertHysteresis so I’ll play with those values and see if it helps (update – doesn’t seem to). Not quite sure what they do, however. I think the issue has something to do with a finger from my right hand still grazing the touchpad surface when I make the click.

On the upside, the palm detection issues I was dealing with seem to be improved. Not sure if they have been solved but I’m not noticing it as much. Could be that I’ve just modified my typing form to avoid the touchpad better.

Overall, I’m pretty pleased with the upgrade. It should set up a nice base for the next LTS release, 16.04. I’m not quite willing to give up Linux Mint on the desktop just yet, and I’ll probably try out Mint 18 when it is released next year, but Ubuntu Gnome 15.10 has at least made switching a possibility.

One final note, I like the new shiny and I’m willing to put up with a lot in order to play with it. I give money to Dell to encourage them to supply more Linux offerings, but the downside is that Dell leads with devices designed for Windows first. If you want a true Linux experience with zero issues, check out the offerings from System 76. Our Sable all-in-one desktops Just Worked™.

Okay, so that wasn’t the final note. While I doubt any of my three readers work for major laptop vendors, I really want to see a push for physical kill switches on things like the camera and the microphone, such as on the Librem 15. I considered getting one of those but they are a little sketchy on what “PureOS” actually is, and so I’ll wait to see what others think of it first.

Solution for One Trackpad Issue for the XPS 13

My new laptop is the beautiful new Dell XPS 13 running Ubuntu Gnome 15.04.

It is not perfect, but it is getting close. Lightweight, beautiful screen and awesome battery life (nearly 8 hours the way I use it).

One thing that was killing me, though, was that after a certain amount of time (on the order of tens of minutes and not hours), the trackpad/clickpad thingie would start misbehaving under Gnome Shell, registering bogus clicks. There wasn’t an easy way to fix it outside of a) reboot or b) use an external mouse.

It seems that this issue has been addressed in the 4.1 kernel, so I decided to try it. I’m not sure if Ubuntu is going to support the 4 kernel series officially before 15.10 so I didn’t want to wait.

I downloaded the 4.1.1 kernel here (you’ll need three debs: the “all” headers deb and the image and headers debs for your CPU – I used “generic” and “amd64”), installed them with “sudo dpkg -i” and rebooted. The problem seems to be fixed.

But, my Broadcom wireless driver wouldn’t work. I had to download one more deb from here (via my phone – never play with kernels when you are on a long road trip), install it and now wireless is back.

Now if we could just get palm detection fixed …

Review: Dell XPS 13 (9343) Ubuntu Edition

Okay. When it comes to tech, I want the latest and greatest. To me, the “greatest” must include as much open software as possible. As an ex-Apple user, I want the same experience I used to get with that gear, but with free and open source software.

It can be hard. Rarely is the open source world involved in new hardware decisions by the major vendors, so we learn about new devices after the fact. Thus there is an inevitable delay between when a product is announced and when it properly works with FOSS.

Such was the case with the new XPS 13 laptop from Dell.

Now, I vote with my wallet, so back in 2012 when I needed a laptop I bought the second edition “Sputnik” Dell XPS 13, which shipped with Ubuntu. It served me well for many years and currently runs Linux Mint 17.1 with no problems. When the latest edition XPS 13 was announced, I immediately ordered it, but it didn’t work out so well.

When I discovered that the other option from Dell, the M3800, wasn’t for me, I decided to wait until they officially supported Ubuntu on the new XPS 13. I didn’t have to wait long, and I placed my order the day I learned it was available (I was happy to learn that they had to fix some kernel-level issues and it wasn’t just me).

Why didn’t I wait longer? The XPS 13 is gorgeous. I haven’t felt this strongly about a laptop since my 12-inch Powerbook back in 2013. Others seem to agree, with even Forbes praising this machine.

Anyway, the order process was simple. I got the XPS 13 with the i7 processor, 8GB of RAM, 512GB SSD and the HiDPI touchscreen. The laptop arrived about a week before it was scheduled. Go Dell.

Now for the obligatory unboxing pictures. The outer box arrived undamaged:

Dell XPS 13 Unboxing Pic 1

The laptop itself came in a separate box:

Dell XPS 13 Unboxing Pic 2

with the accessories shipped in a cardboard “square tube”:

Dell XPS 13 Unboxing Pic 3

While the small power supply came with a longer power cable with a “mickey mouse” connector, the XPS 13 comes with a small adapter that gets rid of the cable entirely (like the Apple laptop power bricks).

Dell XPS 13 Unboxing Pic 4

The laptop pretty much fills up its box:

Dell XPS 13 Unboxing Pic 5

Like with my original XPS, there is a cool little intro video that plays when you first start it up:

Please note that it only runs on the first start – you will not have to wait 40+ seconds to boot your system (usually less than 10).

The XPS 13 Ubuntu Developer Edition ships with 14.04, but I had some issues with it. First, it didn’t have the option for encrypting the home directory. I’m not sure how or why that got removed. The system also crashed when I attempted to make a backup image to a USB stick. Finally, there are apparently still outstanding issues with 14.04:

Ubuntu 14.04 includes kernel 3.13. The touchpad will run in PS2 mode and the soundcard will run in HDA mode. Currently (4/15) out of the box the HDA microphone will not work, and you will need some packages from the factory shipped image to make it work properly.

While I knew I was going to base the system, I logged in to the stock image to check out the apt repository. There really wasn’t anything outside of the vanilla Ubuntu (the few Dell packages seem to be just for recovery) so I felt fairly safe in reinstalling.

I immediately went to my default distro, Linux Mint 17.1, but found that a lot of things, especially the touchpad, didn’t work as expected. It did handle HiDPI screens just fine (you could actually see the mouse pointer increase in size when logging in). I figured I’d wait until 17.2 comes out and try it again.

On a side note, I don’t know why it is so hard to get a decent touchpad under Linux. We’re getting closer, but still, it tends to be the weakest point of the Linux laptop experience.

In search of a solution, I found Barton’s Blog and read the following:

With BIOS A00 or BIOS A01 the touchpad will run in I2C mode and the sound will not function. Please update to at least BIOS A02 and the touchpad will run in I2C mode and the sound in HDA mode. (4/15) All of the relevant patches have been backported and all functions will work out of the box.

I really liked the “will work out of the box” bit, so I installed Ubuntu 15.04.

It had been awhile since I’d used Unity, and it has really matured. I especially liked the little touches. When I changed my desktop background, the background of the Dock changed color to match it. Neat.

Where Unity still has some way to go is in HiDPI support. There is a scaling factor you can set, but it only applies to a small part of the UI. I still ended up having to customize many of my apps. For example, if you look at the settings page with scaling, a lot of the text under the icons are cropped:

Dell XPS 13 Ubuntu Text Cropped

Not a show stopper, and I used it for over a month without getting too annoyed.

Last week I saw that the release candidate for Mint 17.2 was out, so I dutifully backed up my Ubuntu install, based the system and installed Mint. Things seems to work better (although HiDPI support was not working by default), but I ran into a weird problem with trying to click and drag.

While everyone seems to deal with trackpads differently, the way I click and drag is to use the index finger of my left hand to click and hold the lower left corner of the trackpad, and then I use the index finger of my right hand to move the mouse pointer. This works fine under most desktop environments, but under Cinnamon it seems to interpret it as a right click (which usually causes a menu to drop down). If I just used a single finger to click on the window header and then move it, it worked as expected, but I couldn’t get used to it enough to continue to use it.

Oh well. I’ve posted a question on the Mint forums but no one has been able to help.

Anyhoos, since my system was based I decided to try out some other 15.04-based distros while I had the chance. I had heard great things about the new Plasma interface in KDE, so Kubuntu was next.

I can’t say much about Kubuntu since its HiDPI support is worse than Unity. Everything was so tiny I couldn’t spend much time in the UI. Oh well, what I saw was pretty.

And I should stress that this was a recurring theme in my experiments with desktop environments. Every UI I’ve tried has been beautiful and more than able to compete with, say, OS X.

By this point I decided to punt and just search on “Linux Desktop HiDPI”. Several of the results touted that Ubuntu Gnome was the best desktop to use for HiDPI systems. So, before going back to Unity I decided to give it a shot.


I haven’t used Gnome 3 in awhile, but I was encouraged in that even the install process handled the HiDPI screen well. It has become really mature, and so far has provided by far the best experience with the XPS 13. I’ve had to do little to get it to work for me.

Is it flawless? No. There is an issue with the touchpad where it occasionally translates touches into click (kernel patch approved). If you sleep the system, the touchscreen will stop working (but you can reload its module). Sometimes, the system doesn’t sleep when you close the screen, which can cause the laptop to get really, really hot.

But these are minor issues and I expect them to be addressed in the near future. I am confident that I’ve found a great combination of software and hardware, and that it will only get better from here.

I have just a few more notes to share. The battery life is outstanding – I can get 6-7 hours of use without recharging. The “infinity screen” is beautiful and bright, but by having almost no bezel they had to move the camera to the lower left corner, which creates a slightly odd viewing angle.

Dell XPS 13 Camera Angle

In closing, here are a couple of shots comparing the XPS 13 with the M3800.

Dell XPS 13 vs. M3800 Pic 1

Dell XPS 13 vs. M3800 Pic 2

Dell XPS 13 vs. M3800 Pic 3

Building an Open Source PVR: Step Three – Electronic Program Guide

The most frustrating thing about this project was getting the Electronic Program Guide (EPG) to work. Unfortunately, it isn’t easy.

This was one of the things that TiVo excels at doing. You are basically paying for a very up to date program guide. They also offered something called a “Season Pass” which would cause all of the episodes of a particular program to be recorded without having to explicitly select them.

When I got my EyeTV system, this part was a snap. They partner with TV Guide to provide the service, and unlike TiVo’s $14.95/month fee it is a yearly fee of $19.95 (with the first year being included with the unit).

Even my Sony Bravia is able to get over the air EPG information, but I wasn’t able to get that to work with OpenELEC.

The actual EPG configuration occurs in the Tvheadend software. You get a screen like this:

There are three main areas of configuration: the “Internal Grabber”, “Over-the-air Grabbers” and “External Interfaces”.

The internal section displayed a cron job but no module options. None of the OTA grabbers seemed to work, and there wasn’t a module for North America. That left the external grabbers.

I started digging around and found that it really isn’t easy to get this running.

One tool that kept coming up was XMLTV. On the frontend configuration for the Tvheadend client in OpenELEC they even have a section on it:

XMLTV is a number of things, such as a format for representing TV listings, but it is mainly a set of tools “to obtain, manipulate, and search TV Listings”. It contains programs that will connect to an external source to gather EPG data.

Unfortunately, OpenELEC doesn’t ship with it. There is a script called “tv_grab_file” which is used to manage the XMLTV data, but not to actually acquire that data.

For me the easiest solution was to install XMLTV via apt on my home Debian server. It comes with a script called tv_grab_na_dd that can be used to fetch the data.

But I still wasn’t done. I needed a data source. It looks like all the cool kids use Schedules Direct. They are a non-profit that promotes open source software and provides, for a fee, access to EPG information. Since they had a free trial I signed up, configured my tv_grab_na_dd script to access their information, and voilà, I had an XML file with what appeared to be useful information.

I placed that in the webroot of my server, and then configured OpenELEC to point to it. Nothing happened. So I copied the file to the OpenELEC server, modified the client to use the “FILE” method (see screenshot above) and nothing happened.

I finally had to uncheck the XMLTV checkbox under “External Interfaces”. When I did that I finally had something under the “Internal Grabbers” section.

The last chore was to associate the channels I had discovered with the program guide.

Prior to getting all of that to work, the drop down for “EPG Source” had been blank.

So, to summarize my steps:

  1. Get an account at
  2. Install the XMLTV tools somewhere (I used a Debian box)
  3. Configure XMLTV to access your Schedules Direct account
  4. Set up a cron job to periodically grab the updated EPG information and store it in a web root:
     0 1,13 * * * /usr/bin/tv_grab_na_dd --config-file ~/xmltv.conf --days 7 > /secure/html/xmltv.xml
  5. On the OpenELEC box, set up a cron to fetch the data:
    0 2,14 * * * wget -O /storage/xmltv.xml

Whew. So far everything has been working well. You want to be sure not to fetch the data too often as that can overwhelm the Schedules Direct servers. My current seven day XML file is about 10MB.

I went ahead and signed up for a year account for $25, bringing my total to $705.92 (the hardware was $680.92 and the software was, yup, $0). It’s quite possible to shave off about $200 by going with less memory and a smaller SSD (or using an HDD) or if you already have a server to run the Tvheadend backend you could get by with a Raspberry Pi.

My next steps are to play with all the cool add-ons and to try and organize my pictures in a fashion where they would be usable with the system. More fun for me.

Building an Open Source PVR: Step Two – Software

So in my quest to replace my Mac-based PVR I wanted something lightweight that could be controlled via a remote. I had issues with my current setup when a keyboard or a mouse just had to be used, and I wanted to avoid that. Since this system would be dedicated to the PVR, I didn’t want to install anything that wasn’t necessary.

This left me two options: Kodi (formerly XBMC) and MythTV. I decided to try out Kodi via the OpenELEC project. OpenELEC aims to create a very lightweight instance of Kodi that can be installed (and probably even run) from a USB stick. Sounded like just want I needed.

The easiest way it install OpenELEC is to create a bootable install USB stick. This is pretty easy, if you read the instructions correctly. I actually spent a lot more time on this than I needed because of a failure to do so. Once you download the image you need (I used the new bundled “generic” version which works with Intel-based devices as well as most others), you insert your USB stick and then run the “create_livestick” command. You pass a parameter to that command which indicates the path to the USB stick, i.e. /dev/sdX where X is the drive letter.

This is where I screwed up. I could easily tell that the stick I used was mounted on /dev/sdh1, so that’s what I used. The problem was by adding the “1” I was specifying a partition and not the whole drive. It took me an embarrassingly long time to figure out what I was doing wrong.

Once the stick was created, I just booted the Intel NUC with it and followed the on-screen instructions. Pretty soon I had a working OpenELEC system.

Now let me stress that OpenELEC is not designed as a dedicated video recorder. It is designed to run Kodi which is a media center, so most of its functions are aimed at managing libraries of media and not recording television. The menu is organized by media type:

You have a Pictures menu, and if you have the PVR add-ons installed, a TV menu item. Videos are movies, whereas TV Shows are media files that have been identified as TV Shows (different than things you have recorded on your PVR).

Then you have your Music files, any Programs you have added to the system (as in software programs) and the System menu itself.

The system information screen gives you a read-only overview of the system, including memory usage and frames per second.

To actually change things you need to go to the configuration menu:

You can add media sources via pretty much every network protocol currently in use. As I have a couple of UPnP servers on my home network I used that format, but I found that when I added new content the system wouldn’t pick up the changes. So I installed an add-on to update my library but it didn’t help:

I searched but couldn’t find any way to get the changes to show up. There is a menu that comes up when you hit the left arrow that is supposed to update the library but it wouldn’t work for me. Since my UPnP servers can also serve files over SMB, I tried that and it not only fixed the issue but opened up a whole new level of coolness.

You can scan for TV Shows in your media files, and when you do Kodi will try to “scrape” information off of the Internet for such things as artwork and episode synopsis. You have to have your library named in a particular fashion (which I do) but then it is pretty automatic:

This didn’t happen when I was using UPnP.

This is all well and good, but I still get a lot of content through Over the Air (OTA) television broadcasts and the whole purpose of this exercise was to get that working. In order to add PVR functionality to OpenELEC you need to install add-ons. This usually consists of a “backend” application that does all of the heavy lifting with respect to video capture and encoding, and a “frontend” or client application that connects with the backend and displays the video. I specifically chose more powerful hardware as I wanted both features on the same unit.

First I needed to install the backend, which is a piece of software called “Tvheadend“. It was a little hard to find in the menus as it is a “service” and not a normal add-on, so you have to find the “services” section of the add-ons menu:

and then you can find and enable your services:

Like most add-ons within Kodi, you will have an “information” screen:

and a configuration screen:

The configuration screen comes into play when you set up the Electronic Program Guide (EPG) but I’ve reserved that for a separate post.

To access the Tvheadend software, you have to browse to it via http://[openelec-server]:9981. That would be a different URL, of course, if you installed it on a server other than the OpenELEC box. This is where it got difficult as most of the documentation on-line is out of date and the menu options have changed. I’ll post what I did in the hope that it might help someone else out.

First, you want to go to the Configuration tab:

You don’t have to do anything on the “General” tab if you don’t want to, but you do need to see a TV capture device on the “DVB Inputs” tab:

If you have chosen a supported capture unit, it should be displayed here. If not you’ll need to either figure out why it isn’t or get another unit. My Kworld UB435-Q showed up with support for both DVB-C and ATSC formats. Since I am interested in OTA broadcasts in the United States, I chose to enable the ATSC interface as the other is used for cable, which I don’t have.

Note in this screenshot that there is a Network entry called “OTA”. This was not there when I first enabled the interface. I had to go and set it up on the “Networks” tab and then add it.

This took me a rather long time to figure out. You need to tell the Network what multiplexers to use, and it looks like you would need to add them individually under the “Muxes” tab. It turns out that there are a number of pre-defined muxes including one for North America ATSC called “United States: us-ATSC-center-frequencies-8VSB” so I just chose that for my “OTA” network:

Once I associated it with my adapter in the “DVB Inputs” tab, I had a list of television channels:

Tvheadend is pretty cool on its own. If you notice on the screenshot there is a “play” button next to the channel name and if you click it you get a stream that will play on your computer. We recently had a bad weather day and I worked from home, and I was able to keep the local news up in a window while I worked. I haven’t really explored all of the features of Tvheadend, but once I got to this point it was time to head back to OpenELEC and set up the frontend client.

Going back to the configuration menu and looking through the Kodi add-ons, there is a section called “PVR Clients”:

I wanted the Tvheadend HTSP client:

Just like the backend, there is an information screen:

and a configuration screen:

In this configuration screen, you have to point to the Tvheadend backend, which in my case is on localhost.

If you get to this point, then you should see a new “TV” menu item:

You can watch live TV:

But the main reason I wanted a PVR was to time shift and store TV programs so that I could a) skip the commercials and b) make sure I didn’t miss anything. This requires access to the Electronic Program Guide and I could not figure out how to get it to work. I spend days worth of my limited free time working on this. The forums and the existing documentation were not much help.

I got so frustrated that I based the system and installed Mythbuntu – an Ubuntu-based distribution that focuses on MythTV in the same fashion OpenELEC focuses on Kodi. I figured that since MythTV was designed to be a PVR from the start, it might be easier.

There were a number of differences apparent right away. Mythbuntu is huge compared to OpenELEC. It includes a number of things that just aren’t required. It was, however, easy to install, and building on my newly earned knowledge with OpenELEC I was able to navigate the initial setup easily. I found that the MythTV documentation was slightly better than OpenELEC/Kodi/Tvheadend, but I still hit snags.

The first was that MythTV wouldn’t recognize the Kworld tuner I was using. It did, however, see the EyeTV tuner from my Mac-based install. But using it and having it scan for channels turned up nothing. The channel scan seemed to complete as expected but nothing was discovered.

I spent another day’s worth of free time trying to get that to work, but I gave up pretty easily. I wanted to use the Kworld tuner and possibly sell the EyeTV unit, so it bothered me that it wasn’t recognized. Plus, Mythbuntu just wasn’t the lightweight install I wanted, so I decided to go back to OpenELEC.

I did finally get the EPG working, but I’m going to reserve that story for the third and final post in this series. Once that happened I could see the guide in OpenELEC:


Is it perfect? No. The OpenELEC TV frontend is pretty limited. While I can schedule a show for recording through the EPG by setting a “timer”, I have not found a way through the GUI to record a whole series. I can do it through the Tvheadend web interface by selecting the show in the EPG and choosing “Record Series”:

and then it will show up on the “Timers” section of OpenELEC:

You can access saved recordings through the menu as well:

but it frustrates me that there doesn’t seem to be a way to delete the recording once I’ve seen it. I have to do that through the Tvheadend web page.


Overall I’m happy with my new OpenELEC Kodi install. There are a large number of add-ons that I haven’t explored yet, and perhaps I’ll have the time one day. When I was younger and got a new piece of technology I would try out every single feature. Now I tend to do the bare minimum I need to have a viable solution and then stop. (grin)

If you don’t care about OTA television then OpenELEC is a breeze to set up. The only issue I see is that there are a number of closed solutions, such as Google’s Chromecast and Amazon’s FireTV that do pretty much the same thing, at least with respect to video, and they cost about as much as a nice meal versus several hundred dollars.

But I like OTA television. Between it and other services I have like Netflix and Amazon Prime Instant Video, I always have something to watch. Plus, OTA HDTV signals aren’t compressed like those from cable and satellite providers, so the quality is excellent.

This experiment to create an open source PVR both emphasizes the good and the bad about free software. I consider myself pretty technically savvy but I had a lot of issues getting this to work. But I also learned a whole lot about four open source communities (OpenELEC, Kodi, Tvheadend and MythTV) and how OTA television actually works. My PVR is not some magical black box but a tool that I can control and manipulate to my benefit.

Technology is key to personal freedom and ceding the understanding of how it works to third parties can be dangerous. I know it sounds silly to sow fear about something as trivial as the ability to record “The Big Bang Theory“, but rarely does societal change happen in a huge way all at once. It is more a series of small things, chipping away at our freedoms over time, and getting this to work just made me feel like, at least in my life, that I was making a difference.

Many thanks to the people behind OpenELEC, Kodi, Tvheadend and their communities for making this possible.

Building an Open Source PVR: Step One – Hardware

Many years ago I was jealous of my friends that had a TiVo. Since I live out on a farm and get my television the old fashioned way via an antenna, TiVo didn’t work for me. I was stuck with using a VCR to record my television programs.

UPDATE: My friend Tanner pointed out that TiVo does offer an Over the Air (OTA) option now. It runs $49.99 for the hardware plus a monthly charge of $14.99 (a “lifetime” option is available as well). Looks pretty cool and I’m sure it is easier to set up than what I did. If I wasn’t all about open source I would have seriously considered this.

Then I got introduced to a product called EyeTV. This is a product designed for OS X that looks like an old school USB stick with a coax connector at one end. Connect that to your antenna, plug the USB end into your Mac, install the software and, voilà, you have your own personal video recorder (PVR). The hardware is actually a Hauppauge WinTV HVR-980 but the secret sauce is the software to make it easy to use.

Note: it appears they don’t make that unit for the US anymore and instead sell an external unit.

The EyeTV setup works fine, but as I’ve moved away from Apple products I have been wanting to replace it with an open source PVR. There are a number of open source media suites, including Kodi (previously XBMC) and MythTV, and I’ll cover that in my second post. I decided to explore OpenElec, which is a lightweight packaging of Kodi, and since I could not find an “all in one” guide for setting something like this up, I wanted to document the process I went through in the hope that it will help someone else.

The first challenge was finding suitable hardware. I run EyeTV on a Mac Mini. It’s small and quiet and has enough horsepower to drive the EyeTV software at HD resolutions. But, it is a full operating system and has some frustrating issues. First, I use Front Row, which Apple dropped with OS X Lion. Next, it is connected to a UPS and when the power blips (which happens frequently on the farm) I get a little pop up on the screen that requires you to click “ok”, and that can’t be disabled. Almost every time I go to watch a program I have to dig out a wireless mouse just so I can acknowledge the dialog box. I wanted something that could be run entirely by remote and was as lightweight as possible.

Noise from the unit was a big consideration for me. If you want to do a lot of video processing you need a rather powerful machine, but those tend to need cooling and cooling means noise. To deal with this, most PVR software comes with the concept of a “frontend” or playback client that talks to a “backend” that actually acquires and manipulates the video stream. A lot of people have had success in using a Raspberry Pi as the frontend, but I wanted a unit that was both powerful enough to act as both the frontend and backend while not generating much noise.

Yeah, I know, first world problem.

The first unit I tried was the Z3RO Pro from Xi3. My friend Donnie works at WDL Systems and they specialize in embedded devices, so he tends to be pretty up to date on the latest new shiny and recommended I check it out. I ordered one from Amazon preconfigured with an SSD hard drive and 4GB of RAM.

It’s a very stylish and compact unit, and the one I bought came preloaded with SuSE Linux. It did have noticeable fan noise, however. Nothing too obnoxious, but in my quiet office I could definitely hear it.

That wasn’t a show stopper, but what did kill it for me was the fact that I couldn’t get it to talk to any of my HDMI devices. The Z3RO Pro comes with two video ports, one for DisplayPort and a combo DP/HDMI connection. You have to be very careful when putting an HDMI plug into that port as there is nothing to really guide the orientation (you put it in upside down) and if you force it in the wrong way you can damage it. I didn’t have a HDMI cable at the office, so I used an HDMI to DVI one and it worked fine, but when I got home the unit wouldn’t recognize the monitor with a straight HDMI cable. I tried three different cables and three different monitors and not even the boot screen would show up, so I had to return the unit. There is no audio out on the Z3RO Pro outside of HDMI and audio is a big part of this experiment.

The next unit I tried was the Intel NUC (Next Unit of Computing). This looks like a taller, smaller version of the original Mac Mini. It comes as a kit and you have to add your own storage and memory. Since my current setup has a 350GB HDD, I opted for a 512GB SSD and 8GB of Crucial RAM. The NUC also requires a mini-HDMI connection, so I bought a mini-HDMI to HDMI cable as well.

When the unit arrived, there was a cute little Easter Egg upon opening the box:

In addition to the computer hardware, I had to obtain a compatible TV tuner and a remote.

For the tuner I went with a Kworld UB435-Q, which is supported by OpenElec, and an Ortek VRC-1100 for the remote control. I didn’t really care about the remote itself as I have a Harmony 900 universal remote and my plan was to use it, but I needed an IR receiver and the Ortek was cheap.

I did manage to get this setup working well with OpenElec, but it wasn’t easy and I’ll cover the details in the next post. One funny thing I found was when I was programming the Harmony it appears that the NUC includes an IR receiver so the Ortek is unnecessary. You might be able to understand my confusion when the remote was working but the little red LED in the Ortek receiver wasn’t blinking. I ended up unplugging it and when I could still manipulate OpenElec I concluded that the NUC must have it. During this process, however, I ended up blanking my Harmony and I had to reconfigure everything, and at some point the NUC’s IR stopped working. Since I got it working with the Ortek’s receiver and I’m not in any need of extra USB ports, I kept it.

The cost of this experiment so far:

Intel NUC BOXD54250WYKH1 $351.00
Crucial MX100 512GB SATA 2.5″ SSD $213.99
Crucial 8GB Kit (4GBx2) DDR3 1600 $66.99
Kworld UB435-Q USB ATSC TV Tuner $24.99
Ortek Windows 7 Vista XP Media Center MCE PC Remote Control $16.96
AmazonBasics High-Speed A to C Type, HDMI to Mini-HDMI Cable $6.99
Total: $680.92

You can shave $100 to $150 off of that price by using a smaller SSD. The Intel NUC I bought also supports laptop-size HDDs, but if you plan to use an SSD and just want a smaller unit, there is an SSD-only version of the NUC that is a little smaller and a couple of dollars cheaper.

This is much more expensive than a lot of media players out there, and the main cost has to do with my requirement to capture live over-the-air digital television. For many people, this is an archaic concept (I was talking about this in the office and Ben jokingly asked “What’s a channel?”) and thus of limited value, but for those of us still using the old paradigm a PVR like this is useful.

Finally, while my NUC is based on Intel’s Haswell chip, they just announced that the Broadwell-based units will ship soon. In an ironic twist, they are being manufactured via a partnership with Xi3. As soon as that happens, you can expect the price of my unit to drop, so if you are considering a project like this, you have options.

And that’s what open source is all about to me: options.

♫ It’s Hard Out Here for a (Free Software) Pimp ♫

In thinking about a title for this particular screed, I almost went with “Papa’s Got a Brand New Phone” but that didn’t really encompass what I was after as much as a play on the Oscar winning “Best Original Song” by the Three 6 Posse.

When I first got involved in free software, I thought it was too good to be true. I thought “free” implied “no work” but I was confusing free (gratis) with free (libre).

Sometimes freedom takes work.

It takes effort and no small commitment to run as much free software as possible, and no where is that more evident than when it comes to choosing hardware.

I used to be a big Apple fanboy, and thus my personal technology decisions were easy: buy the newest shiny from Apple. When I decided to divorce myself from them, it took awhile to adjust to the fact that, quite frequently, the new shiny is not the best choice for a free software advocate.

But I’ve been stymied time and time again. When looking for a new laptop, I bought the latest Lenevo X1 Carbon and ended up sending it back. It was just too new to support my operating system of choice, whereas my old, second generation Dell XPS 13 “Sputnik” runs Linux Mint Debian Edition (LMDE) just fine.

So I tried a new tact.

When I was in the market for a new phone, I figured the best bet was to work backwards.

I had been using a Samsung Galaxy S3 running Cyanogenmod. However, right after I upgraded the baseband to run Kit Kat, the phone would constantly and randomly reboot. I tried everything I knew of to fix it and tried out just about every major ROM there was but it would still crash. Only by running Jelly Bean could I mitigate the issue somewhat. Then instead of bouncing every hour or so, it would only reboot once or twice a day.

Now I play a game called Ingress … a lot. It is a heavy user of the display, the CPU, the network and the GPS. While these reboots might have been acceptable to a casual user, they were killing me. While I may have somehow corrupted my S3, it was probably due to some other hardware problem, so I decided to get a new phone.

One of the pluses about putting in the time to use free software is quite frequently you learn how things work. I would never have even known about baseband versions, bootloaders, recovery, etc. if I hadn’t played with my phone. I also get a lot of options, such as which ROM to run. In all my research I decided that my philosophy matches up best with the team behind OmniROM.

OmniROM doesn’t have as many options as, say, AOKP, but they are dedicated to keeping it as open as possible and I admire that. Plus they have a pretty decent OpenDelta update application that makes staying on the latest release pretty simple.

Once I decided that I wanted to run OmniROM, I just worked backwards to pick out a phone.

Here’s where I had to make a choice about freedom.

What I loved about my S3 was that it had a replaceable battery and a microSD slot. Some days I’m a heavy user of my phone and even the best phones can’t last the day on a single charge. The microSD slot made it easy to transfer data from my phone to my computer as well as easily and cheaply expanding the available memory.

Not many phones have these two features. In fact, the only modern phones I could find were both from Samsung: the S4 and the S5.

The S5 is not supported by OmniROM, so my choice was simple: get the S4. I ordered an unlocked S4 from Amazon and got ready to enjoy the new-ish shiny.

It was not to be.

While the description on Amazon said that it was “unlocked” it turns out that Samsung has decided to block third party bootloaders, even on the S4, with an update issued last November, so it is impossible to replace their default operating system with a free one. While there are some ways to “dual boot” the phone, this was unacceptable to me, so I sent it back with the reason “item did not match web site description”. Just being carrier unlocked is not enough to merit the term “unlocked”.

In looking over the remaining options, I ended up settling on last year’s HTC One (m7). And I do mean settle: the One has no microSD slot nor does it have a replaceable battery. But these are things I can work around in the pursuit of freedom. I got a microSD to microUSB connector and an external battery pack that can keep my phone running for days. It also has a somewhat lo-rez camera at 4 megapixels, but it seems to take pictures just fine.

You do have to jump through an extra hoop in order to unlock the bootloader, but HTC made it pretty simple. You just have to log in to their developer site and post a code and they’ll send you back a file to run to unlock your particular phone. Not as easy as, say, a Nexus phone, but it isn’t too much extra work.

Now I have the latest Kit Kat running flawlessly on the phone. I’m able to remove the Google search bar, which in my case just takes up space, and I can modify the number of icons displayed per page.

It’s pretty awesome.

Is the HTC One a perfect phone, especially for playing Ingress? No – it is not perfect. But it is pretty darn good. At the Gettysburg anomaly it held up all day with zero reboots, whereas other people were reporting them with usually stable phones such as the Nexus 5. Note that if I didn’t have any other considerations I would have gotten a Nexus phone, but since I play Ingress with my spouse and she has one I wanted another brand in order to diversify the radio technology. In some places her phone gets signal where mine does not, and vice versa, and thus we can tether if needed.

I like to vote with my wallet and I buy products from companies that support freedom. I don’t understand why Samsung felt the need to lock down their devices. In part I think it is Apple-envy, but they just lost out to those of us who want to truly own their hardware. I’m not sure if it is enough to affect the bottom line, but it has soured me on Samsung products as a whole and I do buy a lot of technology.

So, remember that freedom takes work, but it’s worth it in the end.

Upgrades? Upgrades? We don't need no stinkin' …

I am incredibly behind on blog posts, for which I apologize. Three weeks ago (sheesh) I was in the UK for the OUCE, and I owe a post on that. The week after that was filled up with meetings, mainly exciting meetings that I hope to be able to talk about in the near future, and this week I am supposed to be on vacation.

Unfortunately, I caught a nasty cold while in Southampton that I haven’t been able to completely shake and this week I hurt my back, which makes it even hard to type. As George Bernard Shaw said, youth is wasted on the young.

Anyway, apologies once again for disappointing my three readers (one of whom I met on the plane ride back from the conference, hi Greg!) and I hope to do better.

This quick, vacation week post concerns upgrades. I’ve been a bit of an upgrade fool and I thought I’d share some of my stories, most of them actually pretty positive.

The first concerns OpenNMS 1.12.6, which was released this week. That upgrade was the smoothest of the three I did. Upgrading from 1.12.5 only involved two configuration files changing: datacollection-config, which added Cisco Nexus metrics, and, which added a new permission “role” for accessing the Asset Editor UI without being an admin user.

This release also addresses a security bug where an unprivileged user could get a list of user names via ReST. While not a huge issue for most OpenNMS users (How many of you still have admin/admin as the username and password? Be honest) it is still a recommended upgrade if just for all of the other fixes included.

The second upgrade I did was to the latest Ubuntu LTS, Tasty Trollop. It, too, went pretty smoothly.

Many years ago I got frustrated with my laptop and laptops in general. First off, they seemed to be expensive for the performance you received. Second, I would often have to make the “laptop drive of shame” when I forgot it on my way to the office. Finally, I just hated to have to lug it around when I wasn’t traveling.

So I saw a deal on woot for an HP desktop with pretty nice specs, and I bought two of them: one for home and one for the office. While I do have a small laptop for travel, for the most part I use these desktops, and with modern network speeds I can usually access any information I need from either machine.

Now the office machine, which is the one I use most of the time, gets a lot more attention than the one at home. While they both started out running Ubuntu 12.04 (Pastel Pederast), I upgraded the office machine to the newer, non-LTS Ubuntu releases and wasn’t as happy with them. I ended up switching to Linux Mint on both that machine and my laptop, but I left the home machine running Ubuntu.

My initial thought was to wait until Mint 17 came out and then switch to it, but I figured there could be little harm in upgrading to the new 14.04 LTS release in the meantime. The first challenge was actually getting the operating system to realize there was a release out there. I ended up running “sudo do-release-upgrade -d” with the “-d” option finally finding it and getting it started. I run a pretty vanilla setup at home, so there were only a couple of configuration files requiring attention and otherwise the whole thing went smoothly. Took about two hours to download and complete.

So far I’m pretty happy with the new release. No huge new changes, and everything seems to work well together. I did have to re-enable workspaces, and I took advantage of the new option to move application menu bars back to the window versus being in the title bar (I use a 27 inch monitor and it can get a bit tedious swiping the mouse up to the top) but other than that, I don’t see too many changes. Empathy has gotten worse, at least for me, but it was easy to switch to Pidgin. The only bug so far is that if I let the lock screen kick on automatically, a good portion of the time I can never get it to come back up: the screen just remains blank. I usually have to ssh in from another machine and reboot. Other people are reporting the problem (search on “lock screen freeze”) and I have yet to try and restart lightdm (suggested as a way to bring the desktop back), but as a workaround I just manually lock the screen whenever I leave, which is a good habit to be in in any case. I figure they’ll fix this soon.

I still prefer Cinnamon to Unity, but I’m happy using either, and due to the ease of upgrading I’ll probably stick around to using Ubuntu at home for the foreseeable future.

The final upgrade I did this week concerns OS X. I still have three Macintosh computers at home. There is an older Mac Mini that solo boots into Debian that I use for a web and file server. There is an older 24-inch iMac that tri-boots OS X, Ubuntu and Windows 7 that is usually booted to Windows since that is what my wife uses, and there is a newer Mac Mini that runs Snow Leopard and acts as my DVR using the EyeTV product. It also gathers and publishes my weather station data via wview.

I was cleaning up the DVR when an “Upgrade to Mavericks” window popped up. Now I really hated Lion and never used Mountain Lion, so there was no real reason to upgrade, except that I’ve been having an issue where I can’t add any bluetooth devices to the Mini. I really wanted to add a mouse, since some times stupid windows pop up that ruin the DVR aspect of the setup and they can be a pain to close if I have to VNC in. I figured, what could go wrong?

Of course, the first thing I did is make sure I had a full Time Machine backup. I really wish I could find a “bare iron” restore app for Linux that was as easy to use. I do like the Ubuntu backup integration with Déjà Dup, which seems to be missing in Mint so I use BackInTime, but neither offer the ease of Time Machine.

The upgrade to Mavericks didn’t go as smoothly as the others. At some point close to the end, the monitor went blank and wouldn’t come back, so I had to power cycle the system. This caused the install to start over, but the second time it finally completed. I then had to go through and turn off all of the “spyware” that seems to be on by default now. It automatically signed me up for “iCloud” which I turned off (good thing I didn’t have any contacts, etc., on this system or Apple would own them) and I also disabled Facetime, which required deleting a plist file out of the Library directory. My weather station software didn’t start because of a missing USB to Serial driver, but once that got installed things seem to work. I was even able to add a bluetooth mouse with no problem.

Then I found out that Front Row was missing.

Now when I had a Macbook, I hated Front Row. I was always turning it on by accident. But for my DVR, it made a great interface to EyeTV. Apparently Apple has dropped it since Lion, so I spent a couple of hours trying to find a replacement. When nothing I found was acceptable, and with my growing distrust of Apple with respect to the information it was going to capture on my computer, I decided to go back to Snow Leopard. Should be easy, right?


Both the version of Snow Leopard I have on a USB stick and the install disk that shipped with the computer would now gray screen when trying to boot. I know that Mavericks futzes around with the disk partitions, so I figure that is to blame. I was just about to boot to an Ubuntu disk just to repartition the disk when I decided to try and boot into the new “recovery” partition that Mavericks installed. While I didn’t have much hope that it would be able to access a Time Machine backup made with Snow Leopard, I was pleasantly surprised when it worked.

Another surprise came when I found out that my bluetooth mouse was still associated with the computer. I’ve always thought of the term “backup and restore” to mean one puts a set of bits into storage and then puts those same bits back. Apple has a weird interpretation of this, especially when it comes to the iPhone, where “backup and restore” can mean “perform a complete operating system upgrade in the process of putting back user data”. Apparently Time Machine is similar, and my new device settings were remembered.

So in summary, I guess the time I spent playing with Mavericks was worth it. I know now that I don’t ever want to upgrade from Snow Leopard, and I got my bluetooth issue addressed, if not fixed. Ubuntu 14.04 LTS is worth checking out, especially if you are looking to get rid of Lion/Mountain Lion/Mavericks, and do upgrade to OpenNMS 1.12.6.

You’ll be glad you did.