The Technology Choice Struggles of a Freetard

TL;DR: With the demise of CopperheadOS, I’ve had to struggle to find a new mobile operating system. With the choices coming down to Google or Apple, I decided to return to Apple and I bought an iPhone. Learning quickly that it is very hard to manage the iPhone under Linux, I also decided to switch to a Macbook Pro. A month later and after a business trip with the laptop, I am returning to Linux as my primary operating system.

This is a rather long post that I doubt will interest even one of my three readers, but as I expect a small subset of the population agonizes over technology choices as much as I do, perhaps someone will find it useful.

Back in 2011 I decided to stop using Apple gear and switch to running as much free software as possible. It was difficult, but I managed to switch almost all of my technology to open, if not always free, options. The hardest part was mobile.

For years people have been trumpeting each new year as “The Year of the Linux Desktop“. The problem is that more and more people are doing without a desktop entirely, and instead interact via mobile devices, so it is becoming more like “The Year of the Free Buggy Whip”. The broader free and open source community totally missed the boat when it came to mobile.

Seriously, where is the “Linux” of mobile? We don’t have it. Our choices are pretty much limited to Apple and Google.

Apple is pretty straightforward. They control the whole experience so you buy devices from them and you are allowed to run the software they let you. The freetard in me chafes at these limitations.

So that leaves Android. The problem with Android is that it is pretty much Google. Almost all of the Android Open Source Project (AOSP) derivatives rely on Google for both security updates and device drivers (which are rarely open). They start from a platform over which they have little control, unlike Linux.

Google is becoming more and more intrusive when it comes to surveillance. When you first sign in you are asked “Do you want to improve your Android experience?” Well, who doesn’t, but what I failed to realize is that if you turn that on (it is on by default) you end up sending pretty much every thing you do to Google: every app you open and how long you use it, every phone call you take, every text you send in addition to every link you visit. Turn it off and then your experience is greatly limited. For example, Google Maps won’t store your recent searches unless that feature is turned on. Recently I was in a private Google Hangout when the other person pasted a link. Although the link showed up normally in the chat window, the URL itself first went through Google when you clicked in it. Seriously? Google needs to track your activity down to the level of a link in a private Hangout?

But, Android is open source, unlike iOS, so for years I focused my mobile platform on Android but using alternative versions, often called “custom ROMs”.

Running custom ROMs is not for the faint of heart. Probably the most famous was CyanogenMod, but unfortunately that organization imploded spectacularly (but lives on in LineageOS). While I originally ran CyanogenMod, I found a really nice solution and community in OmniROM. In addition to the O/S, you need to install Google applications (GApps) separately, and projects like Open GApps let you control exactly what you install. I really liked that, and it worked well for awhile.

But there are two main issues with custom ROMs. The first is that almost all of them are volunteer organizations, thus the attention level of any one maintainer can vary greatly. They don’t have huge test organizations and the number of handsets supported can be limited. Find a good ROM with an active maintainer for your handset and you’re golden, but you can be up for a world of disappointment if not.

The second is that Google is getting more and more aggressive about having their applications run on these operating systems. Certain apps won’t run well (or run at all) if the underlying operating system isn’t “Google Approved”.

Thus I started running into problems. All of my older handsets are no longer being maintained (farewell Nexus 6) and OmniROM doesn’t support the Pixel (sailfish) or Pixel XL (marlin) which were released two years ago, so that option is out for me. I also like to play games like Pokémon Go, but it started behaving badly (or not running) on devices that weren’t vanilla Google.

I thought I had found a solution in CopperheadOS. This is (was) an organization out of Canada that made an extremely hardened version of Android. Unlike most custom ROMs where you replace the recovery partition or enable root access, Copperhead took the opposite approach and provided a very locked down, security conscious operating system. You would think this would be in opposition to free software, but it turns out their default software repository was F-Droid, which only features open source software, and in fact it was impossible to run the Google Play Store on the device (you allow Google the right to install any software they want without explicit permission when you use GApps and this went against the Copperhead philosophy).

This appealed to me, so I decided to try it out. I found I could do over 90% of what I needed to do without Google, and for things like Pokémon Go, I just got a second phone running stock Google (with a lot of the surveillance features turned off). So, my personal information lived on my Copperhead phone, and my “toy” phone let me do things like use Google Maps and call a Lyft.

Carrying two handsets wasn’t optimal, but I got used to it, and I found myself using the “Google” phone less and less. I loved the fact that security updates often hit my Copperhead phone a day or two before my Google phone, and I slept soundly knowing that my personal data was about as secure as I could make it (and still actually use a mobile device).

Then came June and the apparent demise of Copperhead (thanks Bryan Lunduke, for telling me about this and ruining my life, again). I needed to find another mobile solution.

About this time, privacy had come to the forefront with the impending implementation of the GDPR in Europe. The amount and level of surveillance being done by Google became even more transparent. There was a high profile study done in Norway that showed not only were companies like Google impacting your privacy, they were being pretty sneaky about it. The study also called out Facebook and Microsoft.

Surprisingly absent from that article was Apple. In fact, the news out of Apple-land was pretty positive. Due to the GDPR Apple made it possible for European users to download all of the tracking data Apple had on a given user and it was rather minuscule. Since Apple makes money on hardware, its business model makes it much more privacy friendly, even if it isn’t exactly a freetard’s best friend.

So I bought an iPhone.

A lot had changed in seven years. The iPhone is much more powerful but it is also a lot less intuitive. Even now I prefer the Android interface to iOS, but I didn’t find the transition too difficult.

No, the difficult part was trying to use the iPhone with Linux. While I found ways to mount the iPhone to my Linux desktop, you can’t manage music without iTunes, and iTunes doesn’t run natively on Linux.

(sigh)

Well, in for a penny, in for a pound. We had a spare 2017 13-inch Macbook Pro at the office, so I conscripted it to be my new laptop/desktop. Remember that the last Apple O/S I used regularly was Snow Leopard, so there was a second learning curve to climb.

Part of it was real easy. Free software on OSX has come a long way, so I simply installed Thunderbird, moved my profile over, and I was in business for e-mail. Similarly, Firefox was up and running with an install and a sync. The wonderful Homebrew project brought most of the rest of the stuff I needed to OSX land.

But I wasn’t super happy with the interface. I’ve tried a large number of desktop environments, and for my needs Cinnamon on Linux Mint works best. Little things about the OSX desktop just seemed to get in the way.

For example, I use a little tool called “onmsblink” that takes a ThingM blink1 USB dongle and changes its color based on the highest current alarm in my OpenNMS system. I launch it from the command line, but because it is Java it shows up in the dock and I can’t make it go away. Also, I’m used to clicking on an icon, say the Finder, and having a new window pop up. In OSX, it brings all open windows to the front, even if it is in another workspace. Is this “wrong” behavior? I don’t think so, but it is different for me and it interrupts my workflow.

Speaking of different, I’m also stuck with using a number of apps where I used to use one. I use the tool gscan2pdf constantly to scan in paper so I can shred and dispose of it. I have two scanners, a Brother ADS-3000N with the document feeder (works amazingly well under Linux) and a Canon LiDE 210 flatbed scanner. On OSX I ended up loading in two separate vendor-supplied applications to use them, and both of them feel really cluttered.

Plus, you would think an ecosystem like iOS would have a real mail client. One of the best mobile apps ever is K9 Mail, and I really miss it. I finally settled on Altamail, which has a yearly subscription but it was the only app that would easily handle nested folders. For example, I have a Customer folder with over 3000 subfolders. I can’t be scrolling through that on a mobile device. I don’t like it all that much, but it is the only option I could find.

Then there’s iTunes. Man, I used to think iTunes was a pig and now it is much, much worse. It took me longer than I would expect to get back to the interface I wanted (specifically, Songs with Browser View enabled). And, since I was playing around with a number of iTunes libraries, I ended up having to wipe the music off of my iPhone a couple of times since Apple won’t let you sync one devices to more than one library.

There are some good things about iTunes, I specifically like the way you can sync playlists, but I’ve been happier with my free music managers.

One app I really do like on OSX is iMessage. I am not a good typist on mobile devices, and being able to send and respond to a text from the desktop is awesome. And nobody comes close to making a trackpad that works as well as those on Apple laptops.

And thus I became an Apple laptop guy. Before I used two desktops, pretty much identical, with one at home and one at the office, with my laptop reserved for trips. Now I had to make sure I brought my laptop between both places (no laptop “drive of shame” so far). It was nice to have all of my information in one place, but the downside is that I did have all of my information in one place and it made the possible loss of my laptop that much worse.

I had resigned myself to being an Apple guy from here on out, but then I went on a business trip to Seattle where I used the laptop for several days and it was then I decided that I just couldn’t continue to use it.

The main issue that soured me was the keyboard. This was a 2017 model with one of those fancy “touch bar” thingies. Now everyone thinks that Apple is a great innovator, and in many cases they are, but the touch bar is something other companies have tried and discarded. I returned a Lenovo X1 Carbon laptop back in early 2014 that had one and they removed the feature from future editions. I use that top row of keys. I like having an escape key I can feel, and having real function keys is useful for things like games. Plus it is a lot easier to change the volume with an “up” or “down” key versus having to click on the volume icon and then use a slider.

But that wasn’t a deal breaker. When the “2” key started sticking, sometimes printing a character, sometimes printing many characters with one key press, and finally often not printing anything at all, I got discourage, nay depressed.

The issues with this generation of Apple keyboards are well known, but as I rarely use the keyboard on the laptop itself (I’m almost always connected to an external monitor and keyboard) I couldn’t believe it would get dirty enough to exhibit the issue that fast. Plus, the keyboard even when working just isn’t that good. I really miss the keyboard I had on my Powerbook.

This weekend when I got back home I decided to go back to Linux. I dragged my desktop out of the closet, booted it up, and decided to bring it up to date. During my hiatus a new version of Mint had been released, Mint 19, so I upgraded.

Man, that is one beautiful desktop. Seriously, I can’t remember using a nicer looking desktop environment on any platform. The tweaks the Mint team has made to Cinnamon have moved it from great to outstanding.

Please note that this is from my perspective. If you aren’t using Mint that doesn’t mean you suck or that your choices are wrong. The one thing I love most about the Linux desktop is that there exists a flavor for almost every taste and need.

It was as easy to move back to Mint from OSX as it was to move from it in the first place, so it has only cost me a few hours of time mainly waiting for the upgrade to download on my slow connection at home. I also installed a fresh copy on my fifth generation Dell XPS 13 and was pleasantly surprised at how much better the new trackpad driver, libinput works. That was the main complaint I had about my Linux laptop, and I’m eager to try it out when I am next on the road.

Moving back to Linux made me question my mobile O/S choice one more time. Searching around it looks like it is currently possible to run Pokémon Go on a custom ROM as long as it is not rooted, so I downloaded TWRP and LineageOS for my Pixel XL, as well as the “pico” version of Open GApps. I was thinking I could get back to, basically, my Copperhead environment with a minimal amount of Google and be happy.

Lineage Install Error

Bam, right out the door my phone started screaming about the phone driver not working. The memory of issues I experienced running alternative ROMs came flooding back, and I simply restored the Pixel to factory and decided to stay with my iPhone.

I feel much happier that I’ve gone back to Linux, at least part of the way. It should make it easier to go free on mobile as soon as the technology catches up. I’m very eagerly following the work of the /e/ foundation but as of yet they haven’t released any code. Plus it looks like they are going for an all-out Google replacement. I’m pretty happy running my own mail server and Nextcloud instances, so I’m more interested in a secure mobile device that can run apps from F-Droid versus a whole ecosystem replacement. Purism is also coming out with a phone. I really like the philosophy behind that company, but I’ve been stung by enough Kickstarters that I’m taking a wait and see attitude.

The problem with free and open source mobile will be the apps. As I mentioned, I was able to do 90% of what I needed using F-Droid, which bodes well for the /e/ solution but not so much for the Purism one, and both will faces challenges with adoption.

Until then, feel free to Facetime me and check out my growing collection of chins.

Dealing with Docker Interfaces

We run a lot of instances of OpenNMS (‘natch) and lately we’ve seen issues with disk space being used up faster than expected.

We tracked the issue down to Docker. If Docker is running on a machine, SNMP will discover a Docker interface, usually labelled “docker0”. When that instance is stopped and restarted, or another Docker instance is created, another interface will be created. This will create a lot of RRD files of limited usefulness, so here is how to address it.

First, we want to tell OpenNMS not to discover those interfaces in the first place. This is done using a “policy” in the foreign source definition for the devices in question. Here is what it looks like in the webUI:

Skip Docker Interfaces Policy

The “SNMP Interface Policy” will match on various fields in the snmpinterface table in the database, which includes ifDescr. The regular expression will match any ifDescr that starts with the string “docker” and it will not persist (add) it to the database. This policy has only one parameter, so either “Match All Parameters” or “Match Any Parameter” will work.

If you want to use the command line, or have a lot of custom foreign source definitions, you can paste this into the proper file:

   <policies>
      <policy name="Ignore Docker interfaces" class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy">
         <parameter key="action" value="DO_NOT_PERSIST"/>
         <parameter key="ifDescr" value="~^docker.*$"/>
         <parameter key="matchBehavior" value="ALL_PARAMETERS"/>
      </policy>
   </policies>

This will not deal with any existing interfaces, however. For that there are two steps: delete the interfaces from the database and delete them from the file system.

For the database, with OpenNMS stopped access PostgreSQL (usually with psql -U opennms opennms) and run:

delete from ipinterface where snmpinterfaceid in (select id from snmpinterface where snmpifdescr like 'docker%');

and restart OpenNMS.

For the filesystem, navigate to where your RRDs are stored (usually /opt/opennms/share/rrd/snmp) and run:

find . -type d -name "docker*" -exec rm -r {} \;

That should get rid of existing Docker interfaces, free up disk space and prevent new Docker interfaces from being discovered.