Brain dump, January 2022

There’s a HAM radio enthusiast living in Belarus (and previously Russia) who designed a telegraph keyer and named it after my grandfather, Lev Naumovich Pekler (Лев Наумович Пеклер), after meeting him briefly at the polar research base (Остров Голомянный) where he was stationed. Unfortunately this model is no longer in production, and has been superseded by newer designs, available from his website.


Our cat Bissel was featured as part of the “best cat breeds for apartment living” in The Spruce Pets.


Played around with the Lichee Pi Zero and the Lichee Nano boards, which are nifty and powerful little things. Here are links for getting started with the Zero, as well as SD card images for booting it. And here the documentation for the Nano, and how to get embedded Linux working on it. I simply developed a few random scripts that drew patterns on the framebuffer (i.e. a TFT screen attached to it). Quick tip: to disable the blinking cursor when drawing to the framebuffer:

echo 0 > /sys/class/graphics/fbcon/cursor_blink

Software update round-up

It’s high time to give some love to a few of my older and less-maintained software projects, and bring them up to date with a few much-needed and requested features!

DiskImager

DiskImager is a tool that I’ve used “internally” for a few years now to read and write raw disk images. I’ve simply never found the time to polish it up and make it production-ready, until now. This is a small standalone tool that will dump the contents of any drive connected to your PC to a file on another (larger) drive. It can also write a disk image file to a physical disk. Furthermore, when selecting a disk image to write to a physical disk, you can choose from several types of image formats (besides raw images) including VDI, VMDK, VHD, and E01.

FileSystemAnalyzer

The FileSystemAnalyzer tool has gotten a huge number of bug fixes, as well as these enhancements:

  • Improved compatibility with FAT, exFAT, NTFS, ext4, and UDF filesystems in various states of corruption.
  • Improved previews and metadata for more file types.
  • The main file tree view now has columns with file size and date, similar to Windows Explorer. These columns are clickable to sort the file list by ascending or descending order of the column type.
  • Directories can now be saved from the file tree view (recursively), in addition to individual files.

Outlook PST viewer

The PST viewer tool has been updated to be more compatible with a wider range of PST files from different versions of Outlook, and to be more forgiving of corrupted PST files. There is also a new option to save individual messages as .MSG files.

Minimalist programming, Android edition

Suppose you open Android Studio and create a new project from a template, say, a blank Activity with a button. Then build the project, and look at the APK file that is produced. You’ll notice that the APK is over 3 MB in size. These days we don’t bat an eye at these kinds of numbers, and indeed this size is pretty modest in the grand scheme of today’s software ecosystem. However, objectively speaking, 3 MB is a lot! Let’s take a deep dive into what these 3 MB actually consist of, and see how much we can reduce that size while maintaining the same functionality.

The app I’ve built for this exercise is slightly more complex than “hello world”; it’s an app that could actually be minimally useful: a simple tip calculator.

It’s literally a single input field (an EditText component) where the user enters a number, followed by a few lines of text that tell the user different percentages of that number — 15%, 18%, and 20%, which are the most common tipping percentages in the U.S.

Once again, with the default project settings generated by Android Studio, this app comes out to about 3.2 MB when it’s built. Let’s examine the generated APK file and see what’s taking up all that space:

Right away we can see that the heaviest dependency by far is the AndroidX library, followed by the Kotlin standard library and the Material library (under the com.google package). In fact the code that actually belongs to our package (com.dmitrybrant.tipcalc) is a mere 76 KB, dwarfed by the library dependencies that it’s referencing.

To be fair it’s possible to reduce the size of the APK by a good amount by using the minifyEnabled directive, which is not enabled by default. In fact it would probably optimize away most of the “kotlin” dependency and much of the “androidx” dependency. However, even with minifyEnabled our APK size would still be on the order of megabytes. For the purpose of this exercise I left minifyEnabled off, so that we can see exactly which packages are contributing to the code sizes in our APK.

In any case, let’s start whittling away at this extra weight, and see how lean we can get.

The Kotlin tax

As we can see, merely using Kotlin in our app causes the Kotlin standard library to get bundled into the APK. If we don’t want this library to get bundled, we must no longer use Kotlin. (Although I’ll repeat that if we use minifyEnabled, then Kotlin would pretty much be optimized away, so this is more of an observation than a “tax.”)

After converting the code to plain Java and rebuilding the app, our APK is now 2.7 MB:

That’s a little better! But now can we remove the bulkiest dependency, namely the androidx library?

The AndroidX tax

AndroidX is a fabulous library that ensures your app will run consistently on a huge range of different devices (but not all of them!) and different versions of the Android OS. It makes perfect sense that AndroidX is used by default for new projects, and I’m not saying that you should reject it when building your next app. Buuuut… could we actually get away with not using it? How would our app look and run without it? And would our app still run on the same range of devices?

Getting rid of AndroidX means that our app will rely solely on the SDK libraries that are part of the operating system on the user’s device itself. To get rid of AndroidX in our project, we need to do the following:

  • Our Activity can no longer inherit from AppCompatActivity, and will now simply inherit from the standard Activity class from the SDK.
  • We can no longer use fancy things like ConstraintLayout, and will be limited to using basic components like LinearLayout.
  • Our theme definitions can no longer inherit from predefined Material themes. We will need to apply any color and style overrides ourselves.

After these modifications are all done, here’s what our APK looks like:

That’s right, you’re not dreaming, the app is now 88 KB. That’s kilobytes! Now we’re getting somewhere. And if we look closely at those numbers, we see that the bulk of the size is now taken up by the resources that are bundled in the app. What are those resources, you ask?

Launcher icons

By default Android Studio generates a launcher icon for our app that takes several forms: a mipmap resource, which is a series of PNG files at different scales, which will be chosen by the launcher to match the pixel density and resolution of the device, and also a vector resource that will be used instead of the mipmap on newer devices (Android 8 and higher).

This is all very useful stuff if you need your launcher icon to appear pixel-perfect across all devices. But since our goal is minimalism, we can dispense with all of these things, and instead use just a single PNG file as our launcher icon. I created a 32×32 icon and saved it as a 4-bit PNG file, making it take up a total of 236 bytes. It doesn’t look perfect, but it gets the point across:

So, after getting rid of all that extraneous baggage, how are we looking now?

We’ve arrived at 10.5 KB! This is more like it. It may be possible to squeeze it down even further, but that would necessitate doing even more hacky and inconvenient things, such as removing all XML resources and creating layouts programmatically in our code. While I’m going for minimalism, I do still want the app to be straightforward to develop further, so I’m happy to make this a good stopping point.

This is definitely closer to the size that I would “expect” a tip calculator app for a mobile device to be. Speaking of mobile devices, which devices will this app be able to run on?

Compatibility

By default Android Studio sets our minimum SDK to 21, making our app compatible with Lollipop and above. There are plenty of good reasons to set your minimum SDK to 21, but now that we’ve removed our dependency on AndroidX, as well as our dependency on vector graphics, there’s nothing stopping us from reducing our minimum SDK even lower. How much lower? How about… 1? That’s right, we can set our minimum SDK version to 1. This would make our app compatible with literally every Android device ever made.

I don’t own any devices that actually run Android 1.0, but here is my Tip Calculator app running on the oldest device I own, a Samsung Galaxy Ace from 2011, running Android 2.3 (API 9):

And here is the same app running on my current personal phone in 2021, a Google Pixel 3 XL running Android 11 (API 30):

Takeaways

Aside from being an interesting random exercise, there’s a point I hope to convey here:

As time goes by, software seems to be getting more and more bloated. I believe this might be because developers aren’t always cognizant of the cost of the dependencies they’re using in their projects, whether it’s third-party libraries that provide some kind of convenience over standard functionality, or even the standard libraries of their chosen programming environment that the developer has gotten used to relying upon.

As with anything in life, there should be a balance here — a balance between convenience offered by libraries that might add bloat, and lower-level optimization and active reduction of bloat. However, it feels like this balance is currently not in a healthy place. The overwhelming emphasis seems to be on convenience and abstraction ad infinitum, and virtually no emphasis on stepping back and taking account of the costs that these conveniences incur.

Android is far from the worst offender in the world of bloat, and even though a 3 MB binary may be totally acceptable, it doesn’t have to be that way. Even though bulky standard libraries should be used in the majority of cases, they don’t need to be used all the time, and there may even be cases where the app would benefit from not using them. If only developers would maintain a better sense of how their dependencies are impacting the size of their apps, or indeed what dependencies they’re even using in the first place, we can begin to restore the balance of bloat in our lives.

The lost art of system-level thinking

Our house has steam heat – you know, those old cast iron radiators that are heated by a boiler that sits in the basement. When we first moved in, I was skeptical of how well steam heat actually works, and I began to dread the idea (and the cost!) of having to replace the whole thing with modern forced-air heat. And when we powered on the heat for the first time, my suspicions were confirmed: some of the radiators were making strange noises, and sometimes the pipes themselves would make loud bangs that reverberate through the whole house.

But now, after just a few months, nearly all of these issues are solved, and I am a believer in steam heat and its many benefits.   This is all thanks to a book called The Lost Art of Steam Heating by Dan Holohan.   The book allowed me to solve our heating issues after roughly 20 pages (the pressuretrol was set way too high, if you’re reading, Dan), but I couldn’t put it down and kept reading until the end, for an entirely different reason: the book does a great job of emphasizing system-level thinking, as opposed to narrow immediate problem solving. Even though the book is intended more for HVAC contractors than individual homeowners, I believe that any serious engineer would find the stories and advice relatable.

With every heating “war story” that Holohan recounts, the solution invariably relies on widening one’s perspective beyond the single component that seems to be broken. By the end, I had forgotten that I was even reading about steam heat, but was simply enthralled by Dan’s ability to re-frame the problem from the highest level down. I wish that more engineers in my own field would do the same.      

Problems with steam heat turn out to be easy to diagnose.   It’s simple physics, if at times slightly counterintuitive. The harder lesson is to always think about the big picture. Don’t just replace a faulty component with a new one; think about how the new component affects the rest of the system. Don’t remove a component that you think is unnecessary without understanding why it was there to begin with. And so on.

The lost art, it would seem, is not really steam heat; the lost art is big-picture thinking itself.

“Think about the system.”   Well said, Dan.

Discovering little worlds

Like so many other people during the COVID lockdown, I’ve been looking for additional hobbies that could be done from home, which would occupy my time and help keep my mind off the collapse of civilization as we know it, and maybe even ground my thoughts and keep them away from hyperbole and catastrophizing.

While cleaning and organizing my basement I came across this small USB device. It’s barely larger than a flash drive, but it’s no flash drive at all – it’s a software-defined radio (SDR), namely an RTL-SDR V3 dongle.

I don’t even recall how this device ended up in my possession;  I think it was probably from one of my previous jobs:  they were throwing away a bunch of equipment that was no longer useful, and allowed me to keep some of the items.  Regardless, I had never actually used the SDR, and hadn’t really thought about what the SDR would be useful for.  I had a vague understanding that the SDR can let me tune in to any random radio frequency, but how interesting can that be?  Well, it turns out that playing around with this device led me into a rabbit hole of epic proportions.

Once I found the right software to work with the SDR device (SDRSharp for Windows, and CubicSDR for Mac), I was up and running.  The first rather trivial thing to do was to tune in to the local FM radio stations. Here they are, as viewed through a spectrogram:

FM radio

But that’s kind of boring. I wonder what lies outside of the FM radio band? Well, the next obvious destination is the local police frequencies, which are around 460 MHz to 490 MHz in my area. These are narrow-band FM (NFM) stations, so we adjust our software settings accordingly. In mere moments, I’m listening to police dispatchers communicating with units and telling them about robberies, car accidents, and the like:

And of course there are a few local HAM radio repeaters nearby, which tells me that the HAM community is very much alive and well.  Since I can’t transmit anything using my tiny SDR device, I can only listen in on the HAM conversations, but that’s okay since the conversation wasn’t particularly scintillating anyway, and I’m not sure that getting into the little world of HAM radio is really my goal here. As much as I salute the enthusiasts who keep HAM radio going, they can party on without me.

Mind you, all of this was using the cheap tiny antenna that came with the SDR itself.  But then I discovered that the SDR can be used for something else entirely: receiving signals from satellites!

Arguably the easiest satellites to pick up signals are the NOAA 15/18/19 satellites, which are weather satellites that transmit images of cloud cover over the ground. By “easiest” I mean requiring the least amount of additional equipment:  it only requires a rabbit-ear (V-dipole) antenna connected to your SDR, and a cloud-free day to get a good signal. Here is the signal at 137.9 MHz, and the resulting image, which is produced by special software that demodulates the “audio” data that was recorded:

The downside is that these satellites are in a sun-synchronous orbit, and will only pass by your location for ~15-minute intervals at the most, and can only be caught at very early or late hours of the day. The other downside is that the NOAA satellites are aging, and will probably be decommissioned in the coming years. And anyway, the images they transmit are not the highest quality. Time to step it up to the next level, namely the GOES satellites!

The GOES-16 satellite is a newer weather satellite that is geostationary, and is positioned permanently above the Americas. In fact its longitude is almost exactly over the East coast, which is perfect for my purposes, and its inclination from my location is about 45 degrees (because it orbits around the equator, as all geostationary satellites must do).

But because it’s geostationary it’s also much farther away, and therefore its signal is much weaker, and requires additional equipment:

The setup consists of an old WiFi grid antenna, which feeds into a SAW filter and amplifier, which then feeds into the SDR that’s now connected to a Raspberry Pi (the total cost was about $100).  The Raspberry Pi is running a package called goestools which demodulates the signal from the satellite in real time, and translates the signal into images. The satellite transmits images of Earth in many different spectral bands, ranging from visible light to deep infrared.

And so, the final little world I discovered on this adventure is this one:

The full resolution of these images is 10K, which is mind-blowing, and my next step is to create animations from these images, which are sent by the satellite every 15 minutes.

I think I’ll leave this antenna setup as a permanent installation in my house, so that I can grab these signals anytime I like. Even though this imagery is available on the web if you know where to look, there is something profoundly awesome in knowing that you personally can receive selfies of our world, from 35,000 kilometers away, using about $100 worth of equipment. It’s been a very satisfying few weeks, in spite of everything else that’s happening this year.