Blog

I went down a deep rabbit hole on the history of Amiga computers, and specifically the types of image encodings used in old Amiga software. As an enthusiast of digital preservation, I want to make sure that any images saved using that software can still be decoded and converted using modern-day tools.

What I found is a rich history of software developers squeezing every last bit of performance out of underpowered hardware and limited storage, using techniques that blur the line between laughably hacky and downright ingenious.

A good example is the so-called Extended HAM (HAM-E) graphics mode, which is basically designed to enable a higher color depth, i.e. more color bit planes, at the expense of horizontal resolution. However, the native format of Amiga bitmap images didn’t support this kind of color encoding in the metadata of the bitmap file, so how do we encode our color palette? The solution: put the palette information literally into the actual image!

That’s right, the color palette actually appears as several lines of seemingly “garbage” pixels at the top of the image, and a proper decoder needs to be aware of this hack. Below is an example image that is decoded incorrectly, i.e. using a decoder that reads the image literally, without taking the extra color information into account:

Notice in the above image that we can vaguely discern a stretched shape of a person’s face, and we can also discern a few lines of pixels at the top that don’t seem to go along with the rest of the image. Now here’s the same image decoded correctly, with the color palette lines consumed and blanked out:

But doesn’t that sound wrong? Even if a proper decoder recognizes and blanks out the garbage lines, won’t the bitmap always have unused lines at the top? Not to worry — those lines will fall into the overscan region of the monitor! (You know, because a CRT monitor has overscan margins.) Therefore the palette lines will be automatically hidden from view, unless the user explicitly adjusts the overscan on their monitor. Ah, the good old days.

There are a few modern software packages that handle these types of images and a plethora of others, including abydos, recoil, and grafx2, so the preservation of old Amiga image formats is in good shape. And in my own ImageFormats library for .NET, I implemented support for some even more obscure Amiga formats, notably ACBM (Amiga Contiguous BitMap), which seems to be unsupported by other decoders.

As the happy owners of several chickens, we’ve been finding that non-chicken-owners often have a comically flawed understanding of chicken anatomy and chicken reproduction. Specifically, people seem to assume that if a hen lays an egg, it implies that the egg contains an embryo, and definitely means that a chick will eventually hatch from the egg, unless the egg is taken from the hen and eaten before the embryo grows to a visible size. This assumption is maintained even when people are told that there’s no rooster in our flock, to which the response is “What does a rooster have to do with it?”

I remind my non-chicken-owning friends that a female human routinely produces eggs that fall out of her uterus. These eggs will never develop into a human embryo, unless they are fertilized by the human equivalent of a rooster.

---

I worked part-time while attending college (a decently mid-tier state school) in the early 2000s, and paid for the entirety of my tuition with my salary. I never took out any loans to pay for my education, and finished college without any debt. This is a choice that I made, and this choice is not for everyone. Despite this fact, I am overjoyed that many people will now have an easier time paying for college than I did, and I hold no resentment towards those people for whom the $10,000 loan forgiveness package will make a life-altering difference.

Thinking about it in the most abstract terms, the whole point of civilization is for the current generation to improve the quality of life for the next generation. Parents make sacrifices all the time to improve their children’s future, and good parents don’t resent their children just because their children’s lives are easier than theirs were.

The only minor annoyance I feel is that there’s still no attempt to fix the underlying root cause of the issue, which is twofold: 1) Colleges are experiencing unprecedented levels of administrative bloat, which is driving up the cost of education to the point where college is a luxury product that a seventeen-year-old is expected to purchase without thinking twice, and 2) We’ve created a culture which insists that going to college is the only way to succeed in life, when in fact there are a hundred different trades that are looking desperately for more people, and provide a very comfortable living and life satisfaction.

(Reminder: I perform a service of recovering data from old media such as magnetic tapes, floppies, weird flash memory formats, etc. Get in touch with anything you’d like to recover.)

Performed a very interesting data recovery case:
I received a number of QIC-150 tapes, and one QIC-525 tape, from a client in Tasmania (!) who made backups onto these tapes from an IBM AS/400 system in 1991. This presented a couple of challenges:

• The tapes were written with variable-length block sizes, so we need to configure the tape driver accordingly by running mt -f /dev/nst0 setblk 0.
• And then, when using dd to dump the tape contents, specify a sufficiently large block size to read the largest of the variable block sizes, which turned out to be 32760, so the command was dd … bs=32k.
• After the data was dumped successfully, I determined that the data was saved using the SAVLIB command on the AS/400, which serializes the data onto the tape in a proprietary format. Unfortunately I couldn’t find a lot of public documentation on this format, so I had to do a bit of reverse-engineering. My client specifically wanted to locate some COBOL source files in these archives, which turned out to be easy to pinpoint and extract. Most of the COBOL code was uncompressed, but some of the files were compressed using SCB (string control byte) encoding, which is basically a form of simple run-length compression.
• Finally, since the data came from an AS/400 system, the text contents of the files were EBCDIC-encoded, which required a conversion from EBCDIC to ASCII.

---

One of my first professional programming contracts was done in 1997, while I was still in high school. I built an MS-DOS application that communicated over a serial port with an industrial iron powder machine (i.e. a large apparatus that melts scrap iron and turns it into powder, to be recycled for other uses). This program would display real-time telemetry from the machine, and then output it to a daily log file. It was a very simple and rudimentary program, but it worked well enough that the company continued to use it for years. It was also enough to land me a full-time job at the same company, but that’s a subject for a longer post.

Although I still have the original executable file that I provided, the unfortunate thing is that I’ve lost the source code for this program. The only thing about it that I remember is that it was written in C, and compiled with Borland C++ v3. Therefore I’ve had a mini-quest in the back of my mind to either find the lost source code, or decompile the executable file and reconstruct the source code.

Recently I’ve come the closest I’ve gotten so far to reconstructing the code, all thanks to the excellent Reko decompiler. I opened the executable in Reko; it detected the Borland C v3 runtime effortlessly; it showed a list of function calls (which I recognized! finally!), and I was up and running navigating the disassembled functions:

The hurdle now will be to rewrite the source code based on the disassembly, so that it compiles as closely as possible to the original executable. There’s still a good amount of work left to do, but it’s now much more manageable because of Reko.

In case anyone’s interested, here is the program actually running in DosBox, albeit stuck in an “error” state because it’s not receiving any data from the aforementioned iron powder machine: