Author: Dmitry Brant

Ulam’s spiral in your browser

My day-to-day work is focused mostly on Android and Windows development, so I often find myself a bit disconnected from web development. I thought I’d go through a few random exercises in JavaScript, and simultaneously bring some of my oldie-but-goodie projects “up to date,” as it were. A long time ago I made a Windows application that displays the Ulam prime number spiral, but there’s no reason it can’t be done in the browser today, so here we go:

Zoom:
Highlight twin primes
Highlight Mersenne primes

The above picture is dynamically generated in your browser. Go ahead and interact with it: you can use your mouse scroll wheel to zoom in and out, and use the checkboxes to highlight certain special types of primes.

Send me all your floppies! They nourish me.

Recently I had another data recovery case that involved a comically large number of floppy disks, as in… more than five hundred (split evenly between 3.5” and 5.25” disks). We’re talking several large USPS boxes packed to the brim with floppies.

Of the numerous 3.5” floppies, only about 10% had one or more bad sectors, and none of them were completely unreadable.  The same was true for the 5.25” floppies, even though some of them were physically bent or warped, to the point where I had to cut them open and transplant the disk itself into a new container.  Some of the oldest files on these disks dated all the way back to 1986!

The recovery was performed using two older PCs, each of which have both 3.5” and 5.25” internal floppy drives, allowing the reading to be done somewhat in parallel.

There are actually plenty of cheapo floppy drives that connect over USB that can be purchased even now for as little as $15, but these drives are not, I repeat not suitable for recovering data from actual old floppy disks.  They must be read by a proper original floppy drive, preferably from the same era as the disks themselves.

Anyway, when floppy disks were in widespread use in the 1980s and 1990s, they weren’t really intended or marketed as a long-term storage solution, but they’re proving to be quite resilient as time goes by.  I’m not nearly as optimistic that today’s USB flash drives or SD cards will be readable in 30 years.

To be fair, these old disks have a much lower data density than modern storage media, so it makes sense that they would be more resilient to wear and tear. But still, it’s impressive that even what seems like mediocre-quality floppy disks still hold up to this day.

Despite these excellent outcomes, this still underscores how important it is to recover this data now, rather than waiting any longer and risking these disks developing any more bad sectors. So, let this be a call to action: if you have any old floppies lying around (or old tapes, Zip disks, Jaz disks, or anything else!), contact me for details on how to send them over, and I’ll recover the data from them for a fraction of the cost of other companies.

Reverse-engineering the QICStream tape backup format

TLDR: I developed an open-source tool to read tape backup images that were made using the QICStream tool, and extract the original files from them.

During a recent data recovery contract, I needed to recover files from some old QIC tapes. However, after reading the raw data from the tapes, I couldn’t recognize the format in which the backup was encoded, and none of the usual software I use to read the backups seemed to be compatible with it.

After briefly examining the backup in a hex editor, it was evident that the backup was fortunately not compressed or encrypted, and there were signs that the backup was made using a tool called QICStream.  There doesn’t seem to be any documentation regarding this utility (or the format of the backup it saves) on the web. It’s easy enough to find the tool itself on ancient DOS download sites, and it may have been an interesting project to create an emulated DOS environment where the QICStream tool reads the backup from an emulated tape media, but it turned out to be much easier to reverse-engineer the backup structure and decode the files from the actual raw data.

The binary format of the backup is very simple, once you realize one important thing:  every block of 0x8000 bytes ends with 0x402 bytes of extra data (that’s right, 0x402 bytes, not 0x400). In other words, for every successive block of 0x8000 bytes, only the first 0x7BFE bytes are useful data, and the last 0x402 bytes are some kind of additional data, possibly for parity checking or some other form of error-correcting logic. (I did not reverse-engineer the true purpose of these bytes; they did not turn out to be important in the end.)

Other than that, the format is very straightforward, and basically consists of a sequence of files and directories arranged one after the other, with a short header in front of each file, and “control codes” that determine whether to descend into a subdirectory or to navigate back out of it.

Anyway, I put all of these findings into a small open-source utility that we can now use to extract the original files from QICStream backups. Feel free to look through my code for additional details of the structure of the file headers, and how the large-scale structure of the backup is handled.

More data recovery war stories: recovering QIC-80 tape backups

I’ve got another recent data recovery job that is worth mentioning! (Quick reminder: I offer a service to read data from super-old media such as tapes and floppies)  This particular client had a number of QIC-80 tapes from the mid-1990s that contained backups of a desktop workstation.

To read QIC-80 tapes, I have an ancient Colorado 250MB drive which I acquired a while ago, but here’s the problem: this drive connects to the floppy disk controller on the motherboard, which means that it must need very specific drivers or software to communicate with the drive.

In Linux, the way to communicate with such drives is using the ftape driver, which used to be included with the Linux kernel. However, ftape was removed from the kernel in 2006, citing too many bugs and too few users.

Since the tape’s label indicated that it came from a PC system, I decided to try keep it simple, and to recreate an actual DOS PC with this drive attached to the floppy controller. All I would need is to find the software that might communicate with this drive properly. After some serious dredging of old internet forums, I found a download site that contains the oldest versions of the Colorado backup software.

Setting up the boot disk

As I’ve done in the past, I used bochs to create a generously-sized disk image of 1 GB, and installed MS-DOS 6.22 onto it within the emulated environment of bochs. Then I copied over the installation files for the Colorado backup software. (I planned to do the actual installing of the Colorado software on the live PC instead of the emulated PC, since it might do some of the hardware detection during setup.)

Once the emulated disk image was complete, I wrote it onto a USB flash drive, plugged the drive into the old PC with the tape drive attached, and booted from the USB drive. (Thankfully the old PC has a very versatile motherboard that can boot from pretty much anything.)

The old PC booted successfully into MS-DOS, and I proceeded to set up the Colorado backup software. This did not present any issues, and the setup completed successfully without any non-default configuration.

Restoring

When I ran the backup software, it went through a first-time setup process where the first sign of hope appeared: the software said that the tape drive was detected!

The next step was to rebuild the backup catalog from the tape, while praying that the catalog is compatible with this version of the backup software.  And what do you know – the catalog rebuilt successfully, and I could see the directory tree of the backup. The final step is to perform the actual restoring of the files, which I did directly onto the current boot disk.

There was not a single hiccup during the actual reading of the tape. I continue to be amazed by the resiliency of tape backup media, as well as the durability of the drive hardware, which still works flawlessly after 25 years.

This tape was a full backup of a PC workstation in 1996, so the final “bonus” step is to boot into the backup within an emulated environment, and see this workstation running in all its glory:


There are people alive today…

…who were born before there was a single artificial satellite in space.

…who were born before there was commercial air travel.

…who were born before plastic was invented.

…who were born before DNA was discovered.

…who were born before a single bit of digital information was stored anywhere.

We’ve made such a staggering amount of progress in a single human lifetime, and this progress continues to accelerate. This includes technologies with which we haven’t even evolved to cope, such as social media. Are we sure that we should continue progressing at this pace without taking a breath and taking the time to understand how these things are impacting our society, our psyche, and our future?