Author: Dmitry Brant

Reverse-engineering the QICStream tape backup format

TLDR: I developed an open-source tool to read tape backup images that were made using the QICStream tool, and extract the original files from them.

During a recent data recovery contract, I needed to recover files from some old QIC tapes. However, after reading the raw data from the tapes, I couldn’t recognize the format in which the backup was encoded, and none of the usual software I use to read the backups seemed to be compatible with it.

After briefly examining the backup in a hex editor, it was evident that the backup was fortunately not compressed or encrypted, and there were signs that the backup was made using a tool called QICStream.  There doesn’t seem to be any documentation regarding this utility (or the format of the backup it saves) on the web. It’s easy enough to find the tool itself on ancient DOS download sites, and it may have been an interesting project to create an emulated DOS environment where the QICStream tool reads the backup from an emulated tape media, but it turned out to be much easier to reverse-engineer the backup structure and decode the files from the actual raw data.

The binary format of the backup is very simple, once you realize one important thing:  every block of 0x8000 bytes ends with 0x402 bytes of extra data (that’s right, 0x402 bytes, not 0x400). In other words, for every successive block of 0x8000 bytes, only the first 0x7BFE bytes are useful data, and the last 0x402 bytes are some kind of additional data, possibly for parity checking or some other form of error-correcting logic. (I did not reverse-engineer the true purpose of these bytes; they did not turn out to be important in the end.)

Other than that, the format is very straightforward, and basically consists of a sequence of files and directories arranged one after the other, with a short header in front of each file, and “control codes” that determine whether to descend into a subdirectory or to navigate back out of it.

Anyway, I put all of these findings into a small open-source utility that we can now use to extract the original files from QICStream backups. Feel free to look through my code for additional details of the structure of the file headers, and how the large-scale structure of the backup is handled.

More data recovery war stories: recovering QIC-80 tape backups

I’ve got another recent data recovery job that is worth mentioning! (Quick reminder: I offer a service to read data from super-old media such as tapes and floppies)  This particular client had a number of QIC-80 tapes from the mid-1990s that contained backups of a desktop workstation.

To read QIC-80 tapes, I have an ancient Colorado 250MB drive which I acquired a while ago, but here’s the problem: this drive connects to the floppy disk controller on the motherboard, which means that it must need very specific drivers or software to communicate with the drive.

In Linux, the way to communicate with such drives is using the ftape driver, which used to be included with the Linux kernel. However, ftape was removed from the kernel in 2006, citing too many bugs and too few users.

Since the tape’s label indicated that it came from a PC system, I decided to try keep it simple, and to recreate an actual DOS PC with this drive attached to the floppy controller. All I would need is to find the software that might communicate with this drive properly. After some serious dredging of old internet forums, I found a download site that contains the oldest versions of the Colorado backup software.

Setting up the boot disk

As I’ve done in the past, I used bochs to create a generously-sized disk image of 1 GB, and installed MS-DOS 6.22 onto it within the emulated environment of bochs. Then I copied over the installation files for the Colorado backup software. (I planned to do the actual installing of the Colorado software on the live PC instead of the emulated PC, since it might do some of the hardware detection during setup.)

Once the emulated disk image was complete, I wrote it onto a USB flash drive, plugged the drive into the old PC with the tape drive attached, and booted from the USB drive. (Thankfully the old PC has a very versatile motherboard that can boot from pretty much anything.)

The old PC booted successfully into MS-DOS, and I proceeded to set up the Colorado backup software. This did not present any issues, and the setup completed successfully without any non-default configuration.

Restoring

When I ran the backup software, it went through a first-time setup process where the first sign of hope appeared: the software said that the tape drive was detected!

The next step was to rebuild the backup catalog from the tape, while praying that the catalog is compatible with this version of the backup software.  And what do you know – the catalog rebuilt successfully, and I could see the directory tree of the backup. The final step is to perform the actual restoring of the files, which I did directly onto the current boot disk.

There was not a single hiccup during the actual reading of the tape. I continue to be amazed by the resiliency of tape backup media, as well as the durability of the drive hardware, which still works flawlessly after 25 years.

This tape was a full backup of a PC workstation in 1996, so the final “bonus” step is to boot into the backup within an emulated environment, and see this workstation running in all its glory:


There are people alive today…

…who were born before there was a single artificial satellite in space.

…who were born before there was commercial air travel.

…who were born before plastic was invented.

…who were born before DNA was discovered.

…who were born before a single bit of digital information was stored anywhere.

We’ve made such a staggering amount of progress in a single human lifetime, and this progress continues to accelerate. This includes technologies with which we haven’t even evolved to cope, such as social media. Are we sure that we should continue progressing at this pace without taking a breath and taking the time to understand how these things are impacting our society, our psyche, and our future?

Chinese intellectual property theft hits home

Rampant piracy is pretty much a “feature” of publishing software on the internet, and any author of a semi-popular app will be well-acquainted with it. Not a day goes by when I don’t see pirated copies of my app shared via Google Drive, Dropbox, and all kinds of shadier file sharing sites.

But now I’ve discovered an even more disturbing dimension to this seediness. The problem with Android apps (and any Java-based software, really) is that it’s very easy to reverse-engineer, even when the code is obfuscated. Given enough time and manpower, it’s possible to recreate nearly the original source code from the compiled app. It’s analogous to reconstructing a shredded piece of paper – it simply takes some time to find all the strips and glue them together.

So I recently received a communication from a Chinese user who alerted me that this is precisely what’s happening in China. My app is being deconstructed and repackaged under a different name. But worst of all, they have inserted their own payment mechanism into the app, which requires the user to submit a payment before the app can be used!  According to my Chinese whistle blower, this counterfeit version of DiskDigger is being used by tech support departments that send the app to the user, make them install it, and then wait for payment before assisting the user further.

   

I generally turn a blind eye to most of the piracy of my apps, since it’s humanly impossible to continually track down violators, but this new kind of perversion makes me feel truly powerless. Without having any legal representation in China, there is literally nothing I can do to combat these bad actors. Perhaps it’s time to research how to establish a legal presence in China. If you have any experience with this, feel free to contact me. And in the meantime, when looking for DiskDigger, insist on the original!

When you need some perspective

Whenever I get too caught up in reading the news, or sucked into hopelessly unproductive political discussions, I like to relax with some images from the Hubble Space Telescope and similar telescopic wonders, which give us a glimpse into the wider universe outside the minuscule mote of dust on which we live.

Take, for example, the Ultra Deep Field image, which contains roughly ten thousand galaxies within an area of the sky that is about 2×2 arcminutes. Do you know how small a portion of the sky that is? If you take a 1×1 millimeter square, hold it at arm’s length, and peer through it, that’s the area of the sky captured in the Ultra Deep Field.  The UDF was taken in a deliberately ordinary and uninteresting portion of the sky, which implies that for every 1x1mm portion of the sky, we could get a similarly breathtaking image of thousands of galaxies. Nearly every bright spot in the image is a galaxy, and some of the galaxies are over 12 billion years old.

How about some images that are impressive not in their depth, but in their resolution? Here is the sharpest ever image of the Andromeda galaxy (a composite of numerous other images), where we can zoom in to see literally every individual star in a galaxy outside of our own. If you look closely, you can also find globular clusters and nebulas, all in a different galaxy. There is a similarly high-resolution image of the Triangulum galaxy, as well as a super high-resolution image of the Orion nebula.

Or how about some images of gravitational lenses, where gargantuan clusters of galaxies create such strong curvature in spacetime that the light from more distant galaxies bends around them and becomes distorted, or even splits into multiple images of the same galaxies in different spots.

And of course there’s the recent image of an actual black hole (or rather an image of matter accreting around it) at the center of Messier 87, an extremely massive galaxy with a central black hole of 5 billion solar masses, making it easier to observe than the black hole in our own galaxy.

via Wikimedia Commons

All of this provides a pleasant counterbalance to our daily political bickering, which all seems laughably “local” by contrast.  To channel Christopher Hitchens for a moment, take some time to let these awe-inspiring images sink in, and then compare them to the story of Moses and his “burning bush.”