Data recovery war stories: more fun with QIC-150 backup tapes

In case you’re not aware, I offer a service for recovering data from old or obsolete storage media, such as magnetic tapes, Zip drives, floppies, etc. One of my recent cases, from a rather prominent client, was interesting enough to warrant a detailed write-up. The task was to recover a backup from a QIC-150 tape. I’m no stranger to recovering data from these tapes, so I thought that this case would be fairly routine. However, it turned out to be a bit more involved, in a number of fascinating ways.

First was the matter of hardware: my go-to drive for reading the tapes was my Archive Viper 2150S. However, when I unpackaged it from its static bag, I noticed a black goo all over it. It turned out that the rubber wheel attached to the capstan that drives the tape reels had completely melted! I must have stored the tape drive in excess heat, causing the rubber to melt, and rendering the drive unusable. I proceeded by replacing the drive with a Tandberg TDC 3600, a mere $50 on eBay. (Lesson learned: store tape drives in a cool, dry place!)

20160724183424

Anyway, I proceeded to attach the tape drive to my usual SCSI setup, which includes a Adaptec AHA 2940UW host adapter, which had served me well in the past, and proceeded to boot into the latest version of xubuntu (16.04), thinking that all I’d need to do is dump the tape contents by calling:

$ sudo mt -f /dev/nst0 rewind
$ sudo dd if=/dev/nst0 of=tape.bin

The above commands worked, but it only retrieved about 20MB of data, whereas my client was certain that the backup was at least 100MB (and the tape capacity is 250MB). I quickly realized that the tape contains multiple “files” worth of data. Even though tapes do not have a file system of their own, they do indeed have the notion of “files” stored sequentially along the tape stream, sometimes referred to as “records”. This tape turned out to have five of these files, each 20MB in size, totaling just over 100MB as expected. Tip: to continue reading the “next” file on the tape once the previous file is finished, it suffices to simply execute the same dd command above, with a different output file name. It’s also possible to fast-forward or rewind to any file location by calling:

Go backward one record:
$ sudo mt -f /dev/nst0 bsfm 1
Go forward one record:
$ sudo mt -f /dev/nst0 fsf 1

Having the tape contents dumped, next came the task of identifying the format of the data contained in the dumps. My client initially thought that the tapes were Unix backups saved with the tar command, which would have been convenient, but this turned out not to be the case. The binary format of the dumps was not any format that I recognized. Fortunately, after the client looked at the binary data, he guessed that the backups may have been from an old Macintosh computer, and that the backup software was Retrospect.

My thinking then was to set up an emulated Mac environment, install an old version of Retrospect on it, and try using it to “restore” the data from the binary dumps. The most robust emulator of old Mac architecture is Basilisk II, and fortunately I already had a basic pre-made emulated environment running System 7.5. I then managed to find an ancient version of Retrospect on an abandonware website, and installed it in the emulated system. However, when trying to restore data from the backup — no luck. Retrospect was actually able to read the internal name of the backup (which means that it definitely came from Retrospect), but couldn’t read its catalog contents. It was becoming clear that Retrospect must have had a special proprietary format for saving the data onto a tape. If only there was a way to “trick” the software into thinking that it was reading the backup directly from a tape drive…

After some more researching and digging, I found that Basilisk II has an extremely powerful option that makes it able to have pass-through access to the physical SCSI bus on the host machine! All it needs is to add this line to its initialization file (.basilisk_ii_prefs):

scsi0 /dev/sg2

In the above line, /dev/sg2 points to my actual tape drive, represented as a generic SCSI device in Linux. Amazingly (almost miraculously), this worked, to the point of the emulated Retrospect being able to detect the SCSI tape drive, and read the catalog contents of the backup!

Screenshot_2016-07-21_21-39-31

However, next came the final hurdle. Although it was able to read the catalog, when I asked it to actually restore the files, it did nothing. It just sat there indefinitely. It was so close! It was literally showing me the contents of the backup, but it wouldn’t budge on actually recovering them. Is it a faulty tape drive? Is it an incompatibility between the emulated SCSI driver and the actual hardware? Is it a bug in the backup software?

I continued by recompiling Basilisk II with debug options turned on, particularly enabling debug output in the SCSI modules, since I wanted to see exactly what kind of SCSI commands the emulator was trying to send. All of the I/O was looking normal, until I made the request to restore the backup contents (which was making it hang). The emulator requested to read a block of 6MB of data from the tape. I thought, “that’s a rather large amount.” Does the emulator actually have enough RAM to hold that much data? Yes indeed, in the emulator’s configuration, I was giving it 32MB of RAM.

And then, it finally dawned on me: does the tape drive actually support returning that much data at once? Probably not… but how do I reduce the amount of data that the backup software asks for? The answer: give the emulator less RAM, which will constrain the maximum buffer size that it can request! I tried giving it 4MB, but this was too little to allow the backup software to read the catalog contents into memory. I bumped it to 6MB and… everything worked. I watched in amazement (and relief) as the restoration was really happening — emulated backup software was communicating with a real SCSI device!

Screenshot_2016-07-24_14-05-31

All that was left was to copy the restored files out of the emulator and into my “real” PC, which is easy since Basilisk II can actually mount a Linux directory as a drive within the emulated Mac. That’s it!

Screenshot_2016-07-24_14-26-03

During this experience, what impressed me the most was the quality of the Mac emulator (Basilisk II), and how powerful its features actually are. The author of this software deserves huge kudos for creating it, and making it available for everyone. And of course, this is further proof that my obsession with emulators sometimes pays off!

Share this article:
  • Facebook
  • LinkedIn
  • Reddit
  • Twitter

The state of emulation on Android

For whatever reason, I have a strange fascination with emulators, and making sure that I can emulate as many machines and game consoles as possible on my current PC.

The one thing I haven’t researched until now is emulating various systems on my Android device. Now that I’ve received my shiny new Galaxy S7, I figured it must be powerful enough to emulate a good fraction of older hardware. And indeed, I was pleasantly surprised by the “state” of emulation on Android.  Emulators for NES and Super NES are readily available on Google Play, along with emulators for Atari and Commodore 64, and hardly require the latest hardware to run well.

But what about emulators of actual PC hardware? Well, there’s the excellent DosBox Turbo app, which is an optimized DosBox port to Android. This is not a free app, but it’s well worth it. However, what I was looking for is a true PC emulator, such as the great bochs emulator that I’ve trusted for a long time to boot into my disk images of Windows 3.1 and Windows 95. After a bit of looking, it turns out that this is available, too!

Thanks to some fantastic work by Pelya (for porting the SDL 1.2 library to Android, thus making it easy to port any SDL-based application to Android) and by Lyubomyr (for actually adapting bochs to use pelya’s SDL port), we now have a perfectly working version of bochs for Android! Using their build instructions, I was able to build my own bochs APK, load it onto my device, along with a few disk images. And after a few configuration tweaks, I’m now able to run my emulated images nearly flawlessly:

Here it is, running Windows 3.11:
20160320103603

…and Windows 95:
20160320103424

…and Windows XP:
20160320103213

And here it is running one of my favorite old DOS games, Return to Zork!
20160320103902

Share this article:
  • Facebook
  • LinkedIn
  • Reddit
  • Twitter

Planetarium app for Samsung Gear VR!

As a fun side-project during the last few weeks, I’ve been developing a planetarium application for the Samsung Gear VR headset (one of the neater gadgets I’ve gotten my hands on this year), along the lines of Stellarium or Google Sky Map. It’s pretty rudimentary so far, but nevertheless packs most of the features you would expect from a planetarium application, such as constellations, accurate planet positions based on current time, and a selection of Messier objects.

You can “look at” any object in the virtual night sky within the app, and see a pop-up description of its name. While looking at an object, you can also tap on the trackpad on the Gear VR device to retrieve a full description of the object from Wikipedia (using Wikipedia’s new RESTBase API, which I also help develop).

The app is not yet deployed to the Oculus app store (working on it…), but you’re welcome to check it out on GitHub and build it for your own device. Here are some screenshots:

device-2016-01-26-211603

device-2016-01-26-211851

device-2016-01-26-211815

Share this article:
  • Facebook
  • LinkedIn
  • Reddit
  • Twitter

Star Wars: The Force Awakens — Obligatory Commentary

The new Star Wars movie is really great, but it’s by no means a perfect movie. It strikes a relatively good balance between capitalizing on nostalgia or “fan service,” and delivering fresh, new characters. I’m also thankful that it makes zero references to the prequel trilogy (Episodes I through III). In fact, I firmly believe that one of the planets destroyed by the new Starkiller Base is Coruscant, in a symbolic gesture of destroying any remaining evidence of the prequels. In this way, this movie fully redeems the entire franchise from the sins of the prequels.

But, now that the initial excitement and giddiness have worn off, I’ve crystallized some thoughts on what exactly my issues with this movie are, and I’ll focus on just one:   by far, my biggest complaint is that our lead character, Rey, is implausibly too good at what she does, for the mere convenience of the plot.

This seems to be a recurring theme in J.J. Abrams’ movies. If we look at Star Trek (2009), we see that all the characters are super-geniuses: James Kirk graduates from the Academy in three years, and gets promoted to Captain on his first mission. Uhura speaks a hundred languages, and can learn a new one by listening to it for a few minutes. You get the idea. To me, this suggests that Abrams is more interested in jumping as quickly as possible to the action by endowing his characters with super-human abilities, rather than taking the time to develop his characters in a richer way, by allowing them to be vulnerable human beings with weaknesses.

The Force Awakens obviously invites us to draw comparisons between Rey and Luke Skywalker from the original trilogy, so let’s examine these comparisons:

When we first meet Luke Skywalker, we learn that he’s a good pilot, but that’s about all he is. When it came to most other things, he was pretty much clueless. He had no knowledge of the Force, he was useless in a fight, and had to be rescued multiple times by supporting characters. This made us care about Luke in a much deeper way, because there were times when he was in real danger.

In fact, we can even make the argument that when Luke destroyed the first Death Star, he did it by being a good pilot and a good shot (and being very lucky that Han Solo joined the battle in the end). The Force may have helped him a little, but he certainly didn’t use the Force in the kind of “direct” way that a Jedi would use after sufficient training. Indeed, it wasn’t until Luke’s training with Yoda that he learned to harness the Force directly to manipulate physical objects, as well as to manipulate the actions of weak-minded fools.

Compare this with Rey, who goes from being a lowly scrap scavenger struggling to put food on the table, to doing all of the following in the course of a single movie:

  • Pilots and repairs the Millennium Falcon better than Han Solo ever could, having never touched it before.
  • Resists Kylo Ren’s mind probing, despite Kylo Ren having been trained by Snoke and presumably by Luke Skywalker.
  • Performs a Jedi mind trick on a stormtrooper without ever having heard that Jedi mind tricks are even a thing.
  • Wins against Kylo Ren in a tug-of-war to pull the lightsaber out of the snow, without ever learning that that’s possible.
  • Nearly defeats Kylo Ren in a lightsaber duel, having never used a lightsaber before. (Sure, admittedly Kylo Ren was weakened, but still: he’s a dark-side Force user, trained by Snoke and by Luke Skywalker!)

Because of all of these implausible outcomes, and a ton of other coincidences, we never get a sense that Rey is in any actual danger, which distances us from being able to relate to her. I compare all this to using cheat codes in a video game: “Just get to the boss fight! I don’t have time to go through the rest of this storyline…”

Anyway, once again, The Force Awakens is a great movie overall, and I look forward to watching it again many times. The shortcuts that J.J. Abrams took with the Rey character are certainly not enough to ruin the whole thing.  I just hope that the next installment (Episode VIII) feels more like a true sequel, rather than a sequel-reboot hybrid. I also hope that we get to delve much deeper into the stories and vulnerabilities of the new characters that we’ve met. I want to fall in love with them — but I haven’t yet.

P.S.
On a completely different subject, one other nitpick I’ll mention is a “scientific” one:
When the Starkiller Base fires its weapon, it destroys five planets at once, and we see all five planets in a single frame! If five planets were that close together, they would gravitate towards each other, and eventually combine violently into a single larger planet. Speaking of which, are all the planets in the Star Wars universe inside a single solar system now? The Starkiller Base weapon took less than a minute to reach its targets, meaning either that the weapon travels faster than light, or that Starkiller Base is very close to the other planets. Did the citizens of Coruscant fail to notice the Starkiller Base being built next door?

Share this article:
  • Facebook
  • LinkedIn
  • Reddit
  • Twitter

DiskDigger for Android no longer requires rooting! (with caveats)

Huge news! The DiskDigger app for Android no longer requires the device to be rooted. To be more precise, the app will still work better on rooted devices (it will scan more thoroughly, and rooting is still recommended), but it now has basic functionality to recover photos even on regular non-rooted devices!

If your device is not rooted, DiskDigger will now perform an exhaustive scan of the various caches that your device maintains. These caches often contain lower-resolution versions of the photos on your device. When one of the original photos is deleted, the cached version does not get deleted, allowing DiskDigger to find and recover it for you. This means that, on non-rooted devices, DiskDigger will generally recover lower-resolution versions of your deleted photos. This is a limitation that cannot be overcome at this time (or without rooting).

Of course, if your device is rooted, DiskDigger will continue to perform as it has been, scanning the entirety of your device’s memory for all traces of photos and other types of files.

The app has also been updated to fully support Android Marshmallow (6.0). So, what are you waiting for? Install DiskDigger on your Android device, and see what it can recover!

device-2015-12-13-210046

Share this article:
  • Facebook
  • LinkedIn
  • Reddit
  • Twitter