Category: Programming

Time to mention some great updates in the newest versions of DiskDigger, as well as its cousin FileSystemAnalyzer, which is my “internal” tool for tinkering with various file systems.

ZFS

I’ve been embarking on a bit of self-study of the ZFS file system, specifically its on-disk structures, for the purpose of forensic analysis and opportunities for data recovery. DiskDigger (and FileSystemAnalyzer) now supports ZFS partitions on physical disks and disk images. At the moment it can only parse the current disk’s worth of the ZFS pool, i.e. it does not yet fully support pools that span multiple disks, but this will be updated soon.

FileSystemAnalyzer now lets you visualize and parse a ZFS partition in a couple of unique ways. First, you can select the uberblock from which to start parsing the file system. ZFS uses a round-robin list of uberblocks, where every new “transaction group” causes a new uberblock to be written or updated, and the uberblock with the highest transaction group number is the “active” one. This implies that “older” uberblocks could potentially point to file system structures that are deleted, or forensically interesting in other ways. FileSystemAnalyzer presents the list of potentially parseable uberblocks as separate “partitions”, which you can select:

I have definitely observed cases where deleting files causes a new “transaction group” to be recorded, which creates a new uberblock; and then parsing from the previous uberblock allows the deleted files to be seen.

Then when viewing the actual files in the ZFS partition, FileSystemAnalyzer lets you browse the raw “object list”, which is a flat list of objects that represent all the files and directories organized in the file system tree, but might also include items that are not present in the tree.

As an aside, although the design of ZFS is mostly very sound, I’m a bit taken aback by the complexity of certain portions of ZFS. For example, ZFS uses at least four different ways of storing key-value pairs, each for different purposes:

• nvlist, for storing header metadata at the beginning of the file system.
• SA (system attributes), for storing attributes for files and directories. Because ZFS is designed to be maximally versatile and cross-platform, the types of attributes associated with files and folders can be defined dynamically in a system-attribute registry stored in the metadata of the filesystem.
• ZAP: this is the main storage mechanism by which the actual filesystem (files, directories, symlinks, etc) are structured, achieving a B-tree-like structure. However (!) ZAP offers two different ways of structuring it:
  • Microzap: When there are few enough entries, the structure becomes completely different, and more compact.
  • Fatzap: The actual, full-fledged mechanism for storing the file system tree.

UFS / UFS2 / Minix / Ultrix / Xenix

…Really, all Unix-like file systems. Or, I should say, all inode-based file systems. DiskDigger and FileSystemAnalyzer now have expanded support for more obscure and legacy Unix-like file systems, like Ultrix and Xenix, some of which have different versions that are mutually incompatible. If you have disk images of these types of very old operating systems, send them to me! I’m always looking for obscure stuff to test with.

Pick R83

The Pick Operating System, created by a guy who was actually called Dick Pick, was a super interesting blip in computing history. Everything about this operating system is database-driven, including the file system, if you can call it that. The “account” that a user signs into is really a database, a “file” is really a table in the database, and then each file contains “attributes” and “records” that correspond to columns and rows. The Pick OS found its way into some niche markets, but obviously didn’t last against its larger competitors. It was, however, a bit ahead of its time with its database-centric design, and these ideas are arguably “coming back” in the form of NoSQL.

I’d like to add more meaningful support for Pick partitions in the future, but for now, you can in fact browse a Pick file system in a minimal way, and also see a list of raw on-disk “frames” that make up the Pick database:

Macintosh PowerBook 100

Restored another vintage laptop! This time, the patient is a Macintosh PowerBook 100, which came from a dear friend of mine who allowed me to restore it after recovering the data from its hard drive. The PowerBook 100 was clearly intended to be a “lower-end” model (even though it still had a price tag of $2,500 when it was launched in 1991), with a very minimal design, cheaper-feeling plastic, small monochrome 640×400 LCD display, and no built-in floppy drive. The upshot is that the inexpensive no-nonsense construction allowed for a fairly easy restoration!

When I attempted to power up the laptop as-is, it just made a few crackling noises through the speaker, and not much else. Otherwise it appeared dead. Time to open it up!

The top and bottom halves of the plastic casing are held together by three screws (!), and once these are removed, the entire thing pops open effortlessly.

Looking closely at the humble motherboard, I see the potential culprit right away: failed capacitors that have leaked and corroded. Hopefully the extent of the corrosion is minimal and didn’t affect any of the chips or other components besides the capacitors themselves. I’m hoping this could be as easy as re-capping the board, i.e. replacing the capacitors.

I proceeded to remove all the capacitors that had the slightest indication of corrosion, that is, any capacitor whose solder joints didn’t look totally pristine and shiny. And after removing each one, I cleaned the surface of any residue with alcohol, and then installed a new capacitor with the same value. In all, I replaced 10 bad caps, all of which were a small surface-mount variety, and were either 10µF/16V or 1µF/50V. My new caps are a bit longer than the old ones, so I oriented them horizontally on the board:

And, after reconnecting the display and keyboard back onto the motherboard, let’s try applying power again:

Hey presto, it’s alive!
A bit more cleaning of dust under the keyboard, removing the gunk from inside the trackball mechanism, and a general wipe-down of the exterior, and we’re ready to reassemble!

And there we have it, a lovingly restored PowerBook 100, with 2 MB of RAM, running System 7.0.1. The only unusual thing about it is the hard drive, which is a whopping 1 GB! This was clearly an upgrade from whatever hard drive it had originally (probably something like 40 MB), which must have been installed many years after it was purchased. This implies that the user of this laptop got quite a lot of mileage out of it, probably well into the late 1990s or even 2000s, which makes me happy.

Finally, to round out this restoration, let’s remove this hard drive and replace it with a CompactFlash card, preloaded with tons of vintage games and apps for the Macintosh.

The hard drive interface on the laptop is technically SCSI, so the original hard drive must have been a SCSI drive. The newer 1GB drive is an IDE drive, and came with an adapter board that fits underneath the drive, which translates between SCSI and IDE. This is quite convenient, since we can now plug in a cheap IDE-to-CF adapter, with a generously large CF card that will become our new hard drive. At last, let the retro gaming commence.

MC-3020-Extra tapes!

Recovered data from several MC-3020-Extra and QIC-3020 tapes. These “Extra” tapes have the same front-facing “interface” as their smaller QIC-3020 cousin, except these are larger length-wise, allowing for larger spools inside the cartridge, and therefore a higher data capacity. Of course these cartridges have the same fatal flaw as all other QIC tapes, which is the flimsy tension belt inside the cartridge that drives the motion of the spools. This belt is virtually guaranteed to fail over a long enough time, and since these “Extra” cartridges have even more moving parts inside, they are even more prone to failure.

This batch of tapes was in particularly rough shape: the tension belt in each tape was broken, and was also adhered to the surface of the tape medium. This was likely because the tapes were stored under excessive heat or humidity, which will cause the belt to break down and react with the tape itself. This required pretty extensive cleaning of all the gunk and pieces of belt that were stuck onto the tape.

Fortunately the tapes had been rewound properly, and the damaged portions of the tape were at a spot that was “beyond” the data area of the tape. After throwing on a fresh tension belt, and using one of my trusty Iomega Ditto drives (compatible with a wide range of this family of cartridges), I was able to dump and decode 100% of the data from them.

(As always, get in touch if you have any kind of vintage tapes or other media that you’d like recovered.)

Custom IDE adapter for CP-4021 drive

As part of a special data recovery project, I needed to read the contents of the hard drive from an ancient Compaq LTE 8086 laptop. This hard disk, which is a Conner CP-4021, is quite an oddball. It has an unusual form factor: it’s a 3.5″ drive, but it’s slimmer and shorter than a regular drive, as if it was specifically made to fit into the LTE laptop.

Perhaps most annoyingly, the IDE connector on the drive is a very nonstandard half-pitch connector, where the pins have a horizontal pitch of 1.27mm, and a vertical pitch of 2.54mm. Inside the LTE laptop, this connects to a twin ribbon cable that goes directly to the motherboard, which basically means I have no way of connecting this drive to a “standard” modern IDE controller. (And it’s definitely an IDE drive, since the connector has 44 pins, just like any other laptop hard drive, and it has the same missing “key” pin as a standard IDE connector.)

Time to build a custom connector! After a whole lot of searching, I found a blank 50-pin header on AliExpress that should match the pitch of the drive’s connector. The 50-pin header will overshoot the 44-pin connector by a few pins, but it should still fit without issues. In an ultimate test of my fine motor skills, I soldered a spare 44-pin ribbon cable onto this 50-pin header, checking painstakingly that each pin on the female end matches the corresponding pin on the male end. To secure the delicate soldering onto the header, I covered it with clear epoxy, and let it harden overnight. And just like that, I have an adapter for connecting this ancient drive to a modern PC:

I then connected the drive to my workstation PC, and was encouraged when the drive spun up, and the PC detected the drive successfully! However, the PC didn’t seem to be able to read any actual data from the drive. After a good bit of head-scratching, and double-checking the continuity of all the pins of my adapter, and trying to connect the drive to a few other PCs, I had a last-ditch idea to rule out stuck heads or a locked spindle, which were rather common problems with older drives. I removed the top cover of the drive, exposing the heads and platter, and as the drive was spinning up, I gently turned the spindle manually away from its resting position. And wouldn’t you know it — this caused the spindle to come alive, and the drive became fully functional!

I was able to acquire an image of the drive in Linux with minimal effort (not a single bad sector!), and I’ll be keeping my fancy custom adapter in case I come across another drive like this in the future. I did a few more random Frankenstein experiments with the drive, including booting another vintage laptop from it:

…and connecting that laptop to another laptop via INTERLNK.EXE, which allows the C: drive on the “server” laptop to map as the D: drive on the “client” laptop, with the goal of transferring files from one to the other, or even dumping the entire partition, which I did with the SAVEPART tool.

The connection between the two laptops is a serial cable that I cobbled together from whatever I had on hand, which turned out to be an annoyingly short cable and, thankfully, a null-modem adapter, necessary for communication between the two serial ports. This resulted in a rather slow connection between the laptops; a parallel connection would be significantly faster, but I don’t have the appropriate cable. The INTERLNK tool (bundled with MS-DOS) worked just fine, and automatically detected the connection over the COM port.

T48 chip programmer, finally

I splurged on a proper chip programmer, the T48 by XGecu. Even though there is already a newer model of their programmer (the T56), the extra cost didn’t justify the few additional chips and features it supports, at least for me.

Along with the T48, I purchased a batch of random EEPROM chips for some initial testing and verifying of the T48 itself, and possibly for actual use in future projects. These are Winbond W27C512 chips, which have a rather unusual erase voltage of 14V and programming voltage of 12V, which will be a good exercise for the chip programmer.

The software for the T48 is for Windows only (there is an open-source alternative that supports an earlier model of this programmer (the TL866II), but it still has very limited support for the T48). And because the software is for Windows only, and especially because it requires a special driver which can only be installed with elevated privileges, I prefer to run it in a virtual machine, for reasons that I hope are obvious. Fortunately this is done very easily in VirtualBox, which supports USB passthrough effortlessly. Here it is, running in a Windows 7 (32-bit) VM, and communicating with the T48:

Since my batch of Winbond chips was suspiciously cheap, I assumed they were not “new”, but rather pulled from existing boards. And I was not mistaken: reading the chips, which worked absolutely fine, revealed that they already had contents in them:

And then, erasing and reprogramming the chips also turned out to be a breeze, which gives the T48 programmer a thumbs-up from me. As a special bonus, I removed the firmware chip from the Conner CP-4021 drive (mentioned above!) and was able to read the firmware, using one of the myriad adapters that were included with the T48 programmer.

The above chip is a Microchip 27C256 (32KB), in a PLCC-32 socket. Again, the T48 was able to read this 35-year-old chip without any issues, which makes me look forward to reading and programming many vintage and newer chips in the future.

I’ve finished some major updates to DiskDigger, as well as its companion tool FileSystemAnalyzer, to support a few more filesystems, some more obscure than others!

ReFS support

One of these filesystems represents a serious and substantial update: DiskDigger now has expanded support for ReFS, the Resilient File System introduced in recent versions of Windows Server and Windows Enterprise editions. ReFS remains totally proprietary and undocumented, so it required quite a bit of reverse-engineering to nail down the structures that it uses. I’m happy to report that DiskDigger now supports versions of ReFS starting from 3.0 (introduced in Windows Server 2016) through the very latest version 3.12 (in the latest insider build of Windows 11 Enterprise).

To be clear, DiskDigger had already been able to recover data from ReFS partitions by performing a heuristic (carving) search, which is independent of the actual filesystem on the disk. But now that it understands the data structures of ReFS, it can employ additional specific techniques to recover files more accurately from such partitions.

---

And on a lighter, more whimsical note, DiskDigger and FileSystemAnalyzer now support two other filesystem types that you’ll likely never encounter in everyday life:

RedSea filesystem

The RedSea filesystem was created by the late Terry Davis as part of his TempleOS operating system. If you’re not familiar with TempleOS, it’s an interesting rabbit hole to delve into. Literally an entire operating system built by a single person over the course of many years, TempleOS is intended to be “god’s third temple” in the form of an operating system, due to the guiding principles behind the operating system that Davis believed he was receiving from god. These principles are largely based around simplicity and purity, which is something that even the most hardened atheist like myself can appreciate. There is an expansive volume of videos in which Davis provides tutorials and explains the various features and design choices of TempleOS.

Terry was a troubled soul: he was living with uncontrolled schizophrenia which led to his eventual demise, and his videos occasionally contain some bizarre and horribly racist commentary, all of which make him more pitiable than admirable as a person. However, he was an undeniable savant at building an operating system, and I will defend the idea that we can learn something from his kernel, his compiler, and his insistence on simplicity. As a tribute to his work, I’m including support for the RedSea filesystem in DiskDigger and FileSystemAnalyzer.

The RedSea filesystem is, in many ways, the simplest filesystem possible:

• All files are contiguous! There’s no concept of fragmentation.
• There are no B-trees, no journaling, no symbolic links, no encryption, etc.
• There’s no concept of clusters; block sizes are the same as sector sizes, i.e. 512 bytes.
• Directory blocks are just a sequential list of directory entries.
• For determining where to write new files, there is simply an allocation bitmap, where each bit represents whether the corresponding block is allocated.

One other interesting feature of the RedSea filesystem is that it performs a sort of semi-automatic compression of files, using a form of LZW compression. If you give a file a name that ends with a “.Z” extension, it will be compressed when it’s written to the disk, and then uncompressed the next time it’s read from the disk (transparently to the user). This compression is also supported in DiskDigger, i.e. files recovered from a RedSea partition will be automatically uncompressed.

Commodore 64 disk images

As another fun diversion, I also added support for Commodore 64 disk images (D64 files)! The file system on these disks is thoroughly documented, and is also very simple: files are represented as a linked list of blocks (a primitive “block chain”, as it were). If you have these disk images lying around, you can now peruse their contents!

---

As with all filesystems supported by FileSystemAnalyzer and DiskDigger, these additions are read-only, since these are intended to be tools for forensic analysis, and not intended for two-way interoperability.