Blog

The age of instant nostalgia

Nostalgia is an emotion that should be felt sparingly. At least for someone like me, it’s very easy to get lost in rabbit holes of nostalgia. On days when I’m feeling sentimental, I can spend hours browsing old photo albums and reflecting on past events that I’m powerless to change.

But nostalgia can go much deeper than that. With today’s technology, we can have experiences of nostalgia that were unheard of in previous generations. Remember those candies you loved as a child that were only available in the town where you grew up? Now you can buy them on Amazon and have them shipped in two days. Remember your trip to Stonehenge twenty years ago? Now you can see it in VR any time you like. Remember the games you used to play on your old Atari console? Now you can play them on your mobile phone while waiting for the train. Remember your old friends from elementary school with whom you haven’t connected in years? Now you can find them with a few clicks and talk to them instantly.

We’re living in an age when you can evoke feelings of nostalgia at a moment’s notice, and on demand. This fact is well-known to Hollywood, which capitalizes on nostalgia with abominations like Pixels which do nothing but literally tell the audience, “Remember Q*bert?” “Remember Pac-Man?” “Remember Galaga?”, or its endless obsession with sequels and prequels and reboots — anything to jog your memory of an old movie you used to love.  It’s also known in the literary world, with the success of novels like Ready Player One, where any semblance of a plot is merely an afterthought, and most of the book’s real purpose is to barrage the reader with a rapid-fire of pop culture references from the 1980s.

The problem with nostalgia is that it’s not a productive emotion, in the sense that it romanticizes the past, locks your thinking into it, as if your past experiences were somehow better than the present or future. This kind of thinking is counterproductive whether or not your past was in fact better than your present:  If your past really was better than the present, then you should be actively working to improve your present life instead of dwelling on past events. And if, instead, your life is better today than yesterday, then romanticizing the past cheapens the goodness of your life in the present day, and takes away time you could be spending enjoying the present and planning an even better future.

Nostalgia should be treated like a sweet, rich dessert — great in small quantities, but bad for you if you indulge in it too much.

img082

Premature optimization of Android View hierarchies

In my day job, one of my responsibilities is to oversee the technical evolution of our product and to optimize its performance. I’ve even written a few guidelines that detail numerous recommendations for maximizing performance of Android apps in general.

One of the things I have always recommended is to reduce the complexity of View hierarchies, and try not to overcrowd or have too many nesting levels of your Views, since this can supposedly have a negative impact on performance.  However, I made these statements based on common sentiment on the web, and based on the Android documentation, instead of on actual hard evidence.  So I  looked into it from an interesting perspective:  I dug into View hierarchies as they are used by other major apps, and compared them with our own usage.  This isn’t a totally “scientific” analysis, and it only looks at a single facet of proper View usage. Nevertheless the findings are rather surprising, and are actually challenging my insistence on View minimalism.

I looked at the Twitter, Facebook, and Slack apps, and compared each of their “feed” screens to the feed screen of our own Wikipedia app. (The reason I chose these apps is that the “performance” of their feeds is nearly perfectly smooth, especially considering that some of their content includes auto-playing videos and animations.)  I used the superbly useful and little-known UI Automator Viewer tool, which is bundled with the Android SDK, to explore these view hierarchies.

For reference, the deepest nesting that I found in the feed of the Wikipedia app is seven (7) levels deep:

layout_dump3

But get ready:  The deepest nesting in the Slack app is… eighteen (18) levels deep. And yet it performs perfectly smoothly:

layout_dump2

The deepest nesting in the Facebook app is twenty (20) levels deep. And yet it works just fine:

layout_dump4

The deepest nesting in the Twitter app is twenty three (23) levels deep, which includes eight (8) nesting levels for each item in their ListView (it’s not even a RecyclerView!). And yet I’m able to scroll the Twitter feed infinitely without a single hiccup.

layout_dump1

Therefore I’m compelled to reevaluate the importance we should be placing on optimizing View hierarchies, at least from the perspective of “nesting.”  Indeed, this seems to be yet another case for balancing reasonable performance guidelines with more immediate product goals, or put more simply, avoiding premature optimization.

Higher-level illusions

Most of us have seen optical illusions, and witnessed firsthand how a simple but specially crafted illustration can completely trick our brain, whether it’s an illusion involving depth perception, motion perception, color perception, etc.  One of my favorites is this illusion involving checkered squares with alternating shades of gray (Is square A darker than square B?):

Credit: Wikimedia Commons.

When I first saw the above illusion, I found it unfathomable that squares A and B are actually the same color, and yet it’s true. The illusion is so powerful, I had to open the image in Photoshop and literally look at the pixel color values of the squares to convince myself that they are the same.

But actually, we don’t even need to resort to any specially contrived images to fool our visual circuits, since our eyes themselves have a built-in defect — a consequence of the eye’s evolutionary history — a blind spot that gets patched over in real time by the software of our consciousness.  This allows us to go on with our lives being completely oblivious of this defect (unless we consciously look for it), but it basically means that we experience this genuine illusion during every waking moment.

These kinds of illusions powerfully illustrate how a simple misfire of our sensory perceptions can send our understanding of the world completely astray, and how our consciousness has adapted to compensate for the laughable fallibility of our senses.

My question is the following:  If it’s this easy to fool our visual processing circuits, what kinds of illusions might be at work at higher levels of our consciousness?  What other blind spots are auto-filled by the software of our brain, making us oblivious to their true nature?

The key to uncovering and understanding illusions, I think, is cognitive effort.  It takes cognitive effort to realize that the illustration at the top of this article is, in fact, an illusion.  It takes cognitive effort to expose and become aware of the blind spot in your own eyes. What other illusions might we uncover if we keep building up the muscles of cognitive effort?

Perhaps we might discover that the Earth, instead of being a flat plane with a dome covering it, is actually a spheroidal mass that orbits the Sun, contrary to all of our intuition.

Perhaps we’ll discover that the Sun is actually one of billions of other suns, and is by no means unique among them, and that our galaxy is one of billions of other galaxies, with similarly little uniqueness about it.

We might also discover that the folk tales and mythologies of our ancestors are not literally true, but are merely expressions of the fears, aspirations, ideals, and desires that we all share, especially the desire to find meaning and purpose in a world that doesn’t grant us purpose on its own.

And perhaps we’ll discover that free will itself, far from being a gift bestowed on us by a creator or even a self-evident property that emerges from our consciousness, is actually the grandest illusion of all:  that all of our thoughts and actions are consequences of deterministic physical laws.

But all of these realizations need not lead us towards fatalism or nihilism, for these too are illusions.  If the universe doesn’t grant us a purpose ex nihilo, it shouldn’t stop us from being able to create our own purpose.  And if it really is true that the laws of physics underlie all of our choices, it doesn’t make our choices any less meaningful or consequential, and it doesn’t mean that we should stop striving to make better choices that improve the lives of current and future generations.

And of course, none of this takes into account the illusions of higher and higher order that we’re bound to uncover in the future, and all the consequences of those discoveries that we can’t even fathom in the present.  The one thing we must not stop doing is exerting our cognitive effort to keep discovering and untangling illusions, wherever we might find them. The immortal words of Stephen Jay Gould come to mind:

We are the offspring of history, and must establish our own paths in this most diverse and interesting of conceivable universes — one indifferent to our suffering, and therefore offering us maximal freedom to thrive, or to fail, in our own chosen way.

Digital hoarding

I have a confession to make: I’m a hoarder. Not a hoarder of material possessions — oh no, my house is almost entirely free of unnecessary stuff. I take pride in actively reducing the amount of physical crap that I own, and donate items I no longer need. My family and I have even made a gift-giving agreement among ourselves where any gifts must either be consumable (specialty foods, restaurant gift cards, etc) or experiences (tickets to a show, subscription to a service, etc).

My hoarding, on the other hand, is of the digital variety. My “collection” only spans a few external hard drives of 1 TB each, occasionally backed up or synced to a duplicate set of external hard drives. The physical space occupied by these drives is less than one cubic foot, but the vastness of the digital stuff that they contain is… considerable. Just to give you a rough idea of what I’m dealing with:

  • Archives of old emails and correspondence from previous jobs and contracts.
  • Archives of Instant Messenger conversations with old friends and ex-girlfriends, dating back to 1997.
  • A collection of viruses and trojans for MS-DOS from the 80s and 90s, originally for research purposes.
  • A huge library of shareware games and programs from the MS-DOS era.
  • An archive of articles, papers, and textbooks (in PDF form) relating to computer science, mathematics, and physics.
  • An archive of high-resolution NASA imagery of planets, nebulas, and galaxies.
  • An extensive library of file formats (i.e. sample files saved by all kinds of different software) and documentation for every file format specification.
  • Emulators and system images of virtually every computer system and game console ever built.
  • My complete genome, which I’ve had sequenced a few years ago.
  • And of course, my personal photo and video library, from my birth to the present day, and also photos from my parents’ and grandparents’ old albums that I have digitized and saved.

So yeah… recently I’ve been asking myself whether digital hoarding is a problem of the same magnitude as physical hoarding. On the surface, one might ask “What’s the problem?” These things aren’t taking up any physical space, and you don’t have to give them another thought after you save them to the disk. And yet, perhaps there is an emotional toll that comes with the mere knowledge that all of this old data still exists, and remains your responsibility. If anything, this is surely at odds with my attitude towards physical possessions, which is quite minimalist.

A basic litmus test for hoarding behavior consists of a simple question: How would you feel about throwing away any random item that you see around you? Will you use this twisty-tie for anything? How about this pen cap? Do you really need three different cheese graters? How about this pile of old magazines? A hoarder will answer these questions with something like, “You never know when it will come in handy.” And if I’m being honest, that’s exactly how I feel about all the digital items I listed above. I feel the same hesitation about deleting any of them as a “physical” hoarder might feel about donating old clothes, or throwing away expired spices from the pantry.

When I look at my enormous pile of digital junk, I see in myself all the symptoms of real hoarding, albeit confined to the digital realm, which is likely why it’s been able to go on for so long. I’m also reminded of how freeing and cathartic it feels to let go of unnecessary possessions, and I theorize that a similar feeling of freedom will result from permanently letting go of digital baggage.

Therefore, I have resolved to stop being hypocritical in this regard, and start practicing digitally what I practice physically.  Many of the items I mentioned in my list are actually replaceable (easily found on the web, or generated with minimal effort).  Some of the items are technically “irreplaceable,” such as my old emails and IM archives, but represent unnecessary cognitive and emotional baggage, and have no real nostalgic value.  The only things that seem to have actual meaning, and are objectively worth keeping, are my personal photos and videos, and even those can probably be trimmed down a bit.

It’s time to let the past go, and embrace the future without anything weighing you down. Let the cleaning begin.

FileSystemAnalyzer: a tool that does what it says

Today I’m happy to release a tool for low-level analysis of file systems, which includes digging through file system structures that aren’t normally visible when exploring your disks, looking at metadata of files and folders that isn’t normally accessible, and even browsing file systems that aren’t supported by Windows or your PC.

Download FileSystemAnalyzer

This is actually the software that I use “internally” to test and experiment with new features for DiskDigger, but I thought that it might be useful enough to release this tool its own. It accesses the storage devices on your PC and reads them at the lowest level, bypassing the file system drivers of the OS.

On the surface, this software is very simple: it allows you to browse the files and folders on any storage device connected to your PC, and supports a number of file systems, including some that aren’t supported by Windows itself. However, the power of this tool comes from what else it shows you in addition to the files and folders:

FAT

The program supports FAT12, FAT16, and FAT32 partitions. When looking at FAT partitions, you can see the file and directory structure, and detailed metadata and previews of files that you select. When selecting a file or folder, you can also see its position in the FAT table. And indeed you can explore the entire FAT table independently of the directory structure, to see how the table is structured and how it relates to the files and directories:

filesystemanalyzer5

exFAT

Similarly to FAT, this lets you explore the exFAT file system, while also letting you look at the actual FAT table and see how each file corresponds to each FAT entry.

NTFS

In addition to exploring the NTFS file and folder structure, you can also see the MFT table, and which MFT entry corresponds to which file or folder:

filesystemanalyzer3

HFS+

HFS+ is the default file system used in macOS (although it is slowly being superseded by APFS), and is fully supported in FileSystemAnalyzer. You can explore the folders and files in an HFS+ partition, and you can also see the actual B-Tree nodes and node contents that correspond to each file:

filesystemanalyzer2

ext4

Ext4 partitions (used in Linux and other *nix operating systems) are also supported in FileSystemAnalyzer. In addition to exploring the folders and files, you can also see the actual inode table, and observe how the inodes correspond to the directory structure:

filesystemanalyzer4

APFS

Support for APFS is still very rudimentary, since there isn’t yet any official documentation on its internal structure, and requires some reverse engineering. Nevertheless, support for APFS is planned for a near-future update.

Disk images

The program supports E01 (EWF) disk images, as well as VHD (Microsoft Virtual Hard Disk), VDI (from VirtualBox), VMDK (from VMware), and of course plain dd images.

Creating reports

Given the exhaustiveness of the information that this tool presents about the file system that it’s reading, there’s no end to the types of reports that it could generate. Before committing to specific type(s) of reports for the program to create, I’d like to get some feedback from other forensics specialists on what kind of information would be the most useful. If you have any suggestions on what to include in reports, please contact me.

Limitations

For now, FileSystemAnalyzer is strictly a read-only tool: it lets you read files and folders from a multitude of partitions, but does not let you write new data to them. In some ways this can actually be beneficial (especially for forensics purposes), but is clearly a limitation for users who might want to use the tool to add or modify files in partitions that are not supported natively by the operating system.

Feedback

I’d love to hear what you think of FileSystemAnalyzer so far, and any ideas that you might have for new features or improvements. If you have any suggestions, feel free to contact me!

Download FileSystemAnalyzer