Knock-off apps: the sincerest form of flattery

As the developer of a relatively popular app, there are numerous things I have to worry about on a regular basis. One of these things, however, is something for which I was kind of unprepared, and still not sure how to deal with: the emergence of knock-off apps — apps that have a suspicious and often hilarious resemblance to the original.

There were, of course, counterfeit apps that were blatantly using my trademark in their name (“DiskDigger” is a registered trademark in the U.S.), or my graphics in their store listings, in which case the Google Play Store thankfully took them down upon request.

But then there are apps that don’t quite use my trademark, and don’t quite have the same icon and screenshots. But because they are close enough, they can take advantage of being near the top search results for this category of apps, and lead numerous unsuspecting users to install them and be greeted with a barrage of ads and spyware.

Most of these apps seem to come from Indian and Middle-Eastern developers, which makes the horribly broken English in their app verbiage even more amusing to read. A small part of me even applauds their enterprising spirit, and I don’t fault them for wanting to make a buck, but I wish they’d find ways of doing it more honestly.

Take a look! Can you tell the knock-off apps from the real ones?

The problem is that because these apps don’t explicitly violate my trademark, Google refuses to take them down, which is unfortunate because these apps do literally nothing except shove ads in the user’s face, and therefore actively harm the ecosystem of the Play Store. Shouldn’t Google care more about preventing the Play Store from becoming a cesspool of bottom-feeding cash grabbers?

Premature optimization of Android View hierarchies

In my day job, one of my responsibilities is to oversee the technical evolution of our product and to optimize its performance. I’ve even written a few guidelines that detail numerous recommendations for maximizing performance of Android apps in general.

One of the things I have always recommended is to reduce the complexity of View hierarchies, and try not to overcrowd or have too many nesting levels of your Views, since this can supposedly have a negative impact on performance.  However, I made these statements based on common sentiment on the web, and based on the Android documentation, instead of on actual hard evidence.  So I  looked into it from an interesting perspective:  I dug into View hierarchies as they are used by other major apps, and compared them with our own usage.  This isn’t a totally “scientific” analysis, and it only looks at a single facet of proper View usage. Nevertheless the findings are rather surprising, and are actually challenging my insistence on View minimalism.

I looked at the Twitter, Facebook, and Slack apps, and compared each of their “feed” screens to the feed screen of our own Wikipedia app. (The reason I chose these apps is that the “performance” of their feeds is nearly perfectly smooth, especially considering that some of their content includes auto-playing videos and animations.)  I used the superbly useful and little-known UI Automator Viewer tool, which is bundled with the Android SDK, to explore these view hierarchies.

For reference, the deepest nesting that I found in the feed of the Wikipedia app is seven (7) levels deep:

But get ready:  The deepest nesting in the Slack app is… eighteen (18) levels deep. And yet it performs perfectly smoothly:

The deepest nesting in the Facebook app is twenty (20) levels deep. And yet it works just fine:

The deepest nesting in the Twitter app is twenty three (23) levels deep, which includes eight (8) nesting levels for each item in their ListView (it’s not even a RecyclerView!). And yet I’m able to scroll the Twitter feed infinitely without a single hiccup.

Therefore I’m compelled to reevaluate the importance we should be placing on optimizing View hierarchies, at least from the perspective of “nesting.”  Indeed, this seems to be yet another case for balancing reasonable performance guidelines with more immediate product goals, or put more simply, avoiding premature optimization.

FileSystemAnalyzer: a tool that does what it says

Today I’m happy to release a tool for low-level analysis of file systems, which includes digging through file system structures that aren’t normally visible when exploring your disks, looking at metadata of files and folders that isn’t normally accessible, and even browsing file systems that aren’t supported by Windows or your PC.

Download FileSystemAnalyzer

This is actually the software that I use “internally” to test and experiment with new features for DiskDigger, but I thought that it might be useful enough to release this tool its own. It accesses the storage devices on your PC and reads them at the lowest level, bypassing the file system drivers of the OS.

On the surface, this software is very simple: it allows you to browse the files and folders on any storage device connected to your PC, and supports a number of file systems, including some that aren’t supported by Windows itself. However, the power of this tool comes from what else it shows you in addition to the files and folders:

FAT

The program supports FAT12, FAT16, and FAT32 partitions. When looking at FAT partitions, you can see the file and directory structure, and detailed metadata and previews of files that you select. When selecting a file or folder, you can also see its position in the FAT table. And indeed you can explore the entire FAT table independently of the directory structure, to see how the table is structured and how it relates to the files and directories:

exFAT

Similarly to FAT, this lets you explore the exFAT file system, while also letting you look at the actual FAT table and see how each file corresponds to each FAT entry.

NTFS

In addition to exploring the NTFS file and folder structure, you can also see the MFT table, and which MFT entry corresponds to which file or folder:

HFS+

HFS+ is the default file system used in macOS (although it is slowly being superseded by APFS), and is fully supported in FileSystemAnalyzer. You can explore the folders and files in an HFS+ partition, and you can also see the actual B-Tree nodes and node contents that correspond to each file:

ext4

Ext4 partitions (used in Linux and other *nix operating systems) are also supported in FileSystemAnalyzer. In addition to exploring the folders and files, you can also see the actual inode table, and observe how the inodes correspond to the directory structure:

ISO 9660 / Joliet

ISO 9660 is the original and simplest file system on CD and DVD disks, and Joliet is Microsoft’s extension onto ISO 9660 that adds support for Unicode file names. FileSystemAnalyzer allows you to browse a Joliet file system as either Joliet or ISO 9660 by letting you select which volume descriptor to use.

UDF

The UDF file system is also fully supported, which is the modern file system generally used on DVD and Blu-ray disks. When opening a disk or disk image, you can choose to open the UDF file system or the stub ISO 9660 volume that usually accompanies it.

UDF can also be used on regular disks, not just optical disks. (Pro tip: it’s actually possible to format any disk as UDF by executing this command in an elevated command prompt: format <drive>: /fs:UDF)

APFS

Support for APFS is still very rudimentary, since there isn’t yet any official documentation on its internal structure, and requires some reverse engineering. Nevertheless, support for APFS is planned for a near-future update.

Disk images

The program supports E01 (EWF) disk images, as well as VHD (Microsoft Virtual Hard Disk), VDI (from VirtualBox), VMDK (from VMware), ISO files (CD/DVD images) and of course plain dd images.

Creating reports

Given the exhaustiveness of the information that this tool presents about the file system that it’s reading, there’s no end to the types of reports that it could generate. Before committing to specific type(s) of reports for the program to create, I’d like to get some feedback from other forensics specialists on what kind of information would be the most useful. If you have any suggestions on what to include in reports, please contact me.

Limitations

For now, FileSystemAnalyzer is strictly a read-only tool: it lets you read files and folders from a multitude of partitions, but does not let you write new data to them. In some ways this can actually be beneficial (especially for forensics purposes), but is clearly a limitation for users who might want to use the tool to add or modify files in partitions that are not supported natively by the operating system.

Feedback

I’d love to hear what you think of FileSystemAnalyzer so far, and any ideas that you might have for new features or improvements. If you have any suggestions, feel free to contact me!

Download FileSystemAnalyzer

Revisiting the Windows 10 thumbnail cache

When you look at a folder full of pictures, and enable the display of thumbnails in your folders, Windows will show a thumbnail that represents each of the pictures. Creating these thumbnails is an expensive operation — the system needs to open each file, render the image in memory, and resize it to the desired size of the thumbnail. Therefore, Windows maintains a cache of thumbnails, saved in the following location: [User folder]\AppData\Local\Microsoft\Windows\Explorer.

A few years ago I published a utility for examining the thumbnail cache in Windows Vista and Windows 7.  However, in Windows 8 and Windows 10, Microsoft seems to have made some slight modifications to the file format of the thumbnail cache which was preventing my utility from working properly.  So, I revisited the thumbnail cache on the most recent version of Windows, and made sure the utility works correctly with it, as well as with all previous versions of the cache.

My updated ThumbCacheViewer supports thumbnail cache files from all versions of Windows after XP.  It automatically detects cache files associated with the current user’s account, and it also allows you to explicitly open thumbnail cache files from any other location. Once the file is opened, the utility will show a simple list of all the images contained in the selected cache. If you select an individual image, it will also show some useful metadata about the image:

You can see that the cached images include thumbnails of individual photos, as well as thumbnail representations of folders that contain photos. Both of these can be forensically interesting, since the folder thumbnails still contain plenty of detail in the images. You can also see that there are separate caches for different resolutions of thumbnails, some of which are strikingly high-resolution (up to 2560 pixels wide, which almost defeats the purpose of a cache).

I’ll also point out that you can perform forensic analysis on thumbnail caches using DiskDigger, by opening the cache file as a disk image. You can do this by following these steps:

  • Launch DiskDigger, and go to the “Advanced” tab in the drive selection screen.
  • In the “Bytes per sector” field, enter “1”.
  • Click the “Scan disk image” button, and find the thumbnail cache file that you want to scan.
  • Select “Dig deeper” mode, and proceed with the scan.

Here is the same cache file as in the screenshot above, but viewed using DiskDigger (note the numerous .BMP images detected by the scan):

Either way, this is now a relatively complete solution for analyzing thumbnail cache files, whether you’re a professional forensics specialist, or a home user who’s interested in how Windows preserves thumbnails in more ways than you might think!

The dead dream of chatbots

A long time ago I wrote about the Loebner Prize, and how it seemed like this competition isn’t so much a test of machine intelligence, but rather a test of how well the programmers can fool the judges into thinking they’re talking to a human. Not that anyone has actually done this successfully — none of the judges have ever been convinced that any of the chatbots were really a human, and the annual “winner” of the Prize is decided by sympathy points awarded by some very charitable judges.

In that previous post, I remember being struck by how little has changed in the quality of the chatbots that are entered into this competition:  it hadn’t improved since the inception of the prize.  So, today I thought I’d randomly “check in” on how the chatbots are doing, and read the chat transcripts from the competition from more recent years. (I encourage you to read the transcripts for yourself, to get a taste of the “state of the art” of today’s chatbots.)

And can you guess how they’re doing? Of course you can:  they still haven’t improved by any margin, except perhaps a larger repertoire of cleverly crafted catch-all responses. Even the winner and close runner-up of the 2017 competition are comically robotic, and can be effortlessly pegged as artificial by a human judge.

My goal in this post, however, is not to bash the authors of these chatbots — I’m sure they’re competent developers doing the best they can. My goal is to discuss why chatbot technology hasn’t moved forward since the days of ELIZA. (Note: When I refer to chatbots, I’m not referring to today’s virtual assistants like Siri or Alexa, which have made big strides in a different way, but rather to human-approximating, Turing-test-passing machines, which these chatbots attempt to be.)

I think much of it has to do with a lack of corporate support. Chatbots have never really found a good business use case, so we haven’t seen any major companies devote significant resources to chatbot development. If someone like Google or Amazon had put their weight behind this technology, we might have seen an advancement or two by now.  Instead, the competitors in the Loebner Prize still consist of individual hobbyists and hackers with no corporate backing.

Interest in chatbots seems to have peaked around 2012, when the cool thing was to add a customized bot to your website and let it talk to your users, but thankfully this died down very shortly thereafter, because apparently people prefer to communicate with other people, not lame attempts at imitation. We can theorize that someday we may hit upon an “uncanny valley” effect with chatbots, where the conversation is eerily close to seeming human (but not quite), which will cause a different kind of revulsion, but we’re still very far from that point.

Another thing to note is the actual technology behind today’s (and yesterday’s) chatbots. Most of these bots, and indeed all the bots that have won the Loebner Prize in recent years, are constructed using a language called AIML, which is a markup language based on XML.  Now, there have been plenty of ways that people have abused and misused XML in our collective memory, but this has to be one of the worst!  AIML attempts to augment XML with variables, stacks, macros, conditional statements, and even loops. And the result is an unwieldy mess that is completely unreadable and unmaintainable. If chatbot technology is to move forward, this has to be the first thing to throw out and replace with something more modern.

And finally, building a chatbot is one of those endeavors that seems tantalizingly simple on the surface:  if you look at the past chat logs of the prize-winning bots, it’s easy to think to yourself, “I can build a better bot than that!”  But, once you actually start to think seriously about building a bot that approximates human conversation, you quickly come up against research-level problems like natural language processing, context awareness, and of course human psychology, behavior, and consciousness in general.  These are most definitely not problems that can be solved with XML markup.  They likely can’t even be solved with today’s neural networks and “deep learning” algorithms.  It will probably require a quantum leap in AI technology.  That is, it will require building a machine that is truly intelligent in a more general way, such that its conversations with humans are a by-product of its intelligence, instead of its primary goal.

For now, however, the dream of chatbots has been laid to rest in the mid-2010s, and will probably not come back until the technology allows it, and until they’re actually wanted or needed.