Home security with Raspberry Pi

The versatility of the Raspberry Pi seems to know no bounds. For a while I’ve been wanting to set up a DIY home security system in my house, and it turns out that the Raspberry Pi is the perfect choice for this task, and more. (The desire for a security system isn’t because we live in a particularly unsafe neighborhood or anything like that, but just because it’s an interesting technical challenge that provides a little extra peace of mind in the end.)

Camera integration

I began with a couple of IP cameras, namely the Anpviz Bullet 5MP cameras, which I mounted on the outside of the house, next to the front door and side door.  The cameras use PoE (power over Ethernet), so I only needed to route an Ethernet cable from the cameras to my PoE-capable switch sitting in a closet in the basement.

At first I assumed that I would need to configure my Raspberry Pi (3) to subscribe to the video streams from the two cameras, do the motion detection on each one, re-encode the video onto disk, and then upload the video to cloud storage.  And in fact this is how the first iteration of my setup worked, using the free MotionEye software.  However, the whole thing was very sluggish, since the RPi doesn’t quite have the horsepower to be doing decoding, encoding, and motion detection of multiple streams at once (and I didn’t want to compromise by decreasing the video quality coming from the cameras), so my final output video was less than 1 frame per second, with my RPi running at full load and getting quite warm. Definitely not a sustainable solution.

But then I realized that a much simpler solution is possible. The Anpviz cameras are actually pretty versatile themselves, and can perform their own motion detection. Furthermore, they can write the video stream directly onto a shared NFS folder!  Therefore, all I need to do is set up the RPi to be an NFS server, and direct the cameras to write to the NFS share whenever motion is detected.

And that’s exactly what I did, with a little twist:  I attached two 16 GB USB flash drives to the RPi, with each USB drive becoming an NFS share for each respective camera. That way I’ll get the maximum throughput of data from the cameras directly to USB storage. With this completed setup, the Raspberry Pi barely reaches 1% CPU load, and stays completely cool.

I wrote a Python script that runs continuously in the background and checks for any new video files being written onto the USB drives. If it detects a new file, it automatically uploads it to my Google Drive account, using the Google Drive API which turned out to be fairly easy to work with, once I got the hang of it. The script automatically creates subfolders in Google Drive corresponding to the current day of the week, and which camera the video is from. It also automatically purges videos that are more than a week old.

I have to heap some more praise onto the cameras for supporting H.265 encoding, which compresses the video files very nicely. All in all, with the amount of motion that is typical on a given day, I’m averaging about 1 GB per day of video being recorded (at 1080p resolution!), which makes 7 GB in a rolling week’s worth of video, which is small enough to fit comfortably in my free Google Drive account, without needing to upgrade to a paid tier of storage.

Water sensor

Since my Raspberry Pi still had nearly all of its processing power still left over, I decided to give it some more responsibility.

About a month ago the sewer drain in the house became clogged, which caused it to back up and spill out into the basement.  Fortunately I was in the basement while this was happening and caught it before it could do much more damage. An emergency plumber was called, and the drain was snaked successfully (turned out to be old tree roots).  However, from now on I wanted to be warned immediately in case this kind of thing happens again.

So I built a very simple water sensor and connected it to the Raspberry Pi.  In fact “very simple” is an understatement: the sensor is literally two wires, close together, which will short out if they come into contact with water.  I used some very cheap speaker wire, and routed it from the RPi to the drain from where the water can potentially spill out.

On the Raspberry Pi, one wire is connected to ground, and the other is connected to a GPIO pin with a pull-up resistor enabled. This means that if the wires are shorted out, the GPIO input will go from HIGH to LOW, and this will be an indication that water is present. The sensor is being monitored by the same Python script that monitors and uploads the camera footage, and will automatically send me an email when the sensor is triggered.

For good measure, I installed a second water sensor next to our hot water tank, since these have also been known to fail and leak at the most inconvenient times.

And that’s all for now. The Raspberry Pi still has plenty of GPIO pins left over, so I’ll be able to expand it with additional sensors and other devices in the future.

Notes

Here are just a few random notes related to getting this kind of system up and running:

Enable shared NFS folder(s)

Install the necessary NFS components:
$ sudo apt-get install nfs-kernel-server portmap nfs-common
Add one or more lines to the file /etc/exports:
/folder/path_to_share *(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)
And then run the following:
$ sudo exportfs -ra
For good measure, restart the NFS service:
$ sudo /etc/init.d/nfs-kernel-server restart

Run script(s) on startup

  • Add line(s) to /etc/rc.local
  • If it’s a long-running script, or continuously-running, then make sure to put an ampersand at the end of the line, so that the boot process can continue.

Automatically mount USB drive(s) on boot

When the Raspberry Pi is configured to boot into the desktop GUI, it will auto-mount USB drives, mounting them into the /media/pi directory, with the mount points named after the volume label of the drive. However, if the Pi is configured to boot into the console only (not desktop), then it will not auto-mount USB drives, and they will need to be added to /etc/fstab:
/dev/sda1 /media/mount_path vfat defaults,auto,users,rw,nofail,umask=000 0 0

(The umask=000 parameter enables write access to the entire disk.)

Set the network interface to a static IP

Edit the file /etc/dhcpcd.conf. The file contains commented-out example lines for setting a static IP, gateway, DNS server, etc.

And lastly, here are a couple of Gists for sending an email from within Python, and uploading files to a specific folder on Google Drive.

Reverse-engineering the QICStream tape backup format

TLDR: I developed an open-source tool to read tape backup images that were made using the QICStream tool, and extract the original files from them.

During a recent data recovery contract, I needed to recover files from some old QIC tapes. However, after reading the raw data from the tapes, I couldn’t recognize the format in which the backup was encoded, and none of the usual software I use to read the backups seemed to be compatible with it.

After briefly examining the backup in a hex editor, it was evident that the backup was fortunately not compressed or encrypted, and there were signs that the backup was made using a tool called QICStream.  There doesn’t seem to be any documentation regarding this utility (or the format of the backup it saves) on the web. It’s easy enough to find the tool itself on ancient DOS download sites, and it may have been an interesting project to create an emulated DOS environment where the QICStream tool reads the backup from an emulated tape media, but it turned out to be much easier to reverse-engineer the backup structure and decode the files from the actual raw data.

The binary format of the backup is very simple, once you realize one important thing:  every block of 0x8000 bytes ends with 0x402 bytes of extra data (that’s right, 0x402 bytes, not 0x400). In other words, for every successive block of 0x8000 bytes, only the first 0x7BFE bytes are useful data, and the last 0x402 bytes are some kind of additional data, possibly for parity checking or some other form of error-correcting logic. (I did not reverse-engineer the true purpose of these bytes; they did not turn out to be important in the end.)

Other than that, the format is very straightforward, and basically consists of a sequence of files and directories arranged one after the other, with a short header in front of each file, and “control codes” that determine whether to descend into a subdirectory or to navigate back out of it.

Anyway, I put all of these findings into a small open-source utility that we can now use to extract the original files from QICStream backups. Feel free to look through my code for additional details of the structure of the file headers, and how the large-scale structure of the backup is handled.

When you need some perspective

Whenever I get too caught up in reading the news, or sucked into hopelessly unproductive political discussions, I like to relax with some images from the Hubble Space Telescope and similar telescopic wonders, which give us a glimpse into the wider universe outside the minuscule mote of dust on which we live.

Take, for example, the Ultra Deep Field image, which contains roughly ten thousand galaxies within an area of the sky that is about 2×2 arcminutes. Do you know how small a portion of the sky that is? If you take a 1×1 millimeter square, hold it at arm’s length, and peer through it, that’s the area of the sky captured in the Ultra Deep Field.  The UDF was taken in a deliberately ordinary and uninteresting portion of the sky, which implies that for every 1x1mm portion of the sky, we could get a similarly breathtaking image of thousands of galaxies. Nearly every bright spot in the image is a galaxy, and some of the galaxies are over 12 billion years old.

How about some images that are impressive not in their depth, but in their resolution? Here is the sharpest ever image of the Andromeda galaxy (a composite of numerous other images), where we can zoom in to see literally every individual star in a galaxy outside of our own. If you look closely, you can also find globular clusters and nebulas, all in a different galaxy. There is a similarly high-resolution image of the Triangulum galaxy, as well as a super high-resolution image of the Orion nebula.

Or how about some images of gravitational lenses, where gargantuan clusters of galaxies create such strong curvature in spacetime that the light from more distant galaxies bends around them and becomes distorted, or even splits into multiple images of the same galaxies in different spots.

And of course there’s the recent image of an actual black hole (or rather an image of matter accreting around it) at the center of Messier 87, an extremely massive galaxy with a central black hole of 5 billion solar masses, making it easier to observe than the black hole in our own galaxy.

All of this provides a pleasant counterbalance to our daily political bickering, which all seems laughably “local” by contrast.  To channel Christopher Hitchens for a moment, take some time to let these awe-inspiring images sink in, and then compare them to the story of Moses and his “burning bush.”

Digital hoarding

I have a confession to make: I’m a hoarder. Not a hoarder of material possessions — oh no, my house is almost entirely free of unnecessary stuff. I take pride in actively reducing the amount of physical crap that I own, and donate items I no longer need. My family and I have even made a gift-giving agreement among ourselves where any gifts must either be consumable (specialty foods, restaurant gift cards, etc) or experiences (tickets to a show, subscription to a service, etc).

My hoarding, on the other hand, is of the digital variety. My “collection” only spans a few external hard drives of 1 TB each, occasionally backed up or synced to a duplicate set of external hard drives. The physical space occupied by these drives is less than one cubic foot, but the vastness of the digital stuff that they contain is… considerable. Just to give you a rough idea of what I’m dealing with:

  • Archives of old emails and correspondence from previous jobs and contracts.
  • Archives of Instant Messenger conversations with old friends and ex-girlfriends, dating back to 1997.
  • A collection of viruses and trojans for MS-DOS from the 80s and 90s, originally for research purposes.
  • A huge library of shareware games and programs from the MS-DOS era.
  • An archive of articles, papers, and textbooks (in PDF form) relating to computer science, mathematics, and physics.
  • An archive of high-resolution NASA imagery of planets, nebulas, and galaxies.
  • An extensive library of file formats (i.e. sample files saved by all kinds of different software) and documentation for every file format specification.
  • Emulators and system images of virtually every computer system and game console ever built.
  • My complete genome, which I’ve had sequenced a few years ago.
  • And of course, my personal photo and video library, from my birth to the present day, and also photos from my parents’ and grandparents’ old albums that I have digitized and saved.

So yeah… recently I’ve been asking myself whether digital hoarding is a problem of the same magnitude as physical hoarding. On the surface, one might ask “What’s the problem?” These things aren’t taking up any physical space, and you don’t have to give them another thought after you save them to the disk. And yet, perhaps there is an emotional toll that comes with the mere knowledge that all of this old data still exists, and remains your responsibility. If anything, this is surely at odds with my attitude towards physical possessions, which is quite minimalist.

A basic litmus test for hoarding behavior consists of a simple question: How would you feel about throwing away any random item that you see around you? Will you use this twisty-tie for anything? How about this pen cap? Do you really need three different cheese graters? How about this pile of old magazines? A hoarder will answer these questions with something like, “You never know when it will come in handy.” And if I’m being honest, that’s exactly how I feel about all the digital items I listed above. I feel the same hesitation about deleting any of them as a “physical” hoarder might feel about donating old clothes, or throwing away expired spices from the pantry.

When I look at my enormous pile of digital junk, I see in myself all the symptoms of real hoarding, albeit confined to the digital realm, which is likely why it’s been able to go on for so long. I’m also reminded of how freeing and cathartic it feels to let go of unnecessary possessions, and I theorize that a similar feeling of freedom will result from permanently letting go of digital baggage.

Therefore, I have resolved to stop being hypocritical in this regard, and start practicing digitally what I practice physically.  Many of the items I mentioned in my list are actually replaceable (easily found on the web, or generated with minimal effort).  Some of the items are technically “irreplaceable,” such as my old emails and IM archives, but represent unnecessary cognitive and emotional baggage, and have no real nostalgic value.  The only things that seem to have actual meaning, and are objectively worth keeping, are my personal photos and videos, and even those can probably be trimmed down a bit.

It’s time to let the past go, and embrace the future without anything weighing you down. Let the cleaning begin.

A quick utility for SQLite forensics

When performing forensics on SQLite database files, it’s simple enough to browse through the database directly using a tool like sqlitebrowser, which provides a nice visual interface for exploring the data. However, I’d like to create a tool that goes one step further:  a tool that shows the contents of unallocated or freed blocks within the database, so that it’s possible to see data from rows that once existed, but were later deleted (this can be used, for example, in recovering deleted text messages from an Android device, which usually stores SMS messages in a .sqlite file).

This new utility, which I’ll tentatively call SqliteCarve, represents the minimum solution for accomplishing this task:  it loads a SQLite file, and parses its pages and B-tree structure. While doing this, it detects the portions of the structure that contain unallocated bytes. It then reads these bytes and parses any strings from them.

The tool presents all the strings found in the unallocated space visually, with a quick way to search for keywords within the strings:

A couple of TODOs for this utility:

  • Support strings encoded with UTF-16 and UTF-16BE, in addition to the default UTF-8.
  • Make better inferences about the type of content present in the unallocated areas, to be able to extract strings more precisely.