Machine Learning with MXNet to Recognize Household Appliances

I trained convolutional neural networks on images of household appliances to simulate a home inspector with machine learning.

The end result is a video like this:

When you purchase a house, you pay $400-$800 for an inspection. The inspector walks through the house looking for defects in the home, out of date construction techniques, environmental hazards, and the like. They then write a report, which is effectively a manual for your home.

The inspection is primarily a visual process. An experienced inspector recognizes many rare or old objects. Essentially, this process is a reverse image search – many, though not all of the items could be recognized from photos.

When I began this project, I was unable to find a pre-made dataset to match my needs. While there are many datasets for testing machine learning, most are challenge datasets (i.e. used to test algorithms), lacking the quality or edge cases required to solve a business problem.

Data Setup

I built a simple app to record stock videos with WebRTC, select a tag, and upload them to S3. Files in S3 are stoed in folders named for the tags (washing_machine, sump_pump, furnace, etc).

The training script downloads the videos, extract frames with ffmpeg, applies pre-processing, then does transfer learning with Apache mxnet / gluon.

mkdir -p /data/videos
aws s3 sync --delete s3://gsieling-video-bucket /data/videos
find /data/videos

/data/videos
/data/videos/stairs
/data/videos/stairs.VID_1mp4
/data/videos/stairs/VID_2.mp4

It also takes a significant amount of trial and error to set up an effective training process.
It’s important to independently debug and test each stage, to avoid costly mistakes. When I began the project, I trained an algorithm on 2-3 objects, using a pre-made model called MobileNet. This trains in minutes on a CPU, and allowed me to validate my approach. With a more robust dataset of tens of thousands of images, training on a CPU takes days.

I did most of my work in Apache Zeppelin notebooks (see my previous post on why I chose Zeppelin). Once the codebase was working and stable, I moved to a standalone training script.

The tags this tool learns are: breaker_box, furnace, hot_water_tank, oil_tank, pillar, smoke_detector, stairs, sump_pump, wall, washer, and water_softener. You get better results with more tags. Object classification models always output a tag (which thing it thinks is most likely) – i.e. there’s no clear way to say that it sees nothing.

Pre-trained models are a good starting point, and can be obtained from Apache gluon’s model zoo. These have been trained on some existing dataset of images. We just change the tags they output and tune the model to detect our tags.

There is some data setup required to do the actual training (see the full script here). For low-accuracy models and a few hundred images, this trains in minutes, which is enough to prove out that the problem is possible.

Setting up all of the dependencies turns out to be a significant amount of work (i.e. mxnet, opencv, CUDA). It is really valuable to dockerize as much as you can – NVIDIA provides a Docker runtime that helps with this. That being said, their tooling is exceptionally difficult to set up.

After training, we then run the model against all of the images in a video. Shown below is a notebook the views the video frame-by-frame in our notebook, so we can compare the results of different models:

Decision Points

mxnet

I chose mxnet because AWS seems to be pushing it. For instance, their DeepLens camera comes with it pre-installed.

It has bindings for multiple languages – I like having the ability to convert python code to Scala as my project matures.

MobileNet, ResNet, or VGG16?

The mxnet documentation includes a visualization of model training time vs. accuracy. Models on the outer edge of the curve express the trade-off between accuracy and training time.

When I started this project I used Mobilenet, which trains quickly, but isn’t super-accurate. Once I got to the point where I wanted to produce a demo, I used other models.

It’s worth nothing that these are not all trained with the same image datasets. Each model has different behaviors when they glitch out (e.g. mobilenet defaulted to thinking everything was a sump pump, and could never detect a wall).

It’s also worth noting that while this chart denotes “samples per second”, the models don’t seem to learn information at the same rate – VGG16 consumed images very slowly, but seemed to still improve in accuracy very rapidly.

Image Sources

If you build an image dataset, you can either collect the images yourself or crawl what’s publicly available. For this project I took videos in a couple different houses.

Before landing on this approach, I evalauted the Bing Image API. While this didn’t give me great results, it did help me understand the problem I’m trying to solve much better.

If you search for a term like “cherry flooring”, you quickly see a lot of similarity in the images. Search engines try to find results closest to the search query, which doesn’t give you a lot of diversity in the results.

These houses also appear to professional photos of houses be staged for sale. Our use case is taking cell phone videos as you walk around a home, which is unlikely to ever produce an image of similar quality.

Images of smoke detectors show a different class of problem – these are all stock photos (i.e. no backgrounds), without much variety of angles.

These are also product photos, which suggests they are likely items currently for sale. If you buy a house, you’re more interested in products that are no longer on the market, for instance items subject to recalls.

Environmental hazards also present interesting challenges. Shown here are images of asbestos fabric, which used to be a common way to insulate pipes, as it doesn’t burn.

From a distance it just looks like a white paper / fabric. You can tell what it is from the texture, but this is unlikely to be visible in a cell phone image.

Other asbestos items can only be recognized through a product database, e.g. 9×9 tile that dates from a certain time period, but without having an object of a known size in the images, we don’t know how big anything is, whereas a home inspector would have a tape measure.

Fire damage presents an interesting machine learning challenge – this could be detected, but it’s really a change from the natural state of a room. Most of these images online represent total destruction, which is unlikely what you’d see if you bought a house that had some damage.

Unlike other scenarios, we also can’t easily generate our own training data.

3D Images

A third interesting technique for generating images is to create 3-D models, then virtually photograph the objects we care about.

Shown below is an image I generate from stock videos of a sump pump.

The photogrammetry software did a great job rendering the basin around the pump, but lost the top. I found generally that it loses smooth, solid color surfaces, making it a poor choice for rendering appliances.

However, there are depth sensors on the market (e.g. the Intel Realsense devkit cameras). If these make their way into phones, it may become possible to generate a higher quality renderings.

Classification, Object Detection, or Segmentation?

One of the major choices in machine learning is what you want the machine to “learn”. Classification gives you the most likely label for each image (whether or not something is really there). Object detection gives you bounding boxes, and segmentation gives you a boundary line.

Each of those requires successively more detailed training data.

Aside from the labor cost of generating the data, there are is a clear reasons to prefer classification for this problem.

As seen below, many household objects will be take up the entire frame – pipes, wiring, and framing. It would be difficult to draw segmentation lines around these.

A future technique that virtually photographs 3D models might make it possible to auto-generate segmentation lines. Photogrammetry tools create point clouds from images, and meshes from the point clouds – so tis assumes that either the points or mesh triangles can be labelled.

Alternately, one could build and run multiple models of different types, or a model with a combined architecture – one part to do classification on things that can only be classified, and one part to produce bounding boxes where possible.

Debugging Tools

Notebooks are ideal environments for the initial phases of a machine learning project. You can construct a small dataset, write a paragraph for each step, and carefully test each piece. Because each paragraph has access to the global state, you can start running from any point.

Notebooks also allow you to build small debugging tools into your process, which we’ll discuss here. The following examples use Apache Zeppelin, which lets you log a macro that tells what kind of output your paragraph produces (%table gives you an excel-like charting tool, %html assumes the output is html, and can include base 64 encoded images).

Zeppelin also lets you create UI controls (checkboxes).

Spot checking tags

After we’ve pulled a bunch of data, we want to make sure the frames extracted from the videos are being placed in the right locations (i.e. that a folder named “washing machine” in fact has washing machines).

To check this using our notebook, we output a combobox containing a list of tags. When the notebook operator selects a tag, it posts back to the notebook and chooses 25 randomly selected images from the folder, base64 encodes them, and displays the HTML output:

Dataset size

We also want to see stats on the size of our dataset – is it balanced? Are any of the tags completely empty or lacking data?

Processed Image Preview

Before we hand images off to the machine learning algorithm, they go through a series of pre-processing steps. They must be resized down to 224 x 224. This can be done by resizing or by cropping. We also apply filters to discourage the algorithm from becoming too sensitive to lighting conditions.

Tag an image

After the algorithm finishes we want to test some example tags on images.

Confusion Matrix

A confusion matrix tells you which tags the algorithm mixes up.

Read a TSV file