Experiments in Machine Vision

In between technical writing contracts I write Android apps. My first was a free STL file viewer which is now used by thousands of engineers, modellers and designers, all over the world. I write mainly in Java, dropping into C/C++ where speed demands or to use a library like OpenCV.

My current project is in the field of machine vision. This is a dynamic and growing area of work, used in automotive systems, robotics, the military, medicine, manufacturing and a wide range of other sectors.

My most recent Android app (now version 2.0) uses an open source machine vision library, OpenCV, to process frames from the device's video camera. The app takes the video feed frame by frame, processes the data and overlays the results onto the frame before displaying it on the screen. Scan the QR code here to go to the app on Google Play Store.

The app displays the results of the following processes:

  • Canny edge detector / Hough circle detector / Hough line detector
  • Colour range: contour outline
  • Colour range: quadrilateral search
  • Face detection
  • Swap camera (if second camera is available)
  • Good-features-to-track.

When viewing the video feed in any of these modes, the user can grab a screenshot at any time by just touching the screen, which is saved to the SD card. You can also toggle on and off the title and FPS display.

Source Code

I offer snippets of source code here. The code is based on the sample apps which come with the Android OpenCV 2.4.4 library. The snippets I provide can be dropped into those apps with a bit of work to define some added variables and support methods. If you would like a free copy of the entire source code, click here.

Click on any thumbnail to see the original screen capture. The images were taken from a Nexus 4 (1280 x 768 resolution) using the front camera.

Frame rate: The frame rate is displayed on the screen when viewing the video feed. Some of these processes are very computation-intensive and your Android device is unlikely to be as fast as your desktop computer (unless the latter was built in 1990). However, in all modes the rate improves as the lighting of the scene gets brighter. Use the app in the best lighting available and the frame rate improves significantly.

Canny Edge Detector

Many processes and tools in the OpenCV library use an edge detector process as their first step. When the edges in the video frame have been identified, the features in that image can more readily be identified and used. This mode simply uses the Canny algorithm to process each frame and displays it on the screen.

Hough Circle Detector

The Hough algorithm is used here to detect circular features in the (Canny processed) frame, and then draws a red circle at that location and shows the centre of the circle with a cross. The maximum number of circles outlined is 10 but this can be changed in the code.

Hough Line Detector

Colour Range: Contour Mapper

This process converts the (BGR) frame to HSV, finds all areas of the image where the hue is more or less yellow (values are 'in range'), creates a bit mask of those areas, finds the contours of the areas, creates an approximate polygon for each and overlays them on the original image.

Colour Range: Quadrilaterals

This process carries out the same process as that above, but at the final step checks the contours to see if any are quadrilateral. If so it outlines the contour and draws the diagonals.

Face detection

The Haar cascade used in the app recognises only frontal views, not profile. Having identified a number of faces in the frame the app draws a rectangle over the main features. The largest face is copied to the top left of the screen. If that face is markedly different to the previous largest face it is saved to the SD card of the device. As you scan around a number of faces, you should get one or two saved copies of each face seen.

Good-Features-To-Track

This process is used as first step in a number of other MV procedures. The frame is searched for 'corners', prominent features of the image which are likely to be strong candidates for things to track from one frame to the next, or used when calculating optical flow.

Optical Flow

Uses Optical Flow to compare strong features in each frame with those in the next, then draws a line between matching points in each frame, overlaid on the most recent image.