Hand Mouse

C++ / OpenCV / MFC
Computer Vision-Based Hand Mouse – Created a hand motion recognition tool that captures hand gestures, extract features, and analyze/recognize the intended user’s activity by using C++ and Open Source Computer Vision library (OpenCV)

1. Pre-processing

Before Pre-processing
YCrCb Image
Binary morphology
After Pre-processing
Pre-processing process for hand recognition
  • 1.1 GaussianBlur

    The GaussianBlur is performed to remove unwanted noises from the image coming through camera that the smoothly filtered image.

  • 1.2 Color filter (RGB -> YCrCb)

    After receiving image with RGB color model, the color filter converts it into YCrCb image.

  • 1.3 Binarization (Binary Image)

    Binarization recognizes and distinguishes objects by dividing into black and white using threshold that distinguishes black and white.

  • 1.4 Binary morphology

    It is used for pre-processing such as noise elimination or feature extraction using erosion and expansion calculation.

2. Feature extraction

ROI - GaussianBlur
ROI - Morphology
ROI Area
Region segmentation and feature extraction for identifying hands
  • 2.1 Contour detection (findContours)
  • 2.2 Hand contour drawing (drawContours)
  • 2.3 Contour area

    It calculates the area inside the contour -> The largest area is recognized as hand -> Save large area only

  • 2.4 Drawing square, minimum square, circle based on hand contour
  • 2.5 Central point detection of hand (moments)

    It calculates the center of weight central point by calculating the weighted mean

  • 2.6 ROI (Region of Interest) processing of hand region
  • 2.7 Central point detection of palm (ROI area)

    After drawing a circle with a consistent length based on the center of gravity, count the number of the objects and calculate the number of fingers by holding the furthest coordinate as fingertip point from the center of gravity among the boundary coordinates of the object (finger) outside the circle.

  • 2.8 Convex hull, minimum-size polygon including given all dots
  • 2.9 Convex hull, Convex defect -> Start point, End point

3. Motion detection

Color Selection
Stop drawing
Delete drawing
Operation principle and design for motion detection
  • 3.1 Motion

    1. Index finger : Mouse Pointer
    2. Palm : Drawing / Stop Drawing
    3. Fist : Delete Drawing

  • 3.2 Color Selection

    Move the fingertip point without moving hands.

  • 3.3 Drawing

    Utilize unfolding the palm of hands as a trigger.
    1. Unfold the thumb with an angle of ninety degrees. (recognition as a palm)
    2. Drawing from the moment that unfolds index finger (or one finger)

  • 3.4 Menu

    Recognize the moment of clenching the fist as a trigger.
    Menu should be selected if moving a certain distance from the center coordinate.
    1. Clench the fist.
    • A cross mark is displayed and divided into quadrants.
    • Draw menus in each quadrant (4 menus)
    • Basic status / Clear screen / Drawing / Pen thickness
    2. Largely move to the direction of the desired menu.
    3. Menu selection is completed when moving over a certain distance.
    When a menu is selected, a message is shown.

Program UI

Face recognition and tracking

It is detected with Haar Cascade Classifier using Cascade Training data provided by OpenCV.
However, since the CPU can not achieve the desired performance, the Motion History is used to compare the previous image with the current image, and only the area (red square area) that is updated on the screen is calculated. Face detection using the most hardware resources in the program is handled by CUDA operation in the GPU.
CPU usage before and after improvement
GPU usage has been reduced by half after applying Motion History.
Camshift was used for face tracking. Whenever a new face is recognized, the tracking area is adjusted to improve accuracy.
  • OS : Windows 10 (64bit)
  • IDE : Visual Studio 2015
  • Language : C++
  • Library : OpenCV 3.1.0 / MFC
YoungEun Lee, SeungRi Kim