Algorithms

Hand segmentation

Vibe

• ViBe background subtraction to adaptively learn the keyboard background and separate the hand as foreground.
• Idea is to keep a few past "frames" made of a non deterministic selection of points from the past, where each point may also be updated by surrounding pixels (spatial and temporal awareness).
• Unfortunately, we either can't deal with a camera shake, or if we try to deal with it, we end up with the hand becoming a part of the background if it stays still.
• Attempted to do smart adaptive bgsub, which reinitializes the background if more than 50% of the image is foreground. This works slightly better, but is still an issue if the camera is moving often (if user is typing and screen hinge is weak).

Skin tone changing

• Some webcams do weird hue changes which makes skin detection difficult. However, in a constrained case, this works very well.

Orientation gradients

• Attempted to get orientation gradients and look for curved objects (fingertips)
• Can't think of a way to get just the curved edges.
• However, we could do very good ML with this, and it's very fast.

Textons

• Not attempted, may help

Key blocking

• Detect all the keys in a keyboard, and then check which ones are blocked to get some idea about the fingertip location.
• Main issue: finding squares is not trivial, especially since they're not always closed, may be various colors, etc.

Palettization

• Get the palette of colors in a reference image, then find things that don't fit in that palette later.
• Theoretically sound, but palettization is slow :(
• Haven't tried implementing entirely, will try

Fingertip locator

Specifically for finding the tips of your fingers, once the hand segmentation has been abstracted away.

Fast March Method

• Start at the boundary of the image, then "march" inwards until you hit what's called a 'collapse of a boundary segment'. It seems this means when the "marchers" from two different boundary segments collide into each other, forming a skeleton of the object.
• Seems promising, but not clear how to extract the tips from the skeleton, or if the skeleton will be ideal.
• Paper

SimpleCV skeletonize

• Likely employs something similar but inferior to FMM.
• Works decently well, can use this to get finger tips.