A unified framework for gesture recognition and spatiotemporal gesture segmentation.

Full Abstract

Within the context of hand gesture recognition, spatiotemporal gesture segmentation is the task of determining, in a video sequence, where the gesturing hand is located and when the gesture starts and ends. Existing gesture recognition methods typically assume either known spatial segmentation or known temporal segmentation, or both. This paper introduces a unified framework for simultaneously performing spatial segmentation, temporal segmentation, and recognition. In the proposed framework, information flows both bottom-up and top-down. A gesture can be recognized even when the hand location is highly ambiguous and when information about when the gesture begins and ends is unavailable. Thus, the method can be applied to continuous image streams where gestures are performed in front of moving, cluttered backgrounds. The proposed method consists of three novel contributions: a spatiotemporal matching algorithm that can accommodate multiple candidate hand detections in every frame, a classifier-based pruning framework that enables accurate and early rejection of poor matches to gesture models, and a subgesture reasoning algorithm that learns which gesture models can falsely match parts of other longer gestures. The performance of the approach is evaluated on two challenging applications: recognition of hand-signed digits gestured by users wearing short-sleeved shirts, in front of a cluttered background, and retrieval of occurrences of signs of interest in a video database containing continuous, unsegmented signing in American Sign Language (ASL).

Author information

Author/s: Alon, Jonathan (J); Athitsos, Vassilis (V); Yuan, Quan (Q); Sclaroff, Stan (S);

Affiliation: Computer Science Department, Boston University, Boston, MA 02215, USA. jalon(-atsign-)cs.bu.edu

Journal and publication information

Publication Type: Journal Article; Research Support, U.S. Gov't, Non-P.H.S.

Journal: IEEE transactions on pattern analysis and machine intelligence (IEEE Trans Pattern Anal Mach Intell), published in United States. (Language: eng)

Reference: 2009-Sep; vol 31 (issue 9) : pp 1685-99

Dates: Created 2009/07/03; Completed 2009/10/06;

PMID: 19574627, status: MEDLINE (last retrieval date: 10/6/2009, IMS Date: )

Sourced from the National Library of Medicine. Abstract text and other information may be subject to copyright.

External Links for this article
(including full text providers, if available):

Click Electronic Full-text Provider Links to see options for finding the electronic full text links to this article. Note there may be a subscription or fee required for access to the full text. See our FAQ for information on finding FREE full text articles.

This article may also be located in paper journal collections available in many libraries. Use the Journal and Publication Information above to find the full article.