3D-LEX Semi-Automatic Annotation of Phonemes in Sign Language

This repository contains the code to perform semi-autmatic annotations of phonetic properties based on the 3D-LEX dataset, and to reproduce the results of the paper: 3D-LEX v1.0: 3D Lexicons for American Sign Language and Sign Language of the Netherlands.

Experiments currently available for the following phonetic classes:

Handshapes

Distribution of Handshape labels This figure showcases the distribution of new handshape labels produced using a k-means clustering algorithm on the StretchSense glove data for 1000 glosses, as compared to the distribution of handshape labels provided by expert linguists (ASL-LEX 2.0).

Time-series visualization of handshape classification: Classification of the ASL sign "zero" with expert label "o". The bars denote captured frames, triggered by positional shifts, with colors denoting handshapes identified by the Euclidean distance method. Our segmentation pipeline identifies handshapes "5", "f", "c", and "o", selecting frames corresponding to "o" as the characteristic signal of "zero".

t-SNE projection: A projection of average hand poses to two dimensions from the selected temporal ranges, color-coded with a k(=50)-means cluster label for visualization purposes.

Evaluation on the isolated sign recognition task

a1N	a1E	a1A
0.44±0.01	0.48±0.01	0.49±0.01

Top-1 recognition accuracy: Accuracy using no (N) handshape labels, expert (E) labels, and automatic (A) labels. The accuracies are averaged across 8 runs. Standard deviation across measurements is provided in the subscripts.

Setup (Linux)

1. Download 3D-LEX dataset (instructions coming soon)
2. Download the WLASL/SemLEX data files (instructions below)
3. setup environment:

bash setup/setup.sh
conda activate saa

Datasets

3D-LEX

Data will be made available upon publication of the associated paper.

Evaluation: WLASL & SemLEX

Evaluations of new phonetic labels are currently available with WLASL 2000 ISLR benchmark (Li et al., 2020), which has been combinet with the phonological annotations in ASL-LEX 2.0 (Sehyr et al., 2021) in Kezar et al., 2023. Please obtain the metadata for the merged ASL-LEX and WLASL benchmark from the original repository (Kezar et al., 2023).

For performing the evaluations on an ISR task please download the WLASL pose data from the openhands repositroy or by running the following commands:

mkdir data/wlasl/
wget https://zenodo.org/record/6674324/files/WLASL.zip?download=1
mv WLASL.zip?download=1 data/wlasl/
cd data/wlasl/
unzip WLASL.zip?download=1
rm WLASL.zip?download=1

Evaluations on SemLEX will be available soon.

Demos

Phonetic Labeling with StretchSense Gloves

To produce new handshape labels run either

python main_ED.py --mode annotate
python main_KMeans.py --mode annotate

Evaluation

To perform evaluations using openhands on the wlasl benchmark, please run:

cd evaluation/isolated_sign_recognition/
python train.py
# Edit model path in evaluation/isolated_sign_recognition/configs/gcn_test.yaml
python test.py

Citation

Coming soon