Reads data from @Bluebie/NZSL-Dictionary dataset, and uses ffmpeg to extract video frames, posenet & tensorflow.js to classify poses, and some manual filtering to assess quality. At the output stage, images of features like hands and faces can be extracted from the videos and labeled with metadata from the NZSL-Dictionary dataset like location and handshape.
This big mess is intended to be a tool for generating training datasets to teach a convolutional neural network to recognise BANZLAN sign language handshapes from video. I hope it can be useful to analise the linguistic features of related languages like Auslan and BSL, or build better models for computers to communicate directly using sign languages.
video-reader.js: a simple ffmpeg interface, to convert video files in to a stack of PNG images in a temporary directorypose-machine.js: a wrapper around PoseNet, with some aditional filtering to improve quality signals and reject bad classificationsnzsl.js: a really simple model to read in data from the NZSL-Dictionary dataset, and conveniently access related videos and image filesextractor-bot.js: the main script, which queries the NZSL-Dictionary dataset and bulk exports labeled training samplesrun-extractor-bot.js: example of extractor bot in usetest.js: a really bad test script that just pokes a few apis so i can see they're working, desperately in need of replacement with a propper testing system