In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents.
Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.
From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.
From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.
This was my first thought, too (after “which sign language?”).
That said, the video does show the model identifying facial and other body features, so it’d be interesting To know if this model _could_ be capable of working with a full sign language (ASL or otherwise).
It also got me thinking about how LLMs might handle signs as tokens, and it seems you did too! Would you just detect unique handshapes and hope the LLM could do the rest? Could you somehow encode orientation/movement/direction/locations and facial expressions? I presume current LLMs don’t have much training input on this kind of language - would we need to train new LLMs to even have a hope of them “understanding” 3-dimensional language?
fingerspelling != sign language
In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents. Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.
From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.
From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.
This was my first thought, too (after “which sign language?”).
That said, the video does show the model identifying facial and other body features, so it’d be interesting To know if this model _could_ be capable of working with a full sign language (ASL or otherwise).
It also got me thinking about how LLMs might handle signs as tokens, and it seems you did too! Would you just detect unique handshapes and hope the LLM could do the rest? Could you somehow encode orientation/movement/direction/locations and facial expressions? I presume current LLMs don’t have much training input on this kind of language - would we need to train new LLMs to even have a hope of them “understanding” 3-dimensional language?
anything but learn sl
You could make the same argument about spoken languages. There isn't a single sign language