Sign Language–Based Notepad Op. 3

I. A place for expression

The project explored a note-taking tool for Deaf users, bringing drawing and sign-language input into one interface. Its starting point was a use for recognition: helping someone capture a thought as a note.

II. Two ways to make a note

In drawing mode, tracked hand position controls the mark, while hand openness and distance influence its radius. In sign-language mode, camera-derived features and sensor data feed a recognition and text-generation pipeline. Buttons on the two SPIKE hubs start and stop data collection.

III. Connecting the system

Sensing

Camera, MediaPipe Holistic, two LEGO SPIKE hubs

Transport

Raspberry Pi 3 Model B+, Bluetooth, TCP sockets

Model

MSKA-based vector and IMU fusion, gloss prediction, mBART text generation

Interface

Drawing and text modes in a desktop notepad

IV. Model architecture

The model adapts Multi-Streaming Keypoint Attention (MSKA) to combine body and hand vectors with IMU features. MediaPipe Holistic extracts 67 landmarks per frame: 21 for each hand and 25 for the body.

The architecture uses self-attention, feed-forward layers, and temporal convolution to encode the feature streams, then concatenates and linearly fuses their representations. A CTC head predicts glosses, while an encoder projection connects the fused features to an mBART decoder for text generation. Training combines text-generation loss with a weighted CTC loss.

V. Dataset & reported result

Dataset

RWTH-PHOENIX-Weather 2014 T · German Sign Language

Split

7,096 training · 519 development · 642 test samples

Glosses

1,086 vocabulary entries

Training

70 epochs · vector and IMU features derived from video landmarks

Reported

67.79 BLEU

The final presentation reports 67.79 BLEU under “Training Result.” BLEU is a translation metric, not a percentage accuracy. The slide does not specify the evaluation split or BLEU variant.

VI. Scope & limitations

The motivation discusses Korean Sign Language, while the experiments use RWTH-PHOENIX-Weather 2014 T, a German Sign Language dataset. These experiments do not establish Korean Sign Language translation performance.

Training features derived from video and sensor readings collected during use are different input sources. Their alignment and evaluation on live input remain important questions for this prototype.

VII. Contribution

I worked on the sensors, communication, and hardware construction.

VIII. Team

주준영 · 허제현 · 탁한진 · 한유정 · 황윤정