2
Models
pre-trained tracker plus trained recogniser
academic · 2025
Write in the air with your finger and get digital text. A contribution to an existing open-source computer vision project, extended and presented at university.
2
Models
pre-trained tracker plus trained recogniser
Local
Processing
camera feed never leaves the machine
Academic
Context
contribution to an open-source project
The problem
Turning a fingertip traced through the air into recognised text joins two machine learning problems that do not naturally connect. Hand tracking produces a noisy time series of coordinates; handwriting recognition expects an image in the distribution it was trained on. The interesting work is the conversion between them.
The approach
Took an existing open-source project and worked on top of it, extending the pipeline and presenting the result academically. The architecture combines a pre-trained hand landmark model with a handwriting recogniser trained on a standard academic dataset augmented with synthetic data, since off-the-shelf recognition is trained on scanned documents rather than thin single-stroke characters drawn in the air. The middle step is the hard one: a fingertip trajectory has to be smoothed against frame-to-frame jitter, resampled for frame-rate independence, and rendered and normalised into the canvas the recogniser expects.
Architecture
Two models in sequence: a pre-trained landmark detector for hand tracking, a purpose-trained recogniser for characters. Everything runs locally, so the camera feed never leaves the machine.
Each with what it cost
A decision without a stated tradeoff is a preference.
01
Why
Hand landmark detection is solved and freely available. Recognising air-drawn strokes is not, because the input distribution is unlike any scanned handwriting dataset. Buy what is solved, build what is novel.
What it costs
A dependency on someone else model for the first stage, which is the correct trade against months spent reproducing it.
02
Why
Real handwriting datasets are pen-on-tablet: steadier, with pen-up and pen-down signals air-writing does not have. Synthetic augmentation covers the gap the real data under-represents.
What it costs
Synthetic data can encode its own assumptions, which the validation split is there to expose.
Outcome
A real-time pipeline from webcam to recognised text, with reported accuracy on both training and validation splits rather than a single headline number.