Skip to content
TECAA

Case study // 2026

Extended Reality Interfaces for Speech Assistance in People with Non-Functional Hearing

Live subtitles, translation and fingerspelling on HoloLens 2.

Year
2026
Role
Graduation project, team of 3
Areas
XR // CV // NLP
Stack
Unity, C#, AR Foundation, UWP, Python, PyTorch, CTranslate2, FastAPI, Hugging Face Spaces

The app, HoloAssist, runs on HoloLens 2 for people with non-functional hearing. It shows live subtitles from speech, translates between English and Italian in both directions, and reads ASL fingerspelling through the headset's own camera.

It was our FENS 402 Engineering Design Project at Kadir Has University, supervised by Prof. Dr. Fabio Stroppa.

The system

Speech to subtitles

  1. Voice

    Cloud ASR plus on-device Italian WinRT recognition

  2. Raw PCM

    Strict audio contract

  3. Fine-tuned ASR

    Served from a Hugging Face Space

  4. MenKan

    EN ↔ IT translation

  5. Subtitles

    World-space UI Toolkit panels

Fingerspelling

  1. Headset camera

    Frames from XRCpuImage

  2. Hand region

    Detection and projection

  3. Inference service

    Letter classifier on its own Space

  4. Spelled output

    With confidence and history

The numbers

MenKan translation quality on FLORES-200 devtest
DirectionBLEUchrF++
English → Italian26.9256.16
Italian → English33.6861.15

NLLB-200 (distilled 600M), fine-tuned with LoRA and quantised to INT8 for CTranslate2. English → Italian scored on 1,000 devtest sentences, Italian → English on 200.

The headset client

The client is Unity with MRTK and AR Foundation. Panels are UI Toolkit documents placed in world space, so subtitles stay readable without covering the speaker.

A hybrid voice manager uses cloud speech recognition and falls back to on-device Italian recognition through WinRT.

MenKan, the translation engine

MenKan (Bambara for “hear and understand”) is the translation component, which I built. It's NLLB-200 distilled 600M, fine-tuned with LoRA, quantised to INT8 for CTranslate2 and served by FastAPI.

The evaluation goes beyond BLEU: part-of-speech and formality confusion matrices, word-sense experiments, and paired significance tests (Wilcoxon, bootstrap, chi-square).

  • Language-tag-safe tokenisation with SentencePiece
  • A validated API with clear error codes
  • Docker image, plus latency and TCP batching benchmarks

Reading fingerspelling

Frames come straight from the HoloLens camera through XRCpuImage. The hand region is detected and projected, sent to the fingerspelling classifier (Aiana Mederbek kyzy's Sign-Language Space on Hugging Face), and the spelled result comes back with its confidence and a running history.

Native vs WebXR

We compared the native HoloLens Unity build against a WebXR build on frame rate and p95 frame time, with statistical tests on the frame data. It was one recorded run per platform, so it is a first measurement rather than a benchmark.

Credits

Team: Hadiza Muhammed Avidime, Ahmed Marcolino Teca Kanadji, Aiana Mederbek kyzy. Supervisor: Prof. Dr. Fabio Stroppa, Kadir Has University.