06/Worklog

LocalSubs: On-Device Real-Time Subtitle Translation

A fully local English-to-Taiwan-Traditional-Chinese subtitle translator, built around a workload-optimized 0.6B model and a Chrome-to-native inference path.

LLMMachine LearningGoInfra / Reproducibility
Year
2026
Role
Solo Open-Source Research Project
Status
Released
Code/Demo
aiden1020/localsubs
Problem

Streaming subtitles arrive as short, sequential, batch-size-one requests that must finish before the cue disappears. General-purpose translation models carry large multilingual vocabularies and cloud dependencies that are poorly matched to this latency-sensitive, privacy-sensitive workload.

Approach

Replaced the base model's 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, migrated the embedding space, then applied embedding calibration and full supervised fine-tuning. The released system connects a Chrome extension to a local llama.cpp runtime through Native Messaging, keeping every subtitle cue on-device.

Deployable quality and latency
91.6%
Of GPT-4o mini's shared-anchor score
56.8ms
Mean inference latency on Apple M2
1.63×
Speedup over 151k-vocab baseline
233,088
Quality-filtered SFT examples
End-to-end deployment boundary
A tokenizer designed around the workload

Original multilingual model

151k vocabulary tokens
1.05 Chinese characters / token
~596M parameters

LocalSubs

64k vocabulary tokens
1.40 Chinese characters / token
~506M parameters

The smaller vocabulary cuts the output-projection dimension by 57.9%. Denser Chinese tokenization also reduces the decode steps needed for the same subtitle text.

Model adaptation strategy
Embedding migration
Existing token strings copy their embeddings directly; new tokens initialize from the mean of their original-tokenizer sub-token embeddings.
Embedding calibration
Transformer layers stay frozen while embeddings, the output head, and RMSNorm adapt the new token interface to the existing representation space.
Full supervised fine-tuning
The full model then trains on 233,088 subtitle-domain examples for Taiwan usage, concise cue style, and up to three preceding context cues.

Read the full technical report

Notes
  • Reached 91.6% of GPT-4o mini's shared-anchor win-rate score on the same subtitle translation evaluation (59.2% vs. 64.6%).
  • Reduced average Apple M2 inference latency from 92.7 ms to 56.8 ms, a 1.63x system-level speedup.
  • Reduced vocabulary projection width by 57.9%, from 151k to 64k tokens, while increasing Chinese character density from 1.05 to 1.40 characters per token.
  • Filtered 14,237,823 raw subtitle pairs into 233,088 context-aware SFT examples for Taiwan Traditional Chinese.
Skill roles
LLM

Adapted a 0.6B translation model to short contextual subtitle cues through tokenizer replacement, embedding migration, calibration, and full SFT.

Machine Learning

Built a 233,088-example quality-filtered SFT corpus and evaluated translation quality with randomized pairwise judgments, context ablation, and cue-length analysis.

Go

Implemented the native host and CLI lifecycle for setup, model downloads, browser registration, diagnostics, and on-demand inference process management.

Infra / Reproducibility

Packaged the model as GGUF Q5_K_M, integrated llama.cpp with Apple Metal, bridged Chrome through Native Messaging, and distributed the system through Homebrew and the Chrome Web Store.