Why a 0.6B Model Still Felt Slow

When I started building LocalSubs, the performance plan seemed straightforward: use a small language model, run it locally on Apple Silicon, and translate each subtitle before the next one appeared.

I chose a 0.6B-parameter model. Compared with most modern language models, it was tiny. Subtitle outputs were also short, so I assumed inference speed would be one of the easier parts of the project.

It was not.

The backend mattered more than I expected

Since LocalSubs targets macOS, MLX was my first choice. It is designed specifically for Apple Silicon, so I expected it to be the fastest option—or at least close enough that the native ecosystem would make it the obvious choice.

I tested MLX and llama.cpp using the same subtitle workload:

Backend	Quantization	Mean	P50	P95	Max
MLX	8-bit	163.54 ms	151.50 ms	208.22 ms	313.43 ms
llama.cpp	Q8_0	72.79 ms	59.05 ms	126.71 ms	236.42 ms

For this workload, llama.cpp was about 2.25 times faster on average. The median latency fell from 151.50 ms to 59.05 ms.

This does not mean llama.cpp is universally faster than MLX. The two configurations use different quantization implementations, and the result may change with the model, framework version, prompt length, or generation settings.

But for LocalSubs, the result was clear enough to change the backend.

Subtitle translation is also a slightly unusual inference workload. Most requests contain a short prompt, produce a short output, and run at batch size one. Generic throughput benchmarks do not fully represent that pattern. What matters is whether each individual subtitle finishes before the timing window closes.

Quantization moved the bottleneck

Switching to llama.cpp helped significantly, and quantization reduced the cost of running the transformer layers. Even then, the delay was still noticeable during playback.

At that point, further tuning without measurement was mostly guesswork, so I profiled the generation pipeline.

The LM head stood out.

The original model used a vocabulary of roughly 151,000 tokens. At every decoding step, the LM head projected the hidden state across that entire vocabulary to produce the next-token distribution.

Before quantization, the transformer layers dominated inference time. Once those layers became cheaper, the LM head accounted for a much larger share of the remaining latency.

The LM head had not suddenly become slower. Everything around it had become faster.

That explained why further transformer-side optimization was producing diminishing returns. The next useful target was no longer the transformer itself, but the output vocabulary.

The tokenizer reduced two kinds of cost

A 151K vocabulary is reasonable for a general multilingual model. It needs to cover many languages, scripts, code patterns, rare symbols, and domains.

LocalSubs is much narrower: English dialogue in, Taiwan Traditional Chinese subtitles out.

My initial plan was to replace the tokenizer with a smaller, domain-specific one so that the LM head would compute fewer logits at every decoding step.

That part worked as expected. A smaller vocabulary meant a smaller output projection.

What I had underestimated was the effect on sequence length.

A tokenizer trained around subtitle data could represent common English dialogue and Traditional Chinese subtitle patterns more efficiently. The same translation often required fewer output tokens.

So the tokenizer affected inference in two places:

Smaller vocabulary
→ lower cost per decoding step

More efficient segmentation
→ fewer decoding steps

This made the tokenizer more than a compression detail. It changed the workload itself.

Replacing it was not as simple as removing unused vocabulary rows, however. A new tokenizer changes how text is segmented, which also changes the meaning of token IDs. The original embedding table can no longer be reused directly without adaptation.

That led to a separate embedding migration and training process, which I describe in more detail in the technical report. For the project itself, the important change was that tokenizer design became part of the model architecture rather than an external preprocessing choice.

The data looked aligned until I inspected it

Once latency improved, dataset quality became much harder to ignore.

The training data consisted of English and Chinese subtitle pairs, but subtitle pairs are not necessarily sentence pairs. A cue may contain only half a sentence. A translated cue may include text belonging to the next line. Multiple English cues may be merged into one Chinese subtitle, or the alignment may be close in time but incomplete in meaning.

A simplified example looks like this:

English:
I was going to tell you—

Chinese:
我本來想告訴你，但後來他突然出現了。

The Chinese sentence is related to the English, but it contains information that has not appeared in the current cue.

For ordinary sentence translation, this may look like a slightly loose alignment. For LocalSubs, it is a bad training example. The model is supposed to translate the current subtitle while using previous cues as context. It should not complete unseen dialogue or pull information from the next cue.

I first handled this with rule-based filtering. It was fast and effective for obvious problems:

• empty or malformed samples
• extreme length ratios
• incorrect languages
• duplicated text
• corrupted characters
• clearly abnormal alignments

The harder cases still passed these checks. Their lengths looked reasonable, the text was fluent, and the source and target appeared related. The problem was semantic: the target contained too much, too little, or content from the wrong cue boundary.

Rules could flag suspicious patterns, but they could not reliably determine whether a pair was valid for current-cue-only translation.

I eventually added an LLM-based filtering stage after the rule-based pass. The rules removed inexpensive, obvious failures first. The LLM then handled questions that required meaning:

• Does the target translate only the current cue?
• Does it contain information from a future subtitle?
• Are the source and target semantically aligned?
• Is the pair usable as a standalone training example?

This hybrid pipeline improved the dataset much more than simply adding another round of fine-tuning on the original data.

Where the project ended up

LocalSubs started as a small-model deployment project.

I expected the main challenge to be choosing a model that could fit comfortably on a Mac. Instead, the work moved through several different layers of the system.

MLX seemed like the natural backend, but llama.cpp was much faster on the actual subtitle workload. Quantization reduced transformer cost, which made the LM head visible as the next bottleneck. Reducing the vocabulary lowered the cost of each decoding step, while a subtitle-specific tokenizer also reduced the number of steps required. Once inference became fast enough, cue alignment and dataset filtering became the larger problem.

No single optimization made LocalSubs usable. Each measurement changed what needed attention next.

The model was small. The rest of the system was not.

---

The accompanying technical report contains the full methodology, implementation details, and quantitative evaluation of the tokenizer and inference optimizations.