Building a Streaming Pipeline for BLIP-2–Flan-T5

From batch generation to incremental output in a badminton video QA system

The first version of my badminton video question-answering system used batch generation. After receiving a question, the model generated the complete answer, decoded the full output sequence, and only then returned the text to the interface.

The pipeline was simple:

User question
→ Generate the complete answer
→ Decode the output sequence
→ Display the full response

A typical response took around eleven seconds. The model was working throughout that period, but the interface remained unchanged until generation finished.

For offline evaluation, this was acceptable. In an interactive demo, however, eleven seconds of silence made the system appear unresponsive. I wanted the interface to display the answer progressively while the model continued generating it.

The model did not fit the usual serving stack

The system was built on BLIP-2 with Flan-T5. Its inference path included several components:

Video frames
→ Vision encoder
→ Q-Former
→ Flan-T5 encoder
→ Flan-T5 decoder
→ Answer

This was more complicated than the decoder-only language models targeted by most optimized inference frameworks I evaluated at the time.

Hugging Face Transformers could run the complete model, but the available serving frameworks did not provide a drop-in streaming path for the BLIP-2–Flan-T5 architecture. I could not move the checkpoint into an inference server and enable streaming through a configuration option.

The visual features first had to pass through the vision encoder and Q-Former. Their outputs then became part of the Flan-T5 encoder input. The decoder subsequently generated the answer autoregressively while maintaining its generation state.

Because this complete path was not supported by the serving tools I had available, I kept the original PyTorch implementation and built the streaming generation pipeline directly around it.

Separating generation from display

The batch implementation treated generation and interface rendering as one blocking operation. To stream the response, I separated them into two execution paths.

A background thread handled model generation, while the main thread remained responsible for the interface:

Background thread
Generate the next token
        ↓
Send an update event
        ↓
Main thread
Receive and display the latest text

After encoding the video and question, the background thread repeatedly ran the decoder and produced the next token. Each update was passed to the main thread, allowing the interface to refresh without waiting for the complete answer.

The first visible output could now appear in under one second.

The model still required several seconds to finish the response. Streaming did not reduce the total amount of inference work by eleven times. It moved the first visible output from the end of generation to the beginning.

That distinction was important:

Batch mode

0 s                              11 s
│─────────────────────────────────│
             no output             full answer

Streaming mode

0 s       <1 s                              11 s
│──────────│─────────────────────────────────│
 waiting     answer appears progressively

The full response was not necessarily completed much sooner, but users could begin reading while the remaining tokens were still being generated.

A generated token was not a displayable word

My first decoding loop emitted each token as soon as it was produced:

next_token_id = generate_next_token()
text = tokenizer.decode([next_token_id])
send_to_interface(text)

I initially assumed that each generated token would correspond roughly to a complete word or character sequence.

The output immediately showed otherwise.

Flan-T5 uses subword tokenization. A token may represent a full word, part of a word, punctuation, or a whitespace boundary. A single visible word can therefore span multiple generated tokens.

For example, a word conceptually resembling:

player

might be represented internally as multiple pieces:

play + er

Decoding each token independently exposed those pieces directly in the interface. Words appeared temporarily split, and spacing was sometimes inconsistent because an isolated token did not contain enough surrounding context for the tokenizer to reconstruct the intended text.

The generation itself was correct. The problem was treating an internal token boundary as a safe display boundary.

Re-decoding the generated prefix

I changed the pipeline to retain all token IDs generated so far.

Whenever a new token arrived, it was appended to the sequence, and the complete generated prefix was decoded again:

generated_ids.append(next_token_id)

current_text = tokenizer.decode(
    generated_ids,
    skip_special_tokens=True,
)

send_to_interface(current_text)

The interface then replaced the existing response with the latest decoded text.

The updated process became:

Generate one token
→ Append it to the generated sequence
→ Decode the complete prefix
→ Refresh the displayed answer

With the full prefix available, the tokenizer could reconstruct subword boundaries, punctuation, and spacing more reliably.

Re-decoding the entire sequence after every token is not the most efficient possible implementation. Its detokenization cost grows with the response length. In this system, however, the answers were short, and tokenizer decoding was inexpensive compared with multimodal model inference.

The simpler approach was sufficient for the demo and avoided implementing custom rules for deciding when a sequence of subword tokens had formed stable display text.

For a system producing much longer responses, I would instead consider buffered updates or a dedicated incremental detokenizer.

The resulting pipeline

The final streaming path handled several operations that would normally be hidden inside a supported inference runtime:

• encoding the video frames;
• passing visual features through the Q-Former;
• preparing the Flan-T5 encoder inputs;
• controlling the autoregressive decoding loop;
• maintaining generation state;
• running generation outside the UI thread;
• passing incremental updates back to the interface;
• detecting the end of generation;
• converting subword tokens into stable display text.

From the user’s perspective, the visible change was simple: the answer began appearing almost immediately instead of arriving all at once after eleven seconds.

The underlying model and answer quality remained the same. The main change was how generation and display were scheduled.

The original implementation hid all progress until the model had completed its work. The streaming pipeline exposed that progress as it happened, turning a blocked interface into an interactive one.