How Baidu Solved Long-Document AI

About a month ago, Baidu (often called the “Google of China”) introduced Unlimited-OCR, an advancement over DeepSeek OCR. The model was designed to transcribe long, multi-page documents with high accuracy while delivering fast and stable inference.

Unlike conventional vision-language OCR systems, Unlimited-OCR addresses a major bottleneck in long-document transcription: the rapidly growing Key-Value (KV) cache, whose memory usage rises as more output tokens are generated. In this article, we examine how Baidu’s Unlimited-OCR improves efficiency and scalability for long-document OCR tasks.

How Vision-Language Models Read a Page

Visual token patch representation

Unlike traditional OCR systems, Vision-Language Models do not read raw pixels directly. Instead, every document page is divided into a grid of small square regions known as patches. Each patch is converted into a numerical embedding and treated as a visual token, similar to how words are converted into text tokens inside a language model.

As a result, an entire document page becomes a sequence of visual tokens that the transformer can process.

Imagine reading a newspaper through a small window. Instead of looking at the entire page at once, you inspect one small square after another until you’ve seen the complete document. Vision models perform something very similar, except they represent every square mathematically. The number of patches here directly determines the number of input tokens it gives out.

Using more patches means:

• Better preservation of the finer details
• Higher OCR accuracy
• Better understanding of tables and handwriting

However, it also means:

• More input tokens
• Higher computational cost
• Increased GPU memory usage
• Slower inference

Using fewer patches speeds up inference but sacrifices detail, making it harder to recognize small fonts or densely packed documents. Like many machine learning problems, this becomes a trade-off between speed and accuracy.

The Two-Sided Token Problem

Processing long documents introduces two independent computational challenges of its own.

1. Input Tokens

First, the system must convert every page into visual tokens before the model can understand a document.

A high-resolution page can easily generate thousands of visual tokens. For a document containing dozens of pages, the number of input tokens quickly becomes enormous, making inference computationally expensive. The first challenge therefore becomes:

“How do we reduce the number of input tokens without losing important visual information?”

2. Output Tokens

After processing the document, the model must generate the transcription token by token. This creates a second and often more difficult challenge.

Suppose a document contains 100 pages or more. The generated transcription could easily contain tens of thousands of output tokens. In conventional transformer decoders, every newly generated token is stored inside the Key-Value (KV) cache so that future tokens can attend to it.

As the transcription grows longer:

• More tokens are generated and appended.
• The KV cache keeps expanding each token.
• GPU memory consumption continuously increases.
• Attention computation becomes progressively slower.

Unlike the input side, where the cost is fixed once the document has been encoded, the output side grows throughout decoding. For sufficiently long documents, the decoder eventually becomes the primary bottleneck.

Therefore, long-document OCR presents two separate optimization problems:

• Reduce the number of input tokens while preserving document quality.
• Prevent the decoder’s memory from growing as more output tokens are generated.

How DeepSeek OCR Solved the Input Side

As we discussed earlier, the first challenge in long-document OCR is the sheer number of input tokens. The system converts every document page into visual tokens before it can process it with the language model, and high-resolution pages can easily generate thousands of tokens.

Large white spaces, page margins, and simple paragraphs contain far less information than dense tables, handwritten annotations, or complex diagrams. Processing every region at the same resolution therefore wastes valuable computation and resources.

Image compression strategy implementation

To address this, DeepSeek OCR introduced a compression strategy here say a page which translates to 1000 textual tokens normally but we compress the image up to 16x times smaller and feed it to the model which would end up to be of 128 textual tokens. In fact, if we passed this page as text to the model we would have burnt 1000 tokens, whereas converting it to these vision tokens decreases the size by a factor of 16. So we get detail of a high resolution page and everything on it but at a fraction of a cost, this is called optical compression.

Now this isn’t completely loss less so we do pay a price for compressing the image so according to the DeepSeek paper at 10x compression it retains around 97% accuracy as we compress more and more quite a sharp drop in accuracy.

When Deepseek OCR came out it came out with 5 different resolution modes to offer. Fewer tokens – lesser detail and vice versa.

Consequently, the lower the number of visual tokens, the faster and cheaper the inference becomes, but the model may lose some fine-grained details. Likewise, allocating more visual tokens preserves more information at the cost of increased computation.

This optical compression strategy was one of the key innovations that made DeepSeek OCR significantly more efficient than earlier vision-language OCR models. However, it primarily addresses only the input side of the problem.

Baidu’s Unlimited-OCR builds directly on top of DeepSeek OCR, using its visual encoder (specifically trained upon the Base and Gundam variants) to efficiently compress document pages. Rather than redesigning the visual encoder, Baidu focused on solving the second and arguably more difficult challenge: the ever-growing KV cache during autoregressive decoding.

How Baidu Solved the Output Side

DeepSeek OCR significantly reduced the cost of processing the input, but another bottleneck still remained the decoder. Imagine asking the model to transcribe a 40-page document in a single pass. Although the system has already compressed the document into a small number of visual tokens, the model still has to generate thousands of output tokens, one token at a time.

This naturally raises an important question:

“Why do long-document OCR models become slower and eventually run out of memory?”

The answer lies in something called the Key-Value (KV) caching.

Understanding the KV Cache

Large Language Models and Vision-Language Models generate text autoregressively, meaning they predict one token at a time.

Furthermore, the transformer stores its Key and Value representations inside memory every time it generates a new token. This KV cache allows the model to efficiently attend to everything that it has already generated instead of recomputing the entire sequence from scratch. Without a KV cache, generating long responses would be prohibitively slow. However, the KV cache comes with an important drawback here in our case.

It grows with every output token.

If the model generates:

• 100 tokens → small KV cache

Moreover, a much larger KV cache results from 1,000 tokens.

Finally, 10,000 tokens may already occupy several gigabytes of GPU memory.

Meanwhile, as the model produces more text:

• GPU memory continuously increases
• Attention computations become more expensive
• Generation becomes progressively slower
• Eventually, the model may run out of memory altogether

This means that even if DeepSeek OCR compresses the input document into only a few hundred visual tokens, generating the transcription of a long document still causes the decoder’s memory usage to grow without any limitations. This output-side bottleneck is exactly what Baidu Unlimited-OCR was designed to solve

A Simple Analogy

Imagine you’re copying a chapter from a book by hand. Every time you write the next word, you don’t reread every sentence you’ve already written. Instead, you mainly look at the original book you’re copying from and the last few words you’ve written to maintain continuity so you dont break aay from the flow.

Similarly, the model rarely needs everything you wrote several pages ago. Humans naturally keep their attention focused on the source document while only remembering a short recent history of their own writing.

Baidu observed that document transcription works in much the same way.

Unlike chatbots, which often require access to the entire conversation history, OCR models are simply copying information from a document. They rarely need to revisit thousands of previously generated output tokens. This observation became the foundation of Reference Sliding Window Attention (R-SWA).

Reference Sliding Window Attention (R-SWA)

Traditional transformer attention allows every newly generated token to attend to all previous output tokens. While this produces excellent language modelling capabilities, it also causes the KV cache to grow linearly with the length of the generated text.

Standard Sliding Window Attention attempts to solve this by keeping only a fixed window of recent tokens. Nevertheless, if we applied this directly to OCR, the model would eventually forget the document image itself—the very information it is trying to transcribe—which we provided at the starting.

Baidu therefore introduced Reference Sliding Window Attention (R-SWA). Instead of treating every token equally, R-SWA divides attention into two separate segments:

1. Static Reference Prefix

This contains the visual tokens representing the document pages along with the system prompt. These tokens never change during decoding and remain permanently visible to every generated token.

2. Causal Sliding Window

This contains only the most recently generated output tokens (typically around 128 tokens). As the model continues decoding, this window slides forward. As a result, the system discards older output tokens from the attention window.

Sliding window output management

This means that every generated token can always attend to the entire document and only the recent output history. Older generated text gradually falls out of the attention window because it is no longer needed for accurate transcription.

R-SWA maintains two fixed memory regions:

• Static Prefix (m): The compressed visual tokens produced by the DeepSeek visual encoder.
• Sliding Window (n): The most recent generated text tokens.

The total KV cache therefore becomes m + n

Since both m and n remain fixed throughout decoding, the KV cache no longer grows with the output length. Whether the model generates 500 tokens or 30,000 tokens, the decoder maintains approximately the same memory footprint.

This is the key innovation behind Unlimited-OCR.

Running Unlimited-OCR

Now that we understand how Unlimited-OCR works under the hood, let’s actually run it. Baidu has open-sourced both the model weights and inference code, making Unlimited-OCR available through Hugging Face and its official GitHub repository. The model can be run using Transformers, vLLM, or SGLang depending on whether you want a simple local experiment or a production-grade serving setup.

Running with Transformers

In short, users can get started most simply through Hugging Face Transformers. The official repository provides a Transformers-based inference implementation tested on NVIDIA GPUs. The model can be loaded directly using AutoModel with trust_remote_code=True.

First, install the required dependencies:

pip install torch torchvision transformers Pillow matplotlib einops addict easydict pymupdf psutil

Then load the model:

import torch
from transformers import AutoModel, AutoTokenizer

model_name = "baidu/Unlimited-OCR"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True
)

model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
    use_safetensors=True,
    device_map="auto"
)

The official repository also provides a higher-level Transformers pipeline:

from transformers import pipeline

pipe = pipeline(
    "image-text-to-text",
    model="baidu/Unlimited-OCR",
    trust_remote_code=True
)

From here, you can provide document images to the model and ask it to parse or transcribe their contents. For multi-page documents, the official repository also provides utilities and examples for processing PDFs by converting their pages into images before inference.

Running with vLLM

If the goal is to serve Unlimited-OCR as an inference API rather than simply experimenting with it locally, vLLM is another option. Baidu’s repository now includes official vLLM support, and the Hugging Face model card provides a basic serving example.

pip install vllm
vllm serve "baidu/Unlimited-OCR"

Once the server is running, it exposes an OpenAI-compatible API endpoint that can be called from your application. This makes it much easier to integrate Unlimited-OCR into an existing document-processing or RAG pipeline.

You can find the complete implementation in the official Unlimited-OCR GitHub repository and the model weights on Hugging Face.

Conclusion

Unlimited-OCR is compelling because it reframes long-document OCR as an information-management problem rather than a pure model-scaling problem. Optical compression reduces the cost of reading high-resolution pages, while R-SWA keeps decoding memory from growing uncontrollably by preserving access to the document and retaining only a small recent history of generated text.

Together, these ideas address both the input and output sides of long-context VLM inference. “Unlimited” is not a literal claim, but the broader lesson is important. Better long-context systems may come less from adding more memory and more from deciding, intelligently, what information truly deserves to stay in memory. That is what makes this approach feel like a meaningful step forward.

Frequently Asked Questions

GenAI Intern @ Analytics Vidhya | Final Year @ VIT Chennai
Passionate about AI and machine learning, I’m eager to dive into roles as an AI/ML Engineer or Data Scientist where I can make a real impact. With a knack for quick learning and a love for teamwork, I’m excited to bring innovative solutions and cutting-edge advancements to the table. My curiosity drives me to explore AI across various fields and take the initiative to delve into data engineering, ensuring I stay ahead and deliver impactful projects.