OCR in 2026: Classical Pipelines, VLMs, and Document AI

Viacheslav Dubrov

2026-03-04 · Updated 2026-09-06 · 15 min · Retrieval and Language Systems

Article update

Originally published on 4 March 2026. Reviewed and updated on 6 September 2026. The update adds newer OCR and vision-model candidates, benchmark references, and guidance on interpreting document-extraction results.

OCR leaderboards disagree because they test different documents, outputs, and judges. OmniDocBench’s versioned benchmark table and OCR Arena’s live preference votes can produce different orderings; the scores are not interchangeable, but the disagreement is useful. A production choice needs documents and metrics from the actual workload.

Vision-language models (VLMs) can handle layout, handwriting, tables, and degraded images that break a plain text-recognition pipeline. Traditional engines remain competitive on clean print, especially when CPU latency and operating cost matter. The dated snapshot below includes PaddleOCR-VL 1.6 and dots.mocr, with different hardware, privacy, and output trade-offs. Check each project before making a current choice.

Companion repo: The OCR Gauntlet contains three instructional notebooks and five downloaded samples. It is a smoke demo, not a validated engine ranking. In this inspected revision, Docling labels do not reliably select the advertised pipeline, receipt recognition references include annotation keys, failures disappear from averages, and the Gemini request uses gemini-2.5-flash while the notebook labels it Gemini 3 Flash. Its whole-page ANLS and cost scenarios also need separate interpretation. These implementation defects must be repaired before using its scores to choose a model.

For the compact model-selection version, see Best OCR Models in 2026: Classical OCR, PaddleOCR-VL, VLMs.

OCR now sets downstream quality

OCR has long powered archives, postal systems, accessibility tools, and document management. RAG and document agents made its failure modes visible to a wider group of engineers: a downstream model cannot recover text or table structure that extraction discarded.

Your RAG system’s retrieval quality is capped by OCR quality. If extraction garbles a table, misreads a date, or drops a paragraph, later chunking and embedding changes cannot recover the missing information. Those errors can hide a contract clause, change an invoice total, or corrupt a medical record.

OCR is therefore part of retrieval and agent infrastructure, alongside parsing, chunking, embedding, and indexing. Its errors need their own evaluation rather than being absorbed into a single end-to-end score.

What OCR means in the age of foundation models

OCR converts text in an image into machine-readable characters. Document AI is the broader system around it: layout analysis, table and formula parsing, field extraction, semantic reasoning, provenance, and validation. Some papers use “OCR-2.0” for end-to-end models that combine several of those stages, but that label should not erase the distinction between recognition and document understanding.

The traditional OCR pipeline has three core stages:

1. Text detection: locate regions containing text (e.g., CRAFT, DBNet).
2. Text recognition: convert detected regions into character sequences (e.g., CRNN).
3. Post-processing: spell-checking and language-model correction.

This works well for clean documents, but detection, recognition, and post-processing errors can compound. Measure both character accuracy and downstream field accuracy so a readable page does not hide a wrong total or identifier.

Some end-to-end VLM paths collapse more of that pipeline into a vision encoder plus language decoder. Models such as GOT-OCR 2.0 can emit text and structure together, while general VLMs can also map fields to a requested schema. The trade-offs are workload-specific latency, GPU or API cost, and the risk of plausible text that is absent from the image.

A practical caveat: an image-only model needs rasterized pages, while a service accepting PDF may handle conversion internally. Deskewing, rescaling and cleanup are pipeline-specific experiments, not mandatory improvements. AWS Textract advises preserving supported inputs rather than indiscriminately converting or downsampling them. Output parsing and validation still belong to the application.

What OCR benchmarks measure and miss

The following datasets illustrate how OCR tasks produce different metrics. Dataset sizes and metrics describe the named dataset version; use each project’s current leaderboard for model scores.

IAM line counts depend on the chosen writer split and recognition protocol; no single test-line count is assumed here.

The “benchmark vs. arena” gap

Automated benchmark rankings can conflict with human preference because the input distribution and judging criteria differ.

In the OCR Arena, users vote blindly on head-to-head outputs. Its live ordering changes as new battles arrive, so it is not a reproducible historical benchmark snapshot. The versioned OmniDocBench table later in this article answers a different question with dataset metrics. Do not combine the two leaderboards into one score.

The likely drivers include document mix, output formatting, language coverage, and judge criteria. Published numbers are useful for screening, but final selection needs a held-out set from the target workload.

Traditional OCR engines: still relevant

If traditional engines are worse on complex data, why use them? Because they’re fast and cheap on clean structured data.

Traditional engines are useful baselines because they can run locally on CPU. Their latency and accuracy depend on the selected model, page resolution, language, preprocessing, and hardware, so benchmark them on the same labeled pages used for VLM evaluation.

Tesseract for clean print

Tesseract (v5.5.x, Apache 2.0) is a mature, primarily CPU engine with 100+ language packs. Clean-print accuracy can be high after appropriate rasterization and preprocessing, but handwriting, scene text, and complex layouts need separate testing. Its main advantage is a small, local CPU deployment. Experimental OpenCL support does not establish a general performance advantage.

EasyOCR

EasyOCR pairs a CRAFT detector with a CRNN recognizer. With full PyTorch GPU acceleration, it’s a fast option for quick prototyping and scene text.

The snippet requires pip install easyocr, PyTorch, EasyOCR’s downloaded model weights, and a local receipt.jpg. It is syntax-checked but not executed by the repository’s Markdown runner.

import easyocr

# Three lines for complete OCR
reader = easyocr.Reader(["en"])
result = reader.readtext("receipt.jpg")
# Returns: [(bbox, text, confidence), ...]

PaddleOCR 3

PaddleOCR 3 packages maintained OCR, document parsing, and deployment pipelines. The current quick start uses the predict() API and explicit orientation settings. Pin the paddleocr package and selected pipeline because 2.x examples using .ocr(..., cls=True) do not match the 3.x API.

The snippet requires pip install "paddleocr>=3,<4", a compatible PaddlePaddle runtime, downloaded model weights, and a local receipt.jpg. It is syntax-checked but not executed by the repository’s Markdown runner.

from paddleocr import PaddleOCR

ocr = PaddleOCR(
    use_doc_orientation_classify=False,
    use_doc_unwarping=False,
    use_textline_orientation=False,
)

for result in ocr.predict("receipt.jpg"):
    result.print()
    result.save_to_json("output")

Specialized and general VLM options

Crooked receipts, skewed product labels, handwriting, and dense layouts are where specialized or general VLMs become worth testing against traditional engines.

The specialized OCR wave

The model list in this section was checked on 2026-09-06. It is not a current ranking. Specialized document-parsing models published from 2024 through 2026 include:

• PaddleOCR-VL 1.6: A two-stage pipeline that performs layout analysis, then uses a 0.9B VLM component on detected regions. PaddleOCR reports 109 languages and 96.3 on OmniDocBench v1.6; keep that vendor result attached to the named pipeline and benchmark version.
• dots.mocr (3B): The March 2026 rebrand of dots.ocr-1.5 parses text and structured graphics, including an SVG-oriented variant. The original dots.ocr remains a separate 2025 model.
• GOT-OCR 2.0: A 580M-parameter unified model that emits plain text and formatted outputs such as Markdown and LaTeX. Its official repository does not publish a minimum VRAM figure, so measure peak memory with the chosen runtime, precision, image size, and output limit.
• DeepSeek-OCR2: The checkpoint documented in the official repository at the snapshot date, succeeding the original 3B-class DeepSeek-OCR model that introduced “contextual optical compression.” Treat throughput figures for either generation as hardware- and dataset-specific.
• Mistral OCR 4.1 (mistral-ocr-4-1): Its model card dates release to July 16, 2026; the changelog records general availability on August 31. It adds block confidence alongside paragraph boxes and structural labels. Calibrate confidence against field correctness before automatic acceptance. The companion’s OCR 3 identifier is historical execution context, not the current candidate or price. Pin an explicit version for comparisons; a latest alias can change.
• Granite-Docling 258M: Emits DocTags that can be converted into a structured DoclingDocument. Docling’s VLM pipeline must be selected explicitly; a default converter is not evidence that this model ran.
• MinerU and olmOCR: Candidates for structured conversion and document linearization respectively. Inspect their complete pipelines and selected model licenses. MinerU’s model terms add conditions to an Apache 2.0 basis, so the library’s license alone does not settle deployment rights.

Frontier VLMs

General VLMs are another option when the task combines extraction with visual or semantic reasoning. These candidates were checked on 2026-09-06 and are used in the gateway example below. They are a starting shortlist, not an OCR ranking:

• Gemini 3.8 Flash (Google): A generally available multimodal model with image input and structured outputs. Use its stable ID when starting a new comparison instead of the older Gemini 3 Flash Preview example.
• Claude Sonnet 5 (Anthropic): A newer Sonnet generation for a hosted comparison. Test transcription fidelity and field extraction on your own documents; general reasoning improvements do not establish OCR accuracy.
• Qwen3.8-27B (Alibaba): An open-weight vision-language model that can be tested through a gateway or self-hosted. Its 27B size makes it a different deployment choice from the earlier Qwen3-VL 8B, which remains a useful smaller baseline when memory is constrained.

A newer release earns a place in the test set, not an automatic promotion to production. Compare field correctness, abstention, latency, and cost per accepted document under the same protocol.

Measure latency by tier

Measure page latency with the actual resolution, batch size, hardware or provider region, and output length. Keep preprocessing and retries in the total; a model-only timing cannot price a successful page.

Metrics: measuring what matters

Pick a metric that matches the output type:

• CER and WER for plain text. Character and Word Error Rate depend on normalization choices such as case, whitespace, and punctuation, so fix the comparison protocol before comparing models.
• Field exact match and Field F1 for forms and receipts. Exact match is binary for one field; its rate is the fraction of evaluated fields that pass. Report document-level all-fields correctness separately. For Field F1, define key/value matching, normalization, duplicate and missing fields, and aggregation; a correct amount assigned to the wrong row is an error.
• TEDS for tables. Tree-Edit-Distance-based Similarity compares predicted and reference HTML trees, catching structural and cell-content errors that CER hides.
• Reading-order comparison when the consumer needs one linear sequence. Compare ordered blocks or spans directly, or use an order-aware metric; OmniDocBench evaluates reading order separately. Correct characters and table cells do not establish the order of a multi-column page.
• ANLS for document VQA. Average Normalized Levenshtein Similarity scores answers against accepted references. The original ST-VQA definition assigns 1 − normalized_distance only when the distance is strictly below 0.5, otherwise zero; it takes the best accepted-reference score and averages across questions. DocVQA adopts ANLS. Pin the evaluator’s case, whitespace and distance normalization when reproducing a score. The companion instead scores whole-page strings and includes the 0.5 boundary, so its variant is not the benchmark protocol.

CER/WER count substitutions, deletions and insertions divided by reference characters or words. They can exceed one when insertions dominate. State whether aggregation sums corpus errors and reference lengths or averages document scores. Retain page, block, bounding box, original text, and normalization history so a wrong field can be traced back to the image.

For implementations: jiwer handles CER/WER out of the box, and TEDS implementations live in the OmniDocBench repo.

Testing VLMs with OpenRouter

OpenRouter provides an OpenAI-compatible gateway to models from several providers. Model IDs and supported request features change, so verify them against the gateway’s current catalog before running the example.

The snippet requires pip install openai, an OPENROUTER_API_KEY, network access, a local receipt.jpg, and model IDs still supported by OpenRouter. It is syntax-checked but not executed by the repository’s Markdown runner.

import base64
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

def extract_text(image_path: str, model: str) -> str:
    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract all text from this image, preserving layout as markdown."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
            ],
        }],
        max_tokens=4096,
    )
    choice = response.choices[0]
    if choice.finish_reason != "stop":
        raise RuntimeError(f"{model} did not finish: {choice.finish_reason}")
    if not choice.message.content:
        raise RuntimeError(f"{model} returned no text (finish_reason={choice.finish_reason})")
    return choice.message.content

# Compare models by changing one string
models = [
    "google/gemini-3.8-flash",
    "anthropic/claude-sonnet-5",
    "qwen/qwen3.8-27b",
]
for model in models:
    print(f"\n--- {model} ---\n{extract_text('receipt.jpg', model)[:200]}...")

The OpenRouter model catalog listed google/gemini-3.8-flash, anthropic/claude-sonnet-5, and qwen/qwen3.8-27b with image input and structured-output parameters on 2026-09-06. Catalog support does not prove that this exact request succeeds at every routed endpoint; no paid inference was run for this refresh.

For structured extraction, use response_format with a JSON schema when the selected model and gateway support it. This can make the response parseable; it does not validate extracted values against the image. With OpenRouter, provider.require_parameters makes the request fail when no routed endpoint supports every requested parameter, instead of falling back to an endpoint that cannot honor the schema. The following block repeats its setup so it is independently readable. It still requires pip install openai, an OPENROUTER_API_KEY, network access, a local receipt.jpg, and a model that supports JSON Schema; the repository runner only syntax-checks it.

import base64
import json
import os
from openai import OpenAI

with open("receipt.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemini-3.8-flash",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract only visible receipt fields. Copy amounts as source strings. Use null for absent or unreadable values; do not infer them. Use null for an unreadable item list, and [] only when no items are present."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
        ],
    }],
    max_tokens=4096,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "receipt",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "vendor": {"type": ["string", "null"]},
                    "date": {"type": ["string", "null"]},
                    "items": {
                        "type": ["array", "null"],
                        "items": {
                            "type": "object",
                            "properties": {
                                "description": {"type": ["string", "null"]},
                                "amount": {"type": ["string", "null"]},
                            },
                            "required": ["description", "amount"],
                            "additionalProperties": False,
                        },
                    },
                    "total": {"type": ["string", "null"]},
                },
                "required": ["vendor", "date", "items", "total"],
                "additionalProperties": False,
            },
        },
    },
    extra_body={"provider": {"require_parameters": True}},
)

def completed_text(response, label: str) -> str:
    choice = response.choices[0]
    if choice.finish_reason != "stop":
        raise RuntimeError(f"{label} did not finish: {choice.finish_reason}")
    if not choice.message.content:
        raise RuntimeError(
            f"{label} returned no text (finish_reason={choice.finish_reason})"
        )
    return choice.message.content

receipt = json.loads(completed_text(response, "receipt extraction"))

Keep the copied amount strings alongside normalized values. Parse money downstream with decimal or integer minor-unit arithmetic under an explicit currency and locale; schema validity does not validate a payment total. Route null critical fields to review.

Benchmark results: what the numbers actually show

The table below is one versioned snapshot: OmniDocBench v1.6_full, official README at commit 09ba2b606662695b16aafe5f5e36b7ef020e11a8, published 2026-04-10 and accessed 2026-08-09. All four rows come from that pinned table. The table does not mix values from earlier paper tables or other leaderboards.

The conclusion is limited to this OmniDocBench version: model size alone does not predict document-parsing score. OmniDocBench subsequently recorded v1.7 and EvalScope integration. Changes to prediction/reference matching can change scores even with identical model output. Preserve the historical rows above; compare new runs only under one pinned evaluator. The composite includes text edit distance, table TEDS and formula CDM; reading order needs its separate result.

The companion computes illustrative metrics on five samples. Correct pipeline identity, recognition references, failure accounting, model names, metric protocol and cost assumptions before interpreting any row as comparative evidence.

Deploying OCR in production

A tiered architecture can separate CPU preprocessing from GPU or API inference and reserve expensive paths for documents that need them.

Note on orchestration: Tools such as Docling can coordinate conversion and batch processing. Retry policy still belongs in the surrounding application or service, and routing still needs its own quality labels and thresholds.

The tiered fallback pattern

Start with the cheapest path that meets the quality target, then calibrate routing on labeled pages:

0. Check for embedded text (Tier 0). For PDFs, inspect the text layer with PyMuPDF or pdfplumber before rasterizing, but validate that the layer is complete and correctly ordered.
1. Attempt with a fast model. Use a traditional engine for document classes on which it meets the target.
2. Evaluate calibrated confidence. Combine model confidence with document class, field criticality, and validation rules.
3. Escalate to a stronger model. Route uncertain pages to a specialized or general VLM.
4. Escalate high-risk failures to a human. Human review is a separate tier for values whose cost of error exceeds the automation benefit.

Note on confidence: Raw character probabilities are not automatically calibrated to field correctness. The area-weighted function below is a baseline for page-level aggregation, not a universal router. Calibrate it against labeled pages, and give critical fields their own rules because a page average can hide a wrong ID or total.

def area_weighted_confidence(page):
    """Compute area-weighted confidence from one PaddleOCR 3.x result."""
    total_area, weighted_sum = 0, 0
    for (x_min, y_min, x_max, y_max), score in zip(
        page["rec_boxes"], page["rec_scores"]
    ):
        width = int(x_max) - int(x_min)
        height = int(y_max) - int(y_min)
        area = width * height
        weighted_sum += score * area
        total_area += area
    return weighted_sum / total_area if total_area > 0 else 0

assert area_weighted_confidence({
    "rec_boxes": [[0, 0, 300, 300]], "rec_scores": [0.5]
}) == 0.5

For each planned engine/document pair, retain a result with success, error or skipped status. Failed, empty and truncated outputs remain in the attempted-document denominator, along with their spent cost. Report completion rate, critical-field errors among automatic acceptances, review fraction, p50/p95 end-to-end latency, and cost per successfully processed document. Reuse converter objects for warm timings and record initialization separately.

Cost analysis at scale

There is no universal page-volume break-even between an API and self-hosting. Build the comparison from the same workload:

Use a shared formula: monthly pages × cost per successful page + review cost + fixed operating cost. A “successful page” must meet the same text, table, and field criteria on both paths. Provider prices and GPU rentals change too quickly to embed as a durable procurement estimate.

Error handling: the hallucination problem

VLM errors can be contextually plausible and factually wrong. A receipt total of “$42.50” might become “$45.20”: syntactically valid, but invisible to a spell-checker.

Synthetic failure example: A VLM extracts three receipt line items and a stated total that agree with one another, but one digit differs from the image. Internal arithmetic passes even though the extraction is wrong. This is why validation needs image-grounded labels or an independent review path, not only consistency checks.

A few practical mitigations:

• Arithmetic reconciliation. When the schema exposes them, verify subtotal + tax + fees + shipping - discounts, within the currency’s rounding tolerance, against the stated total. Route missing components or mismatches to review.
• Regex sanity checks for dates (no month 13), phone numbers (correct digit count), and currency formats.
• Cross-model verification. Run critical fields through two different models and flag disagreements.
• Independent OCR cross-check. Run a second extraction path on critical figures and flag disagreements. Agreement raises confidence only when the two paths have sufficiently different failure modes; it is not proof of correctness.

Key takeaways

1. Match the model tier to a labeled document class. Traditional engines can be sufficient for clean text; specialized and general VLMs should earn their added cost on harder pages.
2. Do not merge unlike leaderboards. OmniDocBench metrics and OCR Arena preferences answer different questions.
3. Calibrate routing. Confidence thresholds, document classes, field criticality, and human-review policy belong in one evaluation.
4. Validate plausible output. Schema conformance and internal arithmetic cannot prove that a value appears in the image.
5. Price successful pages. Include retries, review, fixed operations, and quality checks when comparing APIs with self-hosting.

Preprocessing and detection still matter, but production OCR now also requires routing, task-specific evaluation, and defenses against plausible extraction errors.

References

• OCR Arena Leaderboard - Crowdsourced, head-to-head model battles
• The OCR Gauntlet repo - Runnable notebooks for comparing OCR engines, inspecting Docling output, and estimating cost
• OmniDocBench - End-to-end document parsing eval
• dots.ocr - About 3B total parameters, including a 1.7B language model
• PaddleOCR - Traditional OCR toolkit and models
• OpenRouter - Unified access gateway for A/B testing models