NER Guide 2026: GLiNER, spaCy, Transformers, and LLMs

Viacheslav Dubrov

2026-04-02 · Updated 2026-09-06 · 35 min · Retrieval and Language Systems

Article update

Originally published on 2 April 2026. Reviewed and updated on 6 September 2026. The update covers newer GLiNER models, streaming PII detection, structured-extraction APIs, and benchmark interpretation.

Named entity recognition (NER) turns text into labeled pieces that a program can use. Given “Bill Gates founded Microsoft on April 4, 1975,” it can return the contiguous text spans Bill Gates, Microsoft, and April 4, 1975 with the types person, organization, and date. In a span-based encoder such as GLiNER, the model reads the text once and scores candidate spans against the labels you supply. Choosing an NER system means deciding how much latency, cost, label flexibility, and reasoning that extraction needs.

The companion repository provides smoke examples for GLiNER, ONNX export, annotation generation, and structured extraction. Its three-sentence comparison is not a model ranking: the scorer collapses repeated text/type pairs and can ignore trailing gold documents when predictions are missing. The teacher script also lacks a demonstrated conversion into reviewed training spans. Use these examples to inspect APIs, not to claim a completed training or evaluation workflow. For workloads dominated by explicit spans, compact encoders are usually the faster and cheaper option. LLMs remain useful for producing training data and handling cases that require inference or normalization. The benchmark sections explain the published comparisons and their limits; none is a new benchmark run for this guide.

For a retrieval or privacy pipeline, I would first ask whether the output must be a literal mention. That decision separates a span recognizer from systems that also normalize values, link records, or infer facts.

Companion repo: ner-field-guide, with runnable demos for GLiNER, ONNX export, the LLM-as-teacher pipeline, and structured extraction with Instructor.

For the short model comparison, see Best NER Models in 2026.

What is named entity recognition?

Named entity recognition finds spans in text and assigns types such as person, organization, date, product, or domain-specific labels. A span is a contiguous piece of the original text. NER identifies the mention; entity linking is the separate step that resolves it to a canonical record or ontology concept.

Where modern systems use NER

NER still finds spans of text and assigns labels to them. What changed is its position in the system. It now supplies filters for RAG, structured arguments for agent tools, and fields for document-processing pipelines. Those uses make latency, cost, and schema flexibility as important as benchmark accuracy.

RAG: better retrieval through entity extraction

Similarity search alone struggles when a question contains exact entities. For “what did Anthropic say about model safety in Q4 2024?”, entity extraction can propose “Anthropic” and “Q4 2024” as filters. Apply a hard filter only when the indexed metadata and date semantics support it; otherwise use it as a ranking signal or retain an unfiltered retrieval path. A missed alias or an incorrectly inferred date range can exclude the answer.

During indexing, you extract entities from each chunk and store them as metadata: {"organizations": ["Anthropic"], "dates": ["Q4 2024"], ...}. This lets you filter by entity before running vector search. Knowledge graph RAG (GraphRAG, LlamaIndex property graphs) adds named relationships that retrieval can follow across several hops. Compare it with repeated vector searches on the questions and corpus you need to support.

At query time, entities extracted from the user’s question drive routing. A question mentioning a company name goes to a finance index; one mentioning drug names goes to a clinical knowledge base. GLiNER is useful when the query-time schema or entity types change. Unseen company or drug names alone do not require open-vocabulary labels; a closed-label model can still recognize new mentions of known types.

AI agents: turning text into structured facts

Agents receive unstructured text such as web pages, API responses, and user messages. NER converts that text into structured facts the agent can reason over, store, or pass to tools.

For tool routing, a request such as “schedule a meeting with Sarah Chen from Accenture on Thursday at 2pm” can yield PERSON: Sarah Chen, ORGANIZATION: Accenture, DATE: Thursday, and TIME: 2pm. A calendar resolver then combines the date and time into the API’s required timestamp. A local encoder avoids the API round trip and is often substantially faster, but latency depends on the model, runtime, hardware, batch size, and label count. Measure both paths on the calendar workload instead of assuming a fixed millisecond gap.

NER also supports entity tracking across conversations. Agent memory systems need to know that “Sarah” in turn 3 and “Ms. Chen” in turn 12 are the same person. NER identifies the spans; coreference resolution decides whether the mentions refer to the same person in context. Entity linking then maps that person to a canonical record, when one exists.

The constraint in both cases is latency. If each of ten sequential steps makes a 200 ms NER call, those NER calls add 2 seconds of perceived delay. One call adds 200 ms. Encoder models usually fit entity work inside agent loops better than LLM-based extraction.

Document intelligence: from images to structured data

OCR turns images into text. NER turns that text into structured fields.

A standard pipeline first uses OCR, such as Tesseract, Azure Document Intelligence, or AWS Textract, to produce text and bounding boxes. NER then finds spans for fields such as invoice_number, vendor_name, total, and due_date. A schema or relation-extraction step must group line-item descriptions, quantities, and prices into line_items. The same sequence applies to contracts, medical records, and regulatory filings.

Modern document pipelines may combine layout understanding, entity extraction, and relation extraction. OCR or a layout-aware document model still supplies text, reading order, tables, and bounding boxes. Current GLiNER2.5 exposes entities, relations, classification, and structured records through a schema interface. Evaluate each output separately; the older GLiNER2 paper did not benchmark every task now exposed by the library.

Cost decides most of these pipelines. Price yours at the actual monthly document volume, including retries and review. A compact encoder can run on CPU, while an API-based LLM adds per-document inference cost and latency. A practical test is to label a representative invoice set with an LLM. Fine-tune GLiNER on the reviewed records, then compare both paths on field-level F1, latency, and total cost.

PII detection and LLM guardrails

GDPR’s data-protection principles and security duties (Articles 5, 25, and 32), HIPAA’s technology-neutral Security Rule (HHS guidance), and California’s CCPA as amended by CPRA impose different rights and risk-based safeguards. The cited provisions do not prescribe NER or a specific pre-model scanning architecture. This is not legal advice; have counsel review the requirements that apply to your data and jurisdiction. NER can support data inventory, minimization, or de-identification, but it is one control whose recall must be validated for the relevant data and jurisdiction.

NER handles this directly. De-identification models find PERSON, SSN, PHONE, EMAIL, and ADDRESS spans and either redact them or replace them with synthetic equivalents. Microsoft Presidio combines recognizers with anonymization operators, and its samples include GLiNER as a recognizer. The 0.3B-parameter GLiNER2-PII is another candidate: its paper covers 42 PII types at character-span resolution. Neither is compliance evidence. Validate recall by data format, jurisdiction, language, and PII class before using any detector as a control.

In John Snow Labs’ vendor-run comparison, 48 expert-annotated documents cover six PHI classes. Provider labels were remapped and unmappable predictions excluded, which limits cross-provider interpretation. The Providence report describes 281 leaked PHI events in a review of 1,000 notes containing 34,701 sentences. Events are not distinct affected sentences: count both leaked instances and affected documents in a privacy evaluation.

For LLM guardrails, NER works as a pre-screening layer: scan user input for PII before sending it to an external API, then block or anonymize. It may be faster or simpler than asking the LLM to self-moderate. Treat that as a deployment hypothesis: measure both paths on your model, hardware, input mix, and recall target. False negatives remain possible, so add another control for the PII exposure your system cannot accept. GLiNER is especially useful here because PII categories vary by jurisdiction. An API can accept a new label such as “genetic information” without retraining. That does not establish recall for the new category; validate it on reviewed examples before relying on it.

GLiNER: span-to-label matching for open-vocabulary NER

GLiNER (NAACL 2024, Zaratiana et al.) made encoder-based NER competitive with LLMs at a fraction of the cost. Instead of treating NER as sequence labeling or text generation, GLiNER treats it as a matching problem. It scores every candidate text span (each contiguous sequence of words like “Bill Gates” or “Microsoft”) against every entity type label, then keeps the high-scoring pairs.

The model takes entity type labels and input text as a single sequence: [ENT] person [ENT] organization [ENT] date [SEP] Bill Gates founded Microsoft.... A bidirectional transformer (DeBERTa-v3) encodes everything together.

From the output, the model builds two sets of representations. One refines each entity-type representation from an [ENT] token position through a small feed-forward network (FFN) that transforms the encoder’s vector. The other represents text spans by combining start and end word vectors through another FFN. A dot product between a span representation and an entity type representation gives a score.

Apply sigmoid and you get the probability that the span from word position $i$ to word position $j$ belongs to entity type $t$: $\phi(i, j, t) = \sigma(S_{ij}^T \cdot q_t)$. Here $S_{ij}$ is the span vector produced by the FFN and $q_t$ is the FFN-refined entity type embedding from the corresponding [ENT] token (Zaratiana et al., 2024, Eqs. 1–2). Candidate spans are capped at 12 words; the tokenizer may split a word into several subword tokens.

GLiNER accepts natural-language label descriptions at inference time without retraining, but extraction quality depends on label wording and domain fit. You pass in entity types such as “person,” “adverse drug reaction,” or “financial instrument,” and the model scores spans against them. The 50M, 90M, and 300M configurations below are the original paper models. The v2.1 model card instead lists 166M, 209M, and 459M English checkpoints, plus a 209M multilingual checkpoint, all under Apache 2.0. Do not compare latency or memory across those generations as if the parameter names were identical (GLiNER v2.1 model card).

For a true hard zero-shot test, hold out target types as well as target examples: the model has no labeled examples for the target task. A type description such as “a medically confirmed adverse effect caused by a treatment” gives the model more information than adverse event alone. That is not a guarantee of transfer, but description-driven ZeroNER outperformed name-only baselines on its held-out-type benchmarks (Cocchieri et al., 2025).

Training data for the original model came from the Pile-NER dataset: 44,889 passages with 240K entity spans across 13K entity types, all labeled by ChatGPT. Training GLiNER-L took about 5 hours on a single A100 (Zaratiana et al., 2024).

Benchmark results

Zero-shot results from Zaratiana et al. (2024), Tables 1 and 2. F1 combines precision and recall into one score; a higher score is better only under the same task and evaluation rules. The seven-dataset average combines five CrossNER datasets with MIT Movie and MIT Restaurant:

GLiNER-M at 90M parameters nearly matches UniNER-13B in the paper’s seven-dataset average (55.4% vs. 55.6% F1) while using roughly 140 times fewer parameters. The 50M GLiNER-S exceeds the reported ChatGPT (GPT-3.5) result by 5 F1 points. The multilingual variant, trained only on English data, exceeds that same ChatGPT baseline in 8 of 10 non-English languages (Zaratiana et al., 2024). These comparisons use the paper’s model versions and evaluation harness; they do not establish a ranking against newer LLMs.

GLiNER variants cover biomedical text, PII detection, news, and multilingual support.

The companion retains v2.1 for its quickstart. For a new cross-encoder comparison, include the maintained GLiNER v2.5 checkpoints alongside it; version numbers are not evidence of a domain F1 gain. GLiNER v2.5 and Fastino GLiNER2.5 below are different model families.

From scripts/01_gliner_quickstart.py:

from gliner import GLiNER

model = GLiNER.from_pretrained("urchade/gliner_medium-v2.1")
text = "Bill Gates founded Microsoft on April 4, 1975."
labels = ["person", "organization", "date"]
entities = model.predict_entities(text, labels, threshold=0.5)

for entity in entities:
    print(f"  {entity['text']} => {entity['label']}")
# Bill Gates => person
# Microsoft => organization
# April 4, 1975 => date

How GLiNER compares to spaCy

spaCy is one of the most established NLP libraries in production. It also works under different architectural constraints than GLiNER.

spaCy’s pipelines (en_core_web_sm, en_core_web_trf) do closed-vocabulary NER: a fixed set of entity types (PERSON, ORG, GPE, DATE, etc.) defined at training time. Expanding the trained ner component requires labeled examples and training. For identifiers or a controlled dictionary, EntityRuler can add custom labels through patterns without retraining; test its recall outside those patterns. Pin the maintained 3.8 model package rather than assuming an unreleased major line is a production upgrade. The en_core_web_trf 3.8.0 model card reports 90.19 NER F1 on OntoNotes 5.0, but only for its 18 predefined types (spaCy model card).

GLiNER does open-vocabulary NER: it accepts new label text at inference time without retraining. That does not guarantee useful extraction for every label; wording and domain fit still matter. It is a strong candidate when entity types are unknown in advance, change often, or are domain-specific (“adverse drug reaction”, “financial instrument”, “threat indicator”).

My recommendation: use spaCy for standard entity types where pretrained pipelines are well validated. Use GLiNER when you need flexible, zero-shot types or when your pipeline must adapt without retraining. They can share a pipeline, with spaCy handling tokenization and sentence splitting and GLiNER handling entity extraction.

A supervised Transformer baseline

For stable labels with representative annotated spans, start with a fine-tuned token classifier such as RoBERTa or DeBERTa as the supervised baseline. It trades label flexibility for task-specific accuracy. Compare it with spaCy and GLiNER on exact-span F1, per-label recall, calibration, latency, and cost on the same domain set.

UniNER and NuNER: how small can you go?

UniNER (ICLR 2024, Zhou et al.) and NuNER (EMNLP 2024, Bogdanov et al.) both distill LLM annotations into smaller NER models — but they disagree on how small you can go.

UniNER: the maximalist path

UniNER fine-tunes LLaMA-7B/13B on 45,889 input-output pairs generated by ChatGPT. For each entity type, the model answers “What describes [type] in the text?” and outputs JSON lists. A key training trick: frequency-based negative sampling boosts F1 from 31.5% to 53.4% (Zhou et al., 2024).

UniNER-7B hits 41.7% zero-shot F1 across 43 datasets — beating ChatGPT’s 34.9% by 7 points. The 13B variant reaches 43.4%, only 1.7 points more for nearly double the parameters (Zhou et al., 2024).

The production trade-off: the paper’s higher-scoring UniNER-type setup queries each entity type sequentially. Its all-in-one variant uses one response but averaged 3.3% lower. At FP16, a 7B checkpoint needs roughly 14GB just for weights; lower-bit quantization can reduce that footprint. The UniNER-7B-all checkpoint lists CC BY-NC 4.0; check the exact checkpoint rather than transferring that label to every variant.

NuNER: the minimalist path

NuNER starts from RoBERTa-base (125M parameters) and uses contrastive training with 4.38 million GPT-3.5 annotations across 200K concepts. After training, the concept encoder is thrown away; the text encoder slots into any standard NER pipeline as a RoBERTa replacement (Bogdanov et al., 2024).

NuNER’s Table 3 reports roughly 6.1–16.2 points over RoBERTa in macro-average token-classification F1 with frozen encoders and linear heads, across four datasets and sampled training sizes. The paper also compares task-specific fine-tuning with UniNER-7B; its separate discussion of more than a dozen examples concerns GPT-4 in-context learning, not a threshold for matching UniNER (Bogdanov et al., 2024).

Both papers support learning from LLM annotations. They do not establish that any smaller model will outperform its teacher on a new domain. Keep a separately reviewed test set.

GLiNER2 and GLiNER2.5: separate the paper from the checkpoint

The original GLiNER ecosystem split NER, relation extraction, classification, and document-level extraction across separate models. The EMNLP 2025 GLiNER2 paper unified NER, classification, and hierarchical extraction in one 205M-parameter model; current releases later added relation extraction to the same schema interface.

That 2025 architecture keeps the cross-encoder design but extends context to 2,048 tokens (4x the original) and adds declarative schemas for defining extraction tasks. Training uses 135,698 real documents annotated with GPT-4o plus 118,636 synthetic examples (Zaratiana et al., 2025).

On zero-shot CrossNER, GLiNER 2 scores 0.590 F1, close to GPT-4o’s 0.599 in the paper’s mid-2025 benchmark. For classification, it averages 0.72 across 7 benchmarks, compared with 0.69 for DeBERTa-v3-large. On CPU, the paper reports 130-208 ms classification latency across its tested label counts. The zero-shot DeBERTa baseline needs a forward pass per candidate label and rises from 1,714 ms for 5 labels to 16,897 ms for 50; this is not the runtime of a supervised DeBERTa classifier (Zaratiana et al., 2025).

The example below instead loads GLiNER2.5: 194M parameters, a boundary architecture, and a configured 4,096-token window. AutoExtractor selects BoundaryExtractor; the legacy GLiNER2 loader is inappropriate. Sparse start/end pairing changes which spans are scored, not the finite context limit. Use documented overlapping-chunk helpers for long documents and check remapped offsets. The 2025 scores above do not describe this checkpoint.

from gliner2 import AutoExtractor

# Requires `pip install gliner2[local]` and downloads the checkpoint.
extractor = AutoExtractor.from_pretrained("fastino/gliner2.5-base-v1")

# Compose tasks through the current schema interface
schema = (extractor.create_schema()
    .entities({"person": "Names of people", "company": "Organization names"})
    .classification("sentiment", ["positive", "negative", "neutral"])
    .relations(["works_for", "founded", "located_in"])
    .structure("product_info")
        .field("name", dtype="str")
        .field("price", dtype="str"))
text = "Acme launched a $19 widget in Berlin."
results = extractor.extract(text, schema)

Current GLiNER2 releases expose entity recognition, classification, hierarchical extraction, and relation extraction through one schema. The EMNLP paper evaluates NER and classification; it reports no hierarchical-extraction benchmark and does not cover the later relation API. Treat the four-task path as a deployment capability to benchmark, not evidence that one model preserves four specialized models’ accuracy.

2026 additions: choose the architecture that matches the bottleneck

The GLiNER ecosystem now has distinct architectures. They are candidates for a domain comparison, not a single leaderboard.

The original GLiNER paper, the bi-encoder paper, and GLiNER-Relex each use their own models and harnesses. GLiNER-X and GLiNER Decoder model cards describe released checkpoints, but they are not a peer-reviewed, comparable benchmark. Keep that distinction in an architecture decision record.

Streaming PII changes when you can release text

GLiNER Streaming PII adds incremental detection with a Qwen3-0.6B causal backbone and cached session state. Its API returns the whole current session snapshot, with offsets into accumulated text. Preserve chunk boundaries, keep labels fixed until a full recompute, and clear finished sessions. It requires GLiNER 0.2.28 or later.

For streaming redaction, I would buffer unreleased text and test entities split across chunks. Finding a phone number after sending its first half cannot retract those bytes. Measure leaked characters before release and added delay, as well as final-span recall. The model card reports PIIMB masking F2 separately from strict typed-span F1; they answer different questions. Its multilingual weakness also rules out treating the released checkpoint as a universal privacy filter.

Checkpoint licenses are part of the model choice

Verify the exact code, weight, and dataset terms before deployment. The current cited releases are not interchangeable:

License labels are not legal advice. A production review must include base-model, training-data, and provider terms.

The bi-encoder: scaling to million-label NER

The original GLiNER encodes labels and text together. Joint encoding becomes progressively more expensive as label text consumes the context and must be re-encoded with every document. The crossover depends on the checkpoint, label descriptions, and hardware. When the same large type inventory is reused across many documents, make the GLiNER bi-encoder the default comparison. It splits text and label encoding into two separate transformers (Stepanov et al., 2026).

The text encoder uses ModernBERT (Ettin family), the label encoder uses sentence transformers (BGE or MiniLM). Spans and labels are scored via dot product. That split means entity type embeddings can be pre-computed once and cached. At inference, cached labels avoid repeated label encoding. Scoring candidate spans against those labels still takes compute and memory; a million-label application also needs bounded candidate retrieval or batching, with recall measured for labels excluded before scoring.

Four model sizes are available, all benchmarked on CrossNER (Stepanov et al., 2026, Table 1):

At 1,024 entity types, the pre-computed gliner-bi-edge-v2.0 bi-encoder loses only 5.2% throughput versus a single label (19.3 → 18.3 ex/s). The comparable gliner_small-v2.5 uni-encoder loses 98.7% (10.7 → 0.14 ex/s). In the paper’s single-H100, batch-size-1 tests across 64-, 256-, and 512-token inputs, the pre-computed bi-encoder reaches an up to 130× throughput advantage over gliner_small-v2.5. With 100 entity types on a single H100, the bi-encoder processes 1.96 million predictions per day versus 368K for the cross-encoder (Stepanov et al., 2026).

The bi-encoder paper reports 61.5% for gliner-bi-large-v2.0 versus 60.9% for gliner_large-v2.5 on the same CrossNER-plus-MIT average. The original GLiNER paper also reports 60.9%, but for a different checkpoint and evaluation. Treat the bi-paper comparison as a paper result, then test both architectures on the domain set. The authors recommend bi-base-v2.0 (194M) as the sweet spot, hitting 98% of the large model’s accuracy at 2.6x the speed (Stepanov et al., 2026).

from gliner import GLiNER

model = GLiNER.from_pretrained("knowledgator/gliner-bi-base-v2.0")

# Pre-compute embeddings for a reused label set; rebuild this cache when labels or the checkpoint change.
entity_types = ["person", "organization", "date"]  # Small example; large inventories need bounded candidate selection
entity_embeddings = model.encode_labels(entity_types, batch_size=8)

# Encode text and score spans against the cached labels
texts = ["Bill Gates founded Microsoft on April 4, 1975."]
outputs = model.batch_predict_with_embeds(texts, entity_embeddings, entity_types)

The timing sweep stops at 1,024 labels, using an H100, batch size one, and ten forward passes per configuration. It does not measure a deployed million-label service. Larger biomedical or enterprise inventories are applications to evaluate; entity linking is available through the companion GLiNKER framework.

LLMs as teachers: a $70 case study and a deployable pipeline

The LLM-as-teacher pattern separates expensive annotation from cheaper inference. Two published case studies show how teams have applied it under different conditions.

The CFM case study

In a Hugging Face case study, Capital Fund Management extracted company names from roughly 900,000 financial news headlines. Zero-shot GLiNER scored 87.0% F1. The team used Llama 3.1-70B to annotate the dataset in roughly 8 hours for about $70, then reviewed 2,714 samples through Argilla in another 8 hours.

Fine-tuning GLiNER on this data reached 93.4% F1 in the case study, compared with the Llama-70B teacher’s 92.7%. The authors report $0.10 per hour on CPU for the fine-tuned model and $8 per hour for the teacher (CFM case study). Those figures describe one financial-news task and one infrastructure setup.

The Refuel AI study

Refuel AI’s technical report benchmarks LLM labeling across 8 NLP datasets, including CoNLL-2003. It reports 88.4% agreement with ground truth for GPT-4 (March 2023) and 86.2% for the human annotators in its setup, along with 20 times faster and 7 times cheaper labeling. A separate proprietary-data ensemble experiment exceeded 95% agreement, with its best individual LLM at 89%. Confidence-based routing to cheaper or stronger models is a proposed use, not the measured mechanism behind the eight-dataset result (Refuel AI technical report). Treat these as vendor-reported results under that study’s annotation protocol.

A production pipeline

A practical production flow has six steps:

1. Write annotation guidelines in natural language
2. Create human-labeled validation and held-out test sets sized from entity prevalence, per-label slice requirements, and the desired confidence-interval width. A 50-200-document pilot can calibrate guidelines, but it is not a default production evaluation size.
3. Use an LLM with a versioned prompt and explicit output schema to propose bulk training labels. Retain the requested and actual model, prompt, source text and fallback status. Reject unknown types and spans that cannot be aligned to the source; the companion currently falls back silently and must be corrected before its output is treated as training gold.
4. Review a subset via Argilla or Label Studio
5. Fine-tune a compact encoder (GLiNER, SpanMarker, RoBERTa)
6. Deploy only after the encoder passes a held-out quality and total-cost check. CFM reported 16-80x lower hourly infrastructure cost in its setup; include annotation, review, serving, and retraining costs in your comparison.

The LLM can reduce the volume of manual annotation, but the team still owns the validation set, annotation guidelines, targeted review, and error analysis.

Where GLiNER fails and LLMs remain useful

The Sease benchmark (October 2025) tested GLiNER against GPT-4.1-mini on 30 query-parsing tasks. GPT-4.1-mini got 100% fully correct. GLiNER got 53% (16 of 30). But GLiNER responded in 0.08 seconds versus the LLM’s 1.21 seconds — 15x faster.

In this 30-task benchmark, GLiNER failed in three recurring patterns:

1. Implicit entities: extracting “event” from “Elton John performed at Madison Square Garden” — no text literally says “event,” but the LLM infers “concert”
2. Label phrasing sensitivity: “2022” scores 0.388 against “date” but 0.958 against “year” — small label changes cause large score swings
3. Value mapping: GLiNER returns the exact surface text (“family houses”) instead of the canonical value (“Single family house”). An LLM can perform that normalization when its prompt and schema define the target values.

Nested and overlapping entities

GLiNER defaults to flat decoding, which suppresses overlapping spans. Its API also supports flat_ner=False, so nested predictions are possible, although quality depends on the checkpoint, labels, and domain data. Benchmark both decoding modes on a nested, span-level test set before choosing a specialized model.

Use GLiNER for explicit entity extraction and route cases requiring inference, reasoning, or mapping to predefined ontologies to an LLM. The routing threshold should come from a labeled domain set.

Evaluating NER: metrics, pitfalls, and test sets

A model can score 95% F1 on a curated test set and still fail on the document mix it sees after deployment. Build the evaluation set from the production distribution and keep slices for the rare formats and entity types that aggregate F1 can hide.

The core metrics

Identify each occurrence by document ID, half-open start/end offsets, and type, and verify that text[start:end] recovers the mention. Two occurrences of “John” are two targets. Validate document counts before matching: missing predictions must create false negatives, not disappear through zip(). Count wrong types or offsets as an unmatched prediction and an unmatched gold span. Report micro precision, recall and F1, per-type support and recall, and define macro aggregation and empty cases.

• Entity-level F1: The standard metric. A prediction is correct only if both the span boundaries and the type match ground truth exactly. This is what most papers report.
• Token-level F1: Scores each token independently. It can make a partial boundary look better than an exact-span score, so report exact-span F1 when boundary correctness matters; use token-level metrics only when the downstream decision is token-level.
• Precision vs Recall: These often have asymmetric costs. For de-identification, recall matters more — missing a name is worse than over-redacting. For database extraction, precision matters more — false entries corrupt downstream analysis.

Common evaluation pitfalls

1. Partial match inflation: “Bill” extracted when the gold label is “Bill Gates” — some scripts count this as a partial match. Use exact span matching unless you have a reason not to.
2. Type confusion: “Microsoft” correctly identified as a span but labeled PERSON instead of ORG should score zero. Check your evaluation code handles this.
3. Test set leakage: Keep duplicate and near-duplicate records, documents, and annotations out of the test set. For a hard zero-shot claim, also keep target entity types out of training; repeated surface strings alone are not leakage.
4. Uncontrolled label prompts: A short type name and a tested type description are different inputs. Version label descriptions, thresholds, checkpoint revisions, and decoding mode with the score.
5. One-language zero-shot claims: Do not infer multilingual quality from English. OpenNER spans 36 corpora and 52 languages, and its baselines found no single model best in every language (Palen-Michel et al., 2025). In FiNERweb experiments, switching from English to target-language labels changed F1 by 0.02–0.09 depending on the setting (Golde et al., 2026). Test both label languages when the product uses local terminology.
6. One run is a verdict: Report variation across random seeds where applicable, threshold sweeps, and repeated production samples. Small slice counts make a model ranking unstable.

Keep normalization, entity linking, record grouping and relation endpoints separate from extractive F1. For privacy, report leaked instances and affected auto-accepted documents, false redactions, and review fraction. For serving, report completion rate, p50/p95 document latency, cold and warm runs, batch and label sizes, peak memory, and cost per accepted document.

Building a domain test set

For production evaluation, I recommend:

1. Sample from production data, not curated examples. Include the messy documents your model will actually see.
2. Size the test set for the estimate you need. Choose the count from entity prevalence, per-label slice sizes, and the desired confidence-interval width. Report bootstrap or analytic confidence intervals.
3. Use at least two annotators on a calibration subset. Adjudicate disagreements and report a span-aware agreement measure. Agreement diagnoses ambiguity and guideline quality; it is not a model-performance ceiling.
4. Stratify by difficulty — easy cases (clean text, standard types) and hard cases (ambiguous entities, jargon, noisy text).
5. Keep privacy and fairness slices. For PII, report recall by PII type, language, locale, document format, and relevant demographic or name-origin slices. Minimize evaluator access to raw sensitive text, set retention limits, and review false negatives.

Continual NER needs an immutable regression set

Production taxonomies change. Add new types without silently changing what an old type means. Keep a versioned, immutable regression set for existing types, a separate test set for the new type, and a changelog for annotation-guideline changes. Report old-type and new-type scores separately before replacing a checkpoint. This is the simplest way to detect forgetting and taxonomy drift.

Production NER across four industries

These are selected industry examples with specific numbers under their reported conditions. The sources mix vendor-reported comparisons, company- or project-reported case studies, and peer-reviewed papers or preprints. Treat them as practical examples, not a maturity ranking.

Healthcare

The John Snow Labs and Providence examples above show why de-identification needs both entity-level and document-level reporting. Their evaluation protocols are more useful for designing a review than an unqualified provider ranking.

OpenMed’s retrospective, published January 6, 2026, reports 481 models; 380+ describes its July launch inventory. Downloads measure distribution, not deployments. Its claim of leading results on 10 of 12 benchmarks comes from a separate August 2025 author preprint, not independently verified production results.

Financial NER

Financial extraction includes company mentions in news and fields in filings; their schemas and document lengths differ. The CFM case study covers the former. FinBERT-MRC formulates extraction as machine reading comprehension and reports 92.78±0.56 F1 on ChFinAnn and 96.80±0.38 on AdminPunish, both Chinese datasets. Those results do not establish accuracy on English SEC filings. Test long documents and nested financial entities in the intended language.

E-commerce

Walmart’s KDD 2023 paper uses QU-965M multitask query data; its roughly 60 NER labels are IOB2 classes. Section 6.8 attributes the 0.51% GMV lift to the MTDNN baseline trained on that data, while EAMT’s online evaluation remained future work. The business delta cannot be attributed solely to NER. Home Depot’s TripleLearn reports held-out F1 improving from 69.5 to 93.3, an online experiment, and more than nine months in production. The transferable lesson is iterative domain supervision, not a current architecture ranking.

Cybersecurity

iACE is a historical example of automated threat-intelligence extraction. CyNER’s paper combines neural recognition with other extraction sources and reports 76.66 span micro-F1 for XLM-RoBERTa-large, evaluated with seqeval. The CyberNER preprint harmonizes four datasets into 21 STIX 2.1-aligned labels and reports RoBERTa at 0.736 F1. These are research results, not evidence of production uptake.

Deployment optimization: from Python to lower-latency inference

The companion repo demonstrates GLiNER ONNX export and INT8 packaging. It records artifact sizes, but it does not reproduce the latency or F1 figures reported by external projects.

Native GLiNER serving

Before moving to a new runtime, test the project’s Ray Serve path. gliner[serve] provides dynamic batching, memory-aware batch sizing, multi-replica scaling, and an HTTP client. Batching and replicas can improve throughput while keeping the model code; batching can also add waiting time and replicas do not eliminate queueing. Benchmark queue latency, warm latency, throughput, and exact-span F1 under your request mix before comparing it with ONNX or Rust.

ONNX export

GLiNER has native ONNX conversion, and pre-converted models exist on Hugging Face (onnx-community/gliner_small-v2.1). Measure latency against the same PyTorch checkpoint, batch size, hardware, and warm-up protocol.

Run scripts/02_onnx_export.py from the companion repository to export a model and dynamically quantize its ONNX artifact. It needs that environment and downloads a checkpoint, so it is a command to run there rather than a self-contained article snippet:

uv run python scripts/02_onnx_export.py

INT8 quantization

Dynamic quantization can reduce an ONNX model’s storage and memory requirements. The effect on latency and per-label F1 depends on the checkpoint and CPU, so the export script is packaging evidence rather than a deployment benchmark.

from onnxruntime.quantization import quantize_dynamic, QuantType

# Quantize weights dynamically, then evaluate the result
quantize_dynamic("gliner.onnx", "gliner_int8.onnx", weight_type=QuantType.QInt8)

gline-rs: Rust reimplementation

gline-rs (Apache 2.0) removes the Python runtime from the inference path. Its v0.9.0 token-mode benchmark on an Intel i9 with three labels reports 6.67 seq/s versus Python’s 1.61, on 100 NuNER samples. A separate v0.9.1 RTX 4080 experiment on 1,000 samples reports 248.75 seq/s, without a matched Python GPU comparison. Those are the project’s own conditions, not results reproduced by the companion repo. It supports span and token models, GPU/NPU via ONNX Runtime, and ships as a crate on crates.io.

use gliner::{GLiNER, TokenMode, Parameters, RuntimeParameters, TextInput};

let model = GLiNER::<TokenMode>::new(
    Parameters::default(), RuntimeParameters::default(),
    "tokenizer.json", "model.onnx")?;

let input = TextInput::from_str(
    &["My name is James Bond."], &["person", "vehicle"])?;
let output = model.inference(input)?;
// => "James Bond" : "person" (99.7%)

The fast-gliner package provides Python bindings via PyO3.

What the optimization evidence covers

Structured extraction: native schemas, Instructor, and local decoders

When you need more flexibility than encoder models offer — implicit entities, reasoning, ontology mapping — begin with the provider’s native schema mechanism. OpenAI supports strict json_schema response formats, Anthropic structured outputs supports JSON outputs and strict tool inputs, and the Gemini API supports a JSON Schema subset. A shared Pydantic or Zod model can describe the contract, but each provider accepts a different schema subset and has different refusal and complexity behavior.

Schema conformance makes parsing reliable. It does not prove that an extracted span exists in the source text or that a normalized value is correct. Validate field values, preserve offsets or quotations when possible, and score semantic accuracy on a labeled set.

Instructor wraps provider clients with Pydantic validation and optional retries after validation failures.

Adapted from the Instructor pattern in scripts/05_structured_extraction.py. It requires the listed packages and an OPENAI_API_KEY; the companion script falls back to GLiNER when no key is available. This article now uses GPT-5.6 Terra, whose model card supports Chat Completions, function calling, structured outputs, and none reasoning effort. Compare it with the cheaper older baseline before switching a high-volume extractor; a new generation does not establish better NER accuracy.

import instructor
from pydantic import BaseModel
from typing import List, Literal
from openai import OpenAI

class Entity(BaseModel):
    name: str
    label: Literal["PERSON", "ORGANIZATION", "LOCATION"]

class ExtractEntities(BaseModel):
    entities: List[Entity]

client = instructor.from_openai(OpenAI())
result = client.chat.completions.create(
    model="gpt-5.6-terra", reasoning_effort="none",
    response_model=ExtractEntities,
    messages=[{"role": "user", "content": "BioNTech SE acquired InstaDeep in the U.K."}])
# entities=[Entity(name='BioNTech SE', label='ORGANIZATION'), ...]

Outlines by dottxt takes a different approach: constrained token generation through finite-state machines. The decoder masks tokens that would violate the target grammar instead of waiting for a validation failure and retrying. An AWS overview cites 98% schema adherence versus 76% for post-generation validation. It separately repeats .txt Engineering’s claim of up to 5 times faster generation from its coalescence approach; the page does not publish enough methodology to treat both figures as one controlled benchmark.

The following small local example keeps Phi-3 as a historical decoder-integration example, not a 2026 model recommendation. For a new self-hosted extraction comparison, include current Qwen3.8-27B or the smaller Qwen3.5 checkpoints. Use their documented model loader, chat template, and serving backend; replacing the string below alone does not establish compatibility with a multimodal architecture or its reasoning format.

import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
from pydantic import BaseModel
from typing import Literal

class Entity(BaseModel):
    name: str
    label: Literal["PERSON", "ORGANIZATION", "LOCATION"]

class ExtractEntities(BaseModel):
    entities: list[Entity]

model_id = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(model_id),
    AutoTokenizer.from_pretrained(model_id),
)
result = model(
    "Extract entities from: BioNTech SE acquired InstaDeep in the U.K.",
    ExtractEntities,
)

LangExtract is useful when a generated field must be grounded in the source: it returns character intervals and supports hosted or local LLM backends. For self-hosted document extraction, NuExtract converts JSON schemas to templates and includes multimodal document models. Treat both as structured-extraction systems, not automatic replacements for span NER. Their field, offset, document-layout, and latency targets need their own tests.

The choice depends on where you run your models. Native schemas are the lowest-friction path for a supported provider. Instructor adds a provider-agnostic Pydantic validation and retry layer. Outlines constrains local generation against a schema. LangExtract prioritizes source grounding, while NuExtract targets self-hosted document extraction. All LLM paths still include autoregressive generation. Benchmark each path against an encoder on the same batch size, hardware, entity schema, and semantic-accuracy rubric.

The three-tier production architecture

I would route production NER by task shape rather than by one model ranking.

Tier 1: encoder models for explicit spans. Use a GLiNER cross-encoder for a small type inventory. When a large inventory is reused, compare the bi-encoder with cached type embeddings. Fine-tune through the LLM-as-teacher pipeline, then deploy with native serving, ONNX, INT8, or gline-rs only when that path passes the domain benchmark.

Tier 2: multi-task or relation extraction. When one request needs NER, classification, and hierarchical fields, test the current 194M-parameter GLiNER2.5 checkpoint against separate per-task baselines. When the central requirement is joint spans plus relations, test GLiNER-Relex. The GLiNER2 paper reports 130-208 ms CPU classification latency across its tested label counts; it is not evidence for the later relation API or for a different deployment.

Tier 3: LLMs for reasoning-heavy extraction. Route implicit entities, contextual inference, and ontology mapping to a native schema API or Instructor for cloud APIs, and to Outlines for local constrained output. Use LangExtract when source intervals are essential and NuExtract when the document itself is the input. Log these cases for review. Only explicit, source-aligned spans can become ordinary Tier 1 training targets; inferred facts and normalized values need their own targets and evaluation.

The CFM case study gives one cost reference for Tier 1: 93.4% F1 at a reported $0.10 per hour on CPU, compared with 92.7% F1 and $8 per hour for its Llama-70B teacher. Hourly instance prices do not determine cost per document without throughput. Recalculate with your hardware, teacher model, label set, utilization, and review cost.

Trade-offs and limitations

For each trade-off below, the useful questions are where it shows up and whether you can measure it before deployment.

LLM-as-teacher errors propagate. If the LLM consistently gets a specific entity type wrong (e.g., confusing subsidiary names with parent companies), the fine-tuned encoder inherits that bias. Review low-confidence or inconsistent types heavily, and keep a stratified random sample to catch confident systematic errors.

A valid schema can contain false facts. Native structured output, Instructor, and constrained decoders can make a response parseable. They cannot ensure that every field is grounded, that a span boundary is correct, or that a normalized value maps to the right record. Retain source evidence and validate semantics separately.

Quantization losses are checkpoint- and data-dependent. The companion creates a dynamically quantized INT8 artifact but does not measure its F1. Current GLiNER guidance recommends quantization-aware training when INT8 accuracy must be preserved. Compare the quantized and original checkpoints on exact-span F1 and per-label recall before deployment.

When the three-tier architecture is overkill. A single domain with stable entity types and enough labeled examples may need only a fine-tuned RoBERTa or spaCy pipeline. The three-tier pattern fits multiple domains, evolving entity types, or a measured mix of explicit and reasoning-heavy extraction. A narrow invoice pipeline that extracts names and dates may stop at Tier 1.

Bi-encoder quality varies by dataset. Joint encoding can help on some datasets, while the bi-encoder wins the paper’s CrossNER comparison and the uni-encoder edges it on CoNLL-2003. Benchmark both on the domain set; choose from measured exact-span quality, calibration, label count, and throughput rather than using “high stakes” as a model-family rule.

PII and multilingual claims require slices. A high aggregate score can hide a dangerous recall loss for a locale, name form, document layout, or rare PII class. Treat a privacy model as one defense in depth, set a false-negative response process, and re-evaluate when the taxonomy, language mix, or data source changes.

Key takeaways

1. Use a compact encoder for explicit spans only after it passes a domain test set with per-type support and confidence intervals.
2. Use GLiNER for changing label vocabularies. Compare its bi-encoder first when a large type inventory is reused; enable flat_ner=False only after measuring nested-span quality.
3. Use tested type descriptions for hard zero-shot claims, and test English and localized label wording separately for multilingual products.
4. Keep OCR, NER, relation extraction, and entity linking as distinct evaluation stages even when one model exposes several tasks.
5. Treat an LLM teacher as an annotation proposal system. Human guidelines, adjudication, immutable regression sets, and a held-out test set remain required.
6. Use native schemas for supported LLM providers, but validate semantic correctness and source grounding separately from JSON validity.
7. Benchmark native serving, ONNX, quantization, and Rust paths separately and together. Never multiply unmeasured speedups.

References

Papers

• GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer - Zaratiana et al., NAACL 2024. The foundational span-entity matching architecture.
• GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface - Zaratiana et al., EMNLP 2025 System Demonstrations. Unifies NER, classification, and hierarchical extraction; current releases later added relation extraction.
• GLiNER Bi-Encoder: Scalable Named Entity Recognition with Bi-Encoder Architecture - Stepanov et al., February 2026. Decoupled encoding for million-label scale.
• GLiNER-Relex: A Unified Framework for Joint Named Entity Recognition and Relation Extraction - Stepanov et al., 2026. Joint, zero-shot entity and relation extraction.
• GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction - Zaratiana et al., 2026. 42 PII types at character-span resolution.
• ZeroNER: Fueling Zero-Shot Named Entity Recognition via Entity Type Descriptions - Cocchieri et al., ACL 2025. Hard zero-shot evaluation with type descriptions.
• OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages - Palen-Michel et al., EMNLP 2025. 36 corpora, 52 languages, and multi-ontology baselines.
• FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition - Golde et al., EACL 2026. Multilingual labels and transfer evaluation.
• UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition - Zhou et al., ICLR 2024. LLM-based universal NER via distillation from ChatGPT.
• NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data - Bogdanov et al., EMNLP 2024. Studies a 125M encoder pretrained on LLM annotations.

Industry papers

• EAMT: Entity-Aware Multi-Task Learning for Query Understanding - Walmart, KDD 2023. Multitask query study; the online result belongs to its MTDNN baseline.
• TripleLearn: End-to-End NER for E-Commerce Search - Home Depot, AAAI 2021. F1 from 69.5 to 93.3.
• Acing the IOC Game: Toward Automatic Discovery and Analysis of Open-Source Cyber Threat Intelligence - Liao et al., CCS 2016. Historical extraction approach.
• CyberNER: A Harmonized STIX Corpus for Cybersecurity NER - 21 STIX 2.1-aligned entity types.
• FinBERT-MRC: Financial NER via Machine Reading Comprehension - MRC formulation evaluated on Chinese financial datasets.

Case studies

• CFM Case Study: Fine-tuning GLiNER for Financial NER - Capital Fund Management’s $70 LLM labeling pipeline achieving 93.4% F1.
• Refuel AI: LLM Labeling Technical Report - GPT-4 achieving 88.4% annotation agreement, exceeding human annotators.
• Sease: GLiNER as an Alternative to LLMs for Query Parsing - Where GLiNER fails and LLMs are still needed.
• John Snow Labs: Medical Text De-Identification Benchmark - Vendor study with label remapping and exclusions.
• OpenMed: Year in Review 2025 - January 2026 retrospective; inventory and downloads are not deployments.

Tools and frameworks

• ner-field-guide demo repo - Companion demos for this article: GLiNER quickstart, ONNX export, LLM-as-teacher, and structured extraction.
• gline-rs: Rust reimplementation of GLiNER - Separate maintainer CPU and GPU experiments; Apache 2.0 code.
• GLiNER serving - Ray Serve path with dynamic batching and multi-replica deployment.
• spaCy English pipelines - Versioned closed-label pipeline metrics.
• Microsoft Presidio - PII analysis and anonymization with custom recognizers.
• Instructor - Structured LLM extraction via Pydantic models.
• Outlines - Constrained token generation via FSMs, with a reported schema-adherence benchmark.
• LangExtract - Source-grounded structured extraction with character intervals.
• NuExtract - Self-hosted, schema-to-template document extraction.
• OpenAI Structured Outputs - Strict JSON-schema response format reference.
• Anthropic Structured Outputs - JSON outputs and strict tool inputs.
• Gemini Structured Outputs - JSON Schema subset for structured responses.
• AWS: Structured Output with Outlines - 98% schema adherence benchmark.