Docling in Production RAG: Where PDF Chunking and Table Extraction Break

Why We Tested Docling for PDF Retrieval#

We tested Docling because PDF extraction errors hurt retrieval, the process of finding source material for an answer. Our vector database stores numerical representations of text for similarity searches, but it could not compensate for malformed input.

Our PDF extraction process read columns in the wrong order and reduced tables to unstructured text. It also separated section titles from their paragraphs. Repeated headers cluttered chunks, the pieces of text we indexed for search. Embedding model changes did not fix those problems. Neither did switching vector databases. We were indexing malformed source material and then asking retrieval infrastructure to compensate.

Docling prepares documents before search indexing. We used it to create a DoclingDocument, a structured record of each PDF’s text, headings, and tables. We exported Markdown for readable, formatted text and JSON for named fields and values that software can process. An embedding represents text as numbers for similarity searches. Docling’s HybridChunker divides documents into pieces sized for the model that creates those numbers. We fixed the package version at 2.92.0 because our loading process depends on its document structure and serialization interfaces. Serialization turns those structures into text or files that other software can use.

Docling requires more processing than a basic PDF text extractor. Our pipeline could analyze page layouts, identify table rows and columns, and create page images. It could also use optical character recognition, or OCR, to read text from images before building a structured document. These steps improved document structure but required model downloads, more processing power from the central processing unit, or CPU, and more memory. They also gave us more deployment settings to manage.

Our evaluation focused on digitally generated manuals, table-heavy reports, mixed text-and-scan PDFs, and scanned documents.

We evaluated representative tables and answer-bearing document regions. For tables, we compared normalized cell text, meaning text put into a consistent form for comparison. We also compared row order, column order, and merged-cell placement. For retrieval, we checked whether the expected answer-bearing region appeared in the first five returned chunks. We tested document preparation for retrieval-augmented generation, or RAG, which supplies retrieved source material to a language model before it answers. We evaluated the output our services use, rather than OCR alone.

We treated the pinned Docling chunking implementation and examples as an API reference while building the harness. We also exercised the advanced chunking and serialization path because default Markdown export was not sufficient for every table-heavy document.

The main benefit was running Docling on infrastructure we controlled. We could keep enterprise documents out of an external parsing service and retain their structured JSON records. With the same input and settings, we could also generate the same chunks each time. That control matters when regulations or contracts restrict document handling.

The core warning was equally clear: Docling is not a drop-in replacement for a lightweight PDF text library. We had to operate it as a model-backed document-processing service.

Hands-On Walkthrough: Setup, Execution & Output#

We tested Docling 2.92.0 in a Python 3.11 virtual environment and in a Debian-based container. For repeatable evaluation, we need to record CPU capacity, available memory, and whether GPU acceleration is enabled. We stored downloaded model files in a cache that survived container restarts, so each restart did not download them again.

Our minimal environment setup was:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install "docling==2.92.0" transformers psutil

A tokenizer divides text into tokens, the units a model counts when processing input. We used the embedding model’s Hugging Face tokenizer inside HybridChunker. Character or word counts did not reliably predict whether a chunk would fit the embedding service’s token limit.

The following script converts one PDF and saves a primary JSON record plus readable Markdown. It also adds headings and captions to chunks and writes JSON Lines, a format with one JSON record per line:

# ingest_docling.py
import argparse
import json
import time
from pathlib import Path

import psutil
from transformers import AutoTokenizer

from docling.chunking import HybridChunker
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions, TableFormerMode
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling_core.transforms.chunker.tokenizer.huggingface import HuggingFaceTokenizer


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("pdf", type=Path)
    parser.add_argument("--out", type=Path, default=Path("artifacts"))
    parser.add_argument(
        "--tokenizer",
        default="sentence-transformers/all-MiniLM-L6-v2",
    )
    parser.add_argument("--max-tokens", type=int, default=384)
    parser.add_argument("--ocr", action="store_true")
    args = parser.parse_args()

    args.out.mkdir(parents=True, exist_ok=True)

    pipeline = PdfPipelineOptions()
    pipeline.do_ocr = args.ocr
    pipeline.do_table_structure = True
    pipeline.table_structure_options.mode = TableFormerMode.ACCURATE

    converter = DocumentConverter(
        format_options={
            InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline)
        }
    )

    started = time.perf_counter()
    result = converter.convert(args.pdf)
    document = result.document

    hf_tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
    tokenizer = HuggingFaceTokenizer(
        tokenizer=hf_tokenizer,
        max_tokens=args.max_tokens,
    )
    chunker = HybridChunker(tokenizer=tokenizer, merge_peers=True)

    chunks = list(chunker.chunk(dl_doc=document))

    json_path = args.out / f"{args.pdf.stem}.docling.json"
    md_path = args.out / f"{args.pdf.stem}.md"
    chunks_path = args.out / f"{args.pdf.stem}.chunks.jsonl"

    json_path.write_text(
        json.dumps(document.export_to_dict(), indent=2, ensure_ascii=False),
        encoding="utf-8",
    )
    md_path.write_text(document.export_to_markdown(), encoding="utf-8")

    with chunks_path.open("w", encoding="utf-8") as handle:
        for index, chunk in enumerate(chunks):
            contextualized = chunker.contextualize(chunk=chunk)
            handle.write(
                json.dumps(
                    {
                        "chunk_id": index,
                        "text": contextualized,
                        "meta": chunk.meta.export_json_dict(),
                    },
                    ensure_ascii=False,
                )
                + "\n"
            )

    process = psutil.Process()
    elapsed = time.perf_counter() - started

    print(f"input={args.pdf}")
    print(f"pages={len(document.pages)}")
    print(f"chunks={len(chunks)}")
    print(f"elapsed_seconds={elapsed:.2f}")
    print(f"current_rss_mb={process.memory_info().rss / 1024 / 1024:.1f}")
    print(f"markdown={md_path}")
    print(f"json={json_path}")
    print(f"chunk_jsonl={chunks_path}")


if __name__ == "__main__":
    main()

We ran it as follows:

python ingest_docling.py samples/quarterly-operations.pdf \
  --out artifacts \
  --ocr \
  --max-tokens 384

Illustrative output format only; the values below are placeholders, not verified measurements:

input=samples/quarterly-operations.pdf
pages=47
chunks=136
elapsed_seconds=93.84
current_rss_mb=2874.6
markdown=artifacts/quarterly-operations.md
json=artifacts/quarterly-operations.docling.json
chunk_jsonl=artifacts/quarterly-operations.chunks.jsonl

The JSONL output retained contextual material that plain text splitting lost:

{
  "chunk_id": 41,
  "text": "Operations Review > Regional Performance\n\nNorth America revenue increased 8.4% while support cost per account declined 3.1%.",
  "meta": {
    "headings": ["Operations Review", "Regional Performance"],
    "page_numbers": [19]
  }
}

We stored DoclingDocument JSON as the authoritative record and treated Markdown as a secondary export. That let us regenerate chunks without rerunning PDF parsing when we changed the embedding tokenizer or chunk-size policy.

For custom serialization, we used Docling’s serializer layer rather than applying regular expressions to generated Markdown. Regular expressions are rules for matching patterns in text. The relevant extension points were BaseDocSerializer, BaseTableSerializer, and SerializationResult. We wired a custom table serializer into MarkdownDocSerializer so table objects could become HTML while ordinary prose remained Markdown. HTML’s rowspan and colspan attributes let a table cell span multiple rows or columns. They preserved merged cells more reliably for later display than Markdown tables drawn with vertical bars.

Our application called a small wrapper rather than Docling’s serialization code directly. The wrapper accepted a DoclingDocument and returned output files with version labels. When Docling’s API changed, we could update the wrapper in one place.

Deployment Problems and Extraction Limitations#

The first deployment problem was the size of Docling’s required software packages. pip install docling was easy, but the resulting runtime was not small. Parsing with machine-learning models required deep-learning and image-processing packages. The first real conversion also required model files that were missing from a clean container.

Our first isolated deployment failed because it had no outbound network access. We downloaded the models while building the container image and included those cached files in the deployed image. We then configured production processing to run offline. We fixed the Docling version and saved a lock file recording dependency versions. Installing whichever release was available into an old image did not reliably reproduce the same environment.

OCR created the next issue. Enabling it indiscriminately increased latency and occasionally replaced correct embedded text with slightly worse recognized text. We applied three OCR policies based on document type:

• Digital PDFs ran with OCR disabled.
• Fully scanned PDFs ran with OCR enabled.
• For mixed documents, we checked how much embedded text each page contained before deciding whether to use OCR.

That routing reduced wasted work and prevented OCR from touching pages that already had trustworthy text.

Table extraction worked well on ordinary tables with or without visible grid lines, but it was not flawless. It failed on some rotated headers, tables inside other tables, tables spanning pages, and footnotes inside the areas identified as tables. One annual report produced a visually plausible Markdown table whose last two columns were shifted by one cell after a merged header.

We did not accept extracted tables without checks. We added structural checks for inconsistent row widths, abrupt header changes, suspiciously empty columns, and tables split across adjacent pages. We retained tables that failed our checks as structured JSON with a reference to the page image. For high-value financial material, we routed flagged tables to review rather than embedding incorrect flattened text.

Exporting to Markdown also lost information. Tables drawn with vertical bars cannot faithfully represent every arrangement of merged cells. Repeated whitespace, footnote markers, and line breaks inside cells changed during export. We therefore indexed contextualized table text but preserved the JSON document as the audit source. Where table layout affected meaning, our custom serializer produced HTML tables.

Tokenizer mismatch was the easiest mistake to introduce and one of the most consequential. We initially chunked with one tokenizer and embedded with another. Chunks that appeared to fit a 384-token budget exceeded the embedding service’s effective limit, so part of the text was cut off when we sent the request. The missing text was often the table row itself or the paragraph containing the answer.

We corrected this by loading the exact tokenizer associated with the embedding model and passing it to HybridChunker. For each document-loading run, we saved a record of the tokenizer identifier, maximum token count, Docling version, and serializer version. When we change embedding models, we generate new chunks from the saved document structure without extracting the PDF again.

HybridChunker.contextualize() made individual chunks easier to understand by adding headings and captions, but that context consumes tokens. We found several table chunks where repeated heading context displaced useful rows. Our serializer now budgets context and content separately, with shorter heading paths for deeply nested documents.

Large PDFs exposed the operational limit. Peak memory did not track final text size; it tracked rendered pages, intermediate images, OCR state, and table models. We treated large scanned PDFs as a memory-pressure risk and limited concurrent conversions rather than sizing workers from the final JSON artifact alone.

We limited how many queued jobs could start at once rather than relying on automatic capacity increases. Each CPU worker’s limit depended on document type, page count, and whether OCR was enabled. Files above our threshold ran in isolated processes so memory returned to the operating system after each conversion. We also rejected encrypted or malformed PDFs early and imposed page and file-size limits at upload.

Scale, Latency & Cost vs. Alternatives#

For a repeatable performance comparison, we would separate cold-start runs from warmed runs with models already downloaded and loaded. We would measure time per page, peak resident memory, and table fidelity on the same corpus and hardware. The locked evidence does not establish numerical results for that comparison.

For clean embedded text with simple reading order and no meaningful tables, we would benchmark Docling against a lightweight extractor such as PyMuPDF before choosing the processing path. The difference was output structure. Docling retained headings, source-location information, captions, and structured tables. Otherwise, we would need custom rules to recover that information.

We would compare Docling with an Unstructured high-resolution configuration on the same corpus, scoring table boundaries and document hierarchy before selecting either tool.

With commercial parsing, we weighed per-page costs and operational convenience instead of infrastructure requirements. We modeled a managed service at $0.01 per page and a CPU worker at $0.34 per hour. We would calculate raw compute cost per page from measured end-to-end throughput and the worker's hourly price, then account separately for storage, queueing, engineering, and idle capacity.

We would not justify self-hosting from infrastructure prices alone. A break-even estimate requires measured throughput, actual engineering effort, and comparable managed-service pricing; the available evidence does not establish a page-volume threshold.

We would also account for ongoing maintenance. Requirements about where documents stay, repeatable output, and control over saved chunk formats may justify self-hosting independently of cost savings. Those benefits concern data handling and oversight, not lower computing costs.

At lower volumes, we would consider a managed parser if it removes operational work. At higher volumes, a controlled Docling service may become more compelling, but the crossover depends on measured throughput, worker utilization, and engineering costs. Teams can review our broader AI infrastructure tools collection or use our implementation services when comparing build and buy paths.

We also keep this pre-query RAG pipeline checklist alongside our own ingestion reviews because teams should check text extraction, formatting consistency, and document labels before tuning search.

Our Final Verdict: When to Deploy, When to Skip#

Our verdict is positive, with operational conditions.

Docling 2.92.0 produced substantially better output for ingestion than plain PDF text extraction. The combination of DoclingDocument, table-aware conversion, HybridChunker, embedding-tokenizer alignment, and custom serializers addressed real retrieval failures in our test harness. It worked best when we treated parsing output as reusable data with tracked versions rather than disposable text.

We would deploy Docling if:

• We need local or private processing for sensitive documents.
• Our corpus contains tables, headings, captions, and multi-column layouts.
• We can pin dependencies and prepackage model assets.
• We are willing to store canonical Docling JSON alongside Markdown.
• We can use the exact embedding tokenizer during chunk construction.
• We can route digital, mixed, and scanned PDFs through different policies.
• We have enough volume or governance pressure to justify operating workers.
• We can validate tables instead of assuming every plausible-looking result is correct.

We would hold off or avoid Docling if:

• Our PDFs are clean, text-native files with simple reading order.
• We only need raw text and maximum CPU throughput.
• Our hosting platform allows only a small deployment package or little startup time after the service has been idle.
• We cannot accommodate the memory pressure that large scanned PDFs may create.
• We want zero model-cache or dependency management.
• We require perfect reconstruction of complex financial tables without review.
• Our ingestion volume is low enough that paying a managed service per page costs less than having our engineers operate the pipeline.

Before production, we would require fixed package versions, cached models available offline, and a record of which tokenizer each loading run used. We would also limit simultaneous jobs by document type and keep difficult tables as repeatable tests to catch problems introduced by later changes. We would retain source-page references so reviewers could trace every retrieved chunk to its original page.

The biggest mistake would be to evaluate Docling only by opening one generated Markdown file. A readable document can still contain shifted table cells, silent OCR substitutions, or chunks that exceed the embedding model’s token budget. We found the tool valuable because we tested the entire path from PDF bytes to retrievable chunk, not because the first export looked polished.

For table-heavy, privacy-sensitive RAG ingestion, we would deploy it behind a queue with explicit resource controls. For ordinary digital PDFs, we would keep a lightweight fast path and invoke Docling only where document structure justifies the cost.

If the decision depends on a private corpus rather than generic benchmark files, contact our team. We would compare representative documents and score how well retrieved chunks contain the answers before choosing the document-processing setup.

What This Article Could Not Verify#

These tests do not establish performance on documents outside our corpus or on different hardware.