Multimodal Document Parsing: Extracting PDFs, Tables, and Receipts into Clean JSON

Overview #

Multimodal document parsing leverages vision-language models to extract complex nested tables, multi-column text, handwritten annotations, and visual diagrams from PDF and TIFF files into deterministic, validated JSON schemas.

Why Traditional OCR (Tesseract) Fails on Modern Documents #

Traditional OCR engines analyze character shapes in isolation without semantic awareness. When encountering complex financial statements, rotated receipts, or multi-column newspaper layouts, traditional OCR scrambles reading order, collapses table cells, and loses context. Vision LLMs comprehend spatial typography, understanding that an invoice total in the bottom-right belongs to a line item above it.

Extracting Complex Nested Tables and Merged Cells #

By combining vision prompting with strict JSON schemas, developers can instruct models to extract merged headers, currency symbols, and multi-row line items into structured arrays. Even faint watermarks or skewed scans are accurately parsed.

PDF-to-Image Conversion Pipelines & Cost Scaling #

For massive 500-page enterprise archives, rendering every page at maximum resolution can incur high API costs. High-efficiency pipelines use local tools (such as pdf2image and PyMuPDF) to detect whether a page is pure digital text (routed through cheap regex or embedding extraction) or visual layout (routed to Vision LLM APIs).

Code Example: Extracting Tabular Invoices into Pydantic / JSON Schemas #

from openai import OpenAI

client = OpenAI(
    base_url="https://api.apihundred.com/v1",
    api_key="your_api100_key"
)

schema = {
    "type": "object",
    "properties": {
        "vendor_name": {"type": "string"},
        "invoice_number": {"type": "string"},
        "date": {"type": "string"},
        "total_amount": {"type": "number"},
        "items": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "description": {"type": "string"},
                    "quantity": {"type": "integer"},
                    "price": {"type": "number"}
                },
                "required": ["description", "quantity", "price"]
            }
        }
    },
    "required": ["vendor_name", "invoice_number", "total_amount", "items"]
}

response = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=[
        {"role": "system", "content": "Extract invoice details strictly following the JSON schema."},
        {"role": "user", "content": [{"type": "text", "text": "Parse this document."}, {"type": "image_url", "image_url": {"url": "https://storage.apihundred.com/invoices/sample.png"}}]}
    ],
    response_format={"type": "json_object"}
)
print(response.choices[0].message.content)

Frequently Asked Questions #

Q: How do I process multi-page PDFs with Vision APIs?

Convert each PDF page into a JPEG or PNG image using pdf2image, then dispatch them concurrently in batched API calls.

Q: Can LLMs reliably calculate table sums during extraction?

While LLMs can extract figures, it is best practice to perform mathematical verification in post-processing code rather than relying on LLM arithmetic.

Q: What is the best model for dense document OCR?

Claude Sonnet 5 and GPT-6 Astra lead benchmarks for complex document and financial table layout comprehension.