Multimodal & AudioIntermediate
Multimodal Document Parsing: Extracting PDFs, Tables, and Receipts into Clean JSON
Direct Answer & Overview
Multimodal document parsing leverages vision-language models to extract complex nested tables, multi-column text, handwritten annotations, and visual diagrams from PDF and TIFF files into deterministic, validated JSON schemas.
1.Why Traditional OCR (Tesseract) Fails on Modern Documents
Traditional OCR engines analyze character shapes in isolation without semantic awareness. When encountering complex financial statements, rotated receipts, or multi-column newspaper layouts, traditional OCR scrambles reading order, collapses table cells, and loses context. Vision LLMs comprehend spatial typography, understanding that an invoice total in the bottom-right belongs to a line item above it.
2.Extracting Complex Nested Tables and Merged Cells
By combining vision prompting with strict JSON schemas, developers can instruct models to extract merged headers, currency symbols, and multi-row line items into structured arrays. Even faint watermarks or skewed scans are accurately parsed.
3.PDF-to-Image Conversion Pipelines & Cost Scaling
For massive 500-page enterprise archives, rendering every page at maximum resolution can incur high API costs. High-efficiency pipelines use local tools (such as pdf2image and PyMuPDF) to detect whether a page is pure digital text (routed through cheap regex or embedding extraction) or visual layout (routed to Vision LLM APIs).
Extracting Tabular Invoices into Pydantic / JSON Schemaspython
from openai import OpenAI
client = OpenAI(
base_url="https://api.apihundred.com/v1",
api_key="your_api100_key"
)
schema = {
"type": "object",
"properties": {
"vendor_name": {"type": "string"},
"invoice_number": {"type": "string"},
"date": {"type": "string"},
"total_amount": {"type": "number"},
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"description": {"type": "string"},
"quantity": {"type": "integer"},
"price": {"type": "number"}
},
"required": ["description", "quantity", "price"]
}
}
},
"required": ["vendor_name", "invoice_number", "total_amount", "items"]
}
response = client.chat.completions.create(
model="claude-sonnet-5",
messages=[
{"role": "system", "content": "Extract invoice details strictly following the JSON schema."},
{"role": "user", "content": [{"type": "text", "text": "Parse this document."}, {"type": "image_url", "image_url": {"url": "https://storage.apihundred.com/invoices/sample.png"}}]}
],
response_format={"type": "json_object"}
)
print(response.choices[0].message.content)Frequently Asked Questions
How do I process multi-page PDFs with Vision APIs?
Convert each PDF page into a JPEG or PNG image using pdf2image, then dispatch them concurrently in batched API calls.
Can LLMs reliably calculate table sums during extraction?
While LLMs can extract figures, it is best practice to perform mathematical verification in post-processing code rather than relying on LLM arithmetic.
What is the best model for dense document OCR?
Claude Sonnet 5 and GPT-6 Astra lead benchmarks for complex document and financial table layout comprehension.
A100
API100 Engineering Team
Infrastructure & Latency Research