Vision LLM APIs: Image Analysis, Spatial Grounding, and OCR

Overview #

Vision LLM APIs allow developers to pass images and visual documents directly into language models alongside text prompts, transforming raw pixels into structured data, semantic descriptions, and spatial coordinate annotations.

How Vision Transformers Tokenize Images (Patching & Resolution) #

Vision language models do not process images as 2D pixel grids directly. Instead, vision encoders (such as Vision Transformers, or ViT) divide the image into a grid of uniform patches (typically 14x14 or 16x16 pixels). Each patch is linearly projected into a vector embedding and concatenated with positional embeddings, effectively converting the image into a sequence of 'visual tokens' that the LLM's self-attention layers process identically to text tokens.

Detail Modes: Low vs. High Resolution Token Economics #

In APIs like GPT-6 and Claude 5, images can be ingested in 'low' or 'high' detail modes. Low-detail mode resizes the image to 512x512 pixels and consumes a flat fee of 85 tokens. High-detail mode breaks the image into multiple 512x512 tiles plus a base thumbnail, consuming 170 tokens per tile. Understanding this distinction is essential for cost management in high-throughput OCR and image catalog processing.

Spatial Reasoning, Object Grounding, and Bounding Boxes #

Modern vision models can output normalized coordinate bounding boxes [ymin, xmin, ymax, xmax] ranging from 0 to 1000. This enables automated UI testing, robotic manipulation, and defect detection in manufacturing assemblies directly from camera snapshots.

Code Example: Passing Base64 and Hosted Image URLs to Vision API #

import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.apihundred.com/v1",
    api_key="your_api100_key"
)

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Identify any defects or scratches on this automotive panel with coordinates."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://images.unsplash.com/photo-1503376780353-7e6692767b70?auto=format&fit=crop&w=800&q=80",
                        "detail": "high"
                    }
                }
            ]
        }
    ],
    max_tokens=500
)
print(response.choices[0].message.content)

Frequently Asked Questions #

Q: What image formats are supported by Vision APIs?

Most vision APIs accept PNG, JPEG, WEBP, and non-animated GIF formats, either as base64-encoded strings or public HTTPS URLs.

Q: How are image tokens billed in AI APIs?

Image tokens are counted as input tokens. A standard high-detail 1080p image typically generates between 800 and 1,600 input tokens depending on aspect ratio.

Q: Can Vision LLMs read handwritten text?

Yes, models like Claude Opus 5.5 and GPT-6 Astra exhibit strong accuracy on handwritten medical records, cursive notes, and weathered historical documents.