Batch API vs Real-Time Inference: Getting 50% Discounts on Non-Urgent AI Tasks

Overview #

Batch APIs allow developers to queue large volumes of asynchronous inference requests to be processed within 24 hours at a 50% discount compared to real-time rates, dramatically lowering costs for evaluations, offline OCR, and data enrichment.

The Mechanics of Asynchronous Batch Processing #

In real-time inference, cloud providers must maintain reserve GPU capacity to respond within hundreds of milliseconds, resulting in expensive idle compute. Batch APIs take advantage of diurnal valley periods (e.g. overnight off-peak GPU hours). Developers upload a JSONL file of requests; the batch engine queues and executes them whenever surplus GPU capacity becomes available, guaranteeing results within a 24-hour turnaround window.

50% Token Price Reductions & Higher Rate Limits #

Both input and output tokens are billed at a flat 50% discount across major foundation models (such as GPT-4o and Claude 3.5 Sonnet). Furthermore, batch jobs operate against separate, vastly higher token pools, avoiding real-time TPM rate limit throttling during massive catalog migrations.

Ideal Workloads for Batch Processing #

• Offline Document Extraction: Parsing 500,000 scanned PDFs or historical invoices.
• Synthetic Dataset Generation: Distilling teacher model outputs for fine-tuning.
• AI System Evaluations (Evals): Running automated test suites across 10,000 prompt variations before releases.
• Content Personalization: Generating weekly email summaries or localized marketing copy.

Code Example: Submitting an Asynchronous Batch Job with JSONL #

from openai import OpenAI

client = OpenAI(
    base_url="https://api.apihundred.com/v1",
    api_key="your_api100_key"
)

# 1. Upload batch file containing formatted requests
batch_file = client.files.create(
    file=open("batch_queries.jsonl", "rb"),
    purpose="batch"
)

# 2. Create batch job with 24h completion window
batch_job = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Batch Job Created: {batch_job.id}, Status: {batch_job.status}")
# Tokens billed at 50% discount upon job execution

Frequently Asked Questions #

Q: Can I cancel a running batch job?

Yes, batch jobs can be cancelled via API, and you are only billed for requests that were already processed prior to cancellation.

Q: How long does a batch job typically take?

While the SLA window is 24 hours, most batch jobs complete in 15 minutes to 3 hours depending on cluster queue depth.

Q: Does Batch API support structured JSON outputs?

Yes, batch requests support the exact same parameters as real-time requests, including response_format and function calling.