FundamentalsIntermediate
What is an AI Inference API? GPU Compute, Model Hosting, and Execution
Direct Answer & Overview
An AI inference API is a cloud service that executes forward-pass neural network calculations on remote high-performance GPUs (like NVIDIA H100s or Google TPUs) to generate predictions or text from trained machine learning weights on demand.
1.Training vs. Inference: Understanding the Difference
• Training: The massive, months-long process where billions of tokens are fed into an uninitialized neural network to learn linguistic patterns, requiring tens of thousands of GPUs and millions of dollars in electricity.
• Inference: The operational execution phase where a user sends a prompt into the frozen, already-trained model weights. The model performs a single forward-pass matrix multiplication to predict the next word. Inference APIs allow developers to access trained intelligence without footing the multi-million-dollar training bill.
2.Prefill Phase vs. Decode Phase in LLM Inference
Inference consists of two distinct computational stages:
1. Prefill Phase: The model processes the entire incoming prompt in a single parallel matrix multiplication (compute-bound, high GPU utilization).
2. Decode Phase: The model generates completion tokens one-by-one sequentially, reading all model weights from GPU memory for every single token produced (memory-bandwidth-bound).
3.Serverless Inference vs. Dedicated GPU Hosting
Self-hosting a dedicated 8x H100 node costs $15,000-$25,000/month regardless of whether you send 1 request or 1 million requests. Inference APIs (like API100) are serverless: you pay strictly per token consumed, converting fixed capital infrastructure into variable operational efficiency.
Measuring TTFT (Time-To-First-Token) on Inference APIspython
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.apihundred.com/v1", api_key="your_key")
t0 = time.time()
stream = client.chat.completions.create(
model="gemini-3.8-flash",
messages=[{"role": "user", "content": "Explain gravity in one sentence."}],
stream=True
)
first_token_time = None
for chunk in stream:
if first_token_time is None:
first_token_time = time.time()
print(f"Time-To-First-Token (TTFT): {(first_token_time - t0)*1000:.2f}ms")
print(chunk.choices[0].delta.content or "", end="", flush=True)Frequently Asked Questions
What hardware runs AI inference APIs?
Inference APIs run on enterprise AI accelerators including NVIDIA H100/H200/B200 GPUs, AMD MI300X, and Google Trillium TPUs.
What is continuous batching in inference servers?
Continuous batching dynamically groups requests from different users as soon as older requests finish, maximizing GPU core utilization without waiting for static batch boundaries.
Why is inference speed measured in tokens per second (TPS)?
Because tokens are generated sequentially during the decode phase, TPS reflects how fast the user perceives streaming text on their screen.
A100
API100 Engineering Team
Infrastructure & Latency Research