Semantic Caching for AI APIs: Slashing Latency and API Bills with Vector Redis

Overview #

Semantic caching uses vector embeddings and similarity thresholds to intercept and serve semantically equivalent user queries from an in-memory cache, reducing LLM API costs by up to 60% and returning responses in sub-10ms.

Why Exact-Match Caching (HTTP Cache) Fails in AI #

Traditional web caching relies on exact string hashing (MD5/SHA256 of URL and body). In conversational AI, users ask identical questions using different phrasing: 'How do I cancel my plan?' vs. 'I want to cancel my subscription'. Traditional caches miss 100% of these variations. Semantic caching embeds the incoming prompt and performs vector similarity search against previously stored queries.

Setting the Distance Threshold (Cosine vs. Euclidean) #

The core operational knob is the similarity threshold (typically 0.90 to 0.95 cosine similarity). Setting the threshold too low causes false positive cache hits (returning answers to conceptually distinct questions). Setting it too high degrades cache hit ratio. Dynamic thresholds based on query classification maximize efficiency.

Time-to-Live (TTL) & Cache Invalidation #

Cached answers must expire periodically (TTL of 24h to 7d) or invalidate immediately when underlying knowledge changes (e.g. after a documentation or policy update), ensuring users never receive obsolete answers.

Code Example: Implementing Semantic Caching with Redis and Embeddings #

import numpy as np

class SimpleSemanticCache:
    def __init__(self, threshold=0.92):
        self.cache = [] # List of (vector, response_text)
        self.threshold = threshold

    def get(self, query_vector):
        for cached_vec, response in self.cache:
            similarity = np.dot(query_vector, cached_vec) / (np.linalg.norm(query_vector) * np.linalg.norm(cached_vec))
            if similarity >= self.threshold:
                return response, similarity
        return None, 0.0

    def set(self, query_vector, response_text):
        self.cache.append((query_vector, response_text))

# Usage
cache = SimpleSemanticCache()
# If hit: returns in 2ms without invoking LLM API

Frequently Asked Questions #

Q: How much latency does a semantic cache save?

A semantic cache hit resolves in 5ms to 15ms from Redis, compared to 800ms to 3,000ms for a round-trip LLM API completion.

Q: Does semantic caching work for streaming responses?

Yes, cached responses can be stored as pre-chunked SSE streams or replayed as simulated streams for smooth client rendering.

Q: Is semantic caching safe for personalized user data?

For personalized queries, the cache key must partition by tenant_id or user_id to prevent cross-account data leakage.