What is Semantic Caching?
Semantic caching stores responses alongside vector representations of prompts so applications can retrieve them for similar requests.
Prologue
Caching saves time and money. Semantic caching can speed up responses from a large language model (LLM) application when a similar request has been answered before.
How Does Semantic Caching Work?
Semantic refers to meaning. Semantic caching stores responses with embeddings, which represent the meaning of prompts as vectors of numbers. How does that help us retrieve a response?
Traditional key-value caching relies on exact keys. It treats prompts such as "I want to buy the latest MacBook Pro" and "purchase the latest MBP" as distinct requests, even though they express a similar intent.
To connect these prompts, we convert each one into an embedding and measure how similar the embeddings are. If the similarity exceeds a chosen threshold, the application can return a cached response.
A Simplified Workflow
Benefits
Faster response
For a cache hit, the application can return a stored response without waiting for another LLM call. This can reduce response times from seconds to milliseconds and let the same system serve more requests.
Lower cost
LLM calls can be expensive. Reusing a response for similar prompts reduces the number of model calls and their associated costs.
Better efficiency
By tuning the similarity threshold, you can match prompts with similar intent. A vector store helps the application find cached responses so it does not have to send every prompt through the LLM pipeline.
Reference: