A familiar setup: you have a large system instruction, a schema description or documentation that goes into every request. The model reads it fresh each time, and you pay each time.
When caching pays off
Three conditions have to hold together:
- The repeated part is large. Saving on a couple of paragraphs is pointless.
- It is genuinely identical between requests, down to the character.
- Requests come frequently. Caches expire, and if too long passes between calls it will not help.
Typical fits: an agent with a long rule set, a chatbot over large documentation, batch processing of similar documents under a shared instruction.
Where caching will not help
If the repeated part changes by even one character, the cache misses. A common mistake is injecting the current date or a request id into the instruction: you break the cache on every call and never notice.
Keep the variable part separate from the constant part. Constant first, variable last.
Order matters
Caching works from the start of the request. So the stable part must come first and the changing part last.
Put the variable part at the front and everything after it stops being cached too.
What it comes to in money
Work it out like this: take the size of the repeated part, multiply by requests per month and by the input rate. That is what you currently spend re-sending the same thing.
Then compare against the cached-read rate, which is markedly lower than normal input.
There is a calculator on the site to try it on your own volumes.
About cache writes
The first write costs more than a normal read. So caching is a loss if you touch it once or twice. It starts paying from the third or fourth hit onward.
Hence the rule: do not cache what is used once.
In short
Caching pays on a large stable prefix with frequent access. Stable content first, variable content last, and watch that dates and ids do not leak into the constant part.