Prompt Caching

How prompt caching works on TensorX.

Does TensorX offer prompt caching?

Yes. Our serving stack uses KV caching at the inference layer, which is how modern inference engines work in general. When repeated requests share a common prefix, or land on the same replica that has handled a similar prefix recently, reuse of cached state makes those requests faster.

This is best-effort rather than guaranteed. We run multiple replicas of popular models for load and resilience, so a follow-up request can land on a different replica without that prefix warm, in which case it is recomputed from scratch.

Cache-hit pricing

Models that support prompt caching have separate, lower rates for cached input tokens. When a request hits a warm replica with a matching prefix, the cached portion of the input is billed at the model's cache-read rate rather than the standard input rate. Across the catalogue the cache-read rate is roughly a quarter of the input rate (about a 75% discount); the exact per-model figure is available from the API shown below. Populating the cache on the first hit uses the cache-write rate. You don't need to do anything to opt in; pricing is applied automatically based on what the serving layer reports back to the billing layer.

Because cache hits are best-effort across replicas, the saving applies on a per-request basis rather than as a guaranteed discount on every call. In steady traffic with consistent prefixes you can expect the cache-read rate to apply most of the time; under heavy load or after idle periods, more of your requests will be billed at the standard input rate.

We will surface the per-model cache rates on the public pricing page shortly. In the meantime, you can retrieve them yourself today directly from the API:

curl -H "Authorization: Bearer $TENSORX_API_KEY" \
  https://api.tensorx.ai/v1/model/info \
  | jq '.data[] | select(.model_info.supports_prompt_caching == true) | {
      model: .model_name,
      input_cost_per_token: .model_info.input_cost_per_token,
      cache_read_input_token_cost: .model_info.cache_read_input_token_cost,
      cache_creation_input_token_cost: .model_info.cache_creation_input_token_cost
    }'

The supports_prompt_caching flag tells you whether a given model has caching enabled. Audio models and a small number of LLMs don't expose caching today.

Measuring cache usage

Because hits are best-effort, you can confirm what actually got cached from the usage object the API returns on each response. On chat completions, prompt_tokens_details.cached_tokens reports how many of the prompt tokens were served from cache; comparing that against prompt_tokens tells you the share of the input that hit the cache (and so was billed at the cheaper cache-read rate). Aggregate these fields across calls to see how often steady, shared-prefix traffic is hitting the cache.

What this means in practice

• Repeated identical or near-identical prompts, especially when sent in close succession, are likely to benefit from KV cache reuse on a warm replica.
• Long shared system prompts across many calls (a common pattern for agentic workflows) will often hit the cache when traffic is steady, and will miss when load spreads requests across replicas.
• Cold starts (the first request to a freshly scaled replica, or after a long idle period) will not benefit from caching.

Caching and zero data retention

Prompt caching reuses a matching prompt prefix when a recent request has warmed it, on a best-effort basis across the serving fleet, and is unrelated to our zero-data-retention policy. ZDR governs what we do not retain after a request completes (prompts, completions, logs of either).

KV state lives in GPU memory only and is never written to disk or included in any retained dataset. Prompt caching works by reusing this state, so cached blocks do persist in memory beyond the single request that created them: a later request that shares a prefix can reuse them. They remain transient in-memory state, evicted under memory pressure or after idle periods, and nothing about the prompt or completion is persisted to storage at any point.