Prompt Caching

How prompt caching works on TensorX.

Does TensorX offer prompt caching?

Yes. TensorX uses KV caching at the inference layer. Requests with a matching prefix can reuse cached state when they reach a replica where that prefix is available.

This is best-effort rather than guaranteed. We run multiple replicas of popular models for load and resilience, so a follow-up request can land on a different replica without that prefix warm, in which case it is recomputed from scratch.

Cache-hit pricing

Models that support prompt caching have separate rates for cached input tokens. When a request hits a warm replica with a matching prefix, the cached portion is billed at the model's cache-read rate. Populating the cache uses the cache-write rate. Pricing is applied automatically from the usage reported by the serving layer.

Cache hits are best-effort across replicas, so cache-read billing is determined per request and is not guaranteed for repeated input.

The public model catalogue displays each model's cache-read price where caching is available. Retrieve the same pricing and capability metadata programmatically from the API:

curl -H "Authorization: Bearer $TENSORX_API_KEY" \
  https://api.tensorx.ai/v1/model/info \
  | jq '.data[] | select(.model_info.supports_prompt_caching == true) | {
      model: .model_name,
      input_cost_per_token: .model_info.input_cost_per_token,
      cache_read_input_token_cost: .model_info.cache_read_input_token_cost,
      cache_creation_input_token_cost: .model_info.cache_creation_input_token_cost
    }'

The supports_prompt_caching flag identifies models that expose prompt caching.

Measuring cache usage

Because hits are best-effort, you can confirm what actually got cached from the usage object the API returns on each response. On chat completions, prompt_tokens_details.cached_tokens reports how many of the prompt tokens were served from cache; comparing that against prompt_tokens gives the cached share of the input. Aggregate these fields across calls to see how often steady, shared-prefix traffic is hitting the cache.

What this means in practice

• Repeated identical or near-identical prompts, especially when sent in close succession, are likely to benefit from KV cache reuse on a warm replica.

• Long shared system prompts across many calls (a common pattern for agentic workflows) will often hit the cache when traffic is steady, and will miss when load spreads requests across replicas.

• Cold starts (the first request to a freshly scaled replica, or after a long idle period) will not benefit from caching.

Caching and zero data retention

Prompt caching reuses a matching prompt prefix when a recent request has warmed it, on a best-effort basis across the serving fleet, and is unrelated to our zero-data-retention policy. ZDR governs what we do not retain after a request completes (prompts, completions, logs of either).

KV state lives in GPU memory only and is never written to disk or included in any retained dataset. Prompt caching works by reusing this state, so cached blocks do persist in memory beyond the single request that created them: a later request that shares a prefix can reuse them. They remain transient in-memory state, evicted under memory pressure or after idle periods, and nothing about the prompt or completion is persisted to storage at any point.