> For the complete documentation index, see [llms.txt](https://docs.tensorx.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tensorx.ai/api-reference/prompt-caching.md).

# Prompt Caching

How prompt caching works on TensorX.

## Does TensorX offer prompt caching?

Yes. TensorX uses KV caching at the inference layer. Requests with a matching prefix can reuse cached state when they reach a replica where that prefix is available.

This is **best-effort rather than guaranteed**. We run multiple replicas of popular models for load and resilience, so a follow-up request can land on a different replica without that prefix warm, in which case it is recomputed from scratch.

## Cache-hit pricing

Models that support prompt caching have separate rates for cached input tokens. When a request hits a warm replica with a matching prefix, the cached portion is billed at the model's **cache-read** rate. Populating the cache uses the **cache-write** rate. Pricing is applied automatically from the usage reported by the serving layer.

Cache hits are best-effort across replicas, so cache-read billing is determined per request and is not guaranteed for repeated input.

The public [model catalogue](https://tensorx.ai/models) displays each model's cache-read price where caching is available. Retrieve the same pricing and capability metadata programmatically from the API:

```bash
curl -H "Authorization: Bearer $TENSORX_API_KEY" \
  https://api.tensorx.ai/v1/model/info \
  | jq '.data[] | select(.model_info.supports_prompt_caching == true) | {
      model: .model_name,
      input_cost_per_token: .model_info.input_cost_per_token,
      cache_read_input_token_cost: .model_info.cache_read_input_token_cost,
      cache_creation_input_token_cost: .model_info.cache_creation_input_token_cost
    }'
```

The `supports_prompt_caching` flag identifies models that expose prompt caching.

## Measuring cache usage

Because hits are best-effort, you can confirm what actually got cached from the `usage` object the API returns on each response. On chat completions, `prompt_tokens_details.cached_tokens` reports how many of the prompt tokens were served from cache; comparing that against `prompt_tokens` gives the cached share of the input. Aggregate these fields across calls to see how often steady, shared-prefix traffic is hitting the cache.

## What this means in practice

* **Repeated identical or near-identical prompts**, especially when sent in close succession, are likely to benefit from KV cache reuse on a warm replica.
* **Long shared system prompts** across many calls (a common pattern for agentic workflows) will often hit the cache when traffic is steady, and will miss when load spreads requests across replicas.
* **Cold starts** (the first request to a freshly scaled replica, or after a long idle period) will not benefit from caching.

## Caching and zero data retention

Prompt caching reuses a matching prompt prefix when a recent request has warmed it, on a best-effort basis across the serving fleet, and is unrelated to our zero-data-retention policy. ZDR governs what we do not retain after a request completes (prompts, completions, logs of either).

KV state lives in GPU memory only and is never written to disk or included in any retained dataset. Prompt caching works by reusing this state, so cached blocks do persist in memory beyond the single request that created them: a later request that shares a prefix can reuse them. They remain transient in-memory state, evicted under memory pressure or after idle periods, and nothing about the prompt or completion is persisted to storage at any point.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.tensorx.ai/api-reference/prompt-caching.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.