Quantisation
How models are quantised on TensorX, and how to check the quantisation for any specific model.
Overview
Every Large Language Model on the TensorX platform is served in a quantised format. There is no separate "full-precision" or "non-quantised" variant - the published deployment is already quantised to the right precision for production serving on TensorX hardware.
This page explains which quantisation levels are in use, why, and how to read the value for any specific model.
TL;DR. Almost every LLM on TensorX runs at fp8. The Kimi K2 family runs at native int4, and the deepseek-v4-flash-backup model runs at fp4. Audio models do not carry a quantisation label.
Quantisation levels in use
The TensorX catalogue uses three quantisation levels today:
fp8 (8-bit floating point)
Almost every LLM on the platform, including the GLM family (z-ai/glm-4.6, z-ai/glm-4.7, z-ai/glm-5, z-ai/glm-5-turbo, z-ai/glm-5.1, z-ai/glm-5v-turbo), DeepSeek, Llama, Qwen, Mixtral, MiniMax, and the open-source GPT-OSS models
Best precision-vs-throughput trade-off on NVIDIA B200 and B300 hardware. Quality difference versus higher precisions is negligible on standard evaluation suites.
int4 (4-bit integer)
The Kimi K2 family only (moonshotai/Kimi-K2.6, moonshotai/kimi-k2.5, moonshotai/kimi-k2.7-code)
These models ship with Moonshot's native INT4 quantisation-aware-trained weights. Using the QAT weights preserves quality at much lower memory and latency than post-training quantisation.
fp4 (4-bit floating point)
The deepseek-v4-flash-backup model only
Served at fp4 for maximum throughput on the backup deployment.
Non-LLM deployments (text-to-speech, speech-to-text) do not carry a quantisation label.
A small number of LLM deployments also report no quantisation label on /v1/model/info (the field is null).
Checking the quantisation for a specific model
The model catalogue endpoint exposes the quantisation per model. This is the canonical answer and will reflect any future change to how a model is served.
Example response (excerpt)
Each entry in the full /v1/model/info response carries a model_info.quantization field. If the field is null, the model is a non-LLM deployment (audio).
FAQ
Do you offer this model at a different quantisation level?
Not on the shared platform. TensorX does not publish the same model at multiple quantisation levels - there is one served version per model ID, at the quantisation shown on /v1/model/info.
If you have a specific requirement (for example, a particular model served at bf16 or fp16 for a regulated workload or for a benchmark methodology that requires it), that is a Dedicated Inference engagement rather than a parallel SKU on the shared platform.
Is there a quality difference between fp8 and higher precisions?
fp8 and higher precisions?For the models on the TensorX catalogue, the quality difference between the fp8 deployment and a higher-precision version of the same weights is negligible on standard evaluation suites (MMLU, HumanEval, GSM8K, AIME, and similar). Where benchmarks are run on quantised serving, we report numbers from the served fp8 version, not from a hypothetical higher-precision copy.
Why is Kimi K2 served at int4 rather than fp8?
int4 rather than fp8?The Kimi K2 family is published by Moonshot with native INT4 quantisation-aware-trained weights. Using the QAT version preserves the model's quality while reducing memory footprint and latency significantly. Quantising those weights up to fp8 would offer no quality benefit and would cost throughput.
Will the quantisation for a model ever change?
It can. If a model provider releases an updated weight format, or if a new quantisation method gives a clear quality-vs-throughput improvement, we will update the served version. The model_info.quantization field on /v1/model/info is always the source of truth.
See also
Models - Available models on the platform
Chat Completions - API endpoint documentation
Rate Limits - Throughput and request limits
Last updated
Was this helpful?