Glossary
Quantisation
Reducing the numeric precision of model weights to cut memory and increase speed.
Running weights at 8-bit or 4-bit instead of 16-bit dramatically reduces memory, often making a model fit hardware it otherwise could not.
Quality loss is usually small but is never zero. It should be measured on your task rather than assumed negligible.
Quality loss is also uneven. Quantisation tends to degrade long-form reasoning and rare-token handling more than short classification, so a benchmark on your easiest task will mislead you. Measure on the hardest thing you intend to ship.
The practical workflow is to quantise progressively and stop at the first level that fails your evaluation set. Going straight to the most aggressive setting because it fits the hardware you have is how teams end up shipping a model that is measurably worse without ever noticing.
Quantise the deployment, not the evaluation. Benchmarking a full-precision model and then serving a quantised one is a common way to ship something measurably worse than what was signed off.
Related terms, in context
The concepts you almost always meet alongside quantisation.
- Open-weight models
- Models whose parameters you can download and run yourself, on your own infrastructure.
- LoRA
- A fine-tuning method that trains a small set of extra parameters instead of the whole model.
- Edge inference
- Running models on local hardware near the data, instead of calling a cloud API.
Where this shows up in our work
Quantisation is not an abstraction for us. It is a decision we make on live projects. It shows up most directly in ai infrastructure & mlops, custom model fine-tuning, where getting it wrong has a cost someone can measure.
If you are evaluating a vendor on this, the useful question is not whether they can define the term. It is what they measure, what they would refuse to do, and what happens in their system when the assumption behind quantisation stops holding.
Questions
What is Quantisation?
Reducing the numeric precision of model weights to cut memory and increase speed.
Does Orqent Labs build this?
Yes, AI Infrastructure & MLOps and Custom Model Fine-tuning. We work across India, covering all 19,238 PIN codes remotely.
Building something that involves quantisation?
We will tell you honestly whether it is the right approach for your problem.
Or email bd@dtrasglobal.com · call +91 74118 77878