# Introduction
TurboQuant is a novel algorithmic suite and library recently launched by Google. Its goal is to apply advanced quantization and compression to large language models (LLMs) and vector search engines — indispensable elements of retrieval-augmented generation (RAG) systems — to improve their efficiency drastically. TurboQuant has been shown to successfully reduce cache memory consumption down to just 3 bits, without requiring model retraining or sacrificing accuracy.
How does it do that, and is it really worth the hype? This article aims to answer these questions through a description and practical example of its use.
# TurboQuant in a Nutshell
While LLMs and vector search engines use high-dimensional vectors to process information with impressive results, this effort requires vast amounts of memory, potentially causing major bottlenecks in the so-called key-value (KV) cache — a quick-access “digital cheat sheet” containing frequently utilized information for real-time retrieval. Managing larger context lengths scales up KV cache access in a linear fashion, which severely limits memory capacity and computing speed.
Vector quantization (VQ) techniques used in recent years help reduce the size of text vectors to dissipate bottlenecks, but they often introduce a side “memory overhead” and require computing full-precision quantization constants on small blocks of data, thereby partly undermining the reason for compression.
TurboQuant is a set of next-generation algorithms for advanced compression with zero loss of accuracy. It optimally tackles the memory overhead issue by employing a two-stage process aided by two techniques that complement each other:
- PolarQuant: This is the compression technique applied at the first stage. It compresses high-quality data by mapping vector coordinates to a polar coordinate system. This simplifies data geometry and removes the need for storing extra quantization constants — the main cause behind memory overhead.
- QJL (Quantized Johnson-Lindenstrauss): The second stage of the compression process. It focuses on removing possible biases introduced in the previous stage, acting as a mathematical checker that applies a small, one-bit compression to remove hidden errors or residual biases resulting from applying PolarQuant.
Is TurboQuant Worth the Hype?
According to experimental results and evidence, the short answer is yes. By avoiding the expensive data normalization required in traditional quantization approaches, 3-bit TurboQuant yields an 8x performance increase over 32-bit unquantized keys on an H100 GPU-based accelerator.
# Evaluating TurboQuant
The following Python code example illustrates how developers can evaluate this locally. The program can be executed in a local IDE or a Google Colab notebook environment, providing a conceptual comparison between unquantized vectors and TurboQuant’s fast compression.
TurboQuant repositories require specific kernels to operate. To make this example work, perform the following installs first — preferably in a notebook environment, unless you have ample disk space on your local machine.
First, install TurboQuant:
In a Google Colab environment, simply install the library and make sure your runtime hardware accelerator is set to a T4 GPU — available on Colab’s free tier — so the following code executes properly.
The following code illustrates a simple comparison of performance and memory usage when using a pre-trained language model with and without TurboQuant’s KV compression. First and foremost, the imports we will need:
import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
from turboquant import TurboQuantCache
We will load a not-so-big LLM like TinyLlama/TinyLlama-1.1B-Chat-v1.0, trained for text generation, and its respective tokenizer. We specify using 16-bit decimal float precision: this option is usually more efficient in modern hardware.
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.float16)
Next, we define the scenario, simulating a large model input string, as TurboQuant truly shines as context windows become larger. Don’t worry about repeating the same content 20 times across the input: here what matters is the size being managed, not the language itself.
prompt = "Explain the history of the universe in great detail. " * 20
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
The following function is key to measure and compare execution time and memory usage across the text generation process, with TurboQuant’s 3-bit quantization being used, use_tq=True or deactivated, use_tq=False. The cache is first emptied to ensure clean measurements.
def run_unified_benchmark(use_tq=False):
torch.cuda.empty_cache()
# Initializing the specific cache type
cache = TurboQuantCache(bits=3) if use_tq else None
start_time = time.time()
with torch.no_grad():
# Running the model to generate output tokens
outputs = model.generate(**inputs, max_new_tokens=100, past_key_values=cache)
duration = time.time() - start_time
# Isolating the Cache Memory
# Instead of measuring the whole 2GB model, we measure the generated Cache size
# For a 1.1B model: [Layers: 22, Heads: 32, Head_Dim: 64]
num_tokens = outputs.shape[1]
elements = 22 * 32 * 64 * num_tokens * 2 # Key + Value
if use_tq:
mem_mb = (elements * 3) / (8 * 1024 * 1024) # 3-bit calculation
else:
mem_mb = (elements * 16) / (8 * 1024 * 1024) # 16-bit calculation
return duration, mem_mb
We finally execute the process twice — once with each of the two specified settings — and compare the results:
base_time, base_mem = run_unified_benchmark(use_tq=False)
tq_time, tq_mem = run_unified_benchmark(use_tq=True)
print(f"--- THE VERDICT ---")
print(f"Baseline (FP16) Cache: {base_mem:.2f} MB")
print(f"TurboQuant (3-bit) Cache: {tq_mem:.2f} MB")
print(f"Speedup: {base_time / tq_time:.2f}x")
print(f"Memory Saved: {base_mem - tq_mem:.2f} MB")
Results:
--- THE VERDICT ---
Baseline (FP16) Cache: 42.45 MB
TurboQuant (3-bit) Cache: 7.86 MB
Speedup: 0.61x
Memory Saved: 34.59 MB
The compression ratio is impressively up to 5.4x with regard to KV cache memory footprint. But how about the speedup? Is it as expected with TurboQuant? Not quite, but this is normal, as the sequence we used is still deemed as short for the large-scale scenarios TurboQuant is intended for, and we are running this in a local, not large-scale infrastructure. The true speed gain with TurboQuant happens as the context length and hardware accelerators used scale together. Take an enterprise-level cluster of H100 GPUs and long-form RAG prompts containing over 32K tokens: in such scenarios, memory traffic is significantly reduced, and a throughput increase of up to 8x in speed can be expected with TurboQuant.
In sum, there is a tradeoff between memory bandwith and computing latency, and you can further confirm this by trying other settings for the input and output sizes, e.g. multiplying the input string by 200 and setting max_new_tokens=250, you may get something like:
--- THE VERDICT ---
Baseline (FP16) Cache: 421.44 MB
TurboQuant (3-bit) Cache: 79.02 MB
Speedup: 0.57x
Memory Saved: 342.42 MB
Ultimately, the transformative performance of TurboQuant for AI models is proven by its ability to maintain high precision while operating at 3-bit-level system efficiency in large-scale environments.
# Wrapping Up
This article introduced TurboQuant and addressed the question of whether it is worth the hype, concerning compression and performance compared to other traditional quantization methods used in LLMs and other large-scale inference models.
Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.
