Reduce LLM Costs maintaining Quality

Large language models often receive more information than they need. A prompt may include long instructions, retrieved documents, chat history, examples, and tool descriptions. This increases token usage, cost, and response time. It can also make important details harder for the model to identify.

Prompt compression reduces the prompt while keeping the key meaning, instructions, and evidence. It helps the model focus on useful information and avoid unnecessary context. This is valuable in RAG systems, AI agents, customer support, document analysis, and long conversations. In this article, we will explore the main prompt compression techniques, how they work, and how to use them without reducing answer quality.

What is Prompt Compression?

Prompt compression is the process of shortening a prompt without removing the information needed to complete the task. It removes repeated, irrelevant, or low-value content while keeping the main instructions, important facts, and useful context.

The goal is not to create the shortest possible prompt. The goal is to reduce tokens without reducing answer quality. A well-compressed prompt should still help the model understand the task, follow the instructions, and produce an accurate response.

Types of Prompt Compression Techniques

Prompt compression can be done in several ways. Some methods remove complete sentences or documents. Others shorten phrases, remove low-value tokens, or rewrite the full context as a summary.

The main techniques include:

1. Manual Prompt Rewriting

Manual prompt rewriting is the simplest compression technique. It removes repeated instructions, filler words, and unnecessary explanations.

For example:

You should carefully read the information given below and provide an answer based only on the available context.

Compressed version:

Answer using only the provided context.

This method works well for system prompts, reusable templates, evaluation prompts, and tool instructions. It is easy to apply and keeps the prompt readable. However, it requires manual effort and may not scale well for large or changing inputs.

2. Structural Compression

Structural compression changes long text into a compact format. It uses bullet points, tables, key-value pairs, JSON, or YAML.

For example:

The customer is on the Gold plan. They joined 18 months ago. Their monthly spending is $240. They contacted support five times last month.

Compressed version:

plan: Gold
tenure_months: 18
monthly_spend: 240
support_contacts_30d: 5

This method works well for customer records, claims, product details, and tool outputs. It reduces extra words while keeping important facts clear. However, field names should remain easy to understand.

3. Sentence-Level Filtering

Sentence-level filtering removes complete sentences that are not relevant to the task. It keeps only the sentences that help answer the user’s question.

For example, a retrieved document may contain ten sentences, but only three may be useful. The system can rank each sentence and keep the most relevant ones.

This technique is useful in RAG systems, reports, policies, and long conversations. It keeps the compressed text readable. However, removing too many sentences may also remove important supporting context.

4. Phrase-Level Compression

Phrase-level compression removes unnecessary words from a sentence while keeping its main meaning.

For example:

Due to the fact that the application was submitted after the deadline, it cannot be processed.

Compressed version:

The application was submitted after the deadline and cannot be processed.

This method removes filler words, repeated phrases, and long expressions. It keeps the text readable and reduces token usage. However, careless compression may remove important details or change the meaning of the sentence.

5. Token-Level Filtering

Token-level filtering removes individual words or tokens that carry little value. It keeps the terms that are most important for understanding the task.

For example:

The customer has a Gold membership and has contacted support five times during the last month.

Compressed version:

Customer Gold membership, five support contacts last month.

This method can reduce prompts more aggressively than sentence-level filtering. It is useful when working with very long contexts. However, it may make the text less readable. Important words such as “not,” “unless,” or “never” must be protected because removing them can completely change the meaning.

Extractive compression selects the most important parts of the original prompt. It keeps useful sentences, passages, examples, or facts without rewriting them.

For example, a long policy document may contain several sections. The system can extract only the clauses related to the user’s question.

This method is useful for RAG systems, document analysis, and policy-based applications. It reduces the risk of adding incorrect information because the selected text comes directly from the source. However, it may miss important details when the required information is spread across multiple sections.

7. Abstractive Compression

Abstractive compression rewrites long content into a shorter summary. It does not copy the original text directly. Instead, it keeps the main meaning in fewer words.

For example, a long conversation can be compressed into a short summary of the user’s problem, actions already taken, and the next required step.

This method is useful for chat history, reports, meeting notes, and long documents. It produces clear and readable text. However, the summarizing model may remove important details or add information that was not present in the original content.

8. Query-Aware Compression

Query-aware compression keeps information based on the user’s current question. It removes content that may be useful in general but is not relevant to the specific task.

For example, if a user asks about a product’s refund policy, the system keeps refund conditions, deadlines, and exceptions. It removes unrelated details about installation or maintenance.

This technique is highly useful in RAG systems and document question answering. It helps the model focus on the most relevant evidence. However, its quality depends on how accurately the system understands the user’s query.

9. Coarse-to-Fine Compression

Coarse-to-fine compression reduces a prompt in multiple stages. It first removes large irrelevant sections, such as documents or paragraphs. It then compresses the remaining content at the sentence, phrase, or token level.

For example, a RAG system may first remove unrelated documents. It can then select useful paragraphs and shorten them further.

This method gives better control over compression. It is also safer than removing tokens from the full prompt at once. However, it requires multiple processing steps and may increase system complexity.

10. Soft Prompt Compression

Soft prompt compression converts long text into a small set of learned vectors. These vectors represent the important information without using normal readable words.

This method can reduce context size significantly. It is useful in custom models and research systems where developers can access model embeddings or internal representations.

However, soft prompts are difficult to inspect and debug. They may also require model training and may not work easily with closed API models that only accept text input.

Prompt Compression in RAG Systems

Prompt compression is highly useful in Retrieval-Augmented Generation, or RAG. A RAG system retrieves documents and adds them to the prompt before generating an answer.

The retrieved content may contain repeated, outdated, or irrelevant information. Sending all of it increases token usage and can distract the model.

A compressed RAG pipeline works like this:

Compression can remove unrelated documents, select important paragraphs, and retain only the sentences that support the answer. This helps reduce cost and keeps the response focused. Source details should still be preserved when citations are required.

Prompt Compression for AI Agents

AI agents often build long prompts over time. They may include system instructions, tool descriptions, previous actions, tool outputs, errors, and conversation history.

Prompt compression helps agents keep only the information needed for the next step. Old tool outputs can be summarized, completed actions can be stored as short state updates, and repeated instructions can be removed.

For example:

Completed:
- Customer record retrieved
- Eligibility verified
- Email selected as the best channel
Pending:
- Generate the final message

This approach reduces token usage and prevents the agent’s context from growing continuously. However, important rules, safety instructions, decisions, and unresolved errors should never be removed.

How to Measure Prompt Compression

Prompt compression should not be measured by token reduction alone. A shorter prompt is useful only when the model still produces an accurate and complete response.

The main metrics include:

• Token reduction
• Compression factor
• Task accuracy
• Information preservation
• Hallucination rate
• Response time
• Total cost

Token reduction shows how many tokens were removed.

Token reduction = 1 - (compressed tokens / original tokens)

For example, reducing a prompt from 1,000 tokens to 400 tokens gives a 60% token reduction. Compression factor shows how many times smaller the new prompt is.

Compression factor = original tokens / compressed tokens

A prompt reduced from 1,000 tokens to 250 tokens has a compression factor of 4x.The compressed prompt should also preserve names, numbers, dates, instructions, conditions, and negative words. It should be tested using the same task and model as the original prompt.

Hands-On: Simple Prompt Compression in Python

The following example removes less relevant sentences from a long context. It keeps sentences that share important words with the user’s query.

import re

def split_sentences(text):
    return re.split(r"(?<=[.!?])\s+", text.strip())


def compress_prompt(context, query, keep_count=3):
    query_words = set(re.findall(r"\w+", query.lower()))
    sentences = split_sentences(context)

    scored_sentences = []

    for index, sentence in enumerate(sentences):
        sentence_words = set(re.findall(r"\w+", sentence.lower()))
        score = len(query_words.intersection(sentence_words))

        scored_sentences.append(
            {
                "index": index,
                "sentence": sentence,
                "score": score,
            }
        )

    selected = sorted(
        scored_sentences,
        key=lambda item: item["score"],
        reverse=True,
    )[:keep_count]

    selected = sorted(selected, key=lambda item: item["index"])

    return " ".join(item["sentence"] for item in selected)


context = """
The premium plan includes advanced analytics.
Customers can cancel their subscription at any time.
Refund requests must be submitted within 14 days.
The annual plan costs $240.
Refunds are not available after the 14-day period.
The mobile application was updated last month.
"""

query = "Can I get a refund after 20 days?"

compressed_prompt = compress_prompt(context, query)

print(compressed_prompt)

Output:

This method is simple and easy to understand. It works well for basic demonstrations. Production systems should use embeddings, rerankers, or trained compression models for better relevance detection.

Practical Prompt Compression Workflow

A good compression workflow should reduce tokens without removing important information.

Step 1: Identify protected content

Mark the information that must remain unchanged.

This may include:

• System instructions
• User questions
• Output format
• Names and IDs
• Dates and numbers
• Negative words
• Safety rules
• Important evidence

Step 2: Remove repeated content

Delete duplicate instructions, repeated documents, and unnecessary explanations.

Step 3: Filter irrelevant sections

Remove documents, paragraphs, or examples that do not support the current task.

Step 4: Compress the remaining text

Apply sentence-level, phrase-level, or token-level compression.

Step 5: Compare the results

Run the task using both the original and compressed prompts. Check whether the answers remain accurate and complete.

Step 6: Test different compression levels

Start with light compression. Increase it slowly while measuring quality, cost, and response time.

Step 7: Add a fallback

Use the original prompt when compression removes important information or produces low-confidence results.

Advantages and Limitations of Prompt Compression

Prompt compression offers several benefits along with some limitations:

Prompt compression works best when the cost savings are greater than the risk of losing useful context.

Conclusion

Prompt compression helps reduce token usage, API cost, and response time by removing unnecessary content from prompts. It is especially useful in RAG systems, AI agents, long conversations, and document-based applications.

Different techniques offer different levels of compression. Manual rewriting and structural compression are simple and readable. Sentence filtering, token filtering, and learned methods provide stronger compression but require more testing.

The goal should not be to create the shortest prompt. The goal should be to keep the prompt accurate, clear, and useful. Always protect important instructions, facts, conditions, and evidence. Test compressed prompts against the original before using them in production.

Frequently Asked Questions

Hi, I am Janvi, a passionate data science enthusiast currently working at Analytics Vidhya. My journey into the world of data began with a deep curiosity about how we can extract meaningful insights from complex datasets.