RAG Query Rewriting Prompts for Better Retrieval

A retrieval-augmented generation (RAG) system can produce a polished answer while still retrieving the wrong evidence. A user may ask, “Can I cancel after the trial?” while your help center only uses “subscription termination” and “billing period.”

RAG query rewriting closes that language gap before retrieval starts. With disciplined prompts, you can turn vague, conversational, or domain-specific requests into search queries that match your indexed knowledge without changing the user’s meaning.

The goal isn’t to make every query longer. It’s to retrieve the smallest useful set of evidence for a grounded answer.

Key Takeaways

• RAG query rewriting bridges the gap between conversational user language and the terminology used in indexed knowledge, while preserving the user’s intent, entities, constraints, and requested action.
• Use rewriting for unclear wording, query expansion for vocabulary mismatch, and decomposition when a request contains multiple claims that require separate evidence.
• Pair bounded rewrites with hybrid search: lexical search handles exact terms and identifiers, vector search captures semantic similarity, and semantic ranking restores precision.
• Evaluate every prompt and model change using retrieval, faithfulness, grounded-answer, and operational metrics to detect semantic drift, increased cost, and weaker evidence.
• Production systems should limit rewrite budgets, preserve the original query, control approved vocabulary, protect sensitive data, and log transformations for auditing and regression testing.

Why query rewriting improves retrieval accuracy

A RAG pipeline depends on a simple chain: query, retrieved evidence, answer. If the first link is weak, even a strong language model will receive incomplete context.

Users rarely phrase questions like your documentation. They use synonyms, omit product names, refer to prior messages, make spelling mistakes, and mix several requests together. Vector search can bridge some semantic gaps, while lexical search can catch exact terminology. Neither vector search nor lexical search consistently resolves every ambiguous request alone.

Query transformation converts the original request into one or more retrieval-ready forms. A good rewrite preserves user intent while adding context already present in the conversation or permitted by your domain rules.

For example, a customer asks:

“Why is it doing the thing it did yesterday?”

A weak retrieval query repeats that sentence. A useful rewrite might be: “troubleshooting repeated payment failure after a successful payment on the previous day,” but only if earlier chat messages establish that “it” means a payment.

That distinction matters. A rewrite model should clarify references using known context, not fill missing facts with guesses. Meilisearch’s guide to RAG query rewriting describes the technique as a way to bridge the gap between user phrasing and the wording present in your knowledge base.

Query rewriting, expansion, and decomposition solve different problems

These techniques often appear together, yet they make different retrieval decisions.

Technique	What it changes	Best use case	Main risk
Query rewriting	Rephrases one request into clearer search language	Conversational, ambiguous, or poorly phrased questions	Semantic drift
Query expansion	Adds related terms or alternate phrasings	Sparse indexes and vocabulary mismatch	Lower precision
Query decomposition	Splits a compound question into smaller searches	Multi-part questions requiring separate evidence	Fragmented answers
step-back prompting	Produces a broader conceptual question	Questions that depend on policy or first principles	Retrieves material that is too general
HyDE	Generates a hypothetical answer or document for vector search	Semantic retrieval with brief or underspecified queries	Hallucinated details bias retrieval

Use rewriting when the original request has one intent but poor wording. Use query expansion when terminology varies across documents. Use query decomposition only when each part needs separate evidence.

For example, “Does the enterprise plan support SSO, and can contractors access audit logs?” needs two searches. By contrast, “Can outside people see audit history?” probably needs one rewrite plus controlled synonym expansion for “contractors,” “external users,” and “audit logs.”

A rewrite can increase recall while lowering precision. Treat every added term as a retrieval hypothesis that must earn its place in evaluation.

Copy-ready retrieval prompts

A prompt template works when it tells the model what it may change, what it must preserve, and what structured output your retrieval layer expects. These templates provide practical query rewriting strategies, and the rewrite task stays separate from answer generation. The rewriter should not answer the user, cite sources, or invent product behavior.

Canonical rewrite prompt for clear but informal requests

Use this template when the request has a single intent but includes casual wording, typos, or phrasing that differs from your documentation.

You rewrite user requests for document retrieval. Preserve the request’s intent, entities, restrictions, dates, and requested action. Replace informal wording with concise domain-neutral search terms. Do not answer the question. Do not add facts, assumptions, product features, or entities not stated in the conversation. Return JSON with: rewritten_query, key_entities, and uncertainties.

Conversation context: {conversation_context}
User request: {user_query}

If the user asks, “Can I move my team stuff to a different workspace without losing anything?”, the output might be:

rewritten_query: “transfer team data to another workspace while retaining existing content and permissions”
key_entities: [“team data”, “workspace”, “content”, “permissions”]
uncertainties: [“The type of data to transfer is not specified.”]

In some applications, small language models may handle tightly constrained JSON rewriting when evaluation shows they are reliable.

That uncertainty field is useful. Your application can ask a follow-up question when the uncertainty blocks safe retrieval, or it can run a broader query without pretending it knows the answer.

Ambiguous and conversational query prompt

Pronouns and chat shorthand create retrieval failures because the important noun may appear five messages earlier. Use the next template when your application retains conversation state.

Convert the latest user message into a standalone retrieval query. Resolve pronouns and shorthand only with facts stated in the conversation context. If a reference has more than one plausible meaning, keep the ambiguity explicit instead of choosing one. Preserve exact names, model numbers, locations, dates, and policy terms. Return one query and a confidence score from 0 to 1.

Conversation context: {conversation_context}
Latest message: {user_query}

Suppose the previous exchange concerns a Shopify store’s abandoned-cart emails. The user then asks, “Can I change that without starting over?” A safe query becomes: “change an existing Shopify abandoned-cart email automation without recreating the automation.”

Without the prior context, the model should return a low-confidence query such as: “change the previously discussed configuration without recreating it.” Your router can then ask what “that” refers to. Low-confidence rewrites should not silently launch a broad search across unrelated collections.

Multi-turn and multi-part query decomposition prompt

Use decomposition when a single answer requires evidence from separate documentation areas. It prevents one broad query from returning documents that only address the easiest clause.

Break the user request into the minimum number of independent retrieval queries. Each query must seek evidence for one answerable claim. Keep shared constraints in every relevant query. Do not create sub-questions that require unstated assumptions. Return JSON with subqueries, dependencies, and combined_answer_requirements.

Conversation context: {conversation_context}
User request: {user_query}

For “Can a user on our Pro plan export data, and does that export include deleted records?” you might retrieve:

1. “Pro plan data export availability and permissions”
2. “data export contents, including treatment of deleted records”

Your final answer should only combine these results after checking that both documents apply to the same product version and account tier. Decomposition raises recall, but it also raises latency and can create false confidence when one subquery has no evidence.

Domain-specific rewrite prompt with controlled vocabulary

Enterprise systems often have legal, medical, financial, security, or internal product language. Generic rewrites may replace exact terms with everyday synonyms, which can harm matching or change meaning.

Use an approved glossary and force the model to select terms from it. This controlled query expansion supports lexical search, while lexical keyword enrichment stays limited to glossary-approved additions rather than unrestricted synonym generation.

Rewrite the request for retrieval in the {domain_name} knowledge base. Preserve all regulated, legal, technical, and product terms exactly. You may add only approved synonyms from the glossary. If the request needs a missing identifier, jurisdiction, date, or product version, list it as a clarification instead of inferring it. Return JSON with query, approved_expansions, and required_clarifications.

Approved glossary: {glossary}
User request: {user_query}

For a security knowledge base, “Can vendors get into the audit area?” might become “third-party vendor access to audit logs, role-based access control, and access review policy.” The prompt must not assume the vendor is an employee, administrator, or processor.

This approach is also useful when you maintain a prompt repository. Track prompt, glossary, and test-set changes together through one evaluation pipeline. A prompt library download may be convenient for creators, but production prompt files need owners, release notes, and tests.

Step-back prompting and HyDE prompts for difficult retrieval

A broader prompt helps when a narrow question depends on a broader rule. For example, a user might ask whether a specific employee can approve an expense. The key retrieval target may be the organization’s approval policy rather than an employee record.

Create one broader retrieval query that identifies the governing principle behind the user’s request. Keep it within the same domain. Do not answer the user or introduce facts.

User request: {user_query}

HyDE, short for hypothetical document embeddings, takes a different route. It uses pseudo-answer generation to draft a plausible passage, embeds that passage, and performs vector search against it. It can help when queries are short and documents use descriptive prose.

Write a short, generic passage that a relevant knowledge-base document might contain in response to the request. Do not state facts as true. Do not add names, dates, numbers, or policies not present in the request. Use neutral domain terminology.

User request: {user_query}

Keep the original query in the retrieval set when using HyDE. This pseudo-answer generation can pull retrieval toward plausible but nonexistent details. The generated passage is a retrieval aid, not evidence.

Pair rewrites with hybrid search and semantic ranking

A rewrite improves the query. It doesn’t replace a well-built retrieval stack.

Hybrid search combines lexical search with vector search. Lexical search catches exact names, error codes, policy titles, and identifiers. Vector retrieval catches conceptually similar content even when wording differs. Reciprocal rank fusion can merge the result lists before ranking.

Microsoft positions query rewriting as a pre-retrieval feature in Azure AI Search. It can generate up to 10 alternate queries, then send the original and rewritten forms into L1 retrieval. The Azure AI Search semantic overview explains how its semantic ranker then rescores the initial result set, adding an L2 relevance layer.

Microsoft reports that query rewriting can add 4 NDCG@3 points for low-recall, term-based search cases. Its newer semantic ranking capability, combined with query rewriting, achieved up to 22 additional NDCG@3 points in testing across more than 90 datasets and 19 languages. Those are product-specific results, not a guaranteed result for your index.

This multistage retrieval flow shows why hybrid search works. A framework such as LangChain can coordinate the original query, rewrites, fusion, and ranking stages:

1. Generate a small number of bounded rewrites.
2. Run the original and rewritten queries through lexical search and vector search.
3. Fuse and deduplicate the candidate documents.
4. Apply a semantic ranker or cross-encoder during reranking to the top candidates.
5. Pass only the highest-scoring, relevant chunks to large language models for answer generation.

Your implementation details differ, but the principle stays the same. Use rewrites to widen candidate recall, then let a stronger semantic ranker restore precision.

Elasticsearch supports similar patterns through its query DSL, analyzers, synonym handling, lexical search, vector fields, and reranking options. An Elasticsearch implementation can vary by analyzer, synonym configuration, and vector configuration.

Avoid generating ten rewrites for every request by default. A direct query for a unique policy number or error code often needs neither rewriting nor a semantic ranker. Route only uncertain, conversational, or multi-part queries through the rewrite model.

Measure retrieval accuracy before and after every prompt change

A prompt that reads well can still hurt retrieval. Build an evaluation pipeline with an evaluation set that covers real user language, known relevant documents, difficult paraphrases, ambiguous requests, and cases where asking for clarification is correct. Compare it with a lexical search baseline, and record each query transformation alongside retrieved documents and the final answer.

ZenML’s evaluation guidance for query rewriting emphasizes a practical problem: polished rewrites can introduce semantic drift while moving away from user intent. Track the rewrite itself, retrieved documents, and answer together.

Measure at least five dimensions:

• Recall@k is a recall measure that shows whether relevant evidence appears among the first k retrieved results.
• NDCG@k rewards systems that rank the most useful documents higher. It supports reranking comparisons with a semantic ranker.
• Rewrite faithfulness checks whether the rewrite retained the original request’s entities, constraints, and intent.
• Grounded answer quality checks whether the final response is supported by retrieved passages.
• Operational cost includes rewrite latency, retrieval time, token spend, and the number of candidate documents.

Log the original query, each rewrite, retrieval scores, document IDs, semantic ranker scores, final citations, latency, and prompt version. Then use the evaluation pipeline to compare a control pipeline built on a lexical search baseline with the rewrite pipeline on the same test set.

A ranking-feedback approach can also improve the rewriter over time. The RaFe research paper describes using reranker feedback to train query rewriting models. Even without training your own model, you can use reranker and human-review signals to identify rewrite patterns that consistently retrieve weak evidence.

Run regression tests through the evaluation pipeline whenever you change the rewrite prompt, model, embedding model, chunking policy, glossary, lexical search settings, Elasticsearch configuration, or semantic ranker. Different query rewriting strategies may perform differently across support, legal-policy, and product corpora. A prompt update that helps support articles may damage legal-policy retrieval because its synonym choices become too broad.

Control cost, latency, and semantic drift in production

Generative rewriting adds a model call before retrieval. It can increase rewrite latency, retrieval time, token latency, and token spend. Multi-query generation can multiply searches, embeddings, fusion work, and reranking load. Treat the system as a multistage retrieval pipeline, and set a rewrite budget per request. Allow one canonical rewrite and no more than two expansions for normal traffic.

Small language models can handle bounded, schema-constrained rewrites when tests support that choice. Reserve large language models for harder ambiguity or context-resolution cases.

Cache rewrites for repeated normalized queries, but include the knowledge-base version and relevant conversation state in the cache key. Otherwise, an old rewrite may persist after a terminology change or policy update.

You should also block unsafe transformations. Don’t expand customer names, account numbers, or personal data beyond what the user supplied. For account numbers, names, and policy identifiers, use an exact-match lexical search path when possible. In regulated domains, use deterministic mappings for approved abbreviations and preserve the original query for audit review.

Keep rewrite logs, prompt versions, and approval rules tied to each release. Review failures for privacy violations, unsupported additions, and missed context before raising the budget.

Frequently Asked Questions

What is RAG query rewriting?

RAG query rewriting transforms a user’s original request into a clearer retrieval query before document search begins. It can resolve conversational wording, add permitted context, and align the query with terminology in the knowledge base without answering the user.

When should I use query rewriting instead of query expansion or decomposition?

Use rewriting when one request has a single intent but poor or ambiguous wording. Use expansion for vocabulary differences and decomposition when separate parts of a compound question require independent evidence.

Does query rewriting replace hybrid search?

No. Rewriting improves the search input, while hybrid search combines lexical matching for exact terms with vector retrieval for semantic similarity. A semantic ranker can then rerank the combined candidates to improve precision.

How can I prevent rewrites from changing the user’s meaning?

Constrain the prompt to preserve entities, dates, restrictions, and requested actions, and prohibit invented facts or unsupported synonyms. Track uncertainties, retain the original query, and use approved glossaries or deterministic mappings in sensitive domains.

How should I evaluate a query rewriting prompt?

Compare the rewrite pipeline with a lexical-search baseline on the same evaluation set, measuring Recall@k, NDCG@k, rewrite faithfulness, grounded answer quality, and operational cost. Log prompt versions, rewrites, retrieved documents, ranking scores, citations, and latency so regressions and semantic drift are visible.

Build a retrieval system that earns trust

RAG query rewriting works when it makes the user’s meaning easier to find while preserving user intent. Start with a constrained canonical prompt, then apply query rewriting strategies only where evaluation data shows a clear need.

The strongest pipeline is a multistage retrieval architecture that combines careful rewrites with hybrid search, using lexical search for exact matches, vector search for semantic matches, and a semantic ranker for reranking. Together, these stages improve retrieval accuracy without sacrificing grounded answers, while an evaluation pipeline keeps every transformation observable and measurable.