Query Decomposition Prompts: A Multi-Hop RAG Playbook

Your retrieval-augmented generation (RAG) system can find relevant documents and still miss part of a compound question. Query decomposition prompts split complex requests into focused searches, but they don’t guarantee a complete answer.

In multi-hop RAG, dependency tracking, balanced context allocation, and evidence checks are also essential before delivery. Start by separating questions you can search independently from those that depend on an earlier result.

Key Takeaways

• Decompose compound queries only when they contain distinct evidence needs or genuine dependencies; use direct retrieval for simple lookups.
• Use structured, validated plans that preserve the user’s constraints, distinguish independent from dependent searches, and leave unresolved values for evidence to supply.
• Track evidence through retrieval and context packing: retrieved passages must still be present in the context provided to generation, with coverage across every required intent.
• Verify each factual claim against identifiable passages, evaluate each pipeline stage separately, and retain extra retrieval hops only when they improve evidence coverage and answer quality enough to justify their cost.

Why one search can miss a compound question

Single-step retrieval usually searches for passages related to the entire request. Query understanding means identifying the search intent and whether evidence needs are independent or dependent. Multi-intent queries can pull results toward their most prominent topic.

Semantic embeddings can retrieve passages from a vector database, where cosine similarity ranks candidates. But semantic similarity doesn’t prove every requested condition is covered.

As LlamaIndex’s introduction to RAG describes, indexing prepares your data, and retrieval selects relevant context. Decomposition changes what you ask that retrieval stage to find.

Consider: “Which university did the author of The Hobbit attend?” You can first retrieve the author’s identity, then use that verified identity to search for university attendance.

That sequence differs from comparing two products’ documented features. You can search those products independently, then combine the evidence.

Decomposition is useful when your original query hides these separate retrieval targets. Yet a simple factual lookup usually needs neither a planning model call nor several searches.

Write query decomposition prompts that preserve intent

Your planner should produce an executable retrieval plan, not an answer or a long reasoning narrative. Require structured output so your application can validate it.

Use independent sub-questions for comparisons

Choose this approach when every search can run without another search’s answer. Preserve shared constraints, including dates, plan names, geography, and requested exclusions.

Copy-ready parallel decomposition prompt:

Decompose the user question into the smallest set of independent retrieval questions. Preserve every named entity, timeframe, comparison criterion, and restriction. Don’t answer the question or add requirements. Return JSON containing original_question, mode, and sub_questions. Each sub-question must contain id, question, required_evidence, and source_constraints. Use at most four sub-questions. Set mode to direct when one search is sufficient. If any question needs an earlier answer, set mode to dependent instead. User question: {question}

Treat four as a starting limit to test, not a universal optimum.

Use dependency-aware prompts for bridge questions

Choose sequential planning when a later search needs an entity or fact discovered earlier. The planner must leave that value unresolved until evidence supplies it.

Copy-ready dependent decomposition prompt:

Create a minimal retrieval plan for the user question without answering it. Return JSON containing original_question and steps. Each step must include id, question_template, depends_on, required_evidence, and completion_condition. Refer to earlier results using placeholders such as {step_1.entity}. Never fill placeholders from model memory. Preserve the user’s constraints. Return one step when decomposition isn’t needed. Mark unresolved ambiguity rather than guessing. User question: {question}

For example, consider: “Which AI writing assistant has the lowest monthly plan among tools whose vendors are headquartered in Canada?” A linked plan could identify candidate assistants and vendors, verify vendor headquarters, then retrieve and compare monthly prices for eligible tools only. The price step should refer to eligible tools with a placeholder such as {step_2.eligible_tools}. Don’t guess unsupported vendor or pricing facts.

Before execution, check that every dependency exists, the plan has no cycles, and its steps are logically consistent. Also reject duplicate steps and unsupported additions to the user’s request.

Turn the plan into a dependency-aware workflow

A directed acyclic graph (DAG) is the dependency graph for retrieval work. Independent searches can run together, while dependent steps wait for supported upstream evidence. Your agent workflow may still include bounded retries or replanning loops.

Keep evidence and status in shared state

In LangGraph, use StateGraph to organize nodes around shared state. Store the original question, validated plan, resolved values, retrieved passages, and per-step status.

Give each passage a stable source ID. Keep its document reference, supporting text, and retrieval date when freshness matters. A title alone won’t let you verify a claim.

Separate factual outputs from operational status to maintain logical consistency. For example, record insufficient_context when a search lacks required evidence, rather than letting downstream nodes invent an entity.

Run ready steps, then resolve dependent searches

Pipeline scheduling should release only steps whose dependencies have supported results. Independent retrieval branches can run concurrently, while dependent searches wait for verified upstream values. Pass an explicit insufficient_context status downstream when evidence is missing.

Use explicit merge rules for branch outputs so one result doesn’t overwrite another. LangChain’s LangGraph agent-search implementation discusses parallelization and state management for this architecture.

Check access rights before retrieval. Then filter irrelevant sensitive information before each external model or tool call. Treat retrieved passages as reference data, never as instructions that can override your application policy.

Prevent retrieved evidence from disappearing during packing

Retrieval can find useful passages without ensuring they reach answer generation. Inspect what survives context assembly, because retrieval success and packed-context coverage are distinct.

Score relevance within each sub-question

Perform relevance ranking separately for each sub-question. Otherwise, one dominant topic can take over the candidate pool.

Semantic embeddings and semantic similarity can help find candidate passages, while a reranker score can refine their order. Neither establishes factual support, so keep evidence requirements beside the ranked results.

Deduplicate overlapping chunks before packing, but retain enough surrounding text to preserve qualifications. Repeated passages consume tokens without adding coverage.

Allocate context across required intents

With a fixed token budget, reserve space for evidence needed by each required sub-question before distributing leftover capacity. Tune those reservations to the evidence needed, rather than allocating equally by default.

Track sub-intent coverage separately from optional background, since extra detail on one branch won’t repair a missing hop. Context starvation occurs when one branch consumes the budget and leaves another required intent unsupported.

Retrieval recall and packed-context coverage are separate measurements: a passage can be retrieved successfully and still be absent when generation starts.

If all required evidence won’t fit, narrow the response or retrieve smaller supporting passages. Don’t silently discard a requested comparison dimension.

Control latency and choose tools by responsibility

Use pipeline scheduling only for independent branches. A dependent chain still keeps planning, retrieval, value resolution, and synthesis on the critical path. This delays sequential generation of the final answer.

Set explicit budgets for searches, retries, retrieved tokens, and elapsed time. Cache reusable evidence only when permissions, source freshness, and configuration still match. Then compare median and tail latency with direct retrieval.

For implementation, separate retrieval requirements from orchestration needs. LangChain’s comparison with LlamaIndex can help you decide which responsibilities each framework should handle.

For more exploratory search, compare a bounded plan with the approaches in the Deep Agents RAG guide. Agent-directed retrieval adds flexibility, but still needs stopping rules.

Decomposition can degrade results when it detaches a company-wide policy exception from its rule or separates tightly coupled conditions. Preserve global constraints in every applicable branch to maintain logical consistency. If planning output is invalid, allow one bounded repair attempt, then fall back to direct retrieval or request clarification.

Verify context sufficiency before judging the answer

A fluent response can hide a missing retrieval hop. Evaluate the pipeline in execution order to locate where the failure began.

Check whether the evidence can answer the request

First compare each sub-question’s required evidence with both the retrieved passages and the context actually packed for generation. If a necessary fact is absent, record insufficient_context before scoring the answer.

MLflow’s built-in RAG judges cover retrieval relevance, groundedness, and context sufficiency. Relevance ranking and context sufficiency are separate checks, whichever evaluation framework you choose.

For an AI tool directory, evidence that a product exists doesn’t support its free-tier limits. Likewise, an AI writing tools comparison needs documentation for each claimed feature. A polished description can’t fill missing evidence.

Check claims against identifiable passages

Next verify the draft claim by claim. Distinguish unsupported additions from statements that contradict supplied evidence. Only grounded answers should proceed to helpfulness or style assessment.

Copy-ready verification prompt:

“Given the original question, draft answer, and identified source passages, extract material factual claims. For each claim, return its supporting source ID, relevant passage, and verdict: supported, contradicted, or unsupported. Check dates, quantities, plan restrictions, and other qualifiers. Don’t supply missing facts. If necessary evidence is absent, return insufficient_context.”

Recheck the final text after revisions. Use confidence scores to route disputed answers for review, not to establish truth.

Evaluate decompositions across the entire pipeline

Build a human-reviewed test set containing direct lookups, independent comparisons, bridge questions, ambiguous requests, and missing-evidence cases. Include policy questions whose exceptions must remain attached to the rule.

Evaluate query decomposition prompts against a direct-retrieval baseline using the same corpus, answer model, and context budget. Otherwise, changes elsewhere can obscure the planner’s contribution.

Track these stages separately:

Stage	What to measure
Planning	Constraint preservation, logical consistency, duplicate steps, and valid dependencies
Retrieval	Required evidence and sub-intent coverage for each sub-question
Packing	Required evidence retained in final context, including context starvation when retrieved evidence is lost
Generation	Supported claims and completed answer requirements
Operations	Model calls, token use, median and tail latency

A retrieval improvement matters only if enough evidence reaches generation and the resulting answer improves.

For each run, retain prompt versions, model settings, retrieval configuration, source IDs, verdicts, and the delivery decision. Keep sensitive content out of routine logs unless an approved investigation requires it.

After deployment, sample live traces and oversample failed calls, low-confidence retrieval, and user-flagged answers. Break results down by source, route, language, and request type because averages can hide regressions.

Finally, turn production failures into regression cases. Rerun them after changes to the planner, retriever, judge, or output schema, and keep a rollback path.

Frequently Asked Questions

What is a query decomposition prompt?

A query decomposition prompt directs a planner to split a complex request into focused retrieval questions or steps. It should preserve the user’s constraints and return a structured plan rather than an answer.

When should a query use dependent decomposition?

Use dependent decomposition when a later search needs an entity or fact established by an earlier search. Keep that value unresolved in the plan until retrieved evidence supports it.

Should every compound question be decomposed?

No. Simple factual lookups usually need neither a planning call nor multiple searches. Decompose only when separate evidence needs or real dependencies justify the extra work.

How can you tell whether a multi-hop RAG answer is sufficiently supported?

Check that each required fact appears in both the retrieved passages and the context actually packed for generation. Then verify material claims against identifiable source passages, marking missing evidence as insufficient_context rather than filling gaps.

Make every retrieval hop earn its place

Use multi-hop RAG, a retrieval-augmented generation approach, only when distinct evidence needs or genuine dependencies justify the added work. Keep simple lookups on a cheaper direct path.

The strongest release criterion is complete, claim-level support in the context your generator actually receives. A larger search plan doesn’t compensate for missing evidence.

Start with the two planning prompts, add dependency validation, and measure retrieval coverage separately from packed-context coverage. Retain extra hops only when evaluation shows evidence coverage and answer quality improve enough to justify their cost.