Conversational Queries: "What About the Pro Plan?"

Scenario

Example company: a pricing assistant on a SaaS website. Users chat naturally: "How much is the Team plan?", then "And does it include SSO?", then "What about the Pro plan?"

What you see in production

First questions are answered well. Follow-ups are answered badly or with "I couldn't find that". Logs show the retrieval query for the third message was literally "What about the Pro plan?", which matches the Pro plan's price page, not its SSO details.

The naive code

def answer(history, message):
    chunks = search(message, k=5)                   # only the latest message is searched
    return llm(history + [message], chunks)

Reproduce it

Search with the raw follow-up and with a standalone rewrite:

docs = {
    "team-pricing": "team plan price 12 per user per month",
    "team-sso": "team plan includes sso with google and okta",
    "pro-pricing": "pro plan price 29 per user per month",
    "pro-sso": "pro plan includes sso with saml and scim provisioning",
}

def search(query, k=1):
    words = set(query.lower().replace("?", "").split())
    return sorted(docs, key=lambda d: -len(words & set(docs[d].split())))[:k]

history = ["How much is the Team plan?", "And does it include SSO?"]
follow_up = "What about the Pro plan?"

def rewrite(history, message):
    # in production a small model does this; here: carry the topic of the previous question forward
    topic = "sso" if "sso" in history[-1].lower() else ""
    return f"Does the pro plan include {topic}?".replace("  ", " ")

standalone = rewrite(history, follow_up)
print("raw query      :", follow_up, "->", search(follow_up))
print("rewritten query:", standalone, "->", search(standalone))

raw query      : What about the Pro plan? -> ['pro-pricing']
rewritten query: Does the pro plan include sso? -> ['pro-sso']

Root cause

The meaning of a follow-up depends on the conversation, but retrieval sees only the latest message. "What about the Pro plan?" means "does the Pro plan include SSO?" only in context. Embedding the whole conversation instead mixes topics and dilutes the query.

The fix

• Rewrite the query: before retrieval, a small, fast model turns the latest message plus recent history into a standalone search query.
• Multi-query for broad or ambiguous questions: generate 2 or 3 phrasings, search each, and fuse the results.
• Keep the user's own words in the final prompt; the rewrite is only for retrieval.
• Log both the original message and the rewritten query to debug failures.

REWRITE = ("Rewrite the user's last message as a standalone search query using the conversation. "
           "Return only the query.")
query = small_model(system=REWRITE, messages=recent_history + [message])
chunks = search(query, k=20)

Trade-offs

• An extra model call adds latency (often 200 to 500 ms); skip it for first messages, which are already standalone.
• A bad rewrite sends retrieval in the wrong direction; evaluate rewrites on real conversations.
• Multi-query improves recall but multiplies search cost.

Detect it in production

• Retrieval quality for follow-up messages versus first messages.
• "Not found" answers that occur mostly on second and later turns.
• Logged pairs of original message and rewritten query, sampled for review.

Key points

• Follow-up questions are not standalone search queries.
• Rewrite them with a small model before retrieval; consider multi-query for broad questions.
• Log and evaluate rewrites like any other model step.

Further reading: Conversation state in Integrating LLM APIs · Reciprocal rank fusion (for multi-query)