Forward Deployed Engineering Tutorial · AI Deployments · Lesson 18 of 24

Integrating LLMs into Customer Workflows

Put a large language model into a real workflow: structured outputs, validation, fallbacks, human review and cost control.

Create a free account to save this page, take notes and track your progress. Sign in

From chat demo to workflow

Customers rarely need a chat window; they need a step in a workflow done better: classify a ticket, extract fields from a contract, draft a reply for review. Design it like any other integration:

Structured output: ask for JSON matching a schema, and validate it.
Grounding: give the model the customer's documents or data (retrieval), not just its general knowledge.
Fallbacks: retry once on invalid output, then route to a human.
Human in the loop where mistakes are costly: the model drafts, a person approves.
Costs and limits: tokens per request, rate limits, latency budgets, caching.

Validate model output before using itTry it yourself

import json

SCHEMA = {"category": {"billing", "technical", "account"}, "priority": {"low", "medium", "high"}}

def validate(raw):
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        return None, "not JSON"
    for field, allowed in SCHEMA.items():
        if data.get(field) not in allowed:
            return None, f"invalid {field}: {data.get(field)!r}"
    return data, "ok"

model_outputs = ['{"category": "billing", "priority": "high"}',
                 'Sure! Here is the JSON: {"category": "billing"}',
                 '{"category": "legal", "priority": "low"}']
for raw in model_outputs:
    data, status = validate(raw)
    print(f"{status:24} -> {'use result' if data else 'retry once, then send to a human'}")

Output

ok                       -> use result
not JSON                 -> retry once, then send to a human
invalid category: 'legal' -> retry once, then send to a human

For the API details, see the course Integrating LLM APIs.

Test yourself

A model sometimes returns invalid JSON for an extraction task. What's the robust approach?

Trust the output
Validate against a schema, retry once, then route to human review
Raise the temperature
Remove the schema

Show answer

Answer: Validate against a schema, retry once, then route to human review

Treat model output as untrusted input, like any external API response.

Was this page helpful?