Remote MCP Servers in Production

Scenario

Example company: the customer-database MCP server from earlier is now used by twenty agents. The team moves it from stdio on developers' machines to a remote Streamable HTTP server behind a load balancer with two instances.

What you see in production

About half of all tool calls fail with "session not found" after the move. The failures disappear when the team scales down to one instance. A week later, one runaway agent sends 40 calls per second and slows the database for everyone.

The naive code

sessions = {}                                        # in this process's memory only

def handle(request):
    if request.method == "initialize":
        sid = new_session_id()
        sessions[sid] = {"client": request.client_info}
        return reply(request, headers={"MCP-Session-Id": sid})
    if request.headers["MCP-Session-Id"] not in sessions:
        return error(404, "session not found")
    ...

Reproduce it

Requests for one session are spread round-robin across two instances:

from itertools import cycle

class Instance:
    def __init__(self, name, store):
        self.name, self.store = name, store
    def handle(self, method, sid):
        if method == "initialize":
            self.store[sid] = {"initialized": True}
            return "ok"
        return "ok" if sid in self.store else "404 session not found"

def run(shared):
    store = {}
    instances = [Instance("a", store if shared else {}), Instance("b", store if shared else {})]
    balancer = cycle(instances)
    results = [next(balancer).handle(m, "s1") for m in ["initialize", "tools/list", "tools/call", "tools/call"]]
    return results

print("per-instance memory:", run(shared=False))
print("shared store:      ", run(shared=True))

per-instance memory: ['ok', '404 session not found', 'ok', '404 session not found']
shared store:       ['ok', 'ok', 'ok', 'ok']

Root cause

Streamable HTTP can be used with sessions: the server returns an MCP-Session-Id on initialization and the client sends it on later requests. If session state lives in one process's memory, any request routed to another instance fails. Remote servers also face problems stdio servers never had: authentication, many clients at once, and one client starving others.

The fix

• Prefer stateless. If your tools do not need per-session state, run the server stateless so any instance can answer any request:

app = mcp.streamable_http_app(stateless_http=True, json_response=True)   # an ASGI app for uvicorn

• If you need sessions, keep their state in a shared store (Redis, a database) or use sticky routing on the session ID, and expire idle sessions.
• Authenticate as the spec describes: the server is an OAuth resource server, publishes its protected resource metadata, and accepts only tokens issued for it.
• Limit and isolate: per-client rate limits, timeouts per tool, request size limits, and concurrency caps on expensive tools.
• Version tools carefully: adding optional parameters is safe; renaming or removing tools breaks agents, so deprecate first.

Trade-offs

• Stateless servers cannot push server-initiated messages tied to a session or resume interrupted streams; most tool servers do not need either.
• A shared session store is another dependency to run and secure.
• Rate limits protect the backend but surface as tool errors; return a clear "retry after" message so agents back off.

Detect it in production

• 404 "session not found" rate, by instance.
• Calls per client per minute, latency per tool, and error rate per tool.
• Auth failures by reason (expired, wrong audience) to catch misconfigured clients.

Key points

• Session state in process memory breaks behind a load balancer; go stateless or share the state.
• Remote MCP servers are OAuth resource servers: validate audience, never accept tokens meant for others.
• Rate-limit per client, time out tools, and version tools without breaking agents.

Further reading: MCP transports: Streamable HTTP and sessions · MCP authorization · MCP Python SDK