Locked Out on Launch Day

Scenario

Example deployment: a 2,000-user rollout at an insurer. The app uses SAML SSO with the insurer's identity provider and SCIM to provision users. It was tested with the vendor's own test identity provider and three test accounts.

What you see in production

At 08:00 on launch day, about a third of sign-ins fail with "assertion not yet valid"; the rest succeed but users see no data, because their IdP groups don't map to any role. By 08:30 the help desk queue has 400 tickets. The only admin account is SSO-bound, so the FDE can't log in to fix role mappings either.

The naive code

def validate(assertion, now):
    return assertion.not_before <= now < assertion.not_on_or_after    # zero tolerance for clock skew

ROLE_MAP = {"app-users": "viewer"}           # the insurer's groups are named "ins-claims-handlers", etc.

Reproduce it

Clock skew between the identity provider and the app's servers, plus group mappings, decide who gets in:

import random

random.seed(4)
ROLE_MAP = {"app-users": "viewer"}
ROLE_MAP_FIXED = {"ins-claims-handlers": "editor", "ins-claims-readonly": "viewer", "ins-it-admins": "admin"}

def sign_in(skew_s, groups, tolerance_s, role_map):
    issued = 0
    now = issued + skew_s                     # our server's clock relative to the IdP
    if now < issued - tolerance_s:
        return "error: assertion not yet valid"
    roles = [role_map[g] for g in groups if g in role_map]
    return f"ok as {roles[0]}" if roles else "signed in, but no role: sees nothing"

users = [(random.choice([-75, -40, 0, 0, 20]), [random.choice(list(ROLE_MAP_FIXED))]) for _ in range(9)]
for label, tolerance, role_map in [("launch config", 0, ROLE_MAP), ("fixed config", 120, ROLE_MAP_FIXED)]:
    results = [sign_in(skew, groups, tolerance, role_map) for skew, groups in users]
    ok = sum(r.startswith("ok") for r in results)
    print(f"{label:13} {ok}/{len(users)} users working; e.g. {sorted(set(results))}")

launch config 0/9 users working; e.g. ['error: assertion not yet valid', 'signed in, but no role: sees nothing']
fixed config  9/9 users working; e.g. ['ok as admin', 'ok as editor', 'ok as viewer']

Root cause

SSO was tested against a friendly test IdP, not the customer's real configuration. The app accepted no clock skew, while some of its servers ran slightly behind the IdP (NTP misconfigured on one node pool). Group-to-role mappings used placeholder names. And there was no break-glass local admin to fix configuration when SSO itself was the problem.

The fix

1. Test with the customer's real IdP in a staging tenant, with real group names and several real (test) users from each group, before launch.
2. Allow a small clock skew (commonly up to a few minutes) and make sure all servers sync time (NTP / chrony).
3. Validate everything else strictly: signature, issuer, audience, recipient, and replay protection.
4. Map groups explicitly, with a default "no access" page that tells users who to contact, and a report of users without roles.
5. Break-glass admin: a local admin account with a strong secret in the customer's vault, monitored, for when SSO fails.
6. SCIM dry run: provision a pilot group first, check attributes and deprovisioning, then expand.

Trade-offs

• Clock-skew tolerance slightly widens the replay window; keep it small and rely on one-time assertion IDs.
• Break-glass accounts are a risk if unmanaged; vault them, alert on use, and rotate after use.
• Phased rollouts (by group) slow the launch but contain problems to a few users.

Detect it in production

• SSO error rates by error type (signature, audience, time window) on a launch dashboard.
• Count of signed-in users without roles.
• Time-sync monitoring on every node.

Key points

• Test SSO with the customer's real IdP, groups and users before launch.
• Allow a small, explicit clock skew and keep servers in time sync; validate everything else strictly.
• Keep a vaulted break-glass admin and roll out in phases.