Sign in Get started

Kubernetes in Production · Capstone

Capstone: Production Readiness Review for a Payments API

Review and fix the Kubernetes setup of a payments API before launch: deploys, resources, security, networking, scaling and upgrades.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

Example company: a fintech is moving its payments API to Kubernetes before a marketing launch expected to triple traffic. The current manifests: one Deployment (2 replicas, no probes, no resources), a LoadBalancer Service, a ConfigMap with feature flags, a Secret created by hand, the default service account (bound to cluster-admin "temporarily"), no network policies, and a manually set replicas: 2. The cluster runs Kubernetes two minor versions behind.

Requirements

  • No user-visible errors during deploys, scale-downs or node upgrades.
  • Handle three times the current peak with acceptable latency.
  • Pass a security review (PCI scope: payments data must be isolated).

Constraints

  • The API takes 25 seconds to warm up and holds connections to PostgreSQL and a card processor.
  • The team has two weeks and no dedicated platform engineer.

Deliverables

  1. Deploy safety: probes, rollout strategy, termination.
  2. Resources and scaling: requests, limits, HPA, disruption budgets, spreading.
  3. Configuration and secrets: how config changes roll out; where secrets come from.
  4. Security: RBAC, pod security, network policies.
  5. Upgrades and operations: catching up on versions, validation in CI, alerts.

Hints

  • Every lesson in this course maps to at least one item here.
  • Prioritise: what could cause an outage at launch?

Draft each deliverable below before revealing the model answer and rubric.

Further reading: Kubernetes production best practices: configuration · Pod Security Standards

Deliverables

Describe deploy safety.

Model answer

  • startupProbe on /ready allowing 60 s; readinessProbe on /ready (app warm, DB pool open; not the card processor); livenessProbe on /live checking only the process.
  • Rolling update with maxUnavailable: 0, maxSurge: 1, minReadySeconds: 10; alert on stalled rollouts.
  • preStop sleep of 5 s, SIGTERM handler that stops accepting and drains, terminationGracePeriodSeconds: 60.

Rubric

Look for startup plus readiness probes with sensible scopes, zero-unavailability rollouts and graceful termination covering propagation delay.

Describe resources and scaling.

Model answer

  • Requests from load-test measurements (for example 300m CPU, 768Mi memory), memory limit 1Gi, no CPU limit; JVM sized to the memory limit.
  • Remove replicas from the applied manifest; HPA min 4, max 20, target 60 % CPU (warm-up time needs headroom), slow scale-down behaviour.
  • PDB maxUnavailable: 1; topology spread across zones; cluster autoscaler confirmed to have capacity for max replicas.

Rubric

Strong answers derive requests from measurements, avoid CPU throttling, let the HPA own replicas and protect availability during drains.

Describe configuration and secrets.

Model answer

  • ConfigMap generated with a content hash (Kustomize) or a checksum annotation (Helm) so changes roll out; runtime feature flags moved to a flag service.
  • Secrets from a cloud secret manager via External Secrets Operator; the hand-made Secret deleted after migration; etcd encryption at rest verified.

Rubric

Look for config changes tied to rollouts and secrets sourced from an external store rather than created by hand or committed.

Describe security.

Model answer

  • Dedicated ServiceAccount with no API permissions and automountServiceAccountToken: false; the cluster-admin binding removed; audit all cluster-admin bindings.
  • Namespace labelled pod-security.kubernetes.io/enforce=restricted; pod runs non-root, drops all capabilities, read-only root filesystem, RuntimeDefault seccomp.
  • NetworkPolicies: default deny ingress and egress; ingress only from the ingress controller; egress to PostgreSQL, the card processor's IP ranges (or an egress gateway) and DNS.

Rubric

Strong answers combine least-privilege RBAC, restricted pod security and default-deny network policies that keep DNS working.

Describe upgrades and operations.

Model answer

  • Upgrade the cluster one minor version at a time in staging then production before launch; scan manifests and charts for removed APIs; validate against the target version in CI with kubeconform.
  • Alerts: 5xx rate, p99 latency, OOMKills, CPU throttling ratio, HPA at max, rollout stalled, certificate and node health.
  • Load test at three times peak including a rolling deploy and a node drain during the test.

Rubric

Look for a concrete upgrade plan, CI validation, meaningful alerts and a load test that exercises deploys and drains, not just steady load.

Open in the app to mark lessons complete, take notes and track your progress.

Was this page helpful?