Continuous Integration Pipelines

What CI does

Continuous integration builds and tests every change automatically, so problems are found minutes after a commit instead of weeks later. A pipeline is defined as code in the repository.

name: ci
on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
          cache: pip                 # reuse downloaded packages between runs
      - run: pip install -r requirements.txt
      - run: ruff check .            # lint
      - run: pytest -q               # tests

  image:
    needs: test                      # only if tests pass
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t ghcr.io/acme/api:${{ github.sha }} .
      - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
      - run: docker push ghcr.io/acme/api:${{ github.sha }}

GitLab CI (.gitlab-ci.yml), Azure Pipelines, Jenkins and CircleCI express the same ideas: triggers, jobs that run on fresh machines (runners), steps, dependencies between jobs, caches and secrets.

Good pipeline habits

• Fast feedback: lint and unit tests first, and parallel jobs; aim for under 10 minutes.
• Build once, deploy many: build the image once, tag it with the commit SHA, and promote the same image through staging to production.
• Least privilege: set permissions: explicitly; use OIDC to the cloud instead of stored keys.
• Pin actions to versions or commit SHAs; third-party actions run with your secrets.
• Required checks on main so nothing merges without a green pipeline.

STEPS = [("checkout", 5, None), ("install deps", 90, "deps"), ("lint", 15, None), ("unit tests", 120, None),
         ("build image", 150, "layers"), ("push image", 30, None)]

def run(cache_warm):
    total = 0
    for name, seconds, cacheable in STEPS:
        if cache_warm and cacheable:
            seconds = seconds // 6          # cache hit: restore instead of rebuild
        total += seconds
    return total

print(f"cold cache: {run(False) / 60:.1f} min")
print(f"warm cache: {run(True) / 60:.1f} min")

cold cache: 6.8 min
warm cache: 3.5 min