Multi-Platform CI/CD Workflows

Last Updated: 2025-11-23 Status: Active Production Related: Platform Compatibility Matrix, Hybrid Runner System


Overview

The Archery Apprentice project uses a hybrid GitHub Actions runner system across three platform types, each with distinct environments, tooling, and compatibility requirements. This guide documents the comprehensive multi-platform strategy, lessons learned from production incidents, and defensive programming patterns that ensure reliable cross-platform execution.

Runner Platform Summary

Runner TypeOSPrimary UseShell DefaultNotable Constraints
GitHub-hostedubuntu-latestQuick checks, validationbashPowerShell not available
Self-hosted WindowsWindows 11 ProAndroid builds, heavy testingPowerShellUTF-8 encoding issues, no bash
Self-hosted macOSmacOS SonomaiOS builds (future), fallbackbashBSD tools, pip3 only (no pip)

Critical Insight: What works on one platform often fails silently on another due to shell differences, encoding variations, and tool availability disparities.


The November 2025 Cascade: A Case Study

Between November 22-23, 2025, three sequential CI/CD failures exposed systemic platform compatibility issues that had been masked by the hybrid runner system’s day-of-month logic.

Timeline of Failures

PR #287: PowerShell on Ubuntu Failure

Merged: 2025-11-23 01:22 UTC Issue: android-ci.yml used PowerShell commands on ubuntu-latest runner Symptom: /usr/bin/bash: line 1: syntax error near unexpected token 'foreach'

# BROKEN - PowerShell on Ubuntu
run: |
  ${{ needs.decide_runner.outputs.runner }} -eq 'ubuntu-latest' | foreach { Write-Host "Runner: $_" }

Root Cause: GitHub-hosted ubuntu-latest does not have PowerShell installed by default. The workflow assumed PowerShell availability based on self-hosted Windows experience.

Fix: Converted all PowerShell logic to bash equivalents

# FIXED - Bash on Ubuntu
run: |
  echo "Runner selected: ${{ needs.decide_runner.outputs.runner }}"
  if [ "${{ needs.decide_runner.outputs.runner }}" = "ubuntu-latest" ]; then
    echo "Using GitHub-hosted runner"
  fi

Incomplete: Only fixed android-ci.yml, missed deploy-to-play-store.yml


PR #288: UTF-8 Encoding + Deploy Workflow

Merged: 2025-11-23 01:57 UTC Issues:

  1. deploy-to-play-store.yml still had PowerShell on bash
  2. Python UTF-8 encoding crash on Windows

Symptoms:

  • Same bash syntax error on deploy workflow
  • Python crash: UnicodeEncodeError: 'charmap' codec can't encode character '\u2705' (checkmark emoji) in position 43: character maps to <undefined>

Root Cause:

  1. Incomplete fix from PR #287
  2. Windows default encoding is cp1252, not UTF-8

Python Error Context:

# update_build_version.py contained:
print("Build version updated successfully ✅")  # Emoji fails on Windows cp1252

Fix Applied:

# Added to all workflows with Python
env:
  PYTHONIOENCODING: utf-8

Still Incomplete: Didn’t test pip/pip3 availability across platforms


PR #289: Comprehensive Platform Audit

Merged: 2025-11-23 02:49 UTC Issues:

  1. macOS has pip3 but not pip symlink
  2. GNU sed vs BSD sed incompatibility
  3. python3 vs python command availability

Symptoms on macOS:

pip --version
# bash: pip: command not found
 
sed -i 's/old/new/' file.txt
# sed: 1: "file.txt": invalid command code f

Root Causes:

  • macOS uses BSD sed requiring -i '' (empty string for in-place)
  • GNU sed (Linux) requires -i (no space)
  • macOS Python installation provides pip3 and python3 but not pip or python

Comprehensive Fix:

# Python runtime detection with fallback
PYTHON_CMD="python3"
if ! command -v python3 &> /dev/null; then
  PYTHON_CMD="python"
fi
 
# pip detection with fallback
PIP_CMD="pip3"
if ! command -v pip3 &> /dev/null; then
  PIP_CMD="pip"
fi
 
# Platform-safe sed in-place editing
if [[ "$OSTYPE" == "darwin"* ]]; then
  # macOS (BSD sed)
  sed -i '' 's/old/new/' file.txt
else
  # Linux (GNU sed)
  sed -i 's/old/new/' file.txt
fi

Cascade Analysis

The three failures represent a progressive discovery pattern:

  1. Surface Issue (PR #287): Shell incompatibility (PowerShell vs bash)
  2. Secondary Issue (PR #288): Encoding incompatibility (UTF-8 vs cp1252)
  3. Deep Issue (PR #289): Tool availability and Unix variant differences

Key Lesson: Platform compatibility issues are fractal - each fix reveals deeper incompatibilities. Defensive programming must assume nothing about the runtime environment.


Defensive Programming Patterns

Pattern 1: Command Availability Detection

Never assume a command exists. Always detect and fallback.

# WRONG - Assumes pip exists
pip install requests
 
# CORRECT - Detect pip3 first, fallback to pip
if command -v pip3 &> /dev/null; then
  pip3 install requests
elif command -v pip &> /dev/null; then
  pip install requests
else
  echo "ERROR: Neither pip3 nor pip found"
  exit 1
fi

Application in Workflows:

- name: Install Python dependencies (defensive)
  shell: bash
  run: |
    # Detect pip command
    if command -v pip3 &> /dev/null; then
      PIP_CMD="pip3"
    elif command -v pip &> /dev/null; then
      PIP_CMD="pip"
    else
      echo "ERROR: No pip installation found"
      exit 1
    fi
 
    echo "Using pip command: $PIP_CMD"
    $PIP_CMD install --upgrade pip
    $PIP_CMD install -r requirements.txt

Pattern 2: Platform Detection for Tool Variants

Different Unix variants have incompatible tool implementations.

# WRONG - Assumes GNU sed
sed -i 's/version=1/version=2/' version.txt
 
# CORRECT - Detect platform and adapt
if [[ "$OSTYPE" == "darwin"* ]]; then
  # macOS uses BSD sed
  sed -i '' 's/version=1/version=2/' version.txt
else
  # Linux uses GNU sed
  sed -i 's/version=1/version=2/' version.txt
fi

Real-World sed Portability Function:

# Create a portable sed in-place function
portable_sed_inplace() {
  local pattern="$1"
  local file="$2"
 
  if [[ "$OSTYPE" == "darwin"* ]]; then
    # BSD sed (macOS)
    sed -i '' "$pattern" "$file"
  else
    # GNU sed (Linux)
    sed -i "$pattern" "$file"
  fi
}
 
# Usage
portable_sed_inplace 's/old/new/g' myfile.txt

Pattern 3: Encoding Declarations

Never rely on system default encoding.

# WRONG - Relies on system encoding
- name: Run Python script
  run: python update_version.py
 
# CORRECT - Explicit UTF-8 encoding
- name: Run Python script with UTF-8
  env:
    PYTHONIOENCODING: utf-8
  run: python update_version.py

Why This Matters:

  • Windows default: cp1252 (Western European)
  • Linux default: UTF-8
  • macOS default: UTF-8
  • Without PYTHONIOENCODING, emoji and special chars crash on Windows

Application Across All Python Steps:

jobs:
  build:
    runs-on: ${{ matrix.os }}
    env:
      # Global encoding for entire job
      PYTHONIOENCODING: utf-8
    steps:
      - name: Python operation 1
        run: python script1.py
 
      - name: Python operation 2
        run: python script2.py
 
      # All inherit UTF-8 encoding

Pattern 4: Shell Specification

Always declare shell explicitly in workflow steps.

# WRONG - Implicit shell (differs by runner OS)
- name: Build app
  run: ./gradlew assembleRelease
 
# CORRECT - Explicit shell declaration
- name: Build app
  shell: bash
  run: ./gradlew assembleRelease

Shell Availability Matrix:

Shellubuntu-latestWindows Self-hostedmacOS Self-hosted
bash✓ Default✓ Git Bash✓ Default
PowerShell✗ Not installed✓ Default✗ Not installed
pwsh (PS Core)✓ Available✓ Available✓ Available
sh✓ Available✗ Limited✓ Available

Recommendation: Use shell: bash for all cross-platform workflows. Install Git for Windows on Windows self-hosted runners to provide bash.


Pattern 5: Multi-Line Command Safety

Use proper quoting and continuation for multi-line commands.

# WRONG - Line breaks can cause parsing issues
- name: Complex build
  run: |
    echo "Starting build"
    ./gradlew clean assembleRelease
    echo "Build complete"
 
# CORRECT - Explicit error handling and status checks
- name: Complex build with error handling
  shell: bash
  run: |
    set -e  # Exit on any error
    set -u  # Exit on undefined variable
    set -o pipefail  # Catch errors in pipes
 
    echo "Starting build"
    ./gradlew clean assembleRelease
    echo "Build complete"

Advanced: Atomic Multi-Command Operations

# Use subshells for atomic operations
(
  set -e
  echo "Step 1: Clean"
  ./gradlew clean
  echo "Step 2: Build"
  ./gradlew assembleRelease
  echo "Step 3: Test"
  ./gradlew test
) || {
  echo "Build pipeline failed"
  exit 1
}

Pattern 6: Runtime Environment Validation

Validate the environment before executing critical operations.

- name: Validate build environment
  shell: bash
  run: |
    set -e
 
    echo "=== Environment Validation ==="
 
    # Check required commands
    command -v java >/dev/null 2>&1 || { echo "ERROR: java not found"; exit 1; }
    command -v gradle >/dev/null 2>&1 || echo "WARNING: gradle not in PATH (using wrapper)"
 
    # Check Java version
    JAVA_VERSION=$(java -version 2>&1 | head -n 1)
    echo "Java: $JAVA_VERSION"
 
    # Check environment variables
    if [ -z "${ANDROID_HOME:-}" ]; then
      echo "ERROR: ANDROID_HOME not set"
      exit 1
    fi
 
    echo "Android SDK: $ANDROID_HOME"
    echo "=== Validation Complete ==="

Pattern 7: Conditional Platform Logic

Use workflow conditionals for platform-specific steps.

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
 
    steps:
      - name: Linux-specific setup
        if: runner.os == 'Linux'
        run: sudo apt-get install -y some-linux-package
 
      - name: Windows-specific setup
        if: runner.os == 'Windows'
        shell: pwsh
        run: choco install some-windows-package
 
      - name: macOS-specific setup
        if: runner.os == 'macOS'
        run: brew install some-macos-package
 
      - name: Cross-platform build
        shell: bash
        run: ./gradlew assembleRelease

Platform-Specific Best Practices

Ubuntu (GitHub-hosted)

Strengths:

  • Fast, consistent, disposable runners
  • Full sudo access
  • Complete GNU toolchain

Limitations:

  • No PowerShell by default
  • 2-hour job limit
  • Counts against quota

Best Practices:

- name: Ubuntu operation
  if: runner.os == 'Linux'
  shell: bash
  run: |
    # Use apt for dependencies
    sudo apt-get update
    sudo apt-get install -y build-essential
 
    # GNU tools available
    sed -i 's/old/new/' file.txt
    grep -r "pattern" .

Windows (Self-hosted)

Strengths:

  • Persistent environment
  • No quota consumption
  • PowerShell native

Limitations:

  • cp1252 encoding default
  • Backslash path separators
  • Case-insensitive filesystem

Best Practices:

- name: Windows operation
  if: runner.os == 'Windows'
  shell: bash  # Use bash from Git for Windows
  env:
    PYTHONIOENCODING: utf-8  # Critical for emoji/unicode
  run: |
    # Use forward slashes in bash
    ./gradlew.bat assembleRelease
 
    # Or use pwsh for PowerShell Core

Encoding Checklist:

  • Always set PYTHONIOENCODING=utf-8 for Python
  • Avoid emoji in output unless encoding is set
  • Use UTF-8 file encoding for all scripts

macOS (Self-hosted)

Strengths:

  • Native iOS build capability
  • BSD toolchain familiarity
  • Persistent environment

Limitations:

  • BSD tools (not GNU)
  • pip3 only (no pip symlink)
  • python3 only (no python symlink)

Best Practices:

- name: macOS operation
  if: runner.os == 'macOS'
  shell: bash
  run: |
    # Use pip3 explicitly
    pip3 install requests
 
    # BSD sed requires empty string for in-place
    sed -i '' 's/old/new/' file.txt
 
    # Use python3 explicitly
    python3 script.py

Critical macOS Functions:

# Detect and use correct Python
if command -v python3 &> /dev/null; then
  PYTHON=python3
elif command -v python &> /dev/null; then
  PYTHON=python
else
  echo "ERROR: No Python found"
  exit 1
fi
 
# Detect and use correct pip
if command -v pip3 &> /dev/null; then
  PIP=pip3
elif command -v pip &> /dev/null; then
  PIP=pip
else
  echo "ERROR: No pip found"
  exit 1
fi
 
echo "Using Python: $PYTHON ($($PYTHON --version))"
echo "Using pip: $PIP ($($PIP --version))"

Cross-Platform Testing Strategy

Local Testing Before Push

Test workflow changes on all three platform types before pushing to main.

Testing Script:

#!/bin/bash
# test-workflow-locally.sh
 
set -e
 
echo "=== Testing Workflow Cross-Platform Compatibility ==="
 
# Test 1: Command availability
echo "Test 1: Command Detection"
for cmd in python3 python pip3 pip java gradle; do
  if command -v "$cmd" &> /dev/null; then
    echo "✓ $cmd: $(command -v $cmd)"
  else
    echo "✗ $cmd: NOT FOUND"
  fi
done
 
# Test 2: sed portability
echo ""
echo "Test 2: sed In-Place Editing"
echo "test" > /tmp/sed-test.txt
if [[ "$OSTYPE" == "darwin"* ]]; then
  sed -i '' 's/test/success/' /tmp/sed-test.txt
else
  sed -i 's/test/success/' /tmp/sed-test.txt
fi
result=$(cat /tmp/sed-test.txt)
if [ "$result" = "success" ]; then
  echo "✓ sed in-place edit works"
else
  echo "✗ sed in-place edit failed"
fi
rm /tmp/sed-test.txt
 
# Test 3: Python encoding
echo ""
echo "Test 3: Python UTF-8 Encoding"
export PYTHONIOENCODING=utf-8
python3 -c "print('Unicode test: ✅ 🎯 📱')"
echo "✓ Python UTF-8 encoding works"
 
# Test 4: Shell detection
echo ""
echo "Test 4: Shell Environment"
echo "SHELL: $SHELL"
echo "OSTYPE: $OSTYPE"
echo "Runner OS: $(uname -s)"
 
echo ""
echo "=== All Tests Complete ==="

Workflow Dispatch Testing

Use workflow_dispatch for manual testing on specific runners.

on:
  workflow_dispatch:
    inputs:
      runner_mode:
        description: 'Runner selection'
        required: true
        type: choice
        options:
          - auto
          - ubuntu-latest
          - self-hosted-windows
          - self-hosted-macos
 
      test_mode:
        description: 'Enable additional debug output'
        required: false
        type: boolean
        default: false
 
jobs:
  test:
    runs-on: ${{ github.event.inputs.runner_mode }}
    steps:
      - name: Debug environment
        if: ${{ github.event.inputs.test_mode }}
        shell: bash
        run: |
          echo "=== Debug Environment ==="
          echo "Runner OS: $RUNNER_OS"
          echo "Runner Arch: $RUNNER_ARCH"
          echo "OSTYPE: $OSTYPE"
          echo "Shell: $SHELL"
          echo "Python: $(command -v python3 || command -v python || echo 'NOT FOUND')"
          echo "pip: $(command -v pip3 || command -v pip || echo 'NOT FOUND')"
          echo "=== End Debug ==="

Migration Guide: PowerShell to Bash

For workflows currently using PowerShell on Windows self-hosted runners that need to support ubuntu-latest.

Variable Declaration

# PowerShell
$COMMIT_MSG = "${{ github.event.head_commit.message }}"
$DAY = (Get-Date).Day
# Bash equivalent
COMMIT_MSG="${{ github.event.head_commit.message }}"
DAY=$(date +%d)

Conditionals

# PowerShell
if ($DAY -ge 22) {
  $RUNNER = "self-hosted"
} else {
  $RUNNER = "ubuntu-latest"
}
# Bash equivalent
if [ $DAY -ge 22 ]; then
  RUNNER="self-hosted"
else
  RUNNER="ubuntu-latest"
fi

String Matching

# PowerShell
if ($COMMIT_MSG -match '\[skip-ci\]') {
  Write-Host "Skipping CI"
}
# Bash equivalent
if echo "$COMMIT_MSG" | grep -q '\[skip-ci\]'; then
  echo "Skipping CI"
fi

Output Variables

# PowerShell
echo "runner=$RUNNER" >> $env:GITHUB_OUTPUT
# Bash equivalent
echo "runner=$RUNNER" >> $GITHUB_OUTPUT

Complete Example Migration

Before (PowerShell only):

- name: Decide runner
  id: decide
  shell: pwsh
  run: |
    $DAY = (Get-Date).Day
    if ($DAY -ge 22) {
      $RUNNER = "self-hosted"
    } else {
      $RUNNER = "ubuntu-latest"
    }
    echo "runner=$RUNNER" >> $env:GITHUB_OUTPUT

After (Cross-platform Bash):

- name: Decide runner
  id: decide
  shell: bash
  run: |
    DAY=$(date +%d | sed 's/^0//')  # Remove leading zero
    if [ $DAY -ge 22 ]; then
      RUNNER="self-hosted"
    else
      RUNNER="ubuntu-latest"
    fi
    echo "runner=$RUNNER" >> $GITHUB_OUTPUT

Common Pitfalls and Solutions

Pitfall 1: Assuming PowerShell Availability

Problem:

run: Get-Date

Error on ubuntu-latest:

/usr/bin/bash: line 1: Get-Date: command not found

Solution:

shell: bash
run: date

Pitfall 2: Forgetting Encoding for Emoji

Problem:

print("Build successful ✅")

Error on Windows:

UnicodeEncodeError: 'charmap' codec can't encode character '\u2705'

Solution:

env:
  PYTHONIOENCODING: utf-8
run: python script.py

Pitfall 3: Using pip on macOS

Problem:

pip install requests

Error on macOS:

bash: pip: command not found

Solution:

# Defensive approach
if command -v pip3 &> /dev/null; then
  pip3 install requests
elif command -v pip &> /dev/null; then
  pip install requests
else
  echo "ERROR: No pip found"
  exit 1
fi

Pitfall 4: GNU sed Syntax on macOS

Problem:

sed -i 's/old/new/' file.txt

Error on macOS:

sed: 1: "file.txt": invalid command code f

Solution:

if [[ "$OSTYPE" == "darwin"* ]]; then
  sed -i '' 's/old/new/' file.txt
else
  sed -i 's/old/new/' file.txt
fi

Pitfall 5: Relying on Default Shell

Problem:

- name: Build
  run: ./gradlew build

Issue: Default shell is PowerShell on Windows, bash on Linux/macOS.

Solution:

- name: Build
  shell: bash
  run: ./gradlew build

Validation Checklist

Before merging workflow changes, verify:

  • Shell declared: All steps have shell: bash or explicit shell
  • Commands exist: No assumptions about pip/python/sed availability
  • Encoding set: PYTHONIOENCODING=utf-8 for Python steps
  • Platform detection: sed, grep, find use platform-safe syntax
  • Tested on all runners: Workflow_dispatch tested on ubuntu/windows/macos
  • Error handling: All critical steps have set -e or error checks
  • Fallbacks defined: Commands have fallback detection (pip3 → pip)
  • No emoji without encoding: Remove emoji or set UTF-8 encoding

Future Improvements

Standardized Workflow Template

Create a reusable workflow template with all defensive patterns built-in.

# .github/workflows/templates/cross-platform-template.yml
name: Cross-Platform Template
 
on:
  workflow_call:
    inputs:
      runner_type:
        required: true
        type: string
 
jobs:
  validate:
    runs-on: ${{ inputs.runner_type }}
    env:
      PYTHONIOENCODING: utf-8
    steps:
      - name: Validate environment
        shell: bash
        run: |
          # Source common validation script
          source .github/scripts/validate-environment.sh
 
      - name: Your build steps here
        shell: bash
        run: |
          # Build logic

Shared Shell Functions

Create a library of portable shell functions.

# .github/scripts/platform-utils.sh
 
# Portable sed in-place
portable_sed() {
  local pattern="$1"
  local file="$2"
  if [[ "$OSTYPE" == "darwin"* ]]; then
    sed -i '' "$pattern" "$file"
  else
    sed -i "$pattern" "$file"
  fi
}
 
# Detect Python
detect_python() {
  if command -v python3 &> /dev/null; then
    echo "python3"
  elif command -v python &> /dev/null; then
    echo "python"
  else
    echo "ERROR: No Python found" >&2
    return 1
  fi
}
 
# Detect pip
detect_pip() {
  if command -v pip3 &> /dev/null; then
    echo "pip3"
  elif command -v pip &> /dev/null; then
    echo "pip"
  else
    echo "ERROR: No pip found" >&2
    return 1
  fi
}
 
# Export for use in workflows
export -f portable_sed detect_python detect_pip

Usage in workflows:

- name: Use portable functions
  shell: bash
  run: |
    source .github/scripts/platform-utils.sh
    PYTHON=$(detect_python)
    PIP=$(detect_pip)
    portable_sed 's/version=1/version=2/' version.txt


Appendix: Complete Defensive Workflow Example

name: Cross-Platform Build
 
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  workflow_dispatch:
    inputs:
      runner_mode:
        type: choice
        options: [auto, ubuntu-latest, self-hosted]
 
jobs:
  decide_runner:
    runs-on: ubuntu-latest
    outputs:
      runner: ${{ steps.decide.outputs.runner }}
    steps:
      - name: Decide runner based on date
        id: decide
        shell: bash
        run: |
          DAY=$(date +%d | sed 's/^0*//')
          if [ -n "${{ github.event.inputs.runner_mode }}" ] && [ "${{ github.event.inputs.runner_mode }}" != "auto" ]; then
            RUNNER="${{ github.event.inputs.runner_mode }}"
          elif [ $DAY -ge 22 ]; then
            RUNNER="self-hosted"
          else
            RUNNER="ubuntu-latest"
          fi
          echo "runner=$RUNNER" >> $GITHUB_OUTPUT
          echo "Selected runner: $RUNNER (day $DAY)"
 
  build:
    needs: decide_runner
    runs-on: ${{ needs.decide_runner.outputs.runner }}
    env:
      PYTHONIOENCODING: utf-8
 
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Validate environment
        shell: bash
        run: |
          set -e
          echo "=== Environment Validation ==="
          echo "Runner OS: $RUNNER_OS"
          echo "OSTYPE: $OSTYPE"
 
          # Detect Python
          if command -v python3 &> /dev/null; then
            PYTHON=python3
          elif command -v python &> /dev/null; then
            PYTHON=python
          else
            echo "ERROR: No Python found"
            exit 1
          fi
          echo "Python: $PYTHON ($($PYTHON --version))"
 
          # Detect pip
          if command -v pip3 &> /dev/null; then
            PIP=pip3
          elif command -v pip &> /dev/null; then
            PIP=pip
          else
            echo "ERROR: No pip found"
            exit 1
          fi
          echo "pip: $PIP ($($PIP --version))"
 
          # Store for later steps
          echo "PYTHON_CMD=$PYTHON" >> $GITHUB_ENV
          echo "PIP_CMD=$PIP" >> $GITHUB_ENV
          echo "=== Validation Complete ==="
 
      - name: Install dependencies
        shell: bash
        run: |
          set -e
          $PIP_CMD install --upgrade pip
          $PIP_CMD install -r requirements.txt
 
      - name: Update version file
        shell: bash
        run: |
          set -e
          VERSION_FILE="version.txt"
          NEW_VERSION="2.0.0"
 
          # Platform-safe sed
          if [[ "$OSTYPE" == "darwin"* ]]; then
            sed -i '' "s/version=.*/version=$NEW_VERSION/" "$VERSION_FILE"
          else
            sed -i "s/version=.*/version=$NEW_VERSION/" "$VERSION_FILE"
          fi
 
          echo "Updated version to: $(grep version $VERSION_FILE)"
 
      - name: Run build script
        shell: bash
        run: |
          set -e
          $PYTHON_CMD build.py
 
      - name: Run tests
        shell: bash
        run: |
          set -e
          ./gradlew test
 
      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: build-${{ runner.os }}
          path: build/outputs/

Document Revision History:

  • 2025-11-23: Initial creation documenting November 2025 cascade failures and defensive patterns