CI/CD Operations

Status: Active Production Last Updated: 2025-11-23

Overview

Documentation for the Archery Apprentice CI/CD infrastructure, covering hybrid runner architecture, multi-platform workflows, and operational best practices.

Documentation Index

Architecture & Strategy

• Hybrid Runner System - Automatic quota management system balancing GitHub-hosted and self-hosted runners
• Runner Selection System - Critical fixes for runner tag detection and multi-label format (PR #298)
• Cross-Platform Runner Patterns - Essential patterns for workflows that must work on Windows, macOS, and Linux (BILLING_FIX_2025_11_27)
• Workflows Overview - Complete reference for all GitHub Actions workflows

Multi-Platform Operations

• Multi-Platform Workflows - Comprehensive guide to cross-platform CI/CD including:

  • November 2025 cascade failure analysis (PRs #287, #288, #289)
  • Defensive programming patterns
  • Shell compatibility strategies
  • Encoding and tool portability
  • Best practices and lessons learned

• Platform Compatibility Matrix - Detailed reference for tool and command availability across:

  • GitHub-hosted Ubuntu runners
  • Self-hosted Windows runners
  • Self-hosted macOS runners

Maintenance & Troubleshooting

• Troubleshooting Guide - Common failure patterns and solutions
• Maintenance Tasks - Routine maintenance procedures

Incident Reports

Recent Incidents:

• BILLING_FIX_2025_11_27 - GitHub billing blocks all runners, fixed by moving decide_runner to self-hosted
• November 2025 Multi-Platform Cascade (PRs 287-289) - Documented in Multi-Platform Workflows

For older incident reports, see the main repository:

• docs/incidents/ directory in archery-apprentice

Quick Start

For Developers

If you’re writing or modifying workflows:

1. Start here: Platform Compatibility Matrix

   • Check what commands are available on each platform
   • Verify tool versions and shell availability

2. Then read: Multi-Platform Workflows

   • Learn defensive programming patterns
   • Understand the November 2025 failures
   • Apply best practices to avoid similar issues

3. Reference: Hybrid Runner System

   • Understand how runner selection works
   • Learn about quota management strategy

For Troubleshooting

If a workflow is failing:

1. Check: Troubleshooting Guide

   • Common error patterns
   • Platform-specific issues

2. Verify: Platform Compatibility Matrix

   • Tool availability on the failing platform
   • Shell and encoding compatibility

3. Review: Multi-Platform Workflows

   • Defensive patterns for cross-platform execution
   • Validation checklist

Core Principles

1. Defensive Programming

Never assume tool availability, encoding, or shell compatibility. Always:

• Detect commands before using them (pip3 vs pip)
• Declare shell explicitly (shell: bash)
• Set encoding for Python (PYTHONIOENCODING: utf-8)
• Use platform detection for variant tools (GNU vs BSD sed)

2. Cross-Platform by Default

All workflows must work on all three runner types:

• GitHub-hosted Ubuntu
• Self-hosted Windows
• Self-hosted macOS

3. Test Before Merge

Use workflow_dispatch to test changes on all platforms before merging to main.

4. Document Failures

When a platform compatibility issue is discovered:

1. Fix the immediate issue
2. Update this documentation
3. Add to Platform Compatibility Matrix
4. Share lessons learned in Multi-Platform Workflows

Platform Summary

See Platform Compatibility Matrix for complete tool availability reference.

Critical Lessons from November 2025

The cascade of PRs #287, #288, and #289 exposed three layers of platform incompatibility:

1. Shell Incompatibility - PowerShell commands on bash-only runners
2. Encoding Incompatibility - UTF-8 characters (emoji) failing on Windows cp1252
3. Tool Variant Incompatibility - pip vs pip3, GNU sed vs BSD sed

Key Takeaway: Platform compatibility issues are fractal. Each fix reveals deeper incompatibilities. Only comprehensive defensive programming prevents cascading failures.

Required Reading: Multi-Platform Workflows - The November 2025 Cascade

Defensive Programming Checklist

Use this checklist for all workflow changes:

• Shell declared - All steps have shell: bash or explicit shell
• Commands exist - No assumptions about pip/python/sed availability
• Encoding set - PYTHONIOENCODING=utf-8 for Python steps
• Platform detection - sed, grep, find use platform-safe syntax
• Tested on all runners - Manual workflow_dispatch on ubuntu/windows/macos
• Error handling - Critical steps have set -e or error checks
• Fallbacks defined - Commands have fallback detection (pip3 → pip)
• No emoji without encoding - Remove emoji or set UTF-8 encoding

See complete checklist in Multi-Platform Workflows - Validation Checklist

Contributing

When adding new CI/CD documentation:

1. Update this index - Add links to new documents
2. Cross-reference - Link related documents to each other
3. Use consistent terminology - Match naming conventions in existing docs
4. Include code examples - Show both wrong and correct patterns
5. Test instructions - Verify all code examples work on all platforms

Related Documentation

Developer Guide

• Branch Protection - GitHub branch protection rules
• Hybrid Runner Implementation Guide - Technical implementation details

Patterns & Best Practices

• PowerShell Emoji Encoding - Specific encoding issue documentation

Main Repository

• Incident Reports: docs/incidents/ in archery-apprentice repo
• Workflow Files: .github/workflows/ in archery-apprentice repo

Document Revision History:

• 2025-11-28: Added Runner Selection System guide (PR #298 fixes)
• 2025-11-27: Added billing block incident and cross-platform patterns
• 2025-11-23: Initial creation with multi-platform workflow documentation index