runner-selection-system

CI/CD Runner Selection System

Implemented: 2025-11-28 Status: Active - Production Context: PR #298 fixes after significant debugging Related: Cross-Platform Runner Patterns, Hybrid Runner System

---

Overview

The runner selection system allows developers to control which GitHub Actions runner executes their CI jobs. This guide documents two critical issues discovered during PR #298 debugging and their solutions.

Key Insight: These issues caused jobs to wait indefinitely for non-existent runners, with no clear error message. Understanding these patterns prevents hours of debugging.

---

Critical Issues and Fixes

Issue 1: Merge Commit vs PR Head Commit

Symptom: Runner override tags like [github], [macos], [windows] are ignored on pull requests.

Root Cause: For pull request events, github.event.head_commit.message returns the merge commit message (e.g., “Merge abc123 into def456”), NOT the actual commit message with your tags.

# ❌ WRONG - Gets merge commit message for PRs
$COMMIT_MSG = "${{ github.event.head_commit.message }}"
# Result: "Merge abc123 into def456" - no tags!

Solution: Checkout the actual PR head commit first, then read the commit message from HEAD:

# ✅ CORRECT - Checkout actual PR head commit first
- name: Checkout code
  uses: actions/checkout@v5
  with:
    ref: ${{ github.event.pull_request.head.sha || github.sha }}
    fetch-depth: 1
 
- name: Determine runner
  shell: bash
  run: |
    # Now read commit message from HEAD (the PR commit, not merge commit)
    COMMIT_MSG=$(git log -1 --pretty=%B HEAD)
    echo "Commit message: $COMMIT_MSG"

Why This Works:

• github.event.pull_request.head.sha is the actual commit SHA from the PR branch
• github.sha is the fallback for non-PR events (pushes, workflow_dispatch)
• After checkout, HEAD points to the correct commit with your tags

---

Issue 2: Multi-Label Runner Format

Symptom: Jobs queue indefinitely waiting for a runner that doesn’t exist.

Root Cause: GitHub Actions runs-on expects either a string OR an array. Comma-separated strings are treated as a single label, not multiple labels.

# ❌ WRONG - Comma-separated string becomes single label "self-hosted,macOS"
echo "runner_label=self-hosted,macOS" >> $GITHUB_OUTPUT
runs-on: ${{ needs.decide_runner.outputs.runner_label }}
# GitHub looks for runner with label literally named "self-hosted,macOS"

Solution: Use JSON array format and fromJSON():

# ✅ CORRECT - JSON array format with fromJSON()
echo 'runner_label=["self-hosted", "macOS"]' >> $GITHUB_OUTPUT
runs-on: ${{ fromJSON(needs.decide_runner.outputs.runner_label) }}
# GitHub correctly looks for runner with BOTH labels: self-hosted AND macOS
 
# For single labels, use JSON string format for consistency
echo 'runner_label="ubuntu-latest"' >> $GITHUB_OUTPUT

Why This Works:

• fromJSON() parses the string as JSON
• Arrays become multiple label requirements (AND logic)
• Quoted strings become single label requirements
• GitHub correctly routes to matching runners

---

Runner Tag System

The CI system supports these commit message or PR title tags:

Tag	Effect	Example Label Output
[github]	Force GitHub-hosted runner	"ubuntu-latest"
[windows]	Force Windows self-hosted	["self-hosted", "Windows"]
[macos]	Force macOS self-hosted	["self-hosted", "macOS"]
[self-hosted]	Force generic self-hosted	"self-hosted"
[skip-ci] / [ci-skip]	Skip CI entirely	N/A
(no tag)	Day-based auto-selection	Varies

Auto Mode (Day-Based Selection)

When no tag is specified, the system uses day-of-month heuristic:

DAY=$(date +%-d)
if [[ $DAY -ge 26 ]]; then
  # End of month - use GitHub-hosted (quota resets soon)
  RUNNER="ubuntu-latest"
else
  # Active development - use self-hosted (fast, no quota cost)
  RUNNER="self-hosted"
fi

Tag Precedence

1. Commit message (highest priority)
2. PR title
3. Manual input (workflow_dispatch)
4. Day-based default (lowest priority)

---

Complete decide_runner Implementation

This is the canonical pattern after PR #298 fixes:

jobs:
  decide_runner:
    runs-on: ubuntu-latest
    outputs:
      runner_label: ${{ steps.decision.outputs.runner_label }}
      should_skip: ${{ steps.decision.outputs.should_skip }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v5
        with:
          # Critical: Checkout PR head commit, not merge commit
          ref: ${{ github.event.pull_request.head.sha || github.sha }}
          fetch-depth: 1
 
      - name: Determine runner
        id: decision
        shell: bash
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          # Get actual commit message (not merge commit)
          COMMIT_MSG=$(git log -1 --pretty=%B HEAD)
 
          # Get PR title for fallback
          PR_NUMBER="${{ github.event.pull_request.number }}"
          PR_TITLE=""
          if [[ -n "$PR_NUMBER" ]]; then
            PR_TITLE=$(gh pr view "$PR_NUMBER" --json title --jq '.title' 2>/dev/null || echo "")
          fi
 
          INPUT_MODE="${{ github.event.inputs.runner_mode || 'auto' }}"
          RUNNER="ubuntu-latest"
 
          echo "::notice::Commit message: $COMMIT_MSG"
          echo "::notice::PR title: $PR_TITLE"
 
          # Check for skip flags
          if [[ "$COMMIT_MSG" =~ \[skip-ci\]|\[ci-skip\] ]] || [[ "$PR_TITLE" =~ \[skip-ci\]|\[ci-skip\] ]] || [[ "$INPUT_MODE" == "skip" ]]; then
            echo "should_skip=true" >> $GITHUB_OUTPUT
            echo "runner_label=ubuntu-latest" >> $GITHUB_OUTPUT
            exit 0
          fi
 
          # Priority: Commit message > PR title > Manual input > Day-based default
          if [[ "$COMMIT_MSG" =~ \[self-hosted\] ]]; then
            RUNNER="self-hosted"
          elif [[ "$COMMIT_MSG" =~ \[github\] ]]; then
            RUNNER="ubuntu-latest"
          elif [[ "$COMMIT_MSG" =~ \[windows\] ]]; then
            RUNNER='["self-hosted", "Windows"]'
          elif [[ "$COMMIT_MSG" =~ \[macos\] ]]; then
            RUNNER='["self-hosted", "macOS"]'
          elif [[ "$PR_TITLE" =~ \[self-hosted\] ]]; then
            RUNNER="self-hosted"
          elif [[ "$PR_TITLE" =~ \[github\] ]]; then
            RUNNER="ubuntu-latest"
          elif [[ "$INPUT_MODE" == "self-hosted" ]]; then
            RUNNER="self-hosted"
          elif [[ "$INPUT_MODE" == "github" ]]; then
            RUNNER="ubuntu-latest"
          else
            # Day-based default
            DAY=$(date +%-d)
            if [[ $DAY -ge 26 ]]; then
              RUNNER="ubuntu-latest"
            else
              RUNNER="self-hosted"
            fi
          fi
 
          echo "should_skip=false" >> $GITHUB_OUTPUT
          echo "runner_label=$RUNNER" >> $GITHUB_OUTPUT
          echo "::notice::Selected runner: $RUNNER"
 
  build:
    runs-on: ${{ fromJSON(needs.decide_runner.outputs.runner_label) }}
    needs: decide_runner
    if: needs.decide_runner.outputs.should_skip != 'true'
    steps:
      # Your build steps here...

---

Cross-Platform Considerations

When using self-hosted runners, your workflow steps must handle different platforms:

# Windows runners can't use WSL bash (when running as SYSTEM service)
# Must use PowerShell for Windows, bash for macOS/Linux
 
- name: Do something (Windows)
  if: runner.os == 'Windows'
  shell: powershell
  run: |
    # PowerShell commands
 
- name: Do something (macOS/Linux)
  if: runner.os != 'Windows'
  shell: bash
  run: |
    # Bash commands

See Cross-Platform Runner Patterns for complete platform handling.

---

Special Character Warning

Commit messages with special characters can break PowerShell parsing:

# ⚠️ WARNING: These characters in commit messages cause issues
# Brackets: [ ]
# Quotes: " '
# Backticks: `
# Dollar signs: $
 
# Example problematic commit message:
# "Added ["self-hosted", "macOS"] support"
# This breaks when interpolated via ${{ github.event.head_commit.message }}

Mitigation: Read commit message via git log after checkout, not from event context.

---

Troubleshooting

Jobs Queue Forever

Symptom: Job shows “Waiting for a runner to pick up this job” for 1+ hours.

Cause: Runner label mismatch - no runner matches the requested labels.

Debug Steps:

1. Check decide_runner job logs for “Selected runner:” notice
2. Verify runner label format (should be JSON for multi-label)
3. Check GitHub Settings → Actions → Runners for available runners and their labels

Tags Ignored on PRs

Symptom: [github] or [macos] tag in commit message has no effect.

Cause: Reading merge commit message instead of PR head commit.

Solution: Ensure checkout step uses ref: ${{ github.event.pull_request.head.sha || github.sha }}

Wrong Runner Selected

Symptom: Jobs run on unexpected runner despite using tags.

Debug Steps:

1. Check which message is being read (commit vs PR title)
2. Verify tag format (must be exactly [github], not github or [ github ])
3. Check precedence order (commit message > PR title > input > default)

---

Debugging Timeline (PR #298)

For context on how these issues were discovered:

1. Initial symptom: Jobs queued for 1+ hour waiting for runners
2. First discovery: Runner labels showing as self-hosted,macOS (single label)
3. Root cause 1: GitHub expects array, not comma-separated string
4. Applied fix: JSON array format + fromJSON()
5. Second symptom: Tags in commit messages still being ignored
6. Root cause 2: github.event.head_commit.message returns merge commit for PRs
7. Applied fix: Checkout PR head SHA, read message via git log
8. Verification: Pushed test commits with [github], [macos], [windows] tags
9. Result: All tags correctly detected and route to proper runners

---

Affected Workflows

These workflows use the runner selection system:

Workflow	File	Labels Used
Android CI	android-ci.yml	ubuntu-latest, self-hosted
Deploy to Play Store	deploy-to-play-store.yml	ubuntu-latest, self-hosted
Docs Check	docs-only-check.yml	ubuntu-latest

---

Related Documentation

• Cross-Platform Runner Patterns - Platform-specific step handling
• Hybrid Runner System - Original quota management architecture
• CD Troubleshooting Guide - General troubleshooting
• Billing Block Incident - Why self-hosted first

---

Tags: ci-cd github-actions runner-selection debugging Status: Active in production Incident Reference: PR #298 debugging session