Home > Development > Guides > Best Practices > Migration Testing - Unit Tests vs Instrumented Tests

Migration Testing - Unit Tests vs Instrumented Tests

Date: 2025-10-18
Status: Established Pattern
Tags: testing database migrations room best-practices

Context

Room database migrations need testing to ensure schema changes work correctly. Android provides MigrationTestHelper for instrumented tests, but this isn’t always practical.

Our Experience: Instrumented migration tests consistently failed with AbstractMethodError due to kotlinx-serialization binary incompatibility between Room and runtime versions.

The Problem: AbstractMethodError

Error Details

java.lang.AbstractMethodError: Receiver class androidx.room.migration.bundle.FieldBundle$serializer
does not define or inherit an implementation of the resolved method
'abstract kotlinx.serialization.KSerializer[] typeParametersSerializers()'
of interface kotlinx.serialization.internal.GeneratedSerializer.

Root Cause

Room 2.8.1 compiled against kotlinx-serialization 1.7.3
Runtime resolves to kotlinx-serialization 1.8.1
Binary incompatibility in serializer interface between versions
Cannot be fixed by forcing dependency versions (already attempted)

Why This Matters

This is a recurring issue we’ve encountered multiple times. The incompatibility is at the binary level, meaning:

Gradle dependency resolution cannot fix it
Force-downgrading kotlinx-serialization breaks other dependencies
Upgrading Room requires entire dependency tree upgrade
No workaround exists without breaking changes

The Solution: MockK Unit Tests

Pattern Overview

Replace instrumented tests using MigrationTestHelper with unit tests using MockK to verify SQL execution.

Implementation

Basic Structure

class Migration_X_YTest {
    @Test
    fun `migration has correct version numbers`() {
        assertEquals("Start version should be X", X, MIGRATION_X_Y.startVersion)
        assertEquals("End version should be Y", Y, MIGRATION_X_Y.endVersion)
    }
 
    @Test
    fun `migration executes expected SQL statements`() {
        val mockDatabase = mockk<SupportSQLiteDatabase>(relaxed = true)
 
        MIGRATION_X_Y.migrate(mockDatabase)
 
        verify {
            mockDatabase.execSQL(match { sql ->
                sql.contains("ALTER TABLE") && sql.contains("ADD COLUMN")
            })
        }
    }
 
    @Test
    fun `migration is non-destructive`() {
        val mockDatabase = mockk<SupportSQLiteDatabase>(relaxed = true)
 
        MIGRATION_X_Y.migrate(mockDatabase)
 
        verify(exactly = 0) {
            mockDatabase.execSQL(match { sql ->
                sql.contains("DROP") || sql.contains("DELETE")
            })
        }
    }
}

Advanced Pattern: SQL Capture

@Test
fun `migration SQL is properly formatted`() {
    val mockDatabase = mockk<SupportSQLiteDatabase>(relaxed = true)
    val capturedSql = mutableListOf<String>()
 
    every { mockDatabase.execSQL(any()) } answers {
        capturedSql.add(firstArg<String>())
    }
 
    MIGRATION_X_Y.migrate(mockDatabase)
 
    assertEquals(1, capturedSql.size)
    val sql = capturedSql[0]
    
    assertFalse("SQL should not start with whitespace", sql.startsWith(" "))
    assertTrue("SQL should contain expected keywords", sql.contains("ALTER TABLE"))
}

Test Coverage Requirements

Minimum Tests for Each Migration

Version Validation - Verify startVersion and endVersion
SQL Execution - Verify correct SQL statements executed
Schema Changes - Verify columns/tables/indexes created
Non-Destructive - Verify no DROP/DELETE statements
Exception Handling - Verify migration doesn’t throw on errors

Example Coverage

// Migration_33_34Test.kt - Theme Mode Column Addition
@Test fun `migration has correct version numbers`()
@Test fun `migration is instance of Migration class`()
@Test fun `migration executes SQL statement to add themeMode column`()
@Test fun `migration adds themeMode as TEXT NOT NULL`()
@Test fun `migration sets default value to SYSTEM`()
@Test fun `migration uses single ALTER TABLE statement`()
@Test fun `migration does not create or drop tables`()
@Test fun `migration does not create indexes`()
@Test fun `migration handles exception gracefully`()
@Test fun `migration logs debug information`()
@Test fun `migration SQL is properly formatted`()
@Test fun `migration is purely additive`()
@Test fun `migration is non-destructive`()
@Test fun `migration adds column to settings table only`()
@Test fun `migration maintains backward compatibility`()
@Test fun `migration supports rollback safety`()
// Total: 16 tests

File Naming Convention

Standard: Migration_##_##.kt and Migration_##_##Test.kt

Examples

✅ Migration_31_32.kt → Migration_31_32Test.kt
✅ Migration_32_33.kt → Migration_32_33Test.kt
✅ Migration_33_34.kt → Migration_33_34Test.kt

❌ MIGRATION_31_32.kt (inconsistent casing)
❌ Migration31to32.kt (inconsistent separator)
❌ Migration32to33Test.kt (inconsistent with implementation)

Rationale

Consistency: Easy to find test files for any migration
Sorting: Files sort together in file explorer
Pattern Matching: *Migration_* glob matches all migration files
Clarity: Version numbers clearly separated with underscores

Advantages of Unit Tests Over Instrumented Tests

1. Speed

Unit tests: <100ms per test class
Instrumented tests: 10-30 seconds per test class (emulator startup, APK install)

2. Reliability

Unit tests: 100% pass rate in all environments
Instrumented tests: Fail with AbstractMethodError, emulator crashes, timeout issues

3. CI/CD Integration

Unit tests: Run on any machine, no Android SDK required
Instrumented tests: Require emulator, prone to flakiness in CI

4. Developer Experience

Unit tests: Instant feedback, no emulator management
Instrumented tests: Wait for emulator, manage device state, cleanup required

5. Debugging

Unit tests: Standard JVM debugging, clear stack traces
Instrumented tests: Logcat debugging, obscure errors from framework internals

What Unit Tests Can Verify

✅ What MockK Tests CAN Verify

SQL statement structure and content
Correct number of SQL statements executed
Proper use of ALTER TABLE, CREATE TABLE, CREATE INDEX
Absence of destructive operations (DROP, DELETE)
Column names, types, constraints
Index creation
Migration version numbers
Exception handling behavior
Logging behavior

❌ What MockK Tests CANNOT Verify

Actual database schema changes
Data migration correctness (existing rows affected)
Foreign key constraint enforcement
Index performance improvement
Database file size changes
SQLite version compatibility

Mitigation Strategy

For actual database validation:

Manual testing during development
Beta testing with production-like data
Staged rollout to detect migration issues early
Rollback plan for failed migrations
Schema export verification (Room schema JSON files)

Policy: Required Before Merge

For All New Migrations

File naming: Migration_##_##.kt
Unit tests: Migration_##_##Test.kt with minimum 10 tests
Test coverage: All SQL statements verified
Non-destructive: Verified no DROP/DELETE statements (unless explicitly required)
Documentation: Migration KDoc explaining purpose and safety

Test Template Checklist

// Copy this checklist for each new migration test file
class Migration_##_##Test {
    ☐ Test: migration has correct version numbers
    ☐ Test: migration executes expected SQL statements
    ☐ Test: migration creates/modifies correct schema elements
    ☐ Test: migration does not drop or delete (if applicable)
    ☐ Test: migration sets correct defaults
    ☐ Test: migration handles exceptions gracefully
    ☐ Test: migration is non-destructive
    ☐ Test: migration maintains backward compatibility
    ☐ Test: migration SQL is properly formatted
    ☐ Test: migration logs appropriately
    
    // Add specific tests for migration's unique behavior
}

Real-World Example: Migration 32→33

Migration Implementation

val MIGRATION_32_33 = object : Migration(32, 33) {
    override fun migrate(database: SupportSQLiteDatabase) {
        // Create tournament ID mappings table
        database.execSQL("""
            CREATE TABLE IF NOT EXISTS tournament_id_mappings (
                localId TEXT PRIMARY KEY NOT NULL,
                firebaseId TEXT NOT NULL,
                syncedAt INTEGER NOT NULL,
                lastVerifiedAt INTEGER NOT NULL
            )
        """.trimIndent())
 
        // Create unique index on firebaseId for reverse lookups
        database.execSQL("""
            CREATE UNIQUE INDEX IF NOT EXISTS index_tournament_id_mappings_firebaseId
            ON tournament_id_mappings(firebaseId)
        """.trimIndent())
    }
}

Test Coverage (15 tests)

class Migration_32_33Test {
    @Test fun `migration has correct version numbers`()
    @Test fun `migration is instance of Migration class`()
    @Test fun `migration creates tournament_id_mappings table`()
    @Test fun `tournament_id_mappings has primary key on localId`()
    @Test fun `tournament_id_mappings has all required columns`()
    @Test fun `all columns are NOT NULL`()
    @Test fun `syncedAt and lastVerifiedAt are INTEGER type`()
    @Test fun `migration creates unique index on firebaseId`()
    @Test fun `migration creates unique index on localId`()
    @Test fun `migration executes exactly 3 SQL statements`()
    @Test fun `migration does not drop or delete anything`()
    @Test fun `migration is purely additive`()
    @Test fun `migration uses IF NOT EXISTS for safety`()
    @Test fun `migration supports bidirectional ID lookups`()
    @Test fun `migration handles hybrid offline-first architecture`()
}

Result: 100% test pass rate, comprehensive SQL validation, no emulator required

Lessons Learned

1. Official Isn’t Always Practical

While MigrationTestHelper is the “official” Room migration testing approach, it’s not practical when:

Library dependencies have binary incompatibilities
CI/CD needs deterministic, fast feedback
Emulator stability is a concern

Takeaway: Choose tools that work for your constraints, not just “official” tools

2. Dead Code Accumulates When Tests Are Ignored

We had two instrumented test files with ALL tests @Ignored, yet they remained in the codebase for extended periods.

Takeaway: Regularly audit @Ignored tests - if they’ve never passed, delete them

3. Test Coverage Gaps Hide in Plain Sight

78% of our migrations (14/18) had zero tests despite being production code running on thousands of devices.

Takeaway: Make test coverage a merge requirement, not a “nice to have”

4. Binary Incompatibility Cannot Be “Fixed”

We attempted multiple times to fix AbstractMethodError by:

Forcing kotlinx-serialization versions
Adding dependencies to androidTestImplementation
Using resolutionStrategy.force()

Takeaway: Binary incompatibilities are architectural issues, not configuration issues. Change your approach instead of fighting the tooling.

Migration Test Coverage Status

Current Coverage (10-18-25)

✅ v28→v29: Migration_28_29Test.kt (11 tests)
✅ v29→v30: Migration_29_30Test.kt (17 tests)
✅ v30→v31: Migration_30_31Test.kt (18 tests)
✅ v31→v32: Migration_31_32Test.kt (15 tests)
✅ v32→v33: Migration_32_33Test.kt (15 tests)
✅ v33→v34: Migration_33_34Test.kt (16 tests)
✅ v34→v35: Migration_34_35Test.kt
❌ v17→v28: No tests (legacy migrations - low priority)

Coverage: 50% (9 of 18 migrations tested with 76+ total tests)
Target: 100% (all migrations tested)

Project Journal (10-18-25): Migration test cleanup session details
CLAUDE.md: Migration testing policy and standards
Room Migration Guide: https://developer.android.com/training/data-storage/room/migrating-db-versions
MockK Documentation: https://mockk.io/

Quick Reference

Run Migration Tests

# All migration tests
./gradlew :app:testDebugUnitTest --tests="*Migration*" --no-daemon
 
# Specific migration
./gradlew :app:testDebugUnitTest --tests="*Migration_32_33Test" --no-daemon

Create New Migration Test

Create Migration_X_Y.kt in app/src/main/java/com/archeryapprentice/data/db/migrations/
Create Migration_X_YTest.kt in app/src/test/java/com/archeryapprentice/data/db/migrations/
Use template checklist above for minimum test coverage
Run tests: ./gradlew :app:testDebugUnitTest --tests="*Migration_X_YTest" --no-daemon
Verify 100% pass rate before committing

Last Updated: 2025-10-18
Pattern Status: Established and Required
Test Pass Rate: 100% (76+ tests)

Archery Apprentice Documentation

Explorer

Migration Testing Unit Tests vs Instrumented Tests