fix: disable HA and PITR on target instance during migration #294
Conversation
DMS migration fails when the source instance has High Availability or Point-in-Time Recovery enabled, because these settings are copied to the target via DeepCopy() in DefineInstance(). The target is then demoted to a read replica by DMS, which is incompatible with HA and PITR. Changes: - Strip HA and PITR from helper app spec in CreateInstance() - Disable PITR and set availability to ZONAL in PrepareTargetInstance() - Add warnings in ValidateSourceInstance() about temporarily disabled features After promotion, HA and PITR are automatically re-enabled when UpdateApplicationInstance() applies the original app spec (which still has these settings) via naiserator reconciliation. Fixes #240
Instead of relying on implicit re-enabling through naiserator reconciliation, explicitly restore AvailabilityType, backup and PITR settings from the source instance onto the target CNRM spec in UpdateTargetInstanceAfterPromotion(). This makes the restore visible in logs and less fragile.
The Cloud SQL Admin API returns databaseRoles in GET responses but rejects them in Update requests with 'Invalid request to update database roles'. Clear the field before sending the update.
Jeg lurer på om problemet kommer etter at man har kjørt Jeg kjørte akkurat en test på å migrere noen databaser hvor jeg hadde skrudd på audit i henhold til vår dokumentasjon, og alle tre feilet på samme sted. Migreringsjobben på Google feiler ved oppstart med en Internal error og anbefaler å kontakte support. Jeg lar test-apper og databaser stå slik de er nå dersom noen ønsker å undersøke nærmere, så skal jeg prøve ( 😅 ) å huske å slette dem etter vinterferien 😄 Testapper/databaser i
|
2 participants
Problem
Migration fails when the source Cloud SQL instance has any of these features enabled (#240):
Root cause
DefineInstance()usesDeepCopy()to copy the source instance spec to the target. This copies all settings, including HA and PITR. The target instance is then created with these settings via the helper app.During setup, DMS calls
demoteTargetInstance()to convert the target to a read replica. This fails because:Fix
Changes in
instance.go:CreateInstance()– After callingDefineInstance(), strip HA and PITR from the helper app spec before creating the target instance. This prevents GCP from creating an instance with incompatible settings.PrepareTargetInstance()– Also explicitly disable PITR (pointInTimeRecoveryEnabled = false) and setavailabilityType = ZONALon the CNRM spec as a safety net.ValidateSourceInstance()– Log clear warnings when the source has HA or PITR enabled, explaining they will be temporarily disabled during migration.UpdateTargetInstanceAfterPromotion()– After DMS promote completes, explicitly restore HA (availabilityType=REGIONAL) and PITR from the source CNRM spec onto the target instance.Changes in
promote.go:UpdateTargetInstanceAfterPromotion()call to pass source instance for settings restoration.Changes in
database.go:SetDatabasePassword()– ClearDatabaseRolesfrom the user object before callingUsers.Update. The Cloud SQL Admin API returnsdatabaseRolesin GET responses but rejects them in Update requests with"Invalid request to update database roles". This is a pre-existing bug unrelated to HA/PITR but was discovered during testing.Re-enabling after migration
After promotion,
UpdateTargetInstanceAfterPromotion()explicitly restores HA and PITR settings by reading the source CNRM spec and applying them to the target. Additionally,UpdateApplicationInstance()applies the original app spec via naiserator reconciliation.Testing
Verified end-to-end with a real migration in
nav-dev/nais:Source instance:
migrator-test-ha-pitrTarget instance:
migrator-test-targetResults
What about audit logging?
The
cloudsql.enable_pgauditdatabase flag is copied viaDeepCopy()and was not stripped. Testing confirmed it does not break the DMS migration – the flag survived the full setup→promote cycle without issues.Fixes #240
/cc @mortenlj