Restore

Use this when a release migrated your database and you need both the previous release and the database as it was before the migration, for example because the migrated data is damaged. It costs data and downtime, so first check whether you need it:

• A failed update doesn't need a restore. CloudFormation rolls it back by itself; see Updates.
• A release that crossed no migration is undone by updating the stack to the previous release's template, without a restore.

Before you startCopy Anchor Link

What you lose: everything written between the upgrade's restart and the moment you stop the service in step 1. Nothing written before the upgrade is lost: the snapshot was taken while the service was stopped.

How long it takes: in our tests, about 25 minutes of work for AWS (most of it restoring a Multi-AZ instance from the snapshot), plus your time between steps. The console is down throughout.

You need:

• The snapshot. Every migration-crossing update takes one, named tracemill-<short id>-<version>-<attempt>-<hash>. The short id is the first block of the DeploymentId output. List them:

  Copy Text

  aws rds describe-db-snapshots --snapshot-type manual \
    --query "DBSnapshots[?starts_with(DBSnapshotIdentifier, 'tracemill-<short id>-')].[DBSnapshotIdentifier,SnapshotCreateTime]"

  Pick the one taken by the upgrade you're undoing. Its name carries that release's version and attempt.

• The previous release's template URL: https://tracemill-releases-prod-us-east-1.s3.us-east-1.amazonaws.com/releases/<previous version>/tracemill-byoc.yaml.

• AWS credentials allowed to update the stack, with AWS_REGION set to the stack's Region.

• The AWS CLI and jq (step 4 uses it to build the parameter list).

Below, <stack> is the stack name, <id> the short id, and the database is tracemill-<id>-db, the one the stack created. If you launched with an existing database, use its identifier and its own settings instead.

Turn auto-apply off first if AutoApply is enabled. Otherwise its next window reinstalls the ring's head, which may be the very release you're undoing. Update the stack with Use existing template, AutoApply set to disabled and every other parameter unchanged, and wait for UPDATE_COMPLETE. Step 4 keeps it disabled. Turn it back on only once the ring no longer offers the release you restored away from.

Don't start a restore while another stack update is in progress.

Don't change or rotate any of the stack's secrets during a restore. The restored database needs the same data-encryption key to read its encrypted fields.

Step 1: Stop the serviceCopy Anchor Link

Scale the service to zero and wait until its tasks have actually stopped. The service's running count reaches zero before its tasks finish stopping, so wait on the tasks themselves.

T=$(aws ecs list-tasks --cluster <stack> --service-name <stack>-server --query taskArns --output text)
aws ecs update-service --cluster <stack> --service <stack>-server --desired-count 0
aws ecs wait tasks-stopped --cluster <stack> --tasks $T

About a minute and a half in our tests. Writes stop here.

Step 2: Move the current database asideCopy Anchor Link

Rename it, so the restored instance can take its identifier and endpoint:

aws rds modify-db-instance --db-instance-identifier tracemill-<id>-db \
  --new-db-instance-identifier tracemill-<id>-db-old --apply-immediately

The original identifier is free about a minute and a half later: wait until aws rds describe-db-instances --db-instance-identifier tracemill-<id>-db fails with DBInstanceNotFound. Keep the old instance until you're satisfied with the restore.

Step 3: Restore the snapshot under the original identifierCopy Anchor Link

Restore with the old instance's settings. Read them first:

aws rds describe-db-instances --db-instance-identifier tracemill-<id>-db-old \
  --query 'DBInstances[0].{Class:DBInstanceClass,MultiAZ:MultiAZ,SubnetGroup:DBSubnetGroup.DBSubnetGroupName,SecurityGroups:VpcSecurityGroups[].VpcSecurityGroupId,ParameterGroup:DBParameterGroups[0].DBParameterGroupName,StorageType:StorageType,AutoMinorVersionUpgrade:AutoMinorVersionUpgrade,BackupRetentionPeriod:BackupRetentionPeriod}'

The command below matches what the stack creates (gp3 storage, automatic minor upgrades, log export, deletion protection). If you launched with an existing database, use your instance's own values instead.

Then restore:

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier tracemill-<id>-db \
  --db-snapshot-identifier <snapshot> \
  --db-instance-class <Class> \
  --multi-az \
  --db-subnet-group-name <SubnetGroup> \
  --vpc-security-group-ids <SecurityGroups> \
  --db-parameter-group-name <ParameterGroup> \
  --storage-type gp3 \
  --copy-tags-to-snapshot \
  --deletion-protection \
  --enable-cloudwatch-logs-exports postgresql \
  --auto-minor-version-upgrade \
  --no-publicly-accessible
aws rds wait db-instance-available --db-instance-identifier tracemill-<id>-db

Use --no-multi-az instead of --multi-az if MultiAZ was false.

This took about 16.5 minutes for a Multi-AZ instance in our tests. The instance reports available a couple of minutes before its standby exists. The endpoint is the same as before, so nothing else needs to change. The restored instance keeps the snapshot's KMS key and master password. In our tests it also kept the backup retention period; check that BackupRetentionPeriod matches the old instance, and if it doesn't, set it back:

aws rds modify-db-instance --db-instance-identifier tracemill-<id>-db \
  --backup-retention-period <BackupRetentionPeriod> --apply-immediately

Step 4: Update the stack to the previous releaseCopy Anchor Link

Update the stack to the previous release's template, with OperationAttempt set to a new value starting with restore- and every other parameter at its previous value. The restore- prefix tells the stack to trust the restored database over its own records of what's installed.

aws cloudformation describe-stacks --stack-name <stack> \
  --query 'Stacks[0].Parameters[].ParameterKey' --output json \
  | jq '[.[] | if . == "OperationAttempt"
               then {ParameterKey: ., ParameterValue: "restore-1"}
               else {ParameterKey: ., UsePreviousValue: true} end]' > restore-params.json
aws cloudformation update-stack --stack-name <stack> \
  --template-url <previous release template URL> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
  --parameters file://restore-params.json
aws cloudformation wait stack-update-complete --stack-name <stack>

Pass every parameter: an update that omits one resets it to the template default.

Keep rollback on. Never pass --disable-rollback: CloudFormation refuses it for a release change, and the failed attempt leaves the stack's Lambda functions on the wrong release's code.

The update stops the service, takes another snapshot (of the restored database), runs the previous release's transition, then starts the previous release. About five minutes in our tests, with the console back after four. While ECS replaces the deployment, it may briefly start a task of the release you're leaving; that task refuses to start on the older schema ("database schema N is below this binary's head M"). That's expected and doesn't fail the update.

Step 5: VerifyCopy Anchor Link

• The stack is UPDATE_COMPLETE.
• https://<ConsoleHostname>/readyz returns 200.
• Settings → Updates shows the previous version and schema, with the restore- attempt as the last operation.

Step 6: Clean upCopy Anchor Link

Once you're satisfied, delete the old instance. It has deletion protection on, and you decide whether to keep a final snapshot of it:

aws rds modify-db-instance --db-instance-identifier tracemill-<id>-db-old \
  --no-deletion-protection --apply-immediately
aws rds delete-db-instance --db-instance-identifier tracemill-<id>-db-old \
  --final-db-snapshot-identifier tracemill-<id>-db-old-final   # or --skip-final-snapshot
aws rds wait db-instance-deleted --db-instance-identifier tracemill-<id>-db-old
aws logs delete-log-group --log-group-name /aws/rds/instance/tracemill-<id>-db-old/postgresql

RDS creates that log group itself and keeps writing to it until the instance is gone, so delete it only after the wait. It isn't part of the stack, so nothing else deletes it.

If the restore update failsCopy Anchor Link

Its transition failsCopy Anchor Link

For example, the database wasn't restored first, so the previous release refuses the newer schema. CloudFormation rolls the update back and ends at UPDATE_ROLLBACK_COMPLETE with the service left stopped. You get one alert ("release rolled back; service not restored"). It took about seven minutes in our tests.

Fix the cause and repeat step 4 with a new attempt, such as restore-2. If instead you want the service back on the current release and database, scale it back up:

aws ecs update-service --cluster <stack> --service <stack>-server --desired-count <DesiredCount>

The restored release doesn't startCopy Anchor Link

If the previous release's tasks never become healthy, CloudFormation rolls back, but its rollback of the service redeploys the newer release onto the restored schema, which refuses to start. The rollback itself then fails: the stack ends at UPDATE_ROLLBACK_FAILED with the service stopped (about 13.5 minutes in our tests). Expect up to three alerts.

Finish the rollback without the service:

aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip Service
aws cloudformation wait stack-rollback-complete --stack-name <stack>

Then fix the cause and repeat step 4 with a new attempt.