Updates

A self-hosted deployment receives two kinds of update:

• Detection content (scenarios and the detection library) arrives automatically, with no stack update and no outage.
• Releases of the Tracemill server and stack change the software and sometimes the database schema. A release goes in only through a stack update, which you start or which auto-apply starts on your schedule.

Both come from your release ring: Ring is general by default, early for design partners.

Content updatesCopy Anchor Link

The server checks the ring's content feed every five minutes and moves to new content releases on its own. Settings → Content shows the content release you run, lets you turn automatic updates off and on, and lists updates that failed. A content release that needs a newer server than yours waits until you apply a release.

Release updatesCopy Anchor Link

The server checks the ring's signed release manifest every six hours. Settings → Updates shows the installed release, the ring's head release and its security floor, and what you can do:

Check now fetches the manifest immediately. The manifest is signed, and the server ignores one whose signature doesn't verify or whose serial number is lower than one it has already seen.

Apply a releaseCopy Anchor Link

From Settings → Updates:

1. Sign in to the AWS console in the stack's account. A link opened in another account fails.
2. Choose Open stack update in the AWS console. The release's template URL is filled in for you.
3. Parameters: set OperationAttempt to the attempt id the page shows, and leave every other parameter unchanged.
4. Stack options: leave Express mode unchecked, set Behavior on provisioning failure to Roll back all resources, and set Delete newly created resources during a rollback to Use deletion policy.
5. Acknowledge that the stack creates IAM resources, and submit.

The page also shows the equivalent aws cloudformation update-stack command. It sets the attempt id and keeps every other parameter at its previous value.

The console never blocks or gates an update; you decide when it runs.

Never use --disable-rollback (in the console, Preserve successfully provisioned resources) for a release update. CloudFormation refuses it: replacing the task definitions isn't supported with rollback disabled, so the update fails before any transition runs. Worse, by then the stack's Lambda functions are already on the new release's code, and rolling the stack back doesn't restore them. Keep rollback on.

What an update doesCopy Anchor Link

A newer release with no database migration rolls out like any ECS service update: new tasks start, and the old ones stop once the new ones are healthy. There's no outage.

Going back to an older release (the page's "Update to an older version") needs no migration, but the stack still stops the service and takes a snapshot first, so nothing running can depend on the release being withdrawn. Expect a few minutes of outage, as below.

A release that crosses a database migration runs a transition first:

1. The stack scales the service to zero and waits until every task has stopped.
2. It takes an RDS snapshot, named tracemill-<short id>-<version>-<attempt>-<hash> (the LastSnapshotNamePattern output) and tagged with the operation.
3. A one-off ECS task from the new release migrates the database.
4. The new release starts.

The console is down from step 1 until the new release is healthy: about five minutes on a small database in our tests. Runs in progress pause and resume afterwards. While ECS swaps deployments, the previous release may run for a minute or two against the migrated schema; migrations are additive, so it keeps working.

TransitionTimeoutSeconds (default 3300) bounds steps 1 to 3. CloudFormation caps it at one hour, so a transition that needs longer can't run. On a small database the whole transition took under four minutes; the snapshot is the step that grows with database size.

Each release change carries an attempt id, OperationAttempt. The Updates page generates a fresh one each time. Never reuse one: the stack treats the same release and attempt as the same operation and won't run it again.

SnapshotsCopy Anchor Link

Every migration-crossing update leaves a snapshot. A daily housekeeping job deletes this stack's transition snapshots older than SnapshotRetentionDays (default 35; 0 keeps them forever), but always keeps the newest one. It never touches snapshots it didn't create. To go back to one, see Restore.

When a release failsCopy Anchor Link

The transition failsCopy Anchor Link

If the migration fails or the new release refuses the database, the stack update fails and CloudFormation rolls back to the release you ran before. The rollback normally brings the service back up. When it can't do that safely, it leaves the service stopped, emails AlertEmail ("release rolled back; service not restored"), and the Updates page says The service was not restored after the rollback. Once no transition task is running, retry the release with a new attempt id, or set the service's desired count back from the ECS console.

The new release doesn't startCopy Anchor Link

If the new release's tasks never become healthy, ECS's deployment circuit breaker fails the deployment, which fails the stack update. CloudFormation rolls back to the release you ran before, which runs on the database as the new release left it. Detecting a release that can't start took about 15 minutes in our tests.

You get an email ("release failed to start and is rolling back") with the command to retry, and the Updates page shows The last release did not complete. Your options:

• Retry the same release with a new attempt id, from the Retry the last release card or the emailed command.
• Apply a fixed release once Tracemill publishes one.
• Restore the snapshot taken before the migration, if the database itself is damaged. See Restore.

If the console is downCopy Anchor Link

The alert email carries the retry command. You can also work from AWS directly. The prefix is the TransitionStatePrefix output.

# The installed release and the last operation
aws ssm get-parameter --name <TransitionStatePrefix>/installed \
  --query Parameter.Value --output text

# The releases your ring offers, with their template URLs
curl -s <ContentFeedBaseUrl>/rings/<Ring>/image.json | python3 -c '
import base64, json, sys
p = json.load(sys.stdin)["payload"]
print(json.dumps(json.loads(base64.urlsafe_b64decode(p + "=" * (-len(p) % 4))), indent=2))'

# Pre-migration snapshots
aws rds describe-db-snapshots --snapshot-type manual \
  --query "DBSnapshots[?starts_with(DBSnapshotIdentifier, 'tracemill-<short id>-')].[DBSnapshotIdentifier,SnapshotCreateTime]"

A release's template URL has the form https://tracemill-releases-prod-us-east-1.s3.us-east-1.amazonaws.com/releases/<version>/tracemill-byoc.yaml, whichever Region the stack runs in.

Auto-applyCopy Anchor Link

With AutoApply set to enabled, the stack applies the ring's head release on AutoApplySchedule (default cron(0 3 ? * SUN *), Sundays at 03:00 UTC). At each window it:

• does nothing unless the stack is in a completed, non-rollback state and the last transition completed; a failed update is a person's call;
• verifies the manifest's signature and serial, and checks the release's template against the digest in the manifest before using it;
• waits while runs are in progress, re-checking every ten minutes, for up to AutoApplyMaxDeferMinutes (default 120), then proceeds anyway; runs in progress pause and resume;
• updates the stack with a new attempt id (auto-<timestamp>) and every other parameter unchanged, and emails AlertEmail that it's applying the release.

Auto-apply updates the stack only through a role you create and pass as AutoApplyRoleArn. The stack holds no role of its own that can update it.

The role must trust the stack's auto-apply function role, <stack>-auto-apply. That role exists only once auto-apply is on, so trust it by condition rather than by name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<account>:root" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "ArnEquals": { "aws:PrincipalArn": "arn:aws:iam::<account>:role/<stack>-auto-apply" }
      }
    }
  ]
}

Its permissions must cover a full update of this stack: cloudformation:DescribeStacks and cloudformation:UpdateStack on the stack, plus every change CloudFormation makes to the stack's resources during a release update, the same rights the person who launched the stack needed. The server's own task role is explicitly denied from assuming this role.

Auto-apply reserves one unit of Lambda concurrency in your account. If enabling it fails on the auto-apply function, check your account's Lambda concurrency quota.