+1 (726) 207-9872
SleekDeploy

When the Deploy Gets Stuck: CloudFormation Rollbacks, Drift, Stack Limits, and Resource Imports with Serverless Framework V4

Serverless Framework deploys are boring right up until the moment CloudFormation refuses to move. Then a team that has never read a stack event in anger is staring at UPDATE_ROLLBACK_FAILED on a production stack at 4 p.m. This is the playbook we hand to clients on project-rescue engagements: how to read the failure, how to recover each of the common stuck states, and how to stop the same class of failure from recurring.

Everything below assumes Serverless Framework V4 deploying to AWS CloudFormation, which is still what serverless deploy does under the hood.

First: read the actual error, not the CLI summary

serverless deploy prints the first failure reason it can find, but CloudFormation knows more. Pull the raw events, newest first:

aws cloudformation describe-stack-events \
  --stack-name my-service-prod \
  --max-items 40 \
  --query 'StackEvents[?ResourceStatus!=`UPDATE_COMPLETE`].[Timestamp,LogicalResourceId,ResourceStatus,ResourceStatusReason]' \
  --output table

Read from the bottom of the failure window upward. The first CREATE_FAILED or UPDATE_FAILED event is the real cause; everything above it is cascade noise from the rollback. Nine times out of ten the reason string is enough: an IAM permission, a name collision, a quota, or a resource that cannot be updated in place.

Also check the stack status itself, because the recovery path depends entirely on which state you are in:

aws cloudformation describe-stacks --stack-name my-service-prod \
  --query 'Stacks[0].[StackStatus,StackStatusReason]' --output text

The five states you will actually hit

1. ROLLBACK_COMPLETE on a brand-new stack

The very first deploy failed, so the stack exists but contains nothing usable. CloudFormation will not let you update it. There is only one fix:

serverless remove --stage dev
# or: aws cloudformation delete-stack --stack-name my-service-dev

Then deploy again. This is safe precisely because the stack never finished creating — there is no live data in it. Do not spend time trying to repair it.

2. UPDATE_ROLLBACK_FAILED

The deploy failed, CloudFormation tried to roll back to the last good template, and the rollback itself failed — usually because a resource was changed or deleted outside CloudFormation, or a Lambda/log group it wanted to restore no longer exists.

The stack is frozen: no deploys until you clear it. Two options.

Option A — continue the rollback after fixing the cause. If you can make reality match what the rollback expects (recreate the deleted log group, restore the IAM permission), then:

aws cloudformation continue-update-rollback --stack-name my-service-prod

Option B — skip the resources that cannot roll back. CloudFormation will mark them UPDATE_COMPLETE without touching them, leaving the stack consistent on paper:

aws cloudformation continue-update-rollback \
  --stack-name my-service-prod \
  --resources-to-skip ApiGatewayDeployment1700000000000 MyCustomResourceLambda

Skipping means CloudFormation's view of that resource may now be wrong. Write down what you skipped and reconcile it in the next deploy — a drift detection run (below) will tell you how far off you are. For nested stacks the logical ID is qualified: NestedStackName.LogicalId.

3. UPDATE_IN_PROGRESS that never ends

Almost always a custom resource whose Lambda never sent a response to the pre-signed S3 URL — a crash, a timeout, or a VPC with no route out to S3. CloudFormation waits the full hour.

Find the custom resource in the events, open its Lambda logs, and if it has already failed you can unblock the stack by signalling manually:

curl -X PUT -H 'Content-Type:' \
  -d '{"Status":"FAILED","Reason":"manual signal","PhysicalResourceId":"manual","StackId":"<StackId>","RequestId":"<RequestId>","LogicalResourceId":"<LogicalResourceId>"}' \
  "<PreSignedResponseURL from the custom resource request event>"

Then fix the function. The usual root causes: the handler throws before its try/finally sends a response, the function timeout is shorter than the work, or it was placed in private subnets without a NAT gateway or S3 endpoint.

4. DELETE_FAILED on serverless remove

Non-empty S3 buckets, ENIs still attached from VPC Lambdas, and log groups recreated by a still-running function are the top three. Empty the bucket, wait out the ENI detach (it can take 20 minutes), then retry. If one resource is genuinely stuck:

aws cloudformation delete-stack --stack-name my-service-dev \
  --retain-resources MyStubbornBucket

5. "No updates are to be performed" when you know there are changes

Your code changed but the template did not, and CloudFormation only sees the template. This happens when the S3 artifact key is identical. Force a rebuild:

serverless deploy --force

If the opposite happens — V4 reports changes on every deploy for a resource you never touch — check for a timestamped logical ID (API Gateway deployments) or a plugin injecting a value such as a build hash.

Drift: the slow version of the same problem

Most emergency deploy failures start weeks earlier, when somebody edits a resource in the console. CloudFormation does not notice until the next update tries to modify that resource and the real state does not match.

Detect it deliberately:

ID=$(aws cloudformation detect-stack-drift --stack-name my-service-prod \
     --query StackDriftDetectionId --output text)
aws cloudformation describe-stack-resource-drifts \
  --stack-name my-service-prod \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus]' --output table

Run this on a schedule — an EventBridge rule plus a small Lambda, or a weekly CI job — and alert on anything MODIFIED or DELETED. Fix drift by either reverting the console change or codifying it in serverless.yml. Both are fine; silently leaving it is not.

The structural fix is to remove console write access in production. Human IAM roles get read-only; everything that changes infrastructure arrives through the CI/CD pipeline with OIDC and a deploy role.

Preview the change before it lands

CloudFormation change sets show you exactly which resources will be modified and, critically, which will be replaced. Replacement of a DynamoDB table or an RDS instance means data loss. V4 can stop and show you:

provider:
  name: aws
  deploymentMethod: changesets   # review instead of blind apply

Or generate the artifacts in CI and inspect them before the apply step:

serverless package --stage prod
# review .serverless/cloudformation-template-update-stack.json

Watch for changes that force replacement: renaming a logical ID, changing a DynamoDB table's partition key or table name, changing an S3 bucket name, moving a resource between stacks. Any of these in a diff should block the deploy until a human has decided how to migrate the data.

Stack limits, and splitting before you hit them

A single CloudFormation stack allows 500 resources, 200 outputs and 200 parameters. Each Serverless function quietly costs you about five resources (function, version, log group, role statement and an event resource or two), so a service hits the ceiling somewhere around 60–80 functions. Options, in the order we recommend them:

  1. Split the service by bounded context and wire the pieces together with Serverless Framework Compose, passing outputs between services rather than sharing one giant stack.
  2. Move shared, long-lived infrastructure — VPCs, tables, buckets — into a separate stack or into Terraform, and reference it by SSM parameter or CloudFormation import.
  3. Trim what each function generates: disable versioning with versionFunctions: false where you do not need aliases, and define log groups centrally with a retention policy.

Splitting a stack that already exists is the painful case, which is where resource import comes in.

Importing existing resources instead of recreating them

If a table, bucket or user pool must move from one stack to another — or came into existence outside CloudFormation entirely — you can import it rather than recreate it.

  1. On the source stack, set a DeletionPolicy: Retain on the resource and deploy. Then remove the resource from the template and deploy again: CloudFormation stops managing it but leaves it alive.
  2. In the target service, declare the resource in resources.Resources with the identical physical name and DeletionPolicy: Retain.
  3. Package the target template and run a CloudFormation import change set:
serverless package --stage prod
aws cloudformation create-change-set \
  --stack-name target-service-prod \
  --change-set-name import-orders-table \
  --change-set-type IMPORT \
  --resources-to-import file://resources-to-import.json \
  --template-body file://.serverless/cloudformation-template-update-stack.json \
  --capabilities CAPABILITY_NAMED_IAM
aws cloudformation execute-change-set --stack-name target-service-prod \
  --change-set-name import-orders-table

Where resources-to-import.json maps logical to physical IDs:

[{ "ResourceType": "AWS::DynamoDB::Table",
   "LogicalResourceId": "OrdersTable",
   "ResourceIdentifier": { "TableName": "orders-prod" } }]

Rehearse this in a non-production account first, and take a backup (PITR snapshot, bucket replication) before you start. The import itself does not move data, but a mistake in step 1 can delete it.

Make the next failure cheaper

A few habits that turn deploy incidents into non-events:

  • Deploy to an ephemeral or staging stack first, from the same pipeline, with the same template. Most CloudFormation failures reproduce there.
  • Keep stateful resources out of the function stack. If tables and buckets live elsewhere, a broken function deploy can never threaten data.
  • Turn on DeletionPolicy: Retain and UpdateReplacePolicy: Retain for every production data store, so even a replacement leaves the old copy behind.
  • Use canary traffic shifting with alarms so a bad code deploy rolls back automatically and never becomes a stack problem. We cover that in zero-downtime serverless deploys.
  • Record the stack name, deploy role and recovery owner in your runbook. Half the delay in a real incident is finding out who can even run continue-update-rollback.

When to call for help

If a production stack has been frozen for more than an hour, the next step involves --resources-to-skip on a stack holding customer data, or you are importing a database between stacks, it is worth having someone who has done it before on the call. Our Serverless Framework project rescue and DevOps for Serverless Apps teams do exactly this work — unsticking stacks, splitting oversized services and rebuilding the pipeline so it does not happen again. Get in touch if you need a second pair of eyes.