+1 (726) 207-9872

Multi-Account, Multi-Region Serverless: Stage Isolation, Deploy Roles, and Failover with Serverless Framework V4

Most Serverless Framework services start life in one AWS account and one region. That is the right place to start. It stops being the right place the moment a client asks two questions we hear on almost every engagement: "How do we make sure a bad deploy to dev cannot touch production?" and "What happens if us-east-1 has a bad day?"

Those are the same problem viewed from two angles — blast-radius control — and they are solved with account boundaries and region strategy, not with more YAML in one stack. This tutorial walks through the setup we deploy for clients: per-stage AWS accounts, a deployment role per account with no long-lived keys, region-parameterised configuration, and a warm-standby failover pattern with Route 53 and DynamoDB global tables.

Why one account per stage

A single account with dev, staging, and prod stages sharing it looks cheaper and simpler. In practice it fails in three predictable ways:

  • IAM is not a real boundary. Getting resource-level, stage-scoped policies right for every service (Lambda, DynamoDB, SQS, EventBridge, Secrets Manager, CloudWatch) is achievable but fragile. One wildcard in one policy and dev code can read production data.
  • Quotas are per account per region. A load test in staging that exhausts concurrent executions or API Gateway throttles takes production down with it.
  • Cost and audit stories get muddy. "What does production cost?" should be one number on one bill, not a tag-filtering exercise.

Separate accounts give you a hard boundary the platform enforces for free. With AWS Organizations and Control Tower the account itself is cheap to create, and the Serverless Framework does not care how many accounts you have — it deploys with whatever credentials it is handed.

Target layout:

Organization
├── ou: workloads-nonprod
│   ├── acct: myapp-dev      (stage: dev)
│   └── acct: myapp-staging  (stage: staging)
├── ou: workloads-prod
│   └── acct: myapp-prod     (stage: prod)
└── ou: security
    └── acct: myapp-shared   (CI/CD, artifacts, central logs)

Step 1: One deployment role per account, assumed from CI

Do not create an IAM user per account and paste keys into your CI provider. Use OIDC federation from your CI system into a single role per target account, as covered in our GitHub Actions and OIDC tutorial, then scope each role to its stage.

deploy-role.yml (deployed once per account with CloudFormation, not with your app stack):

Parameters:
  GitHubOrg:      { Type: String }
  GitHubRepo:     { Type: String }
  AllowedBranch:  { Type: String, Default: refs/heads/main }

Resources:
  GitHubOidcProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://token.actions.githubusercontent.com
      ClientIdList: [ sts.amazonaws.com ]

  DeployRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: serverless-deploy
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: { Federated: !Ref GitHubOidcProvider }
            Action: sts:AssumeRoleWithWebIdentity
            Condition:
              StringEquals:
                token.actions.githubusercontent.com:sub:
                  !Sub 'repo:${GitHubOrg}/${GitHubRepo}:ref:${AllowedBranch}'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/PowerUserAccess   # tighten before go-live

Two details that matter more than they look:

  • The sub condition pins which repository and which ref can assume the role. The production account's role should only trust refs/heads/main (or better, a protected GitHub Environment). Feature branches can never reach production, regardless of what a workflow file says.
  • PowerUserAccess is a starting point for getting the pipeline green. Before go-live, replace it with a policy generated from what your stack actually creates. Deploy once, then read the CloudTrail events for the role and build the policy from that list.

Step 2: Parameterise the service by stage and region

The single most common mistake in multi-region work is treating the region as a deploy-time accident. Make it an explicit input with explicit per-region values.

serverless.yml:

service: myapp

provider:
  name: aws
  runtime: nodejs22.x
  architecture: arm64
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}
  stackName: myapp-${sls:stage}
  environment:
    STAGE: ${sls:stage}
    PRIMARY_REGION: ${param:primaryRegion}
    TABLE_NAME: ${param:tableName}
    IS_PRIMARY: ${param:isPrimary}

stages:
  default:
    params:
      primaryRegion: us-east-1
      tableName: myapp-${sls:stage}
      isPrimary: ${strToBool(${param:thisIsPrimary, 'true'})}
  prod:
    params:
      primaryRegion: us-east-1
      logRetention: 90

params:
  # region-specific overrides selected by the --region flag
  us-east-1:
    thisIsPrimary: 'true'
  eu-west-1:
    thisIsPrimary: 'false'

functions:
  api:
    handler: src/api.handler
    memorySize: 1024
    events:
      - httpApi:
          method: GET
          path: /orders/{id}

Deploys become explicit and boring:

serverless deploy --stage prod --region us-east-1
serverless deploy --stage prod --region eu-west-1

Two rules we enforce in review:

  1. No region names in application code. If a handler needs to know where it is, read it from an environment variable populated by params. Hard-coded ARNs are the number one blocker when a client asks for a second region.
  2. Never serverless remove a stage from a laptop. Removal runs through the pipeline against the same role, or not at all.

Step 3: Decide what "multi-region" actually means for this client

There is no single multi-region architecture; there is a spectrum with very different price tags. Pick a point on it deliberately, based on the recovery objectives the business will actually sign off on.

PatternRTORPORelative costGood fit
Backup & restoreHoursMinutes–hoursLowestInternal tools, batch
Pilot light~30 minSecondsLowBack-office APIs
Warm standbyMinutesSecondsMediumMost customer-facing APIs
Active–activeSecondsSecondsHighestPayments, global low latency

Warm standby is the sweet spot for the majority of serverless workloads, and it is genuinely cheap here: Lambda and API Gateway cost nothing when idle, so the secondary region's compute tier bills approximately zero until traffic arrives. The recurring cost is data replication, not idle capacity — which is exactly the argument that makes serverless DR an easier sell than the EC2 equivalent.

Active–active is where the difficulty jumps, because you inherit write conflicts. Do not choose it because it sounds better.

Step 4: Replicate the data

DynamoDB global tables are the path of least resistance. Declare the table once in a shared stack (not in the app stack — you do not want a stack rollback destroying replicated data) with replicas in both regions:

resources:
  Resources:
    OrdersTable:
      Type: AWS::DynamoDB::GlobalTable
      DeletionPolicy: Retain
      UpdateReplacePolicy: Retain
      Properties:
        TableName: myapp-${sls:stage}-orders
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          - { AttributeName: pk, AttributeType: S }
          - { AttributeName: sk, AttributeType: S }
        KeySchema:
          - { AttributeName: pk, KeyType: HASH }
          - { AttributeName: sk, KeyType: RANGE }
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES
        Replicas:
          - Region: us-east-1
            PointInTimeRecoverySpecification: { PointInTimeRecoveryEnabled: true }
          - Region: eu-west-1
            PointInTimeRecoverySpecification: { PointInTimeRecoveryEnabled: true }

Global tables replicate asynchronously with last-writer-wins conflict resolution. Two consequences to design around:

  • A read in the secondary region can be a second or two behind. Fine for order history; not fine for "did this idempotency key already run?" if both regions accept writes.
  • Under active–active, two simultaneous writes to the same item silently keep one. If your domain cannot tolerate that, keep writes pinned to the primary region and let the secondary serve reads until failover.

Other stores need their own answer: Aurora Global Database for Postgres (fast replication, one writer), S3 Cross-Region Replication for objects, and — easy to forget — Secrets Manager replicas plus SSM parameters, which are regional and will fail your standby's cold start if you skip them.

Step 5: Route traffic and fail over

Give each region a regional API Gateway custom domain, then put Route 53 in front with a failover record set and a health check against a real dependency-touching endpoint.

resources:
  Resources:
    HealthCheck:
      Type: AWS::Route53::HealthCheck
      Properties:
        HealthCheckConfig:
          Type: HTTPS
          FullyQualifiedDomainName: ${sls:stage}-${aws:region}.api.example.com
          ResourcePath: /health
          RequestInterval: 30
          FailureThreshold: 3

    ApiDnsRecord:
      Type: AWS::Route53::RecordSet
      Properties:
        HostedZoneId: ${param:hostedZoneId}
        Name: api.example.com
        Type: A
        SetIdentifier: ${aws:region}
        Failover: ${param:isPrimary, 'PRIMARY'}
        HealthCheckId: !Ref HealthCheck
        AliasTarget:
          DNSName: !GetAtt HttpApiDomainName.RegionalDomainName
          HostedZoneId: !GetAtt HttpApiDomainName.RegionalHostedZoneId

Make /health shallow but honest: it should verify the function can reach its primary data store, and nothing more. A health check that only returns 200 OK from the function itself will happily keep routing traffic into a region whose database replica is broken. A health check that fans out to six downstream services will flap.

Set the record TTL to 60 seconds and accept that DNS failover is measured in minutes, not milliseconds. If you need faster, use CloudFront with origin failover in front of both regional endpoints, or Global Accelerator for a fixed-anycast entry point.

Step 6: Test the failover, on a schedule

An untested standby is a rumour. Put a game day in the calendar quarterly and run it as a drill:

  1. Disable the primary health check (aws route53 update-health-check --health-check-id ... --disabled) rather than breaking anything real.
  2. Watch traffic move: request counts on the secondary's API Gateway, 5xx on the client side, and how long the shift actually took.
  3. Confirm writes land and replicate back once the primary returns.
  4. Re-enable, and write down the measured RTO. That number, not the diagram, is what you tell the business.

Also verify the boring things that break standbys: does the secondary region have the same Lambda concurrency quota? The same Secrets Manager entries? An up-to-date deployment of the current code? A standby three releases behind is a standby that will not work. Deploy every region on every release from the same pipeline — sequentially, primary last, so a broken build stops before it reaches the region serving traffic.

Where this usually goes wrong

  • Only the compute is multi-region. Teams replicate Lambda and forget SSM parameters, Secrets Manager, ECR images for container-packaged functions, and ACM certificates (which must exist in each region — and in us-east-1 for CloudFront).
  • Cross-region calls sneak in. A function in eu-west-1 calling an SQS queue in us-east-1 has quietly re-coupled you to the region you were trying to survive. Grep for hard-coded ARNs in CI.
  • Idempotency tables are regional. If your dedupe store does not replicate, a failover can reprocess messages. Point idempotency at the global table.
  • Stage isolation stops at IAM. If a developer can assume the production deploy role from a laptop, you have organisational separation without technical separation.

Where to start

If you are running a single-account, single-region service today, the highest-value sequence is: split production into its own account with an OIDC deployment role, remove every hard-coded region and ARN from the service, then pick a DR pattern that matches an RTO someone has actually signed off on. The first two steps pay for themselves whether or not you ever deploy a second region.

Multi-account and multi-region work touches IAM, DNS, data replication, and your deployment pipeline at once, and it is easy to end up with a standby that only works in the diagram. If you want a second pair of eyes on the design — or a measured game day before you promise a number to the business — get in touch or read more about our DevOps for Serverless Apps and Serverless Architecture Design engagements.