SleekDeploy | Orchestrating Long-Running Workflows with AWS Step Functions and Serverless Framework V4

Most serverless applications start with a single Lambda handling a request. Then the process grows: validate the order, charge the card, reserve inventory, render a PDF, email the customer, and retry sensibly when the payment provider times out. Teams usually solve this by having Lambda A invoke Lambda B, or by chaining SQS queues, and end up with business logic scattered across five functions and no way to answer "where is order 4471 stuck?"

This is the point where orchestration beats choreography. Our event-driven serverless tutorial covers the choreographed side — EventBridge, SQS, idempotency — and it is the right tool for loosely coupled fan-out. This post covers the other half: modelling a known multi-step process as an AWS Step Functions state machine, deployed with Serverless Framework V4.

When to reach for Step Functions

Use a state machine when at least two of these are true:

The process has more than three steps with real branching or error handling.
Steps can fail independently and need different retry or compensation behaviour.
The process takes longer than 15 minutes, or waits on a human or an external callback.
Someone will eventually ask "which step did this fail on, and can we resume it?"

Stay with plain Lambda + SQS when the work is a single unit, latency-sensitive, and the retry story is "retry the whole thing".

Standard vs Express: pick before you build

This choice affects cost, duration and observability, and changing it later means re-testing everything.

	Standard	Express
Max duration	1 year	5 minutes
Billing	per state transition	per request + duration/memory
Execution history	full, in the console/API	CloudWatch Logs only
Semantics	exactly-once state transitions	at-least-once (async)
Good for	order fulfilment, ETL, human approval, migrations	high-volume ingestion, per-event transforms

Rule of thumb: business processes are Standard, streaming-style work is Express. A Standard workflow with 12 transitions costs roughly $0.0003 per execution — irrelevant at thousands per day, painful at tens of millions. Express at very high volume is usually an order of magnitude cheaper, but you lose the execution history that makes support tickets answerable.

Setting up the plugin

Step Functions are not native to Serverless Framework's core schema; the community plugin is still the standard way to declare them, and it works fine under V4.

npm install --save-dev serverless-step-functions

# serverless.yml
service: order-fulfilment
frameworkVersion: '4'

plugins:
  - serverless-step-functions

provider:
  name: aws
  runtime: nodejs22.x
  architecture: arm64
  region: eu-west-1
  stage: ${opt:stage, 'dev'}
  environment:
    ORDERS_TABLE: !Ref OrdersTable
    POWERTOOLS_SERVICE_NAME: order-fulfilment

build:
  esbuild:
    bundle: true
    minify: true
    target: node22

The plugin adds serverless deploy support for a top-level stepFunctions: block, plus serverless invoke stepf --name <name> --data '{...}' for manual runs.

The workflow

A realistic order fulfilment flow: validate, charge, reserve stock, then notify — with compensation if stock reservation fails after the card was charged.

functions:
  validateOrder:
    handler: src/validate.handler
  chargePayment:
    handler: src/charge.handler
    timeout: 20
  reserveInventory:
    handler: src/reserve.handler
  refundPayment:
    handler: src/refund.handler

stepFunctions:
  stateMachines:
    fulfilOrder:
      name: ${self:service}-${sls:stage}-fulfil-order
      type: STANDARD
      tracingConfig:
        enabled: true
      loggingConfig:
        level: ERROR
        includeExecutionData: true
        destinations:
          - !GetAtt FulfilLogGroup.Arn
      definition:
        QueryLanguage: JSONata
        Comment: Validate, charge, reserve, notify
        StartAt: ValidateOrder
        States:
          ValidateOrder:
            Type: Task
            Resource: !GetAtt ValidateOrderLambdaFunction.Arn
            Arguments:
              orderId: '{% $states.input.orderId %}'
            Assign:
              order: '{% $states.result %}'
            Retry:
              - ErrorEquals: [Lambda.ServiceException, Lambda.TooManyRequestsException]
                IntervalSeconds: 1
                MaxAttempts: 3
                BackoffRate: 2
            Catch:
              - ErrorEquals: [ValidationError]
                Next: RejectOrder
            Next: ChargePayment

          ChargePayment:
            Type: Task
            Resource: !GetAtt ChargePaymentLambdaFunction.Arn
            Arguments:
              orderId: '{% $order.orderId %}'
              amount: '{% $order.totalCents %}'
              idempotencyKey: '{% $states.context.Execution.Name %}'
            Assign:
              payment: '{% $states.result %}'
            Retry:
              - ErrorEquals: [States.Timeout, PaymentProviderUnavailable]
                IntervalSeconds: 2
                MaxAttempts: 4
                BackoffRate: 2
                JitterStrategy: FULL
            Catch:
              - ErrorEquals: [States.ALL]
                Next: RejectOrder
            Next: ReserveInventory

          ReserveInventory:
            Type: Task
            Resource: !GetAtt ReserveInventoryLambdaFunction.Arn
            Arguments:
              orderId: '{% $order.orderId %}'
              lines: '{% $order.lines %}'
            Catch:
              - ErrorEquals: [States.ALL]
                Next: RefundPayment
            Next: NotifyCustomer

          RefundPayment:
            Type: Task
            Resource: !GetAtt RefundPaymentLambdaFunction.Arn
            Arguments:
              paymentId: '{% $payment.id %}'
              reason: inventory_unavailable
            Next: RejectOrder

          NotifyCustomer:
            Type: Task
            Resource: arn:aws:states:::aws-sdk:sns:publish
            Arguments:
              TopicArn: !Ref OrderEventsTopic
              Subject: 'Order confirmed'
              Message: '{% "Order " & $order.orderId & " is confirmed." %}'
            End: true

          RejectOrder:
            Type: Task
            Resource: arn:aws:states:::dynamodb:updateItem
            Arguments:
              TableName: ${self:provider.environment.ORDERS_TABLE}
              Key:
                pk: { S: '{% $states.input.orderId %}' }
              UpdateExpression: 'SET #s = :s'
              ExpressionAttributeNames: { '#s': status }
              ExpressionAttributeValues: { ':s': { S: rejected } }
            End: true

Three things in there are worth calling out, because they remove code most teams still write by hand.

1. Variables (`Assign`) kill the payload-passing dance

Historically, every state's output had to be merged into a growing JSON blob using ResultPath and Parameters gymnastics, or you added a "glue" Lambda whose only job was reshaping. Assign stores a value in a workflow variable that any later state can read as $order, $payment, and so on. State input stays small, and you stay well clear of the 256 KB payload limit.

2. JSONata replaces most glue Lambdas

Setting QueryLanguage: JSONata lets you write real expressions — string concatenation, arithmetic, filtering, date formatting — inside Arguments between {% %}. Formatting a message, summing line items or picking fields no longer justifies a Lambda. Fewer functions means less IAM, fewer cold starts, less to test. You can set the query language per state if you are migrating an existing JSONPath machine gradually.

3. Optimised SDK integrations

arn:aws:states:::aws-sdk:sns:publish and arn:aws:states:::dynamodb:updateItem call AWS APIs directly from the state machine. Anything that is "one AWS API call" — put an item, publish, start a job, send a message — should not be a Lambda.

IAM: the state machine needs its own role

The plugin generates a role with permission to invoke the Lambdas in the definition, but direct SDK integrations need explicit grants. Declare them:

stepFunctions:
  stateMachines:
    fulfilOrder:
      role: !GetAtt FulfilOrderRole.Arn
      # ...definition as above

resources:
  Resources:
    FulfilLogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        LogGroupName: /aws/vendedlogs/states/${self:service}-${sls:stage}
        RetentionInDays: 30

    FulfilOrderRole:
      Type: AWS::IAM::Role
      Properties:
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal: { Service: states.amazonaws.com }
              Action: sts:AssumeRole
        Policies:
          - PolicyName: fulfil-order
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action: lambda:InvokeFunction
                  Resource:
                    - !GetAtt ValidateOrderLambdaFunction.Arn
                    - !GetAtt ChargePaymentLambdaFunction.Arn
                    - !GetAtt ReserveInventoryLambdaFunction.Arn
                    - !GetAtt RefundPaymentLambdaFunction.Arn
                - Effect: Allow
                  Action: sns:Publish
                  Resource: !Ref OrderEventsTopic
                - Effect: Allow
                  Action: dynamodb:UpdateItem
                  Resource: !GetAtt OrdersTable.Arn
                - Effect: Allow
                  Action:
                    - logs:CreateLogDelivery
                    - logs:GetLogDelivery
                    - logs:UpdateLogDelivery
                    - logs:DeleteLogDelivery
                    - logs:ListLogDeliveries
                    - logs:PutResourcePolicy
                    - logs:DescribeResourcePolicies
                    - logs:DescribeLogGroups
                  Resource: '*'
                - Effect: Allow
                  Action:
                    - xray:PutTraceSegments
                    - xray:PutTelemetryRecords
                    - xray:GetSamplingRules
                    - xray:GetSamplingTargets
                  Resource: '*'

The same principles as our least-privilege IAM post apply: scope to ARNs, and only the logging and X-Ray actions get * because the APIs require it.

Idempotency still matters

A retry in a state machine re-invokes your Lambda. If ChargePayment succeeded but the response was lost to a timeout, attempt two must not charge twice. Two defences, both in the example above:

Pass a stable idempotency key from the execution context — $states.context.Execution.Name is unique per execution and identical across retries of the same state.
Use that key in the downstream provider call, and/or wrap the handler with Powertools idempotency backed by DynamoDB.

Never rely on "the retry probably won't happen".

Starting executions

Triggering from EventBridge keeps the API path fast and the workflow decoupled:

stepFunctions:
  stateMachines:
    fulfilOrder:
      events:
        - eventBridge:
            pattern:
              source: ['orders.api']
              detail-type: ['OrderPlaced']

For synchronous work — a request that must wait for a result — use an Express workflow and StartSyncExecution from API Gateway or a thin Lambda. Do not call StartSyncExecution on a Standard workflow; it is not supported.

Two patterns worth knowing

Distributed Map for large batches. A Map state in DISTRIBUTED mode runs up to 10,000 concurrent child executions and can read directly from an S3 object listing or a CSV/JSONL file. This is how you process a million-row file without writing a batching Lambda:

ProcessFile:
  Type: Map
  ItemReader:
    Resource: arn:aws:states:::s3:getObject
    ReaderConfig: { InputType: CSV, CSVHeaderLocation: FIRST_ROW }
    Arguments:
      Bucket: '{% $states.input.bucket %}'
      Key: '{% $states.input.key %}'
  ItemProcessor:
    ProcessorConfig: { Mode: DISTRIBUTED, ExecutionType: EXPRESS }
    StartAt: ProcessRow
    States:
      ProcessRow:
        Type: Task
        Resource: !GetAtt ProcessRowLambdaFunction.Arn
        End: true
  MaxConcurrency: 200
  ToleratedFailurePercentage: 1
  Next: Summarise

Set MaxConcurrency deliberately — 10,000 concurrent Lambdas will happily exhaust your account concurrency and throttle your production API. Set reserved concurrency on the worker function too.

Wait-for-callback for human steps. Resource: arn:aws:states:::lambda:invoke.waitForTaskToken pauses the execution (up to a year) until something calls SendTaskSuccess with the token. This is how approval flows, third-party webhooks and "wait for the file to arrive" work without polling.

Testing and local development

Step Functions is the one part of a serverless stack where local emulation genuinely pays off, because iterating on a definition through full deploys is slow.

Definition linting: run the definition through Step Functions Local (Docker image amazon/aws-stepfunctions-local) with mocked task results in MockConfigFile.json. You can assert branching and error paths without deploying a single Lambda.
TestState API: aws stepfunctions test-state evaluates a single state — including its JSONata expressions and IAM permissions — against sample input. Fastest way to debug a {% %} expression.
Unit-test handlers normally, then test the wiring with one deployed execution per PR in an ephemeral stage, as described in our testing tutorial.

Operating it

Alarm on ExecutionsFailed and ExecutionsTimedOut per state machine; those are the two that mean orders are stuck.
Alarm on ExecutionThrottled — Standard workflows have a state transition rate quota that surprises people during backfills.
Log level ERROR plus includeExecutionData: true is the pragmatic default; ALL on a high-volume Express workflow can cost more than the workflow itself.
Standard executions can be redriven from the failed state after you fix the bug, instead of re-running from the start. Design steps to be resumable and this becomes your standard incident response.

Common mistakes we get called in to fix

Everything is a Lambda. Half the states in a typical hand-written machine are one AWS API call or a data reshape. Replace them with SDK integrations and JSONata.
Express chosen for a business process. Six months later there is no execution history to answer a customer complaint, and at-least-once semantics have caused a duplicate charge.
No compensation path. Money moved, then a later step failed, and there is no RefundPayment state. Model the saga explicitly.
Giant payloads. Passing whole documents between states hits the 256 KB quota. Put the payload in S3 and pass the key.
Retry with no jitter. BackoffRate: 2 without JitterStrategy: FULL synchronises thousands of retries into a thundering herd against the same downstream API.

Where this fits

Step Functions is the piece that turns a pile of Lambdas into a business process you can explain to a non-engineer and debug at 3 a.m. Combined with EventBridge for decoupling and per-function least-privilege IAM, it is the backbone of most production serverless systems we design.

If you are staring at a chain of Lambdas that nobody fully understands any more, that is exactly the kind of work our serverless architecture design and project rescue teams do. Get in touch with a sketch of the process and we will tell you what it should look like as a state machine.

Orchestrating Long-Running Workflows with AWS Step Functions and Serverless Framework V4