Serverless observability in 2026 is mostly a solved problem, but only if you assemble the pieces deliberately. The default (unstructured console.log lines, no tracing, and an alarm on a Lambda error count) tells you something is wrong after users do. This post sets up the stack we deploy on every client engagement: structured logs, custom metrics, and distributed traces, with Powertools for AWS Lambda doing most of the work, on a Serverless Framework V4 project.
The three signals, and what each is for
- Logs answer "what happened in this request?" They must be structured JSON with a correlation ID, or you cannot query them.
- Metrics answer "how is the system doing right now?" and drive alarms. Lambda gives you invocations, errors, duration, throttles, and concurrency for free; business metrics (orders created, tokens consumed, payments failed) you emit yourself.
- Traces answer "where did the time go?" across API Gateway, Lambda, DynamoDB, SQS, and external calls. This is what turns "the API is slow" into "DynamoDB p99 went from 8 ms to 400 ms at 14:02."
Powertools for AWS Lambda provides a Logger, Metrics, and Tracer utility for each of TypeScript, Python, Java, and .NET. The TypeScript docs are the reference for what follows.
1. Install
npm install @aws-lambda-powertools/logger @aws-lambda-powertools/metrics @aws-lambda-powertools/tracer
npm install @middy/core
Middy is a small middleware engine Powertools integrates with so you do not hand-wire the utilities into every handler.
2. Configure the service
service: orders-api
frameworkVersion: '4'
provider:
name: aws
runtime: nodejs22.x
architecture: arm64
tracing:
apiGateway: true
lambda: true
logs:
httpApi: true
environment:
POWERTOOLS_SERVICE_NAME: orders-api
POWERTOOLS_METRICS_NAMESPACE: OrdersApi
POWERTOOLS_LOG_LEVEL: ${param:logLevel}
POWERTOOLS_LOGGER_SAMPLE_RATE: ${param:debugSampleRate}
iam:
role:
statements:
- Effect: Allow
Action:
- xray:PutTraceSegments
- xray:PutTelemetryRecords
Resource: '*'
stages:
default:
params:
logLevel: INFO
debugSampleRate: '0.1'
prod:
params:
debugSampleRate: '0.01'
build:
esbuild:
bundle: true
minify: true
target: node22
exclude:
- '@aws-sdk/*'
functions:
createOrder:
handler: src/orders.create
events:
- httpApi:
path: /orders
method: post
tracing.lambda: true enables X-Ray active tracing on the functions; tracing.apiGateway: true makes API Gateway start the trace so you see the full request. The sample rate means 10% of requests in dev (1% in prod) log at DEBUG even when the level is INFO, which is the cheap way to keep detailed logs available for a slice of traffic.
3. The handler
src/orders.ts:
import middy from '@middy/core';
import { Logger } from '@aws-lambda-powertools/logger';
import { injectLambdaContext } from '@aws-lambda-powertools/logger/middleware';
import { Metrics, MetricUnit } from '@aws-lambda-powertools/metrics';
import { logMetrics } from '@aws-lambda-powertools/metrics/middleware';
import { Tracer } from '@aws-lambda-powertools/tracer';
import { captureLambdaHandler } from '@aws-lambda-powertools/tracer/middleware';
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb';
import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from 'aws-lambda';
const logger = new Logger();
const metrics = new Metrics();
const tracer = new Tracer();
// captureAWSv3Client adds a subsegment for every DynamoDB call.
const ddb = tracer.captureAWSv3Client(DynamoDBDocumentClient.from(new DynamoDBClient({})));
const createHandler = async (event: APIGatewayProxyEventV2): Promise<APIGatewayProxyResultV2> => {
const body = JSON.parse(event.body ?? '{}');
const orderId = crypto.randomUUID();
logger.appendKeys({ orderId, customerId: body.customerId });
logger.info('creating order', { itemCount: body.items?.length ?? 0 });
await ddb.send(new PutCommand({
TableName: process.env.TABLE_NAME,
Item: { pk: `CUSTOMER#${body.customerId}`, sk: `ORDER#${orderId}`, ...body },
}));
metrics.addMetric('OrderCreated', MetricUnit.Count, 1);
metrics.addMetric('OrderValue', MetricUnit.None, Number(body.total ?? 0));
return { statusCode: 201, body: JSON.stringify({ orderId }) };
};
export const create = middy(createHandler)
.use(captureLambdaHandler(tracer))
.use(injectLambdaContext(logger, { logEvent: false }))
.use(logMetrics(metrics, { captureColdStartMetric: true }));
What you get from those forty lines:
- Every log line is JSON with
service,function_name,function_request_id,cold_start,xray_trace_id, and whatever keys you appended. Logs Insights can queryfields orderId, customerId | filter level = "ERROR". OrderCreatedandOrderValueare emitted via CloudWatch Embedded Metric Format in a single log line at the end of the invocation; no extra API calls, no latency.captureColdStartMetricgives you aColdStartmetric for free, which is how you measure whether the cold-start work paid off.- The X-Ray trace shows API Gateway, the function, and a DynamoDB subsegment with its latency. Add
tracer.captureAWSv3Clientto any other SDK client (SQS, Bedrock, S3) for the same.
4. Alarms that mean something
Lambda Errors > 0 is a noisy alarm. The alarms we actually keep:
- Error rate as a metric math expression:
errors / invocations > 0.02for five minutes. - p99 duration against the function's timeout: if p99 is within 20% of the timeout you are about to see failures.
- Throttles > 0 on anything user-facing.
- Iterator age on stream-triggered functions (Kinesis, DynamoDB Streams) and ApproximateAgeOfOldestMessage on every SQS queue, with a dead-letter queue alarm on
ApproximateNumberOfMessagesVisible > 0. - Business metrics:
OrderCreateddropping to zero during business hours is the alarm that catches a broken frontend that Lambda metrics never will.
Define them in the resources block of serverless.yml as AWS::CloudWatch::Alarm resources so they deploy with the service, or use the serverless-plugin-aws-alerts plugin if you prefer its shorthand.
5. Logs: retention and cost
CloudWatch Logs ingestion is a meaningful line item on busy APIs. Three controls:
provider:
logRetentionInDays: 30
Set it; the default is forever. Keep POWERTOOLS_LOGGER_SAMPLE_RATE low in production. And use log level INFO for normal operation; DEBUG on a high-volume function can cost more than the function.
For long-term analysis, route logs to S3 via a subscription filter and query with Athena, or ship to your existing platform (Datadog, Grafana Cloud, Splunk, Axiom) through a CloudWatch subscription. The Elastic Stack (formerly ELK) remains a common self-hosted destination.
6. Correlating across services
In an event-driven system a request crosses several functions. Two practices keep it traceable:
- Propagate the trace. X-Ray propagates automatically through SQS, SNS, and EventBridge when tracing is on. For anything custom (HTTP calls between services), forward the
X-Amzn-Trace-Idheader. - Carry a correlation ID in the payload. Put
correlationIdin every event you publish andlogger.appendKeys({ correlationId })in every consumer. A trace ID is for X-Ray; a correlation ID is for humans querying logs across services.
What this replaces
Compared to the "CloudWatch, Azure Monitor, or Stackdriver" guidance that dated serverless content still gives, the current AWS-native stack is CloudWatch Logs Insights, CloudWatch Metrics with EMF, and X-Ray, with Powertools as the glue. If you want a single-pane vendor on top, Datadog and Grafana Cloud both ingest all three signals from the same setup.
Our Serverless Monitoring and Logging service installs this stack, writes the alarms, and hands your team a runbook. Contact us.