+1 (726) 207-9872
SleekDeploy

Direct-to-S3 File Uploads and Event-Driven Media Processing with Serverless Framework V4

Every serverless project eventually needs to accept files: customer document uploads, profile images, CSV imports, video clips. The instinct is to POST the file to an API endpoint, and that instinct is wrong. API Gateway caps payloads at 10 MB, Lambda caps its request payload at 6 MB synchronously, and you pay Lambda duration for the time spent shovelling bytes that S3 could have received for free.

The correct shape is: the browser uploads straight to S3 with a short-lived presigned credential, S3 emits an event, and a separate Lambda does the processing. This tutorial builds that pipeline end to end with Serverless Framework V4 — presigned POST generation, a hardened bucket, an image-processing consumer on arm64 with Sharp, a DLQ, idempotency, and the lifecycle rules that keep the bucket from becoming a landfill.

Stack: Node.js 22, TypeScript, @aws-sdk/client-s3 plus @aws-sdk/s3-presigned-post, Serverless Framework V4.

The architecture

browser  --POST /uploads/sign-->  Lambda (sign)  --> presigned POST fields
browser  --POST form-data------->  S3 uploads bucket (uploads/ prefix)
S3 event --------------------->  SQS queue  -->  Lambda (process)  --> S3 derived bucket
                                     |
                                     +--> DLQ after 5 failed receives

Two details that matter more than they look:

  • SQS between S3 and the processor. Wiring S3 notifications directly to Lambda works until you get a burst of 5,000 uploads and your processor throttles, or until a poison-pill file retries forever. A queue gives you buffering, batching, a redrive policy, and a real DLQ.
  • Two buckets, or at minimum two prefixes with different notification filters. If the processor writes its output back into the bucket it is watching, it triggers itself. That recursion has burned real money for real teams. A separate derived bucket removes the possibility entirely.

1. serverless.yml

service: uploads-pipeline
frameworkVersion: '4'

provider:
  name: aws
  runtime: nodejs22.x
  region: us-east-1
  architecture: arm64
  memorySize: 1024
  timeout: 29
  environment:
    UPLOAD_BUCKET: !Ref UploadBucket
    DERIVED_BUCKET: !Ref DerivedBucket
    MAX_UPLOAD_BYTES: '26214400' # 25 MB
  iam:
    role:
      statements:
        - Effect: Allow
          Action: s3:PutObject
          Resource: !Sub '${UploadBucket.Arn}/uploads/*'
        - Effect: Allow
          Action: s3:GetObject
          Resource: !Sub '${UploadBucket.Arn}/uploads/*'
        - Effect: Allow
          Action: s3:PutObject
          Resource: !Sub '${DerivedBucket.Arn}/*'

build:
  esbuild:
    bundle: true
    minify: true
    target: node22
    exclude:
      - '@aws-sdk/*'
    external:
      - sharp

functions:
  sign:
    handler: src/sign.handler
    memorySize: 512
    timeout: 10
    events:
      - httpApi:
          path: /uploads/sign
          method: post

  process:
    handler: src/process.handler
    memorySize: 2048   # Sharp is CPU-bound; more memory means more vCPU
    timeout: 120
    layers:
      - arn:aws:lambda:${aws:region}:XXXXXXXXXXXX:layer:sharp-arm64:1
    events:
      - sqs:
          arn: !GetAtt UploadQueue.Arn
          batchSize: 5
          maximumBatchingWindow: 10
          functionResponseType: ReportBatchItemFailures

ReportBatchItemFailures is the single most important line in that block. Without it, one bad message in a batch of five causes all five to be retried.

2. The buckets and the queue

Add these under resources:. They are ordinary CloudFormation, and V4 resolves !Ref and !GetAtt against them normally.

resources:
  Resources:
    UploadBucket:
      Type: AWS::S3::Bucket
      Properties:
        PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true
        BucketEncryption:
          ServerSideEncryptionConfiguration:
            - ServerSideEncryptionByDefault:
                SSEAlgorithm: AES256
        CorsConfiguration:
          CorsRules:
            - AllowedMethods: [POST]
              AllowedOrigins: ['https://app.example.com']
              AllowedHeaders: ['*']
              MaxAge: 3000
        LifecycleConfiguration:
          Rules:
            - Id: expire-raw-uploads
              Status: Enabled
              Prefix: uploads/
              ExpirationInDays: 7
            - Id: abort-incomplete-multipart
              Status: Enabled
              AbortIncompleteMultipartUpload:
                DaysAfterInitiation: 1
        NotificationConfiguration:
          QueueConfigurations:
            - Event: 's3:ObjectCreated:*'
              Filter:
                S3Key:
                  Rules:
                    - Name: prefix
                      Value: uploads/
              Queue: !GetAtt UploadQueue.Arn

    DerivedBucket:
      Type: AWS::S3::Bucket
      Properties:
        PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true

    UploadQueue:
      Type: AWS::SQS::Queue
      Properties:
        VisibilityTimeout: 720   # at least 6x the process function timeout
        RedrivePolicy:
          deadLetterTargetArn: !GetAtt UploadDlq.Arn
          maxReceiveCount: 5

    UploadDlq:
      Type: AWS::SQS::Queue
      Properties:
        MessageRetentionPeriod: 1209600  # 14 days

    UploadQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        Queues: [!Ref UploadQueue]
        PolicyDocument:
          Statement:
            - Effect: Allow
              Principal: { Service: s3.amazonaws.com }
              Action: sqs:SendMessage
              Resource: !GetAtt UploadQueue.Arn
              Condition:
                StringEquals:
                  'aws:SourceAccount': !Ref AWS::AccountId

Three things people get wrong here:

  1. The queue policy is mandatory. Without it the stack deploys and S3 silently cannot deliver notifications. The aws:SourceAccount condition prevents the confused-deputy problem.
  2. Visibility timeout must exceed the function timeout, by a comfortable margin (AWS recommends 6x). Otherwise a slow message becomes visible again and is processed twice while the first attempt is still running.
  3. The raw-upload lifecycle rule. Uploaded originals are usually worthless once derived output exists. Seven days is generous; the alternative is paying storage on every file anyone ever uploaded, forever.

3. Generating the presigned POST

A presigned POST is preferable to a presigned PUT for browser uploads, because the policy can enforce a content-length range and a content-type condition that the client cannot override. A presigned PUT URL cannot cap file size.

src/sign.ts:

import type { APIGatewayProxyHandlerV2 } from 'aws-lambda';
import { randomUUID } from 'node:crypto';
import { S3Client } from '@aws-sdk/client-s3';
import { createPresignedPost } from '@aws-sdk/s3-presigned-post';

const s3 = new S3Client({});
const MAX = Number(process.env.MAX_UPLOAD_BYTES);

const ALLOWED: Record<string, string> = {
  'image/jpeg': 'jpg',
  'image/png': 'png',
  'image/webp': 'webp',
  'application/pdf': 'pdf',
};

export const handler: APIGatewayProxyHandlerV2 = async (event) => {
  // In a real app, resolve the tenant/user from the JWT claims on the request.
  const userId = event.requestContext.authorizer?.jwt?.claims?.sub as string | undefined;
  if (!userId) return { statusCode: 401, body: JSON.stringify({ error: 'unauthenticated' }) };

  const { contentType } = JSON.parse(event.body ?? '{}');
  const ext = ALLOWED[contentType];
  if (!ext) {
    return { statusCode: 400, body: JSON.stringify({ error: 'unsupported contentType' }) };
  }

  const key = `uploads/${userId}/${randomUUID()}.${ext}`;

  const presigned = await createPresignedPost(s3, {
    Bucket: process.env.UPLOAD_BUCKET!,
    Key: key,
    Expires: 300, // 5 minutes
    Conditions: [
      ['content-length-range', 1, MAX],
      ['eq', '$Content-Type', contentType],
    ],
    Fields: { 'Content-Type': contentType },
  });

  return {
    statusCode: 200,
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ key, ...presigned }),
  };
};

Note what the server controls: the key (so a user cannot overwrite another tenant's object), the expiry, the maximum size, and the allowed content types. The client controls nothing but the bytes.

The browser side is a plain multipart form — no AWS SDK in the front end:

const { url, fields, key } = await fetch('/uploads/sign', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ contentType: file.type }),
}).then((r) => r.json());

const form = new FormData();
Object.entries(fields).forEach(([k, v]) => form.append(k, v));
form.append('Content-Type', file.type);
form.append('file', file); // 'file' must be appended last

const res = await fetch(url, { method: 'POST', body: form });
if (!res.ok) throw new Error(`upload failed: ${res.status}`);

file last is not a style preference. S3 ignores any form field that appears after the file content.

4. Processing the upload

src/process.ts:

import type { SQSHandler, SQSRecord } from 'aws-lambda';
import { GetObjectCommand, PutObjectCommand, S3Client } from '@aws-sdk/client-s3';
import sharp from 'sharp';

const s3 = new S3Client({});
const SIZES = [{ name: 'thumb', width: 240 }, { name: 'medium', width: 1024 }];

type S3Notification = {
  Records?: { s3: { bucket: { name: string }; object: { key: string; size: number } } }[];
};

const processRecord = async (record: SQSRecord) => {
  const body = JSON.parse(record.body) as S3Notification;
  // S3 test events have no Records array; skip them rather than failing.
  for (const r of body.Records ?? []) {
    const bucket = r.s3.bucket.name;
    const key = decodeURIComponent(r.s3.object.key.replace(/\+/g, ' '));

    if (!/\.(jpg|png|webp)$/i.test(key)) {
      console.log(JSON.stringify({ event: 'skip_non_image', key }));
      continue;
    }

    const obj = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
    const input = Buffer.from(await obj.Body!.transformToByteArray());

    // Validate by content, not by extension: Sharp throws on anything that is not an image.
    const meta = await sharp(input).metadata();
    console.log(JSON.stringify({ event: 'image_meta', key, format: meta.format, width: meta.width }));

    for (const size of SIZES) {
      const out = await sharp(input)
        .rotate()                      // honour EXIF orientation
        .resize({ width: size.width, withoutEnlargement: true })
        .webp({ quality: 82 })
        .toBuffer();

      await s3.send(new PutObjectCommand({
        Bucket: process.env.DERIVED_BUCKET!,
        Key: key.replace(/^uploads\//, `${size.name}/`).replace(/\.[^.]+$/, '.webp'),
        Body: out,
        ContentType: 'image/webp',
        CacheControl: 'public, max-age=31536000, immutable',
      }));
    }
  }
};

export const handler: SQSHandler = async (event) => {
  const batchItemFailures: { itemIdentifier: string }[] = [];

  for (const record of event.Records) {
    try {
      await processRecord(record);
    } catch (err) {
      console.error(JSON.stringify({ event: 'process_failed', messageId: record.messageId, err: String(err) }));
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }

  return { batchItemFailures };
};

Points worth internalizing:

  • Keys arrive URL-encoded. A file named my photo.jpg arrives as my+photo.jpg. Forgetting to decode produces a NoSuchKey error that looks like a permissions problem.
  • Never trust the extension or the declared content type. The presigned policy pins Content-Type, but a client can still declare image/png and upload something else entirely. Decoding through Sharp is the validation; if it throws, the message goes to the DLQ and nothing downstream ever sees the bytes.
  • Derived output is immutable, so it can carry a one-year cache header safely.
  • Idempotency: SQS delivery is at-least-once, and the derived key is a pure function of the source key, so a duplicate delivery rewrites identical output. That is idempotency by construction, which is far cheaper than a DynamoDB dedupe table. Prefer it whenever you can arrange it.

5. Files larger than a few hundred megabytes

Presigned POST works up to 5 GB, but a single browser POST of a 2 GB video will fail on any flaky connection. For large media, switch to multipart upload: a create endpoint returning an UploadId, a sign-part endpoint returning a presigned URL per part, and a complete endpoint. The AbortIncompleteMultipartUpload lifecycle rule above is what stops abandoned parts from accruing storage charges invisibly — that rule belongs on every bucket that accepts uploads, whether or not you use multipart today.

For files that need heavy transcoding, do not do it in Lambda. Have the processor submit a MediaConvert job or an ECS/Fargate task and let the 15-minute Lambda limit stop being your problem.

6. What to alarm on

Three CloudWatch alarms cover the realistic failure modes:

  1. ApproximateNumberOfMessagesVisible on the DLQ greater than 0 — something is being rejected and a human needs to look.
  2. ApproximateAgeOfOldestMessage on the main queue above 300 seconds — the processor cannot keep up, or is throttled.
  3. Lambda Throttles greater than 0 on the process function — raise reserved concurrency or lower batchSize.

Add a scheduled reconciliation job if uploads are business-critical: list the uploads/ prefix, check that each object has derived output, and re-enqueue anything orphaned. Notifications are reliable, not guaranteed.

Deploy and test

serverless deploy --stage dev
serverless logs -f process --stage dev --tail

In a second terminal, copy a sample image into the watched prefix and watch for the image_meta log line, then confirm thumb/ and medium/ objects exist in the derived bucket. Then upload a text file renamed to .jpg and confirm the message lands in the DLQ after five attempts rather than crashing the consumer.

Wrapping up

The pipeline is four moving parts — a signing endpoint, a locked-down bucket, a queue with a DLQ, and a stateless processor — and none of them ever puts a large payload through API Gateway. Get the queue policy, the visibility timeout, ReportBatchItemFailures, the lifecycle rules, and the separate output bucket right, and this design scales from ten uploads a day to ten thousand an hour without a code change.

We build this pattern regularly as part of Serverless Application Development and Serverless Integration Services. If you would like a review of an upload path that is already in production, get in touch.