#cloudflare-queues#async#message-broker#backend#finops

Serverless Background Job Processing with Cloudflare Queues and Workflows

"Build cost-effective asynchronous background job processing using Cloudflare Queues and Workflows without idle servers."

By huud
0 views
~3 min read
Serverless Background Job Processing with Cloudflare Queues and Workflows

Serverless Background Job Processing with Cloudflare Queues and Workflows

Asynchronous job execution is essential for building reliable backend systems: transactional emails, invoice generation, webhook dispatching, and search indexing.

Traditional implementations on AWS combine Amazon SQS with EC2/ECS Worker Auto-Scaling Groups or AWS Step Functions. However, keeping EC2 instances running 24/7 to poll message queues produces significant idle compute waste. Furthermore, AWS Step Functions charges per state transition, leading to high bills for multi-step orchestrations.

Cloudflare Queues and Cloudflare Workflows deliver an anti-idle, serverless event-processing model that eliminates standby costs entirely.


1. Quantifying Idle Waste in Traditional Queue Workers

Consider a typical message worker setup on AWS:

  • Message Broker: AWS SQS Standard Queue
  • Consumers: 2x t4g.small EC2 instances in an Auto-Scaling Group for continuous availability.

AWS Monthly Baseline:

  • 2x EC2 t4g.small instances (Always-on 730 hrs): ~$24.50 / month
  • AWS SQS (10 Million API requests): ~$4.00 / month
  • VPC Peering & NAT Data Processing: ~$15.00 / month
  • Total Standby AWS Spend: ~$43.50 / month (even when 0 jobs are queued).

Cloudflare Queues Baseline:

  • Consumer Workers are triggered on-demand upon message arrival. Zero messages = zero active compute cost.
  • Price: $0.40 per 1,000,000 operations.
  • 10 Million processed messages / month = $4.00 / month.

2. Producer-Consumer Architecture with TypeScript

Configuration (wrangler.toml):

toml
name = "job-processor"
main = "src/index.ts"
compatibility_date = "2026-09-01"

[[queues.producers]]
binding = "ORDER_QUEUE"
queue = "order-processing-queue"

[[queues.consumers]]
queue = "order-processing-queue"
max_batch_size = 10
max_batch_timeout = 5
max_retries = 3
dead_letter_queue = "order-dlq"

Application Code:

typescript
export interface Env {
  ORDER_QUEUE: Queue<OrderMessage>;
}

export interface OrderMessage {
  orderId: string;
  customerEmail: string;
  amount: number;
}

export default {
  // HTTP Producer Endpoint
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== 'POST') {
      return new Response('Method Not Allowed', { status: 405 });
    }

    const payload: OrderMessage = await request.json();

    // Push message to queue non-blockingly
    await env.ORDER_QUEUE.send(payload);

    return new Response(JSON.stringify({ status: 'queued', orderId: payload.orderId }), {
      status: 202,
      headers: { 'Content-Type': 'application/json' },
    });
  },

  // Batch Consumer Worker
  async queue(batch: MessageBatch<OrderMessage>, env: Env): Promise<void> {
    console.log(`Processing batch of ${batch.messages.length} messages`);

    for (const msg of batch.messages) {
      try {
        const order = msg.body;
        console.log(`Executing order processing for ${order.orderId}`);

        await executeJob(order);

        // Acknowledge successful execution
        msg.ack();
      } catch (err) {
        console.error(`Failed to process message ${msg.id}:`, err);
        // Retry message based on backoff policy
        msg.retry();
      }
    }
  },
};

async function executeJob(order: OrderMessage): Promise<void> {
  // Job logic here
}

3. Stateful Multi-Step Orchestration with Cloudflare Workflows

For business processes requiring durable sleeping, human approval steps, or multi-day retry routines:

typescript
import { WorkflowEntrypoint, WorkflowStep, WorkflowEvent } from 'cloudflare:workers';

type OnboardingPayload = {
  userId: string;
  email: string;
};

export class OnboardingWorkflow extends WorkflowEntrypoint<Env, OnboardingPayload> {
  async run(event: WorkflowEvent<OnboardingPayload>, step: WorkflowStep) {
    // Step 1: Provision account
    await step.do('create-user-db', async () => {
      return { id: event.payload.userId, status: 'PROVISIONED' };
    });

    // Step 2: Send welcome email
    await step.do('send-welcome-email', async () => {
      // Trigger transactional email
    });

    // Step 3: Sleep 48 hours without paying for idle compute!
    await step.sleep('wait-for-checkin', '48 hours');

    // Step 4: Send follow-up survey
    await step.do('send-survey-email', async () => {
      // Send engagement email
    });
  }
}

4. Strategic FinOps Advantages

  1. Automatic Batch Invocation: Process up to 100 messages per single Worker execution, reducing invocation billings by up to 90%.
  2. Zero Polling Surcharge: Unlike AWS SQS polling calls, Cloudflare operates an event-driven push architecture with no polling request fees.
  3. Integrated Dead Letter Queues: Prevent infinite retry billing loops by rerouting poisoned payloads automatically.

About the Author

huud

huud

@huud

About →

Systems architect and software engineer building high-performance distributed platforms.