Serverless Background Job Processing with Cloudflare Queues and Workflows
"Build cost-effective asynchronous background job processing using Cloudflare Queues and Workflows without idle servers."
Serverless Background Job Processing with Cloudflare Queues and Workflows
Asynchronous job execution is essential for building reliable backend systems: transactional emails, invoice generation, webhook dispatching, and search indexing.
Traditional implementations on AWS combine Amazon SQS with EC2/ECS Worker Auto-Scaling Groups or AWS Step Functions. However, keeping EC2 instances running 24/7 to poll message queues produces significant idle compute waste. Furthermore, AWS Step Functions charges per state transition, leading to high bills for multi-step orchestrations.
Cloudflare Queues and Cloudflare Workflows deliver an anti-idle, serverless event-processing model that eliminates standby costs entirely.
1. Quantifying Idle Waste in Traditional Queue Workers
Consider a typical message worker setup on AWS:
- Message Broker: AWS SQS Standard Queue
- Consumers: 2x
t4g.smallEC2 instances in an Auto-Scaling Group for continuous availability.
AWS Monthly Baseline:
- 2x EC2
t4g.smallinstances (Always-on 730 hrs): ~$24.50 / month - AWS SQS (10 Million API requests): ~$4.00 / month
- VPC Peering & NAT Data Processing: ~$15.00 / month
- Total Standby AWS Spend: ~$43.50 / month (even when 0 jobs are queued).
Cloudflare Queues Baseline:
- Consumer Workers are triggered on-demand upon message arrival. Zero messages = zero active compute cost.
- Price: $0.40 per 1,000,000 operations.
- 10 Million processed messages / month = $4.00 / month.
2. Producer-Consumer Architecture with TypeScript
Configuration (wrangler.toml):
name = "job-processor"
main = "src/index.ts"
compatibility_date = "2026-09-01"
[[queues.producers]]
binding = "ORDER_QUEUE"
queue = "order-processing-queue"
[[queues.consumers]]
queue = "order-processing-queue"
max_batch_size = 10
max_batch_timeout = 5
max_retries = 3
dead_letter_queue = "order-dlq"
Application Code:
export interface Env {
ORDER_QUEUE: Queue<OrderMessage>;
}
export interface OrderMessage {
orderId: string;
customerEmail: string;
amount: number;
}
export default {
// HTTP Producer Endpoint
async fetch(request: Request, env: Env): Promise<Response> {
if (request.method !== 'POST') {
return new Response('Method Not Allowed', { status: 405 });
}
const payload: OrderMessage = await request.json();
// Push message to queue non-blockingly
await env.ORDER_QUEUE.send(payload);
return new Response(JSON.stringify({ status: 'queued', orderId: payload.orderId }), {
status: 202,
headers: { 'Content-Type': 'application/json' },
});
},
// Batch Consumer Worker
async queue(batch: MessageBatch<OrderMessage>, env: Env): Promise<void> {
console.log(`Processing batch of ${batch.messages.length} messages`);
for (const msg of batch.messages) {
try {
const order = msg.body;
console.log(`Executing order processing for ${order.orderId}`);
await executeJob(order);
// Acknowledge successful execution
msg.ack();
} catch (err) {
console.error(`Failed to process message ${msg.id}:`, err);
// Retry message based on backoff policy
msg.retry();
}
}
},
};
async function executeJob(order: OrderMessage): Promise<void> {
// Job logic here
}
3. Stateful Multi-Step Orchestration with Cloudflare Workflows
For business processes requiring durable sleeping, human approval steps, or multi-day retry routines:
import { WorkflowEntrypoint, WorkflowStep, WorkflowEvent } from 'cloudflare:workers';
type OnboardingPayload = {
userId: string;
email: string;
};
export class OnboardingWorkflow extends WorkflowEntrypoint<Env, OnboardingPayload> {
async run(event: WorkflowEvent<OnboardingPayload>, step: WorkflowStep) {
// Step 1: Provision account
await step.do('create-user-db', async () => {
return { id: event.payload.userId, status: 'PROVISIONED' };
});
// Step 2: Send welcome email
await step.do('send-welcome-email', async () => {
// Trigger transactional email
});
// Step 3: Sleep 48 hours without paying for idle compute!
await step.sleep('wait-for-checkin', '48 hours');
// Step 4: Send follow-up survey
await step.do('send-survey-email', async () => {
// Send engagement email
});
}
}
4. Strategic FinOps Advantages
- Automatic Batch Invocation: Process up to 100 messages per single Worker execution, reducing invocation billings by up to 90%.
- Zero Polling Surcharge: Unlike AWS SQS polling calls, Cloudflare operates an event-driven push architecture with no polling request fees.
- Integrated Dead Letter Queues: Prevent infinite retry billing loops by rerouting poisoned payloads automatically.
About the Author
huud
@huud
Systems architect and software engineer building high-performance distributed platforms.