← Back to Articles
#golang#concurrency#backend#performance

Production Go Concurrency: Worker Pools, Contexts, and Preventing Goroutine Leaks

"Production guide to robust Go concurrency: bounded worker pools, context cancellation, and goroutine leak prevention."

•By huud•
0 views
~3 min read
Production Go Concurrency: Worker Pools, Contexts, and Preventing Goroutine Leaks

Production Go Concurrency: Worker Pools, Contexts, and Preventing Goroutine Leaks

Goroutines are lightweight (~2 KB initial stack), but unconstrained go func() invocations frequently cause out-of-memory crashes (OOM) and deadlocks in production systems.

1. The Anti-Pattern: Unbounded Goroutines

Spawning a new goroutine per request or dataset item without bounds creates critical resource exhaustion risks.

Bad:

go 
func handleRequests(jobs []Job) {
    for _, job := range jobs {
        // Danger: Spawns 100,000 goroutines concurrently if slice is large
        // Leads to severe CPU throttling and RAM exhaustion
        go process(job)
    }
}

2. Solution: Bounded Worker Pool Pattern

Implement a bounded worker pool utilizing buffered channels and sync.WaitGroup.

go 
package main

import (
	"context"
	"sync"
	"time"
)

type Job struct {
	ID   int
	Data string
}

type Result struct {
	JobID int
	Err   error
}

func worker(ctx context.Context, id int, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done()

	for {
		select {
		case <-ctx.Done():
			return // Context cancelled, exit gracefully
		case job, ok := <-jobs:
			if !ok {
				return // Channel closed
			}

			err := processJob(job)

			select {
			case results <- Result{JobID: job.ID, Err: err}:
			case <-ctx.Done():
				return
			}
		}
	}
}

func processJob(j Job) error {
	time.Sleep(50 * time.Millisecond)
	return nil
}

func RunPool(ctx context.Context, jobList []Job, numWorkers int) []Result {
	jobs := make(chan Job, len(jobList))
	results := make(chan Result, len(jobList))
	var wg sync.WaitGroup

	// 1. Spawn fixed number of workers
	for w := 1; w <= numWorkers; w++ {
		wg.Add(1)
		go worker(ctx, w, jobs, results, &wg)
	}

	// 2. Feed jobs
	for _, job := range jobList {
		jobs <- job
	}
	close(jobs)

	// 3. Wait and close results channel in background
	go func() {
		wg.Wait()
		close(results)
	}()

	// 4. Collect results
	var out []Result
	for res := range results {
		out = append(out, res)
	}

	return out
}

3. Preventing Goroutine Leaks with Context

A goroutine leak happens when a goroutine remains blocked indefinitely waiting on a channel without active senders or receivers.

Blocked Sender Leak:

go 
// LEAK: If ctx timeouts, sender goroutine blocks forever on unbuffered channel
func queryExternalAPI(ctx context.Context) (string, error) {
    ch := make(chan string) // Unbuffered

    go func() {
        data := fetchHTTP()
        ch <- data // BLOCKS if receiver already timed out and returned
    }()

    select {
    case <-ctx.Done():
        return "", ctx.Err()
    case res := <-ch:
        return res, nil
    }
}

Fix: Use Buffered Channel of Size 1

go 
func queryExternalAPI(ctx context.Context) (string, error) {
    ch := make(chan string, 1) // Buffer allows send without active receiver

    go func() {
        data := fetchHTTP()
        ch <- data // Writes to buffer and exits safely
    }()

    select {
    case <-ctx.Done():
        return "", ctx.Err()
    case res := <-ch:
        return res, nil
    }
}

4. Leak Detection in Tests

Integrate go.uber.org/goleak in unit tests to ensure no dangling goroutines remain:

go 
package mypkg_test

import (
	"testing"
	"go.uber.org/goleak"
)

func TestConcurrentProcessing(t *testing.T) {
	defer goleak.VerifyNone(t)

	// Execute concurrent logic here
}

Summary

  1. Restrict concurrent execution with Worker Pools instead of uncontrolled go func().
  2. Always propagate context.Context for cancellations and timeouts.
  3. Use buffered channels for asynchronous result handoffs to prevent sender blocking.
  4. Enforce goleak checks in CI suites.

About the Author

huud

huud

@huud

About →

Systems architect and software engineer building high-performance distributed platforms.