← Back to Articles
#golang#microservices#reliability#distributed-systems

Resilient Go Microservices: Circuit Breakers, Rate Limiting, and Jitter

"Protect your Go microservices from cascading failures using Circuit Breakers, Token Bucket rate limiting, and Exponential Backoff with Jitter."

•By huud•
0 views
~4 min read
Resilient Go Microservices: Circuit Breakers, Rate Limiting, and Jitter

Resilient Go Microservices: Circuit Breakers, Rate Limiting, and Jitter

In distributed systems, failures are inevitable. Downstream databases will slow down, third-party APIs will experience intermittent outages, and network packets will drop.

Without defensive mechanisms, a slow downstream service causes callers to block, consuming connection pools and worker threads until the entire infrastructure collapses in a cascading outage.

This article covers three foundational stability patterns in Go: Circuit Breakers, Token Bucket Rate Limiting, and Exponential Backoff with Full Jitter.

1. The Circuit Breaker Pattern

A Circuit Breaker acts as an automatic switch with three states:

  1. Closed (Normal): Requests pass through. Failures increment an error counter.
  2. Open (Tripped): When error rate exceeds a threshold, all incoming requests fail immediately without calling the broken dependency.
  3. Half-Open (Testing): After a cooldown period, a limited probe traffic is allowed through to test if the dependency has recovered.
sql 
       ┌─────────── [ Error Rate > Threshold ] ───────────┐
       ▼                                                  │
 ┌───────────┐           Cooldown Elapsed          ┌─────────────┐
 │   OPEN    │ ──────────────────────────────────> │  HALF-OPEN  │
 └───────────┘                                     └─────────────┘
       ▲                                                  │
       │                   Success Rate > Target          ▼
       └────────────────────────────────────────── ┌─────────────┐
                                                   │   CLOSED    │
                                                   └─────────────┘

Implementing Circuit Breakers with sony/gobreaker

go 
package client

import (
	"context"
	"errors"
	"net/http"
	"time"

	"github.com/sony/gobreaker"
)

type ResilientHTTPClient struct {
	cb     *gobreaker.CircuitBreaker
	client *http.Client
}

func NewResilientClient() *ResilientHTTPClient {
	settings := gobreaker.Settings{
		Name:        "PaymentGateway",
		MaxRequests: 5,                // Concurrency allowed in Half-Open state
		Interval:    10 * time.Second, // Cyclic window to clear counters
		Timeout:     5 * time.Second,  // Duration to stay in Open state before Half-Open
		ReadyToTrip: func(counts gobreaker.Counts) bool {
			// Trip if at least 10 requests occurred and failure rate >= 40%
			failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
			return counts.Requests >= 10 && failureRatio >= 0.40
		},
	}

	return &ResilientHTTPClient{
		cb:     gobreaker.NewCircuitBreaker(settings),
		client: &http.Client{Timeout: 3 * time.Second},
	}
}

func (c *ResilientHTTPClient) Execute(ctx context.Context, req *http.Request) (*http.Response, error) {
	// Execute wraps the outbound call inside circuit breaker state logic
	body, err := c.cb.Execute(func() (any, error) {
		req = req.WithContext(ctx)
		resp, err := c.client.Do(req)
		if err != nil {
			return nil, err
		}
		if resp.StatusCode >= 500 {
			return nil, errors.New("downstream 5xx server error")
		}
		return resp, nil
	})

	if err != nil {
		return nil, err
	}
	return body.(*http.Response), nil
}

2. Rate Limiting with golang.org/x/time/rate

The Go standard extended library provides an industrial-grade Token Bucket rate limiter.

go 
package middleware

import (
	"net/http"
	"golang.org/x/time/rate"
)

type IPRateLimiter struct {
	limiter *rate.Limiter
}

func NewRateLimiter(r rate.Limit, b int) *IPRateLimiter {
	// r = tokens replenished per second, b = bucket burst size
	return &IPRateLimiter{
		limiter: rate.NewLimiter(r, b),
	}
}

func (rl *IPRateLimiter) Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Non-blocking token check
		if !rl.limiter.Allow() {
			http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
			return
		}
		next.ServeHTTP(w, r)
	})
}

3. Exponential Backoff with Full Jitter

When a distributed call fails, retrying immediately can overload the recovering server (thundering herd problem). Adding random noise (jitter) disperses retry spikes across time.

AWS "Full Jitter" Formula:

tsleep=random(0,min⁡(Tmax,Tbase×2attempt))t_{\text{sleep}} = \text{random}(0, \min(T_{\text{max}}, T_{\text{base}} \times 2^{\text{attempt}}))

go 
package retry

import (
	"context"
	"math/rand"
	"time"
)

func RetryWithJitter(
	ctx context.Context,
	attempts int,
	baseDelay time.Duration,
	maxDelay time.Duration,
	op func() error,
) error {
	var err error

	for i := 0; i < attempts; i++ {
		if err = op(); err == nil {
			return nil
		}

		// Calculate backoff ceiling: base * 2^i
		multiplier := 1 << i
		backoff := time.Duration(multiplier) * baseDelay
		if backoff > maxDelay {
			backoff = maxDelay
		}

		// Full Jitter: Sleep between [0, backoff]
		jitteredSleep := time.Duration(rand.Int63n(int64(backoff)))

		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(jitteredSleep):
		}
	}

	return err
}

Architecture Summary

Resiliency Layer Purpose Primary Tool
Circuit Breaker Stop hammering broken downstreams gobreaker
Token Bucket Limiter Protect internal resources from bursts golang.org/x/time/rate
Full Jitter Retry Prevent thundering herds on recovery Exponential backoff algorithm

Combining these three defensive primitives transforms fragile services into self-healing, highly available systems.

About the Author

huud

huud

@huud

About →

Systems architect and software engineer building high-performance distributed platforms.