← Back to Articles
#golang#devops#kubernetes#reliability#backend

Zero-Downtime Deployments: Graceful Shutdown in Go HTTP and gRPC Services

"Eliminate deployment downtime and dropped requests by mastering graceful shutdown patterns in Go HTTP and gRPC services."

•By huud•
0 views
~4 min read
Zero-Downtime Deployments: Graceful Shutdown in Go HTTP and gRPC Services

Zero-Downtime Deployments: Graceful Shutdown in Go HTTP and gRPC Services

When a modern web service deploys a new release in Kubernetes or on virtual servers, the process receives an OS termination signal (SIGTERM or SIGINT).

If your Go service exits immediately (os.Exit(0)), in-flight user requests are terminated mid-flight, database transactions are aborted, and background batches are corrupted.

Implementing a Graceful Shutdown pattern guarantees that the application stops accepting new traffic, drains existing connections, finishes active transactions, and exits cleanly.

The Shutdown Lifecycle

A graceful shutdown sequence follows four deterministic steps:

  1. Signal Interception: Trap SIGTERM and SIGINT via an OS notification channel.
  2. Stop Ingress: Close the listening port so load balancers stop routing new traffic.
  3. Drain In-Flight Requests: Allow currently active HTTP/gRPC handlers to complete.
  4. Clean up Resources: Flush log buffers, close database connection pools, and release distributed locks.
css 
OS Signal (SIGTERM)
        │
        ▼
[ Intercept Signal ]
        │
        ▼
[ Stop New Connections ] ──> Load Balancer Routes to Other Pods
        │
        ▼
[ Drain Active Handlers (Deadline: 15s) ]
        │
        ▼
[ Close DB Pools & Flush Logs ]
        │
        ▼
[ Process Exit (Code 0) ]

Idiomatic Go HTTP Graceful Shutdown (Go 1.16+ signal.NotifyContext)

go 
package main

import (
	"context"
	"errors"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

	mux := http.NewServeMux()
	mux.HandleFunc("/work", func(w http.ResponseWriter, r *http.Request) {
		// Simulate long I/O operation
		time.Sleep(3 * time.Second)
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("Work complete"))
	})

	server := &http.Server{
		Addr:         ":8080",
		Handler:      mux,
		ReadTimeout:  5 * time.Second,
		WriteTimeout: 10 * time.Second,
		IdleTimeout:  120 * time.Second,
	}

	// 1. Listen for SIGINT (Ctrl+C) and SIGTERM (Kubernetes pod termination)
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	// 2. Run server in background goroutine
	serverErrors := make(chan error, 1)
	go func() {
		logger.Info("server starting", "addr", server.Addr)
		if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			serverErrors <- err
		}
	}()

	// 3. Block until OS signal or startup error
	select {
	case err := <-serverErrors:
		logger.Error("server fatal startup error", "err", err)
		os.Exit(1)
	case <-ctx.Done():
		logger.Info("shutdown signal received; commencing drain")
	}

	// 4. Create timeout context for the shutdown phase
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel()

	// 5. Shutdown drains in-flight connections
	if err := server.Shutdown(shutdownCtx); err != nil {
		logger.Error("server forced to shutdown due to timeout", "err", err)
		_ = server.Close() // Force-kill sockets
	}

	// 6. Cleanup dependencies (DB connections, caches)
	cleanupDependencies()

	logger.Info("graceful shutdown complete; process exiting")
}

func cleanupDependencies() {
	// db.Close(), redis.Close(), etc.
}

Graceful Shutdown in gRPC Servers

gRPC provides built-in support for graceful drainage via GracefulStop():

go 
package grpcserver

import (
	"log/slog"
	"net"
	"google.golang.org/grpc"
)

func RunGRPC(listener net.Listener, server *grpc.Server, stopChan <-chan struct{}) {
	go func() {
		if err := server.Serve(listener); err != nil {
			slog.Error("gRPC server error", "err", err)
		}
	}()

	// Wait for shutdown trigger
	<-stopChan

	slog.Info("stopping gRPC server gracefully")
	// GracefulStop stops accepting new RPCs and blocks until active RPCs complete
	server.GracefulStop()
	slog.Info("gRPC server stopped")
}

Production Pitfalls to Avoid

1. Hardcoding Insufficient Timeouts

In Kubernetes, pods have a terminationGracePeriodSeconds (default: 30s). Ensure your Go context.WithTimeout(..., 15*time.Second) is smaller than Kubernetes' grace period, giving time for DB disconnects before Kubernetes issues a SIGKILL.

2. Not Handling Context in Handlers

If an HTTP request takes 20 seconds, but your shutdown timeout is 10 seconds, server.Shutdown cancels the request context. Your database queries and HTTP client calls inside the handler must listen to r.Context().Done() to terminate early.

Conclusion

Graceful shutdown is not an optional polish—it is a critical requirement for continuous deployment without customer-facing 502 errors or corrupted database state. Using signal.NotifyContext and server.Shutdown gives your Go applications enterprise-grade reliability.

About the Author

huud

huud

@huud

About →

Systems architect and software engineer building high-performance distributed platforms.