Golang HTTP server middleware for gzip/brotli/zstandard compression

This is a small Go package which wraps HTTP handlers to transparently compress response bodies using zstd, brotli, gzip or deflate - for clients which support them. Although it's usually simpler to leave that to a reverse proxy (like nginx or Varnish), this package is useful when that is undesirable. In addition, this package allows users to extend it by plugging in third-party or custom compression encoders.

Note: This package was recently forked from the dead NYTimes/gziphandler. Maintaining drop-in compatibility is not a goal of this fork, as the scope of this fork is significantly wider than the original package.

⚠️ As we have not reached 1.0 yet, API is still subject to changes.

Features

• gzip, deflate, brotli, and zstd compression by default, alternate (faster) gzip, zstd implementations are optional
• Apply compression only if response body size is greater than a threshold
• Apply compression only to a allowlist/denylist of MIME content types
• Define encoding priority (e.g. give brotli a higher priority than gzip)
• Control whether the client or the server defines the encoder priority
• Plug in third-party/custom compression schemes or implementations
• Custom dictionary compression for zstd and deflate
• Low memory alliocations via transparent encoder reuse

Install

go get github.com/CAFxX/httpcompression

Usage

Call httpcompression.DefaultAdapter to get an adapter that can be used to wrap any handler (an object which implements the http.Handler interface), to transparently provide response body compression. Note that httpcompression automatically compresses using Zstandard, Brotli, Deflate, and Gzip depending on the capabilities of the client (Accept-Encoding) and the configuration of this handler (by default, Zstandard, Brotli and gzip are all enabled and, conditional on client support, used in that order of preference).

As a simple example:

package main

import (
    "io"
    "net/http"
    "github.com/CAFxX/httpcompression"
)

func main() {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain")
        io.WriteString(w, "Hello, World")
    })
    compress, _ := httpcompression.DefaultAdapter() // Use the default configuration
    http.Handle("/", compress(handler))
    http.ListenAndServe("0.0.0.0:8080", nil)
}

Pluggable compressors

It is possible to use custom compressor implementations by specifying a CompressorProvider for each of the encodings the adapter should support. This also allows to support arbitrary Content-Encoding schemes (e.g. lzma, or zstd with a static dictionary - see the examples).

pgz, err := httpcompression.Compressor("gzip", 0, pgzip.New(pgzip.Options{Level: 6}))
if err != nil {
    log.Fatal(err)
}
compress, err := httpcompression.Adapter(
    // use klauspost/pgzip as compressor for the "gzip" content-encoding, with priority 0
    pgz,
)
if err != nil {
    log.Fatal(err)
}
http.Handle("/", compress(handler))

The contrib/ directory contains a number of bundled implementations that are ready for use:

Content-Encoding	Provider package	Implementation package	Notes	Dictionary	Go/cgo	Default	IANA registry
deflate	contrib/compress/zlib	compress/zlib	Slower than klauspost/zlib	✅	Go	✅	✅
deflate	contrib/klauspost/zlib	github.com/klauspost/compress/zlib		✅	Go		✅
gzip	contrib/compress/gzip	compress/gzip	Slower than klauspost/gzip		Go	✅	✅
gzip	contrib/klauspost/gzip	github.com/klauspost/compress/gzip			Go		✅
gzip	contrib/klauspost/pgzip	github.com/klauspost/pgzip	Parallel compression		Go		✅
zstd	contrib/klauspost/zstd	github.com/klauspost/compress/zstd		✅	Go	✅	✅
zstd	contrib/valyala/gozstd	github.com/valyala/gozstd	Slower than klauspost/zstd	✅	cgo		✅
brotli	contrib/andybalholm/brotli	github.com/andybalholm/brotli	Slower than google/brotli		Go	✅	✅
brotli	contrib/google/cbrotli	github.com/google/brotli	Requires brotli libraries to be installed		cgo		✅
lz4	contrib/pierrec/lz4	github.com/pierrec/lz4/v4			Go
xz	contrib/ulikunitz/xz	github.com/ulikunitz/xz			Go

Framework integration

In addition to the default support for net/http, httpcompression provides adapters for the following web frameworks:

Framework	Adapter
github.com/gofiber/fiber/v2	contrib/gofiber/fiber/v2
github.com/labstack/echo	contrib/labstack/echo
github.com/gin-gonic/gin	contrib/gin-gonic/gin

Benchmark

See the benchmark results to get an idea of the relative performance and compression efficiency of gzip, brotli and zstd in the current implementation.

TODO

• Add dictionary support to brotli (zstd and deflate already support it, gzip does not allow dictionaries)
• Allow to choose dictionary based on content-type
• Provide additional implementations based on the bindings to the original native implementations
• Add compressed payload caching (if the same payload has already been compressed and is present in the cache, skip compression)
• Add write buffering (compress larger chunks at once)
• Add decompression (if the payload is already compressed but the client supports better algorithms, or does not support a certain algorithm)
• Add other, non-standardized content encodings (lzma/lzma2/xz, snappy, bzip2, etc.)
• Dynamically tune MinSize (and possibly also ContentTypes, level/quality, ...)
• Automatically generate and serve dictionaries

License

Apache 2.0.