Skip to content

Latest commit

 

History

History
142 lines (112 loc) · 10 KB

README.md

File metadata and controls

142 lines (112 loc) · 10 KB
Raw

Golang HTTP server middleware for gzip/brotli/zstandard compression

Build status codecov Go Report CodeFactor Go Reference

This is a small Go package which wraps HTTP handlers to transparently compress response bodies using zstd, brotli, gzip or deflate - for clients which support them. Although it's usually simpler to leave that to a reverse proxy (like nginx or Varnish), this package is useful when that is undesirable. In addition, this package allows users to extend it by plugging in third-party or custom compression encoders.

Note: This package was recently forked from the dead NYTimes/gziphandler. Maintaining drop-in compatibility is not a goal of this fork, as the scope of this fork is significantly wider than the original package.

⚠️ As we have not reached 1.0 yet, API is still subject to changes.

Features

  • gzip, deflate, brotli, and zstd compression by default, alternate (faster) gzip, zstd implementations are optional
  • Apply compression only if response body size is greater than a threshold
  • Apply compression only to a allowlist/denylist of MIME content types
  • Define encoding priority (e.g. give brotli a higher priority than gzip)
  • Control whether the client or the server defines the encoder priority
  • Plug in third-party/custom compression schemes or implementations
  • Custom dictionary compression for zstd and deflate
  • Low memory alliocations via transparent encoder reuse

Install

go get github.com/CAFxX/httpcompression

Usage

Call httpcompression.DefaultAdapter to get an adapter that can be used to wrap any handler (an object which implements the http.Handler interface), to transparently provide response body compression. Note that httpcompression automatically compresses using Zstandard, Brotli, Deflate, and Gzip depending on the capabilities of the client (Accept-Encoding) and the configuration of this handler (by default, Zstandard, Brotli and gzip are all enabled and, conditional on client support, used in that order of preference).

As a simple example:

package main

import (
    "io"
    "net/http"
    "github.com/CAFxX/httpcompression"
)

func main() {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain")
        io.WriteString(w, "Hello, World")
    })
    compress, _ := httpcompression.DefaultAdapter() // Use the default configuration
    http.Handle("/", compress(handler))
    http.ListenAndServe("0.0.0.0:8080", nil)
}

Pluggable compressors

It is possible to use custom compressor implementations by specifying a CompressorProvider for each of the encodings the adapter should support. This also allows to support arbitrary Content-Encoding schemes (e.g. lzma, or zstd with a static dictionary - see the examples).

pgz, err := httpcompression.Compressor("gzip", 0, pgzip.New(pgzip.Options{Level: 6}))
if err != nil {
    log.Fatal(err)
}
compress, err := httpcompression.Adapter(
    // use klauspost/pgzip as compressor for the "gzip" content-encoding, with priority 0
    pgz,
)
if err != nil {
    log.Fatal(err)
}
http.Handle("/", compress(handler))

The contrib/ directory contains a number of bundled implementations that are ready for use:

Content-Encoding Provider package Implementation package Notes Dictionary Go/cgo Default IANA registry
deflate contrib/compress/zlib compress/zlib Slower than klauspost/zlib ✅ Go ✅ ✅
deflate contrib/klauspost/zlib github.com/klauspost/compress/zlib ✅ Go ✅
gzip contrib/compress/gzip compress/gzip Slower than klauspost/gzip Go ✅ ✅
gzip contrib/klauspost/gzip github.com/klauspost/compress/gzip Go ✅
gzip contrib/klauspost/pgzip github.com/klauspost/pgzip Parallel compression Go ✅
zstd contrib/klauspost/zstd github.com/klauspost/compress/zstd ✅ Go ✅ ✅
zstd contrib/valyala/gozstd github.com/valyala/gozstd Slower than klauspost/zstd ✅ cgo ✅
brotli contrib/andybalholm/brotli github.com/andybalholm/brotli Slower than google/brotli Go ✅ ✅
brotli contrib/google/cbrotli github.com/google/brotli Requires brotli libraries to be installed cgo ✅
lz4 contrib/pierrec/lz4 github.com/pierrec/lz4/v4 Go
xz contrib/ulikunitz/xz github.com/ulikunitz/xz Go

Framework integration

In addition to the default support for net/http, httpcompression provides adapters for the following web frameworks:

Framework Adapter
github.com/gofiber/fiber/v2 contrib/gofiber/fiber/v2
github.com/labstack/echo contrib/labstack/echo
github.com/gin-gonic/gin contrib/gin-gonic/gin

Benchmark

See the benchmark results to get an idea of the relative performance and compression efficiency of gzip, brotli and zstd in the current implementation.

TODO

  • Add dictionary support to brotli (zstd and deflate already support it, gzip does not allow dictionaries)
  • Allow to choose dictionary based on content-type
  • Provide additional implementations based on the bindings to the original native implementations
  • Add compressed payload caching (if the same payload has already been compressed and is present in the cache, skip compression)
  • Add write buffering (compress larger chunks at once)
  • Add decompression (if the payload is already compressed but the client supports better algorithms, or does not support a certain algorithm)
  • Add other, non-standardized content encodings (lzma/lzma2/xz, snappy, bzip2, etc.)
  • Dynamically tune MinSize (and possibly also ContentTypes, level/quality, ...)
  • Automatically generate and serve dictionaries

License

Apache 2.0.