Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support passing FDs (socket activation) #6296

Open
flokli opened this issue May 3, 2024 · 42 comments
Open

Support passing FDs (socket activation) #6296

flokli opened this issue May 3, 2024 · 42 comments
Labels
discussion 💬 The right solution needs to be found feature ⚙️ New feature or request

Comments

@flokli
Copy link

flokli commented May 3, 2024

I'd like to use caddy in a socket-activated environments, using FDs passed down from the service manager, rather than binding on addresses on its own.

Combined with signalling readyness (which caddy already does), this will give zero-downtime (re)deployments on Linux systems using systemd (if .socket files are used), by simply restarting the process - the socket is held open by systemd, and new connections are passed in once caddy is ready to accept new requests. In these cases, there wouldn't be a need for complicated reload logic anymore.

github.com/coreos/go-systemd/activation provides the necessary methods to check whether FDs are passed, including identifying them by their socket name. https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket-activation gives a nice introduction into the feature itself.

In case no explicit listen addresses are specified, caddy could default to do that rather than binding on its own, if it detects it's running in such an environment.
Additionally, Caddyfile could be extended to allow specifying these passed fds as network addresses (something like sd-listen:$name or sd-listen:$idx maybe). This can become useful when you want to expose different things on different sockets.

@francislavoie
Copy link
Member

francislavoie commented May 3, 2024

Are you looking for the bind directive? https://caddyserver.com/docs/caddyfile/directives/bind

And see https://caddyserver.com/docs/conventions#network-addresses, you can use unix sockets in reverse_proxy upstreams.

I'm not sure what you're asking for if not that.

@mholt
Copy link
Member

mholt commented May 3, 2024

It sounds like what is being asked for is graceful upgrades/restarts.

Caddy 1 had this feature, and I quite liked how it worked: pass the socket directly to the next process. It worked on all Unix systems without relying on a separate system service, and it was smart enough to understand Caddy configuration: if the new config didn't use a socket, it wouldn't be kept; rather than blindly moving all the sockets over.

I'd probably rather bring the implementation from Caddy 1 into Caddy 2.

@mholt mholt added feature ⚙️ New feature or request discussion 💬 The right solution needs to be found labels May 3, 2024
@flokli
Copy link
Author

flokli commented May 3, 2024

It sounds like what is being asked for is graceful upgrades/restarts.

No, getting this for free is only one side-effect of supporting socket-activation.

socket-activation will also cause caddy to get started lazily whenever the first connection to the (externally configured) socket address happens, which simplifies declaring service dependencies too.

The article linked from my link elaborates a bit more on this.

Caddy 1 had this feature, and I quite liked how it worked: pass the socket directly to the next process.

This still requires caddy to do manual coordination with its new process and pass it around explicitly. The point of simply taking the FDs passed by the service manager is that caddy does not have to be aware of whether it's the first process being started on the system, or you start a new version with another config. caddy simply gets an FD, where new connections will appear on.

@flokli
Copy link
Author

flokli commented May 3, 2024

Ah yes, and because caddy just takes FDs, it doesn't need to bind() on its own, which allows applying stronger sandboxing from the outside.

@mholt
Copy link
Member

mholt commented May 3, 2024

@flokli

This still requires caddy to do manual coordination with its new process and pass it around explicitly. The point of simply taking the FDs passed by the service manager is that caddy does not have to be aware of whether it's the first process being started on the system, or you start a new version with another config. caddy simply gets an FD, where new connections will appear on.

But what is Caddy supposed to do with that socket? How does it know the configuration associated with it? You can't just hand a server a socket and expect it to know what to do with it, without any configuration... maybe I am missing something about how it works.

@flokli
Copy link
Author

flokli commented May 3, 2024

Sockets can have names attached (so the user can name them http and https for example, or api and metrics), and we could add a syntax to refer to them via these names in Caddyfile. I could say I want a http server on sd-listen:http, which would then expect a listener named http to be passed to caddy.

All these passed FDs also give you a net.Listener interface, so even without explicit config caddy could still check the properties of it and apply some heuristics too (detect port 80 and 443 if you got two unnamed TCP sockets), if we want to apply some out-of-the-box behaviour in these scenarios. But getting the basic support for it (using an externally-passed FD by its name/index) and defining the syntax for it would be a nice first step.

You can play around with this through systemd-socket-activate -l 8088 -l 8089 --fdname=foo:bar -- /path/to-caddy, which will give you two TCP sockets listening on the two ports, named foo and bar.

@mholt
Copy link
Member

mholt commented May 3, 2024

Oh I see, so you'd still have your Caddy config, you'd just specify a different network name for the listener address, and Caddy will then get it from the service manager rather than binding a new socket.

@flokli
Copy link
Author

flokli commented May 3, 2024

Yes! Or well, I don't want caddy to do any bind on its own at all, but pass in every socket via this mechanism.

@mholt
Copy link
Member

mholt commented May 3, 2024

In that case you can use bind in your site blocks to get the socket from the service manager. We'd just need to implement a package that calls caddy.RegisterNetwork(). For example the caddy-tailscale package does this so that Tailscale can provide a listener.

Anyone is welcome to pick this up.

@WeidiDeng
Copy link
Member

@mholt I did some experiments with registering custom network, it's too much trouble to be worth it. Every site block needs an explicit bind and that includes http port and http3 udp socket.

@flokli I'm thinking on unix, we can try preferring socket activation but fallback to the old behavior. What do you think of it? Or should caddy just exit unsuccessfully if socket activation environments variables are found but not sockets matching listening critertia are found? Or if some warning logs are emitted?

As mentioned above, you are responsible to pass every socket yourself, including 80 tcp and 443 udp if auto http->https and http3 are enabled respectively. And admin socket if enabled as well. Assuming you restart caddy instead of reload it.

@climba03003
Copy link

I would really see it happen and it can greatly reduce my network stack complexity.
Currently, I am have two caddy in front of server and I face a lot of instability because of podman networking.
I change to using socket to see if it works better (no more DNS resolution).

flowchart TD
    A[Caddy] -->|Reverse Proxy| B{Container Network}
    B -->|Serve Frontend| C[Caddy]
    C -->|Reverse Proxy| D[Server]
Loading

When socket activation becomes a thing, it can also reduce resources usage. Because the middle caddy can be terminated when no one connected for some time. If the outer one can be socket activated, it will directly pass the socket to inner one and benefit of direct network connection.

@flokli
Copy link
Author

flokli commented May 6, 2024

@mholt I did some experiments with registering custom network, it's too much trouble to be worth it. Every site block needs an explicit bind and that includes http port and http3 udp socket.

@flokli I'm thinking on unix, we can try preferring socket activation but fallback to the old behavior. What do you think of it? Or should caddy just exit unsuccessfully if socket activation environments variables are found but not sockets matching listening critertia are found? Or if some warning logs are emitted?

I think ti makes sense to first land the feature with explicit configuration, which might mean explicit bind statements, and once that's in, think about having more opinionated defaults in case we are in a socket-activated environment.

The good thing is, it's pretty safe to detect whether caddy is running in a socket-activated environment or not, so we are able to change defaults in this case, without breaking existing usecases.

@WeidiDeng
Copy link
Member

@flokli So that means you're fine with mixing passing FD and current binding behavior? And since you will use bind explicitly, it's an error to bind to an non existent FD.

The problem with names is that one name can map to many sockets with different addresses, how do you think caddy handle this situation?

@eliasp
Copy link

eliasp commented May 6, 2024

Until this is implemented: for those that just care about binding to ports <1024 AND not running Caddy as root, can use systemd's SocketBindAllow= (available since systemd 249)

@mohammed90
Copy link
Member

for those that just care about binding to ports <1024 AND not running Caddy as root,

There was never a need to run Caddy as root on Linux. Our standard systemd unit file is shipped with CAP_NET_BIND_SERVICE which allows the service to run without root. The SocketBindAllow and SocketBindDeny allows further restriction to specific ports rather than any port below 1024.

@flokli
Copy link
Author

flokli commented May 8, 2024

I'm aware of CAP_NET_BIND_SERVICE to allow non-root processes to bind to lower ports, that's not why I'm advocating for this feature.

Giving the option to move the whole socket binding business entirely out of caddy is what I'm advocating for, both from a sandboxing (it doesn't need to be allowed to bind() if it doesn't have to, it doesn't even need to have access to the network namespace the bind happens in) and zero downtime restart/configuration update controlled by the service manager.

@flokli So that means you're fine with mixing passing FD and current binding behavior? And since you will use bind explicitly, it's an error to bind to an non existent FD.

Yes, I think the bind syntax should be extended, to allow specifying "use this passed FD rather than binding yourself". Slightly unfortunate name, but well 🤷.

This would also mean, caddy would still bind on its own where we don't explicitly configure it to use the FD(s).

The problem with names is that one name can map to many sockets with different addresses, how do you think caddy handle this situation?

Indeed FileDescriptorName= describes such name applies to all sockets in that .socket file, so sd-listen:http would could identify multiple FDs, not just a single one.

I think I'd be fine landing support for having to explicitly use bind statements everywhere first, working out the syntax for it, and once that's stabilized, I'd think about how a nice out-of-the-box behaviour could look like, if caddy detects it is running in a socket activated scenario.

@WeidiDeng
Copy link
Member

@flokli You can try it with a plugin for now, xcaddy build --with github.com/WeidiDeng/caddy-socket-activation. Let me know what you think.

@caddyserver caddyserver deleted a comment from Karanvarm May 13, 2024
@balki
Copy link

balki commented May 22, 2024

I wrote a small go library to listen on socket activated fds.
https://github.com/balki/anyhttp/blob/main/anyhttp.go#L147

@francislavoie
Copy link
Member

Interesting, this could be turned into a Caddy plugin by using caddy.RegisterNetwork() @balki

@flokli
Copy link
Author

flokli commented May 25, 2024

Interesting, this could be turned into a Caddy plugin by using caddy.RegisterNetwork() @balki

Isn't that exactly what @WeidiDeng's plugin already does? https://github.com/WeidiDeng/caddy-socket-activation/blob/2246ae4a7a00955926ebdf1d557c1530b327bf2f/tcp.go#L12

(I'll try it now, was quite busy before. Did not forget, sorry! Will report back here)

I wrote a small go library to listen on socket activated fds. https://github.com/balki/anyhttp/blob/main/anyhttp.go#L147

There's also github.com/coreos/go-systemd/activation, linked in the initial issue description, which seems a bit more commonly used.

@flokli
Copy link
Author

flokli commented May 25, 2024

I gave https://github.com/WeidiDeng/caddy-socket-activation a try (after WeidiDeng/caddy-socket-activation#1).

Some notes:

I'm using caddy alongside the acme-dns plugin, so don't really need to bind on port 80, however it seems to be quite hard (?) to not bind there (requires disabling autossl) or reconfigure this to another bind, using the new socket-activation functionality (is this even documented at all?).

I ignored the http problem (letting caddy bind on http on its own), and configured a .socket file binding on port 433 tcp (and udp, for http3):

# /etc/systemd/system/caddy.socket
[Unit]

[Socket]
FileDescriptorName=https
ListenDatagram=[::]:443
ListenStream=[::]:443

[Install]
WantedBy=sockets.target

systemd starts up caddy on the first connection, however caddy fails to pick up the FD names, or at least, in some cases. I managed to reproduce the flaky behaviour with systemd-socket-activate too:

[root@n4-rk1:~]# systemd-socket-activate -l '[::]:443' --fdname https /nix/store/fdj05bym30gxrqsn66g1q709d0pirirj-caddy-2.7.6/bin/caddy run --config ./caddy_config --adapter caddyfile
Listening on [::]:443 as 3.
Communication attempt on fd 3.
Execing /nix/store/fdj05bym30gxrqsn66g1q709d0pirirj-caddy-2.7.6/bin/caddy (/nix/store/fdj05bym30gxrqsn66g1q709d0pirirj-caddy-2.7.6/bin/caddy run --config ./caddy_config --adapter caddyfile)
2024/05/25 14:43:57.224	INFO	using provided configuration	{"config_file": "./caddy_config", "config_adapter": "caddyfile"}
2024/05/25 14:43:57.232	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "./caddy_config", "line": 3}
2024/05/25 14:43:57.234	WARN	admin	admin endpoint disabled
2024/05/25 14:43:57.235	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x400053a180"}
2024/05/25 14:43:57.236	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2024/05/25 14:43:57.236	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024/05/25 14:43:57.238	WARN	tls	unable to get instance ID; storage clean stamps will be incomplete	{"error": "open /root/.local/share/caddy/instance.uuid: no such file or directory"}
2024/05/25 14:43:57.238	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/25 14:43:57.238	INFO	tls.cache.maintenance	stopped background certificate maintenance	{"cache": "0x400053a180"}
Error: loading initial config: loading new config: http app module: start: listening on socket-activation/https:443: no file descriptors passed

[root@n4-rk1:~]# systemd-socket-activate -l '[::]:443' --fdname https /nix/store/fdj05bym30gxrqsn66g1q709d0pirirj-caddy-2.7.6/bin/caddy run --config ./caddy_config --adapter caddyfile
Listening on [::]:443 as 3.
Communication attempt on fd 3.
Execing /nix/store/fdj05bym30gxrqsn66g1q709d0pirirj-caddy-2.7.6/bin/caddy (/nix/store/fdj05bym30gxrqsn66g1q709d0pirirj-caddy-2.7.6/bin/caddy run --config ./caddy_config --adapter caddyfile)
2024/05/25 14:44:10.173	INFO	using provided configuration	{"config_file": "./caddy_config", "config_adapter": "caddyfile"}
2024/05/25 14:44:10.176	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "./caddy_config", "line": 3}
2024/05/25 14:44:10.176	WARN	admin	admin endpoint disabled
2024/05/25 14:44:10.177	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x4000551200"}
2024/05/25 14:44:10.177	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2024/05/25 14:44:10.179	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024/05/25 14:44:10.182	INFO	http	enabling HTTP/3 listener	{"addr": "https:443"}
2024/05/25 14:44:11.945	INFO	[INFO][FileStorage:/root/.local/share/caddy] /root/.local/share/caddy/locks/storage_clean.lock: Empty lockfile (EOF) - likely previous process crashed or storage medium failure; treating as stale
2024/05/25 14:44:11.945	INFO	[INFO][FileStorage:/root/.local/share/caddy] Lock for 'storage_clean' is stale (created: 0001-01-01 00:00:00 +0000 UTC, last update: 0001-01-01 00:00:00 +0000 UTC); removing then retrying: /root/.local/share/caddy/locks/storage_clean.lock
2024/05/25 14:44:11.950	INFO	tls	cleaning storage unit	{"storage": "FileStorage:/root/.local/share/caddy"}
2024/05/25 14:44:11.951	INFO	tls	finished cleaning storage units
2024/05/25 14:44:13.995	INFO	tls.cache.maintenance	stopped background certificate maintenance	{"cache": "0x4000551200"}
Error: loading initial config: loading new config: http app module: start: starting HTTP/3 QUIC listener: listen udp: lookup https: no such host

This might also be related to systemd-socket-activate not allowing to listen on tcp and udp simultaneously, but I got the same messages when run as a systemd service:

May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"warn","ts":1716647221.6430447,"logger":"admin","msg":"admin endpoint disabled"}
May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"info","ts":1716647221.6433222,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000393200"}
May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"info","ts":1716647221.643876,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"info","ts":1716647221.643922,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"info","ts":1716647221.64457,"logger":"http","msg":"enabling HTTP/3 listener","addr":"https:443"}
May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"warn","ts":1716647221.6502545,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"d733cc11-103e-4a8e-9d6e-24d4a45822da","try_again":1716733621.650252,"try_again_in":86399.999999416}
May 25 14:27:01 n4-rk1 caddy[253966]: {"level":"info","ts":1716647221.6504123,"logger":"tls","msg":"finished cleaning storage units"}
May 25 14:27:05 n4-rk1 caddy[253966]: {"level":"info","ts":1716647225.4954324,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000393200"}
May 25 14:27:05 n4-rk1 caddy[253966]: Error: loading initial config: loading new config: http app module: start: starting HTTP/3 QUIC listener: listen udp: lookup https: no such host
May 25 14:27:05 n4-rk1 systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
May 25 14:27:05 n4-rk1 systemd[1]: caddy.service: Failed with result 'exit-code'.
May 25 14:27:05 n4-rk1 systemd[1]: Failed to start Caddy.
May 25 14:27:05 n4-rk1 systemd[1]: Starting Caddy...
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"info","ts":1716647225.650479,"msg":"using provided configuration","config_file":"/etc/caddy/caddy_config","config_adapter":"caddyfile"}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"warn","ts":1716647225.654428,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/caddy_config","line":3}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"warn","ts":1716647225.6552162,"logger":"admin","msg":"admin endpoint disabled"}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"info","ts":1716647225.6555943,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000494880"}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"info","ts":1716647225.6560173,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"info","ts":1716647225.6560597,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"info","ts":1716647225.6566808,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
May 25 14:27:05 n4-rk1 caddy[253982]: {"level":"info","ts":1716647225.6567864,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000494880"}
May 25 14:27:05 n4-rk1 caddy[253982]: Error: loading initial config: loading new config: http app module: start: listening on socket-activation/https:443: no file descriptors passed
May 25 14:27:05 n4-rk1 systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE

@rilzbo
Copy link

rilzbo commented May 27, 2024

@flokli systemd-socket-activate can invoke another systemd-socket-activate to bind to both a TCP and UDP socket.
But the current code can anyway not handle two different sockets because it will delete the list of passed fds after processing the first one.

I tweaked the code a bit for testing and got it running using systemd-socket-activate -l 443 --fdname=localhost systemd-socket-activate --datagram -l 443 --fdname=localhost:localhost_UDP caddy run
https://github.com/rilzbo/caddy-socket-activation
I hard-coded the suffix _UDP to let the code distinguish the TCP and UDP sockets.

Also for HTTP3 Caddy will try to DNS resolve the "host" part of the network which is the fd name, so for testing you need to name the fd accordingly (e.g. localhost in my example).

@flokli
Copy link
Author

flokli commented May 30, 2024

Hmmh, this approach now means it'll become impossible to introduce additional domains without restarting everything, because for every new domain we'd need to pass two new FDs...

Also for HTTP3 Caddy will try to DNS resolve the "host" part of the network which is the fd name, so for testing you need to name the fd accordingly (e.g. localhost in my example).

Any idea why caddy does do this? It feels like it should not do this generally for those provided by plugins, only for the ones it knows about.

@WeidiDeng
Copy link
Member

It's not like that, you only need extra FDs if they don't bind to the same network interface. So normally 3 FDs are enough, 80tcp + 443tcp + 443udp.

@flokli
Copy link
Author

flokli commented May 31, 2024

It's not like that, you only need extra FDs if they don't bind to the same network interface. So normally 3 FDs are enough, 80tcp + 443tcp + 443udp.

I'm not sure I understand 100% what you're saying.

I'm having two https:// blocks, matching on two domain patterns (the main domain as well as a wildcard).
In that case, I want to handle these all by passing in a FD to 443/tcp and 443/udp (and 80/tcp)

Also for HTTP3 Caddy will try to DNS resolve the "host" part of the network which is the fd name.

Am I right to assume due to caddy currently trying to resolve the host part of the network I must use bind socket-activation/localhost in both blocks, and name the passed FDs localhost and localhost_UDP in the .socket file? So something like this?

      # Disable the admin API, we don't want to reconfigure Caddy at runtime.
      {
        admin off
      }

      https://*.s3.garage.flokli.io, https://s3.garage.flokli.io {
        bind socket-activation/localhost
        tls {
          dns acmedns {$CREDENTIALS_DIRECTORY}/acme-dns.s3.garage.flokli.io.json
        }

        reverse_proxy unix//run/garage/s3_api.sock
      }

      https://*.web.garage.flokli.io, https://web.garage.flokli.io {
        bind socket-activation/localhost
        tls {
          dns acmedns {$CREDENTIALS_DIRECTORY}/acme-dns.web.garage.flokli.io.json
        }

        reverse_proxy unix//run/garage/s3_web.sock
      }

… as well as

# /etc/systemd/system/caddy-https-tcp.socket
[Socket]
FileDescriptorName=localhost
ListenStream=[::]:443

[Install]
WantedBy=sockets.target
# /etc/systemd/system/caddy-https-udp.socket
[Socket]
FileDescriptorName=localhost_UDP
ListenDatagram=[::]:443

[Install]
WantedBy=sockets.target
# /etc/systemd/system/caddy-http.socket
[Socket]
FileDescriptorName=myhostname
ListenStream=[::]:80

[Install]
WantedBy=sockets.target

I see some problems:

  • The fact that caddy tries to resolve the "hostname" part of the bind value. Calling this localhost smells like a workaround to me, and will become a problem if I want to different socket-activation/… bind statements for different services than I have hostname aliases in /etc/hosts (I already need to use myhostname to have something resolveable to name the FD for port 80)
  • Requiring different names for different FD types. systemd allows adding FileDescriptorName=https, ListenDatagram=[::]:443 and ListenStream=[::]:443 to a single .socket file (and a FileDescriptorName=http, ListenDatagram=[::]:80 to another .socket file), and ListenersWithNames (which I wouldn't vendor/reimplement) maps these names to their listeners. IMHO https/http3 logic could just check .Addr().Network() to distinguish tcp and udp, to decide which listener to pick. And probably take multiple, if there are multiple, so you can just use add multiple Listen* statements in your .socket file if you want to bind on multiple IPs.
  • It's still unclear how to pass port 80 via FD too. My long-term goal is to have caddy not do any binding on its own. Can you extend your config file with this?

@WeidiDeng
Copy link
Member

I'm having two https:// blocks, matching on two domain patterns (the main domain as well as a wildcard).
In that case, I want to handle these all by passing in a FD to 443/tcp and 443/udp (and 80/tcp)

Unless these domains bind to different ips, 443/tcp and 443/udp can be shared because caddy can tell what the domain the client is accessing by tls sni and http hostname, virtual host in nginx terms.

  • The fact that caddy tries to resolve the "hostname" part of the bind value. Calling this localhost smells like a workaround to me, and will become a problem if I want to different socket-activation/… bind statements for different services than I have hostname aliases in /etc/hosts (I already need to use myhostname to have something resolveable to name the FD for port 80)

Caddy 2 is designed to manage sockets by itself instead of by external manager, current approach is a workaround. But as before, unless these domains can use the same bind statement.

  • Requiring different names for different FD types. systemd allows adding FileDescriptorName=https, ListenDatagram=[::]:443 and ListenStream=[::]:443 to a single .socket file (and a FileDescriptorName=http, ListenDatagram=[::]:80 to another .socket file), and ListenersWithNames (which I wouldn't vendor/reimplement) maps these names to their listeners. IMHO https/http3 logic could just check .Addr().Network() to distinguish tcp and udp, to decide which listener to pick. And probably take multiple, if there are multiple, so you can just use add multiple Listen* statements in your .socket file if you want to bind on multiple IPs.

Listening tcp and udp are 2 separate code paths, and using socket activation, udp is a net.PacketConn and that's enough to tell which socket to use. But due to design, caddy will use a different a network for udp, unique in network and address port, to create a socket. Windows doesn't support socket activation, and caddy was not made with that in mind, and current design was best to cope with different platforms.

  • It's still unclear how to pass port 80 via FD too. My long-term goal is to have caddy not do any binding on its own. Can you extend your config file with this?

An explicit http block is needed, like:

http:// {
 bind socket-activation/http
 redir https://{host}{uri} permanent
}

@flokli
Copy link
Author

flokli commented Jun 25, 2024

Caddy 2 is designed to manage sockets by itself instead of by external manager, current approach is a workaround. But as before, unless these domains can use the same bind statement.

I guess it'd be up to the maintainers to decide if these workarounds are gonna be as best as it's gonna get, or if there's plans to add this functionality into caddy directly. @mholt any thoughts on this?

@WeidiDeng
Copy link
Member

@flokli You can take a look at here to add this by yourself. You need to match the address to the passed sockets, and fallback to listening if necessary.

@flokli
Copy link
Author

flokli commented Jul 1, 2024

An explicit http block is needed, like:

http:// {
 bind socket-activation/http
 redir https://{host}{uri} permanent
}

This unfortunately doesn't work, the listener appends a :80 for some reason:

Error: loading initial config: loading new config: http app module: start: listening on socket-activation/http:80: no file descriptors passed

@mholt
Copy link
Member

mholt commented Jul 5, 2024

I'm a little lost as to where we're at with this after reading the thread (I've been catching up on a lot of different threads), but -- I think it would help if we move the discussion to actual JSON config. The Caddyfile is just an adapter; so if we're talking about syntax, then it's helpful to use Caddyfile in the discussion, but if we're talking about a new core change or feature related to listeners, we really should be having the discussion in JSON config, and I think that will clear up some of the confusion.

@flokli
Copy link
Author

flokli commented Jul 10, 2024

I'm a little lost as to where we're at with this after reading the thread (I've been catching up on a lot of different threads), but -- I think it would help if we move the discussion to actual JSON config. The Caddyfile is just an adapter; so if we're talking about syntax, then it's helpful to use Caddyfile in the discussion, but if we're talking about a new core change or feature related to listeners, we really should be having the discussion in JSON config, and I think that will clear up some of the confusion.

I'm sorry, I'm not familiar enough with the JSON config. The things in the initial requests still apply - I'd like to be able to configure caddy to (re-)use externally passed in sockets (through socket activation). Ideally, by identifying them by their name, and have this accessible in Caddyfile config. Once this all works, as a followup, I'd like to establish some nice defaults, caddy detecting it's running in a socket-activated environment, and doing the right thing on its own, without explicit config.

@WeidiDeng has been working on a caddy plugin supporting a custom socket-activation/ type for the bind statement (though technically caddy shouldn't bind anymore on its own at all), and I've been running into some problems using it, and am not sure it's the right layer this should be applied to.

@MayCXC
Copy link

MayCXC commented Aug 5, 2024

I think it would help if we move the discussion to actual JSON config.

To my understanding, Caddy would simply have to associate each of its file descriptors with the network and socket address that they were bound to in apps/http/servers/listen. Then, for both graceful restarts/upgrades and socket activation, old Caddy could pass new Caddy two environment variables, say $CADDY_SOCKET_FDS and $CADDY_SOCKET_ADDRESSES, that specify each fd and each socket addresses they are bound to. This is similar to Caddy 1 upgrade.go, but avoids relying on stdin and gob encoding, so that shell scripts can be used with socket activated systemd services more readily, or other tools like s6-fdholder-retrieve.

A simple shell script can documented for the specific case of socket activated systemd services, that iterates over $LISTEN_FDS and $LISTEN_FDNAMES, sets $CADDY_SOCKET_FDS and $CADDY_SOCKET_ADDRESSES or command line arguments accordingly, and then executes caddy. This is similar to how socket activated nginx works with the $NGINX variable.

Then new Caddy would simply check $CADDY_SOCKET_ADDRESSES before it binds to a socket address, then take its associated fd from $CADDY_SOCKET_FDS if one matched, use it to make a net.FileListener (for tcp/unix/unixpacket networks) or a net.FileConn (for udp/unixgram networks), and optionally close any unused fds after the new config is loaded. I am unsure how to handle ip networks, but it should be similar to those.

Based off of quic-go/quic-go#124, I think the only acceptable way to hand off an http3 server is to serialize the entire state of quic-go, rather than attempt a "graceful" restart that pauses new connections for an arbitrary duration while it sends out GOAWAYs. The recommendations for a QUIC-aware LB or intersected deployments meet different needs than graceful restarts/updates IMO, and someone who knows the protocol well enough could just go through each field of https://github.com/quic-go/quic-go/blob/master/server.go#L63 and identify what needs to be passed, along with whatever new Caddy needs from https://github.com/caddyserver/caddy/blob/master/listeners.go#L440. Then I imagine their gob or JSON states would be piped into additional fds for each http3 server, identified with more environment variables like $CADDY_H3_SERVERS and $CADDY_H3_STATE_FDS. The http3 server can then be restarted from the serialized quic state and socket fds.

It looks like a lot of JSON serialization is already implemented in quic-go here: https://github.com/quic-go/quic-go/blob/master/qlog/event.go and here: https://github.com/quic-go/quic-go/blob/master/internal/wire/transport_parameters.go, so just something like MarshalJSONObject(Server) and a way to unmarshal one is needed. Because UDP packet delivery is never guaranteed, a quic server should be able to stop receiving, hand off its serialized state, and start receiving again with no need to close gracefully. But until it is figured out, a stateless reset for all open http3 connections can do.


I can implement this in stages, it is a core functionality of a project I am working on. I'd be happy to contribute it to Caddy if we can agree on a few details, please tell me what you think:

  • Socket activation should not need to be explicitly enabled per listener in the Caddy config, and socket-activation should not be a separate network from tcp, udp, etc. Passed fds should just be associated 1:1 with the socket addresses they are bound to, and used to produce their listeners unless they are configured otherwise.

  • The only configuration change I would make is to add an optional flag like bind network/address graceful and bind network/address graceless, adapted to graceful: true and graceful: false fields in apps/http/servers/listen elements. This would indicate if Caddy should close that fd or pass it to new Caddy when updating/restarting, and close that fd or listen to it when starting. I would make graceful the default, maybe with an additional graceful_bind: true field in apps/http/servers to override the default.

  • The implementation should not rely on systemd-specific environment variables to declare http server configurations. My highest priority uses systemd socket activation, but there are other cases that I care about that the $LISTEN_FDNAMES variable and go-systemd/activation packages are not precise enough for. Instead I can document a socket activated systemd service that uses a shell script to set up the necessary variables and execute Caddy, so no systemd specific functionality needs to be added.

  • The http3 state needs to be serialized and passed as another fd in memory. This is my lowest priority, so I will do it last, but I do need it. It would be configured with an option like graceful_http3: true in apps/http/servers.

@mholt
Copy link
Member

mholt commented Aug 5, 2024

@MayCXC Welcome!

This is similar to Caddy 1 upgrade.go, but avoids relying on stdin and gob encoding, so that shell scripts can be used with socket activated systemd services more readily

Ah, interesting. I'd probably be OK with that, though env vars feel a bit more hacky to me than a proper data pipe/stream 😅 But yeah, what you proceed to describe does sound similar to what I implemented in Caddy v1 for graceful upgrades.

I can implement this in stages, it is a core functionality of a project I am working on. I'd be happy to contribute it to Caddy if we can agree on a few details, please tell me what you think:

Ooh, this could be an excellent contribution and could speed things up. Thank you!

  • Socket activation should not need to be explicitly enabled per listener in the Caddy config, and socket-activation should not be a separate network from tcp, udp, etc. Passed fds should just be associated 1:1 with the socket addresses they are bound to, and used to produce their listeners unless they are configured otherwise.

This sounds good to me.

  • The only configuration change I would make is to add an optional flag like bind network/address graceful and bind network/address graceless, adapted to graceful: true and graceful: false fields in apps/http/servers/listen elements. This would indicate if Caddy should close that fd or pass it to new Caddy when updating/restarting, and close that fd or listen to it when starting. I would make graceful the default, maybe with an additional graceful_bind: true field in apps/http/servers to override the default.

In Caddy 1, all listeners would automatically be gracefully transferred to the new instance (upon a graceful reload being triggered), without needing configuration. I wonder, is non-graceful listeners even useful/needed? If not, maybe we can forego this config surface entirely.

  • The implementation should not rely on systemd-specific environment variables to declare http server configurations. My highest priority uses systemd socket activation, but there are other cases that I care about that the $LISTEN_FDNAMES variable and go-systemd/activation packages are not precise enough for. Instead I can document a socket activated systemd service that uses a shell script to set up the necessary variables and execute Caddy, so no systemd specific functionality needs to be added.

I am considering whether this is a good time to implement graceful reloads as well (transferring any/all sockets to next Caddy instance -- regardless of systemd) and so I like the systemd-agnostic approach wherever possible.

  • The http3 state needs to be serialized and passed as another fd in memory. This is my lowest priority, so I will do it last, but I do need it. It would be configured with an option like graceful_http3: true in apps/http/servers.

How does passing data as a fd work? I would have thought to use a stdin pipe 😅

@MayCXC
Copy link

MayCXC commented Aug 5, 2024

@MayCXC Welcome!

Thank you :)

This is similar to Caddy 1 upgrade.go, but avoids relying on stdin and gob encoding, so that shell scripts can be used with socket activated systemd services more readily

Ah, interesting. I'd probably be OK with that, though env vars feel a bit more hacky to me than a proper data pipe/stream 😅 But yeah, what you proceed to describe does sound similar to what I implemented in Caddy v1 for graceful upgrades.

I agree they do feel hacky, but otherwise you end up with extra stuff in the pipe that isn't technically configuration data. Env vars are used by systemd-socket-activate itself, so sd-daemon.h and go-systemd ultimately read from them anyways, they are recommended by 12factor.net, blahblahblah. So I do think they are the standardest practice we are gonna get.

  • The only configuration change I would make is to add an optional flag like bind network/address graceful and bind network/address graceless, adapted to graceful: true and graceful: false fields in apps/http/servers/listen elements. This would indicate if Caddy should close that fd or pass it to new Caddy when updating/restarting, and close that fd or listen to it when starting. I would make graceful the default, maybe with an additional graceful_bind: true field in apps/http/servers to override the default.

In Caddy 1, all listeners would automatically be gracefully transferred to the new instance (upon a graceful reload being triggered), without needing configuration. I wonder, is non-graceful listeners even useful/needed? If not, maybe we can forego this config surface entirely.

having it just toggle for startup/reload works for me. that makes the socket activation case a "graceful start", caddy start --graceful and caddy start --graceless, and reloads are caddy upgrade --graceful, and caddy upgrade --graceless. as long as it is clear that the start case means bound sockets are passed from the supervisor/executor script to Caddy gracefully.

  • The implementation should not rely on systemd-specific environment variables to declare http server configurations. My highest priority uses systemd socket activation, but there are other cases that I care about that the $LISTEN_FDNAMES variable and go-systemd/activation packages are not precise enough for. Instead I can document a socket activated systemd service that uses a shell script to set up the necessary variables and execute Caddy, so no systemd specific functionality needs to be added.

I am considering whether this is a good time to implement graceful reloads as well (transferring any/all sockets to next Caddy instance -- regardless of systemd) and so I like the systemd-agnostic approach wherever possible.

yeah it's the same outcome, in the Caddy<->Caddy case it would be easier to use a gob/pipe instead env vars. But if the env vars are implemented already for the shell<->Caddy case, we may as well just reuse them.

  • The http3 state needs to be serialized and passed as another fd in memory. This is my lowest priority, so I will do it last, but I do need it. It would be configured with an option like graceful_http3: true in apps/http/servers.

How does passing data as a fd work? I would have thought to use a stdin pipe 😅

Now that you mention it, this actually would be best to send via gob+stdin pipe. Otherwise you would end up with a tmpfile or fifo that puts the whole http3 state and all its tls stuff somewhere on the disk, which stinks. By the time I can serialize and deserialize a quic server state, figuring out where to pass it should be easy :]

Thanks for your replies, I'll get cookin on a branch that checks the environment for socket fds to do graceful starts and upgrades.

One thing I want to double check, is did caddy 1 upgrade.go orphan its child process after it shut down? It would just be necessary to document that containers need to use use tini or something to collect the zombie caddie in this case, or make it the default entrypoint. I think nginx relies on dumb-init for reference.

@flokli
Copy link
Author

flokli commented Aug 5, 2024

  • The http3 state needs to be serialized and passed as another fd in memory. This is my lowest priority, so I will do it last, but I do need it. It would be configured with an option like graceful_http3: true in apps/http/servers.

How does passing data as a fd work? I would have thought to use a stdin pipe 😅

Now that you mention it, this actually would be best to send via gob+stdin pipe. Otherwise you would end up with a tmpfile or fifo that puts the whole http3 state and all its tls stuff somewhere on the disk, which stinks. By the time I can serialize and deserialize a quic server state, figuring out where to pass it should be easy :]

Thanks for your replies, I'll get cookin on a branch that checks the environment for socket fds to do graceful starts and upgrades.

One thing I want to double check, is did caddy 1 upgrade.go orphan its child process after it shut down? It would just be necessary to document that containers need to use use tini or something to collect the zombie caddie in this case, or make it the default entrypoint. I think nginx relies on dumb-init for reference.

Check https://systemd.io/FILE_DESCRIPTOR_STORE/ and how it specifically talks about serializing state.

@mholt
Copy link
Member

mholt commented Aug 5, 2024

One thing I want to double check, is did caddy 1 upgrade.go orphan its child process after it shut down?

I think so, but I'd have to double-check when I have a chance. Been a while 😅

@MayCXC
Copy link

MayCXC commented Aug 5, 2024

Check https://systemd.io/FILE_DESCRIPTOR_STORE/ and how it specifically talks about serializing state.

Ok, this got me thinking I was wrong here:

Now that you mention it, this actually would be best to send via gob+stdin pipe. Otherwise you would end up with a tmpfile or fifo that puts the whole http3 state and all its tls stuff somewhere on the disk, which stinks.

What Caddy should actually do is keep the fds it received with $CADDY_H3_STATE_FDS, overwrite them with the new state before it upgrades gracefully, and pass them through. Then the example service shell script can set $CADDY_H3_STATE_FDS to the FileDescriptorStoreMax memfds it got from systemd, and another Caddy user can just give it plain old named file fds, the implementation stays the same.

Now there are some cool questions:

  • do we add caddy state --socket-fds "$CADDY_SOCKET_FDS" --socket-addresses "$CADDY_SOCKET_ADDRESSES" <>&$CADDY_STATE_FD? This updates $CADDY_STATE_FD as a state JSON with the socket fds and addresses.
  • then we can do caddy run --config ./Caddyfile --state <>&$CADDY_STATE_FD in the systemd service, which lets us shut down Caddy normally and let systemd restart it like that page describes @flokli. Caddy updates $CADDY_STATE_FD with its serialized http3 state when it stops, at a separate path from the socket bindings.
  • and we still support caddy upgrade --graceful, which spawns CADDY__GRACEFUL_UPGRADE=1 caddy ..., new caddy reads stdin as a gob with a config JSON and state JSON, and signals old caddy to exit similar to old upgrade.go.

I like the introduction of a state fd because it gets rid of all of the environment variables needed for socket activation and graceful http3 restarts @mholt. For a systemd service it can come from the file descriptor store, and in the docker image we make it an actual file in /data/ by default. You can get socket activation and graceful upgrades with systemd, or just graceful upgrades standalone in docker, etc. with the same implementation.

@mholt
Copy link
Member

mholt commented Aug 7, 2024

caddy upgrade --graceful

I feel like that should just be the default, i.e. caddy upgrade should just always be graceful.

As for the first two points, I'm not really sure I understand the need/use for caddy state command or the --state flag; but that's probably my lack of understanding regarding systemd.

In general this sounds fine, but I do recommend that Caddy is as much "just works" as possible. 🙂

@MayCXC
Copy link

MayCXC commented Aug 8, 2024

caddy upgrade --graceful

I feel like that should just be the default, i.e. caddy upgrade should just always be graceful.

As for the first two points, I'm not really sure I understand the need/use for caddy state command or the --state flag; but that's probably my lack of understanding regarding systemd.

In general this sounds fine, but I do recommend that Caddy is as much "just works" as possible. 🙂

sounds good to me, I would definitely expect caddy upgrade to do a graceful restart.

caddy state is like caddy adapt, but for turning environment variables into the state JSON. when using caddy upgrade, a separate command for this and argument for --state are not needed. For systemd, the idea was to restart caddy normally instead of using caddy upgrade, and store state with the systemd file descriptor store as @flokli pointed out, or just a file in the caddy data directory.

I think we can get rid of the caddy state command and use the config to set up socket activation, but Caddy will use state to remember which sockets are bound to which addresses when it restarts and reloads its config. This is so it can close any that become unused, and know which still need to be bound. I'm taking my time thinking of the most caddyish way to do it, when I get it right I think it will click and be clear for you. Here is a way we can do it with a new server option in Caddyfile:

{
  servers tcp/:80 {
    socket {env.CADDY_HTTP_FD}
  }
  servers tcp/:443 {
    socket {env.CADDY_HTTPS_FD}
  }
  servers udp/:443 {
    socket {env.CADDY_HTTP3_FD}
  }
}
...

Then Caddy can supply those env vars with systemd socket activation fds, from a script called exec-start-caddy:

#!/usr/bin/env bash
read -r CADDY_HTTP_FD CADDY_HTTPS_FD CADDY_HTTP3_FD CADDY_STATE_FD <<< $LISTEN_FDS
exec caddy --config ./Caddyfile --state <>&$CADDY_STATE_FD

and a socket unit called caddy.socket:

[Socket]
ListenStream=80
ListenStream=443
ListenDatagram=443

[Install]
WantedBy = sockets.target

to go with https://github.com/caddyserver/dist/blob/master/init/caddy.service, with FileDescriptorStoreMax=1 added. Supplying a separate fd for this with caddy start --config Caddyfile --state <>&$CADDY_STATE_FD lets systemd restart Caddy seamlessly without using caddy upgrade. In this case, systemd holds all the network sockets state socket, while caddy exits and starts again. Later on this is can be used to gracefully resume stateful http3 servers, maintain very strict round robin load balancers, etc.

I think using servers to map out the fds instead of bind gets around #6296 (comment), which was the first way I thought about doing it too. When Caddy starts, it will load its state from --state, or a default location in /data/, start any listeners it can from fds instead of addresses, close any fds that are unused now, etc. Then when it stops or updates, Caddy will save its state into the same fd it loaded it from.

@MayCXC
Copy link

MayCXC commented Aug 11, 2024

I got started by making listeners.go and listen.go socket file aware. If the approach looks good, I can finish up socket activation soon.

@MayCXC
Copy link

MayCXC commented Aug 13, 2024

My PR can now use an arbitrary socket activation FD for each configured bind address.

@mholt
Copy link
Member

mholt commented Aug 13, 2024

Cool, looking forward to reviewing it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion 💬 The right solution needs to be found feature ⚙️ New feature or request
Projects
None yet
Development

No branches or pull requests

10 participants