-
-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support passing FDs (socket activation) #6296
Comments
Are you looking for the And see https://caddyserver.com/docs/conventions#network-addresses, you can use unix sockets in I'm not sure what you're asking for if not that. |
It sounds like what is being asked for is graceful upgrades/restarts. Caddy 1 had this feature, and I quite liked how it worked: pass the socket directly to the next process. It worked on all Unix systems without relying on a separate system service, and it was smart enough to understand Caddy configuration: if the new config didn't use a socket, it wouldn't be kept; rather than blindly moving all the sockets over. I'd probably rather bring the implementation from Caddy 1 into Caddy 2. |
No, getting this for free is only one side-effect of supporting socket-activation. socket-activation will also cause caddy to get started lazily whenever the first connection to the (externally configured) socket address happens, which simplifies declaring service dependencies too. The article linked from my link elaborates a bit more on this.
This still requires caddy to do manual coordination with its new process and pass it around explicitly. The point of simply taking the FDs passed by the service manager is that caddy does not have to be aware of whether it's the first process being started on the system, or you start a new version with another config. caddy simply gets an FD, where new connections will appear on. |
Ah yes, and because caddy just takes FDs, it doesn't need to |
But what is Caddy supposed to do with that socket? How does it know the configuration associated with it? You can't just hand a server a socket and expect it to know what to do with it, without any configuration... maybe I am missing something about how it works. |
Sockets can have names attached (so the user can name them All these passed FDs also give you a You can play around with this through |
Oh I see, so you'd still have your Caddy config, you'd just specify a different network name for the listener address, and Caddy will then get it from the service manager rather than binding a new socket. |
Yes! Or well, I don't want caddy to do any bind on its own at all, but pass in every socket via this mechanism. |
In that case you can use Anyone is welcome to pick this up. |
@mholt I did some experiments with registering custom network, it's too much trouble to be worth it. Every site block needs an explicit @flokli I'm thinking on unix, we can try preferring socket activation but fallback to the old behavior. What do you think of it? Or should caddy just exit unsuccessfully if socket activation environments variables are found but not sockets matching listening critertia are found? Or if some warning logs are emitted? As mentioned above, you are responsible to pass every socket yourself, including 80 tcp and 443 udp if auto http->https and http3 are enabled respectively. And admin socket if enabled as well. Assuming you restart caddy instead of reload it. |
I would really see it happen and it can greatly reduce my network stack complexity. flowchart TD
A[Caddy] -->|Reverse Proxy| B{Container Network}
B -->|Serve Frontend| C[Caddy]
C -->|Reverse Proxy| D[Server]
When |
I think ti makes sense to first land the feature with explicit configuration, which might mean explicit The good thing is, it's pretty safe to detect whether caddy is running in a socket-activated environment or not, so we are able to change defaults in this case, without breaking existing usecases. |
@flokli So that means you're fine with mixing passing FD and current binding behavior? And since you will use The problem with names is that one name can map to many sockets with different addresses, how do you think caddy handle this situation? |
Until this is implemented: for those that just care about binding to ports <1024 AND not running Caddy as root, can use systemd's SocketBindAllow= (available since systemd 249) |
There was never a need to run Caddy as root on Linux. Our standard systemd unit file is shipped with |
I'm aware of Giving the option to move the whole socket binding business entirely out of caddy is what I'm advocating for, both from a sandboxing (it doesn't need to be allowed to
Yes, I think the This would also mean, caddy would still bind on its own where we don't explicitly configure it to use the FD(s).
Indeed FileDescriptorName= describes such name applies to all sockets in that I think I'd be fine landing support for having to explicitly use |
@flokli You can try it with a plugin for now, |
I wrote a small go library to listen on socket activated fds. |
Interesting, this could be turned into a Caddy plugin by using |
Isn't that exactly what @WeidiDeng's plugin already does? https://github.com/WeidiDeng/caddy-socket-activation/blob/2246ae4a7a00955926ebdf1d557c1530b327bf2f/tcp.go#L12 (I'll try it now, was quite busy before. Did not forget, sorry! Will report back here)
There's also github.com/coreos/go-systemd/activation, linked in the initial issue description, which seems a bit more commonly used. |
I gave https://github.com/WeidiDeng/caddy-socket-activation a try (after WeidiDeng/caddy-socket-activation#1). Some notes: I'm using caddy alongside the acme-dns plugin, so don't really need to bind on port 80, however it seems to be quite hard (?) to not bind there (requires disabling autossl) or reconfigure this to another bind, using the new I ignored the http problem (letting caddy bind on http on its own), and configured a
systemd starts up caddy on the first connection, however caddy fails to pick up the FD names, or at least, in some cases. I managed to reproduce the flaky behaviour with
This might also be related to
|
@flokli systemd-socket-activate can invoke another systemd-socket-activate to bind to both a TCP and UDP socket. I tweaked the code a bit for testing and got it running using Also for HTTP3 Caddy will try to DNS resolve the "host" part of the network which is the fd name, so for testing you need to name the fd accordingly (e.g. localhost in my example). |
Hmmh, this approach now means it'll become impossible to introduce additional domains without restarting everything, because for every new domain we'd need to pass two new FDs...
Any idea why caddy does do this? It feels like it should not do this generally for those provided by plugins, only for the ones it knows about. |
It's not like that, you only need extra FDs if they don't bind to the same network interface. So normally 3 FDs are enough, 80tcp + 443tcp + 443udp. |
I'm not sure I understand 100% what you're saying. I'm having two
Am I right to assume due to caddy currently trying to resolve the host part of the network I must use
… as well as
I see some problems:
|
Unless these domains bind to different ips, 443/tcp and 443/udp can be shared because caddy can tell what the domain the client is accessing by tls sni and http hostname, virtual host in nginx terms.
Caddy 2 is designed to manage sockets by itself instead of by external manager, current approach is a workaround. But as before, unless these domains can use the same bind statement.
Listening tcp and udp are 2 separate code paths, and using socket activation, udp is a
An explicit http block is needed, like:
|
I guess it'd be up to the maintainers to decide if these workarounds are gonna be as best as it's gonna get, or if there's plans to add this functionality into caddy directly. @mholt any thoughts on this? |
This unfortunately doesn't work, the listener appends a
|
I'm a little lost as to where we're at with this after reading the thread (I've been catching up on a lot of different threads), but -- I think it would help if we move the discussion to actual JSON config. The Caddyfile is just an adapter; so if we're talking about syntax, then it's helpful to use Caddyfile in the discussion, but if we're talking about a new core change or feature related to listeners, we really should be having the discussion in JSON config, and I think that will clear up some of the confusion. |
I'm sorry, I'm not familiar enough with the JSON config. The things in the initial requests still apply - I'd like to be able to configure caddy to (re-)use externally passed in sockets (through socket activation). Ideally, by identifying them by their name, and have this accessible in @WeidiDeng has been working on a caddy plugin supporting a custom |
To my understanding, Caddy would simply have to associate each of its file descriptors with the network and socket address that they were bound to in apps/http/servers/listen. Then, for both graceful restarts/upgrades and socket activation, old Caddy could pass new Caddy two environment variables, say A simple shell script can documented for the specific case of socket activated systemd services, that iterates over Then new Caddy would simply check Based off of quic-go/quic-go#124, I think the only acceptable way to hand off an http3 server is to serialize the entire state of quic-go, rather than attempt a "graceful" restart that pauses new connections for an arbitrary duration while it sends out GOAWAYs. The recommendations for a QUIC-aware LB or intersected deployments meet different needs than graceful restarts/updates IMO, and someone who knows the protocol well enough could just go through each field of https://github.com/quic-go/quic-go/blob/master/server.go#L63 and identify what needs to be passed, along with whatever new Caddy needs from https://github.com/caddyserver/caddy/blob/master/listeners.go#L440. Then I imagine their gob or JSON states would be piped into additional fds for each http3 server, identified with more environment variables like It looks like a lot of JSON serialization is already implemented in quic-go here: https://github.com/quic-go/quic-go/blob/master/qlog/event.go and here: https://github.com/quic-go/quic-go/blob/master/internal/wire/transport_parameters.go, so just something like I can implement this in stages, it is a core functionality of a project I am working on. I'd be happy to contribute it to Caddy if we can agree on a few details, please tell me what you think:
|
@MayCXC Welcome!
Ah, interesting. I'd probably be OK with that, though env vars feel a bit more hacky to me than a proper data pipe/stream 😅 But yeah, what you proceed to describe does sound similar to what I implemented in Caddy v1 for graceful upgrades.
Ooh, this could be an excellent contribution and could speed things up. Thank you!
This sounds good to me.
In Caddy 1, all listeners would automatically be gracefully transferred to the new instance (upon a graceful reload being triggered), without needing configuration. I wonder, is non-graceful listeners even useful/needed? If not, maybe we can forego this config surface entirely.
I am considering whether this is a good time to implement graceful reloads as well (transferring any/all sockets to next Caddy instance -- regardless of systemd) and so I like the systemd-agnostic approach wherever possible.
How does passing data as a fd work? I would have thought to use a stdin pipe 😅 |
Thank you :)
I agree they do feel hacky, but otherwise you end up with extra stuff in the pipe that isn't technically configuration data. Env vars are used by systemd-socket-activate itself, so
having it just toggle for startup/reload works for me. that makes the socket activation case a "graceful start",
yeah it's the same outcome, in the Caddy<->Caddy case it would be easier to use a gob/pipe instead env vars. But if the env vars are implemented already for the shell<->Caddy case, we may as well just reuse them.
Now that you mention it, this actually would be best to send via gob+stdin pipe. Otherwise you would end up with a tmpfile or fifo that puts the whole http3 state and all its tls stuff somewhere on the disk, which stinks. By the time I can serialize and deserialize a quic server state, figuring out where to pass it should be easy :] Thanks for your replies, I'll get cookin on a branch that checks the environment for socket fds to do graceful starts and upgrades. One thing I want to double check, is did caddy 1 upgrade.go orphan its child process after it shut down? It would just be necessary to document that containers need to use use |
Check https://systemd.io/FILE_DESCRIPTOR_STORE/ and how it specifically talks about serializing state. |
I think so, but I'd have to double-check when I have a chance. Been a while 😅 |
Ok, this got me thinking I was wrong here:
What Caddy should actually do is keep the fds it received with Now there are some cool questions:
I like the introduction of a state fd because it gets rid of all of the environment variables needed for socket activation and graceful http3 restarts @mholt. For a systemd service it can come from the file descriptor store, and in the docker image we make it an actual file in |
I feel like that should just be the default, i.e. As for the first two points, I'm not really sure I understand the need/use for In general this sounds fine, but I do recommend that Caddy is as much "just works" as possible. 🙂 |
sounds good to me, I would definitely expect
I think we can get rid of the
Then Caddy can supply those env vars with systemd socket activation fds, from a script called
and a socket unit called
to go with https://github.com/caddyserver/dist/blob/master/init/caddy.service, with I think using |
I got started by making listeners.go and listen.go socket file aware. If the approach looks good, I can finish up socket activation soon. |
My PR can now use an arbitrary socket activation FD for each configured bind address. |
Cool, looking forward to reviewing it! |
I'd like to use caddy in a socket-activated environments, using FDs passed down from the service manager, rather than binding on addresses on its own.
Combined with signalling readyness (which caddy already does), this will give zero-downtime (re)deployments on Linux systems using systemd (if
.socket
files are used), by simply restarting the process - the socket is held open by systemd, and new connections are passed in once caddy is ready to accept new requests. In these cases, there wouldn't be a need for complicated reload logic anymore.github.com/coreos/go-systemd/activation
provides the necessary methods to check whether FDs are passed, including identifying them by their socket name. https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket-activation gives a nice introduction into the feature itself.In case no explicit listen addresses are specified, caddy could default to do that rather than binding on its own, if it detects it's running in such an environment.
Additionally, Caddyfile could be extended to allow specifying these passed fds as network addresses (something like
sd-listen:$name
orsd-listen:$idx
maybe). This can become useful when you want to expose different things on different sockets.The text was updated successfully, but these errors were encountered: