Resource management with cgroups and friends

[zimbatm]

A tracking issue.

Now that NixOS/nix#11412 has been merged, it might be worth deploying Nix with groups enabled in the next release.

Arian also built this prom exporter that is nicely complementary and could be used to collect metrics and better understand resource utilisation: https://github.com/arianvp/cgroup-exporter

[Mic92]

Part of nix-experimental.

[zowoq]

Maybe I should just apply the cgroups patches to our current nix version so we can try to make some progress on this?

#1466

[zimbatm]

@Mic92 do you know if a new release will be cut soon?

[Mic92]

We are behind schedule actually for a release. I will ask if we can do it next week.

[zowoq]

Remote builds support passing max-jobs, how feasible would it be to add cores so we could set only for hydra jobs? Could be useful as a complement to cgroups and would also work for darwin jobs. If upstream wasn't very keen on it we could even carry the patch (if it isn't too complicated)?

[zowoq]

Interesting error, has happened in a few builds now:

Seems this is the build hitting the 20min max-silent-time but instead of Timed Out it is Aborted and it is added back to the queue for another build attempt.

[Mic92]

@zowoq can you also open a nix issue for this?

[zowoq]

Trying to find a trivial way to reproduce the error but so far haven't been able to.