-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Copy local flakes to the store lazily #3121
Comments
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/flakes-without-git-copies-entire-tree-to-nix-store/10743/2 |
Still important. |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/my-painpoints-with-flakes/9750/20 |
It would be nice if Nix could take advantage of the filesystem's native CoW functionality (if present) in order to speed up copying. |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/is-it-possible-to-make-a-flake-that-has-no-source-tree/16037/2 |
This comment was marked as duplicate.
This comment was marked as duplicate.
Also for context, in my case the flake was not in a git repo, it was just in a folder. Copying to the nix store is unacceptable because the folder contains multiple git repos along with all their build artifacts. Copying a git repo to the Nix store at least would avoid copying untracked files, but in my case it had hundreds of thousands of files and multiple gigabytes of data to look at and copy. |
@lilyball @L-as Taking advantage of CoW doesn't work on Linux either due to a VFS limitation: #5513 One thing I'd like to understand in this issue is why a local flake can't be evaluated "directly" just like the old |
@Atemu from what I've read, it's to help enforce hermetic evaluation and avoid impurities. Presumably it also has advantages for code simplicity, because you don't need to write something separate for local flakes. I agree it's not great UX for those of us who use flakes just to keep track of a dev shell, of course :) |
This comment was marked as duplicate.
This comment was marked as duplicate.
You might not realize you're using local files, accidentally sneak in state, and then be surprised when it doesn't evaluate in deployment (and be all "wait, isn't nix supposed to prevent this?"). Even with fully local files, I'd expect things still to work if I move my directory to a new computer from a restored backup. While I've personally learned when and where local state might happen, it's still a safety net that I consider nice to have. Of course, giant copies for the tiniest delta is way too much of a cost to incur for that, but this is why we're here - to make sure that flakes don't blow up SSDs all over the place when they finally become non-experimental ;) |
I don't understand what you mean by that. How is copying the accidentally added state over to the Nix store first and then evaling it any better than just evaling it directly?
How is the location of the directory related to any of this? A direct eval of the same state of a directory in another location will have the same result. How should copying improve anything? |
IIRC files that are tracked with git already (and changed) are being staged and then copied to the store. I can see how this ensures that at least the files are tracked and marked as updated (by staging them). I also kind of agree that I think this is the wrong solution to the problem, or perhaps a solution in search of a problem? Most of the time it is very expensive to copy my working directory into the store. Since I can see why that feature is useful, I'd argue that it should be configurable if you want your flake repos to be copied to the store or not. As far as I know, the hash of the path that is added to the store is also currently used for the eval caching. Perhaps the current implementation is a nice PoC of how more proper hermetic eval could look like and what it gives us in terms of capabilities (caching, ...). |
That sounds like a sound reason but I can's see how that wouldn't just be possible with direct eval too. @edolstra could we get some insight from you here? |
It's definitely possible, just more work as described in the initial post:
Nix already has a "eval may only access these store paths" logic, but no "may only access tracked files of this Git checkout" logic yet, so using the former was the simplest solution I assume. |
@AleXoundOS note that reflinking is something an application must choose to do; it is not the default. (IMO it really should be but that's a topic for a different project.) I don't think it's worth spending time on this at this point though as it's much more important to push #6530 forward. After that's done, reflinking would be an extremely minor optimisation; probably not signficant enough to be worth considering. |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/using-nix-shells-without-polluting-repositories/37362/6 |
Looks like this is not getting anywhere for the moment, how about allowing listing necessary files in the flake? E.g. the attribute To make it more dev-friendly, the list could be globs instead of paths, so that you can include or exclude. |
If you don't use IFD, can you just first load all nix files imported by flake.nix into memory and eval them then? Can someone tell me why you'd need to copy directories flakes reside in into the store if self is not used as a path? E.g. not used at all or just an attr like What about simple devshell flakes? Why are their directories copied into the store? It's just one nix file that doesn't use I personally use Edit: nevermind the last part, I thought Edit 2: If something like a |
You don't have to commit every time you make a change with the git fetcher ("tree is dirty" is just a warning) and flakes are always copied to the store, even with path URLs. If that doesn't work for you, have a shell.nix until #6530 is complete. |
What about unchanged files within the flake/source dir? Is there any way to keep those from getting duplicated in the store (e.g. have unchanged files in the new If I'm not mistaken, the flake |
@tmillr you can set |
While that reduces the space usage needed asynchronously, it still does a copy which wastes IO time and write endurance. |
Thank you. I guess I forgot that that mechanism works based off of file content and not nix store hash (or something else). I am using that currently, but I have to run a custom It sounds like even using that option causes extra fs writes to occur however (but will work for saving space alone). |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/confused-about-nix-copying-nixpkgs/53737/2 |
@Atemu Reflinking wouldn’t be significant for reducing the copies incurred by the eval, but it could still be significant in reducing the otherwise still necessary copies of inputs for the build itself, right? |
I'm not sure what you mean by that. Reflink copies speed up copying any path to the Nix store aswell as reducing storage cost (though in an unpredictable manner) but the best thing to do on any case is to copy as few things to the Nix store as possible. For reducing copies between existing build inputs in the Nix store, the right tool is Nix store optimisation as @fgaz mentioned and it's filesystem-agnostic. |
getFlake is really slow (~25 seconds) compared to flake-compat (~5 seconds) since getFlake copies the source into the store. There's an issue open for doing the copying lazily[1]. [1]: NixOS/nix#3121
This issue has been mentioned on NixOS Discourse. There might be relevant details there: |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: |
Currently flakes are evaluated from the Nix store, so when using a local flake, it's first copied to the store. This means that
is a lot slower than the non-flake alternative
Ideally, we would copy the flake to the store only when its
outPath
attribute is evaluated. However, we also need to ensure that it's not possible to access untracked files (i.e. we need to check every file againstgit ls-files
).The text was updated successfully, but these errors were encountered: