Unable to build with Cargo.lock referring to git repo with git-lfs enabled in Nix > 2.3

[flokli]

It seems there's some problems / recent breakage when trying to clone repositories with git-lfs enabled from a nix build.

In https://code.tvl.fyi/tree/tvix/eval, we're referring to rnix-parser from Cargo.lock, by git rev.

This seems to fail with more recent versions (>2.3) of Nix, due to some git-lfs issues:

 mg build //tvix/eval
[mg] building target //tvix/eval
Downloading benches/all-packages.nix (803 KB)
Error downloading object: benches/all-packages.nix (935f239): Smudge error: Error downloading benches/all-packages.nix (935f239a576caa554e546072772039f6efcbde2f2e6568733b8857fe476c5604): batch request: missing protocol: ""

Errors logged to '/home/flokli/.cache/nix/gitv3/1qzfb517dymd9fbs5nznalj8803yx2mzy4wzjxfr1wqfc3nj00q2/lfs/logs/20220916T124822.679316329.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: benches/all-packages.nix: smudge filter lfs failed
error: failed to extract archive (Source threw exception: error: program 'git' failed with exit code 128)

       … while fetching the input 'git+https://github.com/nix-community/rnix-parser.git?rev=97b438e34be5211a4b48aeed9cc3ded489b4d6da'

       … while evaluating the attribute 'checkout' of the derivation 'rnix'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'paths' of the derivation 'dependencies'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating 'outputVal'

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:41:15:

           40|
           41|   outputVal = v:
             |               ^
           42|     let

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:39:9:

           38|           "{ ${valsStr} }" else
           39|         outputVal v;
             |         ^
           40|

       … while evaluating 'outputValInner'

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:28:20:

           27|
           28|   outputValInner = v:
             |                    ^
           29|     let

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:69:42:

           68|           vals = mapAttrsToList
           69|             (k': v': "${quoteKey k'} = ${outputValInner v'}") v;
             |                                          ^
           70|           valsStr = concatStringsSep ", " vals;

       … while evaluating anonymous lambda

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:69:18:

           68|           vals = mapAttrsToList
           69|             (k': v': "${quoteKey k'} = ${outputValInner v'}") v;
             |                  ^
           70|           valsStr = concatStringsSep ", " vals;

       … from call site

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/lib/attrsets.nix:356:16:

          355|   mapAttrsToList = f: attrs:
          356|     map (name: f name attrs.${name}) (attrNames attrs);
             |                ^
          357|

       … while evaluating anonymous lambda

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/lib/attrsets.nix:356:10:

          355|   mapAttrsToList = f: attrs:
          356|     map (name: f name attrs.${name}) (attrNames attrs);
             |          ^
          357|

       … from call site

       … while evaluating anonymous lambda

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:133:10:

          132|       concatMapStringsSep "\n"
          133|         (kv: concatStringsSep "\n" (outputKeyVal kv.k kv.v))
             |          ^
          134|         (

       … from call site

       … while evaluating 'concatMapStringsSep'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/lib/strings.nix:111:5:

          110|     # List of input strings
          111|     list: concatStringsSep sep (map f list);
             |     ^
          112|

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:132:7:

          131|     in
          132|       concatMapStringsSep "\n"
             |       ^
          133|         (kv: concatStringsSep "\n" (outputKeyVal kv.k kv.v))

       … while evaluating 'toTOML'

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:119:12:

          118|
          119|   toTOML = attrs:
             |            ^
          120|     assert (typeOf attrs == "set");

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/build.nix:84:19:

           83|     # and git dependencies.
           84|     cargoconfig = builtinz.toTOML {
             |                   ^
           85|       source = {

       … while evaluating the attribute 'cargoconfig' of the derivation 'tvix-eval-deps-0.1.0'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'builtDependencies' of the derivation 'tvix-eval-0.1.0'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

It seems this is caused by some behaviourial change of builtins.fetch, causing the URL used for git-lfs to fetch artifacts lost in translation.

ref: nix-community/rnix-parser#142

[Patryk27]

Somewhat unfortunately, this looks like a missing functionality in Nix itself - for some context, in order to clone repositories, there are two options:

• builtins.fetchGit (supports private repositories, doesn't support LFS),
• nixpkgs' fetchgit (doesn't support private repositories, supports LFS).

Naersk currently uses builtins.fetchGit, since in practice the ability to clone private repositories is more important that supporting LFS.

In theory though it should be possible for Naersk to expose a configuration option using which you'd say for cloning rnix-parser-version, please use fetchgit - but I'd like to avoid implementing workarounds for what should be an improvement to the Nix evaluator itself.

So tl;dr if someone has a use case for this kind of feature (where you can't easily work around it, as compared to that change in rnix-parser), please feel free to re-open this issue (or drop a pull request); but for the time being I'm closing it as as not strictly naersk's fault 😇