Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nix flake not working in git repository with LFS #137998

Closed
laikq opened this issue Sep 15, 2021 · 11 comments
Closed

Nix flake not working in git repository with LFS #137998

laikq opened this issue Sep 15, 2021 · 11 comments
Labels
0.kind: bug Something is broken 6.topic: flakes The experimental Nix feature

Comments

@laikq
Copy link
Contributor

laikq commented Sep 15, 2021

Describe the bug

My nix flake cannot be built. The git repository of the flake uses LFS. Error message:

❯ nix build ".#my-project"
Downloading cvode-2.5.0.tar.gz (1.8 MB)
Error downloading object: cvode-2.5.0.tar.gz (c6f0094): Smudge error: Error downloading cvode-2.5.0.tar.gz (c6f0094d907854488fd68efadce8c4890cc6ba518313bc26e13031838255f976): batch request: missing protocol: ""

Errors logged to /tmp/nix-133890-1/lfs/logs/20210915T123907.095919332.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: cvode-2.5.0.tar.gz: smudge filter lfs failed
error: program 'git' failed with exit code 128
(use '--show-trace' to show detailed location information)

(here, cvode-2.5.0.tar.gz is a file already present & downloaded in the working directory)

Using nix.package = pkgs.nixUnstable; in configuration.nix, where pkgs is from https://nixos.org/channels/nixos-21.05 on 2021-09-15. It also doesn't work if I try to pull the flake from remote: nix build "git+ssh://git@gitlab.com…" still fails with exactly the same error message.

Using https://github.com/nixos/nixpkgs/archive/7cf9185aaeb95f79cfe9cae8689433434576aaff.tar.gz (release-21.05 from 2021-09-05) as pkgs, it works again.

Steps To Reproduce

I don't have a minimal working example yet, sorry :/ (I first wanted to sketch out the issue, don't have much time available right now) I guess it will happen with many git lfs repositories?

Expected behavior

Package should be built.

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
this path will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev
copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.62, NixOS, 21.05.3065.b3083bc6933 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210802_47e96bb`
 - channels(root): `"emacs, nixos-21.05.3065.b3083bc6933, nixos-hardware, nixos-old-20.09.4393.068984c00e0, nixos-unstable-21.11pre314557.364b5555ee0, nixos-unstable-small-21.11pre314602.db88608d8c8, nixos-week-ago"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: nixUnstable
# a list of nixos modules affected by the problem
module:
@laikq laikq added the 0.kind: bug Something is broken label Sep 15, 2021
@mohe2015
Copy link
Contributor

mohe2015 commented Sep 15, 2021

cd /tmp/
mkdir test
cd test/
git init
git lfs install
curl -OL https://github.com/NixOS/nixpkgs/archive/refs/heads/master.zip
git lfs track "*.zip"
git add .gitattributes
git add .
git commit -m "Initial commit"
git lfs ls-files
cat /tmp/test/flake.nix
{
  outputs = inputs@{ self, nixpkgs }:
  {
    legacyPackages = nixpkgs.legacyPackages;
  };
}
nix build .#hello

does not reproduce for me (on nixos-unstable though). Does it for you on your channel? (Maybe you need to download a smaller zip for me that took really long). Does your flake.nix look different?

@mohe2015

This comment has been minimized.

@mohe2015
Copy link
Contributor

mohe2015 commented Sep 15, 2021

Okay got it. This is the exact reproducer you just need to push it somewhere and then clone the remote. See https://github.com/mohe2015/nixpkgs-137998-reproducer

git clone https://github.com/mohe2015/nixpkgs-137998-reproducer.git
cd nixpkgs-137998-reproducer/
nix build .#hello

reproduces.

Edit: I won't work on a fix for now but now it should be easy to reproduce.

@mohe2015
Copy link
Contributor

mohe2015 commented Sep 15, 2021

Can confirm that

rm flake.lock # important
nix run github:NixOS/nixpkgs/7cf9185aaeb95f79cfe9cae8689433434576aaff#nixUnstable build .#hello

works but

rm flake.lock # important
nix run github:NixOS/nixpkgs/release-21.05#nixUnstable build .#hello

doesn't. So it's 21c1c06#diff-b94b72b12e07bed8c4a3b4c50242499d5efb1e32797426d319574fd613bc07ae and therefore one of NixOS/nix@47e96bb...3c56f62

Current bisect status:

git clone git@github.com:NixOS/nix.git
cd nix
git bisect start
git bisect good 47e96bb533f8cacc171bec9b688b134de31a48a9
git bisect bad 3c56f62093601143838af923195f630d8ffae2d4

cd ..
cat test.sh
#!/bin/sh
rm -f ../flake.lock
nix build .#nix || exit 125
nix run .#nix build ..#hello || exit 1

chmod +x test.sh
cd nix

git bisect run ../test.sh

@mohe2015
Copy link
Contributor

Potentially looks like duplicate of NixOS/nix#4623

@mohe2015
Copy link
Contributor

mohe2015 commented Sep 15, 2021

git bisect log
git bisect start
# good: [47e96bb533f8cacc171bec9b688b134de31a48a9] Merge pull request #5082 from manveru/fix-git-init-race
git bisect good 47e96bb533f8cacc171bec9b688b134de31a48a9
# bad: [3c56f62093601143838af923195f630d8ffae2d4] Merge pull request #5225 from ncfavier/patch-1
git bisect bad 3c56f62093601143838af923195f630d8ffae2d4
# good: [323cafcb4ec632881c297f9b3004ea7fb07b678f] Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
git bisect good 323cafcb4ec632881c297f9b3004ea7fb07b678f
# good: [7ee639f9db987a442fdf3820d3681a77b2ee1688] Merge pull request #5066 from Radvendii/master
git bisect good 7ee639f9db987a442fdf3820d3681a77b2ee1688
# good: [a4d02d100afba729b09c43ee19f2398c529dc904] Merge pull request #5204 from lucc/patch-1
git bisect good a4d02d100afba729b09c43ee19f2398c529dc904
# good: [90b2dd570cbd8313a8cf45b3cf66ddef2bb06e07] Add FIXME
git bisect good 90b2dd570cbd8313a8cf45b3cf66ddef2bb06e07
# bad: [6678e98411cd3bfe8143a4ba1b35d1451babce32] Merge pull request #4922 from nrdxp/default-submodules
git bisect bad 6678e98411cd3bfe8143a4ba1b35d1451babce32
# skip: [769ca4e26d973660756de85b51efdb7fc36a00fd] libfetchers/git: fetch submodules by default
git bisect skip 769ca4e26d973660756de85b51efdb7fc36a00fd
# only skipped commits left to test
# possible first bad commit: [6678e98411cd3bfe8143a4ba1b35d1451babce32] Merge pull request #4922 from nrdxp/default-submodules
# possible first bad commit: [769ca4e26d973660756de85b51efdb7fc36a00fd] libfetchers/git: fetch submodules by default
git revert 769ca4e26d973660756de85b51efdb7fc36a00fd

NixOS/nix@769ca4e Really seems to be the problem.
@laikq You could probably revert that commit for now to get this to work. But for long-term fixes see the discussion in the referenced issue.

Edit: @laikq If it's just that .tar.gz file you could probably also fetch it with nix directly and not use lfs at all.

@mohe2015
Copy link
Contributor

It seems like you sometimes need to garbage collect to reproduce btw.

@roberth
Copy link
Member

roberth commented Sep 16, 2021

It seems like you sometimes need to garbage collect to reproduce btw.

This may or may not be like the regular fetchers. If Nix already has the output with the right name and hash in the store, it doesn't need to fetch it again. This is usually a good thing, unless you are testing changes.

@stale
Copy link

stale bot commented Apr 25, 2022

I marked this as stale due to inactivity. → More info

@reivilibre
Copy link
Contributor

reivilibre commented Oct 6, 2022

I have also hit this.
I've specified git+ssh://gitea@myrepo.tld/reivilibre/myname.git?ref=master&rev=f62d1a318184e32f9f30b42683566d18d341bddb as the url of one of my flake inputs and then tried to nix flake lock.

git-lfs/3.1.4 (3.1.4; linux amd64; go 1.17.13; git v3.1.4)
git version 2.36.2

$ git-lfs filter-process
Error downloading object: assets/sfx/Happy/file-sounds-1079-cheerful-1.ogg (16a6034): Smudge error: Error downloading assets/sfx/Happy/file-sounds-1079-cheerful-1.ogg (16a6034bd3e04143907b5b97b6414f93fdea445181ae759e0f4f7e2d29df6324): batch request: missing protocol: ""

missing protocol: ""
github.com/git-lfs/git-lfs/errors.New
        github.com/git-lfs/git-lfs/errors/errors.go:62
github.com/git-lfs/git-lfs/lfshttp.(*Client).NewRequest
        github.com/git-lfs/git-lfs/lfshttp/client.go:134
github.com/git-lfs/git-lfs/lfsapi.(*Client).NewRequest
        github.com/git-lfs/git-lfs/lfsapi/client.go:13
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
        github.com/git-lfs/git-lfs/tq/api.go:84
github.com/git-lfs/git-lfs/tq.Batch
        github.com/git-lfs/git-lfs/tq/api.go:43
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
        github.com/git-lfs/git-lfs/tq/transfer_queue.go:559
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
        github.com/git-lfs/git-lfs/tq/transfer_queue.go:453
runtime.goexit
        runtime/asm_amd64.s:1581
batch request
github.com/git-lfs/git-lfs/errors.newWrappedError
        github.com/git-lfs/git-lfs/errors/types.go:229
github.com/git-lfs/git-lfs/errors.Wrap
        github.com/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
        github.com/git-lfs/git-lfs/tq/api.go:86
github.com/git-lfs/git-lfs/tq.Batch
        github.com/git-lfs/git-lfs/tq/api.go:43
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
        github.com/git-lfs/git-lfs/tq/transfer_queue.go:559
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
        github.com/git-lfs/git-lfs/tq/transfer_queue.go:453
runtime.goexit
        runtime/asm_amd64.s:1581
Error downloading assets/sfx/Happy/file-sounds-1079-cheerful-1.ogg (16a6034bd3e04143907b5b97b6414f93fdea445181ae759e0f4f7e2d29df6324)
github.com/git-lfs/git-lfs/errors.newWrappedError
        github.com/git-lfs/git-lfs/errors/types.go:229
github.com/git-lfs/git-lfs/errors.Wrapf
        github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
        github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:120
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
        github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:79
github.com/git-lfs/git-lfs/commands.smudge
        github.com/git-lfs/git-lfs/commands/command_smudge.go:128
github.com/git-lfs/git-lfs/commands.filterCommand
        github.com/git-lfs/git-lfs/commands/command_filter_process.go:123
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
        github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:766
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:852
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
        github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:800
github.com/git-lfs/git-lfs/commands.Run
        github.com/git-lfs/git-lfs/commands/run.go:107
main.main
        github.com/git-lfs/git-lfs/git-lfs.go:34
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
        github.com/git-lfs/git-lfs/errors/types.go:229
github.com/git-lfs/git-lfs/errors.NewSmudgeError
        github.com/git-lfs/git-lfs/errors/types.go:315
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
        github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:88
github.com/git-lfs/git-lfs/commands.smudge
        github.com/git-lfs/git-lfs/commands/command_smudge.go:128
github.com/git-lfs/git-lfs/commands.filterCommand
        github.com/git-lfs/git-lfs/commands/command_filter_process.go:123
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
        github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:766
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:852
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
        github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:800
github.com/git-lfs/git-lfs/commands.Run
        github.com/git-lfs/git-lfs/commands/run.go:107
main.main
        github.com/git-lfs/git-lfs/git-lfs.go:34
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581

Did you ever manage to resolve this?

@SuperSandro2000
Copy link
Member

This is a problem of the nix package manager and the linked issue, not nixpkgs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken 6.topic: flakes The experimental Nix feature
Projects
None yet
Development

No branches or pull requests

6 participants