Optimize git library installation

[tobil4sk]

Closes #493. For most git installs, a (faster) shallow clone can now be performed with --depth=1, which avoids cloning the entire history and instead only clones one commit's worth of history, including for the submodules. The only exception where a shallow clone cannot be performed is when setting a library to a specific commit with a short commit id, for example:

# Reverts to full non-shallow clone:
haxelib git lime https://github.com/openfl/lime c16f278
# Performs shallow clone:
haxelib git lime https://github.com/openfl/lime c16f27818dcff17e8b5e4648624a64ea6a107119

This is due to how commit ids are resolved.

The data which was used to install the library is now stored in a .gitdata file, to avoid issues of overwriting externally created commits. With this .gitdata file, it is now possible to be smarter when reinstalling the same library, which also fixes #512.

Now when updating a git library, it will now only update if it was installed with the default branch or a specific branch. If it was installed with a specific commit or tag, it remains at that commit/tag, since there is no clear commit to update to.

Finally, this also fixes #593, since the submodules have to be cloned after switching to the correct branch in order to clone them with --depth=1 as well.

Mercurial works differently, so I've just left it as is.

[Simn]

@kLabz Could you check if this makes sense git-wise?

[kLabz]

I don't think I'll have time today, so will look at this tomorrow.

It likely breaks some workflow I use from time to time (using git dependency, and doing git commands in .haxelib/somelib/git folder like changing branches etc.) but I should probably use haxelib dev for that as it's easy to lose changes like that...

[tobil4sk]

Somehow this PR causes #591 to be triggered on Windows with pretty much every haxelib git command :/ Probably requires getting rid of neko to fix it, I can't figure out what exactly is going on...

[dpomier]

The only exception where a shallow clone cannot be performed is when setting a library to a specific commit with a short commit id

It should be possible to get the full commit id from the short one once you have added a git remote:

$ git rev-parse 3cdd5d
3cdd5d19178a54d2e51b5098d43b57571241d0ab

Now when updating a git library, it will now only update if it was installed with the default branch or a specific branch. If it was installed with a specific commit or tag, it remains at that commit/tag, since there is no clear commit to update to.

It might be worth always fetching the commit id so as not to lose your optimisation when a branch or tag is specified instead of a commit id. This can be done this way:

$ git ls-remote https://github.com/HaxeFoundation/haxelib.git HEAD     
98637027327d8cf385d302acaaf104bd6107d2bf        HEAD

$ git ls-remote https://github.com/HaxeFoundation/haxelib.git 4.1.x
f17fffa97554b1bdba37750e3418051f017a5bc2        refs/heads/4.1.x

Hope this helps.

[tobil4sk]

It should be possible to get the full commit id from the short one once you have added a git remote:

This doesn't work if the commit hasn't been fetched:

$ git rev-parse 558fd07 
558fd07
fatal: ambiguous argument '558fd07': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

It might be worth always fetching the commit id so as not to lose your optimisation when a branch or tag is specified instead of a commit id.

Not sure I understand, which optimisation are you talking about that would be lost?

[MAJigsaw77]

Any progress on this?

[tobil4sk]

The main issue that's blocking this is #591, there is no point in optimising git commands if it will break them 😅. Maybe it can work in development with the interp target and hxb, but that would mean that it doesn't work with haxe 4.3.x.