Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a --fast-deps option to comfy install/reinstall that enables new faster/unified/uv-based dependency install #141

Merged
merged 30 commits into from
Aug 16, 2024

Conversation

telamonian
Copy link
Member

@telamonian telamonian commented Jul 26, 2024

This PR adds a new option --fast-deps, to comfy install and comfy reinstall. This option enables the new dependency handling implementation. The features of said new dependency handling currently cover:

  • Uses uv to install all Python deps (very fast)
  • Resolves all core and custom extension dependencies together
    • Any conflicts between core and extensions are settled in core's favor
    • TODO: extension-vs-extension dependency conflict handling
  • Minimizes total number of subprocess calls to external Python dependency-handling tools

This PR depends on ltdrdata/ComfyUI-Manager#886, which adds support for a --no-deps option to the lower level cm-cli interface. Although ComfyUI-Manager#886 has been merged, the relevant code has not yet been pushed to the main branch of ComfyUI-Manager. So if you want to test this PR out yourself, you'll have to manually switch to the feat/cnr branch on the instance of ComfyUI-Manager that comfy-cli installs.

@telamonian telamonian force-pushed the fast-deps branch 2 times, most recently from 1ab6123 to d742327 Compare July 26, 2024 21:14
- output is currently unsatisfactory in the case of a dependency conflict
  - doesn't print the names of conflicting top-level packages. The uv maintainers have expressed interest in fixing this: astral-sh/uv#1854
@telamonian
Copy link
Member Author

Okay, the code in this PR is coming along. Fixed a bunch of bugs/problems, and thanks to #147 this PR is now much easier to test out for yourself. Just install this PR branch, then do:

comfy install --fast-deps --manager-url=https://github.com/ltdrdata/ComfyUI-Manager.git@feat/cnr

specifying any other args as normal. The fast-deps option will now also work with installing custom nodes, eg:

comfy node install --fast-deps comfyui-kjnodes comfyui-animatediff-evolved

@telamonian
Copy link
Member Author

telamonian commented Aug 9, 2024

Here's the features/TODO list for the PR:

  • add --fast-deps option to trigger the new install behavior
    • (optional) ensure that all possible relevant commands support --fast-deps
      • [node] install
      • [node] reinstall
      • others?
  • speed up python dependency installs
    • write uv-based installer
    • minimize calls to subprocess/batch all possible installs
  • Compile all dependencies to explicit versions before install
    • pre-compile step to calculate overrides
    • unified compile (core + exts)
  • UI/UX for conflicts
    • core-vs-extension conflicts are always settled in favor of core
    • resolve extension-vs-extension conflicts via at least one of:
  • add special case handling
  • finalize tests

So the big remaining issues are adding some kind of extension-vs-extension conflict resolution and finishing up any tests for the new installer. Unfortunately the obvious easy path to implement ext-vs-ext behavior is a no-go; due to an upstream issue there's no way currently for us to report to the user which extensions are actually in conflict, so we can't just print out the relevant info and then tell the user to manually resolve the conflict

@robinjhuang robinjhuang changed the title Add a --fast-deps option to comfy install/reintstall that enables new faster/unified/uv-based dependency install Add a --fast-deps option to comfy install/reinstall that enables new faster/unified/uv-based dependency install Aug 10, 2024
comfy_cli/command/install.py Outdated Show resolved Hide resolved
comfy_cli/command/install.py Outdated Show resolved Hide resolved
DEV_README.md Show resolved Hide resolved
comfy_cli/uv.py Outdated Show resolved Hide resolved
@telamonian
Copy link
Member Author

Okay, unittest is now functional. Said test needs a small fix tho (I realized that currently it will break whenver sympy or numpy have a new release, which is not ideal). I'll add the fix and address all of the feedback tomorrow, and then I'll merge

@robinjhuang
Copy link
Member

Okay, unittest is now functional. Said test needs a small fix tho (I realized that currently it will break whenver sympy or numpy have a new release, which is not ideal). I'll add the fix and address all of the feedback tomorrow, and then I'll merge

Sounds great!

@telamonian telamonian merged commit 71cd6ef into Comfy-Org:main Aug 16, 2024
8 checks passed
rocmPytorchUrl = "https://download.pytorch.org/whl/rocm6.0"
nvidiaPytorchUrl = "https://download.pytorch.org/whl/cu121"

overrideGpu = dedent("""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the need for override.txt file?

self.override.unlink(missing_ok=True)

with open(self.override, "w") as f:
if self.gpu is not None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if self.gpu is not None and self.gpuUrl is not None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants