Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow packaging of pnpm workspaces project #119

Open
jonim8or opened this issue Oct 31, 2024 · 12 comments · May be fixed by #121
Open

Slow packaging of pnpm workspaces project #119

jonim8or opened this issue Oct 31, 2024 · 12 comments · May be fixed by #121

Comments

@jonim8or
Copy link

What version of pkg are you using?

6.0.0, 5.16.0

Which version(s) of pkg work for you?

5.15.0

What version of Node.js are you using?

20.18.0

What operating system are you using?

Windows

What CPU architecture are you using?

x86_64

What Node versions, OSs and CPU architectures are you building for?

node20-linux-x64

Describe the Bug

I have a pnpm project using workspaces. build time is significantly slower in pkg 5.16 onwards. Our project builds in 22 seconds using pkg 5.15, and 35 minutes when using pkg 5.16 or later.

Expected Behavior

Build time should be similar for both. Or at least in the same order of magnitude.

To Reproduce

Unzip the reproduce.zip folder.

reproduce.zip

run these commands:

pnpm install
time pnpm -r artifacts

then in packages/foo/package.json change the version to ~5.16.0, and repeat the above commands.
For me this are the execution times:

pkg version build time
~5.15.0 0m5.618s
~5.16.0 0m28.320s
~6.0.0 0m30.630s
@robertsLando
Copy link
Member

@SuperchupuDev could this be due to globby/minimatch replacement?

@SuperchupuDev
Copy link

probably, there are some cases where optimizations can't be applied due to the glob patterns used, a workaround can be ignoring node_modules from globbing. sorry about that. there's some ongoing work in tinyglobby to greatly improve optimizations

@robertsLando
Copy link
Member

@SuperchupuDev could you submit a PR with the suggested optimization please? Otherwise I would revert the glob replacement for now as the time is 5x and is too high

@SuperchupuDev
Copy link

SuperchupuDev commented Nov 4, 2024

should be a one line fix with the ignore option, can do, but first i must ask, is the collect function ever supposed to return files from node_modules? if yes i'll have to fix the problem at its core (finishing the better optimizations in tinyglobby), which shouldn't take long as most of the work is done

@robertsLando
Copy link
Member

@SuperchupuDev Yeah that could return files from node_modules

@SuperchupuDev
Copy link

ah, then i'll try to finish the optimizations and publish a new tinyglobby version. meanwhile, can you try to give me a list of glob patterns commonly used by pkg? just to make sure

@robertsLando
Copy link
Member

@SuperchupuDev Glob patterns used there can be provided by users, them are fetched from scripts and assets provided in pkg configuration in package.json

@SuperchupuDev
Copy link

SuperchupuDev commented Nov 4, 2024

interesting. what's the pattern used in the repro provided in this issue? is it the same one that's actually passed to tinyglobby or is it transformed in some way beforehand?

@robertsLando
Copy link
Member

I have no clue, I think you can try with the provided zip in the issue above?

I also have a feel that the reason caused issue listed in #112 test-80-compression-node-opcua, maybe you can give a try with that? Actually that test takes forever and uses pnpm and has assets. See development.md to understand how to spin tests locally

@SuperchupuDev
Copy link

small update: SuperchupuDev/tinyglobby#69 is in theory done, but it breaks vite's tests. will investigate more tomorrow, and merge once i can make sure it doesn't cause any regressions

@SuperchupuDev
Copy link

debugging the repro zip file made me see the issue with the glob usage done in this package: it is calling globSync many times, when once should be enough. improving this should solve this issue without needing to wait for tinyglobby
image

@SuperchupuDev SuperchupuDev linked a pull request Nov 5, 2024 that will close this issue
@SuperchupuDev
Copy link

i've opened #121 which should solve this issue, but the usage of globSync is still not optimal here until this project stops calling globSync more than once, regardless of the glob library used

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants