Yarn's Future - v2 and beyond

[arcanis]

Hi everyone,

When the Yarn project started back in 2016, our landscape was very different from what it is now. Package locking was far from being a first class citizen in the Javascript ecosystem, and the time needed to run an install was ... well, it was what it was. Yarn's release shook the status quo and started a movement that ended up being beneficial to everyone, other package managers included. Now is the time to assess where we are, reinforce our strengths and patch our weaknesses.

This thread aims to expose our roadmap for the next major Yarn release, and let you know about significant changes that we plan to make regarding Yarn's design. The codename for these changes is Berry - that's how we'll refer to it during the next few months.

Roadmap

This section lists some of the work we have in progress. It's far from being a comprehensive list, but it gives a good idea of the kind of changes you can expect going forward:

Major changes

• The lockfile (and configuration) format will become a strict subset of YAML. In the process, the lockfile information will be slightly changed to account for some long-awaited changes (such as #5892).

• We'll add support for plugins, which will be able to alter various things - from adding new commands to hooking into the resolution / fetching / linking steps to add support for new package sources or install targets.

• Related to the plugin system, Yarn will become an API as much as a CLI. You can expect to be able to require it and start using its components in your script - no need to parse your package.json anymore, no need to run the resolution .. Yarn will abstract all those tedious tasks away.

• Support for both Node 4 and Node 6 will be dropped. We don't expect Berry to have stable releases until after Node 6 gets EOL (expected April 2019), by which point it won't matter anymore.

• The log system will be overhauled - one thing in particular we'll take from Typescript are diagnostic error codes. Each error, warning, and sometimes notice will be given a unique code that will be documented - with explanations to help you understand how to unblock yourself.

• Some features currently in the core (such as autoclean) will be moved into contrib plugins. They'll still be supported, but might have a different release cycle than the standard bundle.

• The codebase will be ported from Flow to TypeScript. To understand the rational please continue reading, but a quick summary is that we hope this will help our community ramp up on Yarn, and will help you build awesome new features on top of it.

• The cache file format will switch from Tar to Zip, which offer better characteristics in terms of random access.

New features

• Nested workspaces (when one of your workspaces itself defines additional workspaces) will be supported out of the box. This will open up new exciting usages, especially when used together with git submodules.

• Running yarn run foo:bar (note the :) will run the foo:bar script regardless of which workspace implements it. This will make working with monorepos a lot easier, as you won't have to constantly cd between directories anymore.

• Running yarn ./packages/my-package add foo will run yarn add foo into my-package. This should also contribute to make monorepos more pleasant, as you won't have to cd to add a new dependency somewhere.

• A new resolution protocol, workspaces:, allows you to force the package manager to link one of your packages against a workspace. This makes it impossible to accidentally use a version of a package obtained from the remote registry.

• A new command will appear, yarn constraints. This command will allow you to enforce constraints across workspaces - for example to require all your declared dependencies to use a unique version of a package.

• A new type of dependencies will be introduced, portals. Portals are like link:, but follow the chain of dependencies (whereas link: dependencies never have any dependencies, but don't require a package.json either).

• The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play 😉).

• Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.

• Scripts will be able to put their arguments anywhere in the command-line (and repeat them if needed) using $@. Similarly, scripts will have access to $1, $2, etc.

• The PnP hook will be able to read files directly from the cache Zip archives - not unlike the phar archives in php.

• Our cache will finally become fully atomic - multiple Yarn instances will be able to run concurrently on the same cache without risking corrupting the data.

And of course, a large amount of bugs are also expected to disappear as we modernize the codebase. A lot of those were caused by intricacies in the various components making up Yarn and, by cleanly separating them into standalone steps, we expect the overall logic to become much clearer and less error prone.

As you might guess, this isn't a comprehensive list - we plan Berry to be the big refactoring that Yarn had needed for a while now so many internal systems will be reworked. After that we expect following majors to become much smaller, as we'll have strong foundation for building new features and reworking the project one component at a time.

Yarn's philosophy

Now that we've discussed about most of the "What", let's discuss about the "Why". First, those are the values we always believed in and around which we continue to design most of our features.

Developing Javascript projects shouldn't leave the door open to surprises

This is our number one motto. We believe that Yarn should have a totally predictable and reproducible behavior. That your package manager has a responsibility to warn you when your project relies on unsafe patterns that could jeopardize your application at the worst possible time.

Developing Javascript projects should be easy

The complexity of the average Javascript project grown during the past years in order to accommodate with the ever increasing number of use cases people found. We believe that the package managers are in a unique position to help decrease some of the cognitive load and make it easy for their users to manage their projects and follow good practices.

Contributing to the Yarn ecosystem should be simple

Because a single project will never be able to satisfy all the needs from all developers counting on us, various projects spawned and grown, reinforcing Yarn rather than competing against it. Some examples are Create-react-app, Lerna, OpenCollective, or patch-package. We like that approach and want to give you the tools to do even more in a more efficient way. Not all features belong to the core, but being able to innovate and experiment is at the core of our philosophy, and we really mean it.

Goals for 2019

Now that our values are clear, what are the next steps we need to do in order to reach our goal? Note that some are long-term, some short-term, and that all are open to discussion (ideally in separate threads, otherwise it might get noisy 🤡).

Yarn will become a development-first tool

We've been saying it for some time now, but now we're ready to start going for it. Package managers are not tools you should ever run on your production servers. The more code run there, the higher are the probabilities that something will eventually go wrong and bring your production systems down. Yarn being development-first means that we'll make it possible for you to reach the state where you clone a repository, and voilà. This includes putting the emphasis on Plug'n'Play, which has been designed with this specific use case in mind.

Yarn will be rewritten in TypeScript

Yarn is currently fully covered by Flow types, and to be honest it works pretty well for us - the key part being “us”. Because we want to make it as easy as possible for third-party contributors to shim in and help us maintain this awesome tool, we'll switch the codebase to TypeScript. We hope this will help making the codebase feel more familiar than the projects your already contribute to.

Yarn will become an API, and its internal components will be split into modular entities

This one is huge. At the moment, when working with Yarn, your only option is the command line interface. We don't offer primitives that allow you to take advantage of the complex logic we've implemented - whether it's the resolvers, linkers, or even accessing the configuration. This will change, and Yarn will be redesign to become an API first and a CLI second.

Yarn will be designed to support different install targets than Node

Package management is a problem that keeps on being reinvented - even Yarn itself did it in the past. We believe the reason for this is that all package managers need to layout the installed packages in slightly different ways in order for them to be read by the host. This unfortunately is rarely possible, and it ends up being easier to rewrite the package manager and discard the already-existing user experience and features. Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Overall compatibility will be preserved when possible

Such semver-major changes are by design backward-incompatible, but we'll make sure to keep them to an acceptable level. In particular, the core command (yarn install, yarn add, yarn remove, yarn run) will keep the same behavior, and your previous yarn.lock will be silently migrated. One important caveat: our installs will now use Plug'n'Play by default. We've proven during the past few months that the approach was sound, worked to trim all potential implementation issues, and finally discussed with project maintainers to figure out whether there was anything we could do to help them make sure everything was ready. Now is the time to take the plunge.

FAQ

When will it be released?

We'll open a new repository early February with the v2, the time to flesh out some remaining interfaces. Starting from then, we'll make Berry accessible simply by running yarn policies set-version berry within your project.

What will happen to the v1?

As mentioned before, Berry won't have a stable release until this summer, and work will continue on the v1 until then. Even after Berry comes live, the v1 branch will still stay well maintained for the foreseeable future, perhaps distributed through a codename. This will ensure that your applications that work now continue to work in the future, until you feel ready to move to the v2.

How can i help?

The biggest way you could help us at the moment would be to enable PnP (yarn --pnp) and report any issue to the maintainers of the packages that don't work yet (ideally with a PR if you have the time). Most of the work they would need to do is to make sure they use the standard require.resolve api, or offer a way to configure a different resolution strategy. At the moment the biggest blocker we have is the tsc default CLI, which doesn't do either (thankfully ts-loader is just fine - we actually use it in order to compile our bundles!).

[Jessidhia]

[*]: Lack of tsc support also affects code completion in VS Code, as it uses the same code as tsc for looking them up.

It may appear to work out of the box sometimes but that's only for types published in DefinitelyTyped which are fetched in the background by the IDE.

---

This small bit of negativity aside, really looking forward to the future, in particular to better workspace workflows 🎉

And, who knows, tsc support for pnp 😇

[felixfbecker]

Love the idea of plugins. Use cases I have are filtering out dependencies that should not be installed (by looking at their package.json metadata or files list) and filtering out files that aren't needed and should not be extracted to disk.

[jleclanche]

Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.

That sounds promising. Does that mean no more need for things like cross-env to be able to do MY_ENV=foo some-command?

[bartlangelaan]

Question!

The lockfile (and configuration) format will become a strict subset of YAML.

Yarn will be designed to support different install targets than Node [...]
Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Does this also mean that the yarn.lock is going to be optional, and Yarn would be able to (with the help of plugins) read and write to (for example) package-lock.json and in the future composer.lock?

Or will Yarn still depend on the yarn.lock file to save the exact state of the Node / PHP dependency tree?

It would be awesome if Yarn can be used on projects built for NPM.

[arcanis]

Ah ah, good question - no (mostly), and yes (but not really) 🙂

The new architecture will see its components moved into various packages - for example plugin-http will only contain the code needed to resolve and fetch the packages when referenced through https?: URLs. Those plugins will all use the API interfaces exposed by the core package in order to work properly - and because (at least at the beginning) the core package will handle the generic resolution algorithm, it will be its task to load the lockfile.

The core package will not support anything else than yarn.lock. We have to put the line somewhere, and given that we don't want people to use Yarn and npm on the same projects (use any of them, but stay consistent) we won't make efforts to support this use case. Additionally, the npm lockfile has characteristics that simply don't make it suitable for us - and the other way around. We've actually discussed it before (#5654).

That being said, I guess you could reimplement the core and read from the package-lock.json file. It's just that it wouldn't map well with the design of the engine, and would (imo) be counter-productive. Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

[mbeaudru]

The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play 😉).

So when a package will be linked into a project, even if the package finds it's dependency in it's surrounding before finding it in the project, it will use the peerDependendy provided by the project anyway? Ref: #6888

Does it means it will be required to use Yarn PnP to use this feature ?

Anyway, thanks for the share this list is awesome!

[arcanis]

So when a package will be linked into a project, even if the package finds it's dependency in it's surrounding before finding it in the project, it will use the peerDependendy provided by the project anyway?

Yep.

Does it means it will be required to use Yarn PnP to use this feature ?

Yes, this is only possible because PnP is aware of the full dependency tree (when it would otherwise be Node that would try to locate the link dependencies, based on the realpath).

[jleclanche]

Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

@arcanis Stepping away from the "using the same package-lock.json file" question; was it considered to use the same format as npm's package lock? Or a superset of it?

I really like yarn.lock becoming programmatically accessible as YAML, but using the same format would make it easier to read both using the same tooling.

[arcanis]

was it considered to use the same format as npm's package lock? Or a superset of it?

Yes, but as I mentioned the npm format has (what we believe are) design flaws that make it incompatible with our goals. We've already got this discussion in the linked thread 😉

[h0ds]

Can't wait!

[bartlangelaan]

That being said, I guess you could reimplement the core and read from the package-lock.json file. It's just that it wouldn't map well with the design of the engine, and would (imo) be counter-productive. Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

I understand completely that it can lead to unexpected results, but as long as its logic can be overwritten with plugins, it can be easier to migrate and convince others to use Yarn on already existing NPM and Composer projects. But of course, only with the help of plugins; this helps keep the core stable.

[flybayer]

Is there any way to add npm’s audit feature to yarn? IMO, that is a huge step toward better security, and I would love to have inside yarn.

[arcanis]

Yarn already implements audit since the 1.12 thanks to @rally25rs's efforts 🙂

[kevinSuttle]

Yarn will be designed to support different install targets than Node

Hoping for solid interop with Rust crates!

[michax]

Yarn will be designed to support different install targets than Node

Really cheering for that one!

[ajbogh]

Would love a future writeup on the Flow=>Typescript conversion from the perspective of Yarn - difficulties/ease, time involved, costs, benefits, etc.

[bondz]

What will the installation story of yarn look like?

If there are different plugins and yarn targeting platforms outside of Node.js.

[erikyuzwa]

totally interested to see where this goes in terms of deployments without running yarn install on production servers...thanks for setting this topic up @arcanis

[jcreamer898]

Yarn will be rewritten in TypeScript

🎉

[soederpop]

This is great.

Yarn will become an API, and its internal components will be split into modular entities

This is by far the best thing IMO especially combined with plugins.

Yarn workspaces transformed how I work with javascript. With an API and the ability to write plugins, I can imagine workspaces being made a lot smarter.

Here is a common problem I run into, and maybe there is a configuration based approach to fixing it but I've always relied on custom scripts

For example, a in workspaces project layout like

├── apps
│   ├── webapp-1/package.json
│   ├── webapp-2/package.json
├── components
│   ├── table/package.json
│   ├── form/package.json
│   ├── field/package.json
├── layouts
│   ├── collapsible-side-menu/package.json
│   ├── long-scrollable/package.json
├── themes
│   ├── light/package.json
│   ├── dark/package.json

If multiple teams or developers are working on this, there are some who are constantly touching layouts, and components, others who primarily work on themes, and others who build apps using these dependencies.

yarn workspace's behavior where it symlinks to the local project instead of the npm dependency is great, however it makes it so if I clone this repo from scratch and want to work on an app, I have to build all the themes, layouts, etc as well

i've worked around this with custom scripts, but they're hard to maintain, feel wrong, and involve a lot of shelling out to yarn to do work.

i think plugins + an api for yarn would make this so much cleaner.

Keep up the great work. This is one of the most exciting roadmaps i've seen in a while for a project.

[3stacks]

Why yaml for the lockfile? Presumably, the v1.x json lockfiles won't be compatible with the v2 yaml ones. Doesn't it make sense for it to be json for the native node support?

Maybe someone can explain the benefit, cause I'm not seeing it.

lockfile just jumped the 🦈

[htoooth]

i hope yarn will add yarn tasks , it can list all tasks in package.json scripts.

[taitung]

Just a quick thought
echo Developing Javascript projects should be easy
Setting up test is a bit tedious regardless of it is jest or mocha...
Any ideas or thoughts incorporating test (should be the unit test) but allow flexibility also.

[asn007]

@htoooth if I recall correctly, invoking yarn run without additional arguments serves this purpose

[xiaoxiangmoe]

Yarn will be designed to support different install targets than Node

Will yarn support deno soon?

[DrSensor]

I hope yarn also support hjson or json5 as an alternative of YAML to maintain compatibility (look&feel) with package.json 😔

[arcanis]

Yes and no. It's likely the publish workflow will be kept relatively similar to what currently exists now, but we'll be looking forward to see different implementations of it (through plugins). If an implementation becomes clearly superior, I could see us moving it into the core.

[samsieber]

Does this issue supersede #5773 (🦁 Yarn 2.0 Working Group ) - if so can we close it?

[grantila]

Fantastic @arcanis, I tried to replace the https?: fetching to my own fetch-h2 (which handles HTTP/1(.1) and HTTP/2 transparently) to see if it'd be faster to use HTTP/2 (most public repositories handles it). This was surely painful, due to the implicit dependency of request, and its arguments being passed around the yarn codebase.

A plugin structure sounds great. I'd be happy to make a yarn-plugin-http2 (and if it proves to be fast, hand it over to you guys if you want it in the core).

[arcanis]

Update: I've published the trunk for the v2 on a separate repository (at least for now, in order to better triage issuers): https://github.com/yarnpkg/berry

It's already working mighty fine, but as you can guess we've still some way to go until we can call it a day. At the moment I'd say most of the remaining work are missing features (like outdated, upgrade-interactive, etc) and documentation, which are two things that anyone can help with! So if you want to have a strong positive impact on a project totalizing more than 34,000 stars on Github, look no further! 🙂

[ExE-Boss]

What about using a more pnpm‑like system (i.e. preventing access to packages that aren’t declared in dependencies, devDependencies, peerDependencies or optionalDependencies).

[creatyvtype]

Do you have an anticipated release date yet for v2?

[arcanis]

When it's done! 😄

More seriously it's quite stable already - it can easily install itself, and I use it daily. There are still some important tasks that need to be done in order to start releasing it more widely, and any help we can get would be seriously impactful. I've listed various tasks in our issues and would be happy to give more details over there! Even documentation writers or early adopters would put us miles ahead 🙂

Overall, my target would be to have an official release somewhere this summer, but it might shift both ways depending on the progress and other external factors.

[dherges]

Following thoughts are more of a feature request.

Right now, yarn is installed via package managers on a machine.

What about a portable yarn wrapper?

What's the problem I'm trying to solve?

• hermecity of builds
• reproducible builds

What's my current idea?

1. Check in an executabe file yarnw to version control repo.
2. Check in portable yarn/node version info in .yarnwrapperrc (filename tbd), example content could be yarn.version=1.15.2 \n node.version=10.15.3
3. Check out repo.
4. Run ./yarnw install, ./yarnw build (and bunch of other scripts from package.json)

Expected behaviour

$ ./yarnw install

Welcome to the portable yarn wrapper - best in town!

Checking ~/.yarn/wrapper/node_versions/10.15.3
Node 10.15.3 already installed. Ready to go!

Checking ~/.yarn/wrapper/yarn_versions/1.15.2
Downloading yarn 1.15.2 from https://yarnpkg.com/<version>.tar.gz
Downloaded.

Running yarn install (would execute ~/.yarn/wrapper/node_versions/10.15.3/node -e "require('~/.yarn/wrapper/yarn_versions/1.15.2/libexec/lib/cli.js').default()" ???  )

yarn install v1.15.2
<usual yarn experience>
<...>
Stars Done in took-some-time.

Creating yarn wrapper

$ yarn wrapper
<create .yarnwrapperrc file in cwd>

Where I am inspired from?

Gradle wrapper: https://docs.gradle.org/current/userguide/gradle_wrapper.html

[Bessonov]

@dherges I'm not sure, but I think this discussion may be interesting for you.

TBH, I'm strongly against such things like gradlew or yarn policies set-version.

[arcanis]

I'm now going to close this thread as I believe I've answered most of the questions related to the plan outlined in the first post. Feature requests should be put into their own threads.

Note that the v2 implementation is going strong on the following repository: github.com/yarnpkg/berry, and we're always happy to receive feedback (and contributions!) from early adopters.

I hope to see you there! 😃

[arcanis]

Hi everyone! After a year of development we've just put Yarn 2 out of beta and into stable. For more information, please read the following post: https://dev.to/arcanis/introducing-yarn-2-4eh1

This is an exciting day - thanks to everyone who helped us on our journey 🌟🌟🌟