Consider future expansion to allow using a registry without new API

[sorvell]

Background

As currently spec'd, registries are coupled to Shadow DOM and require a new opt-in API to facilitate scoped element creation (shadowRoot.createElement/importNode). While this makes good sense for the first version of the API, there are some additional use cases that could be addressed by expanding the API in the future.

Use case

In particular, the micro-frontend architecture (MFE) is a natural fit for web components because of their native interoperability but fundamentally requires scoped registries.

However, MFE's often exist in complex environments where various web frameworks are used together. For MFEs to effectively use scoped registries, ALL frameworks used in them must be aware of and use scoped API for creating DOM. Currently, of course, none of them do. While this can be addressed in the long run, it's sure to be a point of friction and it'd be great to expand the API to more seamlessly support this use case.

Example

An MFE feature is implemented with React. The feature itself (inside the React components) uses custom elements. Currently, this means the React MFE feature should itself be wrapped in a Shadow DOM which uses a scoped registry to define those custom elements. So far so good, but now there's a problem: React doesn't know about shadowRoot.createElement so it doesn't create the correctly scoped elements. An un-upgraded element connected to a shadowRoot with a registry will upgrade with the registry's definition so the only practical problem is a conflict with a globally registered element.

Possible solution

It's tempting to propose adding a way to apply a registry to a DOM subtree that doesn't use Shadow DOM. However, while this might be useful for frameworks that don't expect Shadow DOM for other reasons (e.g. reliance on global styling), it wouldn't typically help solve this problem. That's because it's a very common pattern for frameworks to create elements via document.createElement, and any globally registered element will immediately upgrade on that call.

This should be verified, but I believe the most common pattern generally used by most web frameworks to create DOM is document.createElement + "append." So if there were a general custom elements feature to specify "for this name, only customize the element via upgrade and not creation." Ideally, this could be done independent of the global registration but this could arguably be too powerful. Some API options for how to do add upgradeOnly feature:

• at define time: customElements.define('x-foo', XFoo, {upgradeOnly: true});
  • Pro: explicit
  • Con: requires knowing that this could be a problem
• dynamically: customElements.customizeWhen('x-foo', {upgrade: true})
  • Pro: dynamic and could be done if/when/as needed.
  • Con: changes existing behavior and therefore could violate expectations (perhaps could require an opt-in setting to enable on the registry itself)
• whenever a name is used in a scoped registry: notGlobalRegistry.define('x-foo', XFoo)
  • Pro: seamlessly automatic
  • Con: same changes existing behavior issue

[rniwa]

I think a better solution will be to have createElement function on a custom element registry itself so that the registry can be used independent of shadow trees.

[michaelwarren1106]

as a web component library author, i think a big part of this scoping use case is how to design/implement scoping so that frameworks don’t necessarily have to specifically opt in? if framework updates are required to support scoping, that means that only future applications or app updates can benefit.

is it possible to implement a solution that works with the current document.createElement calls that frameworks make today so that frameworks don’t have to do anything and scoping just works out of the box?

i work on a web component design system library for primarily react consumers. that particular combination of framework + custom elements has been particularly difficult and consumers of the web components often use these kinds of things “not working in react” as negative feedback for web components instead perhaps of being negative feedback for react.

the more features that browser can implement seamlessly without framework adoption the better imo. i don’t have any idea how that could work. but i definitely second the idea of removing the shadowroot restriction on custom registries.

[justinfagnani]

I don't see how this is possible without new API that frameworks use. Ideally frameworks would be flexible enough to show customization of what object/method they use create elements.

The fundamental problem is that existing document.createElement calls do not have enough information to determine a scope if a framework is rendering into a shadow root. There's no way for the platform to determine that either.

I've been able to get React working by patching ownerDocument of the container it's rendering into to return an object that overrides createElement. It's a hack, but I think shows that it would be pretty easy to use a scoped API at that point. Hopefully with little downside to a change, frameworks would accept PRs to used scoped creation APIs when present.

[pascalvos]

out of the box thought, if it would work like a <form> and the inputs inside it. So when you draw a box around it (actual tag) that holds a registry and the custom elements within the tag register to this box, this way you wouldn't need to do anything to do with the creation part.

<registry scope="document"> ... your CE </registry>
<registry scope="my-mfe"> ... your CE </registry>

this also doesn't get in the way of the frameworks and could help solve the problem when you don't have control of the creation.

with the creation using createElement you still have the finer gained control.
this would require a whole new spec of course...

[nolanlawson]

This should be verified, but I believe the most common pattern generally used by most web frameworks to create DOM is document.createElement + "append."

This is not the case – many frameworks such as Solid, Vue Vapor, Svelte v5, and Lit instead use this pattern:

const template = document.createElement('template')
template.innerHTML = htmlString
return template.content.cloneNode(true)

If the htmlString above contains <x-foo></x-foo>, then (AIUI) it would be upgraded in the global scope, since there is no associated shadow root.

I mentioned this in the call, but our (Salesforce's) implementation of scoped registries solves this by conceptually tying a scoped registry to a ShadowRealm, not a ShadowRoot. This indeed requires a lot of global patching, but the upside is that the script can use document.createElement, element.innerHTML, DOMParser, or any other way of creating DOM, and constructed <x-foo> instances are bound to the right "scope" for that particular script.

I don't see how this is possible without new API that frameworks use.

Maybe a v2 of the scoped registries spec could do this instead of boiling the ocean, but for our case at least, we would still need to patch globals to route the global APIs to the right scoped APIs.

[EisenbergEffect]

In the call, I wondered whether we might find value in adding something like this:

const template = document.createElement('template');
registry.run(() => template.innerHTML = htmlString);
return template.content.cloneNode(true);

I think there was agreement that we didn't want to block moving forward with the current proposal, but something like this might be added in round two.

[justinfagnani]

@nolanlawson

Maybe a v2 of the scoped registries spec could do this instead of boiling the ocean

Do what exactly?

[justinfagnani]

registry.run() is an interesting idea... I'd have to think about ways it could break. It's similar to the patch of .ownerDocument I did for React.

One major difference is that in React we can figure out that the framework only creates elements with one specific API, and that it's not reentrant at the point of the patch. Here we'd have to make sure that all element creation APIs invoke the Find a Custom Element Registry step, and that we keep a stack of registries, and we probably want to make this work across async DOM creation operations, so we may need to wait for something like Async Context.

There may be bad cases where undefined elements are upgraded in the global scope because they're not defined in the custom scope and get upgraded in the adopt steps.

[nolanlawson]

Do what exactly?

I mean "expose explicit scoped APIs that the framework must use." E.g. registry.createElement, registry.DOMParser, @EisenbergEffect's registry.run(), etc.

[justinfagnani]

This proposal already does that in the form of ShadowRoot.createElement(), etc. My hope is that when this lands we can update frameworks to use the scoped APIs when available. This should roughly be:

const el = (container.getRootNode().createElement ?? document.createElement)(tagName);

or

const fragment = (container.getRootNode().importNode ?? document.importNode)(template.content, true);

[sorvell]

Just to clarify, I believe in general if there is no global definition, you won't have to use ShadowRoot.createElement. This is because the element will follow the rules of custom element upgrade, meaning it will upgrade when it's attached to the DOM (as long as it's :not(:defined) at that point). So however it's created, as long as it's initially attached to a shadowRoot with a scoped registry, it'll use that registry to upgrade... which is presumably what's expected.

I believe this behavior follows from the Finding a custom element definition section of the proposal and it would also be consistent with how iframes work.

But note, the prototype implemented in Chrome 120 + experimental web platform features does not yet implement this section of the proposal at all, so this doesn't currently work.

This is why I was focusing on a way to preserve the :not(:defined) state for a given, disconnected element via the "upgrade only" concept. This way it'll be a candidate for upgrade when attached to root with the "right" registry.

[michaelwarren1106]

Just to clarify, I believe in general if there is no global definition, you won't have to use ShadowRoot.createElement. This is because the element will follow the rules of custom element upgrade, meaning it will upgrade when it's attached to the DOM (as long as it's :not(:defined) at that point). So however it's created, as long as it's initially attached to a shadowRoot with a scoped registry, it'll use that registry to upgrade... which is presumably what's expected.

This would work for the React MFE use case then I think? Possibly the template.innerHTML approach also as any components would upgrade using whatever registry was associated with the root node of where the cloned template gets added to the DOM?

[nolanlawson]

@justinfagnani Your code example may be a bit contentious for some framework authors, since I imagine it would tank js-framework-benchmark performance (due to having to look up the rootNode for every createElement/<template> clone)… Although maybe it could be mitigated by only running getRootNode for custom elements (e.g. search the tag name for a hyphen).

Just to clarify, I believe in general if there is no global definition, you won't have to use ShadowRoot.createElement. This is because the element will follow the rules of custom element upgrade, meaning it will upgrade when it's attached to the DOM (as long as it's :not(:defined) at that point).

@sorvell This is an interesting solution. I think it might actually solve our use case, since there is no chance that document.createElement('x-foo') would give you a FooComponent that doesn't "belong" to you. (You would need access to a shadowRoot to get access to the scoped custom element definition.)

The downside is that you wouldn't be able to have any top-level custom elements, unless you are willing to expose them globally (to disconnected createElements). In our case, we would probably have to have some kind of dummy <x-app> top-level component that doesn't do anything except encapsulate the whole thing. But maybe that's okay, since script can do document.body.querySelectorAll('*') anyway and get access to those top-level constructors.

[nolanlawson]

OK, I just chatted with @rwaldron (who knows way more about this stuff than I do). Basically, our use case can only be solved if it's possible for legacy code (which uses global customElements.define() and document.createElement()) to use a scoped registry. So this means that findTheShadowRootSomehow().createElement() is not really viable for us, since we can't expect code authors to hook into purely scoped APIs.

In our world, something more like @EisenbergEffect's registry.run(() => {}) would be much closer to what we currently have.

[Westbrook]

What if this is what was on offer for users of w3ctag/design-reviews#334 (or possibly via a new type, e.g. {type: 'scoped-html'} if needed)? In such a case, each HTML module scope would then be known to have had its own customElements so that the registry was scoped, and all the DOM methods were scoped, and all of the villagers were happy.

[sorvell]

This is a nice idea, and I'm thinking it probably should be an explicit choice. The html modules proposal already includes import.meta.document but that almost certainly won't have a defaultView (window object) on which a registry could exist so we may also need import.meta.customElements. And you'd just expose that registry from the module and use it where you need.

I don't think that 100% addresses the needs here, but it is a step in the right direction for sure.

[michaelwarren1106]

i don’t think HTML modules goes very far to supporting scoping in MFE use cases does it? it might if each MFE remote app was itself an html module, but i’m not sure how feasible that is. things like webpack and vite module federation come into play.

HTML modules is a complementary solution i think, it the “how does a framework MFE app/component come to use a registry” would still be an issue i think.

[Westbrook]

If it made sense to use the HTML module boundary as a registry scoping boundary then it would be something that a bundler would be required to manage, much like they are required to manage colliding const declarations when bundling files today. One path (being this is about future expansions) would be to raise an HTML-centric version of @sheet {} or JS module declarations to get exactly the same thing in a single HTML module.

[caridy]

Few notes:

1. Curious what @rniwa and others implementers think about the .run() idea. I think that will be problematic to implement, but also I don't think we have no precedent in the standard libraries for such pattern, even though it is common in the user space.
2. I agree with @rwaldron assessment, it must work for existing registrations. If we create a new API that only scope elements if you control the creation of instances, well, that will be very limiting.
3. @nolanlawson for us at salesforce, I do think that we can work with the registration mechanism because we do control it (via virtualization or via framework when registering pivots). That means something like customElements.define('x-foo', XFoo, {upgradeOnly: true}) from @sorvell might be good enough.

[nolanlawson]

Revisiting this after feedback from @ryansolid on how WCs add extra work for framework authors… This seems like a case where we may want to reconsider the current approach.

This part of the spec would effectively force frameworks to replace current usages of document.createElement and document.importNode, which is a big ask.

Meanwhile, Salesforce has (I believe) the largest deployment of a scoped custom element registry polyfill, and our approach is not based on this design at all but rather on something closer to registry.run(), which is (AFAICT) compatible with how frameworks currently work, since it just implies that document.createElement et al means something different when running in the registry context.

• Related: Idea: Alternative Ways to Control Registry Scoping #1058

[nolanlawson]

Some points that were raised about SCER in the Discord:

• We need a way to represent SCERs in SSR, and if SCERs are tied to Shadow Realms rather than Shadow Roots then that's not possible.
• In our own implementation, we could use our Shadow Realm global distortions to route to the "correct" Shadow Root's SCER. So it's not impossible to build on top of Shadow Realms as a layer on top of SCERs.

[justinfagnani]

We need a way to represent SCERs in SSR, and if SCERs are tied to Shadow Realms rather than Shadow Roots then that's not possible.

To clarify this a bit, it's not impossible, but SSR requires a way to tell some elements to not use the global registry. The current proposal is to use a flag on <template shadowrootmode> that tells the shadow root that it will receive a registry in the future and to not upgrade elements until it does.

This could probably still be done in coordination with shadow realms, but it still needs some kind of scoping within HTML that turns off global upgrades. Shadow roots are our natural scoping boundary here.

[annevk]

const template = document.createElement('template');
registry.run(() => template.innerHTML = htmlString);
return template.content.cloneNode(true);

I take this to mean that as the HTML parser runs, it would use registry to lookup the element names it comes across?

This seems similar to the idea in #1040 (comment) of decoupling shadow trees and registries.

An alternative to that which I would prefer would be that each registry can create built-in elements as well and that the element "remembers" the registry it was created with. This enables you to do:

x = customRegistry.createElement("div");
x.innerHTML = something; // uses customRegistry