Initial WebXR display support for Scene

[pupitetris]

This PR will be limited to making CesiumJS work with WebXR-enabled UAs at the Scene level, and it is considered as a Proof Of Concept due to its caveats, which may be worked on to make it acceptable for merging. The idea is to have a base to continue the discussion regarding WebXR support.

For a demo on how to use this proposal, check the cesium-vr demo. The most imprtant file is htdocs/index.js. This was initially a fork from a NICTA experiment, but most of the code originally found there is not in use by my demo.

Related issue: #3422

• CONTRIBUTORS.md: added an entry for myself.
• Signed the CLA Google Form.
• CHANGES.md: added an entry for Scene.webXRContext property.
• No changes found regarding the test report (26 tests failed in unrelated areas for unrelated assert failures).

TO-DO:

• Currently to achieve display on the Head Mounted Display (HMD) hardware, as a final stage in Scene:executeWebVRCommands, I am doing a low-level GL blit operation from the provided passState GL framebuffer to the GL framebuffer provided by WebXR.
  • This should be done at a Render level, using Context and something like Renderer/MultisampleFramebuffer.
  • The current method additionaly has the limitation that antialiasing cannot be activated in the WebGL context or the Blit will fail, and this is specially bad for HMDs due to the low resolution of the displays. Here the error that is produced:

  [WebGL-0x20036ff00]GL ERROR :GL_INVALID_OPERATION : glBlitFramebufferCHROMIUM: destination framebuffer is multisampled
  

• Also on executeWebVRCommands, make use of the transform and projection matrix that WebXR provides to calculate camera offsets, frustum, etc. Right now we are using the method that was already in place for WebVR, but the parameters provided by WebXR are supposed to be the ideal ones and are provided by the HMD. For example, the Meta Quest 2 is aware of the IPD the user is phisically setting inside the device, and the transform+projection parameters probably obay such adjustment. The WebXR documentation strongly recommends using these params to avoid discomfort for the users.
• Establish a mechanism to allow the containing app to know if pose information from the HMD was not available at a given frame (probably due to lost tracking). Maybe an event handler.

Next steps

After the issues of this POC are resolved, it would be interesting to work on Viewer, where device and API discovery, scene configuration and WebXR session handling would be taking place. Discussion would be necessary regarding HUD widget display inside the VR session, since it is a WebGL pure view and also a default controller scheme that adapts/degrades according to the different VR sets capabilities. Also camera handling, beyond the trivial direct mapping of HMD orientation to camera orientation.

[cesium-concierge]

Thanks for the pull request @pupitetris!

• ✔️ Signed CLA found.
• ❔ Unit tests were not updated.
  • Make sure you've updated tests to reflect your changes, added tests for any new code, and ran the code coverage tool.

Reviewers, don't forget to make sure that:

• Cesium Viewer works.
• Works in 2D/CV.

[ggetz]

Thanks for opening the PR @pupitetris, this is super exciting to see!

We have a few VR headsets around such as the Oculus and are excited to test this out.

Do you have a timeline in mind for addressing the TO DO items in this PR? We're happy to discuss the approach and provide a code review when you are ready.

[pupitetris]

Hi. Thank-you for following up on this.

Long story short:

I can work on the last two points next week. The first point would maybe take me some two weeks. Any help / orientation I could get would be great and probably save me quite some time and effort.

I would like to invite all interested parties to enter the discussion and see if we can tackle this as a team, since there are quite a few issues to solve before CesiumJS gets really good on VR. Lots of room for contributions here!

Now the discussion on the priority task:

Regarding the first TO-DO, I need some orientation since I am not versed on WebGL and I would have to do quite some effort investigating how to copy to a muli-sampled framebuffer, while it must be second-nature for people specialized on the field. I know the blit I am doing right now is not at the right layer: it's too low-level an operation to be carried out at the Scene. But I don't want to intervene lower in the guts of CesiumJS without some kind of approval to the approach I may try.

Not only is the current blit at the wrong level, it is also bad for performance, because I am adding a whole copy stage for every frame. That's why I am calling this a POC, because it works and shows it can be done, but it's not the right way of doing it. Cesium should be drawing directly on the framebuffer that WebXR tells it to.

So a new class will probably have to be implemented, since WebXR makes a fundamental change on the rendering process: CesiumJS is all about creating its own framebuffers with the characteristics it needs and using them to finally draw on the canvas. But WebXR gives you a framebuffer and tells you: "now paint on this". Cesium doesn't have a way of doing that, and that's why developers have been having fundamental problems integrating Cesium with WebXR. So, we need a way of saying, "Create a Cesium.Framebuffer, but use this predefined WebGL framebuffer for that instead of creating a brand-new WebGL framebuffer". Maybe it's just a case of adding a method to Render/Framebuffer.js, not a whole new subclass.

So all of this is quite low-level, and it would take me quite some time to do it by myself and do it right: maybe two weeks if I focus on it, because there would be quite a few things for me to learn about WebGL and Cesium Render internals, and I think someone from within Cesium would probably do it much more efficiently. Of course I would be learning quite a bit, and I have already learned quite a bit, but it all depends how much we may want this.

In my case, I am using an Oculus Quest 2. Have you got one of those at Cesium? In my case, I am finding CesiumJS to be a bit heavy for stand-alone VR for this hardware. You just can't expect to have a CesiumJS page open in the Quest and click on the VRButton and have it working right away... the performance is just not there. You have to reconfigure Cesium, including models, terrain and imagery to achieve a decent experience. Maybe there will be a need to allow the page developer to specify an alternate configuration in case a VR session is started, but we'll see. This continues to be the problem with VR platforms: many things have to be tailor-made, optimized for the particular user and hardware. It continues to be difficult for a developer to offer a page with a VR experience that is not aimed at anyone in particular. And we haven't even started to talk about controller schemes.

[pupitetris]

Hi @ggetz and team in general. I finally got time to work on this.

Camera frustum and translation are now derived from WebXR parameters. When testing on the Meta Quest 2, a more realistic experience is obtained mostly because the calculated FOV goes up from 60deg in the arbitrary calculation to 90 degrees.

Other notes:

Implemented abstractions/factorization that allow the same workflow if
either plain old WebVR is executed, or new WebXR is executed. This
code which is mostly processing and applying the WebXR pose to the
Cesium Camera should be implemented in a class of its own and avoid
Scene.js from getting too large.

Also, the code needs to be optimized, as the HMD and frustum
parameters are not expected to change too often, so recalculating
these transformations on every frame is overkill. We just need for
adjustments on the Camera frustum and IPD parameters to trigger
recalculations on the transformations (on an as-needed basis).

Next steps:

• First, a code reorg because we have quite a few functions now that should go into its own class / module, and others should be complements to existing CesiumJS classes. Scene.js is quite bloated already, so it's good to send code to independent modules where there is opportunity due to identifiable decoupling.
• Second, do a bit of optimization on the new routines to avoid unnecesary calculations for configurations that will not change too often.
• Move on to the bitblit PoC code that is in use now and make it better or more natural for the Cesium pipeline.
• Before moving on to topics related to the viewer (controls and UI overlays), check how much we can comply with the WebXR API (example: makeXrCompatible call)

[pupitetris]

OK, the refactoring is done, and optimizations are in place to avoid unnecesary Camera translation and frustum recalculations. I'll be busy for a couple of weeks so I don´t know if I'll be able to focus much on the bitblit TODO, but we are moving forward with this. Some feedback would be appreciated. Cheers!

[ggetz]

Awesome, thanks @pupitetris!

We're beginning to test this out on the Quest and potentially some other headsets. I understand we should follow directions in https://github.com/pupitetris/cesium-vr?

We'll also take a look shortly at the code in more detail, but the overall approach seems reasonable. I understand that other engines such as Mozilla's A-Frame use blit operation as well.

Before moving on to topics related to the viewer (controls and UI overlays), check how much we can comply with the WebXR API

I would recommend we keep the scope of this PR to a MVP in order to facilitate smaller reviews and a more focused discussion. For instance, UI overlays should probably be deferred to a later PR.

[pupitetris]

Awesome, thanks @pupitetris!

We're beginning to test this out on the Quest and potentially some other headsets. I understand we should follow directions in https://github.com/pupitetris/cesium-vr?

Yeah. I hope the instructions are good enough. I haven´t tried the new code with Cardboard setups: only with the WebXR HMD emulator and my Quest 2.

We'll also take a look shortly at the code in more detail, but the overall approach seems reasonable. I understand that other engines such as Mozilla's A-Frame use blit operation as well.

OK, I'll see if I can find the core part of A-Frame where the blit is performed, to see if we can have a WebGL operation that allows the use of a multisample framebuffer (right now antialiasing must be turned off).

Before moving on to topics related to the viewer (controls and UI overlays), check how much we can comply with the WebXR API

I would recommend we keep the scope of this PR to a MVP in order to facilitate smaller reviews and a more focused discussion. For instance, UI overlays should probably be deferred to a later PR.

Totally. This PR is only about the display part, at the Scene level. Afterwards, a good default input scheme should be discussed for implementation at the Viewer level, and see what can be done to have HUD controls which would be provided by the Viewer as well (probably two independent PRs).

Greetings,
Arturo

Awesome, thanks @pupitetris!

We're beginning to test this out on the Quest and potentially some other headsets. I understand we should follow directions in https://github.com/pupitetris/cesium-vr?

We'll also take a look shortly at the code in more detail, but the overall approach seems reasonable. I understand that other engines such as Mozilla's A-Frame use blit operation as well.

Before moving on to topics related to the viewer (controls and UI overlays), check how much we can comply with the WebXR API

I would recommend we keep the scope of this PR to a MVP in order to facilitate smaller reviews and a more focused discussion. For instance, UI overlays should probably be deferred to a later PR.

[geotom]

Hi @pupitetris and @ggetz , is development/integration stalled or will there be some WebXR support for Cesium?

[ggetz]

Hi @geotom,

We think WebXR support in CesiumJS would be awesome! At least on the Cesium side, we haven't been able prioritize getting this merged.

However, we would would be happen to reengage and discuss a plan of action to get something merged. Let us know your thoughts.

[geotom]

Hi ggetz, thanks for your fast response. I am very new to WebXR and need to first orientate myself, how far this PR goes, etc. What were the issues, not accepting the PR so far? What needs to be improved?