Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full mouse control #1493

Open
zeroxoneafour opened this issue Nov 29, 2020 · 8 comments · May be fixed by #1520
Open

Full mouse control #1493

zeroxoneafour opened this issue Nov 29, 2020 · 8 comments · May be fixed by #1520
Labels

Comments

@zeroxoneafour
Copy link

Is your feature request related to a problem? Please describe.
When I use noVNC for remote gaming, games that require mouse control (such as first-person games like Minecraft) cannot interpret the mouse input correctly, most likely because the application in question is not given full mouse control.

Describe the solution you'd like
It would be great if noVNC was able to fully immerse the mouse in the VNC using mouse control. First-person games online are able to "take control of your pointer" and it would be nice if there was a setting to allow that in the options.

Describe alternatives you've considered
I tried reconfiguring Minecraft's settings, but nothing works.

Additional context
When mouse control is fully taken, it would also probably be a good idea to make a keybind (such as esc) to give mouse control back to the client computer.

@CendioOssman
Copy link
Member

We're only a client so there is limits to what extra features we can add. Do you know of any VNC client/server combination where this already works?

@zeroxoneafour
Copy link
Author

RealVNC seems to support this, or something like this, as Minecraft Pi Edition seems to work over their VNC (see here). I know that RealVNC is proprietary, though, and I think it requires their client as well.

@CendioOssman
Copy link
Member

Unfortunately the fact that they are proprietary does mean it doesn't help us that they have that functionality. We need to know how the extension works, either through documentation or source code.

@zeroxoneafour
Copy link
Author

x11vnc has a option called -pipeinput that can throw the mouse input directly into uinput. It seems to work, but I am met by the limitation of the mouse exiting the area of effect of the vnc client. You might be able to take advantage of the feature for exclusively x11vnc as a server.

@lhchavez
Copy link
Contributor

lhchavez commented Feb 7, 2021

This could in theory be achieved through the Pointer Lock API (demo). Are patches welcome for this? If so, I can maybe cook something up.

lhchavez added a commit to lhchavez/noVNC that referenced this issue Feb 8, 2021
This change adds the following:

a) A new button on the UI to enter full pointer lock mode, which invokes
   the Pointer Lock API[1] on the canvas, which hides the cursor and
   makes mouse events provide relative motion from the previous event
   (through `movementX` and `movementY`). These can be added to the
   previously-known mouse position to convert it back to an absolute
   position.
b) Adds support for the VMware Cursor Position pseudo-encoding[2], which
   servers can use when they make cursor position changes themselves.
   This is done by some APIs like SDL, when they detect that the client
   does not support relative mouse movement[3] and then "warp"[4] the
   cursor to the center of the window, to calculate the relative mouse
   motion themselves.
c) When the canvas is in pointer lock mode and the cursor is not being
   locally displayed, it updates the cursor position with the
   information that the server sends, since the actual position of the
   cursor does not matter locally anymore, since it's not visible.
d) Adds some tests for the above.

You can try this out end-to-end with TigerVNC with
TigerVNC/tigervnc#1198 applied!

Fixes: novnc#1493 under some circumstances (at least all SDL games would now
work).

1: https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Lock_API
2: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#vmware-cursor-position-pseudo-encoding
3: https://hg.libsdl.org/SDL/file/28e3b60e2131/src/events/SDL_mouse.c#l804
4: https://tronche.com/gui/x/xlib/input/XWarpPointer.html
lhchavez added a commit to lhchavez/noVNC that referenced this issue Feb 8, 2021
This change adds the following:

a) A new button on the UI to enter full pointer lock mode, which invokes
   the Pointer Lock API[1] on the canvas, which hides the cursor and
   makes mouse events provide relative motion from the previous event
   (through `movementX` and `movementY`). These can be added to the
   previously-known mouse position to convert it back to an absolute
   position.
b) Adds support for the VMware Cursor Position pseudo-encoding[2], which
   servers can use when they make cursor position changes themselves.
   This is done by some APIs like SDL, when they detect that the client
   does not support relative mouse movement[3] and then "warp"[4] the
   cursor to the center of the window, to calculate the relative mouse
   motion themselves.
c) When the canvas is in pointer lock mode and the cursor is not being
   locally displayed, it updates the cursor position with the
   information that the server sends, since the actual position of the
   cursor does not matter locally anymore, since it's not visible.
d) Adds some tests for the above.

You can try this out end-to-end with TigerVNC with
TigerVNC/tigervnc#1198 applied!

Fixes: novnc#1493 under some circumstances (at least all SDL games would now
work).

1: https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Lock_API
2: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#vmware-cursor-position-pseudo-encoding
3: https://hg.libsdl.org/SDL/file/28e3b60e2131/src/events/SDL_mouse.c#l804
4: https://tronche.com/gui/x/xlib/input/XWarpPointer.html
lhchavez added a commit to lhchavez/noVNC that referenced this issue Feb 8, 2021
This change adds the following:

a) A new button on the UI to enter full pointer lock mode, which invokes
   the Pointer Lock API[1] on the canvas, which hides the cursor and
   makes mouse events provide relative motion from the previous event
   (through `movementX` and `movementY`). These can be added to the
   previously-known mouse position to convert it back to an absolute
   position.
b) Adds support for the VMware Cursor Position pseudo-encoding[2], which
   servers can use when they make cursor position changes themselves.
   This is done by some APIs like SDL, when they detect that the client
   does not support relative mouse movement[3] and then "warp"[4] the
   cursor to the center of the window, to calculate the relative mouse
   motion themselves.
c) When the canvas is in pointer lock mode and the cursor is not being
   locally displayed, it updates the cursor position with the
   information that the server sends, since the actual position of the
   cursor does not matter locally anymore, since it's not visible.
d) Adds some tests for the above.

You can try this out end-to-end with TigerVNC with
TigerVNC/tigervnc#1198 applied!

Fixes: novnc#1493 under some circumstances (at least all SDL games would now
work).

1: https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Lock_API
2: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#vmware-cursor-position-pseudo-encoding
3: https://hg.libsdl.org/SDL/file/28e3b60e2131/src/events/SDL_mouse.c#l804
4: https://tronche.com/gui/x/xlib/input/XWarpPointer.html
@samhed samhed added the feature label Feb 9, 2021
@mteam88
Copy link

mteam88 commented May 3, 2022

Would really love this functionality! Any updates @lhchavez @samhed @CendioOssman @zeroxoneafour @jswanner ?

@Frontesque
Copy link

I'm also looking for a fix for this. I use noVNC as a remote desktop for while im out and it causes some issues with programs that require relative mouse movement.

aseering pushed a commit to aseering/noVNC that referenced this issue Feb 20, 2023
This change adds the following:

a) A new button on the UI to enter full pointer lock mode, which invokes
   the Pointer Lock API[1] on the canvas, which hides the cursor and
   makes mouse events provide relative motion from the previous event
   (through `movementX` and `movementY`). These can be added to the
   previously-known mouse position to convert it back to an absolute
   position.
b) Adds support for the VMware Cursor Position pseudo-encoding[2], which
   servers can use when they make cursor position changes themselves.
   This is done by some APIs like SDL, when they detect that the client
   does not support relative mouse movement[3] and then "warp"[4] the
   cursor to the center of the window, to calculate the relative mouse
   motion themselves.
c) When the canvas is in pointer lock mode and the cursor is not being
   locally displayed, it updates the cursor position with the
   information that the server sends, since the actual position of the
   cursor does not matter locally anymore, since it's not visible.
d) Adds some tests for the above.

You can try this out end-to-end with TigerVNC with
TigerVNC/tigervnc#1198 applied!

Fixes: novnc#1493 under some circumstances (at least all SDL games would now
work).

1: https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Lock_API
2: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#vmware-cursor-position-pseudo-encoding
3: https://hg.libsdl.org/SDL/file/28e3b60e2131/src/events/SDL_mouse.c#l804
4: https://tronche.com/gui/x/xlib/input/XWarpPointer.html
@FinnBaltazar1111
Copy link

I find that turning off Raw Input in Mouse Settings makes it a bit easier to play MineCraft (and of course, use a mouse, not a track pad).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants