Add client side gizmo

[Irev-Dev]

There's a server-side gizmo, but there's problems in that because it's just some magic pixel's it makes it hard for us designing on the frontend to work around it, if we want to put other UI elements in the bottom-right-hand of the screen, we have to be carful to to cover it instead of using normal HTML layout techniques. This will also be something other clients would be interested in doing too.

I think the best approach is to make a small 300x300ish size canvas for the new gizmo to sit inside, and we just synce it with the rest of the scene's camera, this might mean integrating something like existing viewHelper trick to work with because it's not driven off orbit or similar controls, but instead camera details that come from the engine. But that might also be wrong since we already have utils set up for syncing between the engine camera and a three.js camera, but it doesn't happen mid-mouse-drag, so maybe if we make this update more frequent this would work.

Reading back over the above it's clear there's some quirks to our app that come from having a remote engine, I think it's best I outline the steps I think I would take to implement this. While it's most likely I'll get a few details wrong, it will give us a starting point and we can discuss hurdles as we hit them.

• add a small canvas to the bottom rhs of the screen with a tiny threejs scene (can just have a cube for all I care at this point the main thing I want to derisk first is syncing the cameras, we can worry about how it looks later. but ofc if you want to add a proper looking gizmo now that's fine too)
• Look to get the rotation of the gizmo to sync up with the main camera, Probably a good place to start is by adding something within onCameraChange to update the gizmo.
• Doing the above I know will mean the gizmo will not animate smoothly instead will snap into place once the user stops dragging, but once we're at that point we can look into getting more frequent information from the websocket connection mid drag.

[Irev-Dev]

Tagging @max-mrgrsk as I think that might mean I'll be able to assign him?!

Also Max let me know what you think of this approach.

For context max had also put together this proof of concept using r3f and drei, maybe we can use drei, but I'm not sure I want to add another dependency like r3f just for the gizmo, since we're currently using three.js directly, but I could be persuaded.

[Irev-Dev]

Tagging @max-mrgrsk as I think that might mean I'll be able to assign him?!

Nope that didn't work, I think if you comment on the issue it might let me.

[Irev-Dev]

@franknoirot I can see the new bottom right designs are used in a few places in figma, i.e. https://www.figma.com/file/5UUPOawPLEs3D0Gptf42vy/Designs?type=design&node-id=1%3A2&mode=design&t=V4U69BU5ETKwMNlk-1

But is there anything specific I should be linking to for the gizmo itself?

[max-mrgrsk]

@Irev-Dev I've created a small 200x200 canvas that hosts a tiny three.js scene with a cube dummy as a placeholder gizmo. The cube's orientation is currently synchronized with the client-side camera using quaternion values from SceneInfra. The rotation is working correctly and snaps as expected.

Next, I will either integrate the capability to sync the gizmo's rotation based on server-side camera updates

or adjust cameraControls to update the client-camera every 0,1sec while mouse button is pressed.

I like the idea of adjusting the cameraControls because it is a dedicated class for the updates.
It is instantiated in SceneInfra , that provides nice infos about the client-side camera, that we can just use as they are for the gizmo.

engine > cameraControls > SceneInfra > client camera info > gizmo

Afterwards I will integrate lerp to smoothen everything

Plan B : in case if server will not like frequent requests and we will not reach desired smoothness, I could make the client-side camera to move from mouse drag. And use client-side camera mid-drag position for our gizmo. Both engine and client cameras would move simultniously and independently from each other(during drag). Kind of approximation. onMouseUp client-camera would snap in place, same as now.

[franknoirot]

@Irev-Dev potentially the component in Figma to see its different UI states, but I haven't done a good pass of mocking up the different interactions I want the gizmo to have yet so that only really is helpful for gettingcontext about the other UI elements around the gizmo at the moment. I'll make a point to get those made up soon.

Great start @max-mrgrsk thank you so much for contributing! I'll let @Irev-Dev comment on the camera sync approach.

[max-mrgrsk]

I've made some adjustments to the CameraControls. Now, during camera interactions while onMouseDown, we're fetching default_camera_get_settings updates from the server at a 0.5-second interval :

0.1-second interval :

next step - get smooth animation with lerp

[max-mrgrsk]

I've been experimenting with the lerp function to mitigate the step effects. This issue becomes particularly noticeable with infrequent updates, causing the gizmo to appear to lag or "jump" to new positions.

To address this, I adjusted the frequency of updates from the server and experimented with different lerp values to smooth the transitions. Here are my findings:

0.5-second updates: At this interval, the required lerp value to smooth out transitions is quite high, resulting in the gizmo moving sluggishly:

0.2-second updates: Decreasing the update interval improves responsiveness slightly, reducing the necessary lerp value and making the gizmo's motion a bit more fluid. However, there is still a noticeable delay in how the gizmo responds to camera movements:

0.1-second updates: At this rate, the lerp value is significantly lower, and the sluggishness is almost unnoticeable. The gizmo moves more in sync with the server camera, providing a smoother visual experience.

These experiments suggest that shorter update intervals can greatly enhance the visual performance of the gizmo.

I would appreciate any feedback on this approach or any other suggestions that might improve the handling of these issues.

[Irev-Dev]

MAX!

This is great, really happy you've been able to run with this. and apologies I've been slow to respond. We've got a catch up in an hour or so, so we'll talk about it there,

but as far as feedback goes on your approach, my thoughts are that I would probably want to see your code at this point, that would help me reason about it.

I think your experiments with update frequencies and lerping is perfect, is what I would have done to try and strike a balance.

One other thing to note is that as far as commands that are sent to the engine we have two types reliable and unreliable. The reliable ones use TCP to ensure the packets make it through and are sent in order, but there's overhead which makes latency worse than our unreliable commands.

In the case of the rotation here the break down is

• start drag sent reliably
• mouse move during drag sent unreliably for minimal latency
• end drag sent reliably

I bring this up because at one point the engine folk had talked about sending back the camera details for all of the unreliable commands, but I can't remember where that got up to, I might double check on this one, because if we're able to get that data, it might mean we don't need to poll the engine, and potentially it will be lower latency as well.

[Irev-Dev]

Oh and fwiw, if there's a bit of lag in the gizmo, I don't think that's the end of the world, idk what do you think @franknoirot ?

[Irev-Dev]

Okay that data is definitely not coming through, I'll raise it with the engine team because I think that's probably the best.

[Irev-Dev]

Okay @max-mrgrsk this diff here should be useful for getting updated camera data without having to poll
https://github.com/KittyCAD/modeling-app/pull/2332/files

If you want to integrate this into your work you can just copy the same code over, or if you want to use git and the git-cli git cherry-pick cc8cc78bb2d4c200a1a5ccc531d23f53ed22e97f will pull this commit onto your current branch (but don't feel the need if you're more comfortable copying it over.

If you do this the gizmo will be jittery in the short term, until I get something fixed in the engine.
I explain it a bit more in this vid.

Screenshare.-.2024-05-09.4_10_34.PM.mp4

[max-mrgrsk]

Okay @max-mrgrsk this diff here should be useful for getting updated camera data without having to poll https://github.com/KittyCAD/modeling-app/pull/2332/files

Thanks for the diff! I've implemented the changes and everything is working super smoothly now—really nice improvement!

[max-mrgrsk]

Hi @franknoirot, I've implemented your designs for the gizmo. What do you think?

Hi @Irev-Dev, Please check out the diff here: 88d4fb2

[franknoirot]

This looks awesome @max-mrgrsk! I think your canvas size is 100px and in my mockups it's 80px, but that's tiny tweak stuff.

I don't think you need to draw the blue ring in THREEjs, but could instead just give the canvas a round shape and a border, which could be accomplished with a wrapping rounded div element around the canvas:

<div className="w-20 h-20 grid place-content-center rounded-full overflow-hidden border border-solid border-primary/50">
  <canvas className="w-full h-full" {/* rest of canvas element's props... */} />
</div>

That way that blue ring color updates along with all the others when the user's theme color is changed in their settings. We use TailwindCSS's standard sizing setup so w-20 comes out to 80px