Pointer Events for mesh elements #254

trusktr · 2022-12-01T06:10:52Z

Closes #177

Currently only CSS-rendered elements emit events, based on their rectangle shapes.

Now we add click, pointermove, etc, for mesh elements that aren't rectangles (using raycaster and manually emitting the events ourselves to emulate the same event as regular built-in elements).

We will now be able to rely on shapes of meshes for all interactions!

trusktr · 2022-12-01T06:14:07Z

@keywizzle opened PR to track

trusktr · 2022-12-01T06:29:11Z

src/core/Scene.ts

+		if (intersections.length == 0) return
+
+		// Create custom event with detail of the exact XYZ position of the click
+		const newEvent = new CustomEvent('click', {


We can use MouseEvent to emulate the real event and its properties https://developer.mozilla.org/en-US/docs/Web/API/Element/click_event

trusktr · 2022-12-01T06:29:56Z

src/core/Scene.ts

+			detail: {position: intersections[0].point},
+		})
+		// Dispatch custom event on the LUME element saved on the intersection Three.js object
+		intersections[0].object.userData.lumeElement.dispatchEvent(newEvent)


This will need to check up the three.js tree because there can be three.js subnodes and we need to find under which lume element those subnodes are

Not super familiar with the three.js tree, but I assume that not every three.js Object3d will have a corresponding Lume element, so should it just check if lumeElement exists on userData, and if not, check the Object3d's parent until one is found?

Exactly, the element's .three property will be a Three.js node, and it might have Three.js children. This upward check could go in getLumeElementFromThree mentioned in the other comment.

trusktr · 2022-12-01T06:34:31Z

src/core/ImperativeBase.ts

@@ -182,6 +182,8 @@ export class ImperativeBase extends Settable(Transformable) {
 		// Helpful for debugging when looking in devtools.
 		// @prod-prune
 		o.name = `${this.tagName}${this.id ? '#' + this.id : ''} (webgl, ${o.type})`
+		// Save LUME element ref to Three.js object
+		o.userData.lumeElement = this


For this we can use a Map so that it is typesafe. We'll need to eventually once we compile to WebAssembly anyway. We also need to unmap the pair on disconnect. If we use a weak map then we don't need to do the manual unmapping but assembly script doesn't support weak map yet, although we can implement it

Yeah I think just a weak map would be good

Another idea is we can make a subclass mix in or factory function that returns an instance with the lumeElement property

Would the map be stored on the scene to keep track of each Object3d's corresponding Lume element? Then when it's time to dispatch the event, the scene gets the object's element and dispatches it that way?

It can just be in the top level of a module. We can export a function so that anyone can use it, for example:

import {getLumeElementFromThree} from 'lume' getLumeElementFromThree(someThreejsObject)

and what this might do is traverse up until it finds a node associated with a Lume element using that map.

-Dispatches custom event of type click on the first intersected object within a Scene

Co-authored-by: Kyle Bruce <keywizzle@users.noreply.github.com>

@keywizzle

…so that we can stop the native events from happening on the CSS content without breaking devtools support, allowing only a single replicated event to be dispatched fot the 3D content. Also added pointermove, and copied all the needed properties from the native events to our synthetic events. Thanks helping think through this @keywizzle! Co-authored-by: Kyle Bruce <keywizzle@users.noreply.github.com>

…en and cyan planes, and on things blocking those planes, and it works as expected.

trusktr · 2023-05-29T01:04:38Z

src/core/Scene.ts

+	}
+
+	return a
+}


move to a utility file

trusktr · 2023-05-29T01:05:06Z

src/core/SharedAPI.ts

@@ -206,6 +206,8 @@ export class SharedAPI extends DefaultBehaviors(ChildTracker(Settable(Transforma
 		// Helpful for debugging when looking in devtools.
 		// @prod-prune
 		o.name = `${this.tagName}${this.id ? '#' + this.id : ''} (webgl, ${o.type})`
+		// Save LUME element ref to Three.js object
+		o.userData.lumeElement = this


avoid using o.userData which has no types, we can use a Map or WeakMap instead.

trusktr · 2023-05-29T01:05:53Z

src/core/Scene.ts

@@ -1189,6 +1380,8 @@ export class Scene extends SharedAPI {
 		${super.css}

 		:host {
+			pointer-events: auto;


I don't remember why we need to add this (or I forgot to remove it when experimenting). Let's add a comment, or remove if not needed.

I saw this a while ago and don't know why. I know we messed with disabling pointer-events for CSS click events and maybe this was an artifact of that

trusktr · 2023-05-29T01:06:53Z

examples/test.html

@@ -19,161 +19,231 @@



Move this out of test.html into its own demo

trusktr · 2023-05-29T01:07:51Z

src/core/Scene.ts

@@ -685,6 +694,8 @@ export class Scene extends SharedAPI {
 	override disconnectedCallback() {
 		super.disconnectedCallback()
 		this.#stopParentSizeObservation()
+		this.removeEventListener('click', this.#handleClick)
+		this.removeEventListener('pointermove', this.#handlePointermove)


we don't need to explicitly remove the handlers, as event an element with its own event listeners will be GC'd fine.

keywizzle · 2023-05-29T05:24:02Z

InstancedMesh pointer events are a good addition. We know the raycaster has the ability to intersect with an InstancedMesh element already, and thankfully the intersection(s) that the raycaster returns has an instanceId property, which is the index of the instance in the InstancedMesh.

Question is, what's the most user-friendly way of defining specific event listeners of an instance? An easy method might be to add a method to InstancedMesh like addInstanceEventListener or something, which takes in the index of the instance + other event stuff passed when creating an event listener. Something like:

instancedMesh.addInstanceEventListener(0, 'pointer-move', () => {
    // Do whatever
}

Maybe long term, it would be better to have an MeshInstance class of some sort to be able to control individual instances that way, but that's another can of worms on its own (although more declarative and user-friendly IMO). Would look something like:

instancedMesh.instances[0].addEventListener('pointer-move', () => {
    // Do whatever
}

At that point, we might as well add instances to the DOM as a <lume-mesh-instance> and add things like position and rotation to the MeshInstance (but then how to deal with reactivity? oh goodness). I'd imagine some interpretation of the first option is better for now.

trusktr · 2023-09-03T03:18:46Z

@keywizzle

instancedMesh.addInstanceEventListener(0, 'pointer-move', () => {
    // Do whatever
}

I think initially we can start with this:

instancedMesh.addEventListener('pointermove', evt => {
    evt.instanceId
}

Where evt will be a subclass of the PointerEvent specifically for instanced meshes.

We could then provide simpler abstraction built on top of that.

That's a really good idea about the mesh instance elements. Will greatly simplify the user experience. Example:

<lume-instanced-mesh ...>
  <lume-mesh-instance number="100" position=".." color=".." ...></lume-mesh-instance>
</lume-instanced-mesh>

Where the child controls instance 100 specifically. And then:

const instance = document.querySelector('lume-mesh-instance')
instance.addEventListener('pointermove', (evt) => {
  instance.number === evt.instanceId // true

  // ...handle pointer events only for this instance...
})

Naming TBD

keywizzle · 2024-05-22T18:43:22Z

Some thoughts:

For animations, the finished event fired from Three's AnimationMixer when an animation clip is finished. Not sure what element specifically it should be fired on though.
Some of the other events that seem possible require a concept of having focus of a specific 3D element (copy/paste, keypress, drag/drop, etc.) which I could see being useful for the editor (or if anyone wanted to make their own 3D editor).

What other 3D details should be included the event? Currently it's only the world position, but three.js gives more details which we might want to include:

Instance index, in the case of the element being an InstancedMesh
Face/face index, although it would just be the three.js face so maybe we should look into defining a face within Lume
Distance from the ray origin and the intersection
Normal vector

trusktr · 2024-05-22T21:49:59Z

For animations, the finished event fired from Three's AnimationMixer when an animation clip is finished. Not sure what element specifically it should be fired on though.
Some of the other events that seem possible require a concept of having focus of a specific 3D element (copy/paste, keypress, drag/drop, etc.) which I could see being useful for the editor (or if anyone wanted to make their own 3D editor).

These are good thoughts, although we can save this for a separate change and focus this one on the pointer events. But yeah, we'd need to choose between emitting on the animation controller element, or on the model itself, or maybe even both. Anywho let's chat about that in a new issue.

What other 3D details should be included the event? Currently it's only the world position, but three.js gives more details which we might want to include:

Instance index, in the case of the element being an InstancedMesh

Face/face index, although it would just be the three.js face so maybe we should look into defining a face within Lume

Distance from the ray origin and the intersection

Normal vector

That's true. I think first let's get basics working, then we can add more official event properties as needed in new PRs. We can dump any other properties in something like event.threeIntersections and let people do anything custom with that, and then we can determine when/if it would be nice to officially support more event properties.

trusktr marked this pull request as draft December 1, 2022 06:11

trusktr commented Dec 1, 2022

View reviewed changes

trusktr force-pushed the scene-click-events branch 2 times, most recently from dc8cfa0 to 200cdbf Compare April 3, 2023 04:30

trusktr changed the title ~~Click events for Scene~~ mouse/pointer events for mesh elements Apr 20, 2023

trusktr force-pushed the scene-click-events branch from b5ca1cb to 84505cd Compare April 26, 2023 18:52

keywizzle and others added 6 commits May 14, 2023 21:24

Click events for Scene

0516f3d

-Dispatches custom event of type click on the first intersected object within a Scene

WIP experimenting to see how the events work, added TODO comments

458a720

Co-authored-by: Kyle Bruce <keywizzle@users.noreply.github.com>

Update test.html to further verify that it works: try hovering on gre…

07db0c6

…en and cyan planes, and on things blocking those planes, and it works as expected.

Update test.html to show that offsetX/Y still need some work.

ac59835

consolidate duplicate code between click and pointermove events

526ff82

trusktr force-pushed the scene-click-events branch from 84505cd to 526ff82 Compare May 15, 2023 04:30

add pointerup/down events

8e82245

trusktr commented May 29, 2023

View reviewed changes

src/core/Scene.ts

}

return a

}

Copy link

Member Author

trusktr May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to a utility file

trusktr commented May 29, 2023

View reviewed changes

examples/test.html

@@ -19,161 +19,231 @@

Copy link

Member Author

trusktr May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this out of test.html into its own demo

trusktr commented May 29, 2023

View reviewed changes

trusktr linked an issue Oct 16, 2023 that may be closed by this pull request

Implement interaction events as standard events like pointer events (pointerdown, pointermove, click, etc) #177

Open

trusktr changed the title ~~mouse/pointer events for mesh elements~~ Pointer Events for mesh elements Dec 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pointer Events for mesh elements #254

Pointer Events for mesh elements #254

trusktr commented Dec 1, 2022 •

edited

trusktr commented Dec 1, 2022

trusktr Dec 1, 2022

trusktr Dec 1, 2022

keywizzle May 22, 2024

trusktr May 22, 2024

trusktr Dec 1, 2022 •

edited

trusktr Dec 1, 2022

trusktr Dec 1, 2022

keywizzle May 22, 2024

trusktr May 22, 2024

trusktr May 29, 2023

trusktr May 29, 2023

trusktr May 29, 2023

keywizzle May 29, 2023

trusktr May 29, 2023

trusktr May 29, 2023 •

edited

keywizzle commented May 29, 2023

trusktr commented Sep 3, 2023 •

edited

keywizzle commented May 22, 2024

trusktr commented May 22, 2024

Pointer Events for mesh elements #254

Are you sure you want to change the base?

Pointer Events for mesh elements #254

Conversation

trusktr commented Dec 1, 2022 • edited

trusktr commented Dec 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trusktr Dec 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trusktr May 29, 2023 • edited

Choose a reason for hiding this comment

keywizzle commented May 29, 2023

trusktr commented Sep 3, 2023 • edited

keywizzle commented May 22, 2024

trusktr commented May 22, 2024

trusktr commented Dec 1, 2022 •

edited

trusktr Dec 1, 2022 •

edited

trusktr May 29, 2023 •

edited

trusktr commented Sep 3, 2023 •

edited