Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using Enter AR Button Causes Renderering Issues With Markers Resulting In Offset Positions, And AR Markers Move When Headset/Camera Changes Position #298

Open
loganknecht opened this issue Oct 19, 2023 · 0 comments
Open

Using Enter AR Button Causes Renderering Issues With Markers Resulting In Offset Positions, And AR Markers Move When Headset/Camera Changes Position #298

loganknecht opened this issue Oct 19, 2023 · 0 comments

Comments

@loganknecht
Copy link

Hello!

Really love your library! Thank you so much for making it

Environment

  • Magic Leap 2
  • Helio Browser
  • Remote Debugging via Chrome

Expected Behaviour

I expect AR Markers to render correctly where the AR Marker is.

The Problem

I am using a Magic Leap 2 headset to test WebXR development.

I am encountering an issue where AR Markers are being incorrectly rendered when I am in AR Mode after selecting the Enter AR Mode button. This is when the web browser is expanded into a full screen mode.

The issue is that when I am rendering my scene outside of AR mode, the markers are correctly translating and placed correctly in world space. It's perfect. There are no complaints.

However when I enter AR Mode my markers are incorrectly placed and offset in the opposite direction of my camera movement. I have no idea why this is.

Demo Video

I have created a video here to demonstrate the issue

bad_performance_small.mp4

The Code

package.json

{
  "name": "@artcom/react-three-arjs-example",
  "version": "0.1.0",
  "license": "MIT",
  "scripts": {
    "watch": "webpack serve --hot --https --mode=development",
    "build": "webpack --mode=production",
    "start": "http-server -S -C cert.pem dist/"
  },
  "engines": {
    "node": ">= 16.15"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/artcom/react-three-arjs.git"
  },
  "devDependencies": {
    "@babel/cli": "^7.22.6",
    "@babel/core": "^7.22.8",
    "@babel/preset-env": "^7.22.7",
    "@babel/preset-react": "^7.22.5",
    "@pmmmwh/react-refresh-webpack-plugin": "^0.5.10",
    "babel-loader": "^9.1.3",
    "copy-webpack-plugin": "^11.0.0",
    "eslint": "^8.44.0",
    "eslint-config-prettier": "^8.8.0",
    "eslint-plugin-compat": "^4.1.4",
    "eslint-plugin-import": "^2.27.5",
    "eslint-plugin-prettier": "^4.2.1",
    "eslint-plugin-react": "^7.32.2",
    "eslint-plugin-react-hooks": "^4.6.0",
    "html-webpack-plugin": "^5.5.3",
    "http-server": "^14.1.1",
    "react-refresh": "^0.14.0",
    "webpack": "^5.88.1",
    "webpack-cli": "^5.1.4",
    "webpack-dev-server": "^4.15.1"
  },
  "dependencies": {
    "@ar-js-org/ar.js": "^3.4.5",
    "@artcom/react-three-arjs": "0.5.6",
    "@react-three/drei": "^9.88.2",
    "@react-three/fiber": "^8.14.5",
    "@react-three/xr": "^5.7.1",
    "buffer": "^6.0.3",
    "core-js": "^3.31.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "three": "^0.154.0"
  },
  "browserslist": [
    "defaults"
  ]
}

index.html

<html>
    <head>
        <title>@artcom/react-three-arjs example</title>
        <meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0" />

        <style>
            body {
            /* width: 100%;*/
            /* height: 100%;*/
            /* margin: 0px;*/
            /* padding: 0px;*/
            /* overflow: hidden;*/
            }
        </style>
    </head>

    <body>
        <div id="root"></div>
    </body>
</html>

index.js

// Third-Party
import { ARCanvas, ARMarker } from "@artcom/react-three-arjs";
import {
    Box,
    Environment,
    OrbitControls,
    // PerspectiveCamera,
    Stage
} from "@react-three/drei";
import {
    ARButton,
    // Controllers,
    Hands,
    // Interactive,
    XR,
    // useHitTest,
    useXR
} from "@react-three/xr";
import React from "react";
import { createRoot } from "react-dom/client";
// Custom
// N/A

function ConstellationStage(props) {
    // -------------------------------------------------------------------------
    // State
    // -------------------------------------------------------------------------
    const current_xr_state = useXR();

    // -------------------------------------------------------------------------
    // Rendering
    // -------------------------------------------------------------------------
    const placeholder_view = (
        <Stage
            // for formatting
            adjustCamera={true}
            intensity={1}
            preset="rembrandt"
            shadows="contact"
        >
            <Environment preset="forest" background={true} />
            <OrbitControls />
            <ambientLight />
            {/*{current_xr_state.isPresenting === false ? null : null}*/}
        </Stage>
    );
    const ar_view = (
        <>
            <Hands />
            <ambientLight />
            {/*<pointLight position={[10, 10, 0]} intensity={10.0} />*/}
            <ARMarker
                params={{ smooth: true }}
                type={"pattern"}
                patternUrl={"data/patt.hiro"}
                onMarkerFound={() => {
                    console.log("Marker Found");
                }}
            >
                <Box
                    args={[1, 1, 1]}
                    material-color="hotpink"
                    // position={[0, 1.5, -1]}
                />
            </ARMarker>
            <Box args={[0.1, 0.1, 0.1]} material-color="red" position={[0, 1.5, -1]} />
            <Box args={[0.1, 0.1, 0.1]} material-color="blue" position={[0, 1.6, -1.2]} />
            <Box args={[0.1, 0.1, 0.1]} material-color="green" position={[0, 1.7, -1.3]} />
        </>
    );
    const final_render_element = current_xr_state.isPresenting === true ? ar_view : placeholder_view;

    // return final_render_element;
    return ar_view;
}

function ConstellationScene(props) {
    const final_render_element = (
        <div
        // style={{ height: "100vh", width: "100vw" }}
        >
            <ARButton />
            <ARCanvas
                // gl={{ antialias: false, powerPreference: "default", physicallyCorrectLights: true }}
                onCameraStreamReady={() => console.log("Camera stream ready")}
                onCameraStreamError={() => console.error("Camera stream error")}
                onCreated={({ gl }) => {
                    gl.setSize(window.innerWidth, window.innerHeight);
                }}
            >
                <XR>
                    <ConstellationStage />
                </XR>
            </ARCanvas>
        </div>
    );

    return final_render_element;
}

createRoot(document.getElementById("root")).render(<ConstellationScene />);

Code Sandbox

It is available here to reproduce
https://codesandbox.io/s/example-react-ar-js-vx2dj4

Speculation

I have reason to believe that there is something going on in AR Mode that is causing the rendering to get messed up with the camera itself.

I don't know why, or how.

Are there settings that I need to be mindful of when entering AR Mode?

Work Around Request

Is there a way to emulate AR Mode without using your library? I have a sneaking suspicion if I were able to do AR Mode in a different approach I might see the correct behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@loganknecht