Develop your first WebAR app using WebXR and Three.js

WebXR brings together the power of web and the integration of augmented realities, under one umbrella thereby making it easier to create immersive 3D art, interactive environments, VR tools, and much more. WebXR Device API supports developers to create immersive AR VR applications which -can run on a browser.

In this section, we will learn the basics of WebXR. In addition, we will create a web AR app using WebXR and three.js.

Getting Started

WebXR Device API

WebXR Device API provides an interface to access input and output capabilities associated with Virtual Reality (VR) and Augmented Reality (AR) devices such as the pose information of a headset. It enables web developers to create VR or AR applications without having to deal directly with hardware but to experience it on the web. The API enables users to view the AR VR content directly on a browser without having to install extra plugins or software. Developers need to write the code only once and can run it on any device irrespective of brands (Oculus, Vive, Windows Mixed VR, etc.)

Fundamentals of WebXR - Web APIs | MDN

How WebXR works

• Determine XR support in the device using navigator.xr.isSessionSupported
• Wait for the response to start the XR engine.
• Request an XRSession with navigator.xr.requestSession.
• Use the session to render the frame.
• Continue running of XR engine till the session is shut down by the application or user.

Three.js

Three.js is a cross-browser JavaScript library used to create and visualize animated 3D models using WebGL on a web browser. It grants the designing of GPU accelerated 3D animations as part of a website without relying on exclusive browser plugins.

Three.js - JavaScript 3D library

Let’s have a look into some of the key terms and concepts you should be familiar with to better understand what is going on in a Three.js project.

Scene

A scene represents the main 3D environment of the application where you place all your models, lights, and cameras and arrange them in order to provide an immersive experience.

Renderer

The renderer is responsible for drawing our scene onto the screen. We can perform updates on the scene, cameras, lights, and models. To visualize the changes made, the render method will be called. For changes made to the animation, we will create a render-loop, which is a function responsible for repeatedly drawing our scene onto the screen within a given interval.

Camera

Camera represents the user’s view as the eyes are to the human. You can choose the type of camera based on the kind of experience you want to create. In this demo, we are going to use the Perspective Camera, which is designed to replicate how the human eye sees things.

Light

Light is an essential element to view the components in the scene. The scene will appear empty without a source of light. Even though we will be working in the context of augmented reality, lighting is still important to provide naturalism to the scene so as to make the models look their best.

Models & Meshes

Three.js provides the option to load external 3D models, as well as create 3D models from scratch. However, it’s possible to make your own 3D objects in Three.js by specifying the geometry of a shape and applying a material to it thereby creating a Mesh of these two parts. We can import any 3D model created using 3D modeling software such as Blender, Maya into Three.js.

Develop your first Web AR app

So far we have looked into the core building blocks of a web AR application.

Now, it’s time to try it.

Requirements

• Node.js [download]

For building a web AR app, we require only node.js software installed in our machine. It would be easier to follow this tutorial if you have some basics on typescript since the tutorial is designed with typescript. The user also requires an android device with the latest google chrome to run the app.

Experience the Web AR Boilerplate

We are aware of the challenges of setting up a project from scratch. In this tutorial, we will be using the boilerplate code repository provided and hence you need not worry about the project configuration before setting up the project.

Let’s start our web AR development by cloning/downloading the Web AR Boilerplate repository.

GitHub - codemaker2015/web-ar-boilerplate: Web AR boilerplate using WebXR and Three.js

This repository contains two branches — master and production. In this tutorial, we will be focusing more on the master branch code. By the end of the tutorial, you will get the basic idea behind web AR development. If you wish to try out the master branch, clone/download the repository and run it using the following commands,

npm install
npm run start:live

npm install will install all the dependencies of your project.

npm run start:live will run your project and generate a public URL using localtunnel.

To view your demo, load the generated public URL (Here it is https://neat-baboon-76.loca.lt) on your mobile device by using google chrome.

Create your first web AR app

Step 1

Hope you have cloned/downloaded the master branch. We can start the development with that. Open the code in any code editor (preferred editor: VSCode).

Step 2

Now, we will install the dependencies. We can do that by opening a terminal/command prompt and typing the following command.

npm install

So, we have installed all required dependencies — three.js, webpack, localtunnel, typescript.

Let’s start coding!

Step 3

Open index.ts file located under the src folder which is the main entry point to our app. We can find a method named initializeXRApp that is responsible for initializing the app in the index.ts file. Replace the TODO comment from the body and paste the following code snippet into it.

Understand the code,

• After determining the device dimensions (pixel ratio, height and width), create a renderer object using the WebGLRenderer() constructor with the device dimensions.
• Enable xr functionality on the renderer object and add it to the DOM.
• Setup the AR experience by calling the built-in helper function ARButton.createButton() Now, three.js will give a page with a nice little button on the bottom part to enter into the AR experience.
• Pass the renderer that is imported from the scene.ts file into the createScene().

Step 4

Add the following code into the start() method.

Understand the code,

• browserHasImmersiveArCompatibility method is used to call a built-in function on the navigator.xr object to check for WebXR-support.
• showUnsupportedBrowserMessage method renders some HTML to the DOM containing a message to the user if their browser is not supported.

If you run the app by npm run start:live command, you will get the public URL which when opened on the device will show a page with the START AR button on the bottom. Which when pressed will create a camera feed and will be rendered by the WebXR functionality in three.js.

Step 5: Create a scene

Open scene.ts file located under the src folder. Where you will find a createScene() method. Replace the TODO comment from the body and paste the following code snippet into it.

Understand the code,

• Create a scene object using new Scene() which will initialize a new empty scene object. Create a camera object using new PerspectiveCamera() which takes in four parameters — field of view, aspect ratio, near plane, and far plane. Here we will set the field of view as 70, aspect ratio as window.innerWidth / window.innerHeight , near and far plane values as 0.2 meter and 20 meter.
• Finally, we will setup the main render loop for the app, and then pass it into the setAnimationLoop method on the renderer. This will ensure that our renderLoop method is called on every frame, which in turn renders our scene.

Step 6: Mesh rendering

Now, We will be rendering a simple cube to the scene. Update the createScene() method as follows,

Understand the code,

• BoxBufferGeometry is a pre-defined method which can be used to create a cube like mesh whose geometry can be changed by modifying width, height, and depth parameters.
• MeshBasicMaterial class is used to provide material for the 3d models in order for them to be drawn to the scene using a constructor. In this tutorial, we will be making use of the color property with the value 0x00ff00which gives the color green.
• Finally, we can create the box mesh by passing the geometry and material to the Mesh constructor. This will return a nice green color cube that we can render onto our AR view. For that, add it to the scene using the add() method on the scene object.
• We will also learn to create an animation of spinning the cube by manipulating the rotation of the cube within the render method by continuously adding to the x and y values of the object rotation.

Step 7: Run the app

Now, we can run the app by running the following command,

npm run start:live

In the command prompt/terminal, you will get a public URL. Open the same in Google chrome to see the demo.

Rendering a custom 3D model

In the previous section, we learned the environment setup of Three.js, installing dependencies and creating an animation of a green spinning cube. In this section, we will learn some advanced functions such as hit testing and rendering a custom 3D model on the browser.

Step 1: Hit test

Open the hitTest.ts file located under the src/utils folder where you will find handleXRHitTest method. Replace the TODO comment from the body and paste the following code snippet into it.

Understand the code,

• Get the current reference space and session from the xrobject on the renderer, and then declare a variable to hold the value of our hit pose.
• Reference space is like a local coordinate system of the AR space which helps the device to know about its whereabouts and to translate between the coordinate space of the three.js scene and the real world.
• The session variable is simply used to keep a handy reference of the current running XR session along with the pose of the device.
• In the next step, check if a session exists and a hit-test source has been requested. If it is an initial request for a hit-test source then we request a reference space of type viewer from the XR session.
• In the next step, verify if the hit test source has been set or not. If we have a source then get the hit test results from the current frame and pass it as a parameter to the initialization function.
• If the array returned by getHitTestResults is not null then we can get the pose of the device by calling getPose(). The position and orientation of the device is returned through onHitTestResultReady as a transformation matrix. Using these values we can position objects in the scene. We can use onHitTestResultEmpty callback to hide the plane marker.
• As the next step, we will be using the hitTestHelper method inside the render loop.

Step 2: Call the handleXRHitTest() method

Open the scene.ts file again and create a simple circular marker mesh, call the handleXRHitTest method and place the marker over the plane. So, update the scene.ts file as follows (replace the old content with this),

Understand the code,

• The handleXRHitTest method has two callback parameters, one which is called when a hit test result is ready, and the other which is called when the result is empty. We can show or hide the circular marker on any surface using the handleXRHitTest value.
• If the hit test is successful, we set the visible property on the marker mesh to true else we set the visible property to false which in turn will hide the marker to indicate the user that no surfaces were found.

Step 3: Add Lighting

You can add AmbientLight to the scene by using the following code,

Ambient light provides a distributed illumination to the scene. In this example, we pass in the color as white and intensity as 1.

Step 4: Place the model

Three.js is capable of loading various model formats such as OBJ, 3DS, and gLTF. In this example, we will go with the gLTF format.

We have already imported the 3d model and the gLTF loader in the scene.ts file. Add the following lines inside the createScene method to load the 3d model into the scene.

We create an instance of the GLTFLoader class to load the model. Provide model URL as the first parameter of the load function. The second parameter is the onLoad callback which will be called once the model is ready.

Step 5: Adding a WebXR Controller

To listen to any events such as a tap on the screen or a swipe, select the event and attach a listener to the WebXR Controller.

Add the following lines inside the createScene method,

In order to check if the surface is valid or not, we first check whether the marker is being shown or not when we scan the surface in the onSelect() method. And we can place any cloned or downloaded model onto the marker.

Step 6: Run the app

We can run the app by running the following command,

npm run start:live

Open the public URL from the command prompt/terminal to see the demo in your browser (Google chrome).

There you have it! Your first webAR app using WebXR and Three.js :)

Thanks for reading this article.

If you enjoyed this article, please click on the clap button 👏 and share to help others find it!

The full source of the demo and this tutorial is available on

GitHub - codemaker2015/web-ar-boilerplate at production

Here are some useful links,

Fundamentals of WebXR - Web APIs | MDN

three.js