HoloLens Model and Poster Tracker Tutorial

Level: Basic

Tracking with the VisionLib SDK also works on Microsoft HoloLens. This tutorial will show you how to use and customize the HoloLens example scenes of the VisionLib SDK in Unity.

Note: It is not possible to remote control the HoloLens from Unity when using VisionLib at the moment. You have to deploy your application using Visual Studio to test the tracking properly.

Preparation

Please make sure you have installed all required tools and adjusted the settings for HoloLens and Unity Development.

• Windows 10, Windows 10 SDK, Developer Mode activated
• HoloLens with activated Developer Mode
• Visual Studio 2017 (with UWP Tools installed) or higher
• Unity with .NET/ IL2CPP Scripting Backend

Set Up the Unity Project

At first, there are some adjustments of the Unity project needed.

1. Import the vlUnitySDK.unitypackage into a new Unity project.
2. Copy your license file to Assets/StreamingAssets/VisionLib and name it license.xml.
3. Open the HoloLensSimpleModelTracking or HoloLensPosterTracking example scene in Unity from VisionLib/Examples/ModelTracking/Simple or VisionLib/Examples/PosterTracking/Scenes
4. Adjust the build settings (File/Build Settings):
   • Add your scene by clicking the Add Open Scenes button
   • Set platform to: Universal Windows Platform and hit Switch Platform
   • Choose HoloLens as the target device
   • For HoloLens 2, set the architecture to ARM (in Unity 2019 you are able to choose ARM64 to improve performance)
     
5. Adjust the player settings (Edit/Project Settings/Player):
   • XR Settings/Virtual Reality Supported: ON
   • XR Settings/Virtual Reality SDKs: Windows Mixed Reality
     
   • Publishing Settings/Capabilities:
     • WebCam: ON
     • Microphone: ON
     • VideosLibrary: ON
       

Configure the Scene

If you want to use the VisionLib mini car model or the default leaves poster, you can skip to the Deploy on HoloLens section below.

If you are using your own model, we recommend to try out the general smartphone / tablet tutorials first, to test if your poster or 3D model works well.

Important: For HoloLens scenes, the metric of the provided 3D model is very important. You have to specify the units used in your model file in the metric attribute in the parameters section of the tracker. If the 3D model is scaled in a different metric, the interaction between the VisionLib SDK and HoloLens does not work properly. This is crucial because the distance of the object to the camera will be deduced from the specified size of the model. If you are unsure about the metrics of your 3D model, please see Understanding Tracking Parameters for help.

Poster Tracker

1. Copy your image to the StreamingAssets/VisionLib/Examples/PosterTracking folder of your project.

2. At the same location, make a copy of HoloLensPosterTracker.vl and adjust it:
   a) Rename the file accordingly.
   b) Open it with a text editor and change the attribute imageURI from leaves.png to the filename of your own image (e.g. "test.jpg").
   Retain project_dir: in front of the name.
   c) Adjust the attribute realWidth according to the physical size of your poster in meters. The default value matches ISO A4 (210 mm x 297 mm).

   
3. Also make a copy of the Unity scene and inside it, reference your tracking configuration in the VLTrackingConfiguration component of the VLTracking GameObject.
4. Find and select the GameObject Poster_Quad (as a child of SceneContent/Poster) in the hierarchy and enter the scale of your physical poster in meters in the transform component of the GameObject.
5. If you would like to see a digital debug image of your poster like the pink one in the picture below, copy your image to the Materials folder. Select the mat_Poster_Debug material in VisionLib/Examples/Common/Materials in Unity's project panel and change the Albedo map to your image.
   If you do not want a debug image to be shown, disable the Mesh Renderer component of Poster_Quad.
   
6. Insert the 3D content that you would like to associate with the poster into the hierarchy as a child of the Poster_Content GameObject.

Model Tracker

1. Copy your model into the folder StreamingAssets/VisionLib/Examples/ModelTracking of your project.
2. At the same location, make a copy of carHoloLens.vl, open it with a text editor, and change the attribute modelURI to the filename of your own model (e.g. "tutorial.obj"). Retain project_dir: in front of the name.
   
3. Also make a copy of the Unity scene and inside it, reference your tracking configuration in the VLTrackingConfiguration component of the VLTracking GameObject.
4. Also, copy your model into the Models folder of your Unity project. Drag it from the project panel into the hierarchy of your scene as a child of the GameObject SceneContent.
5. If necessary, adjust the initial pose of the tracked model by moving the GameObject VLHoloLensInitCamera. Hint: Adjust your scene view and press GameObject/Align With View while the VLHoloLensInitCamera is selected to transfer the parameters of your scene view to VLHoloLensInitCamera.

Essential: In VLTracking/VLHoloLensTracker the SceneContent GameObject is referenced as the Content attribute. Only the GameObject, which is referenced as Content will be positioned (and thus displayed) correctly in the application. So - as an alternative to drop your model as a child of the SceneContent GameObject, you can also drop it somewhere else. If you do so, you also have to adjust the Content attribute in VLTracking/VLHoloLensTracker to your new GameObject.

Deploy on HoloLens

1. Press the Build button in the File/Build Settings window and select a folder for the HoloLens deployment.
2. Open the generated Visual Studio Solution (*.sln) in that folder.
3. Adjust the deployment settings in Visual Studio:
   • Change the dropdown settings to Release or Debug and x86 (on HoloLens 2 to ARM / ARM64) as seen below.
   • Change from Local Machine to Remote Machine (deploy via Wi-Fi) or Device (deploy via USB connection).

   • If you choose Remote Machine, you will be asked to enter the IP address of the HoloLens first. (On HoloLens: Settings/Network & Internet/Hardware Properties/IPv4 address).

     
4. Make sure, that the HoloLens is turned on and the developer mode is activated (On HoloLens: Settings/Update/For developers).
5. Press Remote Machine or Device to deploy and run the application on the HoloLens.

6. If this is the first deployment for HoloLens on your computer, you will be asked to pair the devices. Find the required PIN on your HoloLens in Settings/Update&Security/For developers.

   

Once the HoloLens application starts, you will be asked for permission to use the microphone and webcam. This is mandatory for tracking and the use of voice commands. Airtap Yes to confirm the dialog. Now gaze at your model to initialize tracking. You will see a debug image that is moving together with your head movement, so you can simply check what might cause tracking issues.

Please consider to improve initialization and re-initialization speed by using the Read/Write Data commands below. To understand the process behind them, have a look at Initialization: Fast Init & Re-initialization.

Voice and Keyboard Commands

The HoloLens example scenes include some simple commands that you can invoke with your voice or a connected bluetooth keyboard. Please note that voice commands only work, if your HoloLens has an internet connection.
You can change or extend the list by editing the GameObjects SpeechInput and KeyboardHandler, which you can find under the Input GameObject in the hierarchy. All commands available by default can be found in the table below.

Voice Command	Key	Description
Reset Soft	S	Resets the tracking to the init pose
Reset Hard	H	Resets the tracking to the init pose and deletes all saved templates
Pause Tracking	P	Stops the tracking, also freezes the debug image
Resume Tracking	R	Resumes tracking and unfreezes the debug image
Hide Debug Image	D	Makes the debug image invisible
Show Debug Image	D	Makes the debug image visible
Read Data	I	Reads and applies saved tracking templates
Write Data	W	Stores dynamically collected tracking templates for later use

For the HoloLensPosterTracking scene, only the first four commands apply.

Trial Licence Restrictions

With the trial license you can only start an application up to five times after successful deployment. Afterwards, it no longer works together with the vlSDK. Then, you will need modify your project and redeploy the application to the device.
You can overcome this technical restriction by purchasing a different license type. Please note that the trial license may only be used for evaluation purposes and does not permit application development.

When deploying on HoloLens with Visual Studio, updates are made instead of a complete reinstallation of the app, which might cause the deployment counter not to be reset correctly. If you still experience licence errors, try to uninstall the app on the HoloLens manually before deploying, or enable the option Uninstall and then re-install my package in Visual Studio Project/Properties/Debug.

Configuration File Parameters

If you want to make further adjustments on the VisionLib tracking behavior, you can use the parameters in the tracking configuration. A typical configuration file can be seen in the following example:

 {
  "type": "VisionLibTrackerConfig",
  "version": 1,
  "meta": {
    "name": "Simple VisionLib Car Tracker for HoloLens",
    "description": "Simple tracking configuration file for Microsoft HoloLens",
    "author": "VisionLib"
  },
  "tracker": {
    "type": "hololensModelTracker",
    "version": 1,
    "parameters": {
      "modelURI": "project_dir:car.obj",
      "useColor": true,
      "metric": "m",
      "keyFrameDistance": 50,
      "laplaceThreshold": 1,
      "normalThreshold": 0.3,
      "maxFramesFeaturePrediction": 3,
      "lineGradientThreshold": 50,
      "lineSearchLengthTrackingRelative": 0.04,
      "minNumOfCorrespondences": 50,
      "minInlierRatioInit": 0.7,
      "minInlierRatioTracking": 0.6,
      "showLineModel": true,
      "showLineModelCriticalColor": [255, 0, 127]
    }
  },
  "input": {
    "useImageSource": "cameraHololens",
    "imageSources": [{
      "name": "cameraHololens",
      "type": "camera",
      "data": {
        "undistort": true,
        "scale": 0.5,
        "framerate": 30,
        "cameramode": 0
      }
    }]
  }
}

All parameters which are valid for the generic Model Tracker are also valid for the HoloLens tracker.

The camera mode of the HoloLens can be defined by the following parameters.

Parameter	Type	Default value	Example
undistort	boolean	false	"undistort": true
	This flag defines whether the HoloLens camera image should be undistorted before further processing. As the HoloLens internal calibration comes with radial distortion parameters, this value should always be set to true, if custom intrinsic camera calibration parameters are not given. If custom intrinsic parameters are given without radial distortion parameters, the flag undistort can be set to false.
scale	float	1.0	"scale": 0.5
	If this value is set to 0.5, only half of the image resolution is used for camera tracking. This increases processing speed but reduces the tracking precision. If it is set to 1, the full image resolution is used.
framerate	int	30	"framerate": 30
	This value defines the framerate of the HoloLens image acquisition. Valid values are 30, 24, 20, 15 and 5.
cameramode	int	0	"cameramode": 3
	This value defines the camera mode of the HoloLens image acquisition. Valid camera modes are: 0: 1280x720, 45 deg FOV 1: not usable 2: 1408x792, 48 deg FOV 3: 1344x756, 67 deg FOV 4: 896x504; 48 deg FOV A way to select a good video mode is to analyse the tracked object in the debug image. If the whole object is always located inside the camera image during a typical use case, the mode seems to be fine. If the object is large, and only a part is visible during the runtime, a larger FOV (mode 3) might be useful.

These parameters may change in future releases.

User Defined Calibration

HoloLens provides intrinsic values of the camera device, which seem to work fine in many cases. However, the internal HoloLens intrinsic parameters rely on a radial distortion model, which is not optimal in many cases. Since the camera image has almost no distortion, a custom intrinsic camera calibration without distortion parameters provides better results in some cases. If a custom intrinsic camera calibration will be used, it can be defined as in the following example.

"input": {
  "useImageSource": "cameraHololens",
  "imageSources": [{
    "name": "cameraHololens",
    "type": "camera",
    "data": {
      "device": "Hololens",
      "undistort": false,
      "scale": 0.5,
      "framerate": 30,
      "cameramode": 3,
      "calibration": { // custom calibration for mode 3
        "width": 1344,
        "height": 756,
        "fx": 0.79193672839,
        "fy": 1.4100414006,
        "cx": 0.51323558113,
        "cy": 0.50764185229
      }
    }
  }]
}

In the current release, there is no user friendly HoloLens calibration process yet. Therefore it is recommended not to use a custom intrinsic calibration.

Finding problems during runtime

Once an application has been deployed to the HoloLens, finding the origin of a problem can be difficult. The vlSDK log messages could possibly give you a hint in the right direction. To see the log messages, you can start the application on the HoloLens with the Visual Studio debugger. The messages will then be shown in the Visual Studio "Output" window. To be able to see the log messages while deploying with IL2CPP, you need to enable ScriptDebugging in the Unity Build Settings and deploy the build via Visual Studio as Debug.

Alternatively, you can see the log messages in the HoloLens Web Portal under "Logging". In order to activate it, you must select the "Microsoft-Windows-Diagnostics-LoggingChannel" provider. The log messages of the vlSDK will then appear in the "Events" section.

HoloLens live view with Miracast

Many users have noticed, that it is not possible to have a live preview of the HoloLens in the HoloLens web portal while using the vlSDK. This is due to the exclusive camera access of the web portals live preview. But taking pictures or recording videos does work.

An alternative to the web portal was introduced with the October Update 2018 (version 1809): It allows connection to a Miracast device (also at least version 1809) and thus streaming of the camera image with displayed virtual content. To enable the streaming onto a Miracast device, you first have to start your application. Inside this application, use the bloom gesture to switch to an intermediate menu.

The button on the right opens a sub-menu for establishing the connection to a miracast device. Immediately after you have selected your device in this list, streaming will start.

Important: You should stop your connection before starting the application. Even if you leave the application via bloom gesture (and selecting home), you should stop the miracast connection. Otherwise there will be two processes competing for the camera and the miracast connection will be corrupted. Then only a restart of the HoloLens can still resolve this problem.