Using the Model Tracking Setup Scene

Level: Basic

The ModelTrackingSetup scene is meant as a tool to optimize the tracking parameters for your custom model. It also allows you to troubleshoot tracking or initialization challenges with your model.

For these purposes, the scene offers a debug-view, along with UI elements enabling direct manipulation of tracking parameters and initialization data at runtime. Once a suitable setup is found, you can save it as a .vl configuration file directly from within the scene.

The debug view and options offered in the UI are covered individually in the following.

Workflow for Creating a custom Tracking Configuration

Disclaimer: You should not derive your own development scene from ModelTrackingSetup: The debug functionality available in this scene comes with a performance penalty. It is not designed or optimized for deployment.

The intended use case is to temporarily set up your model in the ModelTrackingSetup scene, set the desired initial pose, tweak the relevant tracking parameters and finally export a configuration file for use in your separate development scene.
The typical workflow for creating a custom configuration in this manner is outlined in the following:

  1. Replace the model in the scene (which is the VLMiniCar by default) with your own model. You may use the step-by-step summary given here or refer to Changing the Model in the Unity Quick Start Guide:
    • Import your model into the Unity project (see Unity's documentation). For supported model file types see our documentation on Model Tracker Configuration File Parameters.
    • Drag the imported model asset into the SceneContent in the hierarchy.
    • Delete or disable the existing VLMiniCar in the SceneContent.
    • Copy your model file into the StreamingAssets folder. E.g. into a new subfolder like StreamingAssets/MyModel.
    • Reference the copied model file in ModelTrackingSetupConfiguration.vl by replacing the `modelURI` parameter with the path to your model. Using the example path from above, the new path would be: streaming-assets-dir:MyModel/filename.type.
      (ModelTrackingSetupConfiguration.vl is located in StreamingAssets/VisionLib/Examples/ModelTracking)
    • Set the `metric` parameter in ModelTrackingSetupConfiguration.vl to the unit of length used in your model file.
  2. Set the saving directory (see Saving the Current Tracking Settings in a Configuration File below for instructions).
  3. Set the the desired initial pose. To do this:
    • Switch to the Scene View in the editor, make sure you are not in 2D mode.
    • Double click the GameObject containing your model in the hierarchy. This snaps the scene view to the model regardless of its location and size.
    • From there, move the scene view to a pose you would like to initialize from.
    • Select the VLInitCamera under VLTracking in the hierarchy by single clicking it. An inset camera preview window becomes visible in the bottom right of the scene view.
    • Press the Align with view button on the VLInitCamera component. This snaps the selected VLInitCamera to the view you are seeing.
    • Check the camera preview inset and ensure the initial pose's point of view is as desired. Should this not yet be the case, refine your view and align the VLInitCamera again. Repeat this until the result is satisfactory.
  4. Start Tracking with default settings (as provided in the 'ModelTrackingSetupConfiguration.vl' with the examples package.)
  5. Tweak the tracking parameters as necessary using the parameter panel and debug view. Details on the usage of these tools are provided in the section Scene Controls below. In advanced use-cases you might also want to record additional initialization data along the way. How this works is explained in the section Reading/Writing/Resetting the Initialization Data below.
  6. Save the init pose and tracking parameter settings in a .vl configuration file using the Save Configuration.
  7. Move the saved configuration into your project:
    • Collect the .vl configuration file from the location it was written to. Refer to the section Saving the Current Tracking Settings in a Configuration File below for path settings.
    • Place the .vl configuration file in the StreamingAssets folder of your project. Ensure the modelURI parameter in the configuration file is correct if you are using a relative URI. In the simplest case, place the model in the same folder as the configuration file and use the project-dir: scheme (see File Access).
  8. Optional: Move the saved initialization data into your project — if applicable.
    • Collect the .binz init data file from the location it was written to. Refer to the section Reading/Writing/Resetting the Initialization Data below for path settings.
    • Place the .binz init data file in the StreamingAssets folder of your project. Add the parameter initDataURI to your configuration if it does not exist yet. Point this parameter to your init data file. In the simplest case, place the init data file in the same folder as the configuration file and use the project-dir: scheme (see File Access).

Congratulations! At this point your have successfully set up a new project with your custom tracking model and bespoke tracking parameters.

Scene Controls

This section covers the purpose of the controls available in the ModelTrackingSetup scene, all of which are shown in the overview image here:


  1. Start/Stop Tracking
  2. Show/Hide the Line Model
  3. SLAM options
    • 'Extend Tracking with SLAM'
    • 'Constrain to Plane'
    • 'Static Scene'
  4. Pause/Resume Tracking
  5. Reset Tracking (Soft/Hard)
  6. Read/Write/Reset Initialization Data
  7. Toggle Debug Mode
  8. Toggle the Parameter Panel
  9. Pause/Resume the Image Stream
  10. Save the Current Tracking Settings in a Configuration File

Starting/Stopping Tracking

The Start button sets up a new tracking context as specified in the configuration file and starts tracking. A tracking context contains the tracker itself and any tracking data generated since start was last pressed. If tracking is already running, pressing Start again discards the current context and loads a fresh one. The Stop button ends tracking, unloads and tears down the current tracking context. These buttons are located at the top left of the scene (see (1) in the overview image above).

When tracking is running, the sequence Stop > Start is the most complete tracking reset option. With regard to tracking, it is equivalent to restarting the Unity Scene entirely.

Pausing/Resuming Tracking

The Pause button halts tracking and image acquisition. The Resume button resumes both. These buttons are located at the top right of the scene (see (4) in the overview image above).

SLAM options: 'Extend Tracking with SLAM', 'Constrain to Plane' and 'Static Scene'

Checking Extend Tracking with SLAM enables the use of external SLAM on devices offering this feature (see External SLAM (ARKit/ARCore) Support for details). This is relevant during rapid or erratic camera movement, as well as when the tracked object is lost from the camera's field of view. If available on the device, enabling SLAM will prevent loss of tracking in these cases — or at least significantly improve tracking reacquisition. This allows the model-based tracking to pick up again automatically after the tracked object re-enters the field of view or when excessive movement subsides.

The option Constrain to Plane "glues" the tracking model to a plane defined by a normal (up) vector. This removes two rotational degrees of freedom, only allowing rotations of the model around the plane's up vector, which can improve performance. This mode is intended only for use cases where the tracked object is set on a flat surface and remains there during tracking. Note: this only works if the flat surface is level and NOT tilted with respect to the environment. For additional documentation of the plane constrained mode, refer to Model Tracker. The plane constrained mode parameters can be customized under under VLTracking > VLPlaneConstrainedMode in the hierarchy.

The Static Scene mode can be used if the tracked objects are not expected to move at all. SLAM and model-based tracking are decoupled in this mode. This allows for a high tracking pose update frequency by pushing through lower accuracy SLAM poses directly and regularly applying high accuracy pose corrections using VisionLib's model-based tracking (each time a new model tracking result becomes available). In effect, the high update frequency makes the application feel smoother.

The checkboxes controlling Extend Tracking with SLAM, Constrain to Plane and Static Scene are located underneath the Show Line Model checkbox at the top left of the screen (see (3) in the overview image above). Note that changes to these settings only take effect after tracking is restarted. Constrain to Plane and Static Scene only become available if Extend Tracking with SLAM is enabled. All these options fully rely on SLAM being available and hence only work on devices offering external SLAM. On other devices, these settings have no effect.

Showing/Hiding the Line Model

The current tracking line model is displayed by default. It is usually hidden under the rendered augmentation of the tracked model. To make the line model fully visible, disable the model's mesh renderer in the hierarchy.

The line model itself can be toggled on and off using the checkbox reading Show Line Model in left button panel ((2) in the overview image):

Resetting Tracking (Soft/Hard)

The Reset Soft button resets tracking and returns to initialization mode. In other words, the currently tracked pose is dropped and the tracking state switches to lost. In some cases VisionLib is able to re-initialize very quickly after a soft reset. The Reset Hard button additionally deletes the dynamic initialization data recorded in the current session. For details on the init data, refer to Initialization: Fast Init & Re-initialization.

The reset buttons are located at the top right of the scene (see (5) in the overview image above).

Reading/Writing/Resetting the Initialization Data

The Read-, Write- and Reset init data buttons (see (6) in the overview image above) are located underneath the two reset buttons. The functionality of these buttons is explained in Initialization: Fast Init & Re-initialization.

As a short summary:

  • Read init data reads saved init data from a file into the static init data storage. By default this file's URI is local-storage-dir:/VisionLib/initData.binz. You can replace this with a custom URI in the hierarchy under Canvas > RightButtonPanel > ReadInitData > Button > On Click().
  • Write init data moves the current dynamic init data into the static init data container and then saves the combined init data in a file. Note that since the formerly dynamic init data has become static it is now protected from Reset Hard and can only be reset with Reset init data. By default the target file's URI is local-storage-dir:/VisionLib/initData.binz. You can replace this with a custom URI in the hierarchy under Canvas > RightButtonPanel > WriteInitData > Button > On Click().
  • Reset init data clears both the dynamic and the static init data containers. This implicitly also happens if you press Start while tracking is running or if your restart tracking using Stop > Start.

Again, for more details on working with init data, refer to Initialization: Fast Init & Re-initialization.

Enabling/Disabling the Debugging View

The Debug on/off button (see (8) in the overview image above) toggles the debug view. This mode shows the grayscale image on which tracking is performed. The line model and search lines are overlaid on top.

In combination with the parameter panel, this view can help you tweak different aspects of the tracking, including:

  • The Line Model: Check whether the line model is satisfactory and modify it if necessary.
  • Tracking initialization:
    • Understand and fix failures to initialize,
    • Reduce tendencies of the tracking to snap on to your model prematurely or in the wrong orientation,
    • Influence the number of fallback poses for re-initialization (keyframes) generated during successful tracking.
  • Tracking after initialization: Make tracking more robust against movement between frames

To learn how to use the sliders in the parameter panel to achieve these goals, familiarize yourself with the pertinent tracking parameters.

A more complete overview over debugging with VisionLib is provided in the Debugging Options page.

Enabling/Disabling the Parameter Panel

The Parameters on/off button (see (8) in the overview image above) toggles the parameter panel. This panel is shown in the image below. It allows you to change the values of a range of tracking parameters during tracking and observe the effects.
While the panel can also be used outside the debug view, the effects of parameter changes are most easily observed in debug view.

Pausing the Image Stream

The Pause Image Stream button (see (9) in the overview image above) stops forwarding images to the tracker. Tracking is repeated indefinitely on the last frame sent to the tracker. This can be used to observe the effect of tracking parameter modifications on the tracking from a given view of the model. This lends itself to use in combination with the debug view and the parameter panel.

This feature also allows efficient initial pose setup using the following method:

  1. Start Tracking.
  2. Move the physical camera to the desired initial pose. Then hold the camera still.
  3. Pause the image stream. You may now set down the camera.
  4. Adjust the init camera's transform (VLTracking > VLInitCamera in the hierarchy) until tracking snaps on to your model.
  5. Save the current configuration. If you do not do this, you will lose the init pose as the init camera's transform is reset upon leaving play mode.

Saving the Current Tracking Settings in a Configuration File

The Save Configuration button (see (10) in the overview image above) saves the current tracking parameters and init pose for the tracker into a .vl configuration file. Note that the current pose of the InitCam in the scene (VLTracking > VLInitCamera) is used as the init pose in this new file.

To make use of it in your development scene, select the VLInitCamera game object from the Hierarchy there, and enable Use Pose From Tracking Config option.

By default this file's URI is local-storage-dir:VisionLib/SavedTrackingConfiguration.vl. A custom URI can be set via Save Configuration URI in the hierarchy on Model Tracker Parameters on VLTracking > VLModelTracker. VisionLib file schemes (see File Access) are supported for this URI. Note that writing to the assets directory may not be possible on some platforms.