documentation

AutoInit with WorkSpaces (Beta)
Level: Advanced

This tutorial page shows you how to use the VisionLib Auto Initialization feature.

Note: This feature is still beta, and might be modified in upcoming releases.

Automatic Initialization: Introduction

In the past, most applications that were made with VisionLib Model Tracking started with a fix initial pose, to which the user had to align the physical object. There was still the possibility to save init data during the session and load it again, which enabled the user to gain some kind of static auto initialization.
While this workflow is still available however, it has disadvantages: it requires to collect such data manually in advance, it is not stable over time and is not robust against any other device rotation, object movement or changing surroundings compared to the recording.

With the new auto initialization, you can define your working area - which is called WorkSpace - and enable an initialization from any point in this area. Theses poses can be "trained" at the start of the application.
This way, you don't need to align the camera view to a certain pose anymore and start the tracking much faster and a lot easier: simply point your camera at the object and it will be recognized and tracked.

The concept of WorkSpaces in VisionLib is a key feature for multiple use cases in future versions:

  • Definition of the area an object can be initialized from
  • Transferability to other applications and models, allowing quickly scaling use-cases
  • Better tracking results in general
  • Defining 3D spatial UI fences for better user guidance
  • ... and even more to come ...

Note: The AutoInit feature, as described here, is a pure on-device feature and does not connect to any learning service outside of the device. It will not and never send your data anywhere.

Technical Requirements

AutoInit might get performance-intensive and can thus cause problems on older hardware. We recommend using hardware from 2018 or newer.

1. Set up a WorkSpace in Unity

Note: Test your tracking target with the SimpleModelTracking first and adjust the tracking parameters accordingly. Only proceed with this experimental feature as soon as the standard tracking is working fine.

You can find an example implementation of WorkSpaces in the AutoInitTracking scene from the VisionLib/Examples/ModelTracking/AutoInit folder.
In your own scene, drag the VLWorkSpace and VLWorkSpaceManager prefab from VisionLib/Utilities/Prefabs/AutoInit into the hierarchy. You can use multiple VLWorkSpaces but only one VLWorkSpaceManager in your scene.

If you like, you can now remove the VLInitCamera from your scene, or keep it to provide a "fallback" init pose.

WorkSpace

The VLWorkSpace prefab consists of the parent object with some general settings:

  • Up Vector: The upright direction of your model (often the Y-Axis (0,1,0))
  • Display View Direction: If enabled, dotted lines will be drawn in the scene view between all origin and destination points
  • Preview in Camera: Enable this to check the initial poses that will be generated (in the game window)

And it contains the Origin and the Destination geometry as children, which define your area of interaction.
Therefore, the WorkSpace describes a set of possible views from the user on the object which are generated by "looking" from all origin points to all destination points.

The calculated number of poses will be shown in the inspector when selecting the VLWorkSpace, as seen in the picture below.

WorkSpace_SceneView.jpg

Origin Geometry

WorkSpaces can be represented geometrically by multiple predefined shapes. Suitable WorkSpace definitions can be saved as prefabs for reuse.
To achieve a custom parameterization, select the origin or destination geometry (children of the VLWorkSpace prefab) to modify its parameters in the inspector.

You can choose between four shapes: sphere, plane, line and point. Besides the width, length, radius and so on you can adjust the detail level or step of each shape, which later affects the number of initial poses to be learned.

The sphere has some special parameters in addition to the radius:

Parameter Default Value Function
Detail Level 0.1 Amount of Pose Points
Vertical Angle Constraints (Polar):
Polar Start Vertical starting angle
Polar Length 90° Vertical sweep angle size
Horizontal Angle Constraints (Azimuthal):
Azimuth Start Horizontal starting angle
Azimuth Length 360° Horizontal sweep angle size


In the VLWorkSpace prefab, the default origin geometry is set to a half sphere, with PolarStart at 0° and PolarLength at 90°. This is a good preset for objects that stand or lie on a surface and should be initialized from above. Other custom variations (e.g. spherical triangle) can be achieved by constraining the horizontal angles as well.
If the WorkSpace geometry configuration is adequate, place it over the tracking target via Unity's transformation component. The WorkSpace and its origin & destination geometry can be translated and rotated.

Note: The scale of the WorkSpace and its geometries cannot be changed at the moment! Always keep it at (1,1,1).

Important: Keep in mind that the pose amount in the VLWorkSpace object depends on the detail level or step count that you defined in the geometry. As a rule of thumb, a higher amount of poses enables more initialization points but also increases the initial pose learning time and might affect the performance negatively. A decreased performance can overall lead to a slower initialization!
That said, it is always better to only set up initialization areas that are actually needed.

Destination Geometry

Here you can either use the Destination GameObject, which is a child of the VLWorkSpace, and adjust it similar to how you used the Origin. In most cases, it will be enough to use only one target point.
Or you can simply drag your tracking target object from the hierarchy inside this public parameter. In that case, the geometric center of the GameObject will be used as a single target point.

Note: When using a GameObject as destination, the transform of this object must lie in the origin (0,0,0) of the scene.
Note: If a destination point shares its position with an origin point, it won't be possible to create a corresponding initialization pose.

Preview

To preview the poses, that you generated by setting the origin geometry and destination object, you can use the following:

See the gizmos in scene view:

  • The used pose points are drawn as small wireframe spheres in the scene view (origin: white, destination: cyan) if VLWorkSpace is selected
  • Activate the showLines parameter of the VLWorkSpace to show dotted lines from origin to destination points in scene view. To keep the scene tidy, they will only be drawn if the number of poses is smaller than a fix maximum.

Preview poses in camera:

  • Use the slider in the inspector of the VLWorkSpace to preview the generated poses in your game view (make sure you can see the game view window in Unity)
  • Note: The transform of the camera used for preview will be modified by this feature. If you are unsure, don't use your main camera here.

WorkSpaceManager

The VLWorkSpaceManager is mandatory and aggregates one or more VLWorkSpace definitions to process them during runtime.

It contains the following public parameters:

  • Auto Start Learning: Toggle to activate/deactivate an autostart for the pose learning as soon as the VisionLib tracking is started
  • Show Progress Bar: Toggle to activate/deactivate the progress bar, which shows the learning progress

2. Adjust the Tracking Configuration

As always, reference your used tracking configuration (.vl file) on the VLTracking object and your license on the VLCamera.

To enable AutoInit, insert "autoInit": true in your tracking configuration:

"tracker": {
// ... tracker definition
"parameters": {
// Enable AutoInit Feature
"autoInit": true,

Note: You can delete the "initPose" section from your tracking configuration, or keep it to provide a "fallback" pose.

3. Start Tracking with Auto Initialization

The last step is to activate the pose learning. Do that either by using the autostart option in the VLWorkSpaceManager(which is active by default), or by calling the following function e.g. on a button click or from your script: VLWorkSpaceManager.LearnWorkSpaceInitData()

Wait some seconds (activate the progress bar in the VLWorkSpaceManager to show the learning progress) until all poses have been trained.
Now you can point your camera to the physical object from a direction that is included in the VLWorkSpace that you defined and it will be tracked.

Caching

The footprint of the learned data is quite small (several 100kB) and will be cached in local_storage_dir:/VisionLib/AutoInit/initData_filename.binz. The filename is a combination of the hash of your WorkSpace definition and model definition.

You only need to "train" the WorkSpaces once. When you call the learning command a second time with the same WorkSpace definition and model configuration (including transforms and model-constellation), the cached file will be used instead of relearning.

To stop AutoInit or reset its learned data on runtime, you need to call VLModelTrackerBehaviour_v1.ResetInitData(). In the example scene, the Clear button is using this command.
This will not delete the cache, which thus can be loaded again without relearning (e.g. by pressing the Learn button in the example scene).
If you want to force a relearning of the poses, you will need to remove the .binz file from the local storage of your device.

Trouble Shooting

AutoInit is not working as expected? Check the following:

  • Make sure the standard tracking is working fine before proceeding with AutoInit.
  • Point your physical camera to your object from different directions and distances and move it slowly as it sometimes takes some seconds for the object to be found in the camera image.
  • Make sure your object is completely visible and not partly occluded. E.g. holding the example car in your hand might not successfully initialize tracking.
  • Calibrate your movable web-camera (not the built in one) and test your scene from within Unity before deploying to mobile devices. This makes the process much faster.
  • Make sure your Origin geometry points have enough distance from your model to see it entirely in long shot. You can not rely on the camera preview in Unity here, because your physical camera might have a different field of view. Try to increase the distance (e.g. by increasing the sphere radius) by a fair amount.
  • If you still experience problems, you can adjust some optional parameters in your tracking configuration or check the debug images as described below:

Additional AutoInit Parameters

You can add some optional parameters for AutoInit by inserting it as a section in your tracking configuration:

"tracker": {
// ... tracker definition
"parameters": {
// Enable AutoInit Feature
"autoInit": {
"laplaceThreshold": 0.7,
"writeDebug": true
},

The following optional parameters are available:

Parameter Type Default Value Function
laplaceThreshold float laplaceThreshold of ModelTracker * 4 See ModelTracker laplaceThreshold
normalThreshold float normalThreshold of ModelTracker See ModelTracker normalThreshold
writeDebug bool false If set to true, the vlSDK will write images of the renderings (that are an intermediate step when learning features for AutoInit) to the writeDebugPath.
The debug images are especially helpful to understand the effects of laplaceThreshold and normalThreshold to the edge extraction.
writeDebugPath string "local_storage_dir:/VisionLib/AutoInit/DebugImages/EdgeImage_" Path for writing debug images. The default path writes images to local_storage_dir:/VisionLib/AutoInit/DebugImages_[n]/EdgeImage_[m].png
cacheDir string "local_storage_dir:/VisionLib/AutoInit/" The directory in which the cached AutoInit learning results will be stored.

Write Debug Images

When troubleshooting, it might be helpful to activate "writeDebug":true in your tracking configuration.
Have a look at the generated images. Your object should be fully visible and lie in the center of the image. If not, adjust your WorkSpace parameters in Unity.

If the white lines in the images are dotted or messy in some way, you can modify the laplaceThreshold and normalThreshold of the autoInit section to get better results. Usually you will need to take a much higher float value than for the corresponding parameters of the model tracker. If the debug images have clear, tidy lines, the set parameters are fine.

See an example for debug images with good and bad lines below:

DebugImages_compared.jpg
Left: Good lines , Middle: Bad (too cluttered), Right: Bad (only outlines visible)

Adjust Up-Vector

If your written debug images show your model rotated, you should adjust the up vector of your WorkSpace.

RotatedDebugImage.jpg
Left: Up-Vector (0,1,0) , Right: Up-Vector (1,0,0)

Long shot vs. Close-Up

Currently, AutoInit works well on objects that are initialized in a long shot view. Thus, smaller objects which fit into the camera view completely will work much better.
You can check if the object is fully visible in the camera preview. If you are unsure, increase the distance from the VLWorkSpace origin points to the target object.

Close-Up object initialization is still under development, but might already work with the tools provided in this version.

Note: On HoloLens 2, it can make a great difference to set "fieldOfView: narrow" when working with small objects. For more details on that parameter, please refer to HoloLens Optimizations.

Why not always use a whole Sphere as the Origin Geometry?

This is a legit question. Anyway, initializing the tracking from all views might not be necessary and restricting the working area will help VisionLib to find the object more reliably.
Also, you should prevent the definition of views, from where the object is not even recognizable or distinguishable by its form: Initializing a cell phone from the side will probably result in detecting a pencil as well.
So please consider this when defining WorkSpaces.

Define WorkSpaces with VisionLib native SDK

Please contact us and we can help you find a way to integrate WorkSpaces into your product.

Feedback

We are happy if you give us feedback or even provide us image sequences of objects being initialized.
If you have issues or suggestions on how to enhance or integrate this feature, please do not hesitate to contact us on reque.nosp@m.st@v.nosp@m.ision.nosp@m.lib..nosp@m.com.