documentation

AutoInit with WorkSpaces (Beta)

Level: Advanced

This tutorial page shows you how to use the VisionLib Auto Initialization feature.

Note: This feature is still beta, and might be modified in upcoming releases.

Automatic Initialization: Introduction

In the past, most applications that were made with VisionLib Model Tracking started with a fix initial pose, to which the user had to align the physical object. There was still the possibility to save init data during the session and load it again, which enabled the user to gain some kind of static auto initialization.
While this workflow is still available, it requires to collect such data manually in advance. It is not stable over time and is not robust against any other device rotation, object movement or changing surroundings compared to the recording.

With the new auto initialization, you can define your working area - which is called WorkSpace - and enable an initialization from any point in this area. Theses poses can be "trained" at the start of the application.
This way, you don't need to align the camera view to a certain pose anymore and start the tracking much faster and a lot easier: simply point your camera at the object and it will be recognized and tracked.

Note: The AutoInit feature, as described here, is a pure on-device feature and does not connect to any learning service outside of the device. It does not send your data anywhere.

Technical Requirements

AutoInit might get performance-intensive and can thus cause problems on older hardware. We recommend using hardware from 2018 or newer.

1. Set up a WorkSpace in Unity

Note: Test your tracking target with the SimpleModelTracking first and adjust the tracking parameters accordingly. Only proceed with this experimental feature as soon as the standard tracking is working fine.

You can find an example of how to use AutoInit in the AutoInitTracking scene which is located in VisionLib Examples/ModelTracking/AutoInit after importing the VisionLib.SDK.Examples-Unity.unitypackage.

To add AutoInit to your own scene, right click on the GameObject that you are using as an augmentation of your tracking target and then choose VisionLib > AutoInit > Create Simple WorkSpace. This will create a VLSimpleWorkSpace with the selected GameObject as its destination. It will also create a VLWorkSpaceManager GameObject if it doesn't exist yet.

If you like, you can now remove the VLInitCamera from your scene, or keep it to provide a "fallback" init pose.

There are two types of WorkSpaces, the Simple- and the AdvancedWorkSpace.

We strongly recommend to use the SimpleWorkSpace and only choose the AdvancedWorkSpace if the degrees of freedom given by the SimpleWorkSpace do not allow you to model your workspace close enough.

WorkSpace Basics

Both the Simple- and the AdvancedWorkSpace share some general settings:

SettingDescription
UsedCamera VisionLib will use the field of view of this camera when creating renderings of your model. This way, the visualization in the editor matches the actual rendering in the VisionLib SDK. The camera reference is also used for stepping through the positions with the "Switch through Poses" slider.
Up VectorThe upright direction of your model. If the orientation is aligned with the unity world, choose (0,1,0).
Destination ObjectGameObject containing your tracked model as a mesh.
Display View DirectionIf enabled, dotted lines will be drawn in the Scene view between all origin and destination points.
Switch through posesUse this slider for a Game view preview of the poses that you defined for the WorkSpace. This will set the transform of the camera assigned to "UsedCamera".

SimpleWorkSpace

The SimpleWorkSpace allows you to select the angles from which to initialize your tracked model using the settings that are summarized in "Sphere Constraints".

The values are essentially the same as for the Sphere of the AdvancedWorkSpace (see below) - and can also easily be manipulated using the green and red handles in the Scene view.

  • Double click on the VLSimpleWorkSpace GameObject to select it and focus it in the Scene view.
  • Drag the green and red handles to manipulate the area from which initialization should be possible. The values will be saved as vertical and horizontal angle constraints of the spherical WorkSpace.

The radius of the sphere will be adjusted automatically to fit the bounds of the destination object and the field of view of the camera.

Note: To improve recognition results, it is advisable to set the field of view of your camera to a value that roughly matches the one of the physical camera you will use for tracking later.

AdvancedWorkSpace

The AdvancedWorkSpace introduces more degrees of freedom to model the view points you want to cover with the AutoInit. Geometry primitives of different types can be used to describe both the origin and the destination.

When creating an AdvancedWorkSpace via context menu (or using the prefab VLAdvancedWorkSpace), the created GameObject will contain two children - Origin and Destination. Both will come with a WorkSpaceGeometry component attached.

Each WorkSpaceGeometry produces a set of points. The AdvancedWorkSpace will then generate views by "looking" from all origin points to all destination points.

A WorkSpaceGeometry can be one of four primitives, that can be selected with the Shape attribute of that component.

Sphere

Parameter Default Value Function
Radius 1 Radius of the sphere
Detail Level 0.1 Amount of Pose Points
Vertical Angle Constraints (Polar):
Polar Start Vertical starting angle
Polar Length 90° Vertical sweep angle size
Horizontal Angle Constraints (Azimuthal):
Azimuth Start Horizontal starting angle
Azimuth Length 360° Horizontal sweep angle size

The default origin geometry of the AdvancedWorkSpace is set to a half sphere, with PolarStart at 0° and PolarLength at 90°.

This is a good preset for objects that stand or lie on a surface and should be initialized from above. Other custom variations (e.g. spherical triangle) can be achieved by constraining the horizontal angles as well.

Plane

Parameter Default Value Function
Length 1 Horizontal extend of the plane
Width 1 Vertical extend of the plane
Step 3 Number of sample points in each dimension

Line

Parameter Default Value Function
Length 1 Extend of the line
Step 3 Number of sample points

If the WorkSpace geometry configuration is adequate, place it over the tracking target via Unity's transformation component. The WorkSpace and its origin & destination geometry can be translated and rotated.

Note: The scale of the WorkSpace and its geometries cannot be changed at the moment! Always keep it at (1,1,1).

Important: Keep in mind that the pose amount in the WorkSpace component depends on the detail level or step count that you defined in the geometry. As a rule of thumb, a higher amount of poses enables more initialization points but also increases the initial pose learning time and might affect the performance negatively. A decreased performance can overall lead to a slower initialization!
That said, it is always better to only set up initialization areas that are actually needed.

Destination Geometry

Here you can either use the Destination GameObject, which is a child of the VLWorkSpace, and adjust it similar to how you used the Origin. In most cases, it will be enough to use only one target point.
Or you can simply drag your tracking target object from the hierarchy inside this public parameter. In that case, the geometric center of the GameObject will be used as a single target point.

Note: When using a GameObject as destination, the transform of this object must lie in the origin (0,0,0) of the scene.
Note: If a destination point shares its position with an origin point, it won't be possible to create a corresponding initialization pose.

Preview

To preview the poses, that you generated by setting the origin geometry and destination object, you can use the following:

See the gizmos in Scene view:

  • The used pose points are drawn as small wireframe spheres in the Scene view (origin: white, destination: cyan) if the VLWorkSpace is selected.
  • Activate the Display View Direction parameter of the WorkSpace component to show dotted lines from origin to destination points in Scene view. To keep the scene tidy, they will only be drawn if the number of poses is smaller than a fix maximum.

Preview poses in camera:

  • Use the slider in the inspector of the WorkSpace component to preview the generated poses in your game view (make sure you can see the game view window in Unity).
  • Note: The transform of the camera used for preview will be modified by this feature. If you are unsure, don't use your main camera here.

WorkSpaceManager

The VLWorkSpaceManager is mandatory and aggregates one or more VLWorkSpace definitions to process them during runtime.

It contains the following public parameters:

  • Auto Start Learning: Toggle to activate/deactivate an autostart for the pose learning as soon as the VisionLib tracking is started
  • Show Progress Bar: Toggle to activate/deactivate the progress bar, which shows the learning progress

2. Adjust the Tracking Configuration

As always, reference your used tracking configuration (.vl file) and license in the TrackingConfiguration component of the VLTracking object. To enable AutoInit, insert "autoInit": true in your tracking configuration:

"tracker": {
// ... tracker definition
"parameters": {
// Enable AutoInit Feature
"autoInit": true,

Note: You can delete the "initPose" section from your tracking configuration, or keep it to provide a "fallback" pose.

3. Start Tracking with Auto Initialization

The last step is to activate the pose learning. Do that either by using the autostart option in the WorkSpaceManager(which is active by default), or by calling the following function e.g. on a button click or from your script: WorkSpaceManager.LearnWorkSpaceInitData()

Wait some seconds (activate the progress bar in the WorkSpaceManager to show the learning progress) until all poses have been trained.
Now you can point your camera to the physical object from a direction that is included in the VLWorkSpace that you defined and it will be tracked.

Caching

The footprint of the learned data is quite small (several 100kB) and will be cached in local-storage-dir:/VisionLib/AutoInit/initData_filename.binz. The filename is a combination of the hash of your WorkSpace definition and model definition.

You only need to "train" the WorkSpaces once. When you call the learning command a second time with the same WorkSpace definition and model configuration (including transforms and model-constellation), the cached file will be used instead of relearning.

To stop AutoInit or reset its learned data on runtime, you need to call ModelTracker.ResetInitData(). In the example scene, the Clear button is using this command.
This will not delete the cache, which thus can be loaded again without relearning (e.g. by pressing the Learn button in the example scene).
If you want to force a relearning of the poses, you will need to remove the .binz file from the local storage of your device.

Trouble Shooting

AutoInit is not working as expected? Check the following:

  • Make sure the standard tracking is working fine before proceeding with AutoInit.
  • Point your physical camera to your object from different directions and distances and move it slowly as it sometimes takes some seconds for the object to be found in the camera image.
  • Make sure your object is completely visible and not partly occluded. E.g. holding the example car in your hand might not successfully initialize tracking.
  • Calibrate your movable web-camera (not the built in one) and test your scene from within Unity before deploying to mobile devices. This makes the process much faster.
  • Make sure your Origin geometry points have enough distance from your model to see it entirely in long shot. You can not rely on the camera preview in Unity here, because your physical camera might have a different field of view. Try to increase the distance (e.g. by increasing the sphere radius) by a fair amount.
  • If you still experience problems, you can adjust some optional parameters in your tracking configuration or check the debug images as described below:

Additional AutoInit Parameters

You can add some optional parameters for AutoInit by inserting it as a section in your tracking configuration:

"tracker": {
// ... tracker definition
"parameters": {
// Enable AutoInit Feature
"autoInit": {
"laplaceThreshold": 0.7,
"writeDebug": true
},

The following optional parameters are available:

Parameter Type Default Value Function
laplaceThreshold float laplaceThreshold of ModelTracker * 4 See ModelTracker laplaceThreshold
normalThreshold float normalThreshold of ModelTracker See ModelTracker normalThreshold
writeDebug bool false If set to true, the VisionLib SDK will write images of the renderings (that are an intermediate step when learning features for AutoInit) to the writeDebugPath.
The debug images are especially helpful to understand the effects of laplaceThreshold and normalThreshold to the edge extraction.
writeDebugPath string "local-storage-dir:/VisionLib/AutoInit/DebugImages/EdgeImage_" Path for writing debug images. The default path writes images to local-storage-dir:/VisionLib/AutoInit/DebugImages_[n]/EdgeImage_[m].png
cacheDir string "local-storage-dir:/VisionLib/AutoInit/" The directory in which the cached AutoInit learning results will be stored.
maxImageSize int 640 Maximum side-length of the image used for AutoInit. If this is smaller then the tracking image, VisionLib will downscale the image for auto init. Increasing this value will allow to initialize from larger distances.

Write Debug Images

When troubleshooting, it might be helpful to activate "writeDebug":true in your tracking configuration.
Have a look at the generated images. Your object should be fully visible and lie in the center of the image. If not, adjust your WorkSpace parameters in Unity.

If the white lines in the images are dotted or messy in some way, you can modify the laplaceThreshold and normalThreshold of the autoInit section to get better results. Usually you will need to take a much higher float value than for the corresponding parameters of the model tracker. If the debug images have clear, tidy lines, the set parameters are fine.

See an example for debug images with good and bad lines below:

Left: Good lines , Middle: Bad (too cluttered), Right: Bad (only outlines visible)

Adjust Up-Vector

If your written debug images show your model rotated, you should adjust the up vector of your WorkSpace.

Left: Up-Vector (0,1,0) , Right: Up-Vector (1,0,0)

Long shot vs. Close-Up

Currently, AutoInit works well on objects that are initialized in a long shot view. Thus, smaller objects which fit into the camera view completely will work much better.
You can check if the object is fully visible in the camera preview.

Close-Up object initialization is still under development, but might already work with the tools provided in this version.

Note: On HoloLens 2, it can make a great difference to set "fieldOfView: narrow" when working with small objects. For more details on that parameter, please refer to HoloLens Optimizations.

Why not always use a whole Sphere as the Origin Geometry?

This is a legit question. Anyway, initializing the tracking from all views might not be necessary and restricting the working area will help VisionLib to find the object more reliably.
Also, you should prevent the definition of views, from where the object is not even recognizable or distinguishable by its form: Initializing a cell phone from the side will probably result in detecting a pencil as well.
So please consider this when defining WorkSpaces.

Define WorkSpaces with VisionLib native SDK

Please contact us and we can help you find a way to integrate WorkSpaces into your product.

Feedback

We are happy if you give us feedback or even provide us image sequences of objects being initialized.
If you have issues or suggestions on how to enhance or integrate this feature, please do not hesitate to contact us on reque.nosp@m.st@v.nosp@m.ision.nosp@m.lib..nosp@m.com.