VisionLib's Enhanced Model Tracker can handle a multitude of 3D objects types. However, not all objects are the same: some are tiny, other are huge. Some come with nice edges and a well trackable geometric structure, other are almost symmetric, with round and blunt edges. So tracking these objects isn't always a one-click-success. Instead, we need to tell VisionLib some information about our tracking targets to improve tracking quality.
There are many parameters that influence the tracking quality which can be controlled in the tracking config .vlfile. Understanding these properties and how they affect the tracking is subject of this tutorial. We won't have a look at all parameters in detail, but discuss and explore the most important ones that might have most impact on your tracking results.
Setting up your .vl file, first of all make sure to set the correct metric.
Usually you don't need to worry, if e.g. your physical model is centimeter-sized, while your 3D file is measured in millimeters. But especially when combining VisionLib tracking with external SLAM (ARKit, ARCore or HoloLens), metrics must fit and be declared absolutely, match 1 by 1. Fetching a 3D model's metric isn't as straight forward as with a physical ones. If you are not sure if you know the metric of your 3D model, we recommend using some tool like MeshLab. Here, you can digitally measure the length of the model and retrieve the scaling.
Lets do a test with our VisionLib mini car model (you can download here). Load the 3D model in MeshLab and an measure its length. You can do the same with your 3D object-to-be-tracked as well.
The real size of our physical mini car is 10 cm and in the 3D model you can measure 0.1 units. So we have to set the the metric parameter to m in the .vl file.
If your model is not in the right scale try rescaling e.g. in MeshLab, or 'rescale' via the metric parameter in the config file. You might use mm, cm, dm, m. Imperial units are currently not supported.
Once you have configured your metrics correctly, you have set the basis for configuring the rest of the parameters. VisionLib internally calculates using mm – so any other parameters to be set will refer to millimeters.
Two parameters allow you to control the line model and subsequently the 'search lines' for tracking that will be generated. Those are especially used during initialization. For debugging purposes you can either set the showLineModel parameter to true or add "debugLevel":2 in your config in order to show the generated line model. We recommend the latter.
You can configure the depth jump in space to be interpreted as an edge by setting the laplaceThreshold parameter in mm.
In the image you can see an example of a configured sensitivity (laplaceThreshold) of 1, 5 and 10 mm for a 1:18 car model of a Ferrari (without normalThreshold defined). Note how the amount of inlying lines decreases. Ergo, there will be less 'search lines' for the tracker later on to detect inlying structures of the physical model.
NOTE: If you use m in your model, which is very common when dealing with Unity - it is advisable to set your laplaceThreshold to some higher value than 1 mm (default). Depending on your GPU, this can yield to incomplete line models. You might better start with 10-50mm.
If you are unsure of the effect, please use VisLab and load the object to be tracked, set your metric accordingly, enable show lines and change the laplaceThreshold.
You may also add additional edges by defining a threshold for neighboring surfaces and their angle between (curvature). This way generating lines is disabled by default (by setting a high value). However, at significant model edges it might be of use specifying the normalThreshold parameter.
In the second image you can observe the differences for this parameter. We set laplaceThreshold to 1 and then normalThreshold to 0.01, 0.1 and 0.4. Please note that a very low value may result in generation of not detectable lines, which impacts the performance significantly.
Once a nice line model generation has been configured, you will need to observe the trackingInlierRatio and the initInlierRatio which correspond directly to the minTrackingInlinerRatio and minInitInlinerRatio parameters in the .vl file.
Play around with these values using the config file of our mini car model. Open the SimpleModelTracking scene that comes with the vlUnitPackage.unitypackage. After importing it in an empty project, switch to Assets in Unity's Project Panel, go to VisionLib/Examples/ModelTracking/Simple, open the scene and hit Play. In Game Mode press the Debug on button.
Observe the tracking states and focus on inlier ratio values: In the beginning, the values should be quiet low. As a rule of thumb, the minInitInlinerRatio value should always be higher than the minTrackingInlinerRatio value. A good value to start is setting these to 0.6 and 0.5. The inlier ratio depicts the ratio between found and not found parts of edges, so a high value means that a lot of the edges could be found in the actual camera image - making the tracking result more strict. Anyway, this will not mean that the tracking is working well. A single line of an edge of a model might result in a high inlier ratio but will not lead to a significant stable pose estimation. It is always good to have lines in different 3D directions in your camera image for determining a good camera pose.
In order to detect the generated edges, you can specify the tolerance in form of a length to be searched orthogonal to each line as a relative value. You can do this by setting the lineSearchLengthInitRelative and lineSearchLengthTrackingRelative parameters. With the debug image active (i.e. having set debugLevel to 1 or 2), you can see the debug visualization in the Unity, once you've pressed the Debug on button. From our experience the default values are fine. Anyway if you experience a poor init behavior or you expect a more tolerant one, you may increase the lineSearchLengthInitRelative value slightly.
The keyFrameDistance depicts the distance of each camera position in millimeters, when a new hypothesis might be generated for your tracking target in the background. During runtime, keyframes enable you to store more poses beyond your init pose. So, once you loose tracking, you can re-init from those key points. Setting this value very low might stress the performance of your application, while setting it too high may result in unreliable generation of your line model. If you can foresee the target to be observed at a low distance, you will need to lower the keyFrameDistance accordingly.
Usually tracking gets lost when turning away from the actual object that should be tracked. By enabling the extendibleTracking option, you can track beyond the actual view onto the model. VIO, visual inertial odometry, enables this. VisionLib uses platform-dependent available AR capabilities to do this: on HoloLens and iOS the internal tracking is used. On Android, Windows and macOS VisionLib's custom SLAM is used. In order to force the same behavior on ALL platforms, you can enable the disableExternalSLAM parameter. But we recommend working with the platform-capabilities. Only when working with image sequences as input source, use the disableExternalSLAM.
The table below gives a quick overview on the most relevant tracking configuration parameters for the modelTracker pipeline — a more detailed description of the config files is covered in this documentation at Configuration File Reference.
| Name | Description |
|---|---|
| tracker.type="modelTracker" | Do not touch this! This key is created in order to enable certain model tracker features |
| modelURI | An URI to your file, which holds the model to be tracked. The project_dir depicts the directory, where the .vl tracking configuration file is situated. fileFormats |
| metric | The metrics of one unit in the virtual space. |
| laplaceThreshold | Depicts the depth sensitivity in millimeters to be applied for the generation of the edge model from your 3D model. |
| normalThreshold | Depicts the value that describes the minimal curvature of two coherent faces in your 3D model, before generating an edge. Do not make this too small! |
| minInlierRatioInit | This is the minimum quality that is required to initialize when detecting the target. This value can be between 0-1 and makes assumptions about the tolerance of the algorithm to track. This might be a higher value than the minInlierRatioTracking, which describes the continuous threshold when tracking occurs. When doing tests with models, you should first begin changing these two values when starting and setting up a new tracking configuration. Good starting values for 3D objects are 0.6 and 0.5, while you will need to observe the quality feedback shown in the example. |
| minInlierRatioTracking | This depicts the threshold used for determining trackability of an object. |
| initPose | Is an optional object describing the initial pose to be used.. |
| keyFrameDistance | Is an optional value. Use this key in order to change the distance in mm of significant frames to be used for the reinitialization. Thus 50 mm means that every 5 cm reinitialization and tracking data will be updated. A low value means more computational power to be required, but might be required at small distances. |
| minNumOfCorrespondences | This key is optional and depicts the minimum number of correspondences |
| extendibleTracking | If you expect an object to be tracked along with extendible tracking this allows you moving away with your camera from the real object. It will help the tracking to be more stable during runtime. Please see the noted below on restrictions. |