In order to configure and manage the tracking pipeline, VisionLib uses specific tracking configuration files. For instance, here you can set the reference to the tracking target (e.g. the file of a 3D model), control and tweak initial tracking parameters, specify an image source (which can be a live video or an image stream), or define an init pose for the model tracker. In essence, the configuration files define algorithms, parameters and sources and tell the VisionLib tracking engine what to do.

Structure

The tracking configuration file format has a .vl extension and a similar structure as the JSON text. Including additional support for comments using // and no support for unicode characters.

NOTE: If you plan to use .vl or .json files within your own code, beware that not all JSON parsers understand the use of comments (//).

Example of a very large Configuration File

{
    "type": "VisionLibTrackerConfig",
    "version": 1,
    "tracker": {
        "type": "modelTracker",
        "version": 1,
        "parameters": {
            "modelURI": "project-dir:model.ply",
            //"lineModelURI": "project-dir:lines.obj",
            "useColor": true,
            "metric": "mm",
            "initPose": {
                "type": "visionlib",
                // Translation stored as [t_x, t_y, t_z]
                "t": [1.075873932e-05, -2.691710303e-05, 186.6348404],
                // Rotation stored as quaternion [r_x, r_y, r_z, r_w]
                "r": [-0.71556605, -0.008785564998, 0.0007537285788, 0.6984894228]
            },
            "keyFrameDistance": 500,
            "laplaceThreshold": 5,
            "normalThreshold": 1000,
            "lineGradientThreshold": 40,
            "lineSearchLengthInit": 30,
            "lineSearchLengthTracking": 15,
            "minNumOfCorrespondences": 200,
            "minInitQuality": 0.6,
            "minTrackingQuality": 0.5,
            "maxFramesFeaturePrediction": 30,
            "poseFilteringSmoothness": 0.0,
            "extendibleTracking": false,
            "debugLevel": 1
        }
    },
    "input": {
        "useImageSource": "camera0",
        "imageSources": [{
            "name": "camera0",
            "type": "camera",
            "data": {
                "unit": 0
            }
        }, {
            "name": "video0",
            "type": "video",
            "data": {
                "uri": "project-dir:Videos/video0.avi",
                "calibration": {
                    "width": 640,
                    "height": 480,
                    "fx": 0.83512866676,
                    "fy": 1.1153914302,
                    "s": 0,
                    "cx": 0.49543359638,
                    "cy": 0.48924857004,
                    "k1": 0.16713074339,
                    "k2": -0.43314747547,
                    "k3": -0.0022563785648,
                    "k4": -0.00012730804896,
                    "k5": 0.32117836262
                }
            }
        }, {
            "name": "imageSequence0",
            "type": "imageSequence",
            "data": {
                "uri": "project-dir:Records/record_0/*.png",
                "deviceID": "SurfacePro4",
                "initPose": {
                    "type": "visionlib",
                    "uri": "project-dir:IninitialPose.xml"
                }
            }
        }]
    }
}

The following sections contain a detailed explanation of the example above.

Header

Tracking configurations contain a header with a file type and a corresponding file version:

{
    "type": "VisionLibTrackerConfig",
    "version": 1,
    // ... sections ...
}

This allows the tracking system to identify the used file format.

VisionLibTrackerConfig v1

"VisionLibTrackerConfig" files with version 1 contain three sections:

{
    // ...
    "tracker": {
        // ... tracker data ...
    },
    "input": {
        // .. input data ...
    }
}

Tracker (mandatory)

The tracker section is mandatory and describes the tracker. There will be different tracker types in the future, so the tracker section contains a header similar to the file header:

{
    // ...
    "tracker": {
        "type": "modelTracker",
        "version": 1,
        // ...
    }
    // ...
}

The type contains a string with the tracker type. The version contains a version number for the used tracker. This version number makes sure that existing tracking configurations still work the same after updating the VisionLib plugin.

The tracker section also contains a list with parameters:

{
    // ...
    "tracker": {
        // ... tracker header ...
        "parameters": {
            "modelURI": "project-dir:model.ply",
            //"lineModelURI": "project-dir:lines.obj",
            "useColor": true,
            "metric": "mm",
            "initPose": {
                "type": "visionlib",
                // Translation stored as [t_x, t_y, t_z]
                "t": [1.075873932e-05, -2.691710303e-05, 186.6348404],
                // Rotation stored as quaternion [r_x, r_y, r_z, r_w]
                "r": [-0.71556605, -0.008785564998, 0.0007537285788, 0.6984894228]
            },
            "keyFrameDistance": 500,
            "laplaceThreshold": 5,
            "normalThreshold": 1000,
            "lineGradientThreshold": 40,
            "lineSearchLengthInit": 30,
            "lineSearchLengthTracking": 15,
            "minNumOfCorrespondences": 200,
            "minInitQuality": 0.6,
            "minTrackingQuality": 0.5,
            "maxFramesFeaturePrediction": 30,
            "poseFilteringSmoothness": 0.3,
            "extendibleTracking": false,
            "debugLevel": 1
        }
    }
    // ...

The available parameters depend on the tracker type and version:

Input (optional)

The input section is an optional section. It allows you to specify the source of the input images for the tracking:

{
    // ...
    "input": {
        "imageSources": [{
            "name": "camera0",
            "type": "camera",
            "data": {
                "deviceID": "iPhone84_BackCamera",
                "position": "back",
                "unit": 1
            }
        }, {
            "name": "video0",
            "type": "video",
            "data": {
                "uri": "project-dir:Videos/video0.avi",
                "undistort": true,
                "calibration": {
                    "width": 640,
                    "height": 480,
                    "fx": 0.83512866676,
                    "fy": 1.1153914302,
                    "s": 0,
                    "cx": 0.49543359638,
                    "cy": 0.48924857004,
                    "k1": 0.16713074339,
                    "k2": -0.43314747547,
                    "k3": -0.0022563785648,
                    "k4": -0.00012730804896,
                    "k5": 0.32117836262
                }
            }
        }, {
            "name": "imageSequence0",
            "type": "imageSequence",
            "data": {
                "uri": "project-dir:Records/record_0/*.png",
                "deviceID": "SurfacePro4",
                "initPose": {
                    "type": "visionlib",
                    "uri": "project-dir:InitialPose.xml"
                }
            }
        }],
        "useImageSource": "camera0"
    }
}

The example above defines three image sources:

"camera0": Tries to capture images from the camera with device ID "iPhone84_BackCamera" (A list of supported devices can be found at List of devices with legacy camera support). If this camera wasn't found, it will use a back-facing camera instead. If such a camera isn't available, the camera unit 1 will be used instead. deviceID, position and unit are all optional. If no matching camera was found, the first available camera will be used.
"video0": Reads images from a video file. The camera calibration has been specified explicitly. It will undistort the image using the distortion parameters k1 - k5.
"imageSequence0": Reads images from a sequence of image files. In this case, the camera calibration will be loaded from a list of predefined devices. This is done by setting the deviceID to the value of the recording device (in this case SurfacePro4). While the other image sources use the initial pose specified inside the "tracker" section, this instance will use a custom initial pose.

The useImageSource attribute contains the name of the image source which should be used. This simplifies switching between image sources.

If the input section is not defined, the first available camera will be used.

Each image source in the imageSources array has the following structure:

{
    "name": "<customName>",
    "type": "<type>", // (camera, inject, imageSequence, uEye or video)
    "data": {
        // ...
    }
}

The available attributes for data depend on the type of the image source:

Parameter	Type	Default Value	camera	inject	imageSequence	uEye	video
calibration	Camera Calibration		yes	yes	yes	yes	yes
	Specifies the calibration for the given image source. It contains the current resolution and the focal length in x and y direction. It can also contain the parameter for undistortion and extrinsicData for this camera.
calibrationURI	string		yes	yes	yes	yes	yes
	URI of a file containing the camera calibration parameter for the current camera.
device	string		yes	yes	yes	no	yes
	DEPRECATED. Please use `deviceID` instead.
deviceID	string		yes	yes	yes	yes	yes
	Can be used to prefer a specific camera by a name. The camera with the correct `deviceID` will be preferred over `position` and `unit`. Additionally to preferring a camera, the `deviceID` will be used to load the according calibration. This is especially relevant for image sequences and videos.
ignoreImageSequenceIntrinsics	bool	false	no	yes	no	no	no
	If this parameter is set to `true`, VisionLib ignores all `IntrinsicData.xml` files specified in the image sequence. The camera calibration should be specified elsewhere.
initPose	extrinsicData		yes	yes	yes	no	yes
	Can be used to specify an init pose for an input source. This is useful for testing different input sources inside the same tracking configuration. It will only be set in the tracker, if the tracker does not specify an init pose itself. Please take a look at modelTracker for a more detailed description of the init pose property.
position	string		yes	no	no	no	no
	Can be used to prefer a specific camera by its position on the device. Possible values are "front" and "back". This has a lower priority than `deviceID` but is preferred over `unit`.
resolution	string	"default"	yes	no	no	no	no
	Requests a specific resolution for the camera. A detailed description can be found at the "Setting up a higher resolution" section.
scale	float	1.0	no	yes	no	no	no
	DEPRECATED: Use `trackingImageSize` instead.
simulateMobileRecording	bool	false	no	yes	no	no	yes
	Should be set to `true` if one uses an image sequence or video recorded on a mobile device using `SmartDownsamplingDisabled`. This way you can display the large image that was recorded but use a scaled down version for tracking. See also `SmartDownsamplingDisabled` and `trackingImageSize`.
SmartDownsamplingDisabled	bool	false	yes	no	yes	yes	no
	Disables the automatic downsampling algorithm. If this parameter is `true`, the `trackingImageSize` will be activated. For further details see the "Prevent automatic rescaling" subsection.
trackingImageSize	int	640	yes	yes	yes	yes	yes
	The image will be downsampled for tracking to improve performance. This parameter specifies the maximal size of the image for tracking. If the used image is smaller than the specified `trackingImageSize`, it will not be downsampled. A higher resolution will lead to a better precision but reduces the performance.
undistort	bool	false	yes	yes	yes	yes	yes
	The undistort parameter k1 - k5 of the Camera Calibration will be used to remove the distortion of the input image. The parameters are equivalent to the first five `distCoeffs` used by OpenCV (k1, k2, p1, p2 and k3 in that order).
unit	int	0	yes	no	no	no	no
	Index of the camera, which should be used for image capturing. This has a lower priority than `deviceID` and `position`.
uri	string		no	yes	no	no	yes
	URI specifying the source of the image sequence or video. For videos, a lot of formats are supported under non-UWP Windows using the provided ffmpeg dll.
useColor	bool	true	yes	yes	no	yes	no
	This option allows VisionLib to recognize edges between pixels with the same intensity value, if they differ in hue or saturation. It can increase the tracking quality, while having the drawback of needing more resources and processing power. Turn this setting on, if the object can not be distinguished from the background just by intensity or if internal edges should be used, that are easily separated by their hue.

Setting up a higher resolution

This feature is currently only available on iOS, Android and Windows.

By default, VisionLib uses a quiet low resolution (960x540 on iPhone and 640x480 on iPad/Android devices). In order to change the resolution, we introduced the resolution field in the data section of the camera image source. Please add the following lines to your tracking configuration in order to influence the resolution:

"input":{
  "imageSources": [{
          "name": "camera0",
          "type": "camera",
          "data": {
              "resolution": "auto"
          }
        }],
  "useImageSource":"camera0"
},

Possible values for resolution are:

default - Would be the same as not setting the resolution at all (will use the above described resolutions).
auto - Will find the resolution, which suits the performance of the device best. For example, an iPad Air 1/2 will yield in a 640x480 resolution, while devices shipping an A9 processor will use a 1024x768 resolution.
low - Will force the lowest resolution
medium - Will force not to use a low resolution
high - Will use a very high resolution

In general, we recommend to set the resolution to auto to preserve the transportability character of tracking configuration files in future.

Note: The auto setting will be defaulted in more future versions and is preserved for unified backwards compatibility right now.

Prevent automatic rescaling

VisionLib tries to separate the image to be displayed and processed. Using high resolution camera images on devices. VisionLib might decide to downsample the image on certain processes. This leads to a more performant user experience on the costs of tracking quality. A high frame rate is usually a good compensator, on the other hand. In order to force VisionLib to NOT allow downsampling on the tracking pipeline, you can set the SmartDownsamplingDisabled parameter to true (by default false) in the input section. Exceptions that do not downsample by default are loading video or image sequences, and any type of camera calibration.

Schemes (optional)

You can use several integrated schemes when passing URIs to VisionLib. E.g. a common way to reference files in the tracking configuration using schemes is "modelURI": "project-dir:VLMiniCar.obj".

Have a look at File Access for a list of available schemes and more information on the use of URIs with VisionLib.

Overwriting configuration parameters via URI parameters

The VisionLib SDK allows you to overwrite JSON parameters when initializing with a tracking configuration.

Parameters are passed at the end of the URI like in the following example:

scheme:/pathToVL.vl?parameter1=value&parameter2=value

You can address the keys using a dot separated notation or change values inside arrays by using brackets [] (see examples below).

You can overwrite nearly all parameters. Some useful examples are given below:

Show line model and enable extendible tracking:

http://myserver.com/myConfig.vl?tracker.parameters.showLineModel=1&tracker.parameters.extendibleTracking=true

Open the tracking configuration from a web server, select a specified camera and use it:

http://myserver.com/myConfig.vl?input.imageSources[0].data.useDeviceID=macOS_HDWebcamC525&input.useImageSource=camera

Please check the tracking configuration file section for more information on the parameters.

documentation