documentation

Configuration File Reference

In order to configure and manage the tracking pipeline, VisionLib uses specific tracking configuration files. For instance, here you can set the reference to the tracking target (e.g. the file of a 3D model), control and tweak initial tracking parameters, specify an image source (which can be a live video or an image stream), or define an init pose for the model tracker. In essence, the configuration files define algorithms, parameters and sources and tell the VisionLib tracking engine what to do.

Structure

The tracking configuration file format has a .vl extension and a similar structure as the JSON text. Including additional support for comments using // and no support for unicode characters.

NOTE: If you plan to use .vl or .json files within your own code, beware that not all JSON parsers understand the use of comments (//).

Example of a very large Configuration File

{
"type": "VisionLibTrackerConfig",
"version": 1,
"tracker": {
"type": "modelTracker",
"version": 1,
"parameters": {
"modelURI": "project-dir:model.ply",
//"lineModelURI": "project-dir:lines.obj",
"useColor": true,
"metric": "mm",
"initPose": {
"type": "visionlib",
// Translation stored as [t_x, t_y, t_z]
"t": [1.075873932e-05, -2.691710303e-05, 186.6348404],
// Rotation stored as quaternion [q_x, q_y, q_z, q_w]
"q": [-0.71556605, -0.008785564998, 0.0007537285788, 0.6984894228]
},
"keyFrameDistance": 500,
"laplaceThreshold": 5,
"normalThreshold": 1000,
"lineGradientThreshold": 40,
"lineSearchLengthInit": 30,
"lineSearchLengthTracking": 15,
"minNumOfCorrespondences": 200,
"minInlierRatioInit": 0.6,
"minInlierRatioTracking": 0.5,
"maxFramesFeaturePrediction": 30,
"usePoseFiltering": false,
"poseFilteringSmoothness": 0.3,
"extendibleTracking": false,
"debugLevel": 1
}
},
"input": {
"useImageSource": "camera0",
"imageSources": [{
"name": "camera0",
"type": "camera",
"data": {
"unit": 0
}
}, {
"name": "video0",
"type": "video",
"data": {
"uri": "project-dir:Videos/video0.avi",
"calibration": {
"width": 640,
"height": 480,
"fx": 0.83512866676,
"fy": 1.1153914302,
"s": 0,
"cx": 0.49543359638,
"cy": 0.48924857004,
"k1": 0.16713074339,
"k2": -0.43314747547,
"k3": -0.0022563785648,
"k4": -0.00012730804896,
"k5": 0.32117836262
}
}
}, {
"name": "imageSequence0",
"type": "imageSequence",
"data": {
"uri": "project-dir:Records/record_0/*.png",
"deviceID": "SurfacePro4",
"initPose": {
"type": "visionlib",
"uri": "project-dir:IninitialPose.xml"
}
}
}]
}
}

The following sections contain a detailed explanation of the example above.

Header

Tracking configurations contain a header with a file type and a corresponding file version:

{
"type": "VisionLibTrackerConfig",
"version": 1,
// ... sections ...
}

This allows the tracking system to identify the used file format.

VisionLibTrackerConfig v1

"VisionLibTrackerConfig" files with version 1 contain three sections:

{
// ...
"tracker": {
// ... tracker data ...
},
"input": {
// .. input data ...
}
}

Tracker (mandatory)

The tracker section is mandatory and describes the tracker. There will be different tracker types in the future, so the tracker section contains a header similar to the file header:

{
// ...
"tracker": {
"type": "modelTracker",
"version": 1,
// ...
}
// ...
}

The type contains a string with the tracker type. The version contains a version number for the used tracker. This version number makes sure that existing tracking configurations still work the same after updating the VisionLib plugin.

The tracker section also contains a list with parameters:

{
// ...
"tracker": {
// ... tracker header ...
"parameters": {
"modelURI": "project-dir:model.ply",
//"lineModelURI": "project-dir:lines.obj",
"useColor": true,
"metric": "mm",
"initPose": {
"type": "visionlib",
// Translation stored as [t_x, t_y, t_z]
"t": [1.075873932e-05, -2.691710303e-05, 186.6348404],
// Rotation stored as quaternion [q_x, q_y, q_z, q_w]
"q": [-0.71556605, -0.008785564998, 0.0007537285788, 0.6984894228]
},
"keyFrameDistance": 500,
"laplaceThreshold": 5,
"normalThreshold": 1000,
"lineGradientThreshold": 40,
"lineSearchLengthInit": 30,
"lineSearchLengthTracking": 15,
"minNumOfCorrespondences": 200,
"minInlierRatioInit": 0.6,
"minInlierRatioTracking": 0.5,
"maxFramesFeaturePrediction": 30,
"usePoseFiltering": false,
"poseFilteringSmoothness": 0.3,
"extendibleTracking": false,
"debugLevel": 1
}
}
// ...

The available parameters depend on the tracker type and version:

Input (optional)

The input section is an optional section. It allows you to specify the source of the input images for the tracking:

{
// ...
"input": {
"imageSources": [{
"name": "camera0",
"type": "camera",
"data": {
"deviceID": "iPhone84_BackCamera",
"position": "back",
"unit": 1
}
}, {
"name": "video0",
"type": "video",
"data": {
"uri": "project-dir:Videos/video0.avi",
"undistort": true,
"calibration": {
"width": 640,
"height": 480,
"fx": 0.83512866676,
"fy": 1.1153914302,
"s": 0,
"cx": 0.49543359638,
"cy": 0.48924857004,
"k1": 0.16713074339,
"k2": -0.43314747547,
"k3": -0.0022563785648,
"k4": -0.00012730804896,
"k5": 0.32117836262
}
}
}, {
"name": "imageSequence0",
"type": "imageSequence",
"data": {
"uri": "project-dir:Records/record_0/*.png",
"deviceID": "SurfacePro4",
"initPose": {
"type": "visionlib",
"uri": "project-dir:InitialPose.xml"
}
}
}],
"useImageSource": "camera0"
}
}

The example above defines three image sources:

  1. "camera0": Tries to capture images from the camera with device ID "iPhone84_BackCamera" (A list of supported devices can be found at List of devices with legacy camera support). If this camera wasn't found, it will use a back-facing camera instead. If such a camera isn't available, the camera unit 1 will be used instead. deviceID, position and unit are all optional. If no matching camera was found, the first available camera will be used.
  2. "video0": Reads images from a video file. The camera calibration has been specified explicitly. It will undistort the image using the distortion parameters k1 - k5.
  3. "imageSequence0": Reads images from a sequence of image files. In this case, the camera calibration will be loaded from a list of predefined devices. This is done by setting the deviceID to the value of the recording device (in this case SurfacePro4). While the other image sources use the initial pose specified inside the "tracker" section, this instance will use a custom initial pose.

The useImageSource attribute contains the name of the image source which should be used. This simplifies switching between image sources.

If the input section is not defined, the first available camera will be used.

Each image source in the imageSources array has the following structure:

{
"name": "<customName>",
"type": "<type>", // (camera, inject, imageSequence, uEye or video)
"data": {
// ...
}
}

The available attributes for data depend on the type of the image source:

Parameter Type Default Value camera inject imageSequence uEye

video

calibration Camera Calibration yes yes yes yes yes

Specifies the calibration for the given image source. It contains the current resolution and the focal length in x and y direction. It can also contain the parameter for undistortion and extrinsicData for this camera.

calibrationURI string yes yes yes yes yes

URI of a file containing the camera calibration parameter for the current camera.

device string yes yes yes no yes

DEPRECATED. Please use deviceID instead.

deviceID string yes yes yes yes yes

Can be used to prefer a specific camera by a name. The camera with the correct deviceID will be preferred over position and unit. Additionally to preferring a camera, the deviceID will be used to load the according calibration. This is especially relevant for image sequences and videos.

ignoreImageSequenceIntrinsics bool false no yes no no no

If this parameter is set to true, VisionLib ignores all IntrinsicData.xml files specified in the image sequence. The camera calibration should be specified elsewhere.

initPose extrinsicData yes yes yes no yes

Can be used to specify an initPose for an input source. This is useful for testing different input sources inside the same tracking configuration. It will only be set in the tracker, if the tracker does not specify an initPose itself. Please take a look at modelTracker for a more detailed description of the initPose property.

position string yes no no no no

Can be used to prefer a specific camera by its position on the device. Possible values are "front" and "back". This has a lower priority than deviceID but is preferred over unit.

resolution string "default" yes no no no no

Requests a specific resolution for the camera. A detailed description can be found at the "Setting up a higher resolution" section.

scale float 1.0 no yes no no no

DEPRECATED: Use trackingImageSize instead.

simulateMobileRecording bool false no yes no no yes

Should be set to true if one uses an image sequence or video recorded on a mobile device using SmartDownsamplingDisabled. This way you can display the large image that was recorded but use a scaled down version for tracking. See also SmartDownsamplingDisabled and trackingImageSize.

SmartDownsamplingDisabled bool false yes no yes yes no

Disables the automatic downsampling algorithm. If this parameter is true, the trackingImageSize will be activated. For further details see the "Prevent automatic rescaling" subsection.

trackingImageSize int 640 yes yes yes yes yes

The image will be downsampled for tracking to improve performance. This parameter specifies the maximal size of the image for tracking. If the used image is smaller than the specified trackingImageSize, it will not be downsampled. A higher resolution will lead to a better precision but reduces the performance.

undistort bool false yes yes yes yes yes

The undistort parameter k1 - k5 of the Camera Calibration will be used to remove the distortion of the input image. The parameters are equivalent to the first five distCoeffs used by OpenCV (k1, k2, p1, p2 and k3 in that order).

unit int 0 yes no no no no

Index of the camera, which should be used for image capturing. This has a lower priority than deviceID and position.

uri string no yes no no yes
URI specifying the source of the image sequence or video. For videos, a lot of formats are supported under non-UWP Windows using the provided ffmpeg dll.

Setting up a higher resolution

This feature is currently only available on iOS, Android and Windows.

By default, VisionLib uses a quiet low resolution (960x540 on iPhone and 640x480 on iPad/Android devices). In order to change the resolution, we introduced the resolution field in the data section of the camera image source. Please add the following lines to your tracking configuration in order to influence the resolution:

"input":{
"imageSources": [{
"name": "camera0",
"type": "camera",
"data": {
"resolution": "auto"
}
}],
"useImageSource":"camera0"
},

Possible values for resolution are:

  • default - Would be the same as not setting the resolution at all (will use the above described resolutions).
  • auto - Will find the resolution, which suits the performance of the device best. For example, an iPad Air 1/2 will yield in a 640x480 resolution, while devices shipping an A9 processor will use a 1024x768 resolution.
  • low - Will force the lowest resolution
  • medium - Will force not to use a low resolution
  • high - Will use a very high resolution

In general, we recommend to set the resolution to auto to preserve the transportability character of tracking configuration files in future.

Note: The auto setting will be defaulted in more future versions and is preserved for unified backwards compatibility right now.

Prevent automatic rescaling

VisionLib tries to separate the image to be displayed and processed. Using high resolution camera images on devices. VisionLib might decide to downsample the image on certain processes. This leads to a more performant user experience on the costs of tracking quality. A high frame rate is usually a good compensator, on the other hand. In order to force VisionLib to NOT allow downsampling on the tracking pipeline, you can set the SmartDownsamplingDisabled parameter to true (by default false) in the input section. Exceptions that do not downsample by default are loading video or image sequences, and any type of camera calibration.

Schemes (optional)

You can use several integrated schemes when passing URIs to VisionLib. E.g. a common way to reference files in the tracking configuration using schemes is "modelURI": "project-dir:VLMiniCar.obj".

Have a look at File Access for a list of available schemes and more information on the use of URIs with VisionLib.

Overwriting configuration parameters via URI parameters

The VisionLib SDK allows you to overwrite JSON parameters when initializing with a tracking configuration.

Parameters are passed at the end of the URI like in the following example:

scheme:/pathToVL.vl?parameter1=value&parameter2=value

You can address the keys using a dot separated notation or change values inside arrays by using brackets [] (see examples below).

You can overwrite nearly all parameters. Some useful examples are given below:

Show line model and enable extendible tracking:

http://myserver.com/myConfig.vl?tracker.parameters.showLineModel=1&tracker.parameters.extendibleTracking=true

Open the tracking configuration from a web server, select a specified camera and use it:

http://myserver.com/myConfig.vl?input.imageSources[0].data.useDeviceID=macOS_HDWebcamC525&input.useImageSource=camera

Please check the tracking configuration file section for more information on the parameters.