Configuration File Reference

In order to configure and manage the tracking pipeline, VisionLib uses specific tracking configuration files. For instance, here you can set the reference to the tracking target (e.g. the file of a 3D model), control and tweak initial tracking parameters, specify an image source (which can be a live video or an image stream), or define an init pose for the model tracker. In essence, the configuration files define algorithms, parameters and sources and tell the VisionLib tracking engine what to do.

Structure

The tracking configuration file format has a .vl extension and a similar structure as the JSON text. Including additional support for comments using // and no support for unicode characters.

NOTE: If you plan to use .vl or .json files within your own code, beware that not all JSON parsers understand the use of comments (//).

Example of a very large Configuration File

{
    "type": "VisionLibTrackerConfig",
    "version": 1,
    "meta": {
        "name": "SomethingTracker",
        "description": "Custom tracker description from tracking author",
        "author": "<request@visionlib.com>"
    },
    "tracker": {
        "type": "modelTracker",
        "version": 1,
        "parameters": {
            "modelURI": "project_dir:model.ply",
            //"lineModelURI": "project_dir:lines.obj",
            //"lineModelOcclusionTest": true,
            "useColor": true,
            "metric": "mm",
            "initPose": {
                "type": "visionlib",
                // Translation stored as [t_x, t_y, t_z]
                "t": [1.075873932e-05, -2.691710303e-05, 186.6348404],
                // Rotation stored as quaternion [q_x, q_y, q_z, q_w]
                "q": [-0.71556605, -0.008785564998, 0.0007537285788, 0.6984894228]
            },
            "keyFrameDistance": 500,
            "laplaceThreshold": 5,
            "normalThreshold": 1000,
            "lineGradientThreshold": 40,
            "lineSearchLengthInit": 30,
            "lineSearchLengthTracking": 15,
            "minNumOfCorrespondences": 200,
            "minInlierRatioInit": 0.6,
            "minInlierRatioTracking": 0.5,
            "maxFramesFeaturePrediction": 30,
            "usePoseFiltering": false,
            "poseFilteringSmoothness": 0.3,
            "extendibleTracking": false,
            "debugLevel": 1
        }
    },
    "input": {
        "useImageSource": "camera0",
        "imageSources": [{
            "name": "camera0",
            "type": "camera",
            "data": {
                "unit": 0
            }
        }, {
            "name": "video0",
            "type": "video",
            "data": {
                "uri": "project_dir:Videos/video0.avi",
                "scale": 0.5,
                "calibration": {
                    "width": 640,
                    "height": 480,
                    "fx": 0.83512866676,
                    "fy": 1.1153914302,
                    "s": 0,
                    "cx": 0.49543359638,
                    "cy": 0.48924857004,
                    "k1": 0.16713074339,
                    "k2": -0.43314747547,
                    "k3": -0.0022563785648,
                    "k4": -0.00012730804896,
                    "k5": 0.32117836262
                }
            }
        }, {
            "name": "imageSequence0",
            "type": "imageSequence",
            "data": {
                "uri": "project_dir:Records/record_0/*.png",
                "device": "SurfacePro4",
                "initPose": {
                    "type": "metaio",
                    "uri": "project_dir:IninitialPose.xml"
                }
            }
        }]
    }
}

The following sections contain a detailed explanation of the example above.

Header

Tracking configurations contain a header with a file type and a corresponding file version:

{
    "type": "VisionLibTrackerConfig",
    "version": 1,
    // ... sections ...
}

This allows the tracking system to identify the used file format.

VisionLibTrackerConfig v1

"VisionLibTrackerConfig" files with version 1 contain three sections:

{
    // ...
    "meta": {
        // ... meta data ...
    },
    "tracker": {
        // ... tracker data ...
    },
    "input": {
        // .. input data ...
    }
}

Meta (optional)

The meta section is completely optional, but it can be used to describe the content of the file.

{
    // ...
    "meta": {
        "name": "SomethingTracker",
        "description": "Custom tracker description from tracking author",
        "author": "<request@igd.fraunhofer.de>"
    },
    // ...
}

Tracker (mandatory)

The tracker section is mandatory and describes the tracker. There will be different tracker types in the future, so the tracker section contains a header similar to the file header:

{
    // ...
    "tracker": {
        "type": "modelTracker",
        "version": 1,
        // ...
    }
    // ...
}

The type contains a string with the tracker type. The version contains a version number for the used tracker. This version number makes sure that existing tracking configurations still work the same after updating the VisionLib plugin.

The tracker section also contains a list with parameters:

{
    // ...
    "tracker": {
        // ... tracker header ...
        "parameters": {
            "modelURI": "project_dir:model.ply",
            //"lineModelURI": "project_dir:lines.obj",
            //"lineModelOcclusionTest": true,
            "useColor": true,
            "metric": "mm",
            "initPose": {
                "type": "visionlib",
                // Translation stored as [t_x, t_y, t_z]
                "t": [1.075873932e-05, -2.691710303e-05, 186.6348404],
                // Rotation stored as quaternion [q_x, q_y, q_z, q_w]
                "q": [-0.71556605, -0.008785564998, 0.0007537285788, 0.6984894228]
            },
            "keyFrameDistance": 500,
            "laplaceThreshold": 5,
            "normalThreshold": 1000,
            "lineGradientThreshold": 40,
            "lineSearchLengthInit": 30,
            "lineSearchLengthTracking": 15,
            "minNumOfCorrespondences": 200,
            "minInlierRatioInit": 0.6,
            "minInlierRatioTracking": 0.5,
            "maxFramesFeaturePrediction": 30,
            "usePoseFiltering": false,
            "poseFilteringSmoothness": 0.3,
            "extendibleTracking": false,
            "debugLevel": 1
        }
    }
    // ...

The available parameters depend on the tracker type and version:

• Model Tracker
• Poster Tracker
• Image Recorder
• Camera Calibration

Input (optional)

The input section is an optional section. It allows you to specify the source of the input images for the tracking:

{
    // ...
    "input": {
        "imageSources": [{
            "name": "camera0",
            "type": "camera",
            "data": {
                "deviceID": iPhone84_BackCamera,
                "position": "back",
                "unit": 1
            }
        }, {
            "name": "video0",
            "type": "video",
            "data": {
                "uri": "project_dir:Videos/video0.avi",
                "scale": 0.5,
                "calibration": {
                    "width": 640,
                    "height": 480,
                    "fx": 0.83512866676,
                    "fy": 1.1153914302,
                    "s": 0,
                    "cx": 0.49543359638,
                    "cy": 0.48924857004,
                    "k1": 0.16713074339,
                    "k2": -0.43314747547,
                    "k3": -0.0022563785648,
                    "k4": -0.00012730804896,
                    "k5": 0.32117836262
                }
            }
        }, {
            "name": "imageSequence0",
            "type": "imageSequence",
            "data": {
                "uri": "project_dir:Records/record_0/*.png",
                "device": "SurfacePro4",
                "initPose": {
                    "type": "metaio",
                    "uri": "project_dir:IninitialPose.xml"
                }
            }
        }],
        "useImageSource": "camera0"
    }
}

Each image source in the imageSources array has the following structure:

{
    "name": "<customName>",
    "type": "<type>", // (camera, video or imageSequence)
    "data": {
        // ...
    }
}

The available attributes for data depend on the type of the image source:

1. "camera"
   • deviceID
   • position
   • resolution
   • unit
   • calibration
   • device
   • initPose
   • undistort
2. "video"
   • uri (a lot of video formats are supported under non-UWP Windows using the provided ffmpeg dll)
   • scale
   • undistort
   • deviceID
   • calibration
   • device
   • initPose
3. "imageSequence"
   • uri
   • scale
   • undistort
   • deviceID
   • calibration
   • device
   • initPose
   • simulateMobileRecording

The example above defines three image sources:

1. "camera0": Tries to capture images from the camera with device ID "iPhone84_BackCamera" (A list of supported devices can be found at List of available devices). If this camera wasn't found, it will use a back-facing camera instead. If such a camera isn't available, the camera unit 1 will be used instead. deviceID, position and unit are all optional. If no matching camera was found it will use the first available camera.
2. "video0": Reads images from a video file. The camera calibration has been specified explicitly. The scale parameter has also been set. A value of 0.5 has the effect that the input image will get scaled down before processing. This improves the performance but reduces the precision.
3. "imageSequence0": Reads images from a sequence of image files. In this case, the camera calibration will be loaded from a list of predefined devices (SurfacePro4). While the other image sources use the initial pose specified inside the "tracker" section, this instance will use a custom initial pose. Please take a look at modelTracker for a more detailed description of the initPose property.

The useImageSource attribute contains the name of the image source which should be used. This simplifies switching between image sources.

If the input section is not defined, the first available camera will be used.

Setting up a higher resolution on iOS and Android devices

By default, VisionLib uses a quiet low resolution (960x540 on iPhone and 640x480 on iPad/Android devices). In order to change the resolution there is a more generic approach than just setting the resolution fixed. Please add the following lines to your tracking configuration in order to influence the resolution:

"input":{
  "imageSources": [{
          "name": "camera0",
          "type": "camera",
          "data": {
              "resolution": "auto"
          }
        }],
  "useImageSource":"camera0"
},

Possible values for resolution are:

• default - Would be the same as removing the input section at all. (will use the above described resolutions)
• auto - Will find the best resolution, which suites the performance of the device. An iPad Air 1/2 will yield in a 640x480 resolution, while devices shipping an A9 processor will use a 1024x768 resolution for example.
• low - Will force the lowest resolution
• medium - Will force not to use a low resolution
• high - Will use a very high resolution

In general, we recommend to set the resolution to auto to preserve the transportability character of tracking configuration files in future. The auto setting will be defaulted in more future versions and is preserved for unified backwards compatibility right now.

VisionLib tries to separate the image to be displayed and processed. Using high resolution camera images on devices. VisionLib might decide to downsample the image on certain processes. This leads to a more performant user experience on the costs of tracking quality. A high frame rate is usually a good compensator, on the other hand. In order to force VisionLib to NOT allow downsampling on the tracking pipeline, you can set the SmartDownsamplingDisabled parameter to true (by default false) in the input section. Exceptions that do not downsample by default are loading video or image sequences, and any type of camera calibration.

Schemes (optional)

General: URIs and Schemes in VisionLib

All I/O accesses within VisionLib are Uniform Resource Identifier (URI) driven. URIs in the context of VisionLib SDK describe the location of a resource, which can be a directory or a file at the end of the chain. The SDK strictly relies on the URI specification RFC3986.

A URI's format is `scheme:[//[user@]authority[:port]]/path[?arguments1&arguments...][#fragment].

You can take advantage of the URI parameters, when you're passing a tracking configuration as a parameter for the StartTracking function of the SDK. A very common way to reference files in the tracking configuration is to use schemes.

The VisionLib SDK defines various schemes to maintain portability of tracking configurations or configurations sets among the borders of varying systems.

Integrated Schemes

You can not overwrite already internally set schemes. Those are reserved for use with VisionLib. Those schemes are:

• project_dir:Points to the directory where the original .vlfile has been loaded from.
• local_storage_dir: Points to platform specific directory, where data can be stored persistently. These are:
  • Windows: Current users home directory (e.g. C:\\Users\\username).
  • UWP: The applications local directory in the user folder (C:\\Users\\USER_NAME\\AppData\\Local\\Packages\\PACKAGE_FAMILY_NAME\\LocalState).
  • HoloLens: The applications local directory (User Folder\\LocalAppData\\APPLICATION_NAME\\LocalState). You can access this via the Windows Device Portal (section File explorer).
  • MacOS: Current users document directory (e.g. `/Users/username/Documents).
  • iOS: The applications document directory. If you want to access the files through your iTunes or any other tool, you must set the Application supports iTunes file sharing (UIFileSharingEnabled) key in your application's .plist file.
  • Android: The application uses getExternalFilesDir(null). The directory location can differ by your phone settings, but it is application specific. Please have a look into the Android developer documentation for more information.
• file: is reserved to the filesystem access.
• http: is reserved for the access to sources outside the local storage using the http protocol. The usage of the http protocol for accessing files is currently not possible on UWP, which includes HoloLens. If you require this feature, please contact us.
• capture_dir: Points to the Videos/Captures directory in the Internal Storage of the HoloLens, which can be accessed via USB. You have to enable the Videos Library capablity for the application to grant access to this folder. On all other platforms, this file scheme points to local_storage_dir:/records.

Other schemes are to come in future releases. Support for more protocols (e.g. https and ftp) is planned.

Overwriting Schemes

You can define user defined schemes referring to resources in your context. You cannot redefine the system specific schemes mentioned above.

There are generally two ways of overwriting schemes using the SDK:

• define it in the tracking configuration or
• define it in the calling url of the tracking configuration

This allows max flexibility in reconfiguring and maintaining the portability.

Define schemes in the tracking configuration

This is an example of how to define a scheme within your tracking configuration.

{
   "type": "VisionLibTrackerConfig",
    "version": 1,
    "meta": {...},
    "tracker":{...},
    "schemes":{
        "my_image_location":"http://server/path",
        "image_dir1":"my_image_location:/ImageSequences/current/"
        "image_dir2":"project_dir:/ImageSequences/record_2/"
    },
    "input": {
        "useImageSource": "imageSequence",
        "imageSources": [{
            "name": "camera0",
            "type": "camera",
            "data": {
                "deviceID": "someDeviceIDGatheredFromGetInfo"
            }
        }, {
            "name": "imageSequence",
            "type": "imageSequence",
            "data": {
                "device": "SurfacePro3_rear",
                "uri": "image_dir1:/*.png"
            }
        }]}
}

This example defines three custom schemas. my_image_location aliases the URL defined, while image_dir1 uses this location to resolve a path. The image input will now be a sequence of images defined in the location:http://server/path/ImageSequences/current/*.png.

NOTE: An image sequence on a remote server might only be used for testing purposes and should be hosted on an Apache2 server with Indexes option turned on.

Passing schemes through the tracking configuration URI.

An other way to pass a scheme definition more generally, is through the calling URI. In our example above, the following scenario is possible:

http://mySever/path/myCoolTrackingConfig.vl?schemes.image_dir1=project_dir:/ImageSequences/record_2/;image_dir2.my_image_location:/other/dir

In this example there are two schemes passed, separated by ;. The SDK first creates aliases defined by the tracking configuration and overwrites the ones defined through the URI. This allows you to refer to `my_image_location' for instance (like described in the example tracking configuration above).

Overwriting configuration parameters via url parameters

The VisionLib SDK allows you to overwrite JSON parameters when initializing with a tracking configuration. The original parameters of the tracking configuration are always used unless they are overwritten in the calling URI of the main file.

Parameters are passed at the end of the URI like in the following example:

scheme:/pathtovl.vl?parameter1=value&parameter2=value

You can address the keys using a dot separated notation or change values inside arrays by using brackets [] (see examples below).

You can overwrite nearly all parameters. Some useful examples are given below:

Show line model and enable extendible tracking:

http://myserver.com/myConfig.vl?tracker.parameters.showLineModel=1&tracker.parameters.extendibleTracking=true

Open the tracking configuration from a web server, select a specified camera and use it:

http://myserver.com/myConfig.vl?input.imageSources[0].data.useDeviceID=macOS_HDWebcamC525&input.useImageSource=camera

Please check the tracking configuration file section for more information on the parameters.