Using pre-calculated Line Models for Tracking

Level: Advanced

In certain cases it might be useful not to load the 3D model itself, but a pre-compiled (line) model for tracking instead.

This is useful when the developer wants to:

• hide model data from the customer
• avoid loading a big 3D model into memory, e.g. to save memory on a mobile device – this is particularly suitable for HoloLens, because the lack of memory in version one prevents using big objects for tracking (i.e. models >60k polygons).

That said, this of course has indications to and constraints for the model tracking, since the developer has to:

• clarify and define a certain region of action around the model in a prior preparation step
• save the generated data results into some lineModelData file and deploy it along with the application
• be aware, that in contrast to the full ones, pre-calculated line models are a subset of the original 3D data

Preparations

Working with pre-calculated line models feels little like recording only parts of an otherwise full 3D model you would use for tracking.

We try to describe the process as general as possible, so it should work similar on all platforms. As we are working with VisionLib tracking configurations, we assume you are familiar with these .vl files (more on tracking configurations, see › Configuration File Reference).

About Line Model Creation

In brief, when you use model tracking, VisionLib internally generates what we call model hypotheses in form of line models. These 3D models are then created viewpoint dependent. VisionLib allows you to cache these on-the-fly generated models. So whenever a new model pose is estimated for an actual tracking frame, a hypothesis might be generated or poked from the cache. Especially on convex objects these hypothesis do not alter too much, as you will notice later.

Creating Line Models from a 3D Model

To use these stripped line models, we first need to create a cached line model data set, which we can load in our application later on.

We do this by using at first our usual 3D model, which we will track against our (physical) tracking target, and "record" the automatically generated lines models while we go.

Once we have a valid pose while tracking, a suitable line model for the majority of occurring poses will be cached. We can record these positions while we move the camera to look around our target, making it possible to track it from these positions later on.

For this pre-processing step, you can either use a life camera stream of your tracking target, or a recorded image sequence of it.

To start, we need to setup a couple of parameters within the tracking configuration. Let's use the VisionLib paper-craft car example, which comes with the SDK.

Have a look at this tracking configuration: SimpleModelTrackingConfiguration.vl from the VisionLib.SDK.Examples-Unity.unitypackage.

We will remove the modelURI parameter and save it to a new file as CarWithoutModelURIConfiguration.vl.

In order to prepare the "recording" of a line model we will first need to increase the size of the cache depending on how many cached line models you want to save. A good value to start is 250 ("lineModelBufferSize":250). You can verify this in the tracking stats by observing the 'lineModels' tracking state attribute. The quantity of generated lines models is influenced by the keyFrameDistance parameter, which you might set to a very low value (e.g. 10 = 10 mm).

You can use two different commands to have VisionLib write line model data:

• the writeInitData command writes two files, init data files and line model files (e.g. myName_linemodels.binz). In the case when you use a MultiModelTracker, the names of the init data files are extended by an anchor name (e.g. myName_anchorName.binz and myName_anchorName_linemodels.binz)
• the writeLineModelData command writes only line models into a desired file

We provide a location by using URI schemas. This can be project-dir: (if writable) or local-storage-dir: or any location via file:.

A full string would look like this:

"writeLineModelData" : "true",
"lineModelDataURI" : "project-dir:myLineModelData.binz"

Creating & Writing pre-calculated Line Models in Unity

In Unity you can quickly create and write generated lines models using the ModelTrackingSetup scene from the from the VisionLib.SDK.Examples-Unity.unitypackage that is located in VisionLib Examples/ModelTracking/TrackingSetup. The writeInitData is a handy item already available and taken as example for now.

In case of using the writeInitData button inside this scene's gui, you will still need to set the attribute "lineModelPersistence":"true" in the tracking parameters.

Note: The aspect ratio influences generated line models. Later tracking results will be best by using more or less the same aspect ratio while recording, as you have on the target devices.

In Unity, a shortened way of using one single tracking configuration for recording & writing line models, and for later tracking, is using a query string URI to load the tracking configuration. You can use the prior created CarWithoutModelURIConfiguration.vl. Because during the pre-processing step we need the original model, we add it quickly by:

CarWithoutModelURIConfiguration.vl?tracker.parameters.lineModelPersistence=true&tracker.parameters.lineModelBufferSize=250&tracker.parameters.keyFrameDistance=10&tracker.parameters.modelURI=project-dir:VLMiniCar.obj

Using the Recorded Output

Let's assume we have generated and written the lineModelData, we can now use a parameterized query string URI as shortcut in Unity to have our tracking configuration load the generated line models:

CarWithoutModelURIConfiguration.vl?tracker.parameters.lineModelGeneration=false&tracker.parameters.lineModelDataURI=project-dir:myLineModelData.binz

To use our results for tracking, we need to change the following in the tracking configuration:

• remove the modelURI - in order to actively NOT load any 3D model
• add "lineModelGeneration":false - in order to prevent active generation of hypotheses.
• add "lineModelDataURI":"myLineModelData.binz" - in order to load the generated line models instead of the original 3D model

Just like usually, you should now be able to track your tracking target with your recently created line model instead of the original 3D file.