Occluder for Model Tracking

Level: Intermediate

This article explains advantages of using some 3D models as occluder besides usual 3D model tracking references: such occluders enable you to tell VisionLib which parts it should ignore while creating the line model for tracking.

Reasons to use Occlusion Models

As a reminder: model tracking let's you use digital 3D model counterparts of the same physical objects as a tracking references to localize and track them. VisionLib takes these '3D model references' and generates a line model representation of it, which is then used for the actual tracking process.

As a rule of thumb, tracking results are best, whenever the 3D model matches its physical counterpart as accurately as possible. In some cases, however, there are some natural deviations.

A good example might be a car's tires and rims: because the rims are moving in reality, it is hard to almost impossible to estimate their rotation and integrate it into the 3D model upfront. During tracking, this might cause problems to an otherwise great fit, because those rims eventually deviate arbitrarily, while the overall car shape remains. While minimal deviations typically aren't a show stopper, depending e.g. on the viewing angle, others could cause the pose to jiggle, flip or break.

The figure above illustrates various strategies for overcoming these inconsistencies. One possibility could be to remove only the rims from the model. As a disadvantage, however, several structures will also be exposed that would otherwise remain hidden, creating useless and possibly misleading reference edge lines. And in case of removing the wheels completely, the informative external contours that would contribute to more stable tracking will also be missing after line model generation.

Instead of removing any parts of a 3D model at all, we can add a replacement geometry: in our case a disc in place of the rims. Subsequently, the generated line model will neither carry any lines of the rims anymore, nor will lines that lie behind the deleted geometry suddenly pop up. However, all 'desired' lines, like the tire contour, will be kept.

Therefore, when defining such geometry as occluder, non-desired geometry is rendered 'invisible' during the line model generation process.

You can either use the tracking configuration and set reference models as occluder there. Or, in case of Unity, with Mutable Model Tracking you can add the VLModelTrackableBehaviour component to 3D models in the Hierarchy and set the occluder: true property (see › Using Models from Unity Hierarchy for Tracking (Experimental)).

Tagging Models as Occluder

Let's have a look how to define models as occluder in the tracking configuration. There are basically two ways:

• load a second file via occlusionModelURI, when you use modelURI for a single 3D model as tracking reference, or
• via the occluder:true flag, when you use models to declare a list of tracking references.

Using occlusionModelURI

In the first case, the tracking configuration would look like this:

{
    "tracker": {
        "parameters": {
            "modelURI": "project-dir:car-shell.obj",
            "occlusionModelURI": "project-dir:rim-replacement.obj",
            "useColor": true,
            "metric": "m",
            "keyFrameDistance": 5,
            ...
            "showLineModel": true 
        }
    }
}

Using occluder:true with models

In the second case, our tracking configuration would look as outlined below. Please note, as all models are enabled, VisionLib will treat them as one tracking reference with the option to disable parts dynamically at runtime:

{
    "tracker": {
         "parameters": {
              "models":[
                {"name":"Piece000", "uri":"project-dir:/car-exterior.obj","enabled":"true" },
                {"name":"Piece001", "uri":"project-dir:/car-interior-.obj","enabled":"true" },
                {"name":"Piece002", "uri":"project-dir:/tires.obj","enabled":"true" },
                {"name":"Piece003", "uri":"project-dir:/rim-replacement.obj","enabled":"true", "occluder":"true" }
              ],
              ...
          }
      }
}

Occlusion for Tracking vs. Occlusion Geometry for Visualization Purposes

Please notice, that you should not confuse the topics of this article with occluder masks for rendering purposes, as we cover occlusion at the level of vision-based tracking here.

However, in Augmented Reality, occlusion is not only a topic for the tracking, but for rendering and visualization purposes as well. Because, whenever your tracking target is partly occluded in reality, you would want to mask your argumentations likewise, in order to create a AR-enhanced depiction that "makes sense" to the viewer's eyes. That is mainly, accounting for appropriate depth cues and visual consistency.

Say, we had some components inside the greater shell of a machine, which we wanted to appear "selected" and "highlighted" in AR: in order to create a credible AR view, we would need to take the shell and render it to act as a occlusion mask for these inlying components. Or, say we wanted to create an AR xray view to look inside a car's door as in the picture above. By using additional auxiliary geometry and shell parts (here the wing mirrors) around the door as occlusion masks, we obtain such a visual credibility.