Tracking States in Unity

Level: Intermediate

Introduction

When using Model Tracking, showing the line model or giving other visual guidance is a good practice to help users find entry into your AR app. Basically, this is the matching of the camera to the init pose to get tracking started.

To do so, as a developer, you can create custom graphics, set e.g. the "showLineModel":"true", parameter in the tracking configuration, or present the superimposition visuals already at startup.

Especially the latter is often not desired. In contrary, you only want to show your augmentation visuals once you are successfully tracking your target. The same is true for visual guidance: it's nice to have a line model rendered to help users around, but only if tracking hasn't started yet or got lost.

In case of a visual guidance, there is a parameter you can use in your tracking configuration and even further parametrization, like the line model color:

"tracker" : {
  ...
  "showLineModelLost":true,
  "showLineModelLostColor": [255,255,255],
  ...
}

If you want to add a custom behaviour on tracking state changes for your application, you can choose between two options:

• to invoke a simple function via the public inspector interface of the VLTrackedObjectBehaviour or
• to listen to the appropriate events from the VLWorkerBehaviour using your own scripts.

Using Unity Events in the Inspector

VisionLib comes with the VLTrackedObjectBehaviour (You can see it in use within the PosterTracking example scene under VisionLib/Examples/PosterTracking). You can simply drag it on the GameObject, which you are using for superimposition.

Now you can see a list of empty events and add your desired behaviour as a reply to these events.

For example this could be the activation of a GameObject as soon as its tracked and its deactivation when tracking is lost:

• Hit the + under the Tracked Event() of the VlTrackedObjectBehaviour.
• Drag the GameObject you wish to activate/deactivate into the field with the label None(Object).
• From the dropdown menu, change from No Function to GameObject/ SetActive(bool).
• Check the box that appears below.
• Repeat the steps for the Lost Event(), but uncheck the box.

Now the component should look like this:

You can also invoke a custom function from your script that way, as long as it's public and takes no parameters or only basic ones like string, bool or int. Just drag the GameObject that has your script attached in the event field and choose your desired function.

The VLTrackedObjectBehaviour component gives the opportunity to react to the following events:

Event	Description
Tracked Event()	Event fired whenever the tracking state is "tracked".
Just Tracked Event()	Event fired once after the tracking state changed to "tracked".
Critical Event()	Event fired whenever the tracking state is "critical".
Just Critical Event()	Event fired once after the tracking state changed to "critical".
Lost Event()	Event fired whenever the tracking state is "lost".
Just Lost Event()	Event fired once after the tracking state changed to "lost".

Listening to the Tracking States Event in Script

If you need to call a more specific behaviour, you can also access the tracking states within your script by listening to VisionLib's tracking states event.

First, we need to get a reference of the VLWorkerBehaviour and register to its TrackingStates (which are stored in the state object of VLTrackingState.TrackingObject, see VLTrackingState Class Reference) :

...
// Get the VLWorkerBehaviour
public VLWorkerBehaviour workerBehaviour

void OnEnable() {

    //register a custom function to the tracking states
    VLWorkerBehaviour.OnTrackingStates += HandleTrackingStates; 

}

void OnDisable() {

    // unregister from event
    VLWorkerBehaviour.OnTrackingStates -= HandleTrackingStates; 

}

private void HandleTrackingStates(VLTrackingState state) {

}

...

Once you are successfully listening to tracking state changes, you can write your own behaviours within the newly defined HandleTrackingStates function. There are several states available which are listed in the table below.

As an example, we want to toggle the augmentation's visibility and hide it, whenever tracking is lost.

...

// container to store the previous tracking state
private string previousState = ""; 

public GameObject mySuperimposition;

private void HandleTrackingStates(VLTrackingState state) {

    string currentState = state.objects[0].state;
    //Debug.Log("Current tracking state: " + currentState);

    // the state is fired continuously, so only do something, 
    // when it actually changes

    if (currentState == previousState)
        return;

    if(currentState == "tracked") {

        // make augmentation visible
        mySuperimposition.SetActive(true);

    } else {

        mySuperimposition.SetActive(false);
    }

    // store tracking state before next iteration
    previousState = currentState;
}

...

At the moment there are three states to listen to, which are:

state	description
"tracked"	Tracking target is tracked successfully.
"critical"	Tracking target was tracked, but something disturbs tracking (e.g. motion blur or occlusion), making tracking unstable. If the tracking stays critical for too long, then the state might change to "lost".
"lost"	Tracking target was lost and could not be tracked anymore.

Tracking States with External SLAM (ARKit/ARCore)

You should typically listen to the "tracked" state to generate a stable superimposition. Anyway, we recommend listening to "critical", when using external SLAM poses with VisionLib. This is because you may want to continue to track and show your content, although the (original) model tracking target is not at display in the video image anymore. For example: you have a physical tracking target that you want to track and initialize with VisionLib's Model Tracking, but you also want to look around and move sideways in your virtual scene with the help of SLAM tracking. Combining both, VisionLib and ARKit/ARCore, makes this possible and the "critical" state enables you to have your superimposition visible until both, VisionLib and SLAM tracking, are truly "lost".

Distinguish the critical tracking state

The VLTrackingState contains a state and a quality value. If you want to distinguish the "critical" tracking state further, you need to check the combination of those two values, like seen below:

Tracking State	Tracking Quality	Description
lost	> 0	The object has not be found and the tracker is searching for a pose
tracked	> trackingInlierRatio	The tracker is successfully tracking the object
critical	> 0	The tracker is tracking the object only using the SLAM pose, the object is assumed to be in the field of view
critical	0	The tracker is tracking the object only using the SLAM pose, the object is assumed NOT to be in the field of view

Find more details on VisionLib with External SLAM here: External SLAM (ARKit/ARCore) Support.