Updating the VisionLib Package

Level: Basic

Updating to VisionLib SDK 2.0

The VisionLib SDK 2.0 Release comes with major interface changes. Please make a backup of your project before updating to the new release and follow all sections below for migration guidance.

Migration Guide - New UPM Package Structure

Affects: All projects

As the VisionLib SDK for Unity is now distributed as a UPM (Unity Package Manager) package, the import process and the location of the package in your project have changed. Therefore, in order to upgrade, the prior VisionLib version needs to be completely removed from your project. To do so, follow these steps:

• Remove all VisionLib files from the Plugins folder under Assets
  • If you haven't imported other plugins, simply delete the Plugins folder
  • Otherwise, search and delete all files called vlSDK.dll, opencv_ffmpeg320.dll, opencv_ffmpeg320_64.dll, VPVideoUEye.dll and libvlSDK.bundle. Also delete the folder Plugins/ iOS/ vlSDK.framework and under Plugins/ Android remove AndroidManifest.xml, arcore_client.aar, google_ar_optional.aar and vlsdk.aar.

Note: If you have started tracking since you opened your project, you might need to restart Unity to be able to delete the file vlSDK.dll

• Remove the VisionLib folder under Assets
  Note: Check if you have copied or modified one of the example scenes and move them to another folder if you want to keep them

• Finally, remove the Examples folder that is located in StreamingAssets / VisionLib
  Note: Be especially careful to not delete your custom tracking configurations (.vl files, model or poster tracking references) or your license file.

Now that the old version of the VisionLib SDK has been cleared, you can import the new version.

• Download the VisionLib.SDK-Unity.zip from the customer area and extract it locally
• Import the tarball file VisionLib.SDK-Unity.tgz via Unity's Package Manager

Note: UPM package contents are not saved as files in the Unity project (only a reference to the tarball file is stored in Packages/package-lock.json). Therefore the location of the tarball file is important and moving or deleting it will break the dependencies of your Unity project. To prevent that and also include the package in your version control, you can save it to the Packages folder of your project and import it from that location. This will result in a relative path in your package-lock.json. We recommend using git-lfs.

The remainder of this section can be skipped if you are only using the core functionality of the VisionLib SDK. If you have used assets from the example scenes (e.g. the ParameterSlider prefab from the ModelTrackingSetup scene), you might also want to import the example package.

• Import the VisionLib.SDK.Examples-Unity.unitypackage by dragging and dropping the package into the project view or select Assets > Import Package > Custom Package (do not use the Package Manager in this case)
• This will add example configurations to StreamingAssets/VisionLib/Examples and a VisionLib Examples folder directly below Assets which contains the example assets and scenes.

The HoloLens and ARFoundation (experimental) functionality are now distributed as separate extension packages that need to be imported in addition to the VisionLib SDK package. To add the according functionality to your project, follow these steps:

• Within the unpacked VisionLib.SDK-Unity.zip file, open the Extensions folder, which contains the HoloLens and ARFoundation extension packages.
• Import the tarball file VisionLib.SDK.HoloLens-Unity.tgz / VisionLib.SDK.ARFoundation-Unity.tgz via Unity's Package Manager
• Optional: Import the VisionLib.SDK.HoloLens.Examples-Unity.unitypackage / VisionLib.SDK.ARFoundation.Examples-Unity.unitypackage by dragging and dropping the package into the project view or select Assets > Import Package > Custom Package (do not use the Package Manager in this case)

Note: The VisionLib SDK -HoloLens and -ARFoundation package do have additional dependencies to Plugins from Unity. When adding the package to the Package Manager, those dependencies are installed automatically.

On HoloLens some additional steps are required to update the scene for using the XR Plug-in:

• Make sure that the Windows Mixed Reality option in ProjectSettings/XR Plug-in-Management is selected.

• Deactivate the VLHoloLensInitCamera GameObject
• Select GameObject/XR/Convert Main Camera to XR Rig
• Select the XR Rig GameObject and set the Camera Y Offset of the Camera Offset component to 0

Migration Guide - Notifications and LogLevel Updates

Affects: All projects

The handling of notifications and logs from VisionLib has been unified and simplified. The most important changes are:

• Notifications are not displayed in the UI by default, only if a NotificationDisplay component exists in the Canvas
• All notifications and console logs now behave according to a global log level, which can be adjusted by using the GeneralSettings component in your scene
• Initialization issues and VisionLib events are now logged automatically depending on the log level and don't need to have an according component anymore.

To update your existing Unity scene, follow these steps:

• Remove the obsolete VLNotifications GameObject under VLTracking or other occurrences of the components VLInitIssueNotifications and VLLogEventsBehaviour.
• If you have used the VLLogTrackingInfoBehaviour in your scene, remove that component and drag the VLTrackingInfoList prefab from VisionLib/UI/Templates/InfoList/Resources in the hierarchy under your canvas instead. Toggle its visibility by enabling and disabling the component or the whole GameObject.
• Finally, add the GeneralSettings component to the VLTracking GameObject. Change its log level to Info if you want to receive additional information during development, e.g. the selected cameras deviceID or tracking events. Usually, it is recommended to set the log level to Warning.

To additionally display the notifications in the UI (previous behaviour):

• Drag the VLNotifications prefab from VisionLib SDK/UI/Common/Prefabs in the hierarchy under the canvas in which you want the notifications to be displayed.

Migration Guide - Automatic View Point adjustment of AutoInit

Affects: Projects using AutoInit

The distance of view points in AdvancedWorkSpaces is no longer adjusted silently to let the object cover the full view.

Therefore, it might happen, that your viewpoints are too far away to gather recognizable information from them.

If this is the case, the following warning message will be logged when you start your Scene:

To fix this, change the source geometry of your WorkSpace in a way, that the tracked object covers a bigger part of the image, e.g. by reducing the radius of you sphere. Alternatively, use the new SimpleWorkSpace which will automatically set the view point distance according to the bounds of your model. For more details, see AutoInit with WorkSpaces (Beta).

Migration Guide - Missing WorkSpace References in TrackingAnchors

Affects: Projects using MultiModelTracking

Due to the introduction of the new SimpleWorkSpaces, this update will break the reference to WorkSpaces in TrackingAnchors of existing Scenes.

When you start your Scene like this, you will get a NullReferenceException.

To fix this problem, reassign the WorkSpaces to the corresponding TrackingAnchor in your MultiModelTracking Scene.

Migration Guide - Naming changes and new namespaces

Affects: Projects containing custom scripts that access VisionLib scripts

The VL prefix and the Behaviour or Wrapper postfix of many scripts have been removed. Several scripts have been renamed further to better fit their purpose.

Many scripts did just loose the prefix and or postfix and can be replaced like this: VLInitCameraBehaviour -> InitCamera. Find a list of namings that have changed further here:

Former Name	New Name
VLWorkerBehaviour	TrackingManager
VLWorkerReferenceBehaviour	TrackingManagerReference
VLModelTrackableBehaviour	TrackingModel
VLTrackedObjectBehaviour	TrackingStateProvider
CameraCalibrationExampleBehaviour	CameraCalibration
VLRenderBackgroundImageBehaviour	BackgroundImageRenderer
VLDetectScreenChangeBehaviour	ScreenOrientationObserver
VLCameraCalibration	CameraCalibrationResult
VLInitIssuesToEventsAdapter	IssuesToEventsAdapter
VLGeometry	WorkSpaceGeometry
VLWebCamBehaviour	WebCamInputDevice
VLUnitySdk	VLSDK
ChangeModelBehaviour	ModelSwitcher
SetTextBehaviour	InvariantCultureTextField
WindowsSpeechRecognitionBehaviour	WindowsSpeechInput
KeyDownEventDispatcher	KeyboardInput

Note: If you have accessed the reference to the TrackingManager (former VLWorkerBehaviour) through the TrackingManagerReference (former VLWorkerReferenceBehaviour), you will now need to access it via the member trackingManager (former workerBehaviour).

All VisionLib scripts are now also contained in namespaces. For details, see vlUnitySDK. The most common namespaces that you will interact with while developing are:

• Visometry.VisionLib.SDK.Core: Contains components that are used to add tracking functionality and to setup or interact with VisionLib's tracking.
• Visometry.VisionLib.SDK.Core.API: Contains classes that are used to directly communicate with the VisionLib engine, e.g. command classes.

To migrate your scripts to the new version, follow these steps:

• After importing the new SDK like described in the section Migration Guide - New UPM Package Structure, you will now see errors in Unity's console stating that certain class names do no exist.
• Double click the console log to jump to the according line in your IDE (integrated developer environment), e.g. Visual Studio.
• Now, you can make use of the Find and Replace all Occurrences function of your IDE to quickly exchange the namings within your project.

• After that, add the use of the according namespace to your script. Most IDEs will automatically suggest a quick action that applies the right namespace for the class that you want to use.

• Repeat the steps for each compiler error that is shown in Unity's console

Migration Guide - Removed boolean variable from several events

Affects: Projects containing custom scripts that implement a reaction to tracker events

Several events of the TrackingManager (former VLWorkerBehaviour) or the ModelTracker (former VLModelTrackerBehaviour_v1) had been the results of calling some commands in the behaviour. They returned booleans indicating the success of the command. Since it is usually only relevant to react to the event, if the command has been successful, the boolean argument has been removed from those functions. To adjust your classes accordingly, remove the boolean argument from all callback functions registering to one of the following events:

• TrackingManager.OnTrackerInitialized
• TrackingManager.OnTrackerStopped
• TrackingManager.OnTrackerRunning
• TrackingManager.OnTrackerRanOnce
• TrackingManager.OnTrackerPaused
• TrackingManager.OnTrackerResetSoft
• TrackingManager.OnTrackerResetHard
• ModelTracker.OnWriteInitData
• ModelTracker.OnReadInitData
• ModelTracker.OnResetInitData

E.g. you have a script which registered the following function for hiding the augmentation of the application to the OnTrackerStopped event:

public void Hide(bool success)
{
    if (success)
    {
        this.augmentation.SetActive(false);
    }
}

Since the boolean argument will be removed, the function can be reduced to

public void Hide()
{
    this.augmentation.SetActive(false);
}

To get notified about a failed initialization of the tracker, you can register to OnTrackerStopped, which will now get triggered in this case.

Migration Guide - Adjust root transform for dynamic model tracking on HoloLens

Affects: HoloLens projects using dynamic model tracking

When using dynamic model tracking in a HoloLens scene, you will now need to set its root transform (e.g. the SceneContent GameObject) in the inspector of the TrackingModel (former VLModelTrackableBehaviour) component, because it will no longer be set automatically from the content parameter of the HoloLensTracker component.

Migration Guide - Removed command classes in Unity

Affects: Projects containing custom scripts that communicate with the VisionLib engine directly by pushing commands

The command classes for communicating with VisionLib via Worker.PushCommand are now private. Each command namespace has been turned into a class containing several async static functions, one for each command. This change enforces the use of the new async Task functions. These functions provide a Task object which

• contains the correct result object as the result (void if the command has no result)
• can be awaited to receive the result and/or continue the execution after the command has been returned successfully. Therefore all instances of calling Worker.PushCommand(XXXCmd) should be replaced by await XXXAsync() of the same namespace. See also Using Commands and Tasks

If you pushed a command without caring for the result, the code should be replaced like this (to better understand the example, one should note that ResetHard has been moved from VLWorkerCommands to ModelTrackerCommands):

public void ResetTrackingHardOld()
{
    this.worker.PushCommand(
        new VLWorkerCommands.ResetHardCmd(),
        null,
        IntPtr.Zero);
}

public void ResetTrackingHardNew()
{
    TrackingManager.CatchCommandErrors(
        ModelTrackerCommands.ResetHardAsync(this.worker));
}

If you also registered a callback and reacted to the result, the code can be simplified even more. Here is an example of the former way to perform a ResetHard command and log an info message, if the tracking was reset correctly.

private GCHandle gcHandle;
private void Awake()
{
    this.gcHandle = GCHandle.Alloc(this);
}
 
private void OnDestroy()
{
    this.gcHandle.Free();
}
 
private void ResetTrackingHardHandler(string errorJson, string resultJson)
{
    if (errorJson)
    {
        Debug.LogError("Error occurred during call of 'resetHard'");
    }
    else
    {
        Debug.Log("Tracking was reset");
    }
}
 
[MonoPInvokeCallback(typeof(VLWorker.JsonStringCallback))]
private static void DispatchResetTrackingHardCallback(
    string errorJson, string resultJson, IntPtr clientData)
{
    try
    {
        GetInstance(clientData).ResetTrackingHardHandler(
            errorJson, resultJson);
    }
    catch (Exception e) // Catch all exceptions, because this is a callback 
                        // invoked from native code
    {
        Debug.LogException(e);
    }
}
private static VLWorker.JsonStringCallback dispatchResetTrackingHardCallbackDelegate =
    new VLWorker.JsonStringCallback(DispatchResetTrackingHardCallback);
 
public void ResetTrackingHard()
{
    this.worker.PushCommand(
        new VLWorkerCommands.ResetHardCmd(),
        dispatchResetTrackingHardCallbackDelegate,
        GCHandle.ToIntPtr(this.gcHandle));
}

Using the new Task function doesn't require registering a callback function and adding a GCHandle:

private async Task ResetTrackingHardAsync()
{
    await ModelTrackerCommands.ResetHardAsync(this.worker);
    Debug.Log("Tracking was reset");
}
 
public void ResetTrackingHard()
{
    TrackingManager.CatchCommandErrors(ResetTrackingHardAsync());
}

Migration Guide - Updating Camera Calibration

Affects: Projects containing custom scripts that interact with the VisionLib camera calibration process

You no longer have to keep track of the internal state of the camera calibration. Instead almost all commands are now available at any time. The result objects of the commands are now almost always empty. So parsing the result of the command callbacks is only necessary for the error string. The only exception is getResults. It returns the resulting camera calibration of the last optimization. If you didn't optimize, the callback of a getResults command will be an empty string. To receive the camera calibration you have to do the following steps:

• triggering the optimization via the optimize command
• Repeatedly call getResults (at most once per frame). If the callback returns an empty string, push another getResults command. Otherwise it should return a calibration object Adding a calibration to the calibration database can now be done at any time but the calibration object has to be provided as an argument (in general use the json object received by the getResults command). To abort the optimization process, the newly introduced cancel command should be used.

Updating to vlUnitySDK 20.10.1

Migration Guide - Deprecated the use of relative paths in URIs

The public base directory (which has been adjustable via the VLWorkerBehaviour) has been removed with this release. In relation to that, the use of relative paths when specifying the URI of the license file or tracking configuration has been deprecated. For users that already work with the TrackingConfiguration component, everything should be updated automatically without breaking existing Unity scenes.

To remove the deprecation warnings when using relative paths or to fix your paths when you had set a custom base directory before, please change them to refer to absolute locations. E.g. if you have been using a relative URI like Examples/ModelTracking/car.vl to start tracking, you can simply add the new streaming-assets-dir: scheme and the VisionLib/ folder in front of the uri: streaming-assets-dir:VisionLib/Examples/ModelTracking/car.vl. Like this, the path will be interpreted relative to Assets/StreamingAssets/VisionLib again. You can fix the license path of the VLWorkerBehaviour in the same way. The VLWorkerBehaviour will then look similar to this:

Alternatively, simply drag and drop the used tracking configuration, license and calibration files inside the TrackingConfiguration component in your scene and adjust their URIs if needed.

Updating to vlUnitySDK 20.3.1

This update contains some changes in the native interface. The application of these changes made some files obsolete. Additionally, some internal behaviour and default paths have changed.
To avoid complications when updating an existing project, please refer to the migration guides provided below.

Migration Guide - Obsolete Files

Please remove the following obsolete files (and their .meta files) from your project:

• Assets/VisionLib/Core/Scripts/Editor/VLDebugImageBehaviourEditor.cs
• Assets/VisionLib/Core/Scripts/Debug/VLDebugCameraBehaviour.cs
• Assets/VisionLib/Utilities/Prefabs/Debug/VLDebugCamera.prefab
• Assets/VisionLib/Utilities/UI/Templates/Notifications/Scripts/VLEventsNotification.cs

Migration Guide - Default License Path

To keep the places of files related to VisionLib at a minimum, we changed the default folder of the license from StreamingAssets to StreamingAssets/VisionLib.
If you never adjusted the license path in the VLWorkerBehaviour you might thus get an error that your license can not be found anymore.

• To fix this, move your license to StreamingAssets/VisionLib or adjust its path in the VLWorkerBehaviour (located in the scene hierarchy under VLTracking/VLCamera).

Migration Guide - VLDebugImage

The behaviour of the debug image has been changed internally. To be able to use it like before, do the following adjustments:

• Select the VLDebugImage in your canvas. On its component ImageStreamFilter, select DebugImage if you want to see the grey image or CameraImage if you want to see the camera image for debugging.
• To be able to use the grey image, please set the debugLevel to 1 inside your tracking configuration. Don't forget to remove this line before your final deploy to increase performance.

"tracker": {
    "type": "modelTracker",
    "version": 1,
    "parameters": {
        "modelURI": "project-dir:VLMiniCar.obj",
            ...
        "debugLevel": 1
    }

Migration Guide - Access of Image Sequence Parameters

Image sequence parameters can now only be accessed when actually using an image sequence. Trying to access those parameters when using a camera for tracking will print warnings to your console. To fix this:

• Remove the VLImageSequenceParameters prefab from your scene hierarchy (under VLTracking)
• If you are using the ImageSourcePanel from the ModelTrackingSetup scene in your canvas, also remove all of its children except Enabled:
• You will still be able to pause and resume an image sequence by using the remaining enabled checkbox

Updating to vlUnitySDK 19.11.1 - Remove of 180° Flip & New Folder Structure

This update contains important changes regarding the 180° flip of models and the general folder structure of the VisionLib Unity package. To avoid complications when updating an existing project, please refer to the migration guides provided below.

Migration Guide - 180° Flip

Since 19.11.1 we now handle Unity’s coordinate system handedness automatically, so you don’t need to rotate your tracking target 180 degrees around the y-axis anymore. However, in order to use this update in older projects, you can easily turn back to the legacy behaviour. So you don’t need to make changes in your scene setup:

• Activate legacy support:
  • Select the VLCamera GameObject under VLTracking in the hierarchy
  • Browse to the VLWorkerBehaviour component in the inspector
  • Choose the Manually option for the parameter Flip Coordinate System Handedness
  • A warning box will appear, that tells you that this option is deprecated. You can ignore it for now, but keep in mind that this option might be removed in the future.

• Switch to new automatic mode:
  • The Automatically option is activated by default in the component described above
  • Remove the 180° rotation around the y-axis on all scene objects that have been used as augmentations in your scene
  • Update the initial pose. Either move the VLInitCamera in your scene or edit the init pose that is set in your tracking configuration, depending on which one you have used before

Background to the 180° Flip:

The coordinate system used for tracking by VisionLib is right-handed, while the coordinate system used by Unity is left-handed. This caused several issues, when using the same model for tracking and visualization. For example when moving the VLInitCamera, the movement direction in Unity did not match the movement direction of VisionLib. Thus, to match visualization and tracking, models had to be rotated 180° around the Y-axis.

With this release, VisionLib converts the coordinate system automatically in Unity. You are no longer required to manually rotate your models around the Y-axis. This change only affects Unity development; it does not apply for development with VisionLib outside of Unity.

Migration Guide - New Folder Structure

The new folder structure is not a breaking change. If you leave your project as it is and just import the new VisionLib package, Unity will keep your previous folders and will not use the updated structure. Anyway, if new files are added from the VisionLib package, they will appear under the new structure, which leads to an inconsistent mix of folders. Also, some references to .vl-files might break in the example scenes.

Therefore, we recommend to remove all VisionLib assets from your project to make a clean update afterwards.

Note: Please make a backup of your project before deleting assets
Note: Don't delete the entire folders but the single files if you have used the folders for your own assets

• Delete _Scenes folder that is containing the example scenes:
  
• Delete GUI folder (and it's subfolders "LoadingScreen" and "Parameters"):
  
• Delete Materials folder containing three assets:
  
• Delete Models folder containing two assets and two folders:
  
• Delete Scripts folder containing 13 scripts and the subfolder "BuildingBricks":
  
• Delete the VisionLib folder:
  
• Delete the VisionLib folder inside the StreamingAssets
  
• If you own an experimental package, also remove all "Experimental" folders and their contents
• Now you can import the latest vlUnitySDK package into your project. All previous set references should be restored.