ARDK 2.2.0 Release Notes 2022-08-11

What’s New

  • [Experimental Features] In order to speed up our feedback loop ARDK provides experimental features. These features are available for testing and feedback but are in early stages of development, subject to change, and may not reflect final designs or APIs. Experimental features should not be used in production products and are unsupported. We want interested developers to use these experimental features and provide feedback. If you have feedback on an experimental feature, post your feedback in the Lightship Developer Community

  • [Contextual Awareness] “person” is now available as a semantic segmentation channel. Person segmentation detects people, including body parts, hair, and clothes worn.

  • [Contextual Awareness - Experimental] Added experimental implementation of hand tracking using palm detection. Enable the AR Session and palm detection in your session configuration to receive events when palms are detected, undetected, and moved. A rectangle will be provided to locate the palm in the camera feed in 2d coordinate space.

  • [Developer Tools - Experimental] Playback mode has been added to Virtual Studio. Playback mode lets you run your AR application inside the Unity Editor using real AR data. This happens by “playing back” a dataset, which for ARDK 2.2 contains camera pose, frame images, and location data (optional). At runtime, ARDK’s context awareness algorithms are run using the dataset’s frames to surface camera, depth, semantics, and meshing data to the application. VPS localization (with Wayspot Anchor creation and restoration) is also supported in Playback mode. The VirtualStudio/Playback/PlaybackDatasetCapture scene added to ARDK-Examples can be used to capture Playback datasets.

  • [Shared AR] ReliableUnordered and UnreliableUnordered message transport types now use server-relayed TCP and UDP respectively. This improves bandwidth use (decreasing by 50 to 90 percent), reduces latency variance, and addresses packet drop rates when sending messages over 1KB.

Upgrade Guide

Use the following information when upgrading to ARDK 2.2

New Person Semantic Channel

What’s Changing

A new semantic channel, “person” has been added to ARDK’s semantic segmentation. Person segmentation detects people, including body parts, hair, and clothes worn.

How to Determine if your Application will be Affected

If your application refers to semantic channels by index rather than by channel name, your application will need to be updated.

How to Migrate your Application due to this Change

Any code referring to semantic channels at index 5 or higher should be updated to take into account the new channel index. For example, if you have code that accesses channel index 6 expecting the “foliage” channel, you’ll need to update this code to access channel index 7, as index 6 will no longer be “foliage” and instead will be “person”.

We strongly recommend referencing semantics by channel name as opposed to index, as names will be stable, whereas indices of each semantic channel could change in the future.

For example, let’s say we have the following function, which will check if a given semantic channel exists at a given position by using the index of that channel:

// Check if a given semantic channel exists at a position
bool CheckIfChannelAtPosition(int xPosition, int yPosition, int indexToCheck)
{
    // For example, indexToCheck could be 7
    return _semanticManager.SemanticBufferProcessor.DoesChannelExistAt(xPosition,yPosition,indexToCheck);
}

This function uses the index of a channel, and instead, we want to use the name of that channel. We can achieve this by updating our function to this:

// Check if a given semantic channel exists at a position
bool CheckIfChannelAtPosition(int xPosition, int yPosition, string channelToCheck)
{
    // For example, channelToCheck could be "foliage"
    return _semanticManager.SemanticBufferProcessor.DoesChannelExistAt(xPosition,yPosition,channelToCheck);
}

Using Updated IWaySpotAnchor to Track Anchor Events

What’s New

The IWayspotAnchor interface for VPS has a new StatusCode property and StatusCodeUpdated event. Subscribe to the StatusCodeUpdated event to know when Wayspot Anchors have been successfully created or restored by VPS. WayspotAnchorService.CreateWayspotAnchors() has been updated to immediately return a list of IWayspotAnchors in Pending status.

What’s Changing

The IWayspotAnchor.TrackingStateUpdated event has been obsoleted by the new IWayspotAnchor.TransformUpdated event. Subscribe to the new event or use the new IWayspotAnchor.Position and Rotation properties to track anchor position and orientation changes.

How to Determine if your Application will be Affected

If your application currently uses the IWayspotAnchor.TrackingStateUpdated event, it can be replaced with the new IWayspotAnchor.TransformUpdated event.

How to Migrate your Application due to this Change

You can also use or extend the new WayspotAnchorTracker extension class to handle TransformUpdated events automatically. The following example uses WayspotAnchorTracker to instantiate a prefab (_anchorPrefab) and associate it with a created or restored Wayspot anchor via WayspotAnchorTracker.AttachAnchor(), like so:

using Niantic.ARDK.Extensions;

private GameObject CreateWayspotAnchorGameObject(IWayspotAnchor anchor, Vector3 position, Quaternion rotation, bool startActive)
{
    var go = Instantiate(_anchorPrefab, position, rotation);

    var tracker = go.GetComponent<WayspotAnchorTracker>();
    if (tracker == null)
    {
    Debug.Log("Anchor prefab was missing WayspotAnchorTracker, so one will be added.");
    tracker = go.AddComponent<WayspotAnchorTracker>();
    }

    tracker.gameObject.SetActive(startActive);
    tracker.AttachAnchor(anchor);
    // ...Add to a local cache of WayspotAnchorTrackers if needed...

    return go;
}

Once a Wayspot anchor is attached to a WayspotAnchorTracker, the WayspotAnchorTracker will automatically handle IWayspotAnchor.TransformUpdated events and update the game object position and orientation accordingly.

See the User Manual and the WayspotAnchors scene in ARDK-examples for examples of handling these events.

TransportTypes Changes

What’s Changing

With the release of the ARDK version 2.2, we introduced a change regarding the TransportTypes that exist in Lightship.

To date, the Lightship ARDK has offered four different options for TransportTypes when integrating the low level networking APIs. Those four options are:

  • ReliableOrdered [now deprecated]

  • ReliableUnordered

  • UnreliableOrdered [now deprecated]

  • UnreliableUnordered

As noted above, we will be deprecating two of these TransportTypes, simplifying the options available to ReliableUnordered and UnreliableUnordered. These transport options will continue to function as expected for the time being, however will not be supported in a future version of Lightship. With this change, we recommend migrating off of using the ReliableOrdered and UnreliableOrdered transport options and switching to ReliableUnordered and UnreliableUnordered to take advantage of the reduced bandwidth use and reduced latency of the updated types.

How to Determine if your Application will be Affected

At a later date, the two deprecated TransportTypes will be removed from Lightship. If your application currently uses ReliableOrdered or UnreliableOrdered for networking message transport, we recommend switching the transport method you’re using to ReliableUnordered or UnreliableUnordered.

How to Migrate your Application due to this Change

Thankfully, the fix for this is quite simple! You just need to update your application to use the ReliableUnordered and UnreliableUnordered TransportTypes, as opposed to the now deprecated ReliableOrdered and UnreliableOrdered options.

For example, let’s say we have the following function, which sends a single networking message to a single peer:

// A message sent to a single peer in the session
void SendToASinglePeer(IMultipeerNetworking networking, IPeer peer, byte[] data)
{
  networking.SendDataToPeer(tag: 0, data: data, peer: peer, transportType: TransportType.ReliableOrdered);
}

You can see this function uses the ReliableOrdered TransportType, which we want to move away from. Updating this is as simple as a one line code change, and becomes the following:

// A message sent to a single peer in the session
void SendToASinglePeer(IMultipeerNetworking networking, IPeer peer, byte[] data)
{
  networking.SendDataToPeer(tag: 0, data: data, peer: peer, transportType: TransportType.ReliableUnordered);
}

If you’re not sure whether to use ReliableUnordered or UnreliableUnordered, it’s good to understand that reliable messages come with much more overhead than Unreliable messages. In general, reliable messages are good for game state information, such as keeping track of a score, spawning assets, or anything that would degrade the shared experience if not synchronized across all devices. If you do not need to communicate game state information, we recommend using the UnreliableUnordered TransportType.

If the order in which messages are received and processed is crucial for your application, consider including a piece of information in your message that indicates where the message falls in the order of messages you’re sending. For example, each message could contain a SequenceNumber property, so as messages are received you can check that property to ensure you process the messages in the correct order.

Having Your Users Update their Version of your Application

Once you’ve updated your application to use the proper transport types, you will want users to update to the latest version of your app. After you push a new version of your application to the App Store or Google Play, it’s possible for many users to have the updated version installed automatically. For users who do not have their applications updated automatically, you will want to encourage users to obtain the newest version of your application by installing the updated version manually.

Improvements

  • [Developer Platform] Updated ARCore version used by ARDK to 1.32.0

  • [VPS] The IWayspotAnchor interface has been updated with new properties and events to track when the anchor has been resolved and when the anchor position and orientation has been updated by VPS. See Upgrade Guide for more details.

  • [VPS] You can now subscribe to WayspotAnchorService.LocalizationUpdatedEvent to get events for when the VPS session localization state changes.

  • [VPS] Anchors can now be restored before the VPS session is localized, however restored anchors will not be able to resolve position and orientation until after the session is localized.

  • [VPS] Optimizations for anchor tracking during continuous localization.

  • [VPS] Mock VPS coverage areas are now filtered for location and radius.

Bug Fixes

  • [Contextual Awareness] Fixed issue with gameboard tile visualization appearing pink when using URP

  • [Developer Platform] Fixed bug with 2-finger scroll in examples when running in Unity on Windows.

  • [Developer Platform] Removed DepthMeshOcclusion scene in ARDK-Examples, as it duplicated the Depth scene

  • [Developer Platform] Remove missing prefab references and old corrupted meshes from some ARDK-Examples scenes

  • [Developer Tools] Fixed bug where if mock environments were imported into a project that already had a custom layer in the Layer 8 slot, it would override mock environment prefab’s layer and prevent the environment from being rendered.

  • [Developer Tools] Improved error message now surfaced when using Remote mode on M1 or Windows.

  • [VPS] New error code triggered when using the VPS Coverage API with no API key set.

Deprecations

  • [Shared AR] The following transport types will still work with ARDK 2.2, but are marked deprecated and may be removed in a future release: UnreliableOrdered and ReliableOrdered. The UnreliableUnordered and ReliableUnordered types should be used instead. See Upgrade Guide for more details.

  • [VPS] IWayspotAnchor.TrackingStateUpdated has been obsoleted and replaced by IWayspotAnchor.TransformUpdated. The callback and async versions of WayspotAnchorService.CreateWayspotAnchors() have been obsoleted and replaced by a WayspotAnchorService.CreateWayspotAnchors() method that returns immediately with a list of IWayspotAnchors in Pending state. See Upgrade Guide for more details.

Breaking Changes

  • [Contextual Awareness] New “person” semantic segmentation channel added at index 5. Numerical indices for semantic classes “building”, “foliage”, and “grass” have changed. See Upgrade Guide for more details.

  • [Contextual Awareness] Moved MeshObjSaver from ARDK.Assets.ARDK.Extensions.Meshing namespace to Niantic.ARDK.Extensions.Meshing namespace.

  • [VPS] Removed IWayspotAnchorsConfiguration properties intended for debug only: CloudProcessingForced, ClientProcessingForced, ConfigURL, HealthURL, LocalizationURL, GraphSyncURL, WayspotAnchorCreateURL, WayspotAnchorResolveURL, RegisterNodeURL, LookUpNodeURL.

Known Issues

  • [Contextual Awareness] On iOS 16 dev beta 3, the ARDK-Examples PlaneAnchor scene’s camera view shakes and jitters once meshing is rendered.

  • [Contextual Awareness] Semantic segmentation channel indices in Mock Mode differ from indices when running on device.

  • [Contextual Awareness] Person segmentation can be overconfident and include parts of the image that aren’t person, particularly classes that co-occur with person, like chairs or backpacks.

  • [Contextual Awareness - Experimental] A recent MediaPipe change is preventing hand tracking from working.

  • [Developer Platform] Voyage sample app bug in Walkabout scene makes game task not completable.

  • [Developer Tools] When running in Unity Play mode with Virtual Studio, in some circumstances output is rotated 90 degrees.

  • [VPS] “Motion and Orientation” pop-up displays after tapping on GPS “locate me” button in ARDK-Examples VPSCoverageExample sample scene on iOS. This is due to LocationService auto-enabling UnityEngine.Input.Compass.

  • [VPS] Some iPhone 8 Plus devices are encountering unexplained issues while localizing. Investigation is in progress.

  • [VPS] Using async versions of CreateWayspotAnchors() in WayspotAnchorService threw an exception due to bug in internal ARDK code