Semantic Confidence

Note

This feature is an experimental feature and is subject to change. Experimental features should not be used in production products and are unsupported. We are providing these experimental features as a preview of what might be coming in future releases. If you have feedback on an experimental feature post your feedback in the Lightship Developer Community

Semantic confidence is an update that expands the semantic segmentation features of ARDK. Use semantic confidence to determine the likelihood of a given camera frame pixel to belong to a semantic class. For example, a pixel might have a 80% likelihood that it represents “sky” and a 30% likelihood that it represents “foliage”.

Semantic confidence provides an additional layer of visibility into ARDK’s semantic processing. Use semantic confidence for scenarios where you need a specific level of confidence for a particular channel, or need more granular information about the channels a pixel might represent. For example, you could use semantic confidence values to better blend the edges between semantic boundaries and reduce visual aliasing when using semantics for masking. Or, your app might only want to utilize pixels that have a greater than 75% likelihood of being in a particular channel.

The following clip shows the difference between existing thresholded semantic segmentation data from ARDK (the middle clip) vs semantic confidence data (the rightmost clip).

Semantic confidence generation has the same device and OS requirements that semantic segmentation has.

Accessing Confidence Data

You can access semantic confidence data using either ARSemanticSegmentationManager or from IARFrame. Semantic confidence data is provided separately from the thresholded semantic segmentation data ARDK was already providing. Semantic confidence data is not available through the ISemanticBuffer interface, only thresholded semantic segmentation data is provided by ISemanticBuffer.

Confidence values for a semantic channel are represented as per-pixel floats from 0.0 (0% likelihood) to 1.0 (100% likelihood).

Accessing Using ARSemanticSegmentationManager

Using ARSemanticSegmentationManager, you can set which semantic channels to gather semantic confidence data for, and then query for the confidence values for those channels.

Specify which channels ARDK should get confidence values for by passing an array of channel names to ARSemanticSegmentationManager.SetConfidenceChannels().

using Niantic.ARDK.AR;
using Niantic.ARDK.AR.Awareness;
using Niantic.ARDK.AR.Awareness.Semantics;
using Niantic.ARDK.Extensions;

private ARSemanticSegmentationManager _semanticSegmentationManager;

// Channels we want to get semantic confidence data for
private readonly string[] _channels = {
    "artificial_ground", "sky", "building", "person"
};

private void Start()
{
    // Tell the manager what channels does it need to maintain confidences for
    _semanticSegmentationManager.SetConfidenceChannels(_channels);

    // Set event handler for semantic buffer update
    _semanticSegmentationManager.SemanticBufferUpdated += OnSemanticBufferUpdated;
}

To get semantic confidence data, add an event handler for ARSemanticSegmentationManager.SemanticBufferUpdated, and in your event handler get the confidence value buffer for a specific channel using GetConfidences(), or create a float or ARGB Texture2D for a specific channel using GetConfidencesRFloat() or GetConfidencesARGB32().

The following example gets a semantics confidence ARGB Texture2D for a specific channel and renders it to the display.

using Niantic.ARDK.AR;
using Niantic.ARDK.AR.Awareness;
using Niantic.ARDK.AR.Awareness.Semantics;
using Niantic.ARDK.Configuration;
using Niantic.ARDK.Extensions;
using Niantic.ARDK.Rendering;

private ARSemanticSegmentationManager _semanticSegmentationManager;
private Texture2D _semanticTexture;
private Material _overlayMaterial;
// The index of the currently displayed semantics channel in the array
private int _featureChannel;

// Channels we care about in this example
private readonly string[] _channels = {
    "artificial_ground", "sky", "building", "person"
};

// Handle SemanticBufferUpdated event to get confidence values into texture
private void OnSemanticBufferUpdated(ContextAwarenessStreamUpdatedArgs<ISemanticBuffer> args)
{
    if (!args.IsKeyFrame)
    {
        return;
    }

    // Get the name of the observed channel
    var currentChannelName = _channels[_featureChannel];

    // Update the texture using confidence values
    _semanticSegmentationManager.GetConfidencesARGB32
    (
        ref _semanticTexture,
        currentChannelName
    );
}

// We use this callback to overlay semantics on the rendered background
private void OnRenderImage(RenderTexture src, RenderTexture dest)
{
    // Get the transformation to correctly map the texture to the viewport
    var sampler = _semanticSegmentationManager.SemanticBufferProcessor.SamplerTransform;

    // Update the transform
    _overlayMaterial.SetMatrix(PropertyBindings.SemanticsTransform, sampler);

    // Update the texture
    _overlayMaterial.SetTexture(PropertyBindings.SemanticChannel, _semanticTexture);

    // Display semantics
    Graphics.Blit(src, dest, _overlayMaterial);
}

Accessing Using IARFrame

Semantic confidence data for the current frame is provided through the IARFrame.CopySemanticConfidences() method. This method returns a floating point buffer of per-pixel confidence values for a specific semantic channel. Additionally, you can get confidence data using the AwarenessUtils extension. To get a float or ARGB Texture2D representation of the confidence data, use the CopySemanticConfidencesRFloat() or CopySemanticConfidencesARGB32() methods of AwarenessUtils.

using Niantic.ARDK.AR;
using Niantic.ARDK.Extensions;
using Niantic.ARDK.AR.ARSessionEventArgs;
using Niantic.ARDK.AR.Awareness;
using Niantic.ARDK.AR.Awareness.Semantics;

private Texture2D _semanticTexture;
private Matrix4x4 _samplerTransform;

private void ARSession_OnFrameUpdated(FrameUpdatedArgs args)
{
    var frame = args.Frame;
    if (frame == null || frame.Semantics == null)
        return;

    frame.CopySemanticConfidencesARGB32
    (
        "sky",
        desired_width,
        desired_height,
        ref _semanticTexture,
        out var sampler
    );

    // Use sampler transform in your OnRenderImage() as needed
    _samplerTransform = sampler;
}

Get the most recent frame either through the IARSession.CurrentFrame property, or by subscribing to the IARSession.FrameUpdated event.

Additional Examples

See more code examples in SemanticSegmentationExampleManager in the SemanticSegmentation example Unity scene in ARDK-examples under ContextAwareness/Segmentation.