picoxr/face-mediapipe-pipeline

Face MediaPipe Spatial Pipeline

This repository contains a prebuilt SpatialML pipeline package for running MediaPipe Face detection with SecureMR in spatial mode.

The package is more than a model file. It bundles the serialized model, SpatialML pipeline JSON files, package metadata, and rendering assets needed to construct and run the full face-detection pipeline through the SpatialML package loader in spatial mode.

Package file

The main artifact is:

face-mediapipe-pipeline.zip

Extracting the zip produces a folder named face-mediapipe-pipeline/ with this structure:

face-mediapipe-pipeline/
├── manifest.json
├── gltf/
│   └── frame.gltf
├── model/
│   └── face_detector.tflite
└── pipeline/
    ├── face_detection_pipeline.json
    └── face_display_pipeline.json

What is included

• manifest.json: schema v2 package metadata. It declares the package id, pipeline entries, schema version, and Spatial runtime support.
• model/face_detector.tflite: MediaPipe Face detection model in TFLite format.
• pipeline/face_detection_pipeline.json: SpatialML pipeline definition for VST camera access, affine preprocessing, color conversion, LiteRT model inference, and MediaPipe SSD post-processing.
• pipeline/face_display_pipeline.json: SpatialML pipeline definition for consuming detection output, projecting the face center into 3D camera space, and producing a pose for the face-frame scene.
• gltf/frame.gltf: glTF asset rendered by the display pipeline.

Pipeline behavior

The package contains two SpatialML pipelines:

1. detection: reads rectified VST camera tensors, preprocesses the left camera image into a 256x256 RGB float input, runs the MediaPipe Face model, decodes the best SSD detection, and writes the post_det tensor.
2. display: consumes post_det together with VST image, timestamp, and camera-matrix tensors, converts the detected image-space face center into 3D camera space, and outputs frame_pose plus frame_gltf for rendering.

The manifest declares Spatial runtime support:

"runtime": {
  "supported_modes": [
    "spatial"
  ]
}

How to use the package

Spatial SDK

Place the extracted face-mediapipe-pipeline/ folder in your Android app assets. A Spatial SDK app can keep SpatialML assets under a namespaced assets path:

app/src/main/assets/SpatialML/face-mediapipe-pipeline/

Load the package with the SpatialML package loader. In the Spatial SDK, the public loader entry point is:

public fun SpatialMLSession.loadPipelinePackageFromAssets(
    assetRoot: String,
    externalGlobals: Map<String, GlobalTensor> = emptyMap(),
): PipelinePackageBundle

The loader reads manifest.json, validates that runtime.supported_modes includes spatial, builds each declared pipeline, creates or reuses global tensors for package inputs and outputs, and returns a PipelinePackageBundle containing:

• pipelines: package pipelines keyed by manifest id.
• globalTensors: materialized package tensors keyed by tensor name.
• manifest: parsed package metadata.

The Spatial SDK sample uses this package like this:

import com.pico.spatial.ml.securemr.SceneGraphProperty
import com.pico.spatial.ml.securemr.SpatialMLInstance
import com.pico.spatial.ml.securemr.SpatialMLSession
import com.pico.spatial.ml.securemr.loadPipelinePackageFromAssets

val instance = SpatialMLInstance.create(context)
val session =
    instance.createSession(
        SpatialMLSession.InitInfo(
            imageWidth = 580,
            imageHeight = 326,
            containerWidth = 1000,
            containerHeight = 1000,
            containerDepth = 10,
            containerType = SpatialMLSession.ContainerType.VOLUMETRIC,
        )
    )!!

val bundle = session.loadPipelinePackageFromAssets("SpatialML/face-mediapipe-pipeline")
val detection = bundle.pipelines["detection"]!!
val display = bundle.pipelines["display"]!!
val frameScene = bundle.globalTensors["frame_gltf"]!!
val framePose = bundle.globalTensors["frame_pose"]!!

val initTask =
    session.newPipeline().run {
        // The full sample also sets an initial pose and scale before showing the scene.
        switchSceneVisibility(frameScene, newLocalTensor(0xF.toByte()))
        submit(emptyMap(), null, null)
    }

val framePipeline =
    session.newPipeline().apply {
        updateSceneGraphProperty(
            frameScene,
            "/",
            SceneGraphProperty.CameraAnchor.Follow,
            framePose,
        )
    }

val detectionTask = detection.pipeline.submit(detection.submitBindings, null, initTask)
val displayTask = display.pipeline.submit(display.submitBindings, null, detectionTask)
framePipeline.submit(emptyMap(), null, displayTask)

Submit the package pipelines in sequence for each frame. The display pipeline depends on tensors produced by the detection pipeline, so the display submission should wait for the detection submission. The sample also creates a small frame pipeline that applies the generated frame_pose to the packaged frame_gltf scene through SceneGraphProperty.CameraAnchor.Follow.

The package also exposes global tensors such as:

• post_det: decoded face detection output.
• vst_left_image, vst_right_image, vst_timestamp, vst_camera_matrix: VST tensors shared between the detection and display pipelines.
• frame_pose: transform matrix generated by the display pipeline.
• frame_gltf: packaged glTF scene rendered as the face frame.

For Android packaging, make sure pipeline assets are not compressed by the APK packager. The Spatial SDK sample keeps .json, .gltf, and .tflite in androidResources.noCompress. Apps using VST camera tensors and spatial-data readback also need the appropriate app permissions, including com.picovr.permission.SPATIAL_DATA.

Spatial SDK sample

Spatial SDK examples are documented here:

• PICO Spatial SDK examples

A Spatial SDK integration should load this package from assets with SpatialMLSession.loadPipelinePackageFromAssets(...), create a volumetric SpatialML session, submit the detection pipeline first, then submit the display pipeline to update the face-frame pose and glTF output.

Device support

This package is intended for supported PICO devices running a SecureMR runtime in spatial mode that supports the bundled SpatialML operators, LiteRT model inference, VST access, UV-to-3D projection, and glTF scene output.

License

Apache License 2.0.