Sync from huggingface/lerobot-dataset-visualizer via hub-sync

AGENTS.md

@@ -59,7 +59,7 @@ Locating an episode (its video URLs and offsets, its data file and row range) an
 - Create datasets with `leRobotDataset(repoId)` in `fetch-data.ts`: it passes the `DATASET_URL` endpoint and `authHeaders()`.
 - `episodes({ offset, limit })` reads the v3 episode index across every metadata chunk. `offset` is a position in the listing, not an `episode_index`.
 - `episode.data.fromRow` / `toRow` are row offsets **inside `episode.data.url`**, not dataset-wide frame indexes. Pass them straight to `rowStart` / `rowEnd`; never subtract the file's first `index`. `episode.data` is optional: it is absent when the data file can't be read.
-- Episode charts come from `frames()` (numeric and boolean series, bookkeeping columns left out). `loadEpisodeFrames` reads the task and language columns (`TEXT_COLUMNS`) from the same file alongside it, since `frames()` returns numbers only.
+- Episode charts come from `frames()` (numeric and boolean series, bookkeeping columns left out). `loadEpisodeFrames` reads the task and language columns (`TEXT_COLUMNS`) from the same file alongside it, since `frames()` returns numbers only. hyparquet decodes a nested column for its whole row group, so a nested language column is left out past `MAX_NESTED_TEXT_VALUES` values.
 - Camera sizes come from `parseInfo(...).cameras`. Never read them from `shape[0]` / `shape[1]`: some datasets declare channel-first shapes (`[3, H, W]`).
 - To upgrade, bump the version in `package.json` and run `bun install`. Read the package's changes first: 0.0.4 changed what `fromRow` / `toRow` mean.
 
@@ -131,9 +131,9 @@ Reserved/bookkeeping columns from lerobot — see `EXCLUDED_COLUMNS` in `src/uti
 
 - URDFs and meshes are hosted in the HF bucket `lerobot/robot-urdfs` — base URL `https://huggingface.co/buckets/lerobot/robot-urdfs/resolve` (no `/main` segment; buckets are unbranched). Override with `NEXT_PUBLIC_URDF_BASE_URL` for local development.
 - Asset layout under the bucket: `g1/`, `openarm/`, `so101/` (both SO-100 and SO-101 live here).
-- **URDFLoader gotcha**: after our `loadMeshCb` returns, `URDFLoader.js` does `if (obj instanceof THREE.Mesh) obj.material = <urdf-material>`, overwriting any material we set. Workaround: wrap the loaded mesh in a `THREE.Group` so the `instanceof Mesh` check fails. DAE returns a Group already; STL must be wrapped explicitly.
+- SO-100/SO-101 and OpenArm are drawn from the USDs in the dataset [`lerobot/robot-usds`](https://huggingface.co/datasets/lerobot/robot-usds), loaded with `@huggingface/usd-web-viewer`. Their URDFs share the USDs' joint and link names, link frames and zero pose, and only move them: `attachUsdMeshes` puts each USD mesh under the URDF link its prim path names. G1 still loads its URDF's STL meshes.
+- **URDFLoader gotcha**: after our `loadMeshCb` returns, `URDFLoader.js` does `if (obj instanceof THREE.Mesh) obj.material = <urdf-material>`, overwriting any material we set. Workaround: wrap the loaded mesh in a `THREE.Group` so the `instanceof Mesh` check fails.
 - **STLLoader event ordering**: `manager.itemEnd(url)` fires _before_ the user `onLoad` callback, so `manager.onLoad` can fire before meshes are attached to the robot tree. Defer post-load work (auto-fit camera, shadow flags) with `setTimeout(..., 0)`. Don't try to rebuild materials in `manager.onLoad` — pick the archetype color directly inside `loadMeshCb`.
-- **OpenArm DAE files ship 23 stray `PointLight`s** that drown out scene lighting. Strip non-`AmbientLight` lights from `collada.scene` before adding it to the robot.
 - Scene setup: `<Canvas shadows>` with `ACESFilmicToneMapping` (exposure 0.9), 3-point directional + ambient lights, `<Environment preset="studio" background={false} />`, `<color attach="background" args={["#1a2433"]} />`. `<OrbitControls makeDefault />` is required so `useThree().controls` exposes the controls for auto-fit.
 
 ## Design system

README.md

@@ -37,10 +37,33 @@ This tool is designed to help robotics researchers and practitioners quickly ins
 - **Action Insights Panel:** Data-driven analysis tools to guide training configuration — includes autocorrelation, state-action alignment, speed distribution, and cross-episode variance heatmap.
 - **Filtering Panel:** Identify and flag problematic episodes (low movement, jerky motion, outlier length) for removal. Exports flagged episode IDs as a ready-to-run LeRobot CLI command.
 - **3D URDF Viewer:** Visualize robot joint poses frame-by-frame in an interactive 3D scene, with end-effector trail rendering. Supports SO-100, SO-101, and OpenArm bimanual robots.
+- **3D Hands:** Automatically detects native SOMA156 hand annotations. Shows wrists, 21-joint hand skeletons, wrist trails and the saved camera trajectory alongside a synchronized camera overlay, with a Follow camera control.
 - **Annotations Panel:** Hand-edit the v3.1 language schema (`language_persistent` + `language_events`) — subtask, plan, memory, interjection + paired speech, and VQA atoms with bounding-box / keypoint / count / attribute / spatial answers. VQA bboxes and keypoints render as overlays on the video player; drag or click on a camera to draw new ones. Backed by an optional FastAPI service (in `backend/`) for parquet rewrites and HF Hub push.
 - **Efficient Data Loading:** Uses parquet and JSON loading for large dataset support, with pagination, chunking, and lazy-loaded panels for fast initial load.
 - **Responsive UI:** Built with React, Next.js, and Tailwind CSS for a fast, modern user experience.
 
+## Hand annotations
+
+Choose **Advanced → 3D Hands** (below **Annotations**) or open an annotated
+episode with `?tab=hands`. Detection requires the 156D
+`action` plus `annotation.hand_joints_camera` and `annotation.hand_joints_world`
+(126 values each), `annotation.hand_action_valid`, `annotation.hand_observed`
+(left/right) and `annotation.hand_reference_frame`. No model weights or meshes
+are required. Missing or masked poses remain gaps in both views.
+
+When `annotation.camera_world_from_optical` contains saved rigid transforms,
+the viewer reconstructs recording-wide hand positions from camera-space joints,
+anchored to the first saved camera pose. The camera path also breaks at missing
+transforms. Without them, it displays the stored segment-start FLU coordinates
+and breaks wrist trails at reference resets. Positions are in meters.
+
+The overlay uses the source camera and per-source-episode calibration saved in
+`annotation_studio/run.json`, resolving exported episodes through
+`source_episode_map`; a video feature's `info["camera.intrinsics"]` is also
+supported. Pinhole and OpenCV fisheye distortion are projected onto the original
+video at its saved resolution. Unknown cameras, missing calibration and mismatched
+video dimensions disable the overlay. Estimated calibration is labeled.
+
 ## Technologies Used
 
 - **Next.js** (App Router)
@@ -48,6 +71,7 @@ This tool is designed to help robotics researchers and practitioners quickly ins
 - **Recharts** (for data visualization)
 - **Three.js** + **@react-three/fiber** + **@react-three/drei** (for 3D URDF visualization)
 - **urdf-loader** (for parsing URDF robot models)
+- **@huggingface/usd-web-viewer** (for the robot USD models in [lerobot/robot-usds](https://huggingface.co/datasets/lerobot/robot-usds))
 - **hyparquet** (for reading Parquet files)
 - **Tailwind CSS** (styling)
 

bun.lock

@@ -6,9 +6,11 @@
       "name": "lerobot-viewer",
       "dependencies": {
         "@huggingface/hub": "^2.11.0",
-        "@huggingface/lerobot": "^0.0.11",
+        "@huggingface/lerobot": "^0.0.13",
+        "@huggingface/usd-web-viewer": "0.2.0",
         "@react-three/drei": "^10.7.7",
         "@react-three/fiber": "^9.5.0",
+        "fzstd": "0.1.1",
         "hyparquet": "^1.12.1",
         "next": "15.3.6",
         "react": "^19.0.0",
@@ -28,6 +30,7 @@
         "@types/three": "^0.182.0",
         "eslint": "^9",
         "eslint-config-next": "15.3.1",
+        "happy-dom": "21.0.4",
         "prettier": "3.8.2",
         "tailwindcss": "^4",
         "typescript": "^5",
@@ -67,10 +70,12 @@
 
     "@huggingface/hub": ["@huggingface/hub@2.11.0", "", { "dependencies": { "@huggingface/tasks": "^0.19.90" }, "optionalDependencies": { "cli-progress": "^3.12.0" }, "bin": { "hfjs": "dist/cli.js" } }, "sha512-WS6QGaXYeBVFlaB4SOn6z4LGUpLB5kRZNL08uUni4izX353KxiwwZMK5+/AWX86MJh8SMZNa/JFcvFCcQsbszQ=="],
 
-    "@huggingface/lerobot": ["@huggingface/lerobot@0.0.11", "", { "dependencies": { "hyparquet": "^1.31.2" } }, "sha512-HlU2b8PC+khySlnTeRWxIUSB41eN83H0Gpci0GK/folPWrAppLD4v0dqAXq1NDTloh5elBxX+rfhf+TQD12fkw=="],
+    "@huggingface/lerobot": ["@huggingface/lerobot@0.0.13", "", { "dependencies": { "fzstd": "0.1.1", "hyparquet": "^1.31.2" } }, "sha512-FWjICnB9ndm4eRWH52xAybT4bDhK+JQZCMJrF3ivVYtG/b8H81I+vcZ5Yw22NNCQVZ2Ib7FgOLYo0rmyxJrUoQ=="],
 
     "@huggingface/tasks": ["@huggingface/tasks@0.19.90", "", {}, "sha512-nfV9luJbvwGQ/5oKXkKhCV9h4X7mwh1YaGG3ORd6UMLDSwr1OFSSatcBX0O9OtBtmNK19aGSjbLFqqgcIR6+IA=="],
 
+    "@huggingface/usd-web-viewer": ["@huggingface/usd-web-viewer@0.2.0", "", { "peerDependencies": { "@types/three": ">=0.162.0 <0.187.0", "three": ">=0.162.0 <0.187.0" }, "optionalPeers": ["@types/three"] }, "sha512-2VpC0Gwbq0n7cl4/Lb5YpxQeUdFg6rvM2ZpIkU/Hc5eYaM5nsHqcDgwy2RVaZ/DOJcgDpmwkxWAlrVXhq3rpqg=="],
+
     "@humanfs/core": ["@humanfs/core@0.19.1", "", {}, "sha512-5DyQ4+1JEUzejeK1JGICcideyfUbGixgS9jNgex5nqkW+cY7WZhxBigmieN5Qnw9ZosSNVC9KQKyb+GUaGyKUA=="],
 
     "@humanfs/node": ["@humanfs/node@0.16.7", "", { "dependencies": { "@humanfs/core": "^0.19.1", "@humanwhocodes/retry": "^0.4.0" } }, "sha512-/zUx+yOsIrG4Y43Eh2peDeKCxlRt/gET6aHfaKpuq267qXdYDFViVHfMaLyygZOnl0kGWxFIgsBy8QFuTLUXEQ=="],
@@ -263,6 +268,10 @@
 
     "@types/webxr": ["@types/webxr@0.5.24", "", {}, "sha512-h8fgEd/DpoS9CBrjEQXR+dIDraopAEfu4wYVNY2tEPwk60stPWhvZMf4Foo5FakuQ7HFZoa8WceaWFervK2Ovg=="],
 
+    "@types/whatwg-mimetype": ["@types/whatwg-mimetype@3.0.2", "", {}, "sha512-c2AKvDT8ToxLIOUlN51gTiHXflsfIFisS4pO7pDPoKouJCESkhZnEy623gwP9laCy5lnLDAw1vAzu2vM2YLOrA=="],
+
+    "@types/ws": ["@types/ws@8.18.2", "", { "dependencies": { "@types/node": "*" } }, "sha512-67MQl+fpWKVTT1NYdnmo3U4sc/xPo/zQBncVnI74qmQa0z/b+1g6iYqNmGCPbxO+zz2aklb08a0oHfegiVd0/w=="],
+
     "@typescript-eslint/eslint-plugin": ["@typescript-eslint/eslint-plugin@8.55.0", "", { "dependencies": { "@eslint-community/regexpp": "^4.12.2", "@typescript-eslint/scope-manager": "8.55.0", "@typescript-eslint/type-utils": "8.55.0", "@typescript-eslint/utils": "8.55.0", "@typescript-eslint/visitor-keys": "8.55.0", "ignore": "^7.0.5", "natural-compare": "^1.4.0", "ts-api-utils": "^2.4.0" }, "peerDependencies": { "@typescript-eslint/parser": "^8.55.0", "eslint": "^8.57.0 || ^9.0.0", "typescript": ">=4.8.4 <6.0.0" } }, "sha512-1y/MVSz0NglV1ijHC8OT49mPJ4qhPYjiK08YUQVbIOyu+5k862LKUHFkpKHWu//zmr7hDR2rhwUm6gnCGNmGBQ=="],
 
     "@typescript-eslint/parser": ["@typescript-eslint/parser@8.55.0", "", { "dependencies": { "@typescript-eslint/scope-manager": "8.55.0", "@typescript-eslint/types": "8.55.0", "@typescript-eslint/typescript-estree": "8.55.0", "@typescript-eslint/visitor-keys": "8.55.0", "debug": "^4.4.3" }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0", "typescript": ">=4.8.4 <6.0.0" } }, "sha512-4z2nCSBfVIMnbuu8uinj+f0o4qOeggYJLbjpPHka3KH1om7e+H9yLKTYgksTaHcGco+NClhhY2vyO3HsMH1RGw=="],
@@ -379,6 +388,8 @@
 
     "buffer": ["buffer@6.0.3", "", { "dependencies": { "base64-js": "^1.3.1", "ieee754": "^1.2.1" } }, "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA=="],
 
+    "buffer-image-size": ["buffer-image-size@0.6.4", "", { "dependencies": { "@types/node": "*" } }, "sha512-nEh+kZOPY1w+gcCMobZ6ETUp9WfibndnosbpwB1iJk/8Gt5ZF2bhS6+B6bPYz424KtwsR6Rflc3tCz1/ghX2dQ=="],
+
     "bun-types": ["bun-types@1.3.10", "", { "dependencies": { "@types/node": "*" } }, "sha512-tcpfCCl6XWo6nCVnpcVrxQ+9AYN1iqMIzgrSKYMB/fjLtV2eyAVEg7AxQJuCq/26R6HpKWykQXuSOq/21RYcbg=="],
 
     "busboy": ["busboy@1.6.0", "", { "dependencies": { "streamsearch": "^1.1.0" } }, "sha512-8SFQbg/0hQ9xy3UNTB0YEnsNBbWfhf7RtnzpL7TkBiTBRfrQ9Fxcnz7VJsleJpyp6rVLvXiuORqjlHi5q+PYuA=="],
@@ -471,6 +482,8 @@
 
     "enhanced-resolve": ["enhanced-resolve@5.19.0", "", { "dependencies": { "graceful-fs": "^4.2.4", "tapable": "^2.3.0" } }, "sha512-phv3E1Xl4tQOShqSte26C7Fl84EwUdZsyOuSSk9qtAGyyQs2s3jJzComh+Abf4g187lUUAvH+H26omrqia2aGg=="],
 
+    "entities": ["entities@7.0.1", "", {}, "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA=="],
+
     "es-abstract": ["es-abstract@1.24.1", "", { "dependencies": { "array-buffer-byte-length": "^1.0.2", "arraybuffer.prototype.slice": "^1.0.4", "available-typed-arrays": "^1.0.7", "call-bind": "^1.0.8", "call-bound": "^1.0.4", "data-view-buffer": "^1.0.2", "data-view-byte-length": "^1.0.2", "data-view-byte-offset": "^1.0.1", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "es-set-tostringtag": "^2.1.0", "es-to-primitive": "^1.3.0", "function.prototype.name": "^1.1.8", "get-intrinsic": "^1.3.0", "get-proto": "^1.0.1", "get-symbol-description": "^1.1.0", "globalthis": "^1.0.4", "gopd": "^1.2.0", "has-property-descriptors": "^1.0.2", "has-proto": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "internal-slot": "^1.1.0", "is-array-buffer": "^3.0.5", "is-callable": "^1.2.7", "is-data-view": "^1.0.2", "is-negative-zero": "^2.0.3", "is-regex": "^1.2.1", "is-set": "^2.0.3", "is-shared-array-buffer": "^1.0.4", "is-string": "^1.1.1", "is-typed-array": "^1.1.15", "is-weakref": "^1.1.1", "math-intrinsics": "^1.1.0", "object-inspect": "^1.13.4", "object-keys": "^1.1.1", "object.assign": "^4.1.7", "own-keys": "^1.0.1", "regexp.prototype.flags": "^1.5.4", "safe-array-concat": "^1.1.3", "safe-push-apply": "^1.0.0", "safe-regex-test": "^1.1.0", "set-proto": "^1.0.0", "stop-iteration-iterator": "^1.1.0", "string.prototype.trim": "^1.2.10", "string.prototype.trimend": "^1.0.9", "string.prototype.trimstart": "^1.0.8", "typed-array-buffer": "^1.0.3", "typed-array-byte-length": "^1.0.3", "typed-array-byte-offset": "^1.0.4", "typed-array-length": "^1.0.7", "unbox-primitive": "^1.1.0", "which-typed-array": "^1.1.19" } }, "sha512-zHXBLhP+QehSSbsS9Pt23Gg964240DPd6QCf8WpkqEXxQ7fhdZzYsocOr5u7apWonsS5EjZDmTF+/slGMyasvw=="],
 
     "es-define-property": ["es-define-property@1.0.1", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="],
@@ -557,6 +570,8 @@
 
     "functions-have-names": ["functions-have-names@1.2.3", "", {}, "sha512-xckBUXyTIqT97tq2x2AMb+g163b5JFysYk0x4qxNFwbfQkmNZoiRHb6sPzI9/QV33WeuvVYBUIiD4NzNIyqaRQ=="],
 
+    "fzstd": ["fzstd@0.1.1", "", {}, "sha512-dkuVSOKKwh3eas5VkJy1AW1vFpet8TA/fGmVA5krThl8YcOVE/8ZIoEA1+U1vEn5ckxxhLirSdY837azmbaNHA=="],
+
     "generator-function": ["generator-function@2.0.1", "", {}, "sha512-SFdFmIJi+ybC0vjlHN0ZGVGHc3lgE0DxPAT0djjVg+kjOnSqclqmj0KQ7ykTOLP6YxoqOvuAODGdcHJn+43q3g=="],
 
     "get-intrinsic": ["get-intrinsic@1.3.0", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="],
@@ -579,6 +594,8 @@
 
     "graceful-fs": ["graceful-fs@4.2.11", "", {}, "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="],
 
+    "happy-dom": ["happy-dom@21.0.4", "", { "dependencies": { "@types/node": ">=20.0.0", "@types/whatwg-mimetype": "^3.0.2", "@types/ws": "^8.18.1", "buffer-image-size": "^0.6.4", "entities": "^7.0.1", "whatwg-mimetype": "^3.0.0", "ws": "^8.21.0" } }, "sha512-tFm+GaLBgo6o0awHnvHaNoraVo27qJ+2tZEtZ+cIa08hXbeitrxAJajLI05hsUy/2/YTKcUjQYzHe5mCcraLBw=="],
+
     "has-bigints": ["has-bigints@1.1.0", "", {}, "sha512-R3pbpkcIqv2Pm3dUwgjclDRVmWpTJW2DcMzcIhEXEx1oh/CEMObMm3KLmRJOdvhM7o4uQBnwr8pzRK2sJWIqfg=="],
 
     "has-flag": ["has-flag@4.0.0", "", {}, "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ=="],
@@ -981,6 +998,8 @@
 
     "webgl-sdf-generator": ["webgl-sdf-generator@1.1.1", "", {}, "sha512-9Z0JcMTFxeE+b2x1LJTdnaT8rT8aEp7MVxkNwoycNmJWwPdzoXzMh0BjJSh/AEFP+KPYZUli814h8bJZFIZ2jA=="],
 
+    "whatwg-mimetype": ["whatwg-mimetype@3.0.0", "", {}, "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q=="],
+
     "which": ["which@2.0.2", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "./bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="],
 
     "which-boxed-primitive": ["which-boxed-primitive@1.1.1", "", { "dependencies": { "is-bigint": "^1.1.0", "is-boolean-object": "^1.2.1", "is-number-object": "^1.1.1", "is-string": "^1.1.1", "is-symbol": "^1.1.1" } }, "sha512-TbX3mj8n0odCBFVlY8AxkqcHASw3L60jIuF8jFP78az3C2YhmGvqbHBpAjTRH2/xqYunrJ9g1jSyjCjpoWzIAA=="],
@@ -993,6 +1012,8 @@
 
     "word-wrap": ["word-wrap@1.2.5", "", {}, "sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA=="],
 
+    "ws": ["ws@8.22.0", "", { "peerDependencies": { "bufferutil": "^4.0.1", "utf-8-validate": ">=5.0.2" }, "optionalPeers": ["bufferutil", "utf-8-validate"] }, "sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg=="],
+
     "yocto-queue": ["yocto-queue@0.1.0", "", {}, "sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q=="],
 
     "zustand": ["zustand@5.0.11", "", { "peerDependencies": { "@types/react": ">=18.0.0", "immer": ">=9.0.6", "react": ">=18.0.0", "use-sync-external-store": ">=1.2.0" }, "optionalPeers": ["@types/react", "immer", "react", "use-sync-external-store"] }, "sha512-fdZY+dk7zn/vbWNCYmzZULHRrss0jx5pPFiOuMZ/5HJN6Yv3u+1Wswy/4MpZEkEGhtNH+pwxZB8OKgUBPzYAGg=="],

package.json

@@ -16,9 +16,11 @@
   },
   "dependencies": {
     "@huggingface/hub": "^2.11.0",
-    "@huggingface/lerobot": "^0.0.11",
+    "@huggingface/lerobot": "^0.0.13",
+    "@huggingface/usd-web-viewer": "0.2.0",
     "@react-three/drei": "^10.7.7",
     "@react-three/fiber": "^9.5.0",
+    "fzstd": "0.1.1",
     "hyparquet": "^1.12.1",
     "next": "15.3.6",
     "react": "^19.0.0",
@@ -38,6 +40,7 @@
     "@types/three": "^0.182.0",
     "eslint": "^9",
     "eslint-config-next": "15.3.1",
+    "happy-dom": "21.0.4",
     "prettier": "3.8.2",
     "tailwindcss": "^4",
     "typescript": "^5"

src/app/[org]/[dataset]/[episode]/episode-viewer.tsx

@@ -6,6 +6,8 @@ import { postParentMessageWithParams } from "@/utils/postParentMessage";
 import { SimpleVideosPlayer } from "@/components/simple-videos-player";
 import PlaybackBar from "@/components/playback-bar";
 import { TimeProvider, useTime, useTimeControls } from "@/context/time-context";
+import { useAuth } from "@/context/auth-context";
+import { getAuthToken } from "@/utils/auth";
 import { FlaggedEpisodesProvider } from "@/context/flagged-episodes-context";
 import {
   AnnotationsProvider,
@@ -37,6 +39,7 @@ import { getDatasetVersionAndInfo } from "@/utils/versionUtils";
 import type { DatasetMetadata } from "@/utils/parquetUtils";
 
 const URDFViewer = lazy(() => import("@/components/urdf-viewer"));
+const HandReplay = lazy(() => import("@/components/hand-replay"));
 const ActionInsightsPanel = lazy(
   () => import("@/components/action-insights-panel"),
 );
@@ -71,6 +74,7 @@ const TABS = [
   "filtering",
   "doctor",
   "urdf",
+  "hands",
 ] as const;
 type ActiveTab = (typeof TABS)[number];
 
@@ -118,6 +122,11 @@ const ADVANCED_TABS: { tab: ActiveTab; label: string; title?: string }[] = [
     title:
       "Edit subtask / plan / memory / interjection / VQA atoms (lerobot v3.1 schema)",
   },
+  {
+    tab: "hands",
+    label: "3D Hands",
+    title: "Synchronized wrists, hands and camera overlays",
+  },
   { tab: "statistics", label: "Statistics" },
   { tab: "filtering", label: "Filtering" },
   { tab: "frames", label: "Frames" },
@@ -132,13 +141,16 @@ const ADVANCED_TABS: { tab: ActiveTab; label: string; title?: string }[] = [
 function AdvancedTabsMenu({
   activeTab,
   onSelect,
+  handsAvailable,
 }: {
   activeTab: ActiveTab;
   onSelect: (tab: ActiveTab) => void;
+  handsAvailable: boolean;
 }) {
   const [open, setOpen] = useState(false);
   const rootRef = useRef<HTMLDivElement>(null);
-  const active = ADVANCED_TABS.find((t) => t.tab === activeTab);
+  const tabs = ADVANCED_TABS.filter((t) => t.tab !== "hands" || handsAvailable);
+  const active = tabs.find((t) => t.tab === activeTab);
 
   useEffect(() => {
     if (!open) return;
@@ -193,7 +205,7 @@ function AdvancedTabsMenu({
           role="menu"
           className="absolute left-0 top-full z-50 mt-1 min-w-[12rem] panel-raised bg-[var(--surface-1)] p-1.5 shadow-xl animate-menu-pop [transform-origin:top_left]"
         >
-          {ADVANCED_TABS.map(({ tab, label, title }) => (
+          {tabs.map(({ tab, label, title }) => (
             <button
               key={tab}
               role="menuitem"
@@ -265,6 +277,9 @@ export default function EpisodeViewer({
   const [error, setError] = useState<string | null>(null);
   const [noEpisodes, setNoEpisodes] = useState(false);
   const requestIdRef = useRef(0);
+  const loadedRef = useRef<{ key: string; token: string | null } | null>(null);
+  const { oauth } = useAuth();
+  const accessToken = oauth?.accessToken ?? null;
 
   // Opening straight onto 3D Replay: start the robot meshes downloading now,
   // alongside the episode data, rather than once it has loaded and the viewer
@@ -291,6 +306,14 @@ export default function EpisodeViewer({
       setData(null);
       return;
     }
+    // Requests read the token from storage, so the first load already uses a
+    // saved sign-in, which AuthProvider restores into `accessToken` only
+    // afterwards. Reload for a new token, not for that one.
+    const key = `${org}/${dataset}/${episodeId}`;
+    const token = getAuthToken();
+    if (loadedRef.current?.key === key && loadedRef.current.token === token)
+      return;
+    loadedRef.current = { key, token };
     const requestId = ++requestIdRef.current;
     setError(null);
     setNoEpisodes(false);
@@ -315,7 +338,7 @@ export default function EpisodeViewer({
         setError(message || "Unknown error");
         setData(null);
       });
-  }, [org, dataset, episodeId]);
+  }, [org, dataset, episodeId, accessToken]);
 
   if (noEpisodes) {
     return (
@@ -347,6 +370,9 @@ export default function EpisodeViewer({
           <p className="text-sm font-mono whitespace-pre-wrap text-red-700/90 dark:text-red-200/90">
             {error}
           </p>
+          <div className="mt-4">
+            <HfAuthButton variant="ghost" />
+          </div>
         </div>
       </div>
     );
@@ -426,11 +452,19 @@ function EpisodeViewerInner({
   // Safe because EpisodeViewerInner only mounts client-side (behind a loading gate).
   const [activeTab, setActiveTab] = useState<ActiveTab>(() => {
     if (typeof window !== "undefined") {
+      const requested = new URLSearchParams(window.location.search).get("tab");
+      if (
+        TABS.includes(requested as ActiveTab) &&
+        (requested !== "hands" || data.handTracking) &&
+        (requested !== "urdf" || urdfSupported)
+      )
+        return requested as ActiveTab;
       const stored = sessionStorage.getItem("activeTab");
       /// A tab saved on another dataset may not exist here (3D Replay needs a supported robot).
       if (
         TABS.includes(stored as ActiveTab) &&
-        (stored !== "urdf" || urdfSupported)
+        (stored !== "urdf" || urdfSupported) &&
+        (stored !== "hands" || data.handTracking)
       ) {
         return stored as ActiveTab;
       }
@@ -741,7 +775,11 @@ function EpisodeViewerInner({
       <div className="relative z-30 flex items-center border-b border-line-subtle bg-[var(--surface-0)] shrink-0">
         {renderTab("episodes", "Episodes")}
         {urdfSupported && renderTab("urdf", "3D Replay")}
-        <AdvancedTabsMenu activeTab={activeTab} onSelect={handleTabChange} />
+        <AdvancedTabsMenu
+          activeTab={activeTab}
+          onSelect={handleTabChange}
+          handsAvailable={!!data.handTracking}
+        />
         <div className="ml-auto flex shrink-0 items-center gap-1 pl-2">
           <ThemeToggle />
           <HfAuthButton variant="tab" />
@@ -753,6 +791,7 @@ function EpisodeViewerInner({
         {/* Sidebar — on Episodes and 3D Replay tabs */}
         {(activeTab === "episodes" ||
           activeTab === "annotations" ||
+          activeTab === "hands" ||
           activeTab === "urdf") && (
           <Sidebar
             datasetInfo={datasetInfo}
@@ -795,6 +834,7 @@ function EpisodeViewerInner({
                 <SimpleVideosPlayer
                   videosInfo={videosInfo}
                   onVideosReady={() => setVideosReady(true)}
+                  handTracking={data.handTracking}
                 />
               )}
 
@@ -846,6 +886,7 @@ function EpisodeViewerInner({
                     <SimpleVideosPlayer
                       videosInfo={videosInfo}
                       onVideosReady={() => setVideosReady(true)}
+                      handTracking={data.handTracking}
                     />
                   </div>
                 )}
@@ -942,6 +983,20 @@ function EpisodeViewerInner({
               />
             </Suspense>
           )}
+          {activeTab === "hands" && data.handTracking && (
+            <>
+              <EpisodeHeader
+                repoId={datasetInfo.repoId}
+                episodeId={episodeId}
+              />
+              <Suspense fallback={<Loading />}>
+                <HandReplay
+                  tracking={data.handTracking}
+                  videosInfo={videosInfo}
+                />
+              </Suspense>
+            </>
+          )}
         </div>
       </div>
     </div>

src/app/[org]/[dataset]/[episode]/fetch-data.ts

@@ -2,8 +2,10 @@ import {
   DatasetMetadata,
   fetchParquetFile,
   formatStringWithVars,
+  nestedColumnValues,
   readParquetAsObjects,
 } from "@/utils/parquetUtils";
+import { parquetMetadataAsync } from "hyparquet";
 import { pick } from "@/utils/pick";
 import {
   getDatasetVersionAndInfo,
@@ -31,6 +33,13 @@ import {
 } from "@huggingface/lerobot";
 import type { LeRobotEpisode, LeRobotFrames } from "@huggingface/lerobot";
 import { authHeaders, getAuthToken } from "@/utils/auth";
+import {
+  hasHandTracking,
+  handProjectionMetadata,
+  isHandGeometryColumn,
+  readHandTracking,
+  type HandTracking,
+} from "@/lib/hand-tracking";
 
 const SERIES_NAME_DELIMITER = CHART_CONFIG.SERIES_NAME_DELIMITER;
 
@@ -279,6 +288,8 @@ export type EpisodeData = {
    * events compatible with the writer in lerobot#3471.
    */
   frameTimestamps?: number[];
+  /** Full-resolution native hand geometry, independent of sampled chart rows. */
+  handTracking?: HandTracking;
 };
 
 type EpisodeMetadataV3 = {
@@ -745,7 +756,11 @@ async function getEpisodeDataV2(
   const videosInfo =
     info.video_path !== null ? toVideosInfo(leRobotEpisode, info, false) : [];
 
-  const { frames, textRows } = await loadEpisodeFrames(repoId, leRobotEpisode);
+  const { frames, textRows, handTracking } = await loadEpisodeFrames(
+    repoId,
+    leRobotEpisode,
+    info,
+  );
   const frameTimestamps = sortedTimestamps(frames);
   const languageAtoms = extractLanguageAtoms(textRows);
 
@@ -842,6 +857,7 @@ async function getEpisodeDataV2(
     task,
     languageAtoms,
     frameTimestamps,
+    handTracking,
   };
 }
 
@@ -877,6 +893,7 @@ async function getEpisodeDataV3(
     task,
     languageAtoms,
     frameTimestamps,
+    handTracking,
   } = await loadEpisodeDataV3(
     repoId,
     version,
@@ -901,6 +918,7 @@ async function getEpisodeDataV3(
     task,
     languageAtoms,
     frameTimestamps,
+    handTracking,
   };
 }
 
@@ -918,9 +936,14 @@ async function loadEpisodeDataV3(
   task?: string;
   languageAtoms?: import("@/types/language.types").LanguageAtom[];
   frameTimestamps?: number[];
+  handTracking?: HandTracking;
 }> {
   try {
-    const { frames, textRows } = await loadEpisodeFrames(repoId, episode);
+    const { frames, textRows, handTracking } = await loadEpisodeFrames(
+      repoId,
+      episode,
+      info,
+    );
     const frameTimestamps = sortedTimestamps(frames);
 
     if (!frames) {
@@ -931,6 +954,7 @@ async function loadEpisodeDataV3(
         task: undefined,
         languageAtoms: extractLanguageAtoms(textRows),
         frameTimestamps,
+        handTracking,
       };
     }
 
@@ -1041,6 +1065,7 @@ async function loadEpisodeDataV3(
       task,
       languageAtoms: extractLanguageAtoms(textRows),
       frameTimestamps,
+      handTracking,
     };
   } catch {
     return {
@@ -1149,26 +1174,70 @@ const TEXT_COLUMNS = [
   "language_events",
 ];
 
+/**
+ * Past this many values in its row groups, a nested text column is left out: hyparquet decodes it
+ * for the whole row group, and a file written as one row group can hold millions of atoms. 22M
+ * took 2-3 GB of heap; 44M (MinhDuk/final_dataset_annotated) and 66M crashed the tab.
+ */
+const MAX_NESTED_TEXT_VALUES = 25_000_000;
+
 /** The episode's numeric series from `@huggingface/lerobot`, plus its text columns read alongside. */
 async function loadEpisodeFrames(
   repoId: string,
   episode: LeRobotEpisode,
+  info: DatasetMetadata,
 ): Promise<{
   frames: LeRobotFrames | undefined;
   textRows: Record<string, unknown>[];
+  handTracking?: HandTracking;
 }> {
   const data = episode.data;
   if (!data) return { frames: undefined, textRows: [] };
-  const [frames, textRows] = await Promise.all([
+  const [frames, textRows, report] = await Promise.all([
     leRobotDataset(repoId).frames(episode),
-    fetchParquetFile(data.url).then((file) =>
-      readParquetAsObjects(file, TEXT_COLUMNS, {
-        rowStart: data.fromRow,
-        rowEnd: data.toRow,
-      }),
-    ),
+    fetchParquetFile(data.url).then(async (file) => {
+      const values = nestedColumnValues(
+        await parquetMetadataAsync(file),
+        data.fromRow,
+        data.toRow,
+      );
+      const skipped = TEXT_COLUMNS.filter(
+        (column) => (values.get(column) ?? 0) > MAX_NESTED_TEXT_VALUES,
+      );
+      if (skipped.length > 0)
+        console.warn(
+          `[language] ${skipped.join(", ")} left out: its row group is too large to decode`,
+        );
+      return readParquetAsObjects(
+        file,
+        TEXT_COLUMNS.filter((column) => !skipped.includes(column)),
+        { rowStart: data.fromRow, rowEnd: data.toRow },
+      );
+    }),
+    hasHandTracking(info)
+      ? fetch(
+          buildVersionedUrl(
+            repoId,
+            info.codebase_version,
+            "annotation_studio/run.json",
+          ),
+          { headers: authHeaders(), signal: AbortSignal.timeout(10_000) },
+        )
+          .then((response) =>
+            response.ok ? (response.json() as Promise<unknown>) : null,
+          )
+          .catch(() => null)
+      : Promise.resolve(null),
   ]);
-  return { frames, textRows };
+  return {
+    frames,
+    textRows,
+    handTracking: readHandTracking(
+      info,
+      frames,
+      handProjectionMetadata(info, episode.index, report),
+    ),
+  };
 }
 
 /** Source-frame timestamps, sorted, for the annotations editor to snap to. */
@@ -1192,6 +1261,7 @@ function chartDataFromFrames(
   const series: [string, number[]][] = [];
   for (const [key, components] of Object.entries(frames.series)) {
     if (!info.features[key] || excludedColumns.includes(key)) continue;
+    if (hasHandTracking(info) && isHandGeometryColumn(key)) continue;
     const names =
       frames.names[key]?.length === components.length
         ? frames.names[key]

src/components/__tests__/simple-videos-player.test.tsx

@@ -0,0 +1,163 @@
+import {
+  afterAll,
+  afterEach,
+  beforeAll,
+  describe,
+  expect,
+  test,
+} from "bun:test";
+import { Window } from "happy-dom";
+import { act, useEffect, useState } from "react";
+import { createRoot, type Root } from "react-dom/client";
+import { AnnotationsProvider } from "@/context/annotations-context";
+import { TimeProvider, useTime } from "@/context/time-context";
+import type { VideoInfo } from "@/types";
+import type { HandTracking } from "@/lib/hand-tracking";
+import { SimpleVideosPlayer } from "../simple-videos-player";
+
+const dom = new Window({ url: "http://localhost" });
+const originals = new Map<string, PropertyDescriptor | undefined>();
+let root: Root;
+let container: HTMLDivElement;
+let time: ReturnType<typeof useTime>;
+
+beforeAll(() => {
+  const globals = {
+    window: dom,
+    document: dom.document,
+    HTMLVideoElement: dom.HTMLVideoElement,
+    HTMLMediaElement: dom.HTMLMediaElement,
+    Event: dom.Event,
+    ResizeObserver: dom.ResizeObserver,
+    requestAnimationFrame: dom.requestAnimationFrame.bind(dom),
+    cancelAnimationFrame: dom.cancelAnimationFrame.bind(dom),
+    IS_REACT_ACT_ENVIRONMENT: true,
+  };
+  for (const [key, value] of Object.entries(globals)) {
+    originals.set(key, Object.getOwnPropertyDescriptor(globalThis, key));
+    Object.defineProperty(globalThis, key, {
+      configurable: true,
+      writable: true,
+      value,
+    });
+  }
+});
+
+afterEach(async () => {
+  if (root) await act(async () => root.unmount());
+  container?.remove();
+});
+
+afterAll(async () => {
+  await dom.happyDOM.close();
+  for (const [key, descriptor] of originals) {
+    if (descriptor) Object.defineProperty(globalThis, key, descriptor);
+    else Reflect.deleteProperty(globalThis, key);
+  }
+});
+
+const videos: VideoInfo[] = [
+  {
+    filename: "ego",
+    url: "http://localhost/ego.mp4",
+    isSegmented: true,
+    segmentStart: 12.5,
+    segmentEnd: 22.5,
+  },
+  {
+    filename: "other",
+    url: "http://localhost/other.mp4",
+    isSegmented: true,
+    segmentStart: 30,
+    segmentEnd: 40,
+  },
+];
+
+async function mount(initialTime = 0, handTracking?: HandTracking) {
+  let showPlayer: () => void;
+  function Controls() {
+    time = useTime();
+    const { seek } = time;
+    useEffect(() => seek(initialTime), [seek]);
+    return null;
+  }
+  function DeferredPlayer() {
+    const [visible, setVisible] = useState(false);
+    showPlayer = () => setVisible(true);
+    return visible ? (
+      <SimpleVideosPlayer videosInfo={videos} handTracking={handTracking} />
+    ) : null;
+  }
+  container = document.createElement("div");
+  document.body.append(container);
+  root = createRoot(container);
+  await act(async () => {
+    root.render(
+      <TimeProvider duration={10}>
+        <AnnotationsProvider>
+          <Controls />
+          <DeferredPlayer />
+        </AnnotationsProvider>
+      </TimeProvider>,
+    );
+  });
+  // The lazy hands panel can mount after the page has already applied ?t=.
+  await act(async () => showPlayer());
+  return Array.from(container.querySelectorAll("video"));
+}
+
+async function ready(elements: HTMLVideoElement[]) {
+  await act(async () => {
+    for (const video of elements) video.dispatchEvent(new Event("loadeddata"));
+  });
+  await act(async () => time.setIsPlaying(false));
+}
+
+describe("hand replay video synchronization", () => {
+  test("restores the URL timestamp on every camera after media loads", async () => {
+    const elements = await mount(3.25);
+    await ready(elements);
+    expect(time.currentTime).toBe(3.25);
+    expect(elements.map((video) => video.currentTime)).toEqual([15.75, 33.25]);
+  });
+
+  test("uses a seek made during loading instead of the mount timestamp", async () => {
+    const elements = await mount(3);
+    await act(async () => time.seek(5));
+    await ready(elements);
+    expect(time.currentTime).toBe(5);
+    expect(elements.map((video) => video.currentTime)).toEqual([17.5, 35]);
+  });
+
+  test("paused single-frame seeks update both source videos", async () => {
+    const elements = await mount(1);
+    await ready(elements);
+    await act(async () => time.seek(2));
+    await act(async () => time.seek(2 + 1 / 30));
+    expect(elements[0].currentTime).toBeCloseTo(14.5 + 1 / 30, 8);
+    expect(elements[1].currentTime).toBeCloseTo(32 + 1 / 30, 8);
+    expect(time.isPlaying).toBe(false);
+  });
+
+  test("labels estimated calibration beside the overlay controls", async () => {
+    await mount(0, {
+      frames: [],
+      fps: 30,
+      cameraKey: "ego",
+      coordinateFrame: "segment",
+      overlayIssue: null,
+      calibration: {
+        model: "pinhole",
+        width: 640,
+        height: 480,
+        fx: 600,
+        fy: 600,
+        cx: 320,
+        cy: 240,
+        distortion: [],
+        estimated: true,
+      },
+    });
+    expect(container.textContent).toContain("Estimated camera calibration");
+  });
+});

src/components/hand-overlay-canvas.tsx

@@ -0,0 +1,156 @@
+"use client";
+
+import { useEffect, useRef } from "react";
+import {
+  HAND_COLORS,
+  HAND_EDGES,
+  HAND_INDICES,
+  handFrameIndex,
+  projectHandPoint,
+  type HandTracking,
+} from "@/lib/hand-tracking";
+
+/** Follow this camera's displayed frames, including paused seeks and shared-file offsets. */
+export function HandOverlayCanvas({
+  video,
+  tracking,
+  cameraKey,
+  segmentStart = 0,
+  enabled,
+}: {
+  video: HTMLVideoElement | null;
+  tracking: HandTracking;
+  cameraKey: string;
+  segmentStart?: number;
+  enabled: boolean;
+}) {
+  const canvasRef = useRef<HTMLCanvasElement>(null);
+  useEffect(() => {
+    const canvas = canvasRef.current;
+    const calibration = tracking.calibration;
+    if (
+      !canvas ||
+      !video ||
+      !calibration ||
+      cameraKey !== tracking.cameraKey ||
+      !enabled
+    )
+      return;
+    let callback = 0;
+    let mediaTime = video.currentTime;
+    const draw = (time = mediaTime) => {
+      mediaTime = time;
+      const width = video.clientWidth,
+        height = video.clientHeight;
+      if (!width || !height) return;
+      const dpr = Math.min(window.devicePixelRatio || 1, 2);
+      if (
+        canvas.width !== Math.round(width * dpr) ||
+        canvas.height !== Math.round(height * dpr)
+      ) {
+        canvas.width = Math.round(width * dpr);
+        canvas.height = Math.round(height * dpr);
+      }
+      const ctx = canvas.getContext("2d");
+      if (!ctx) return;
+      ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
+      ctx.clearRect(0, 0, width, height);
+      // A changed crop/resolution needs a corresponding calibration, not an implicit guess.
+      if (
+        video.videoWidth !== calibration.width ||
+        video.videoHeight !== calibration.height
+      )
+        return;
+      const scale = Math.min(
+        width / calibration.width,
+        height / calibration.height,
+      );
+      const left = (width - calibration.width * scale) / 2;
+      const top = (height - calibration.height * scale) / 2;
+      const frame =
+        tracking.frames[handFrameIndex(tracking.frames, time - segmentStart)];
+      if (!frame) return;
+      ctx.save();
+      ctx.beginPath();
+      ctx.rect(
+        left,
+        top,
+        calibration.width * scale,
+        calibration.height * scale,
+      );
+      ctx.clip();
+      HAND_INDICES.forEach((hand) => {
+        const joints = frame.camera[hand];
+        if (!joints) return;
+        const points = joints.map((p): [number, number] | null => {
+          const pixel = projectHandPoint(p, calibration);
+          return pixel
+            ? [left + pixel[0] * scale, top + pixel[1] * scale]
+            : null;
+        });
+        const path = () => {
+          ctx.beginPath();
+          for (const [a, b] of HAND_EDGES) {
+            const p = points[a],
+              q = points[b];
+            if (p && q) {
+              ctx.moveTo(p[0], p[1]);
+              ctx.lineTo(q[0], q[1]);
+            }
+          }
+          ctx.stroke();
+        };
+        ctx.strokeStyle = "#111827";
+        ctx.lineWidth = 5;
+        path();
+        ctx.strokeStyle = HAND_COLORS[hand];
+        ctx.lineWidth = 2.5;
+        path();
+        points.forEach((p, joint) => {
+          if (!p) return;
+          ctx.beginPath();
+          ctx.arc(p[0], p[1], joint === 0 ? 5 : 2.5, 0, Math.PI * 2);
+          ctx.fillStyle = HAND_COLORS[hand];
+          ctx.fill();
+        });
+        const wrist = points[0];
+        if (wrist) {
+          ctx.font = "bold 13px sans-serif";
+          ctx.lineWidth = 3;
+          ctx.strokeStyle = "#111827";
+          ctx.strokeText(hand === 0 ? "L" : "R", wrist[0] + 8, wrist[1] - 8);
+          ctx.fillText(hand === 0 ? "L" : "R", wrist[0] + 8, wrist[1] - 8);
+        }
+      });
+      ctx.restore();
+    };
+    const update = () => draw(video.currentTime);
+    const observer = new ResizeObserver(() => draw());
+    observer.observe(video);
+    const events = ["loadeddata", "seeked", "resize"];
+    events.forEach((event) => video.addEventListener(event, update));
+    if ("requestVideoFrameCallback" in HTMLVideoElement.prototype) {
+      const frame = (_now: number, metadata: VideoFrameCallbackMetadata) => {
+        draw(metadata.mediaTime);
+        callback = video.requestVideoFrameCallback(frame);
+      };
+      callback = video.requestVideoFrameCallback(frame);
+    } else video.addEventListener("timeupdate", update);
+    update();
+    return () => {
+      observer.disconnect();
+      events.forEach((event) => video.removeEventListener(event, update));
+      video.removeEventListener("timeupdate", update);
+      if (callback) video.cancelVideoFrameCallback(callback);
+      const ctx = canvas.getContext("2d");
+      ctx?.clearRect(0, 0, canvas.width, canvas.height);
+    };
+  }, [video, tracking, cameraKey, segmentStart, enabled]);
+  return (
+    <canvas
+      ref={canvasRef}
+      aria-hidden="true"
+      className="absolute inset-0 h-full w-full pointer-events-none"
+    />
+  );
+}

src/components/hand-replay.tsx

@@ -0,0 +1,376 @@
+"use client";
+
+import {
+  useEffect,
+  useMemo,
+  useRef,
+  useState,
+  type ComponentRef,
+  type RefObject,
+} from "react";
+import { Canvas, useThree } from "@react-three/fiber";
+import { Bounds, Grid, Html, Line, OrbitControls } from "@react-three/drei";
+import { Vector3 } from "three";
+import { useTime } from "@/context/time-context";
+import { useTheme } from "@/utils/theme";
+import {
+  HAND_COLORS,
+  HAND_EDGES,
+  HAND_INDICES,
+  cameraPathRuns,
+  fluToScene,
+  handFrameIndex,
+  wristTrail,
+  type HandTracking,
+  type HandFrame,
+  type CameraPose,
+  type Point3,
+} from "@/lib/hand-tracking";
+import { SimpleVideosPlayer } from "./simple-videos-player";
+import PlaybackBar from "./playback-bar";
+import type { VideoInfo } from "@/types";
+
+/** A fixed episode-wide fit keeps the orbit camera still while the hands move. */
+function RecordingBounds({ tracking }: { tracking: HandTracking }) {
+  const bounds = useMemo(() => {
+    const lo: Point3 = [Infinity, Infinity, Infinity],
+      hi: Point3 = [-Infinity, -Infinity, -Infinity];
+    for (const frame of tracking.frames) {
+      const positions = frame[tracking.coordinateFrame].flatMap(
+        (hand) => hand ?? [],
+      );
+      if (frame.cameraPose) positions.push(frame.cameraPose.position);
+      for (const p of positions) {
+        const point = fluToScene(p);
+        for (let a = 0; a < 3; a++) {
+          lo[a] = Math.min(lo[a], point[a]);
+          hi[a] = Math.max(hi[a], point[a]);
+        }
+      }
+    }
+    if (!Number.isFinite(lo[0]))
+      return { center: [0, 0, 0] as Point3, size: [1, 1, 1] as Point3 };
+    return {
+      center: lo.map((v, a) => (v + hi[a]) / 2) as Point3,
+      size: lo.map((v, a) => Math.max(hi[a] - v, 0.4)) as Point3,
+    };
+  }, [tracking]);
+  return (
+    <mesh position={bounds.center} visible={false}>
+      <boxGeometry args={bounds.size} />
+      <meshBasicMaterial />
+    </mesh>
+  );
+}
+
+function Hands({
+  tracking,
+  index,
+  trails,
+}: {
+  tracking: HandTracking;
+  index: number;
+  trails: boolean;
+}) {
+  const frame = tracking.frames[index];
+  return (
+    <>
+      {HAND_INDICES.map((hand) => {
+        const joints = frame?.[tracking.coordinateFrame][hand];
+        if (!joints) return null;
+        const points = joints.map(fluToScene),
+          color = HAND_COLORS[hand];
+        const trail = trails
+          ? wristTrail(
+              tracking.frames,
+              index,
+              hand,
+              2,
+              tracking.coordinateFrame,
+            )
+          : [];
+        return (
+          <group key={hand}>
+            {HAND_EDGES.map(([a, b]) => (
+              <Line
+                key={`${a}-${b}`}
+                points={[points[a], points[b]]}
+                color={color}
+                lineWidth={3}
+              />
+            ))}
+            {points.map((p, joint) => (
+              <mesh key={joint} position={p}>
+                <sphereGeometry args={[joint === 0 ? 0.012 : 0.004, 12, 8]} />
+                <meshBasicMaterial color={color} />
+              </mesh>
+            ))}
+            {trail.length > 1 && (
+              <Line
+                points={trail}
+                color={color}
+                lineWidth={1.5}
+                transparent
+                opacity={0.5}
+              />
+            )}
+          </group>
+        );
+      })}
+    </>
+  );
+}
+
+function RecordedCamera({
+  tracking,
+  index,
+}: {
+  tracking: HandTracking;
+  index: number;
+}) {
+  const paths = useMemo(
+    () => cameraPathRuns(tracking.frames, index),
+    [tracking.frames, index],
+  );
+  const pose = tracking.frames[index]?.cameraPose;
+  const origin = pose ? fluToScene(pose.position) : null;
+  const at = (pose: CameraPose, x: number, y: number, z: number): Point3 => {
+    const coordinate = (axis: 0 | 1 | 2) =>
+      pose.position[axis] +
+      pose.axes[0][axis] * x +
+      pose.axes[1][axis] * y +
+      pose.axes[2][axis] * z;
+    return fluToScene([coordinate(0), coordinate(1), coordinate(2)]);
+  };
+  const corners = pose
+    ? [
+        at(pose, -0.055, -0.035, 0.08),
+        at(pose, 0.055, -0.035, 0.08),
+        at(pose, 0.055, 0.035, 0.08),
+        at(pose, -0.055, 0.035, 0.08),
+      ]
+    : [];
+  return (
+    <>
+      {paths
+        .filter((path) => path.length > 1)
+        .map((path, i) => (
+          <Line key={i} points={path} color="#8391a4" lineWidth={1.5} />
+        ))}
+      {origin && pose && (
+        <>
+          <mesh position={origin}>
+            <sphereGeometry args={[0.012, 12, 8]} />
+            <meshBasicMaterial color="#ffffff" />
+          </mesh>
+          {(["#ed6464", "#54ba7c", "#579cff"] as const).map((color, axis) => (
+            <Line
+              key={axis}
+              points={[
+                origin,
+                at(
+                  pose,
+                  axis === 0 ? 0.12 : 0,
+                  axis === 1 ? 0.12 : 0,
+                  axis === 2 ? 0.12 : 0,
+                ),
+              ]}
+              color={color}
+              lineWidth={2}
+            />
+          ))}
+          {corners.map((corner, i) => (
+            <Line
+              key={i}
+              points={[origin, corner, corners[(i + 1) % 4]]}
+              color="#8391a4"
+              lineWidth={1}
+            />
+          ))}
+          <Html position={origin} style={{ pointerEvents: "none" }}>
+            <span className="ml-3 text-xs text-fg-muted">Camera</span>
+          </Html>
+        </>
+      )}
+    </>
+  );
+}
+
+/** Follow translation while preserving the user's orbit angle and zoom. */
+function FollowCamera({
+  frame,
+  enabled,
+  orbitRef,
+}: {
+  frame: HandFrame | undefined;
+  enabled: boolean;
+  orbitRef: RefObject<ComponentRef<typeof OrbitControls> | null>;
+}) {
+  const { camera, invalidate } = useThree();
+  const following = useRef(false);
+  useEffect(() => {
+    if (!enabled) {
+      following.current = false;
+      return;
+    }
+    if (!frame?.cameraPose) return;
+    const orbit = orbitRef.current;
+    if (!orbit) return;
+    const { position, axes } = frame.cameraPose;
+    const focus = new Vector3(
+      ...fluToScene([
+        position[0] + axes[2][0] * 0.4,
+        position[1] + axes[2][1] * 0.4,
+        position[2] + axes[2][2] * 0.4,
+      ]),
+    );
+    if (!following.current) {
+      camera.position
+        .sub(orbit.target)
+        .normalize()
+        .multiplyScalar(1.1)
+        .add(focus);
+      following.current = true;
+    } else camera.position.add(focus.clone().sub(orbit.target));
+    orbit.target.copy(focus);
+    orbit.update();
+    invalidate();
+  }, [camera, orbitRef, enabled, frame, invalidate]);
+  return null;
+}
+
+export default function HandReplay({
+  tracking,
+  videosInfo,
+}: {
+  tracking: HandTracking;
+  videosInfo: VideoInfo[];
+}) {
+  const { currentTime } = useTime();
+  const orbitRef = useRef<ComponentRef<typeof OrbitControls>>(null);
+  const theme = useTheme();
+  const [trails, setTrails] = useState(true);
+  const [follow, setFollow] = useState(false);
+  const index = handFrameIndex(tracking.frames, currentTime),
+    frame = tracking.frames[index];
+  const videos = useMemo(
+    () =>
+      [...videosInfo].sort(
+        (a, b) =>
+          Number(b.filename === tracking.cameraKey) -
+          Number(a.filename === tracking.cameraKey),
+      ),
+    [videosInfo, tracking.cameraKey],
+  );
+  return (
+    <div className="space-y-4">
+      <div className="flex flex-wrap items-center justify-between gap-3 text-sm">
+        <div className="flex items-center gap-4">
+          {HAND_COLORS.map((color, hand) => (
+            <span key={hand} className="flex items-center gap-1.5">
+              <i
+                className="h-2 w-2 rounded-full"
+                style={{ backgroundColor: color }}
+              />
+              {hand === 0 ? "Left" : "Right"} hand
+            </span>
+          ))}
+          <span className="font-mono text-xs text-fg-muted">
+            Frame {index >= 0 ? index : "—"}
+          </span>
+        </div>
+        <div className="flex items-center gap-4">
+          {tracking.coordinateFrame === "recording" && (
+            <button
+              type="button"
+              className={`btn text-xs ${follow ? "btn-active" : ""}`}
+              aria-pressed={follow}
+              onClick={() => setFollow((value) => !value)}
+            >
+              Follow camera
+            </button>
+          )}
+          <label className="flex items-center gap-2 text-fg-soft">
+            <input
+              type="checkbox"
+              checked={trails}
+              onChange={(e) => setTrails(e.target.checked)}
+            />
+            Wrist trails
+          </label>
+        </div>
+      </div>
+      <div className="grid gap-4 md:grid-cols-2 items-start">
+        <div className="panel overflow-hidden">
+          <div
+            className="relative h-[min(45vh,420px)] min-h-72"
+            aria-label="3D recorded hand and wrist positions"
+          >
+            <Canvas
+              frameloop="demand"
+              camera={{ position: [0.8, 0.6, 1.1], near: 0.001, far: 1000 }}
+            >
+              <color
+                attach="background"
+                args={[theme === "dark" ? "#151b24" : "#f3f5f7"]}
+              />
+              <OrbitControls ref={orbitRef} makeDefault />
+              <Bounds fit clip margin={1.5}>
+                <RecordingBounds tracking={tracking} />
+              </Bounds>
+              <Grid
+                infiniteGrid
+                cellSize={0.1}
+                sectionSize={0.5}
+                fadeDistance={12}
+                cellColor={theme === "dark" ? "#354052" : "#cbd2dc"}
+                sectionColor={theme === "dark" ? "#536278" : "#9aa6b5"}
+              />
+              <axesHelper args={[0.15]} />
+              <Hands tracking={tracking} index={index} trails={trails} />
+              <RecordedCamera tracking={tracking} index={index} />
+              <FollowCamera
+                frame={frame}
+                enabled={follow}
+                orbitRef={orbitRef}
+              />
+            </Canvas>
+            {!frame?.[tracking.coordinateFrame].some(Boolean) && (
+              <div className="absolute inset-x-0 top-4 text-center pointer-events-none">
+                <span className="rounded bg-[var(--surface-0)] px-3 py-2 text-sm text-fg-muted">
+                  No observed hand pose at this frame
+                </span>
+              </div>
+            )}
+          </div>
+          <div className="border-t border-line-subtle p-3 text-xs text-fg-muted space-y-1">
+            <p>Drag to orbit · scroll to zoom · grid cells 10 cm</p>
+            <p>
+              {tracking.coordinateFrame === "recording"
+                ? "Recording frame anchored to the first saved camera pose, in meters. Camera paths and wrist trails break where poses are missing."
+                : "Segment-start frame, in meters. Coordinates and trails reset at each valid span. Camera transforms are unavailable."}
+            </p>
+            {(["Left", "Right"] as const).map((side, hand) => (
+              <p key={side} className="font-mono">
+                {side} wrist:{" "}
+                {frame?.[tracking.coordinateFrame][hand]?.[0]
+                  .map((n) => n.toFixed(3))
+                  .join(", ") ?? "unobserved"}{" "}
+                {frame?.[tracking.coordinateFrame][hand]
+                  ? "(forward, left, up)"
+                  : ""}
+              </p>
+            ))}
+          </div>
+        </div>
+        <div className="space-y-3">
+          <SimpleVideosPlayer videosInfo={videos} handTracking={tracking} />
+          {tracking.overlayIssue && (
+            <p className="text-sm text-fg-muted">{tracking.overlayIssue}</p>
+          )}
+        </div>
+      </div>
+      <PlaybackBar />
+    </div>
+  );
+}

src/components/simple-videos-player.tsx

@@ -7,11 +7,10 @@ import type { VideoInfo } from "@/types";
 import { proxyHfUrl } from "@/utils/auth";
 import { VideoOverlayCanvas } from "./video-overlay-canvas";
 import { ColormappedVideo } from "./colormapped-video";
+import { HandOverlayCanvas } from "./hand-overlay-canvas";
+import type { HandTracking } from "@/lib/hand-tracking";
 
-const THRESHOLDS = {
-  VIDEO_SYNC_TOLERANCE: 0.2,
-  VIDEO_SEGMENT_BOUNDARY: 0.05,
-};
+const THRESHOLDS = { VIDEO_SEGMENT_BOUNDARY: 0.05 };
 
 const VIDEO_READY_TIMEOUT_MS = 10_000;
 
@@ -32,6 +31,7 @@ const SUPPORTS_RVFC =
 type VideoPlayerProps = {
   videosInfo: VideoInfo[];
   onVideosReady?: () => void;
+  handTracking?: HandTracking;
 };
 
 const videoEventCleanup = new WeakMap<HTMLVideoElement, () => void>();
@@ -39,7 +39,9 @@ const videoEventCleanup = new WeakMap<HTMLVideoElement, () => void>();
 export const SimpleVideosPlayer = ({
   videosInfo,
   onVideosReady,
+  handTracking,
 }: VideoPlayerProps) => {
+  const [showHands, setShowHands] = React.useState(true);
   const { currentTime, seek, externalSeekVersion, isPlaying, setIsPlaying } =
     useTime();
   const videoRefs = useRef<(HTMLVideoElement | null)[]>([]);
@@ -126,6 +128,20 @@ export const SimpleVideosPlayer = ({
   // context's version moves past this, an external seek happened and we
   // need to drive every video to the new position.
   const lastSeekVersionRef = useRef(externalSeekVersion);
+  // Media may take seconds to load. Keep the latest external seek, including
+  // the initial URL timestamp and seeks made while loading, until it is ready.
+  const pendingSeekRef = useRef({
+    version: externalSeekVersion,
+    time: currentTime,
+  });
+  useEffect(() => {
+    if (pendingSeekRef.current.version !== externalSeekVersion) {
+      pendingSeekRef.current = {
+        version: externalSeekVersion,
+        time: currentTime,
+      };
+    }
+  }, [externalSeekVersion, currentTime]);
 
   // Mirror firstVisibleIdx into a ref so the videos-ready effect doesn't have
   // to depend on it. If it did, hiding the first camera would tear the whole
@@ -166,6 +182,7 @@ export const SimpleVideosPlayer = ({
       if (resolved) return;
       resolved = true;
       setVideosReady(true);
+      seek(pendingSeekRef.current.time);
       onVideosReadyRef.current?.();
       setIsPlaying(true);
     };
@@ -426,10 +443,7 @@ export const SimpleVideosPlayer = ({
         targetTime = (info.segmentStart ?? 0) + currentTime;
       }
 
-      if (
-        Math.abs(video.currentTime - targetTime) >
-        THRESHOLDS.VIDEO_SYNC_TOLERANCE
-      ) {
+      if (video.currentTime !== targetTime) {
         video.currentTime = targetTime;
       }
     });
@@ -437,6 +451,29 @@ export const SimpleVideosPlayer = ({
 
   return (
     <>
+      {handTracking && (
+        <div className="flex flex-wrap items-center gap-3 text-xs text-fg-muted">
+          <label className="flex items-center gap-2 text-fg-soft">
+            <input
+              type="checkbox"
+              checked={showHands}
+              disabled={!handTracking.cameraKey || !handTracking.calibration}
+              onChange={(e) => setShowHands(e.target.checked)}
+            />
+            Hand overlay
+          </label>
+          <span>
+            {handTracking.calibration && handTracking.cameraKey
+              ? handTracking.cameraKey
+              : handTracking.overlayIssue}
+          </span>
+          {handTracking.calibration?.estimated && (
+            <span>
+              Estimated camera calibration · metric accuracy unvalidated
+            </span>
+          )}
+        </div>
+      )}
       {/* Hidden videos menu */}
       {hiddenVideos.length > 0 && (
         <div className="relative mb-4">
@@ -573,6 +610,15 @@ export const SimpleVideosPlayer = ({
                     videoEl={videoEls[idx] ?? null}
                     cameraKey={info.filename}
                   />
+                  {handTracking && (
+                    <HandOverlayCanvas
+                      video={videoEls[idx] ?? null}
+                      tracking={handTracking}
+                      cameraKey={info.filename}
+                      segmentStart={info.isSegmented ? info.segmentStart : 0}
+                      enabled={showHands}
+                    />
+                  )}
                 </div>
               </div>
             </div>

src/components/urdf-viewer.tsx

@@ -18,10 +18,8 @@ import {
 import * as THREE from "three";
 import URDFLoader from "urdf-loader";
 import type { URDFRobot } from "urdf-loader";
+import { loadUsd } from "@huggingface/usd-web-viewer";
 import { STLLoader } from "three/examples/jsm/loaders/STLLoader.js";
-import { ColladaLoader } from "three/examples/jsm/loaders/ColladaLoader.js";
-import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader.js";
-import { MeshoptDecoder } from "three/examples/jsm/libs/meshopt_decoder.module.js";
 import { Line2 } from "three/examples/jsm/lines/Line2.js";
 import { LineMaterial } from "three/examples/jsm/lines/LineMaterial.js";
 import { LineGeometry } from "three/examples/jsm/lines/LineGeometry.js";
@@ -59,136 +57,6 @@ function loadStlGeometry(url: string): Promise<THREE.BufferGeometry> {
   return loading;
 }
 
-/**
- * Rebuilds every mesh with a MeshStandardMaterial bucketed into one of three archetypes (carbon-black,
- * brushed metal, off-white paint) by the original base-colour lightness. `flatShading` is for the GLB
- * meshes, which carry no normals.
- */
-function applyOpenArmMaterials(root: THREE.Object3D, flatShading: boolean) {
-  root.traverse((child) => {
-    if (!(child instanceof THREE.Mesh) || !child.material) return;
-
-    const originals = Array.isArray(child.material)
-      ? child.material
-      : [child.material];
-
-    const rebuilt = originals.map((orig) => {
-      const srcColor =
-        (orig as THREE.MeshStandardMaterial).color ??
-        new THREE.Color("#c0c4cc");
-      const hsl = { h: 0, s: 0, l: 0 };
-      srcColor.getHSL(hsl);
-
-      // Archetype classification by original lightness.
-      let color: THREE.Color;
-      let metalness: number;
-      let roughness: number;
-      let envMapIntensity: number;
-      if (hsl.l < 0.3) {
-        // Carbon / anodised structural parts
-        color = new THREE.Color().setHSL(hsl.h, 0.02, 0.09);
-        metalness = 0.15;
-        roughness = 0.75;
-        envMapIntensity = 0.6;
-      } else if (hsl.l < 0.7) {
-        // Brushed metal joint collars / accents
-        color = new THREE.Color().setHSL(hsl.h, 0.04, 0.42);
-        metalness = 0.75;
-        roughness = 0.35;
-        envMapIntensity = 1.1;
-      } else {
-        // Off-white painted plates
-        color = new THREE.Color().setHSL(hsl.h, 0.03, 0.6);
-        metalness = 0.1;
-        roughness = 0.5;
-        envMapIntensity = 0.9;
-      }
-
-      const mat = new THREE.MeshStandardMaterial({
-        color,
-        metalness,
-        roughness,
-        envMapIntensity,
-        side: THREE.DoubleSide,
-        flatShading,
-      });
-      orig.dispose?.();
-      return mat;
-    });
-
-    child.material = Array.isArray(child.material) ? rebuilt : rebuilt[0];
-    child.castShadow = true;
-    child.receiveShadow = true;
-  });
-}
-
-/// One download and parse per DAE file per session: the OpenArm URDF references finger.dae 4 times
-/// and each arm link twice, and the viewer remounts on every visit to the 3D Replay tab.
-const daeSceneCache = new Map<string, Promise<THREE.Object3D>>();
-
-function loadDaeScene(url: string): Promise<THREE.Object3D> {
-  let loading = daeSceneCache.get(url);
-  if (!loading) {
-    loading = new ColladaLoader()
-      .loadAsync(url)
-      .then((collada) => {
-        // DAE (Collada) files — ColladaLoader yields whatever the .dae author
-        // baked in: flat MeshPhongMaterial/MeshBasicMaterial colors plus, in
-        // OpenArm's case, ~23 per-file PointLight/SpotLight nodes. The stray
-        // lights caused the scene to look pure-white everywhere regardless of
-        // our own lighting, and the flat materials looked cartoonish. We strip
-        // both and rebuild every mesh with a MeshStandardMaterial bucketed into
-        // one of three archetypes (carbon-black, brushed metal, off-white paint)
-        // based on the original base-color lightness.
-        if (url.includes("openarm")) {
-          const strayLights: THREE.Object3D[] = [];
-          collada.scene.traverse((child) => {
-            if (
-              (child as THREE.Light).isLight &&
-              !(child instanceof THREE.AmbientLight)
-            ) {
-              strayLights.push(child);
-            }
-          });
-          for (const l of strayLights) l.parent?.remove(l);
-
-          applyOpenArmMaterials(collada.scene, false);
-        }
-        return collada.scene;
-      })
-      .catch((err) => {
-        daeSceneCache.delete(url);
-        throw err;
-      });
-    daeSceneCache.set(url, loading);
-  }
-  return loading;
-}
-
-/// The bucket holds meshopt-compressed GLB copies of the SO-101 and OpenArm meshes, about 16x and 20x
-/// smaller than the STL/DAE originals. They carry no normals: the originals' are flat per face, and
-/// flatShading derives the same ones.
-const glbSceneCache = new Map<string, Promise<THREE.Object3D>>();
-
-function loadGlbScene(url: string): Promise<THREE.Object3D> {
-  let loading = glbSceneCache.get(url);
-  if (!loading) {
-    loading = new GLTFLoader()
-      .setMeshoptDecoder(MeshoptDecoder)
-      .loadAsync(url)
-      .then((gltf) => {
-        if (url.includes("openarm")) applyOpenArmMaterials(gltf.scene, true);
-        return gltf.scene;
-      })
-      .catch((err) => {
-        glbSceneCache.delete(url);
-        throw err;
-      });
-    glbSceneCache.set(url, loading);
-  }
-  return loading;
-}
-
 // URDFs + meshes are hosted in the Hub bucket at
 // https://huggingface.co/buckets/lerobot/robot-urdfs. URDFLoader resolves
 // relative mesh paths against the URDF's own URL, so the bucket layout
@@ -222,34 +90,90 @@ function loadUrdfText(url: string): Promise<string> {
   return loading;
 }
 
+/// USD models of the robots with their URDF's joint and link names, link frames and zero pose: the
+/// URDF moves the robot and the USD draws it.
+const USD_BASE_URL =
+  "https://huggingface.co/datasets/lerobot/robot-usds/resolve/main";
+
+/// Kept for the session like the URDF text: the viewer remounts on every visit to the 3D Replay tab.
+const usdBlobCache = new Map<string, Promise<Blob>>();
+
+function loadUsdBlob(url: string): Promise<Blob> {
+  let loading = usdBlobCache.get(url);
+  if (!loading) {
+    loading = fetch(url)
+      .then((res) => {
+        if (!res.ok)
+          throw new Error(`Failed to load USD ${url} (HTTP ${res.status})`);
+        return res.blob();
+      })
+      .catch((err) => {
+        usdBlobCache.delete(url);
+        throw err;
+      });
+    usdBlobCache.set(url, loading);
+  }
+  return loading;
+}
+
+/**
+ * Moves each USD mesh under the URDF link its prim path names, where the URDF's zero pose puts it.
+ * USD stage space is the URDF's root frame, so the robot's joints then move the meshes.
+ */
+function attachUsdMeshes(robot: URDFRobot, usdRoot: THREE.Object3D) {
+  for (const name of Object.keys(robot.joints)) robot.setJointValue(name, 0);
+  robot.updateMatrixWorld(true);
+  for (const mesh of [...usdRoot.children]) {
+    const segments = mesh.name.split("/");
+    let link: THREE.Object3D = robot;
+    for (let i = segments.length - 1; i >= 0; i--) {
+      if (robot.links[segments[i]]) {
+        link = robot.links[segments[i]];
+        break;
+      }
+    }
+    mesh.matrix
+      .premultiply(robot.matrixWorld)
+      .premultiply(link.matrixWorld.clone().invert());
+    link.add(mesh);
+  }
+}
+
 /**
  * Starts downloading a robot's visual meshes into the caches. The viewer only mounts once the episode
  * data has loaded, and the meshes used to start downloading only then; calling this earlier overlaps
  * the two.
  */
 export function prefetchRobotModel(robotType: string | null) {
-  const { urdfUrl } = getRobotConfig(robotType);
+  const { urdfUrl, usdUrl } = getRobotConfig(robotType);
   if (prefetchedUrdfs.has(urdfUrl)) return;
   prefetchedUrdfs.add(urdfUrl);
+  if (usdUrl) {
+    Promise.all([loadUrdfText(urdfUrl), loadUsdBlob(usdUrl)]).catch(() =>
+      prefetchedUrdfs.delete(urdfUrl),
+    );
+    return;
+  }
   const base = THREE.LoaderUtils.extractUrlBase(urdfUrl);
   loadUrdfText(urdfUrl)
     .then((text) => {
       const doc = new DOMParser().parseFromString(text, "application/xml");
       for (const mesh of doc.querySelectorAll("visual mesh")) {
         const filename = mesh.getAttribute("filename");
-        /// Same resolution URDFLoader applies to relative paths; package:// and DAE meshes are left to it.
+        /// Same resolution URDFLoader applies to relative paths; package:// meshes are left to it.
         if (!filename || filename.startsWith("package://")) continue;
-        const lower = filename.toLowerCase();
-        if (lower.endsWith(".glb"))
-          loadGlbScene(base + filename).catch(() => {});
-        else if (lower.endsWith(".stl"))
+        if (filename.toLowerCase().endsWith(".stl"))
           loadStlGeometry(base + filename).catch(() => {});
       }
     })
     .catch(() => prefetchedUrdfs.delete(urdfUrl));
 }
 
-function getRobotConfig(robotType: string | null) {
+function getRobotConfig(robotType: string | null): {
+  urdfUrl: string;
+  usdUrl?: string;
+  scale: number;
+} {
   const lower = (robotType ?? "").toLowerCase();
   if (lower.includes("g1") || lower.includes("unitree")) {
     return { urdfUrl: `${URDF_BASE_URL}/g1/g1_body29_hand14.urdf`, scale: 1 };
@@ -257,11 +181,13 @@ function getRobotConfig(robotType: string | null) {
   if (lower.includes("openarm")) {
     return {
       urdfUrl: `${URDF_BASE_URL}/openarm/openarm_bimanual_web.urdf`,
+      usdUrl: `${USD_BASE_URL}/openarm_bimanual/simready_usd/openarm_bimanual.usd`,
       scale: 3,
     };
   }
   return {
     urdfUrl: `${URDF_BASE_URL}/so101/so101_new_calib_web.urdf`,
+    usdUrl: `${USD_BASE_URL}/so101_new_calib/simready_usd/so101_new_calib.usd`,
     scale: 10,
   };
 }
@@ -353,11 +279,14 @@ function autoMatchJoints(
       continue;
     }
 
-    // OpenArm: openarm_(left|right)_joint(\d+) → (left|right)_joint_(\d+)
+    // OpenArm: openarm_(left|right)_joint(\d+) → (left|right)_joint_(\d+) or (left|right)_joint(\d+),
+    // whole words so joint1 doesn't take joint10
     const armMatch = lower.match(/^openarm_(left|right)_joint(\d+)$/);
     if (armMatch) {
-      const pattern = `${armMatch[1]}_joint_${armMatch[2]}`;
-      const idx = suffixes.findIndex((s) => s.includes(pattern));
+      const pattern = new RegExp(
+        `(^|_)${armMatch[1]}_joint_?${armMatch[2]}(\\D|$)`,
+      );
+      const idx = suffixes.findIndex((s) => pattern.test(s));
       if (idx >= 0) {
         mapping[jointName] = columnKeys[idx];
         continue;
@@ -430,6 +359,7 @@ const MAX_FIT_WAIT_FRAMES = 30;
 // ─── Robot scene (imperative, inside Canvas) ───
 function RobotScene({
   urdfUrl,
+  usdUrl,
   jointValues,
   onJointsLoaded,
   trailEnabled,
@@ -437,6 +367,7 @@ function RobotScene({
   scale,
 }: {
   urdfUrl: string;
+  usdUrl?: string;
   jointValues: Record<string, number>;
   onJointsLoaded: (names: string[]) => void;
   trailEnabled: boolean;
@@ -537,20 +468,22 @@ function RobotScene({
     //   `if (obj instanceof THREE.Mesh) obj.material = material;`
     // on every mesh we hand back — overwriting our PBR material with the
     // URDF's <material rgba="...">-derived MeshPhongMaterial. Wrapping the
-    // returned mesh in a Group dodges that check (same way DAE's collada.scene
-    // already avoids it) so our carefully-tuned materials survive.
+    // returned mesh in a Group dodges that check so our materials survive.
     const wrapForUrdf = (obj: THREE.Object3D): THREE.Object3D => {
       const group = new THREE.Group();
       group.add(obj);
       return group;
     };
     loader.loadMeshCb = (url, mgr, onLoad) => {
-      const makeMaterial = (flatShading: boolean) => {
-        // Defaults: neutral off-white plastic, matches OpenArm "light" archetype
+      /// The USD draws these robots: their URDF only moves it.
+      if (usdUrl) {
+        onLoad(new THREE.Object3D());
+        return;
+      }
+      const makeMaterial = () => {
         let color = "#9ba1ab";
         let metalness = 0.1;
         let roughness = 0.5;
-        let side: THREE.Side = THREE.FrontSide;
         if (isG1) {
           const lower = url.toLowerCase();
           const isWhitePart =
@@ -563,54 +496,9 @@ function RobotScene({
           color = isWhitePart ? "#9ca3af" : "#1f2937";
           metalness = 0.25;
           roughness = 0.6;
-        } else if (url.includes("sts3215")) {
-          // SO-arm / any STL servo housing — carbon-black archetype
-          color = "#171a20";
-          metalness = 0.15;
-          roughness = 0.75;
-        } else if (isOpenArm) {
-          color = url.includes("body_link0") ? "#3a3a4a" : "#f5f5f5";
-          metalness = 0.15;
-          roughness = 0.6;
-          side = THREE.DoubleSide;
         }
-        return new THREE.MeshStandardMaterial({
-          color,
-          metalness,
-          roughness,
-          side,
-          flatShading,
-        });
+        return new THREE.MeshStandardMaterial({ color, metalness, roughness });
       };
-      const makeMesh = (geometry: THREE.BufferGeometry) =>
-        new THREE.Mesh(geometry, makeMaterial(false));
-      if (url.endsWith(".glb")) {
-        mgr.itemStart(url);
-        loadGlbScene(url)
-          .then((glbScene) => {
-            const obj = glbScene.clone();
-            if (!isOpenArm) {
-              obj.traverse((child) => {
-                if (child instanceof THREE.Mesh)
-                  child.material = makeMaterial(true);
-              });
-            }
-            onLoad(obj);
-          })
-          .catch((err) => onLoad(new THREE.Object3D(), err as Error))
-          .finally(() => mgr.itemEnd(url));
-        return;
-      }
-      if (url.endsWith(".dae")) {
-        /// URDFLoader re-parents and re-orients what it is handed, so it gets a clone (sharing
-        /// geometry and materials) and the cached scene stays intact.
-        mgr.itemStart(url);
-        loadDaeScene(url)
-          .then((daeScene) => onLoad(daeScene.clone()))
-          .catch((err) => onLoad(new THREE.Object3D(), err as Error))
-          .finally(() => mgr.itemEnd(url));
-        return;
-      }
       // STL files — apply final PBR materials directly here. We used to do a
       // post-load archetype rebuild in manager.onLoad, but STLLoader calls
       // `manager.itemEnd` *before* our Promise resolves — so when the last
@@ -628,7 +516,9 @@ function RobotScene({
       /// with every mesh prefetched it would otherwise never fire.
       mgr.itemStart(url);
       loadStlGeometry(url)
-        .then((geometry) => onLoad(wrapForUrdf(makeMesh(geometry))))
+        .then((geometry) =>
+          onLoad(wrapForUrdf(new THREE.Mesh(geometry, makeMaterial()))),
+        )
         .catch((err) => onLoad(new THREE.Object3D(), err as Error))
         .finally(() => mgr.itemEnd(url));
     };
@@ -654,6 +544,7 @@ function RobotScene({
 
     loader.workingPath = THREE.LoaderUtils.extractUrlBase(urdfUrl);
     let cancelled = false;
+    let disposeUsd: (() => void) | undefined;
     manager.itemStart(urdfUrl);
     loadUrdfText(urdfUrl)
       .then((text) => {
@@ -700,11 +591,24 @@ function RobotScene({
               j.jointType === "prismatic",
           )
           .map((j) => j.name);
-        onJointsLoaded(movable);
+        if (!usdUrl) {
+          onJointsLoaded(movable);
+          return;
+        }
+        /// Loaded from memory, so the session cache spares a download on every visit to the tab.
+        const name = usdUrl.slice(usdUrl.lastIndexOf("/") + 1);
+        return loadUsdBlob(usdUrl)
+          .then((blob) => loadUsd(name, { files: new Map([[name, blob]]) }))
+          .then(({ root, dispose }) => {
+            if (cancelled) return dispose();
+            disposeUsd = dispose;
+            attachUsdMeshes(robot, root);
+            onJointsLoaded(movable);
+          });
       })
       .catch((err) => {
         if (cancelled) return;
-        console.error("Error loading URDF:", err);
+        console.error("Error loading the robot model:", err);
         setError(String(err));
       })
       .finally(() => {
@@ -716,9 +620,10 @@ function RobotScene({
         scene.remove(robotRef.current);
         robotRef.current = null;
       }
+      disposeUsd?.();
       tipLinksRef.current = [];
     };
-  }, [urdfUrl, scale, scene, onJointsLoaded, ensureTrails, invalidate]);
+  }, [urdfUrl, usdUrl, scale, scene, onJointsLoaded, ensureTrails, invalidate]);
 
   const tipWorldPos = useMemo(() => new THREE.Vector3(), []);
 
@@ -811,7 +716,7 @@ function RobotScene({
   if (error)
     return (
       <Html center>
-        <span className="text-red-400">Failed to load URDF</span>
+        <span className="text-red-400">Failed to load the 3D model</span>
       </Html>
     );
   return null;
@@ -921,7 +826,7 @@ export default function URDFViewer({
     () => getRobotConfig(datasetInfo.robot_type),
     [datasetInfo.robot_type],
   );
-  const { urdfUrl, scale } = robotConfig;
+  const { urdfUrl, usdUrl, scale } = robotConfig;
   const isG1 = urdfUrl.includes("g1");
   const isOpenArm = urdfUrl.includes("openarm");
   const repoId = org && dataset ? `${org}/${dataset}` : null;
@@ -1222,6 +1127,7 @@ export default function URDFViewer({
           </mesh>
           <RobotScene
             urdfUrl={urdfUrl}
+            usdUrl={usdUrl}
             jointValues={jointValues}
             onJointsLoaded={onJointsLoaded}
             trailEnabled={trailEnabled}

src/lib/__tests__/hand-tracking.test.ts

@@ -0,0 +1,317 @@
+import { describe, expect, test } from "bun:test";
+import {
+  fluToScene,
+  cameraPathRuns,
+  handFrameIndex,
+  handProjectionMetadata,
+  hasHandTracking,
+  parseHandCalibration,
+  projectHandPoint,
+  readHandTracking,
+  wristTrail,
+} from "../hand-tracking";
+import type { DatasetMetadata } from "@/utils/parquetUtils";
+
+function info(): DatasetMetadata {
+  const widths = {
+    action: 156,
+    "annotation.hand_joints_camera": 126,
+    "annotation.hand_joints_world": 126,
+    "annotation.hand_action_valid": 1,
+    "annotation.hand_observed": 2,
+    "annotation.hand_reference_frame": 1,
+  };
+  return {
+    codebase_version: "v3.0",
+    robot_type: "human_egocentric",
+    fps: 10,
+    total_episodes: 1,
+    total_frames: 6,
+    total_tasks: 1,
+    total_videos: 1,
+    total_chunks: 1,
+    chunks_size: 1000,
+    splits: {},
+    data_path: "",
+    video_path: "",
+    features: {
+      ...Object.fromEntries(
+        Object.entries(widths).map(([key, width]) => [
+          key,
+          { dtype: "float32", shape: [width], names: null },
+        ]),
+      ),
+      "observation.images.ego": {
+        dtype: "video",
+        shape: [480, 640, 3],
+        names: null,
+      },
+    },
+  };
+}
+const calibration = {
+  model: "pinhole",
+  width: 640,
+  height: 480,
+  fx: 600,
+  fy: 610,
+  cx: 320,
+  cy: 240,
+  distortion: [],
+};
+const metadata = {
+  cameraKey: "observation.images.ego",
+  calibration: parseHandCalibration(calibration),
+  overlayIssue: null,
+};
+function numericFrames(): {
+  length: number;
+  timestamps: number[];
+  series: Record<string, number[][]>;
+} {
+  const vector = () =>
+    Array.from({ length: 126 }, (_, component) =>
+      Array.from({ length: 6 }, () =>
+        component % 3 === 2 ? 1 : component / 100,
+      ),
+    );
+  return {
+    length: 6,
+    timestamps: [20, 20.1, 20.2, 20.3, 20.4, 20.5],
+    series: {
+      "annotation.hand_joints_camera": vector(),
+      "annotation.hand_joints_world": vector(),
+      "annotation.hand_action_valid": [[1, 1, 0, 1, 1, 1]],
+      "annotation.hand_observed": [
+        [1, 1, 1, 1, 1, 1],
+        [1, 1, 1, 1, 1, 1],
+      ],
+      "annotation.hand_reference_frame": [[0, 0, -1, 3, 3, 5]],
+    },
+  };
+}
+
+describe("native hand schema and time alignment", () => {
+  test("an arbitrary 156D action is not detected as hand tracking", () => {
+    const metadata = info();
+    expect(hasHandTracking(metadata)).toBe(true);
+    delete metadata.features["annotation.hand_joints_camera"];
+    expect(hasHandTracking(metadata)).toBe(false);
+  });
+  test("all source frames survive, with local timestamps and independently masked hands", () => {
+    const frames = numericFrames();
+    frames.series["annotation.hand_observed"][1][1] = 0;
+    const data = readHandTracking(info(), frames, metadata)!;
+    expect(data.frames).toHaveLength(6);
+    expect(data.frames[1].time).toBeCloseTo(0.1);
+    expect(data.frames[0].camera[0]).toHaveLength(21);
+    expect(data.frames[1].camera[1]).toBeNull();
+    expect(data.frames[2].camera).toEqual([null, null]);
+    expect(data.frames[2].segment).toEqual([null, null]);
+  });
+  test("a non-finite point hides the affected hand in both views", () => {
+    const frames = numericFrames();
+    frames.series["annotation.hand_joints_camera"][0][0] = NaN;
+    const data = readHandTracking(info(), frames, metadata)!;
+    expect(data.frames[0].camera[0]).toBeNull();
+    expect(data.frames[0].segment[0]).toBeNull();
+    expect(data.frames[0].segment[1]).not.toBeNull();
+  });
+  test("selects the displayed frame without interpolating or looking ahead", () => {
+    const frames = readHandTracking(info(), numericFrames(), metadata)!.frames;
+    expect(handFrameIndex(frames, 0.099)).toBe(0);
+    expect(handFrameIndex(frames, 0.1)).toBe(1);
+    expect(handFrameIndex(frames, 0.2)).toBe(2);
+    // A frame at 0.2 s lives at media time 12.7 s in a shared video starting at 12.5 s.
+    expect(handFrameIndex(frames, 12.7 - 12.5)).toBe(2);
+    expect(handFrameIndex(frames, -1)).toBe(-1);
+  });
+  test("trails stop at gaps and segment-reference resets", () => {
+    const frames = readHandTracking(info(), numericFrames(), metadata)!.frames;
+    expect(wristTrail(frames, 1, 0)).toHaveLength(2);
+    expect(wristTrail(frames, 2, 0)).toHaveLength(0);
+    expect(wristTrail(frames, 4, 0)).toHaveLength(2);
+    expect(wristTrail(frames, 5, 0)).toHaveLength(1);
+    expect(fluToScene([1, 2, 3])).toEqual([-2, 3, -1]);
+  });
+});
+
+describe("recording-wide camera and wrist coordinates", () => {
+  const rigid = (x = 0) => [1, 0, 0, x, 0, 1, 0, 20, 0, 0, 1, 30, 0, 0, 0, 1];
+  const withPoses = (poses: number[][]) => {
+    const frames = numericFrames();
+    frames.series["annotation.camera_world_from_optical"] = Array.from(
+      { length: 16 },
+      (_, a) => poses.map((pose) => pose[a]),
+    );
+    return readHandTracking(info(), frames, metadata)!;
+  };
+  test("removes the first camera's translation and rotation from the recording frame", () => {
+    const pose = [0, -1, 0, 10, 1, 0, 0, 20, 0, 0, 1, 30, 0, 0, 0, 1];
+    const moved = [...pose];
+    moved[7] += 1;
+    const tracking = withPoses([pose, moved, pose, pose, pose, pose]);
+    expect(tracking.coordinateFrame).toBe("recording");
+    expect(tracking.frames[0].cameraPose?.position).toEqual([0, -0, -0]);
+    expect(tracking.frames[1].cameraPose?.position).toEqual([0, -1, -0]);
+    expect(tracking.frames[0].recording[0]?.[0][0]).toBeCloseTo(1);
+    expect(tracking.frames[1].recording[0]?.[0][1]).toBeCloseTo(-1);
+    expect(tracking.frames[1].recording[0]?.[0][2]).toBeCloseTo(-0.01);
+  });
+  test("rotates optical joints with each camera pose instead of displaying span-local joints as global", () => {
+    const rotated = [0, -1, 0, 1, 1, 0, 0, 20, 0, 0, 1, 30, 0, 0, 0, 1];
+    const tracking = withPoses([
+      rigid(),
+      rotated,
+      rigid(),
+      rigid(),
+      rigid(),
+      rigid(),
+    ]);
+    expect(tracking.frames[1].recording[0]?.[0][1]).toBeCloseTo(-0.99);
+    expect(tracking.frames[1].recording[0]?.[0][2]).toBeCloseTo(0);
+    expect(tracking.frames[1].cameraPose?.axes[0]).toEqual([0, 0, -1]);
+    // Reference frame changes at frame 5, but the saved global trajectory remains continuous.
+    expect(wristTrail(tracking.frames, 5, 0, 2, "recording")).toHaveLength(3);
+    expect(wristTrail(tracking.frames, 5, 0)).toHaveLength(1);
+  });
+  test("zero, reflected and non-rigid camera placeholders split the trajectory and hide global hands", () => {
+    const reflection = rigid();
+    reflection[0] = -1;
+    const scaled = rigid();
+    scaled[0] = 2;
+    const tracking = withPoses([
+      rigid(),
+      rigid(1),
+      Array(16).fill(0),
+      reflection,
+      scaled,
+      rigid(2),
+    ]);
+    expect(cameraPathRuns(tracking.frames, 5).map((run) => run.length)).toEqual(
+      [2, 1],
+    );
+    for (const i of [2, 3, 4]) {
+      expect(tracking.frames[i].cameraPose).toBeNull();
+      expect(tracking.frames[i].recording).toEqual([null, null]);
+    }
+    expect(tracking.frames[5].recording[0]).not.toBeNull();
+  });
+  test("older native exports without saved camera transforms retain span-local 3D", () => {
+    const tracking = readHandTracking(info(), numericFrames(), metadata)!;
+    expect(tracking.coordinateFrame).toBe("segment");
+    expect(tracking.frames[0].segment[0]).not.toBeNull();
+    expect(cameraPathRuns(tracking.frames, 5)).toEqual([]);
+  });
+});
+
+describe("episode-specific source camera binding", () => {
+  test("an incomplete episode mapping cannot select an unindexed report entry", () => {
+    const selected = handProjectionMetadata(info(), 0, {
+      source_episode_map: [{ episode_index: 1, source_episode_index: 7 }],
+      episodes: [{ hand_actions: { camera_intrinsics: calibration } }],
+    });
+    expect(selected.calibration).toBeNull();
+    expect(selected.overlayIssue).toContain("No camera calibration");
+  });
+  test("uses the source episode map, not completion order or output index", () => {
+    const report = {
+      manifest: { camera: "observation.images.ego" },
+      source_episode_map: [{ episode_index: 0, source_episode_index: 7 }],
+      episodes: [
+        {
+          episode_index: 0,
+          hand_actions: { camera_intrinsics: { ...calibration, fx: 100 } },
+        },
+        {
+          episode_index: 7,
+          hand_actions: {
+            camera_intrinsics: { ...calibration, fx: 700, measured: false },
+          },
+        },
+      ],
+    };
+    const selected = handProjectionMetadata(info(), 0, report);
+    expect(selected.calibration?.fx).toBe(700);
+    expect(selected.calibration?.estimated).toBe(true);
+  });
+  test("never projects source-camera coordinates onto another camera", () => {
+    expect(
+      handProjectionMetadata(info(), 0, {
+        manifest: { camera: "missing.camera" },
+      }).cameraKey,
+    ).toBeNull();
+    const data = info();
+    data.features["observation.images.other"] =
+      data.features["observation.images.ego"];
+    expect(handProjectionMetadata(data, 0, null).cameraKey).toBeNull();
+  });
+  test("missing calibration keeps 3D available and explains why overlays are unavailable", () => {
+    const selected = handProjectionMetadata(info(), 0, null);
+    expect(selected.calibration).toBeNull();
+    expect(selected.overlayIssue).toContain("No camera calibration");
+    expect(
+      readHandTracking(info(), numericFrames(), selected)?.frames[0].segment[0],
+    ).not.toBeNull();
+  });
+  test("accepts measured calibration JSON stored on the video feature", () => {
+    const data = info();
+    data.features["observation.images.ego"].info = {
+      "camera.intrinsics": JSON.stringify(calibration),
+    };
+    expect(handProjectionMetadata(data, 0, null).calibration?.fx).toBe(600);
+    expect(parseHandCalibration({ ...calibration, fx: Infinity })).toBeNull();
+  });
+  test("rejects malformed calibration at the unknown-data boundary", () => {
+    for (const value of [
+      null,
+      [],
+      "invalid JSON",
+      { ...calibration, width: 640.5 },
+      { ...calibration, fx: "600" },
+      { ...calibration, distortion: [0, 0, 0, "0"] },
+    ]) {
+      expect(parseHandCalibration(value)).toBeNull();
+    }
+  });
+});
+
+describe("source camera projection", () => {
+  // Reference outputs generated with OpenCV projectPoints, not the implementation under test.
+  const d = [
+    0.1, -0.02, 0.003, -0.004, 0.01, 0.02, -0.01, 0.005, 0.001, -0.002, 0.003,
+    -0.004, 0.02, -0.03,
+  ];
+  const expected = [
+    [0, 360, 321.3333333333333],
+    [4, 360.04582716049384, 321.57562633744857],
+    [5, 360.04583155006856, 321.5756352629172],
+    [8, 360.02821795025153, 321.53982094328916],
+    [12, 360.0409586909922, 321.5792826716842],
+    [14, 360.2472660915072, 322.00366115201376],
+  ];
+  for (const [count, x, y] of expected)
+    test(`pinhole ${count}-coefficient projection matches OpenCV`, () => {
+      const c = parseHandCalibration({
+        ...calibration,
+        distortion: d.slice(0, count),
+      })!;
+      const p = projectHandPoint([0.1, 0.2, 1.5], c)!;
+      expect(p[0]).toBeCloseTo(x, 8);
+      expect(p[1]).toBeCloseTo(y, 8);
+    });
+  test("fisheye projection matches OpenCV", () => {
+    const c = parseHandCalibration({
+      ...calibration,
+      model: "opencv_fisheye",
+      distortion: d.slice(0, 4),
+    })!;
+    const p = projectHandPoint([0.1, 0.2, 1.5], c)!;
+    expect(p[0]).toBeCloseTo(359.7941667826154, 8);
+    expect(p[1]).toBeCloseTo(320.9148057913179, 8);
+  });
+  test("points behind the image plane never appear", () => {
+    expect(projectHandPoint([0, 0, -1], metadata.calibration!)).toBeNull();
+    expect(projectHandPoint([0, 0, 0], metadata.calibration!)).toBeNull();
+  });
+});

src/lib/hand-tracking.ts

@@ -0,0 +1,447 @@
+import type { DatasetMetadata } from "@/utils/parquetUtils";
+
+export type Point3 = [number, number, number];
+export type HandIndex = 0 | 1;
+export type HandPair = [Point3[] | null, Point3[] | null];
+export type CameraPose = { position: Point3; axes: [Point3, Point3, Point3] };
+export type HandFrame = {
+  time: number;
+  referenceFrame: number;
+  camera: HandPair;
+  segment: HandPair;
+  recording: HandPair;
+  cameraPose: CameraPose | null;
+};
+export type HandCalibration = {
+  model: "pinhole" | "opencv_fisheye";
+  width: number;
+  height: number;
+  fx: number;
+  fy: number;
+  cx: number;
+  cy: number;
+  distortion: number[];
+  estimated: boolean;
+};
+export type HandTracking = {
+  frames: HandFrame[];
+  fps: number;
+  cameraKey: string | null;
+  calibration: HandCalibration | null;
+  overlayIssue: string | null;
+  coordinateFrame: "recording" | "segment";
+};
+type NumericFrames = {
+  length: number;
+  timestamps: number[];
+  series: Record<string, number[][]>;
+};
+
+export const HAND_COLORS = ["#59b5ff", "#ffb454"] as const;
+export const HAND_INDICES = [0, 1] as const;
+export const HAND_EDGES = [1, 5, 9, 13, 17].flatMap(
+  (base): [number, number][] => [
+    [0, base],
+    [base, base + 1],
+    [base + 1, base + 2],
+    [base + 2, base + 3],
+  ],
+);
+const HAND_WIDTHS: Record<string, number> = {
+  "annotation.hand_joints_camera": 126,
+  "annotation.hand_joints_world": 126,
+  "annotation.hand_action_valid": 1,
+  "annotation.hand_observed": 2,
+  "annotation.hand_reference_frame": 1,
+};
+
+/** The SOMA156 contract, rather than a robot name or an arbitrary 156D action. */
+export function hasHandTracking(info: DatasetMetadata): boolean {
+  return (
+    info.features.action?.shape.join() === "156" &&
+    Object.entries(HAND_WIDTHS).every(
+      ([key, width]) => info.features[key]?.shape.join() === String(width),
+    )
+  );
+}
+
+/** Native auxiliary arrays have their own viewer, rather than hundreds of chart series. */
+export function isHandGeometryColumn(key: string): boolean {
+  return (
+    key.startsWith("annotation.hand_") ||
+    key.startsWith("annotation.soma_") ||
+    key === "annotation.wrist_rotation_camera" ||
+    key === "annotation.camera_world_from_optical"
+  );
+}
+
+function record(value: unknown): Record<string, unknown> {
+  return value !== null && typeof value === "object" && !Array.isArray(value)
+    ? (value as Record<string, unknown>)
+    : {};
+}
+
+function isFiniteNumber(value: unknown): value is number {
+  return typeof value === "number" && Number.isFinite(value);
+}
+
+export function parseHandCalibration(value: unknown): HandCalibration | null {
+  if (typeof value === "string") {
+    try {
+      value = JSON.parse(value);
+    } catch {
+      return null;
+    }
+  }
+  const c = record(value);
+  const { width, height, fx, fy, cx, cy } = c;
+  if (
+    !isFiniteNumber(width) ||
+    !Number.isInteger(width) ||
+    width <= 0 ||
+    !isFiniteNumber(height) ||
+    !Number.isInteger(height) ||
+    height <= 0 ||
+    !isFiniteNumber(fx) ||
+    fx <= 0 ||
+    !isFiniteNumber(fy) ||
+    fy <= 0 ||
+    !isFiniteNumber(cx) ||
+    !isFiniteNumber(cy)
+  )
+    return null;
+  if (c.model !== "pinhole" && c.model !== "opencv_fisheye") return null;
+  const d = c.distortion ?? [];
+  if (
+    !Array.isArray(d) ||
+    !d.every(isFiniteNumber) ||
+    !(c.model === "pinhole" ? [0, 4, 5, 8, 12, 14] : [4]).includes(d.length)
+  )
+    return null;
+  return {
+    model: c.model,
+    width,
+    height,
+    fx,
+    fy,
+    cx,
+    cy,
+    distortion: d,
+    estimated: c.measured === false || c.source === "moge-estimated",
+  };
+}
+
+/** Run episodes are source IDs and can finish out of order; exports can be reindexed. */
+export function handProjectionMetadata(
+  info: DatasetMetadata,
+  episodeId: number,
+  reportValue: unknown,
+): Pick<HandTracking, "cameraKey" | "calibration" | "overlayIssue"> {
+  const report = record(reportValue);
+  const manifest = record(report.manifest);
+  const cameras = Object.keys(info.features).filter(
+    (key) => info.features[key].dtype === "video",
+  );
+  const cameraKey =
+    typeof manifest.camera === "string"
+      ? cameras.includes(manifest.camera)
+        ? manifest.camera
+        : null
+      : cameras.length === 1
+        ? cameras[0]
+        : null;
+  const mapping = Array.isArray(report.source_episode_map)
+    ? report.source_episode_map.map(record)
+    : null;
+  const mapped = mapping?.find((entry) => entry.episode_index === episodeId);
+  const sourceId = mapping ? mapped?.source_episode_index : episodeId;
+  const episodes = Array.isArray(report.episodes)
+    ? report.episodes.map(record)
+    : [];
+  const sourceEpisode =
+    isFiniteNumber(sourceId) && Number.isSafeInteger(sourceId) && sourceId >= 0
+      ? episodes.find((entry) => entry.episode_index === sourceId)
+      : undefined;
+  const handResult = record(sourceEpisode?.hand_actions);
+  const calibration =
+    parseHandCalibration(handResult.camera_intrinsics) ??
+    parseHandCalibration(
+      cameraKey ? info.features[cameraKey].info?.["camera.intrinsics"] : null,
+    );
+  return {
+    cameraKey,
+    calibration,
+    overlayIssue: !cameraKey
+      ? "The annotation camera is unknown; camera overlays are unavailable."
+      : !calibration
+        ? "No camera calibration is saved for this episode; 3D hands remain available."
+        : null,
+  };
+}
+
+/** Keep the full source frame grid. Never turn masked zero placeholders into hands. */
+export function readHandTracking(
+  info: DatasetMetadata,
+  frames: NumericFrames | undefined,
+  metadata: ReturnType<typeof handProjectionMetadata>,
+): HandTracking | undefined {
+  if (!hasHandTracking(info)) return undefined;
+  const rows: HandFrame[] = [];
+  const cameraMatrices: (number[] | null)[] = [];
+  const timeOrigin = frames?.timestamps[0] ?? 0;
+  const points = (
+    key: string,
+    frame: number,
+    hand: HandIndex,
+  ): Point3[] | null => {
+    const components = frames?.series[key];
+    if (components?.length !== 126) return null;
+    const joints = Array.from({ length: 21 }, (_, joint): Point3 => {
+      const offset = hand * 63 + joint * 3;
+      return [
+        components[offset]?.[frame],
+        components[offset + 1]?.[frame],
+        components[offset + 2]?.[frame],
+      ];
+    });
+    return joints.every((p) => p.every(Number.isFinite)) ? joints : null;
+  };
+  if (
+    frames &&
+    frames.timestamps.length === frames.length &&
+    Number.isFinite(timeOrigin)
+  ) {
+    for (let i = 0; i < frames.length; i++) {
+      const time = frames.timestamps[i] - timeOrigin;
+      if (!Number.isFinite(time) || (i > 0 && time <= rows[i - 1].time))
+        return undefined;
+      const reference =
+        frames.series["annotation.hand_reference_frame"]?.[0]?.[i];
+      const valid =
+        frames.series["annotation.hand_action_valid"]?.[0]?.[i] === 1 &&
+        Number.isInteger(reference) &&
+        reference >= 0 &&
+        reference <= i;
+      const observed = (hand: HandIndex) =>
+        valid && frames.series["annotation.hand_observed"]?.[hand]?.[i] === 1;
+      const camera: HandPair = [null, null],
+        segment: HandPair = [null, null];
+      for (const hand of HAND_INDICES) {
+        if (!observed(hand)) continue;
+        const cameraPoints = points("annotation.hand_joints_camera", i, hand);
+        const segmentPoints = points("annotation.hand_joints_world", i, hand);
+        if (cameraPoints && segmentPoints) {
+          camera[hand] = cameraPoints;
+          segment[hand] = segmentPoints;
+        }
+      }
+      rows.push({
+        time,
+        referenceFrame: valid ? reference : -1,
+        camera,
+        segment,
+        recording: [null, null],
+        cameraPose: null,
+      });
+      const matrix = frames.series["annotation.camera_world_from_optical"];
+      const pose =
+        matrix?.length === 16 ? matrix.map((component) => component[i]) : null;
+      cameraMatrices.push(pose && isCameraTransform(pose) ? pose : null);
+    }
+  }
+  const origin = cameraMatrices.find((matrix) => matrix !== null);
+  if (origin)
+    rows.forEach((frame, i) => {
+      const pose = cameraMatrices[i];
+      if (!pose) return;
+      // Only saved rigid transforms are used. Masked zero matrices are not camera poses.
+      const relative = (p: Point3): Point3 => {
+        const world = [0, 1, 2].map(
+          (a) =>
+            pose[a * 4] * p[0] +
+            pose[a * 4 + 1] * p[1] +
+            pose[a * 4 + 2] * p[2] +
+            pose[a * 4 + 3] -
+            origin[a * 4 + 3],
+        );
+        const optical = [0, 1, 2].map(
+          (a) =>
+            origin[a] * world[0] +
+            origin[a + 4] * world[1] +
+            origin[a + 8] * world[2],
+        );
+        return [optical[2], -optical[0], -optical[1]];
+      };
+      const position = relative([0, 0, 0]);
+      const axis = (p: Point3): Point3 => {
+        const end = relative(p);
+        return [
+          end[0] - position[0],
+          end[1] - position[1],
+          end[2] - position[2],
+        ];
+      };
+      frame.cameraPose = {
+        position,
+        axes: [axis([1, 0, 0]), axis([0, 1, 0]), axis([0, 0, 1])],
+      };
+      frame.recording = [
+        frame.camera[0]?.map(relative) ?? null,
+        frame.camera[1]?.map(relative) ?? null,
+      ];
+    });
+  return {
+    frames: rows,
+    fps: info.fps,
+    coordinateFrame: origin ? "recording" : "segment",
+    ...metadata,
+  };
+}
+
+function isCameraTransform(m: number[]): boolean {
+  if (
+    !m.every(Number.isFinite) ||
+    [m[12], m[13], m[14], m[15] - 1].some((n) => Math.abs(n) > 1e-4)
+  )
+    return false;
+  for (let a = 0; a < 3; a++)
+    for (let b = 0; b < 3; b++) {
+      const dot = [0, 1, 2].reduce(
+        (sum, k) => sum + m[k * 4 + a] * m[k * 4 + b],
+        0,
+      );
+      if (Math.abs(dot - Number(a === b)) > 2e-3) return false;
+    }
+  const determinant =
+    m[0] * (m[5] * m[10] - m[6] * m[9]) -
+    m[1] * (m[4] * m[10] - m[6] * m[8]) +
+    m[2] * (m[4] * m[9] - m[5] * m[8]);
+  return Math.abs(determinant - 1) < 2e-3;
+}
+
+/** The frame currently on screen, without interpolation or looking ahead to the next pose. */
+export function handFrameIndex(frames: HandFrame[], time: number): number {
+  let lo = 0,
+    hi = frames.length;
+  while (lo < hi) {
+    const mid = (lo + hi) >>> 1;
+    if (frames[mid].time <= time + 1e-7) lo = mid + 1;
+    else hi = mid;
+  }
+  return lo - 1;
+}
+
+/** The stored "world" joints are segment-start FLU, not a continuous global trajectory. */
+export function fluToScene([forward, left, up]: Point3): Point3 {
+  return [-left, up, -forward];
+}
+
+/** Break trails both at validity gaps and at each segment-reference reset. */
+export function wristTrail(
+  frames: HandFrame[],
+  index: number,
+  hand: HandIndex,
+  seconds = 2,
+  space: "segment" | "recording" = "segment",
+): Point3[] {
+  const trail: Point3[] = [];
+  const current = frames[index];
+  if (!current?.[space][hand]) return trail;
+  for (let i = index; i >= 0; i--) {
+    const f = frames[i];
+    const joints = f[space][hand];
+    if (
+      !joints ||
+      (space === "segment" && f.referenceFrame !== current.referenceFrame) ||
+      current.time - f.time > seconds
+    )
+      break;
+    trail.push(fluToScene(joints[0]));
+  }
+  return trail.reverse();
+}
+
+/** Separate camera-path runs wherever no transform was saved; never draw through zero placeholders. */
+export function cameraPathRuns(frames: HandFrame[], index: number): Point3[][] {
+  const runs: Point3[][] = [];
+  let run: Point3[] = [];
+  for (let i = 0; i <= index; i++) {
+    const pose = frames[i]?.cameraPose;
+    if (pose) run.push(fluToScene(pose.position));
+    else if (run.length) {
+      runs.push(run);
+      run = [];
+    }
+  }
+  if (run.length) runs.push(run);
+  return runs;
+}
+
+/** Project optical-camera XYZ onto the original, possibly distorted source image. */
+export function projectHandPoint(
+  point: Point3,
+  c: HandCalibration,
+): [number, number] | null {
+  if (!point.every(Number.isFinite) || point[2] <= 1e-6) return null;
+  let x = point[0] / point[2],
+    y = point[1] / point[2];
+  const d = c.distortion;
+  const r2 = x * x + y * y;
+  if (c.model === "opencv_fisheye") {
+    const r = Math.sqrt(r2),
+      theta = Math.atan(r),
+      t2 = theta * theta;
+    const scale =
+      r < 1e-10
+        ? 1
+        : (theta * (1 + t2 * (d[0] + t2 * (d[1] + t2 * (d[2] + t2 * d[3]))))) /
+          r;
+    x *= scale;
+    y *= scale;
+  } else if (d.length) {
+    const [
+      k1 = 0,
+      k2 = 0,
+      p1 = 0,
+      p2 = 0,
+      k3 = 0,
+      k4 = 0,
+      k5 = 0,
+      k6 = 0,
+      s1 = 0,
+      s2 = 0,
+      s3 = 0,
+      s4 = 0,
+      tx = 0,
+      ty = 0,
+    ] = d;
+    const denominator = 1 + r2 * (k4 + r2 * (k5 + r2 * k6));
+    if (Math.abs(denominator) < 1e-12) return null;
+    const radial = (1 + r2 * (k1 + r2 * (k2 + r2 * k3))) / denominator;
+    const dx =
+      x * radial +
+      2 * p1 * x * y +
+      p2 * (r2 + 2 * x * x) +
+      s1 * r2 +
+      s2 * r2 * r2;
+    const dy =
+      y * radial +
+      p1 * (r2 + 2 * y * y) +
+      2 * p2 * x * y +
+      s3 * r2 +
+      s4 * r2 * r2;
+    // OpenCV's tilted-sensor projection (tau_x, tau_y), after radial/tangential distortion.
+    const sx = Math.sin(tx),
+      cx = Math.cos(tx),
+      sy = Math.sin(ty),
+      cy = Math.cos(ty);
+    const vx = cy * dx + sy * sx * dy - sy * cx;
+    const vy = cx * dy + sx;
+    const vz = sy * dx - cy * sx * dy + cy * cx;
+    if (Math.abs(vz) < 1e-12) return null;
+    x = (cy * cx * vx + sy * cx * vz) / vz;
+    y = (cy * cx * vy - sx * vz) / vz;
+  }
+  const pixel: [number, number] = [c.fx * x + c.cx, c.fy * y + c.cy];
+  return pixel.every(Number.isFinite) ? pixel : null;
+}

src/utils/__tests__/parquetUtils.test.ts

@@ -3,7 +3,9 @@ import {
   formatStringWithVars,
   arrayToCSV,
   getRows,
+  nestedColumnValues,
 } from "@/utils/parquetUtils";
+import type { FileMetaData } from "hyparquet";
 import { PADDING } from "@/utils/constants";
 
 // ---------------------------------------------------------------------------
@@ -145,3 +147,47 @@ describe("getRows", () => {
     expect(rows[0]).toEqual([0.75]);
   });
 });
+
+// ---------------------------------------------------------------------------
+// nestedColumnValues — what decoding a nested column costs for a row range
+// ---------------------------------------------------------------------------
+describe("nestedColumnValues", () => {
+  const chunk = (path: string[], values: number) => ({
+    meta_data: { path_in_schema: path, num_values: BigInt(values) },
+  });
+  // Two row groups of 100 rows: a flat column, a list column and a list of
+  // structs whose leaves are counted together.
+  const metadata = {
+    row_groups: [0, 1].map((group) => ({
+      num_rows: 100n,
+      columns: [
+        chunk(["timestamp"], 100),
+        chunk(["language_events", "list", "element"], 10 + group),
+        chunk(["language_persistent", "list", "element", "role"], 1000),
+        chunk(["language_persistent", "list", "element", "content"], 1000),
+      ],
+    })),
+  } as unknown as FileMetaData;
+
+  test("counts the whole row group, however few rows are read", () => {
+    expect(nestedColumnValues(metadata, 10, 20)).toEqual(
+      new Map([
+        ["language_events", 10],
+        ["language_persistent", 2000],
+      ]),
+    );
+  });
+
+  test("sums every row group the range overlaps", () => {
+    expect(nestedColumnValues(metadata, 90, 110).get("language_events")).toBe(
+      21,
+    );
+    expect(nestedColumnValues(metadata, 100, 200).get("language_events")).toBe(
+      11,
+    );
+  });
+
+  test("leaves flat columns out", () => {
+    expect(nestedColumnValues(metadata, 0, 200).has("timestamp")).toBe(false);
+  });
+});

src/utils/parquetUtils.ts

@@ -4,7 +4,9 @@ import {
   parquetRead,
   parquetReadObjects,
   type AsyncBuffer,
+  type FileMetaData,
 } from "hyparquet";
+import { decompress } from "fzstd";
 import { authHeaders } from "./auth";
 
 export interface DatasetMetadata {
@@ -43,6 +45,10 @@ type ParquetFile = ArrayBuffer | AsyncBuffer;
 
 const parquetFileCache = new Map<string, AsyncBuffer>();
 
+// hyparquet only decodes SNAPPY. No output buffer: fzstd throws on a page of
+// several zstd frames.
+const compressors = { ZSTD: (input: Uint8Array) => decompress(input) };
+
 export async function fetchParquetFile(url: string): Promise<ParquetFile> {
   const cached = parquetFileCache.get(url);
   if (cached) return cached;
@@ -69,6 +75,7 @@ export async function readParquetColumn(
         columns: columns.length > 0 ? columns : undefined,
         rowStart: options?.rowStart,
         rowEnd: options?.rowEnd,
+        compressors,
         onComplete: (data: unknown[][]) => {
           resolve(data);
         },
@@ -89,9 +96,38 @@ export async function readParquetAsObjects(
     columns: columns.length > 0 ? columns : undefined,
     rowStart: options?.rowStart,
     rowEnd: options?.rowEnd,
+    compressors,
   }) as Promise<Record<string, unknown>[]>;
 }
 
+/**
+ * Leaf values per nested column in the row groups overlapping
+ * [rowStart, rowEnd). hyparquet decodes a nested column across its whole row
+ * group, so this is what reading it costs, however few rows are asked for.
+ */
+export function nestedColumnValues(
+  metadata: FileMetaData,
+  rowStart: number,
+  rowEnd: number,
+): Map<string, number> {
+  const values = new Map<string, number>();
+  let groupEnd = 0;
+  for (const group of metadata.row_groups) {
+    const groupStart = groupEnd;
+    groupEnd += Number(group.num_rows);
+    if (groupStart >= rowEnd || groupEnd <= rowStart) continue;
+    for (const { meta_data } of group.columns) {
+      if (!meta_data || meta_data.path_in_schema.length === 1) continue;
+      const [column] = meta_data.path_in_schema;
+      values.set(
+        column,
+        (values.get(column) ?? 0) + Number(meta_data.num_values),
+      );
+    }
+  }
+  return values;
+}
+
 // Convert a 2D array to a CSV string
 export function arrayToCSV(data: (number | string)[][]): string {
   return data.map((row) => row.join(",")).join("\n");

tsconfig.test.json

@@ -6,5 +6,6 @@
     "types": ["bun-types"],
     "incremental": false
   },
-  "include": ["**/__tests__/**/*.ts", "src/**/*.ts"]
+  "include": ["**/__tests__/**/*.ts", "**/__tests__/**/*.tsx", "src/**/*.ts"],
+  "exclude": ["node_modules"]
 }