final checkpoint (114.4 epochs) + model card

Browse files

Files changed (8) hide show

README.md +107 -0
config.json +116 -0
model.safetensors +3 -0
policy_postprocessor.json +33 -0
policy_postprocessor_step_0_unnormalizer_processor.safetensors +3 -0
policy_preprocessor.json +72 -0
policy_preprocessor_step_3_normalizer_processor.safetensors +3 -0
train_config.json +252 -0

README.md ADDED Viewed

	@@ -0,0 +1,107 @@

+---
+license: apache-2.0
+library_name: lerobot
+tags: [robotics, lerobot, diffusion, tactile, manipulation]
+datasets: [aryankakad/tactile_charger_inserting]
+---
+# dp_tactile_charger_50ep
+**Diffusion** policy — **vision + tactile** — trained on **50 episodes**.
+Task: *grab and remove the charger from the socket and put it in the black box*
+Trained with [LeFlexiTac](https://github.com/TNA001-AI/lerobot_tactile), a LeRobot fork adding
+FlexiTac tactile sensing. Docs: <https://tna001-ai.github.io/LeFlexiTac/docs.html>
+## Training data
+| | |
+|---|---|
+| dataset | [`aryankakad/tactile_charger_inserting`](https://huggingface.co/datasets/aryankakad/tactile_charger_inserting) |
+| episodes | 50 (all) |
+| frames | 27,973 @ 30 fps |
+| cameras | `observation.images.top`, `observation.images.gripper` (224×224) |
+| tactile | `observation.tactile.primary` (12×32) |
+| state / action | 6-DoF SO-100 follower |
+## Configuration
+| | |
+|---|---|
+| steps | 200,000 |
+| batch size | 16 |
+| **epochs** | **114.4** |
+| `horizon` | `16` |
+| `n_obs_steps` | `2` |
+| `n_action_steps` | `8` |
+| `frame_stride` | `3` |
+| `resize_shape` | `[144, 192]` |
+| `crop_is_random` | `True` |
+| `optimizer_lr` | `0.0001` |
+| `use_amp` | `True` |
+| `n_tactile_chunks` | `1` |
+| `tactile_feature_dim` | `64` |
+Every model in this series is **epoch-matched at ~114.4 epochs**, so dataset size and
+sensor modality are the only variables across the set.
+## Training command actually used
+Run on 1× AMD Instinct MI300X (ROCm 6.2.4). `HIP_VISIBLE_DEVICES` selected the GPU,
+so `--policy.device=cuda` refers to that single card.
+**Initial run:**
+```bash
+python -u -m lerobot.scripts.lerobot_train \
+  --dataset.repo_id=aryankakad/tactile_charger_inserting \
+  --policy.type=diffusion \
+  --policy.use_tactile=true \
+  --policy.tactile_features='["observation.tactile.primary"]' \
+  --policy.n_tactile_chunks=1 \
+  --policy.tactile_feature_dim=64 \
+  --policy.crop_is_random=true \
+  --policy.resize_shape='[144,192]' \
+  --policy.use_amp=true \
+  --policy.frame_stride=3 \
+  --policy.repo_id=Dimios45/dp_tactile_charger_inserting \
+  --policy.private=true \
+  --policy.device=cuda \
+  --output_dir=outputs/train/dp_tactile_charger_inserting \
+  --job_name=dp_tactile_charger_inserting \
+  --batch_size=16 \
+  --num_workers=8 \
+  --steps=200000 \
+  --save_freq=40000 \
+  --wandb.enable=true
+```
+**Resumed** (the first run was interrupted; resumed from its last checkpoint):
+```bash
+python -u -m lerobot.scripts.lerobot_train \
+  --config_path=outputs/train/dp_tactile_charger_inserting/checkpoints/last/pretrained_model/train_config.json \
+  --resume=true
+```
+## Evaluation / rollout
+Not run here — this machine has no robot attached. To evaluate, run on the machine with the
+SO-100 and sensors, loading the policy with `--policy.path=Dimios45/dp_tactile_charger_50ep`.
+Reference: the `lerobot-record` eval invocations in
+[`tactile_cmd.txt`](https://github.com/TNA001-AI/lerobot_tactile/blob/main/tactile_cmd.txt)
+and the [project docs](https://tna001-ai.github.io/LeFlexiTac/docs.html). You will need to
+supply your own robot port, camera serials, and a `primary` tactile sensor entry matching training.
+## Notes
+- Two ROCm-specific fixes were required in the fork: `persistent_workers=True` on the
+  dataloader (epoch boundaries otherwise stalled ~410 s each), and keeping
+  `cudnn.benchmark` **off** (on ROCm it triggers an exhaustive MIOpen search that can
+  precede step 1 by hours).
+- Training loss is **not** a proxy for task success. Compare policies by rollout success
+  rate, especially on contact-rich phases.
+- The source dataset's task string is labelled `stack cup` — a mislabel carried over from
+  an earlier session. It does not affect Diffusion, which is not language-conditioned.

config.json ADDED Viewed

	@@ -0,0 +1,116 @@

+{
+    "type": "diffusion",
+    "n_obs_steps": 2,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.gripper": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                224,
+                224
+            ]
+        },
+        "observation.images.top": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                224,
+                224
+            ]
+        },
+        "observation.tactile.primary": {
+            "type": "STATE",
+            "shape": [
+                12,
+                32
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": true,
+    "use_peft": false,
+    "push_to_hub": true,
+    "repo_id": "Dimios45/dp_tactile_charger_inserting",
+    "private": true,
+    "tags": null,
+    "license": null,
+    "pretrained_path": "outputs/train/dp_tactile_charger_inserting/checkpoints/last/pretrained_model",
+    "horizon": 16,
+    "n_action_steps": 8,
+    "frame_stride": 3,
+    "normalization_mapping": {
+        "VISUAL": "MEAN_STD",
+        "STATE": "MIN_MAX",
+        "ACTION": "MIN_MAX",
+        "TACTILE": "MEAN_STD"
+    },
+    "drop_n_last_frames": 21,
+    "vision_backbone": "resnet18",
+    "resize_shape": [
+        144,
+        192
+    ],
+    "crop_ratio": 1.0,
+    "crop_shape": null,
+    "crop_is_random": true,
+    "pretrained_backbone_weights": null,
+    "use_group_norm": true,
+    "spatial_softmax_num_keypoints": 32,
+    "use_separate_rgb_encoder_per_camera": false,
+    "down_dims": [
+        512,
+        1024,
+        2048
+    ],
+    "kernel_size": 5,
+    "n_groups": 8,
+    "diffusion_step_embed_dim": 128,
+    "use_film_scale_modulation": true,
+    "noise_scheduler_type": "DDPM",
+    "num_train_timesteps": 100,
+    "beta_schedule": "squaredcos_cap_v2",
+    "beta_start": 0.0001,
+    "beta_end": 0.02,
+    "prediction_type": "epsilon",
+    "clip_sample": true,
+    "clip_sample_range": 1.0,
+    "num_inference_steps": null,
+    "compile_model": false,
+    "compile_mode": "reduce-overhead",
+    "do_mask_loss_for_padding": false,
+    "use_tactile": true,
+    "tactile_encoder_type": "cnn",
+    "tactile_input_shape": [
+        12,
+        32
+    ],
+    "tactile_dropout": 0.3,
+    "tactile_feature_dim": 64,
+    "tactile_features": [
+        "observation.tactile.primary"
+    ],
+    "n_tactile_chunks": 1,
+    "optimizer_lr": 0.0001,
+    "optimizer_betas": [
+        0.95,
+        0.999
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 1e-06,
+    "scheduler_name": "cosine",
+    "scheduler_warmup_steps": 500
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:65f3e3a903e52bf6bb78aad20f44b2a15fc8943122e337a8c37304d9b45dfd42
+size 1082755728

policy_postprocessor.json ADDED Viewed

	@@ -0,0 +1,33 @@

+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX",
+          "TACTILE": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:44d4d7be9139b14b3b0810abf88febcbcc7f3862b89e7f2783278919aab3e232
+size 9644

policy_preprocessor.json ADDED Viewed

	@@ -0,0 +1,72 @@

+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              6
+            ]
+          },
+          "observation.images.gripper": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              224,
+              224
+            ]
+          },
+          "observation.images.top": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              224,
+              224
+            ]
+          },
+          "observation.tactile.primary": {
+            "type": "STATE",
+            "shape": [
+              12,
+              32
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX",
+          "TACTILE": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
+    }
+  ]
+}

policy_preprocessor_step_3_normalizer_processor.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:44d4d7be9139b14b3b0810abf88febcbcc7f3862b89e7f2783278919aab3e232
+size 9644

train_config.json ADDED Viewed

	@@ -0,0 +1,252 @@

+{
+    "dataset": {
+        "repo_id": "aryankakad/tactile_charger_inserting",
+        "root": null,
+        "episodes": null,
+        "image_transforms": {
+            "enable": false,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "affine": {
+                    "weight": 1.0,
+                    "type": "RandomAffine",
+                    "kwargs": {
+                        "degrees": [
+                            -5.0,
+                            5.0
+                        ],
+                        "translate": [
+                            0.05,
+                            0.05
+                        ]
+                    }
+                }
+            }
+        },
+        "revision": null,
+        "use_imagenet_stats": true,
+        "video_backend": "torchcodec",
+        "streaming": false
+    },
+    "env": null,
+    "policy": {
+        "type": "diffusion",
+        "n_obs_steps": 2,
+        "input_features": {
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            },
+            "observation.images.gripper": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    224,
+                    224
+                ]
+            },
+            "observation.images.top": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    224,
+                    224
+                ]
+            },
+            "observation.tactile.primary": {
+                "type": "STATE",
+                "shape": [
+                    12,
+                    32
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    6
+                ]
+            }
+        },
+        "device": "cuda",
+        "use_amp": true,
+        "use_peft": false,
+        "push_to_hub": true,
+        "repo_id": "Dimios45/dp_tactile_charger_inserting",
+        "private": true,
+        "tags": null,
+        "license": null,
+        "pretrained_path": "outputs/train/dp_tactile_charger_inserting/checkpoints/last/pretrained_model",
+        "horizon": 16,
+        "n_action_steps": 8,
+        "frame_stride": 3,
+        "normalization_mapping": {
+            "VISUAL": "MEAN_STD",
+            "STATE": "MIN_MAX",
+            "ACTION": "MIN_MAX",
+            "TACTILE": "MEAN_STD"
+        },
+        "drop_n_last_frames": 21,
+        "vision_backbone": "resnet18",
+        "resize_shape": [
+            144,
+            192
+        ],
+        "crop_ratio": 1.0,
+        "crop_shape": null,
+        "crop_is_random": true,
+        "pretrained_backbone_weights": null,
+        "use_group_norm": true,
+        "spatial_softmax_num_keypoints": 32,
+        "use_separate_rgb_encoder_per_camera": false,
+        "down_dims": [
+            512,
+            1024,
+            2048
+        ],
+        "kernel_size": 5,
+        "n_groups": 8,
+        "diffusion_step_embed_dim": 128,
+        "use_film_scale_modulation": true,
+        "noise_scheduler_type": "DDPM",
+        "num_train_timesteps": 100,
+        "beta_schedule": "squaredcos_cap_v2",
+        "beta_start": 0.0001,
+        "beta_end": 0.02,
+        "prediction_type": "epsilon",
+        "clip_sample": true,
+        "clip_sample_range": 1.0,
+        "num_inference_steps": null,
+        "compile_model": false,
+        "compile_mode": "reduce-overhead",
+        "do_mask_loss_for_padding": false,
+        "use_tactile": true,
+        "tactile_encoder_type": "cnn",
+        "tactile_input_shape": [
+            12,
+            32
+        ],
+        "tactile_dropout": 0.3,
+        "tactile_feature_dim": 64,
+        "tactile_features": [
+            "observation.tactile.primary"
+        ],
+        "n_tactile_chunks": 1,
+        "optimizer_lr": 0.0001,
+        "optimizer_betas": [
+            0.95,
+            0.999
+        ],
+        "optimizer_eps": 1e-08,
+        "optimizer_weight_decay": 1e-06,
+        "scheduler_name": "cosine",
+        "scheduler_warmup_steps": 500
+    },
+    "output_dir": "outputs/train/dp_tactile_charger_inserting",
+    "job_name": "dp_tactile_charger_inserting",
+    "resume": true,
+    "seed": 1000,
+    "cudnn_deterministic": false,
+    "num_workers": 8,
+    "batch_size": 16,
+    "steps": 200000,
+    "eval_freq": 20000,
+    "log_freq": 200,
+    "tolerance_s": 0.0001,
+    "save_checkpoint": true,
+    "save_freq": 40000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adam",
+        "lr": 0.0001,
+        "weight_decay": 1e-06,
+        "grad_clip_norm": 10.0,
+        "betas": [
+            0.95,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": {
+        "type": "diffuser",
+        "num_warmup_steps": 500,
+        "name": "cosine"
+    },
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 50,
+        "use_async_envs": false
+    },
+    "wandb": {
+        "enable": true,
+        "disable_artifact": false,
+        "project": "lerobot",
+        "entity": null,
+        "notes": null,
+        "run_id": "mc1hb0so",
+        "mode": null,
+        "add_tags": true
+    },
+    "peft": null,
+    "use_rabc": false,
+    "rabc_progress_path": null,
+    "rabc_kappa": 0.01,
+    "rabc_epsilon": 1e-06,
+    "rabc_head_mode": "sparse",
+    "rename_map": {},
+    "checkpoint_path": "outputs/train/dp_tactile_charger_inserting/checkpoints/last"
+}