Model Training

1. ACT

The following is a training example for Task 1 using the ACT policy, including the complete set of hyperparameters:

# Concise Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
    --dataset.repo_id=your_org/your_dataset \
    --dataset.video_backend=pyav \
    --policy.type=act \
    --output_dir=challenge2026_baseline/Part_Sorting/act \
    --dataset.root=datasets/Part_Sorting/ \
    --job_name=part_sorting_act \
    --policy.device=cuda \
    --wandb.enable=false \
    --policy.repo_id=none \
    --policy.push_to_hub=false 


# Detailed Training Command for Task 1
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --dataset.repo_id=your_org/your_dataset \
  --dataset.root=datasets/Part_Sorting/ \
  --dataset.video_backend=pyav \
  --policy.type=act \
  --policy.n_obs_steps=1 \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.vision_backbone=resnet18 \
  --policy.pretrained_backbone_weights=ResNet18_Weights.IMAGENET1K_V1 \
  --policy.dim_model=256 \
  --policy.n_heads=4 \
  --policy.dim_feedforward=1024 \
  --policy.n_encoder_layers=4 \
  --policy.n_decoder_layers=1 \
  --policy.use_vae=true \
  --policy.latent_dim=32 \
  --policy.n_vae_encoder_layers=4 \
  --policy.dropout=0.1 \
  --policy.kl_weight=10.0 \
  --policy.optimizer_lr=1e-5 \
  --policy.optimizer_weight_decay=1e-4 \
  --policy.optimizer_lr_backbone=1e-5 \
  --policy.device=cuda \
  --policy.use_amp=true \
  --policy.push_to_hub=false \
  --output_dir=challenge2026_baseline/Part_Sorting/act \
  --job_name=part_sorting_act \
  --resume=false \
  --seed=1000 \
  --num_workers=8 \
  --batch_size=8 \
  --steps=100000 \
  --eval_freq=0 \
  --log_freq=200 \
  --save_checkpoint=true \
  --save_freq=5000 \
  --wandb.entity=your_wandb_entity

Replace your_org/your_dataset with your own dataset repo ID, replace challenge2026_baseline/Part_Sorting/act with the path where you want to save the output, and replace your_wandb_entity with your WandB username or team name. If you do not use WandB, you can remove the --wandb.entity argument. Training ACT requires downloading resnet18-f37072fd.pth.

Dataset and Output Parameters

Parameter	Description	Default / Notes
`dataset.repo_id`	Dataset ID (Hugging Face or a local organization name)	Required
`dataset.root`	Local root path of the dataset	Required
`output_dir`	Directory for saving checkpoints and logs	Required
`job_name`	Job identifier (shown in logs / WandB)	Optional
`resume`	Whether to resume training from the previous checkpoint	`false`
`seed`	Global random seed	`1000`

Training Loop Parameters

Parameter	Description	Default / Notes
`steps`	Total number of training steps	`100000`
`batch_size`	Number of samples per step	`8`
`num_workers`	Number of DataLoader worker processes	`8`
`eval_freq`	Evaluation interval in steps (`0` disables evaluation)	`0`
`log_freq`	Log printing interval in steps	`200`
`save_checkpoint`	Whether to save checkpoints	`true`
`save_freq`	Checkpoint saving interval in steps	`5000`

ACT Policy Parameters

Parameter	Description	Default / Notes
`policy.type`	Policy algorithm type	`act` / `pi0`
`policy.device`	Execution device	`cuda` / `cpu`
`policy.use_amp`	Whether to enable mixed-precision training	`true`
`policy.n_obs_steps`	Number of observation steps	`1`
`policy.chunk_size`	Action chunk length	`50`
`policy.n_action_steps`	Number of action steps executed per inference call	`50`
`policy.vision_backbone`	Vision encoder architecture	`resnet18`
`policy.pretrained_backbone_weights`	Pretrained backbone weights	`ResNet18_Weights.IMAGENET1K_V1`
`policy.dim_model`	Transformer model dimension	`256`
`policy.n_heads`	Number of attention heads	`4`
`policy.dim_feedforward`	Feed-forward network dimension	`1024`
`policy.n_encoder_layers`	Number of encoder layers	`4`

2. Diffusion Policy (DP)

The following is a training example using Diffusion Policy and includes the complete set of hyperparameters:

/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --dataset.repo_id=your_org/your_dataset \
  --dataset.root=datasets/Part_Sorting \
  --dataset.video_backend=pyav \
  --output_dir=challenge2026_baseline/Part_Sorting/diffusion \
  --policy.repo_id=none \
  --policy.type=diffusion \
  --policy.n_obs_steps=2 \
  --policy.horizon=16 \
  --policy.n_action_steps=8 \
  --policy.vision_backbone=resnet18 \
  --policy.pretrained_backbone_weights=null \
  --policy.resize_shape=null \
  --policy.crop_ratio=1.0 \
  --policy.crop_shape=null \
  --policy.crop_is_random=true \
  --policy.use_group_norm=true \
  --policy.spatial_softmax_num_keypoints=32 \
  --policy.use_separate_rgb_encoder_per_camera=false \
  --policy.down_dims='[512,1024,2048]' \
  --policy.kernel_size=5 \
  --policy.n_groups=8 \
  --policy.diffusion_step_embed_dim=128 \
  --policy.use_film_scale_modulation=true \
  --policy.noise_scheduler_type=DDPM \
  --policy.num_train_timesteps=100 \
  --policy.beta_schedule=squaredcos_cap_v2 \
  --policy.beta_start=0.0001 \
  --policy.beta_end=0.02 \
  --policy.prediction_type=epsilon \
  --policy.clip_sample=true \
  --policy.clip_sample_range=1.0 \
  --policy.num_inference_steps=null \
  --policy.compile_model=false \
  --policy.compile_mode=reduce-overhead \
  --policy.do_mask_loss_for_padding=false \
  --policy.optimizer_lr=1e-4 \
  --policy.optimizer_betas='[0.95,0.999]' \
  --policy.optimizer_eps=1e-8 \
  --policy.optimizer_weight_decay=1e-6 \
  --policy.scheduler_name=cosine \
  --policy.scheduler_warmup_steps=500 \
  --job_name=part_sorting_diffusion \
  --resume=false \
  --seed=1000 \
  --num_workers=8 \
  --batch_size=32 \
  --steps=100000 \
  --eval_freq=0 \
  --log_freq=200 \
  --save_checkpoint=true \
  --save_freq=5000

Replace your_org/your_dataset with your own dataset repo ID, replace challenge2026_baseline/Part_Sorting/diffusion with the path where you want to save the output.

Diffusion Policy Parameters - Input/Output Structure

Parameter	Description	Default / Notes
`policy.type`	Policy algorithm type	`diffusion`
`policy.n_obs_steps`	Number of observation steps	`2`
`policy.horizon`	Action prediction horizon	`16`
`policy.n_action_steps`	Number of action steps executed per inference call	`8`

Diffusion Policy Parameters - Vision Backbone

Parameter	Description	Default / Notes
`policy.vision_backbone`	Vision encoder architecture	`resnet18`
`policy.pretrained_backbone_weights`	Pretrained backbone weights	`null`
`policy.resize_shape`	Image preprocessing resize dimensions (H, W)	`null`
`policy.crop_ratio`	Crop size ratio (0, 1]	`1.0`
`policy.crop_shape`	Crop dimensions (H, W)	`null`
`policy.crop_is_random`	Whether to apply a random crop during training	`true`
`policy.use_group_norm`	Whether to use GroupNorm instead of BatchNorm	`true`
`policy.spatial_softmax_num_keypoints`	Number of SpatialSoftmax keypoints	`32`
`policy.use_separate_rgb_encoder_per_camera`	Whether to use a separate encoder for each camera	`false`

Diffusion Policy Parameters - UNet Architecture

Parameter	Description	Default / Notes
`policy.down_dims`	UNet downsampling dimensions	`[512,1024,2048]`
`policy.kernel_size`	Kernel size	`5`
`policy.n_groups`	Number of GroupNorm groups	`8`
`policy.diffusion_step_embed_dim`	Diffusion timestep embedding dimension	`128`
`policy.use_film_scale_modulation`	Whether to use FiLM scale modulation	`true`

Diffusion Policy Parameters - Noise Scheduler

Parameter	Description	Default / Notes
`policy.noise_scheduler_type`	Noise scheduler type	`DDPM` / `DDIM`
`policy.num_train_timesteps`	Number of diffusion steps during training	`100`
`policy.beta_schedule`	Beta schedule	`squaredcos_cap_v2`
`policy.beta_start`	Initial beta value	`0.0001`
`policy.beta_end`	Final beta value	`0.02`
`policy.prediction_type`	Prediction type	`epsilon` / `sample`
`policy.clip_sample`	Whether to clip samples	`true`
`policy.clip_sample_range`	Clipping range	`1.0`
`policy.num_inference_steps`	Number of inference steps	`null` (equivalent to the number of training steps)

Diffusion Policy Parameters - Optimizer and Scheduler

Parameter	Description	Default / Notes
`policy.optimizer_lr`	Learning rate	`1e-4`
`policy.optimizer_betas`	Adam betas	`[0.95,0.999]`
`policy.optimizer_eps`	Adam eps	`1e-8`
`policy.optimizer_weight_decay`	Weight decay	`1e-6`
`policy.scheduler_name`	Learning rate scheduler	`cosine`
`policy.scheduler_warmup_steps`	Number of warmup steps	`500`

Diffusion Policy Parameters - Other Settings

Parameter	Description	Default / Notes
`policy.compile_model`	Whether to compile the model	`false`
`policy.compile_mode`	Compilation mode	`reduce-overhead`
`policy.do_mask_loss_for_padding`	Whether to mask the loss on padding elements	`false`

3. π₀ (PI0)

Download Pretrained Weights

# Download Pretrained Weights
hf download \
  lerobot/pi0_base \
  --local-dir pretrained/pi0_base

hf download \
  lerobot/pi05_base \
  --local-dir pretrained/pi05_base
  
hf download google/paligemma-3b-pt-224 \
  --local-dir pretrained/paligemma-3b-pt-224

In the following file: /workspace/GlobalHumanoidRobotChallenge_2026_Baseline/src/lerobot/processor/tokenizer_processor.py Locate the relevant section and replace the code with:

if self.tokenizer is not None:
            # Use provided tokenizer object directly
            self.input_tokenizer = self.tokenizer
        elif self.tokenizer_name is not None:
            if AutoTokenizer is None:
                raise ImportError("AutoTokenizer is not available")

            # If tokenizer_name contains "paligemma", the model is PI0, so force loading from the local offline path
            if "paligemma" in self.tokenizer_name.lower():
                self.input_tokenizer = AutoTokenizer.from_pretrained(
                    "/root/.cache/huggingface/hub/models--google--paligemma-3b-pt-224/snapshots/35e4f46485b4d07967e7e9935bc3786aad50687c",
                    local_files_only=True
                )
            else:
                # Otherwise (for example, ACT or SmolVLA), load normally from the provided tokenizer_name path
                self.input_tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name)
            
        else:
            raise ValueError(
                "Either 'tokenizer' or 'tokenizer_name' must be provided. "
                "Pass a tokenizer object directly or a tokenizer name to auto-load."
            )

The following is a training example using the π₀ (PI0) policy, including the complete set of hyperparameters:

# Concise Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --policy.path=lerobot/pi0_base \
  --dataset.repo_id=your_org/your_dataset \
  --batch_size=64 \
  --steps=20000 \
  --output_dir=challenge2026_baseline/Part_Sorting/pi0 \
  --job_name=part_sorting_pi0 \
  --policy.device=cuda \
  --wandb.enable=true

# Detailed Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --dataset.repo_id=your_org/your_dataset \
  --dataset.root=datasets/Part_Sorting \
  --policy.type=pi0 \
  --policy.paligemma_variant=gemma_2b \
  --policy.action_expert_variant=gemma_300m \
  --policy.dtype=float32 \
  --policy.n_obs_steps=1 \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=32 \
  --policy.max_action_dim=32 \
  --policy.num_inference_steps=10 \
  --policy.time_sampling_beta_alpha=1.5 \
  --policy.time_sampling_beta_beta=1.0 \
  --policy.time_sampling_scale=0.999 \
  --policy.time_sampling_offset=0.001 \
  --policy.min_period=0.004 \
  --policy.max_period=4.0 \
  --policy.image_resolution='[224,224]' \
  --policy.empty_cameras=0 \
  --policy.gradient_checkpointing=false \
  --policy.compile_model=false \
  --policy.compile_mode=max-autotune \
  --policy.freeze_vision_encoder=false \
  --policy.train_expert_only=false \
  --policy.optimizer_lr=2.5e-5 \
  --policy.optimizer_betas='[0.9,0.95]' \
  --policy.optimizer_eps=1e-8 \
  --policy.optimizer_weight_decay=0.01 \
  --policy.optimizer_grad_clip_norm=1.0 \
  --policy.scheduler_warmup_steps=1000 \
  --policy.scheduler_decay_steps=30000 \
  --policy.scheduler_decay_lr=2.5e-6 \
  --policy.tokenizer_max_length=48 \
  --output_dir=challenge2026_baseline/Part_Sorting/pi0 \
  --job_name=part_sorting_pi0 \
  --resume=false \
  --seed=1000 \
  --num_workers=8 \
  --batch_size=8 \
  --steps=100000 \
  --eval_freq=0 \
  --log_freq=200 \
  --save_checkpoint=true \
  --save_freq=5000 \
  --wandb.entity=your_wandb_entity

Replace your_org/your_dataset with your own dataset repo ID, replace challenge2026_baseline/Part_Sorting/pi0 with the path where you want to save the output, and replace your_wandb_entity with your WandB username or team name. If you do not use WandB, you can remove the --wandb.entity argument.

π₀ Policy Parameters - Model Architecture

Parameter	Description	Default / Notes
`policy.type`	Policy algorithm type	`pi0`
`policy.paligemma_variant`	PaliGemma model variant	`gemma_2b`
`policy.action_expert_variant`	Action Expert variant	`gemma_300m`
`policy.dtype`	Data type	`float32`

π₀ Policy Parameters - Input/Output Structure

Parameter	Description	Default / Notes
`policy.n_obs_steps`	Number of observation steps	`1`
`policy.chunk_size`	Action chunk size	`50`
`policy.n_action_steps`	Number of action steps to execute	`50`
`policy.max_state_dim`	Maximum state dimension (inputs are padded to this size)	`32`
`policy.max_action_dim`	Maximum action dimension (inputs are padded to this size)	`32`

π₀ Policy Parameters - Flow Matching

Parameter	Description	Default / Notes
`policy.num_inference_steps`	Number of denoising steps during inference	`10`
`policy.time_sampling_beta_alpha`	Time-sampling beta α	`1.5`
`policy.time_sampling_beta_beta`	Time-sampling beta β	`1.0`
`policy.time_sampling_scale`	Time-sampling scale	`0.999`
`policy.time_sampling_offset`	Time-sampling offset	`0.001`
`policy.min_period`	Minimum period	`0.004`
`policy.max_period`	Maximum period	`4.0`

π₀ Policy Parameters - Images and Cameras

Parameter	Description	Default / Notes
`policy.image_resolution`	Image resolution (H, W)	`[224,224]`
`policy.empty_cameras`	Number of empty cameras used for padding	`0`

π₀ Policy Parameters - Training Settings

Parameter	Description	Default / Notes
`policy.gradient_checkpointing`	Whether to enable gradient checkpointing	`false`
`policy.compile_model`	Whether to compile the model	`false`
`policy.compile_mode`	Compilation mode	`max-autotune`

π₀ Policy Parameters - Fine-Tuning Settings

Parameter	Description	Default / Notes
`policy.freeze_vision_encoder`	Whether to freeze the vision encoder	`false`
`policy.train_expert_only`	Whether to train only the Action Expert	`false`

π₀ Policy Parameters - Optimizer

Parameter	Description	Default / Notes
`policy.optimizer_lr`	Learning rate	`2.5e-5`
`policy.optimizer_betas`	AdamW betas	`[0.9,0.95]`
`policy.optimizer_eps`	AdamW eps	`1e-8`
`policy.optimizer_weight_decay`	Weight decay	`0.01`
`policy.optimizer_grad_clip_norm`	Gradient clipping norm	`1.0`

π₀ Policy Parameters - Learning Rate Scheduler

Parameter	Description	Default / Notes
`policy.scheduler_warmup_steps`	Number of warmup steps	`1000`
`policy.scheduler_decay_steps`	Number of decay steps	`30000`
`policy.scheduler_decay_lr`	Final learning rate after decay	`2.5e-6`

π₀ Policy Parameters - Tokenizer

Parameter	Description	Default / Notes
`policy.tokenizer_max_length`	Maximum tokenizer length	`48`

4. π₀.₅ (PI05)

The following is a training example using the π₀.₅ (PI05) policy, including the complete set of hyperparameters. π₀.₅ is an enhanced version of π₀ that supports open-world generalization. Its main differences include QUANTILES normalization, a longer tokenizer length, and AdaRMS conditioning.

# Concise Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
    --dataset.repo_id=your_org/your_dataset \
    --policy.type=pi05 \
    --output_dir=challenge2026_baseline/Part_Sorting/pi05 \
    --job_name=part_sorting_pi05 \
    --policy.repo_id=your_repo_id \
    --policy.pretrained_path=lerobot/pi05_base \
    --policy.compile_model=true \
    --policy.gradient_checkpointing=true \
    --wandb.enable=true \
    --policy.dtype=bfloat16 \
    --policy.freeze_vision_encoder=false \
    --policy.train_expert_only=false \
    --steps=3000 \
    --policy.device=cuda \
    --batch_size=32

# Detailed Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --dataset.repo_id=your_org/your_dataset \
  --dataset.root=datasets/Part_Sorting/ \
  --policy.type=pi05 \
  --policy.paligemma_variant=gemma_2b \
  --policy.action_expert_variant=gemma_300m \
  --policy.dtype=float32 \
  --policy.n_obs_steps=1 \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=32 \
  --policy.max_action_dim=32 \
  --policy.num_inference_steps=10 \
  --policy.time_sampling_beta_alpha=1.5 \
  --policy.time_sampling_beta_beta=1.0 \
  --policy.time_sampling_scale=0.999 \
  --policy.time_sampling_offset=0.001 \
  --policy.min_period=0.004 \
  --policy.max_period=4.0 \
  --policy.image_resolution='[224,224]' \
  --policy.empty_cameras=0 \
  --policy.gradient_checkpointing=false \
  --policy.compile_model=false \
  --policy.compile_mode=max-autotune \
  --policy.freeze_vision_encoder=false \
  --policy.train_expert_only=false \
  --policy.optimizer_lr=2.5e-5 \
  --policy.optimizer_betas='[0.9,0.95]' \
  --policy.optimizer_eps=1e-8 \
  --policy.optimizer_weight_decay=0.01 \
  --policy.optimizer_grad_clip_norm=1.0 \
  --policy.scheduler_warmup_steps=1000 \
  --policy.scheduler_decay_steps=30000 \
  --policy.scheduler_decay_lr=2.5e-6 \
  --policy.tokenizer_max_length=200 \
  --output_dir=challenge2026_baseline/Part_Sorting/pi05 \
  --job_name=part_sorting_pi05 \
  --resume=false \
  --seed=1000 \
  --num_workers=8 \
  --batch_size=8 \
  --steps=100000 \
  --eval_freq=0 \
  --log_freq=200 \
  --save_checkpoint=true \
  --save_freq=5000 \
  --wandb.entity=your_wandb_entity

Replace your_org/your_dataset with your own dataset repo ID, replace challenge2026_baseline/Part_Sorting/pi05 with the path where you want to save the output, and replace your_wandb_entity with your WandB username or team name. If you do not use WandB, you can remove the --wandb.entity argument.

π₀.₅ Policy Parameters - Model Architecture

Parameter	Description	Default / Notes
`policy.type`	Policy algorithm type	`pi05`
`policy.paligemma_variant`	PaliGemma model variant	`gemma_2b`
`policy.action_expert_variant`	Action Expert variant	`gemma_300m`
`policy.dtype`	Data type	`float32`

π₀.₅ Policy Parameters - Input/Output Structure

Parameter	Description	Default / Notes
`policy.n_obs_steps`	Number of observation steps	`1`
`policy.chunk_size`	Action chunk size	`50`
`policy.n_action_steps`	Number of action steps to execute	`50`
`policy.max_state_dim`	Maximum state dimension (inputs are padded to this size)	`32`
`policy.max_action_dim`	Maximum action dimension (inputs are padded to this size)	`32`

π₀.₅ Policy Parameters - Flow Matching

Parameter	Description	Default / Notes
`policy.num_inference_steps`	Number of denoising steps during inference	`10`
`policy.time_sampling_beta_alpha`	Time-sampling beta α	`1.5`
`policy.time_sampling_beta_beta`	Time-sampling beta β	`1.0`
`policy.time_sampling_scale`	Time-sampling scale	`0.999`
`policy.time_sampling_offset`	Time-sampling offset	`0.001`
`policy.min_period`	Minimum period	`0.004`
`policy.max_period`	Maximum period	`4.0`

π₀.₅ Policy Parameters - Images and Cameras

Parameter	Description	Default / Notes
`policy.image_resolution`	Image resolution (H, W)	`[224,224]`
`policy.empty_cameras`	Number of empty cameras used for padding	`0`

π₀.₅ Policy Parameters - Training Settings

Parameter	Description	Default / Notes
`policy.gradient_checkpointing`	Whether to enable gradient checkpointing	`false`
`policy.compile_model`	Whether to compile the model	`false`
`policy.compile_mode`	Compilation mode	`max-autotune`

π₀.₅ Policy Parameters - Fine-Tuning Settings

Parameter	Description	Default / Notes
`policy.freeze_vision_encoder`	Whether to freeze the vision encoder	`false`
`policy.train_expert_only`	Whether to train only the Action Expert	`false`

π₀.₅ Policy Parameters - Optimizer

Parameter	Description	Default / Notes
`policy.optimizer_lr`	Learning rate	`2.5e-5`
`policy.optimizer_betas`	AdamW betas	`[0.9,0.95]`
`policy.optimizer_eps`	AdamW eps	`1e-8`
`policy.optimizer_weight_decay`	Weight decay	`0.01`
`policy.optimizer_grad_clip_norm`	Gradient clipping norm	`1.0`

π₀.₅ Policy Parameters - Learning Rate Scheduler

Parameter	Description	Default / Notes
`policy.scheduler_warmup_steps`	Number of warmup steps	`1000`
`policy.scheduler_decay_steps`	Number of decay steps	`30000`
`policy.scheduler_decay_lr`	Final learning rate after decay	`2.5e-6`

π₀.₅ Policy Parameters - Tokenizer

Parameter	Description	Default / Notes
`policy.tokenizer_max_length`	Maximum tokenizer length	`200` (π₀ uses `48`)

Key Differences Between π₀ and π₀.₅

Feature	π₀	π₀.₅
Time conditioning injection	Concatenates time and actions through `action_time_mlp_*`	Uses AdaRMS conditioning through `time_mlp_*`
AdaRMS	Not used	Used in the Action Expert
Tokenizer length	48 tokens	200 tokens
Discrete state input	False (uses the `state_proj` layer)	True
Parameter count	Higher (includes a state embedding layer)	Lower (no state embedding layer)
State normalization	MEAN_STD	QUANTILES
Action normalization	MEAN_STD	QUANTILES

5. SmolVLA

The following is a fine-tuning example using the SmolVLA policy. SmolVLA is built on the SmolVLM2-500M-Video-Instruct vision-language model and supports open-world generalization.

# Concise Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=your_org/your_dataset \
  --batch_size=64 \
  --steps=20000 \
  --output_dir=challenge2026_baseline/Part_Sorting/smolvla \
  --job_name=part_sorting_smolvla \
  --policy.device=cuda \
  --wandb.enable=true

# Detailed Training Command
/isaac-sim/python.sh src/lerobot/scripts/lerobot_train.py \
  --dataset.repo_id=your_org/your_dataset \
  --dataset.root=datasets/Part_Sorting/ \
  --policy.type=smolvla \
  --policy.vlm_model_name=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
  --policy.load_vlm_weights=true \
  --policy.dtype=float32 \
  --policy.n_obs_steps=1 \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=32 \
  --policy.max_action_dim=32 \
  --policy.num_steps=10 \
  --policy.tokenizer_max_length=48 \
  --policy.image_resolution='[224,224]' \
  --policy.empty_cameras=0 \
  --policy.freeze_vision_encoder=true \
  --policy.train_expert_only=true \
  --policy.train_state_proj=true \
  --policy.gradient_checkpointing=false \
  --policy.compile_model=false \
  --policy.compile_mode=max-autotune \
  --policy.attention_mode=cross_attn \
  --policy.num_vlm_layers=16 \
  --policy.self_attn_every_n_layers=2 \
  --policy.expert_width_multiplier=0.75 \
  --policy.optimizer_lr=1e-4 \
  --policy.optimizer_betas='[0.9,0.95]' \
  --policy.optimizer_eps=1e-8 \
  --policy.optimizer_weight_decay=1e-10 \
  --policy.optimizer_grad_clip_norm=10.0 \
  --policy.scheduler_warmup_steps=1000 \
  --policy.scheduler_decay_steps=30000 \
  --policy.scheduler_decay_lr=2.5e-6 \
  --policy.min_period=0.004 \
  --policy.max_period=4.0 \
  --output_dir=challenge2026_baseline/Part_Sorting/smolvla \
  --job_name=part_sorting_smolvla \
  --resume=false \
  --seed=1000 \
  --num_workers=8 \
  --batch_size=8 \
  --steps=100000 \
  --eval_freq=0 \
  --log_freq=200 \
  --save_checkpoint=true \
  --save_freq=5000 \
  --wandb.entity=your_wandb_entity

Replace your_org/your_dataset with your own dataset repo ID, replace challenge2026_baseline/Part_Sorting/smolvla with the path where you want to save the output, and replace your_wandb_entity with your WandB username or team name. If you do not use WandB, you can remove the --wandb.entity argument.

SmolVLA Policy Parameters - Model Architecture

Parameter	Description	Default / Notes
`policy.type`	Policy algorithm type	`smolvla`
`policy.vlm_model_name`	VLM backbone model	`HuggingFaceTB/SmolVLM2-500M-Video-Instruct`
`policy.load_vlm_weights`	Whether to load pretrained VLM weights	`true`
`policy.dtype`	Data type	`float32`

SmolVLA Policy Parameters - Input/Output Structure

Parameter	Description	Default / Notes
`policy.n_obs_steps`	Number of observation steps	`1`
`policy.chunk_size`	Action chunk size	`50`
`policy.n_action_steps`	Number of action steps to execute	`50`
`policy.max_state_dim`	Maximum state dimension (inputs are padded to this size)	`32`
`policy.max_action_dim`	Maximum action dimension (inputs are padded to this size)	`32`

SmolVLA Policy Parameters - Decoding and Tokenizer

Parameter	Description	Default / Notes
`policy.num_steps`	Number of denoising steps during inference	`10`
`policy.tokenizer_max_length`	Maximum tokenizer length	`48`
`policy.use_cache`	Whether to use the attention cache	`true`

SmolVLA Policy Parameters - Images and Cameras

Parameter	Description	Default / Notes
`policy.image_resolution`	Image preprocessing resolution (H, W)	`[224,224]`
`policy.empty_cameras`	Number of empty cameras used for padding	`0`
`policy.add_image_special_tokens`	Whether to use special image tokens	`false`

SmolVLA Policy Parameters - Fine-Tuning Settings

Parameter	Description	Default / Notes
`policy.freeze_vision_encoder`	Whether to freeze the vision encoder	`true`
`policy.train_expert_only`	Whether to train only the Action Expert	`true`
`policy.train_state_proj`	Whether to train the state projection layer	`true`

SmolVLA Policy Parameters - Transformer Architecture

Parameter	Description	Default / Notes
`policy.attention_mode`	Attention mode	`cross_attn`
`policy.num_vlm_layers`	Number of VLM layers used	`16`
`policy.self_attn_every_n_layers`	Insert a self-attention layer every N layers	`2`
`policy.expert_width_multiplier`	Action Expert hidden-layer width multiplier	`0.75`

SmolVLA Policy Parameters - Optimizer

Parameter	Description	Default / Notes
`policy.optimizer_lr`	Learning rate	`1e-4`
`policy.optimizer_betas`	AdamW betas	`[0.9,0.95]`
`policy.optimizer_eps`	AdamW eps	`1e-8`
`policy.optimizer_weight_decay`	Weight decay	`1e-10`
`policy.optimizer_grad_clip_norm`	Gradient clipping norm	`10.0`

SmolVLA Policy Parameters - Learning Rate Scheduler

Parameter	Description	Default / Notes
`policy.scheduler_warmup_steps`	Number of warmup steps	`1000`
`policy.scheduler_decay_steps`	Number of decay steps	`30000`
`policy.scheduler_decay_lr`	Final learning rate after decay	`2.5e-6`

SmolVLA Policy Parameters - Training Settings

Parameter	Description	Default / Notes
`policy.gradient_checkpointing`	Whether to enable gradient checkpointing	`false`
`policy.compile_model`	Whether to compile the model	`false`
`policy.compile_mode`	Compilation mode	`max-autotune`

1. ACT​

2. Diffusion Policy (DP)​

3. π₀ (PI0)​

4. π₀.₅ (PI05)​

5. SmolVLA​

1. ACT

2. Diffusion Policy (DP)

3. π₀ (PI0)

4. π₀.₅ (PI05)

5. SmolVLA