Automatic Data Collection

This document describes the automatic data collection feature provided by the Global Humanoid Robot Challenge 2026 Baseline. In the Walker S2 simulation environment, the system can automatically perform grasping, sorting, foam inlaying, or action replay while saving robot states, actions, and four camera streams as a LeRobotDataset.

Automatic collection does not require keyboard teleoperation. The entry point is src/lerobot/scripts/auto_collect_main.py, and the core implementation is located in src/lerobot/auto_collect/.

1. Overview

1.1 Supported Tasks

Task	Description	Execution
Part_Sorting	Automatically grasps parts and places them into the corresponding bins by type	Right arm only
Conveyor_Sorting	Reads part positions from the conveyor in real time and sorts the parts	Right arm only
Foam_Inlaying	Automatically grasps large and small workpieces and places them into the corresponding cutouts	Dual-arm
Packing_Box	Replays actions from an existing dataset and records them again	Dual-arm action replay

1.2 Collected Data

The following data can be recorded synchronously during each control cycle:

• Walker S2 robot states and control actions
• RGB images from the left and right head cameras
• RGB images from the left and right wrist cameras
• Dataset metadata, including the episode, task description, and timestamps

2. Quick Start

Specify the task name, dataset identifier, local output directory, and number of episodes when starting a collection run. --auto_collect.repo_id uses the format {user-or-organization}/{dataset-name}. This parameter is required even when the dataset is stored only locally; data is not uploaded automatically.

2.1 Task 1: Part Sorting

/isaac-sim/python.sh -m lerobot.scripts.auto_collect_main \
    --robot.type=walker_s2_sim \
    --auto_collect.task=Part_Sorting \
    --auto_collect.repo_id=your_org/part_sorting_auto \
    --auto_collect.root=datasets/Part_Sorting/auto/v0 \
    --auto_collect.num_episodes=100 \
    --auto_collect.single_task="Part Sorting" \
    --auto_collect.record_data=true \
    --auto_collect.video=true \
    --auto_collect.fps=30

The right arm automatically approaches, grasps, lifts, and sorts each part. Data is saved after each successful episode.

2.2 Task 2: Conveyor Sorting

/isaac-sim/python.sh -m lerobot.scripts.auto_collect_main \
    --robot.type=walker_s2_sim \
    --auto_collect.task=Conveyor_Sorting \
    --auto_collect.repo_id=your_org/conveyor_sorting_auto \
    --auto_collect.root=datasets/Conveyor_Sorting/auto/v0 \
    --auto_collect.num_episodes=250 \
    --auto_collect.single_task="Conveyor Sorting" \
    --auto_collect.record_data=true \
    --auto_collect.video=true \
    --auto_collect.fps=30

The program reads the world coordinates of conveyor parts in real time. It places part_a in the right bin and part_b in the left bin. If execution exceeds the timelimit configured in Ubtech_sim/config/Conveyor_Sorting.yaml, the current episode is retried according to the max_retries setting.

Segmented Collection

Use the segmented collection entry point to generate both short, single-part episodes and long, full-process episodes:

/isaac-sim/python.sh -m lerobot.scripts.auto_collect_conveyor_segmented \
    --robot.type=walker_s2_sim \
    --auto_collect.task=Conveyor_Sorting \
    --auto_collect.repo_id=your_org/conveyor_sorting_segmented \
    --auto_collect.root=datasets/Conveyor_Sorting/auto/segmented_v0 \
    --auto_collect.num_episodes=100 \
    --auto_collect.single_task="Conveyor Sorting" \
    --auto_collect.segmented_failure_mode=loosen

The output directory contains separate short_episodes and long_episodes directories:

Failure Mode	Behavior
loosen	Skips the failed part and continues collecting the current long episode
strict	Marks the current long episode as failed

2.3 Task 3: Foam Inlaying

/isaac-sim/python.sh -m lerobot.scripts.auto_collect_main \
    --robot.type=walker_s2_sim \
    --auto_collect.task=Foam_Inlaying \
    --auto_collect.repo_id=your_org/foam_inlaying_auto \
    --auto_collect.root=datasets/Foam_Inlaying/auto/v0 \
    --auto_collect.num_episodes=100 \
    --auto_collect.single_task="Foam Inlaying" \
    --auto_collect.record_data=true \
    --auto_collect.video=true \
    --auto_collect.fps=30 \
    --auto_collect.arm_execution_mode=dual

arm_execution_mode supports the following execution modes:

Value	Execution Order
left_then_right	Executes the left-arm motion first, followed by the right-arm motion
right_then_left	Executes the right-arm motion first, followed by the left-arm motion

2.4 Task 4: Packing-Box Action Replay

/isaac-sim/python.sh -m lerobot.scripts.auto_collect_main \
    --robot.type=walker_s2_sim \
    --auto_collect.task=Packing_Box \
    --auto_collect.repo_id=your_org/packing_box_auto \
    --auto_collect.root=datasets/Packing_Box/auto/v0 \
    --auto_collect.num_episodes=100 \
    --auto_collect.single_task="Packing Box" \
    --auto_collect.record_data=true \
    --auto_collect.video=true \
    --auto_collect.fps=30 \
    --auto_collect.source_repo_id=your_org/packing_box_source \
    --auto_collect.source_root=datasets/Packing_Box/source \
    --auto_collect.source_episode=0 \
    --auto_collect.source_step_interval=1

Packing Box uses actions loaded from the specified source episode instead of IK planning. After each scene reset, the program replays the same source episode and creates a new collection episode until num_episodes is reached.

The 18-dimensional actions in the source dataset are mapped to the 20-dimensional actions used by Walker S2, with left_gripper and right_gripper control values added. source_step_interval=1 replays every frame, whereas 2 selects every second frame. Increasing this value reduces the number of replayed and recorded frames.

3. Configuration

3.1 General Parameters

Parameter	Description	Default
--auto_collect.task	Task name	Part_Sorting
--auto_collect.repo_id	Output dataset identifier	Empty; required during collection
--auto_collect.root	Local output directory	./outputs/auto_collect
--auto_collect.num_episodes	Target number of episodes	1
--auto_collect.single_task	Task description written to the dataset	Uses the task name when omitted
--auto_collect.fps	Control and collection frequency	30
--auto_collect.record_data	Writes data to the dataset; when disabled, runs the process once without recording	true
--auto_collect.video	Encodes camera frames as video	true
--auto_collect.max_retries	Maximum retries for one episode	20
--auto_collect.objects_per_episode	Maximum objects processed per episode; 0 means all objects	0
--auto_collect.push_to_hub	Uploads the completed dataset to the Hugging Face Hub	false
--auto_collect.private	Creates a private dataset when uploading	false

3.2 Randomized Speed

When randomized speed is enabled, a speed scaling factor is sampled from the configured range for each episode to increase data diversity:

--auto_collect.random_speed=true \
--auto_collect.speed_scale_min=0.5 \
--auto_collect.speed_scale_max=2.0

1.0 is the original speed. Values below 1.0 speed up the motion, while values above 1.0 slow it down. Using identical lower and upper bounds produces a fixed speed.

3.3 Video Encoding

Parameter	Description	Default
--auto_collect.vcodec	Video codec; supported values are h264, auto, and libsvtav1	h264
--auto_collect.video_encoding_batch_size	Number of episodes encoded per batch	1
--auto_collect.streaming_encoding	Encodes video as data is collected	false
--auto_collect.encoder_queue_maxsize	Maximum buffered frames per camera during streaming encoding	30
--auto_collect.encoder_threads	Number of threads per encoder	Automatic
--auto_collect.num_image_writer_processes	Number of PNG writer processes	8
--auto_collect.num_image_writer_threads_per_camera	Number of PNG writer threads per camera	4

• h264 offers broad compatibility and is the default.
• auto lets the encoder select an available implementation, including hardware encoders.
• libsvtav1 provides better compression but is generally slower to encode.
• streaming_encoding=true reduces temporary PNG writes, but the queue size must be tuned to available memory and encoding speed.

4. Collection Controls and Workflow

4.1 Runtime Controls

Key	Action
Esc	Saves the frames recorded so far and ends collection
Left Arrow	Clears the current episode buffer and restarts the episode
Ctrl+C	Stops collection; the program attempts to save recorded frames and release resources

4.2 Standard Collection Workflow

Except for Packing Box action replay, the standard collection workflow is as follows:

1. Load the task YAML file, then create and connect the Walker S2 simulated robot.
2. Initialize LeRobotDataset and wait for the physics engine to stabilize.
3. Reset the scene and obtain the randomized part positions.
4. Select a single-arm or dual-arm strategy according to the task, then automatically approach, descend, grasp, lift, move, and place each object.
5. Record robot observations and actions during each control cycle. When video is enabled, record all four camera streams at the same time.
6. If a grasp fails, clear the current episode and retry it. Save the episode after successful completion.
7. After reaching the target episode count, finish video encoding and optionally upload the dataset to the Hugging Face Hub.

5. Output Data

5.1 Dataset Fields

By default, the dataset contains robot states, actions, and the following four RGB camera fields:

• observation.images.head_left
• observation.images.head_right
• observation.images.wrist_left
• observation.images.wrist_right

5.2 Directory Structure

A typical output directory has the following structure:

datasets/<task>/auto/v0/
|-- data/
|   `-- chunk-000/
|       `-- file-000.parquet
|-- meta/
|   |-- episodes/
|   |-- info.json
|   |-- stats.json
|   `-- tasks.parquet
`-- videos/
    |-- observation.images.head_left/
    |-- observation.images.head_right/
    |-- observation.images.wrist_left/
    `-- observation.images.wrist_right/

6. Advanced Configuration and Extension

6.1 Overriding Task Configuration

Scene settings, object counts, grasp parameters, and timeout settings for each task are defined in Ubtech_sim/config/<task-name>.yaml. To temporarily override YAML parameters, pass JSON through the robot configuration:

--robot.task_cfg_overrides='{"part":{"fixed_spawn":{"enabled":false},"num_parts_a":2,"num_parts_b":2}}'

Refer to the YAML file for the selected task for the available fields. Command-line overrides apply only to the current run and do not modify the configuration file.

6.2 Code Structure

src/lerobot/
|-- auto_collect/
|   |-- auto_collect_base.py
|   |-- auto_collect_config.py
|   |-- task_part_sorting.py
|   |-- task_conveyor_sorting.py
|   |-- task_conveyor_sorting_segmented.py
|   |-- task_foam_inlaying.py
|   `-- task_packing_box.py
`-- scripts/
    |-- auto_collect_main.py
    `-- auto_collect_conveyor_segmented.py

AutoCollectBase handles dataset initialization, robot connection, motion interpolation, frame-by-frame recording, failure retries, and episode management. Each task subclass computes grasp and placement poses, evaluates grasp success, and implements its task-specific execution sequence.

6.3 Adding a New Task

1. Add an AutoCollectBase subclass under src/lerobot/auto_collect/.
2. Implement compute_grasp_poses(), check_grasp_success(), get_place_pose(), and _execute_sequence().
3. For a dual-arm task, set is_dual_arm = True and implement get_arm_side() and _on_episode_start() as needed.
4. Register the task name and collector class in _COLLECTOR_REGISTRY in src/lerobot/scripts/auto_collect_main.py.

New tasks should continue to reuse _joint_interpolate_to_pose(), _cartesian_interpolate_to_pose(), gradually_move_gripper(), and _record_frame() from the base class to keep control, recording, and failure-handling behavior consistent.