Planet ROS

Planet ROS - http://planet.ros.org

Planet ROS - http://planet.ros.org http://planet.ros.org

ROS Discourse General: ROS2 multi-machine with WSL2 + Linux (NVIDIA Jetson Orin Nano)

Hi everyone,

I am trying setting up a ROS2 multi-machine communication between WSL2 (Ubuntu 22.04) running on a local Windows11 machine and a Linux-native platform, i.e., NVIDIA Jetson Orin Nano. My objective is generating data on my WSL2 machine, and process them using the Jetson.

However, I am experiencing issues related to the fact that the two machines cannot see the respective topics / nodes. They are both connected to the same network, and I have activated the mirroring mode networking in WSL2 using the flag networkingMode=mirrored. I have also tried using Cyclone and FastDDS using custom configuration files, but nothing worked so far.

I was thus wondering if anybody has succeded in setting up a proper ROS2 communication between WSL2 and a native Linux machine. If you need any further information, please do not hesitate to ask.

Thank you in advance!

Simone

2 posts - 2 participants

Read full topic

https://discourse.openrobotics.org/t/ros2-multi-machine-with-wsl2-linux-nvidia-jetson-orin-nano/55452

ROS Discourse General: STS3215 pan/tilt + LD19: a no-SDK 3D scanning module for ROS 2 Jazzy

I put together a small ROS 2 subsystem that turns a 2-DOF pan/tilt platform and a cheap 2D LiDAR into a stop-and-capture 3D scanner, and figured it might be useful to someone else here.

The setup: two Feetech STS3215 serial-bus servos aim an LDROBOT LD19. A node sweeps the platform and an assembler stacks the 2D scans into a `PointCloud2` using the live TF tree. There’s an optional MQTT bridge so an external controller (in my case a microcontroller mission queue on a rover) can trigger scans and get a completion handshake back.

It’s a *complete* project — it even includes a fix to the LiDAR driver (upstream `ldlidar_stl_ros2` won’t build on recent GCC/glibc; the patched fork is linked below). It talks to the rover over a well-defined set of MQTT messages, but every command also has an equivalent ROS 2 topic, so if you want a pure ROS 2 setup you just don’t launch the bridge. (Personally I love the MQTT side — it lets me drive the whole thing from a tablet.)

No vendor SDK — the Feetech STS/SMS half-duplex protocol is implemented directly over pyserial, including handling the URT-1 adapter’s habit of echoing every TX byte back on the RX line (the kind of thing that eats an evening if you don’t know it’s coming). The assembler is driver-agnostic: it consumes standard `sensor_msgs/LaserScan` on `/scan`, so any conformant 2D LiDAR should work.

It’s running on an RK3588 today and is built to go headless on a Pi 5.

This is the first piece I’m open-sourcing from a larger autonomous rover project, GPL-3.0. I’d genuinely welcome feedback — particularly from anyone who’s done multi-LiDAR or TF-timing work, since the scan-to-TF synchronization was the fussiest part to get right. But it does work! Happy to answer questions about any of it.

- Project: GitHub - aa2mz/pan_tilt_lidar: A ROS 2 (Jazzy) subsystem that turns a 2-DOF pan/tilt platform and a 2D LiDAR into a 3D scanner. · GitHub

- Patched LiDAR driver: GitHub - aa2mz/ldlidar_stl_ros2: LDROBOT DTOF LiDAR ROS2 Package · GitHub

3 posts - 3 participants

Read full topic

https://discourse.openrobotics.org/t/sts3215-pan-tilt-ld19-a-no-sdk-3d-scanning-module-for-ros-2-jazzy/55447

ROS Discourse General: Isaac Lab Teleoperation & Data Collection 2.0: Automated Block Stacking Dataset Generation with AgileX PiPER and NERO

0.Introduction

In our previous article, we introduced a complete teleoperation and demonstration collection pipeline for the AgileX NERO and PiPER robotic arms in Isaac Lab.

The project demonstrated how to:

Import custom robot assets into Isaac Lab
Configure Differential IK control
Perform keyboard-based teleoperation
Record demonstrations into HDF5 datasets
Replay trajectories for verification

Previous article:

Isaac Lab Teleoperation & Data Collection: Controlling the AgileX NERO 7-DoF Robotic Arm

Repository: The full source code is available in the repository.

While teleoperation is an effective approach for collecting expert demonstrations, scaling datasets through manual operation quickly becomes time-consuming and difficult to maintain consistently.

In this follow-up work, we extend the original pipeline with an automated block stacking data generation framework that can autonomously complete manipulation tasks and continuously export successful demonstrations.

What’s New in this Tutorial ?

Compared with the previous teleoperation workflow, this version introduces:

Automated trajectory generation

Autonomous block stacking execution

Differential IK-based task completion

PCHIP trajectory smoothing

Automatic success detection

Continuous dataset generation

Zero human intervention during collection

The result is a scalable demonstration generation pipeline suitable for robot learning workflows.

1.Systerm Architecture

The overall workflow is shown below:


Cube Position Detection

          ↓

Trajectory Planner

          ↓

IK Stack Controller

          ↓

Differential IK (Isaac Lab)

          ↓

Robot Motion

          ↓

Recorder Manager

          ↓

HDF5 Dataset Export

Unlike the teleoperation workflow, no human operator is involved once the simulation starts.

The controller generates Cartesian target motions, while Isaac Lab’s built-in Differential IK module handles inverse kinematics computation internally.

2.Implementation Steps

Step 1.Automated Block Stacking Controller

A new script was introduced:

scripts/tools/record_ik_stack.py

The controller executes a complete stacking sequence:


Approach Cube

      ↓

Descend

      ↓

Grasp

      ↓

Lift

      ↓

Move To Target

      ↓

Place

      ↓

Retreat

Instead of commanding joint angles directly, the controller outputs Cartesian pose increments:

action = [ Δx,Δy,Δz,Δrx,Δry,Δrz,gripper ]

This design allows the same controller logic to be reused across different robot configurations.

The newly introduced block stacking controller is shown below.

sequence = [
    (self._shift_z(src, self.Z_PRE), self.GRIP_OPEN_VAL, 30),
    (self._shift_z(src, self.Z_DOWN), self.GRIP_OPEN_VAL, 30),
    (self._shift_z(src, self.Z_DOWN), self.GRIP_CLOSE_VAL, 30),
    (self._shift_z(src, self.Z_LIFT), self.GRIP_CLOSE_VAL, 30),
    (self._shift_z(place_pos, self.Z_PRE), self.GRIP_CLOSE_VAL, 30),
    (self._shift_z(place_pos, self.Z_PLACE), self.GRIP_CLOSE_VAL, 30),
    (self._shift_z(place_pos, self.Z_PLACE), self.GRIP_OPEN_VAL, 20),
    (self._shift_z(place_pos, self.Z_RETREAT), self.GRIP_OPEN_VAL, 30),
]

Code shown below to realize this step

def plan(self) -> None:
    """plan the full pick-and-place trajectory."""
    cube_height = self.cubes[0].data.root_pos_w[:, 2:3].clone(2.0

    all_waypoints: list[torch.Tensor] = []
    all_gripper_vals: list[float] = []
    all_durations: list[int] = []

    base_pos = self.cubes[0].data.root_pos_w.clone()

    for i in range(1, len(self.cubes)):
        src = self.cubes[i].data.root_pos_w.clone()
        place_pos = self._shift_z(base_pos, cube_height)

        sequence = [
            (self._shift_z(src, self.Z_PRE), self.GRIP_OPEN_VAL, 30),
            (self._shift_z(src, self.Z_DOWN), self.GRIP_OPEN_VAL, 30),
            (self._shift_z(src, self.Z_DOWN), self.GRIP_CLOSE_VAL, 30),
            (self._shift_z(src, self.Z_LIFT), self.GRIP_CLOSE_VAL, 30),
            (self._shift_z(place_pos, self.Z_PRE), self.GRIP_CLOSE_VAL, 30),
            (self._shift_z(place_pos, self.Z_PLACE), self.GRIP_CLOSE_VAL, 30),
            (self._shift_z(place_pos, self.Z_PLACE), self.GRIP_OPEN_VAL, 20),
            (self._shift_z(place_pos, self.Z_RETREAT), self.GRIP_OPEN_VAL, 30),
        ]

        for wp, gv, dur in sequence:
            all_waypoints.append(wp.squeeze(0))
            all_gripper_vals.append(gv)
            all_durations.append(dur)

        all_durations[-1] += 30
        base_pos = self._shift_z(base_pos, cube_height)

    # Generate smooth trajectory
    self._path = self._generate_smooth_trajectory(all_waypoinall_durations)

    # Expand gripper timeline
    self._gripper_timeline = []
    for gv, dur in zip(all_gripper_vals, all_durations):
        self._gripper_timeline.extend([gv] * dur)

    self._total_steps = self._path.shape[0]
    self._gripper_timeline = self._gripper_timeline[:s_total_steps]

    self._step_idx = 0
    self._post_traj_wait_count = 0
    self._last_gripper_val = self.GRIP_OPEN_VAL

    print(f"[IKStackController] Trajectory planned: {s_total_steps} steps")

Step 2.Differential IK-Based Control

Rather than solving inverse kinematics explicitly, the controller outputs Cartesian pose increments and delegates IK computation to Isaac Lab’s built-in Differential IK controller.

def compute_action(self, env: gym.Env) -> torch.Tensor:
    """Return the action for the current step.
    Returns:
        torch.Tensor of shape ``(1, 7)``:
            ``[Δx, Δy, Δz, Δrx, Δry, Δrz, gripper_binary]``
    """
    # trajectory exhausted → hold position
    if self._step_idx >= self._total_steps:
        return torch.tensor(
            [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, self_last_gripper_val]],
            device=env.device,
        )
    # delta position = target − current
    target_pos = self._path[self._step_idx]  # [1, 3]
    current_pos = self.ee_frame.data.target_pos_w[:, 0, :]  # [1,3]
    delta_pos = target_pos - current_pos  # [1, 3]
    # delta rotation = zero (keep orientation)
    delta_rot = torch.zeros(1, 3, device=env.device)
    # gripper binary command
    gv = self._gripper_timeline[self._step_idx]
    grip_tensor = torch.tensor([[gv]], device=env.device)
    action = torch.cat([delta_pos, delta_rot, grip_tensor],dim=-1)  # (1, 7)
    self._step_idx += 1
    self._last_gripper_val = gv
    return action

Step 3.Trajectory Smoothing with PCHIP

One challenge during autonomous execution is trajectory discontinuity between waypoints.

To improve motion quality, the controller uses Hermite-based interpolation with PCHIP-style tangents to generate smooth Cartesian trajectories.

@staticmethod
def _generate_smooth_trajectory(
    waypoints: list[torch.Tensor], durations: list[int]
) -> torch.Tensor:
    """Interpolate waypoints with PCHIP-style tangents.
    Args:
        waypoints: list of ``(3,)`` tensors.
        durations: steps per segment (``len == len(waypoints) -1``).
    Returns:
        ``torch.Tensor`` of shape ``(Total, 1, 3)``.
    """
    n = len(waypoints)
    wp = torch.stack(waypoints)  # [N, 3]
    tangents = torch.zeros_like(wp)
    if n > 1:
        tangents[0] = wp[1] - wp[0]
        tangents[-1] = wp[-1] - wp[-2]
        for i in range(1, n - 1):
            tangents[i] = (wp[i + 1] - wp[i - 1]) / 2.0
    segments = []
    for i in range(n - 1):
        p0, p1 = wp[i], wp[i + 1]
        m0, m1 = tangents[i], tangents[i + 1]
        steps = durations[i]
        if torch.norm(p1 - p0) < 1e-4:
            seg = p0.unsqueeze(0).repeat(steps, 1)
        else:
            s = torch.linspace(0, 1, steps, device=p0.device)unsqueeze(-1)
            h00 = 2 * s**3 - 3 * s**2 + 1
            h10 = s**3 - 2 * s**2 + s
            h01 = -2 * s**3 + 3 * s**2
            h11 = s**3 - s**2
            seg = h00 * p0 + h10 * m0 + h01 * p1 + h11 * m1
        segments.append(seg)
    return torch.cat(segments, dim=0).unsqueeze(1)  # [Total, 1, 3]

Step 4.Automatic Success Verification

To ensure dataset quality, demonstrations are exported only after the stacking task is successfully completed.

The workflow continuously evaluates the stacking success condition:

Failed trajectories are automatically discarded.

def process_success_condition(env: gym.Env, success_term: object | None, success_step_count: int) -> tuple[int, bool]:
    if success_term is None:
        return success_step_count, False
    if bool(success_term.func(env, **success_term.params)[0]):
        success_step_count += 1
        if success_step_count >= args_cli.num_success_steps:
            env.recorder_manager.record_pre_reset([0], force_export_or_skip=False)
            env.recorder_manager.set_success_to_episodes(
                [0], torch.tensor([[True]], dtype=torch.bool, device=env.device)
            )
            env.recorder_manager.export_episodes([0])
            print("Success condition met! Recording completed.")
            return success_step_count, True
    else:
        success_step_count = 0
    return success_step_count, False

Step 5.Dataset Export

Successful demonstrations are exported automatically using Isaac Lab’s Recorder Manager.

Output format:

ik_dataset.hdf5

The exported datasets contain:

Observations
Actions
Episode Metadata
Success Labels

and can be directly integrated into:

Robomimic
LeRobot
ACT
Diffusion Policy
OpenVLA

with minimal conversion effort.

Step6. Running the Pipeline

PiPER

python scripts/tools/record_ik_stack.py \ 
--task Isaac-Stack-Cube-Piper-IK-Rel-v0 \ 
--device cuda \ 
--dataset_file ./datasets/ik_dataset.hdf5 \ 
--num_demos 10

piper_ik

NERO

python scripts/tools/record_ik_stack.py \ 
--task Isaac-Stack-Cube-Nero-IK-Rel-v0 \ 
--device cuda \ 
--dataset_file ./datasets/ik_dataset.hdf5 \ 
--num_demos 10

nero_ik

3.Why This Matters

This work moves the Isaac Lab workflow from:

Human Demonstration Collection

towards

Automated Demonstration Generation

which is an important step for building large-scale robot learning datasets.

Rather than spending hours manually collecting demonstrations, developers can now generate consistent trajectories automatically and focus on downstream policy training.

FAQ

Q1: Is this solution fully compatible with all Sogot desktop robotic arms?

This solution natively supports the two mainstream Sogot desktop robotic arm models: PiPER and NERO. It features dedicated optimizations for joint limits, movement speed, gripper parameters, and workspace of these two models. You can switch between models directly via corresponding task commands. It is not compatible with other arm models out of the box, but compatibility can be achieved by fine-tuning motion offset parameters and trajectory step lengths.

Q2: How to resolve environment mismatch errors during runtime?

First, confirm you have deployed the IsaacLab base environment with CUDA support, and verify matching versions for PyTorch, gymnasium, and Pinocchio dependencies.

Next, ensure the task parameter in your run command matches your Sogot arm model — PiPER and NERO task commands cannot be interchanged.

Finally, check that GPU acceleration is enabled. This project requires CUDA acceleration for robotic arm inverse kinematics (IK) solving and trajectory computation; running on CPU will throw runtime errors immediately.

Q3: Can the collected datasets be directly used for training physical Sogot robotic arms?

Yes. The datasets output by this solution replicate the physical characteristics of real Sogot arms across motion workspace, control logic, and movement trajectories. Simulation data aligns closely with real hardware control logic, so no extra format conversion is required. The datasets can be directly applied to imitation learning training for physical PiPER and NERO arms, effectively cutting the high cost and operational risks of real-world data sampling.

Q4: How to customize data sample volume and sampling frequency?

Flexible configuration is available via runtime command arguments:

num_demos: Sets the total number of successful demonstration samples to collect; set to 0 for infinite loop sampling.
step_hz: Adjusts the simulation sampling frequency (default: 30 Hz). Tune this value based on your Sogot arm’s movement speed to match different operation paces.

Q5: What causes stuttering trajectories or stacking failures on the robotic arm?

This issue most often stems from two root causes:

CUDA acceleration is not active, leading to delayed IK solving and choppy trajectory calculations.

Mismatch between simulation environment frame rate and robotic arm motion step length.

Troubleshooting steps: First check your GPU operational status. Then fine-tune PCHIP trajectory interpolation step length and Z-axis offset parameters to meet precise operation requirements for Sogot robotic arms.

Have Question?

If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/isaac-lab-teleoperation-data-collection-2-0-automated-block-stacking-dataset-generation-with-agilex-piper-and-nero/55439

ROS Discourse General: Invitation to African Robotics Summit 2026 - June 25, 2026

Dear Friends and Colleagues,

On behalf of the African Robotics Network (AfROB), we are delighted to invite you to the AfROB Virtual Summit, taking place on Thursday, June 25, 2026, at 2:00 - 5:00 PM Central African Time (CAT).

This special gathering will mark the launch of our State of Robotics in Africa report and provide an opportunity for our community to reflect, connect, and contribute to shaping the future of robotics and AI across Africa.

Please find the flyer attached, and you are warmly welcome to register and learn more at afrob.org/summit.

Thank you.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/invitation-to-african-robotics-summit-2026-june-25-2026/55432

ROS Discourse General: EpisodeVault: open source tool to debug why your LeRobot model regressed

Been building with LeRobot v3 and kept hitting the same wall: retrain a policy, it gets worse, no clear idea why. DVC tells you files changed. MLflow tells you which run. Nobody tells you which tasks dropped or which episodes degraded between dataset versions.

Built a small open source library to fill that gap.

Four commands

episodevault track ./my_dataset
episodevault commit -m "added kitchen episodes"
episodevault diff v1.0 v2.0
episodevault blame model_v3

What the diff looks like

Ran this against two real LeRobot datasets:

Dataset diff: v1.0 → v2.0
────────────────────────────────────────────────────
Episodes added:    +0
Episodes removed:  -7

Distribution shift:
  factory_pick        2 → 6   ↑ 200%  ⚠️
  kitchen_grasp       4 → 1   ↓ 75%   ⚠️

Quality metrics:
  avg episode length:    3.7s → 3.0s  ↓
  success_rate:          0.88 → 0.38  ↓
  camera_sync_score:     1.00 → 1.00  →

Likely regression cause:
  'kitchen_grasp' episodes dropped 75% (4 → 1). Restore from prior
  version if this task is in your eval benchmark.

The blame command

One line in your training script:

import episodevault as ev
ev.log_training_run(model_version="model_v3", dataset_version="v2.0")

Then later:

episodevault blame model_v3

Traces the model back to the exact dataset version that trained it and shows the diff automatically.

Compatibility

Tested against four real datasets from the Hub:

Robot	Dataset	Episodes	Frames	Parse time
aloha	aloha_static_pro_pencil	25	8,750	0.35s
aloha	aloha_mobile_shrimp	18	67,500	0.38s
so100	svla_so100_stacking	56	22,956	0.63s
aloha	aloha_mobile_cabinet	85	127,500	2.73s

Install

pip install episodevault

Python 3.10+. Works on any local LeRobot v3 dataset.

GitHub: Rohan-Prabhakar/EpisodeVault

If you have a dataset where this breaks or gives a wrong regression hint, open an issue, that’s the most useful feedback right now.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/episodevault-open-source-tool-to-debug-why-your-lerobot-model-regressed/55401

ROS Discourse General: What's the most annoying part of your ROS 2 workflow?

Been spending more time in ROS 2 lately and getting a feel for the daily grind.

For folks working on real robots, what part of going from code change to seeing it run drives you the most nuts? And have you found anything that actually makes it better?

7 posts - 4 participants

Read full topic

https://discourse.openrobotics.org/t/whats-the-most-annoying-part-of-your-ros-2-workflow/55371

ROS Discourse General: Logging and Observability Guide Discussion | Cloud Robotics WG Meeting 2026-06-15

Please come and join us for this coming meeting at Mon, Jun 15, 2026 4:00 PM UTC→Mon, Jun 15, 2026 5:00 PM UTC, where we will be discussing Logging & Observability tips and best practices. The group has been researching in this area for the past several months, so we are preparing to assemble a guide, then invite previous guest speakers to review it before publishing.

Last session, we continued preparing our questions for the State of Cloud Robotics Survey 2026. This survey has now been released and is available here (all developers using the cloud in any way are invited to respond!), so once the survey has received a few more submissions, we will be analysing and discussing the results. If you’re interested in watching the previous session, the meeting recording is available on YouTube.

The meeting link for next meeting is here, and you can sign up to our calendar or our Google Group for meeting notifications or keep an eye on the Cloud Robotics Hub.

Hopefully we will see you there!

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/logging-and-observability-guide-discussion-cloud-robotics-wg-meeting-2026-06-15/55370

ROS Discourse General: Context & Hardware Setup

I am developing an Autonomous Mobile Robot (AMR) designed to operate inside a dairy cattle barn. The locomotion system suffers from high wheel slippage due to the environment’s muddy, uneven, and unpaved terrain.

My current sensor suite consists of:

Wheel Encoders (Differential drive odometry)
IMU (Providing angular velocity and absolute orientation)
Ultra-Wideband (UWB) Positioning System (1 Tag on the robot, multiple Anchors around the barn)
(Upcoming) Range sensors for obstacle avoidance (cattle detection).

For state estimation, I am using the robot_localization package running a Dual EKF setup to broadcast the map -> odom -> base_footprint transform tree.

The Problem

I am experiencing significant localization inaccuracies, specifically with the robot’s heading (yaw) and absolute positioning.

Heading / Yaw Errors: The estimated orientation is frequently wrong or drifts rapidly. The barn structure contains dense steel pillars (headlocks and support frames) which I suspect are causing massive magnetic disturbances to the IMU’s magnetometer.
UWB Inaccuracy & Jumps: Initially, the UWB update rate was 1 Hz, which caused huge position jumps. I managed to optimize the anchor-tag communication to achieve around 10 Hz. While the frequency issue is resolved, I still observe positional errors between 10 cm to 40 cm, which drastically degrade especially when the robot approaches the steel pillars or when cattle walk between the tag and the anchors.

Current Configuration & Attempts

The first EKF instance fuses wheel odometry and IMU (angular velocity) to output continuous odom -> base_footprint.
The second EKF instance fuses the odom data and absolute UWB positions to output map -> odom.
I have noticed that high multipath reflections and Non-Line-of-Sight (NLOS) conditions caused by steel structures and livestock are heavily corrupting the UWB ranges.

Questions for the Community

Given the harsh environment (muddy terrain, heavy metal interference, dynamic obstacles like cows), I would highly appreciate any advice or architectural suggestions:

How should I handle the IMU in a high-metal environment? Should I completely disable the magnetometer fusion in robot_localization and rely solely on differential_drive odometry + IMU angular velocity (z-axis gyro) for heading, even if it drifts over time?
How can I mitigate UWB multipath/NLOS errors in EKF? Are there proven outlier rejection methods or thresholding techniques within robot_localization (like using the covariance matrix dynamically or mahalanobis_distance filtering) to ignore bad UWB fixes near the steel pillars?
Alternative Fusion Strategies: Has anyone successfully implemented robust UWB-based localization in similar agricultural setups? Would moving towards an Unscented Kalman Filter (UKF) or a factor-graph-based optimization approach (like Ceres or GTSAM) yield better results than standard Dual EKF?

Thank you in advance for your insights!

My ekf configuration :

odometry_ekf_filter_node:
    ros__parameters:
        sensor_timeout: 0.1
# The frequency, in Hz, at which the filter will output a position estimate. Note that the filter will not begin
# computation until it receives at least one message from one of the inputs. It will then run continuously at the
# frequency specified here, regardless of whether it receives more measurements. Defaults to 30 if unspecified.
        frequency: 30.0
        permit_corrected_publication: false
# ekf_localization_node and ukf_localization_node both use a 3D omnidirectional motion model. If this parameter is
# set to true, no 3D information will be used in your state estimate. Use this if you are operating in a planar
# environment and want to ignore the effect of small variations in the ground plane that might otherwise be detected
# by, for example, an IMU. Defaults to false if unspecified.
        two_d_mode: true

# Use this parameter to provide an offset to the transform generated by ekf_localization_node. This can be used for
# future dating the transform, which is required for interaction with some other packages. Defaults to 0.0 if
# unspecified.
        transform_time_offset: 0.0

# Use this parameter to provide specify how long the tf listener should wait for a transform to become available.
# Defaults to 0.0 if unspecified.
        transform_timeout: 0.05

# If you're having trouble, try setting this to true, and then echo the /diagnostics_agg topic to see if the node is
# unhappy with any settings or data.
        print_diagnostics: false

# Whether to publish the acceleration state. Defaults to false if unspecified.
        publish_acceleration: false

# Whether to broadcast the transformation over the /tf topic. Defaults to true if unspecified.
        publish_tf: true
        # initial_state: [2.0,  -8.0,  0.0,
        #             0.0,  0.0,  0.0,
        #             0.0,  0.0,  0.0,
        #             0.0,  0.0,  0.0,
        #             0.0,  0.0,  0.0]
# 1. Set the map_frame, odom_frame, and base_link frames to the appropriate frame names for your system.
#     1a. If your system does not have a map_frame, just remove it, and make sure "world_frame" is set to the value of odom_frame.
# 2. If you are fusing continuous position data such as wheel encoder odometry, visual odometry, or IMU data, set "world_frame"
#    to your odom_frame value. This is the default behavior for robot_localization's state estimation nodes.
# 3. If you are fusing global absolute position data that is subject to discrete jumps (e.g., GPS or position updates from landmark
#    observations) then:
#     3a. Set your "world_frame" to your map_frame value
#     3b. MAKE SURE something else is generating the odom->base_link transform. Note that this can even be another state estimation node
#         from robot_localization! However, that instance should *not* fuse the global data.
        #map_frame: map              # Defaults to "map" if unspecified
        odom_frame: odom            # Defaults to "odom" if unspecified
        base_link_frame: base_footprint  # Defaults to "base_link" if unspecified
        world_frame: odom           # Defaults to the value of odom_frame if unspecified

        odom0: /wheel_odom
        # x, y pozisyon differential modda fuse ediliyor.
        # Twist (vx) kapalı - twist verisine güvenilmiyor.
        # differential: true -> mutlak pose değil, adım adım delta alınıyor,
        # böylece birikimli odometri hatası direkt EKF'e yansımaz.
        odom0_config: [true, true, false,
                      false, false, false,
                      false, false, false,
                      false, false, false,
                      false, false, false]
        odom0_differential: true
        odom0_relative: false
        odom0_queue_size: 10
        odom0_pose_rejection_threshold: 5.0
        odom0_twist_rejection_threshold: 5.0

        imu0: /witmotion/imu
        imu0_config: [false, false, false,
                      false, false, false,  # absolute yaw KAPALI — motor EMI manyetometreyi bozuyor, gradual drift yaratıyor
                      false, false, false,
                      false, false, true,   # yaw_rate (jiroskop) — SLAM map→odom global yaw'ı düzeltiyor
                      false, false, false]
        imu0_differential: false
        imu0_relative: false
        imu0_queue_size: 10
        # EMI koruması: 10.0'dan 3.0'a düşürüldü.
        # Motorlardan kaynaklanan magnetik spike'lar bu eşiği aşarsa elenir.
        # Eğer yaw sürekli reddediliyorsa (diagnostics'te görünür) değeri
        # biraz artır (4.0-5.0) ya da imu0_config[5]'i false yap.
        imu0_pose_rejection_threshold: 3.0
        imu0_twist_rejection_threshold: 5.0
        imu0_linear_acceleration_rejection_threshold: 0.8
        imu0_remove_gravitational_acceleration: true

ekf_filter_node:
  ros__parameters:
    frequency: 10.0
    sensor_timeout: 0.5
    two_d_mode: true          # Set false for 3D navigation (drones, etc.)
    transform_time_offset: 0.0
    transform_timeout: 0.0
    print_diagnostics: true
    debug: false
    publish_tf: false

    map_frame: map
    odom_frame: odom
    base_link_frame: base_footprint
    world_frame: map
    # [x, y, z, roll, pitch, yaw, vx, vy, vz, vroll, vpitch, vyaw, ax, ay, az]
    # initial_state: [2.55,  -7.4,  0.0,
    #                 0.0,  0.0,  0.0,
    #                 0.0,  0.0,  0.0,
    #                 0.0,  0.0,  0.0,
    #                 0.0,  0.0,  0.0]
    # -----------------------------------------------------------------------
    # Sensor 0: Wheel Odometry
    # -----------------------------------------------------------------------
    # Fuse: vx, vy, vyaw (velocities from wheel encoders)
    # [x, y, z, roll, pitch, yaw, vx, vy, vz, vroll, vpitch, vyaw, ax, ay, az]
    odom0: /wheel_odom
    # x, y pozisyon differential modda fuse ediliyor.
    # Twist (vx) kapalı - twist verisine güvenilmiyor.
    # differential: true -> mutlak pose değil, adım adım delta alınıyor,
    # böylece birikimli odometri hatası direkt EKF'e yansımaz.
    odom0_config: [true, true, false,
                  false, false, false,
                  false, false, false,
                  false, false, false,
                  false, false, false]
    odom0_differential: true
    odom0_relative: false
    odom0_queue_size: 10
    odom0_pose_rejection_threshold: 5.0
    odom0_twist_rejection_threshold: 5.0

    imu0: /witmotion/imu
    imu0_config: [false, false, false,
                  false, false, false,  # absolute yaw KAPALI — motor EMI manyetometreyi bozuyor, gradual drift yaratıyor
                  false, false, false,
                  false, false, true,   # yaw_rate (jiroskop) — SLAM map→odom global yaw'ı düzeltiyor
                  false, false, false]
    imu0_differential: false
    imu0_relative: false
    imu0_queue_size: 10
    # EMI koruması: 10.0'dan 3.0'a düşürüldü.
    # Motorlardan kaynaklanan magnetik spike'lar bu eşiği aşarsa elenir.
    # Eğer yaw sürekli reddediliyorsa (diagnostics'te görünür) değeri
    # biraz artır (4.0-5.0) ya da imu0_config[5]'i false yap.
    imu0_pose_rejection_threshold: 3.0
    imu0_twist_rejection_threshold: 0.8
    imu0_linear_acceleration_rejection_threshold: 0.8
    imu0_remove_gravitational_acceleration: true

    # -----------------------------------------------------------------------
    # Sensor 1: UWB Trilateration Position
    # -----------------------------------------------------------------------
    # Fuse: x, y, z (absolute position from trilateration)
    # [x, y, z, roll, pitch, yaw, vx, vy, vz, vroll, vpitch, vyaw, ax, ay, az]
    pose0: /uwb/pose
    pose0_config: [true,  true,  false,
                   false, false, false,
                   false, false, false,
                   false, false, false,
                   false, false, false]
    pose0_differential: false
    pose0_queue_size: 10
    pose0_rejection_threshold: 1.0  # Reject UWB jumps > 2m (NLOS protection)

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/context-hardware-setup/55356

ROS Discourse General: Isaac Lab Teleoperation & Data Collection: Controlling the AgileX NERO 7-DoF Robotic Arm

0. Preface

In this tutorial, we walk through the step-by-step implementation of keyboard teleoperation and demonstration data collection for the AgileX NERO (7-DoF) and PiPER robotic arms within NVIDIA Isaac Lab.

By the end of this guide, you will be able to control the robotic arm in a simulated environment to perform a block-stacking (Pick and Place) task, collect human demonstration data (HDF5), and replay the dataset for imitation learning pipeline verification.

Key Capabilities Supported

Multimodal Teleoperation: Direct keyboard control for end-effector SE(3) pose alignment.
Pick-and-Place (Stacking) Task: Multi-block stacking simulation environment.
Demonstration Collection: Seamless HDF5 data logging compatible with Robomimic.
Trajectory Replay: Instant validation of collected human demonstrations.
Cross-Hardware Support: Out-of-the-box support for AgileX NERO (7-DoF) and PiPER (6-DoF) arms.

Project Open-Source Repo

GitHub: GitHub - szyzp/IsaacLab_Data_Collection · GitHub

1.Environment & Hardware Configuration

1.1 Generate the Unified URDF File

To import the NERO arm into Omniverse, we must first merge its separate XACRO description files into a single, clean URDF.

Navigate to your local repository directory:

source/agx_teleop/agx_description/agx_arm_urdf/nero

Merge nero_description.urdf, nero_with_gripper_flange_description.xacro, and nero_with_gripper_description.xacro into a single file named nero_gripper.urdf. Ensure all visual/collision mesh paths pointing to local ROS packages are resolved to relative directory paths (e.g., ../meshes/).

Here is the complete nero_gripper.urdf:


<?xml version="1.0" ?>

<robot name="agx_arm">

  <link name="world"/>

  <joint name="world_to_base_link" type="fixed">

      <parent link="world"/>

      <child link="base_link"/>

      <origin rpy="0 0 3.14" xyz="0 0 0"/>

  </joint>

  <link name="base_link">

    <inertial>

      <origin rpy="0 0 0" xyz="-0.00319465997 -0.00005467608 0.04321758463"/>

      <mass value="1.06458435"/>

      <inertia ixx="0.00102659855152" ixy="0.00000186219753" ixz="-0.00000295298037" iyy="0.00114399299508" iyz="-0.00000078763492" izz="0.00090872933022"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/base_link.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/base_link.stl"/>

      </geometry>

    </collision>

  </link>

  <link name="link1">

    <inertial>

      <origin rpy="0 0 0" xyz="-0.00076810578 -0.00005718031 -0.00686372765"/>

      <mass value="0.76710504"/>

      <inertia ixx="0.00066689512233" ixy="0.00000273938306" ixz="0.00001463284987" iyy="0.00054501235592" iyz="0.00000183168344" izz="0.00051509465737"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link1.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link1.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint1" type="revolute">

    <origin rpy="0 0 0" xyz="0 0 0.138"/>

    <parent link="base_link"/>

    <child link="link1"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-2.70526" upper="2.70526" velocity="5"/>

  </joint>

  <link name="link2">

    <inertial>

      <origin rpy="0 0 0" xyz="0.00079686657 -0.07890325930 0.00111849422"/>

      <mass value="0.69381908"/>

      <inertia ixx="0.00201093857656" ixy="-0.00002399426785" ixz="-0.00000072824714" iyy="0.00063544232555" iyz="0.00005831284343" izz="0.00181943989593"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link2.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link2.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint2" type="revolute">

    <origin rpy="1.5707963  3.1415926 0" xyz="0 0 0"/>

    <parent link="link1"/>

    <child link="link2"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-1.74" upper="1.74" velocity="5"/>

  </joint>

  <link name="link3">

    <inertial>

      <origin rpy="0 0 0" xyz="0.00014513363 -0.00121378766 -0.04861136795"/>

      <mass value="0.67246544"/>

      <inertia ixx="0.00343229961910" ixy="-0.00000026976739" ixz="0.00000305078107" iyy="0.00338662526897" iyz="-0.00003550220027" izz="0.00028853579710"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link3.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link3.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint3" type="revolute">

    <origin rpy="-1.5707963  0 3.1415926926" xyz="0 -0.31 0"/>

    <parent link="link2"/>

    <child link="link3"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-2.75" upper="2.75" velocity="5"/>

  </joint>

  <link name="link4">

    <inertial>

      <origin rpy="0 0 0" xyz="-0.00031334623 -0.05675192422 0.00076056133"/>

      <mass value="0.48451414"/>

      <inertia ixx="0.00050386693505" ixy="-0.00000529675237" ixz="0.00000006134740" iyy="0.00023502068202" iyz="0.00001815932566" izz="0.00040149523118"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link4.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link4.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint4" type="revolute">

    <origin rpy="-1.5707963  0 3.1415926926" xyz="0 0 0"/>

    <parent link="link3"/>

    <child link="link4"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-1.01" upper="2.14" velocity="5"/>

  </joint>

  <link name="link5">

    <inertial>

      <origin rpy="0 0 0" xyz="0.00265591357 0.00201195753 -0.11620855434"/>

      <mass value="0.58368405"/>

      <inertia ixx="0.00089537686723" ixy="0.00000648218232" ixz="-0.00001463346849" iyy="0.00085866122972" iyz="-0.00004035185637" izz="0.00021774946532"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link5.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link5.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint5" type="revolute">

    <origin rpy="1.5707963  -1.5707963  0" xyz="0 -0.27001 0"/>

    <parent link="link4"/>

    <child link="link5"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-2.75" upper="2.75" velocity="5"/>

  </joint>

  <link name="link6">

    <inertial>

      <origin rpy="0 0 0" xyz="0.00009501505 0.00107831174 -0.00019110990"/>

      <mass value="0.3140406"/>

      <inertia ixx="0.00006027211661" ixy="-0.00000024650903" ixz="0.00000015701294" iyy="0.00004699514792" iyz="0.00000004170875" izz="0.00005756771792"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link6.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link6.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint6" type="revolute">

    <origin rpy="1.5707963  -1.5707963  0" xyz="0 0 0"/>

    <parent link="link5"/>

    <child link="link6"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-0.73" upper="0.95" velocity="5"/>

  </joint>

  <link name="link7">

    <inertial>

      <origin rpy="0 0 0" xyz="-0.00014000 -0.00010000 -0.00275000"/>

      <mass value="0.000001"/>

      <inertia ixx="1.1000000000000001e-07" ixy="0" ixz="0" iyy="1.1000000000000001e-07" iyz="0" izz="1.9999999999999999e-07"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/link7.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/link7.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="joint7" type="revolute">

    <origin rpy="1.5707963 0 0" xyz="0 -0.0235 0"/>

    <parent link="link6"/>

    <child link="link7"/>

    <axis xyz="0 0 1"/>

    <limit effort="100" lower="-1.5707963" upper="1.5707963" velocity="5"/>

  </joint>

  <link name="gripper_flange">

    <inertial>

      <origin rpy="0 0 0" xyz="0.0 0.0 -0.00032"/>

      <mass value="0.04771096"/>

      <inertia ixx="2.697e-05" ixy="0" ixz="0" iyy="4.311e-05" iyz="0" izz="2.479e-05"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/gripper_flange.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/gripper_flange.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="gripper_flange_joint" type="fixed">

    <origin rpy="-1.5708 0 -1.5708" xyz="0.032 0 -0.0235"/>

    <parent link="link7"/>

    <child link="gripper_flange"/>

  </joint>

  <link name="gripper_base">

    <inertial>

      <origin rpy="0 0 0" xyz="-0.000183807162235591 8.05033155577911E-05 0.0321436689908876"/>

      <mass value="0.45"/>

      <inertia ixx="0.00092934" ixy="0.00000034" ixz="-0.00000738" iyy="0.00071447" iyz="0.00000005" izz="0.00039442"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/gripper_base.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/gripper_base.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="gripper_base_joint" type="fixed">

    <origin rpy="0 0 0" xyz="0 0 0.0055"/>

    <parent link="gripper_flange"/>

    <child link="gripper_base"/>

  </joint>

  <link name="gripper_link1">

    <inertial>

      <origin rpy="0 0 0" xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025"/>

      <mass value="0.025"/>

      <inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/gripper_link1.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/gripper_link1.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="gripper_joint1" type="prismatic">

    <origin rpy="1.5707963 0 3.1415926" xyz="0 0 0.1358"/>

    <parent link="gripper_base"/>

    <child link="gripper_link1"/>

    <axis xyz="0 0 1"/>

    <limit effort="10" lower="0" upper="0.05" velocity="3"/>

  </joint>

  <link name="gripper_link2">

    <inertial>

      <origin rpy="0 0 0" xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025"/>

      <mass value="0.025"/>

      <inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>

    </inertial>

    <visual>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/dae/gripper_link2.dae"/>

      </geometry>

    </visual>

    <collision>

      <origin rpy="0 0 0" xyz="0 0 0"/>

      <geometry>

        <mesh filename="../meshes/gripper_link2.stl"/>

      </geometry>

    </collision>

  </link>

  <joint name="gripper_joint2" type="prismatic">

    <origin rpy="1.5707963 0 0" xyz="0 0 0.1358"/>

    <parent link="gripper_base"/>

    <child link="gripper_link2"/>

    <axis xyz="0 0 -1"/>

    <limit effort="10" lower="-0.05" upper="0" velocity="3"/>

  </joint>

</robot>

1.2 Compile URDF to USD Format

Convert the generated URDF into an NVIDIA USD (Universal Scene Description) file using Isaac Lab’s built-in exporter utility:


python ../IsaacLab/scripts/tools/convert_urdf.py 

  /home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/nero/urdf/nero_gripper.urdf 

  /home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/nero/usd/nero.usd 

  --fix-base

1.3 Create the Asset Configuration File

Create assets/nero.py to define the simulated joints, limits, PD controller gains, and stiffness/damping coefficients.


import isaaclab.sim as sim_utils

from isaaclab.actuators import ImplicitActuatorCfg

from isaaclab.assets.articulation import ArticulationCfg

from agx_teleop.assets import AGX_TELEOP_DESCRIPTION_DIR



NERO_GRIPPER_CFG = ArticulationCfg(

    spawn=sim_utils.UsdFileCfg(

        usd_path=f"{AGX_TELEOP_DESCRIPTION_DIR}/agx_arm_urdf/nero/usd/nero.usd",

        rigid_props=sim_utils.RigidBodyPropertiesCfg(

            disable_gravity=False,

            max_depenetration_velocity=5.0,

        ),

        articulation_props=sim_utils.ArticulationRootPropertiesCfg(

            enabled_self_collisions=False, solver_position_iteration_count=8, solver_velocity_iteration_count=0

        ),

    ),

    init_state=ArticulationCfg.InitialStateCfg(

        rot=(1.0, 0.0, 0.0, 0.0),

        joint_pos={

            "joint1": 0.0,

            "joint2": 0.0,

            "joint3": 0.0,

            "joint4": 1.22,

            "joint5": 0.0,

            "joint6": 0.0,

            "joint7": 1.31,

            "gripper_joint1": 0.05,

            "gripper_joint2": -0.05

        },

        joint_vel={".*": 0.0},

    ),

    actuators={

        "arm": ImplicitActuatorCfg(

            joint_names_expr=["joint.*"],

            effort_limit=25.0, 

            velocity_limit=1.5,

            stiffness={

                "joint1": 200.0, 

                "joint2": 170.0,

                "joint3": 120.0,

                "joint4": 80.0,

                "joint5": 50.0,

                "joint6": 20.0,

                "joint7": 10.0

            },

            damping={

                "joint1": 100.0,

                "joint2": 60.0,

                "joint3": 70.0,

                "joint4": 24.0,

                "joint5": 20.0,

                "joint6": 10.0,

                "joint7": 5,

            },

        ),

        "gripper": ImplicitActuatorCfg(

            joint_names_expr=\["gripper_joint1","gripper_joint2"\],

            effort_limit_sim=22,

            velocity_limit_sim=1.5,

            stiffness=800.0,

            damping=20.0,

        ),

    },

    soft_joint_pos_limit_factor=0.9,

)



NERO_GRIPPER_HIGH_PD_CFG = NERO_GRIPPER_CFG.copy()

NERO_GRIPPER_HIGH_PD_CFG.spawn.rigid_props.disable_gravity = True

2.Task Environment Configuration

2.1 Configure Directory Tree Structure

Setup your task configurations under the standard Isaac Lab layout:

Adapted from the official Isaac Lab stacking task implementation: isaaclab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack


cd agx_teleop

mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/config/nero/agents/robomimic

Create the required task files:


cd source/agx_teleop/agx_teleop/tasks/manager_based/

touch   manipulation/stack/config/nero/__init__.py 

        manipulation/stack/config/nero/agents/__init__.py 

        manipulation/stack/config/nero/agents/robomimic/bc_rnn_low_dim.json 

        manipulation/stack/config/nero/nero_stack_ik_rel_env_cfg.py 

        manipulation/stack/config/nero/nero_stack_joint_pos_env_cfg.py

3. Implement Configurations

3.1 `bc_rnn_low_dim.json` (Robomimic Behavior Cloning Profile)

This file defines policy network structures (LSTM with Gaussian Mixture Models) for downstream imitation learning. Copy your target JSON config as-is without extra modifications:


{

    "algo_name": "bc",

    "experiment": {

        "name": "bc_rnn_low_dim_franka_stack",

        "validate": false,

        "logging": { "terminal_output_to_txt": true, "log_tb": true },

        "save": {

            "enabled": true,

            "every_n_seconds": null,

            "every_n_epochs": 100,

            "epochs": \[\],

            "on_best_validation": false,

            "on_best_rollout_return": false,

            "on_best_rollout_success_rate": true

        },

        "epoch_every_n_steps": 100,

        "env": null,

        "additional_envs": null,

        "render": false,

        "render_video": false,

        "rollout": { "enabled": false }

    },

    "train": {

        "data": null,

        "num_data_workers": 4,

        "hdf5_cache_mode": "all",

        "hdf5_use_swmr": true,

        "hdf5_normalize_obs": false,

        "hdf5_filter_key": null,

        "hdf5_validation_filter_key": null,

        "seq_length": 10,

        "dataset_keys": [ "actions" ],

        "goal_mode": null,

        "cuda": true,

        "batch_size": 100,

        "num_epochs": 2000,

        "seed": 101

    },

    "algo": {

        "optim_params": {

            "policy": {

                "optimizer_type": "adam",

                "learning_rate": {

                    "initial": 0.001,

                    "decay_factor": 0.1,

                    "epoch_schedule": [],

                    "scheduler_type": "multistep"

                },

                "regularization": { "L2": 0.0 }

            }

        },

        "loss": { "l2_weight": 1.0, "l1_weight": 0.0, "cos_weight": 0.0 },

        "actor_layer_dims": [],

        "gmm": {

            "enabled": true,

            "num_modes": 5,

            "min_std": 0.0001,

            "std_activation": "softplus",

            "low_noise_eval": true

        },

        "rnn": {

            "enabled": true,

            "horizon": 10,

            "hidden_dim": 400,

            "rnn_type": "LSTM",

            "num_layers": 2,

            "open_loop": false,

            "kwargs": { "bidirectional": false }

        }

    },

    "observation": {

        "modalities": {

            "obs": {

                "low_dim": [ "eef_pos", "eef_quat", "gripper_pos", "object" ],

                "rgb": [], "depth": [], "scan": []

            }

        }

    }

}

3.2 `nero_stack_joint_pos_env_cfg.py` (Joint Position Environment)

Sets up state configurations, randomizing object spawn spaces and assigning physical bounds to NERO.


from isaaclab.assets import RigidObjectCfg

from isaaclab.managers import EventTermCfg as EventTerm

from isaaclab.managers import SceneEntityCfg

from isaaclab.sensors import FrameTransformerCfg

from isaaclab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg

from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg

from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg

from isaaclab.utils import configclass

from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR



from agx_teleop.tasks.manager_based.manipulation.stack import mdp

from agx_teleop.tasks.manager_based.manipulation.stack.mdp import piper_stack_events

from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import StackEnvCfg



from isaaclab.markers.config import FRAME_MARKER_CFG  

from agx_teleop.assets.nero import NERO_GRIPPER_CFG  



@configclass

class EventCfg:

    init_nero_arm_pose = EventTerm(

        func=piper_stack_events.set_default_joint_pose,

        mode="reset",

        params={

            "default_pose": [0.0, 0.0, 0.0, 1.22, 0.0, 0.0, 1.31, 0.05, -0.05],   

        },

    )



    randomize_nero_joint_state = EventTerm(

        func=piper_stack_events.randomize_joint_by_gaussian_offset,

        mode="reset",

        params={

            "mean": 0.0,

            "std": 0.02,

            "asset_cfg": SceneEntityCfg("robot"),

        },

    )



    randomize_cube_positions = EventTerm(

        func=piper_stack_events.randomize_object_pose,

        mode="reset",

        params={

            "pose_range": {"x": (0.3, 0.5), "y": (-0.10, 0.10), "z": (0.0203, 0.0203), "yaw": (-1.0, 1, 0)},

            "min_separation": 0.12,

            "asset_cfgs": [SceneEntityCfg("cube_1"), SceneEntityCfg("cube_2"), SceneEntityCfg("cube_3")],

        },

    )



@configclass

class NeroCubeStackEnvCfg(StackEnvCfg):

    def __post_init__(self):

        super().__post_init__()



        self.events = EventCfg()

        self.scene.robot = NERO_GRIPPER_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

        self.scene.robot.spawn.semantic_tags = [("class", "robot")]

        self.scene.table.spawn.semantic_tags = [("class", "table")]

        self.scene.plane.semantic_tags = [("class", "ground")]



        self.actions.arm_action = mdp.JointPositionActionCfg(

            asset_name="robot", joint_names=["joint.*"], scale=0.5, use_default_offset=True

        )

        self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(

            asset_name="robot",

            joint_names=["gripper_joint.*"],

            open_command_expr={"gripper_joint1": 0.05, "gripper_joint2": -0.05},

            close_command_expr={"gripper_joint1": 0.0, "gripper_joint2": 0.0},

        )

        self.gripper_joint_names = ["gripper_joint.*"]

        self.gripper_open_val = [0.05, -0.05]

        self.gripper_threshold = 0.005



        cube_properties = RigidBodyPropertiesCfg(

            solver_position_iteration_count=16,

            solver_velocity_iteration_count=1,

            max_angular_velocity=1000.0,

            max_linear_velocity=1000.0,

            max_depenetration_velocity=5.0,

            disable_gravity=False,

        )



        self.scene.cube_1 = RigidObjectCfg(

            prim_path="{ENV_REGEX_NS}/Cube_1",

            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0.0, 0.0203], rot=[1, 0, 0, 0]),

            spawn=UsdFileCfg(

                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/blue_block.usd",

                scale=(1.0, 1.0, 1.0),

                rigid_props=cube_properties,

                semantic_tags=[("class", "cube_1")],

            ),

        )

        self.scene.cube_2 = RigidObjectCfg(

            prim_path="{ENV_REGEX_NS}/Cube_2",

            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.55, 0.05, 0.0203], rot=[1, 0, 0, 0]),

            spawn=UsdFileCfg(

                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/red_block.usd",

                scale=(1.0, 1.0, 1.0),

                rigid_props=cube_properties,

                semantic_tags=[("class", "cube_2")],

            ),

        )

        self.scene.cube_3 = RigidObjectCfg(

            prim_path="{ENV_REGEX_NS}/Cube_3",

            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.60, -0.1, 0.0203], rot=[1, 0, 0, 0]),

            spawn=UsdFileCfg(

                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/green_block.usd",

                scale=(1.0, 1.0, 1.0),

                rigid_props=cube_properties,

                semantic_tags=[("class", "cube_3")],

            ),

        )



        marker_cfg = FRAME_MARKER_CFG.copy()

        marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)

        marker_cfg.prim_path = "/Visuals/FrameTransformer"

        self.scene.ee_frame = FrameTransformerCfg(

            prim_path="{ENV_REGEX_NS}/Robot/world",

            debug_vis=False,

            visualizer_cfg=marker_cfg,

            target_frames=[

                FrameTransformerCfg.FrameCfg(

                    prim_path="{ENV_REGEX_NS}/Robot/gripper_base",

                    name="end_effector",

                    offset=OffsetCfg(pos=[0.0, 0.0, 0.125\]),

                ),

                FrameTransformerCfg.FrameCfg(

                    prim_path="{ENV_REGEX_NS}/Robot/gripper_link1",

                    name="tool_rightfinger",

                    offset=OffsetCfg(pos=(0.0, 0.0, 0.0)),

                ),

                FrameTransformerCfg.FrameCfg(

                    prim_path="{ENV_REGEX_NS}/Robot/gripper_link2",

                    name="tool_leftfinger",

                    offset=OffsetCfg(pos=(0.0, 0.0, 0.0)),

                ),

            ],

        )

3.3 Teleoperation, Data Collection & Replay

3.3.1 stack/config/nero/nero_stack_ik_rel_env_cfg.py

Copy the corresponding Isaac Lab code and modify the following:

Update imports and naming
Modify the end-effector offset

Complete code for nero_stack_ik_rel_env_cfg.py:


# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).

# All rights reserved.

#

# SPDX-License-Identifier: BSD-3-Clause

from isaaclab.controllers.differential_ik_cfg import DifferentialIKControllerCfg

from isaaclab.devices.device_base import DeviceBase, DevicesCfg

from isaaclab.devices.keyboard import Se3KeyboardCfg

from isaaclab.devices.openxr.openxr_device import OpenXRDeviceCfg

from isaaclab.devices.openxr.retargeters.manipulator.gripper_retargeter import GripperRetargeterCfg

from isaaclab.devices.openxr.retargeters.manipulator.se3_rel_retargeter import Se3RelRetargeterCfg

from isaaclab.envs.mdp.actions.actions_cfg import DifferentialInverseKinematicsActionCfg

from isaaclab.managers import SceneEntityCfg

from isaaclab.managers import TerminationTermCfg as DoneTerm

from isaaclab.utils import configclass



from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import mdp



from . import nero_stack_joint_pos_env_cfg



##

# Pre-defined configs

##

from agx_teleop.assets.nero import NERO_GRIPPER_HIGH_PD_CFG  # isort: skip




@configclass

class NeroCubeStackEnvCfg(nero_stack_joint_pos_env_cfg.NeroCubeStackEnvCfg):

    def __post_init__(self):

        # post init of parent

        super().__post_init__()



        # Set Nero as robot

        # We switch here to a stiffer PD controller for IK tracking to be better.

        self.scene.robot = NERO_GRIPPER_HIGH_PD_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")



        # Set actions for the specific robot type (nero)

        self.actions.arm_action = DifferentialInverseKinematicsActionCfg(

            asset_name="robot",

            joint_names=["joint.*"],

            body_name="gripper_base",

            controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),

            scale=0.5,

            body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.125]),

        )



        self.teleop_devices = DevicesCfg(

            devices={

                "handtracking": OpenXRDeviceCfg(

                    retargeters=[

                        Se3RelRetargeterCfg(

                            bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT,

                            zero_out_xy_rotation=True,

                            use_wrist_rotation=False,

                            use_wrist_position=True,

                            delta_pos_scale_factor=10.0,

                            delta_rot_scale_factor=10.0,

                            sim_device=self.sim.device,

                        ),

                        GripperRetargeterCfg(

                            bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT, sim_device=self.sim.device

                        ),

                    ],

                    sim_device=self.sim.device,

                    xr_cfg=self.xr,

                ),

                "keyboard": Se3KeyboardCfg(

                    pos_sensitivity=0.05,

                    rot_sensitivity=0.05,

                    sim_device=self.sim.device,

                ),

            }

        )




@configclass

class NeroCubeStackRedGreenEnvCfg(NeroCubeStackEnvCfg):

    def __post_init__(self):

        \# post init of parent

        super().__post_init__()



        self.terminations.success = DoneTerm(

            func=mdp.cubes_stacked,

            params={"cube_1_cfg": SceneEntityCfg("cube_2"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},

        )




@configclass

class NeroCubeStackRedGreenBlueEnvCfg(NeroCubeStackEnvCfg):

    def __post_init__(self):

        # post init of parent

        super().__post_init__()



        self.terminations.success = DoneTerm(

            func=mdp.cubes_stacked,

            params={

                "cube_1_cfg": SceneEntityCfg("cube_2"),

                "cube_2_cfg": SceneEntityCfg("cube_3"),

                "cube_3_cfg": SceneEntityCfg("cube_1"),

            },

        )




@configclass

class NeroCubeStackBlueGreenEnvCfg(NeroCubeStackEnvCfg):

    def __post_init__(self):

        # post init of parent

        super().__post_init__()



        self.terminations.success = DoneTerm(

            func=mdp.cubes_stacked,

            params={"cube_1_cfg": SceneEntityCfg("cube_1"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},

        )




@configclass

class NeroCubeStackBlueGreenRedEnvCfg(NeroCubeStackEnvCfg):

    def __post_init__(self):

        # post init of parent

        super().__post_init__()



        self.terminations.success = DoneTerm(

            func=mdp.cubes_stacked,

            params={

                "cube_1_cfg": SceneEntityCfg("cube_1"),

                "cube_2_cfg": SceneEntityCfg("cube_3"),

                "cube_3_cfg": SceneEntityCfg("cube_2"),

            },

        )

3.3.2 stack/config/nero/init.py**


# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).

# All rights reserved.


# SPDX-License-Identifier: BSD-3-Clause

import gymnasium as gym



from isaaclab_tasks.manager_based.manipulation.stack.config.franka import agents

from agx_teleop.tasks.manager_based.manipulation.stack.config.nero import agents



\##

# Register Gym environments.

##



gym.register(

    id="Isaac-Stack-Cube-Nero-IK-Rel-v0",

    entry_point="isaaclab.envs:ManagerBasedRLEnv",

    kwargs={

        "env_cfg_entry_point": f"{__name__}.nero_stack_ik_rel_env_cfg:NeroCubeStackEnvCfg",

        "robomimic_bc_cfg_entry_point": f"{agents.__name__}:robomimic/bc_rnn_low_dim.json",

    },

    disable_env_checker=True,

)

4. Collect Demonstration Data via Teleoperation


# First, create a dataset directory

mkdir -p datasets

# Record demonstrations via teleoperation.

# This example uses keyboard teleoperation.
# Available devices: spacemouse, keyboard, handtracking

python scripts/tools/record_demos.py 

    --task Isaac-Stack-Cube-Nero-IK-Rel-v0 

    --device cpu --teleop_device keyboard 

    --dataset_file ./datasets/nero_dataset.hdf5 

    --num_demos 10


# Replay the recorded demonstrations

python scripts/tools/replay_demos.py 

    --task Isaac-Stack-Cube-Nero-IK-Rel-v0 

    --device cpu --dataset_file ./datasets/nero_dataset.hdf5

Keyboard teleoperation controls:


Keyboard Controller for SE(3): Se3Keyboard



Reset all commands: R

Toggle gripper (open/close): K



Move arm along x-axis: W/S

Move arm along y-axis: A/D

Move arm along z-axis: Q/E



Rotate arm along x-axis: Z/X

Rotate arm along y-axis: T/G

Rotate arm along z-axis: C/V

FAQ

Q1: How do I adapt this pipeline from the 7-DoF NERO arm to the 6-DoF PiPER arm (or any other custom manipulator)?

To swap the robot asset, you need to update three core configuration files in your task directory:

1.Asset Definition (assets/piper.py): Define the new USD file path, update the default joint positions (joint_pos) to match a 6-DoF configuration, and re-tune the stiffness/damping parameters for the 6 active joints.

2.Action Configuration (piper_stack_joint_pos_env_cfg.py): Modify the joint regular expression to match the PiPER joint names (e.g., changing expression targets from 7 joints to 6). Update the gripper joint names and binary open/close limit values.

3.Differential IK Config (piper_stack_ik_rel_env_cfg.py):Update the body_name parameter to point to the correct end-effector link (e.g., link6 or gripper_base) and adjust the body_offset to match the physical dimension of the PiPER gripper flange.

Q2: The robotic gripper slips or fails to lift the blocks during teleoperation. How do I improve grap stability?

Slipping or unstable grasping in Isaac Lab is usually a physics-tuning issue rather than an algorithmic one. Try the following adjustments:

Increase Gripper Stiffness: In your asset configuration (assets/nero.py or piper.py), increase the gripper actuator stiffness (e.g., to 800.0 or 1000.0) and effort_limit to ensure the gripper applies enough physical force.
Verify Collision Geometries: Open your converted .usd asset in Isaac Sim and check that the collision meshes for the gripper fingers are set to “Convex Decomposition” (or Simplfied Box) rather than a loose bounding box.
Adjust Material Friction: Add a high-friction physics material (e.g., static friction coefficient.

Q3: Are the collected HDF5 datasets directly compatible with popular imitation learning codebases like ACT or Diffusion Policy?

Yes, with minor formatting adjustments. The HDF5 files generated by our record_demos.py script closely follow the standard Robomimic data structure, containing hierarchically organized datasets:

data/demo_x/obs/eef_pos & eef_quat (End-effector pose)
data/demo_x/obs/joint_values (Joint angles)
data/demo_x/actions (User control inputs)

To feed this data into ACT (Action Chunking with Transformers) or Diffusion Policy, you can use Robomimic’s conversion scripts (convert_robomimic_to_numpy.py) or write a simple Python wrapper to map the HDF5 dictionary keys to the observation/action expectations of your specific network.

Q4: The simulation occasionally crashes or the robot “explodes” (joints jitter violently) when starting teleoperation. How do I fix this?

“Exploding” physics usually happen when the control gains (PD) are too aggressive, or the solver parameters are too coarse.

Increase Solver Iterations: In your environment configuration, increase the solver iterations on the robot asset:


articulation_props=sim_utils.ArticulationRootPropertiesCfg(

    solver_position_iteration_count=16, # Increase from 8 to 16

    solver_velocity_iteration_count=4    # Increase from 0 to 4

)

Disable Self-Collisions: Ensure that adjacent links on the arm do not have overlapping collision meshes by setting enabled_self_collisions=False.
Lower Stiffness: If the arm vibrates violently upon contact, slightly lower the joint stiffness in your actuator configuration and add proportional damping (ideal ratio is roughly 10:1 to 20:1 for stiffness to damping).

nero_teleop

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/isaac-lab-teleoperation-data-collection-controlling-the-agilex-nero-7-dof-robotic-arm/55348

ROS Discourse General: Pre-built ROS2 Humble Windows SDK + Standalone RViz2 (No Toolchain Required)

# Pre-built ROS2 Humble Windows SDK + Standalone RViz2 (No Toolchain Required)

Hi everyone,

I’d like to share two projects that may help developers working with ROS2 on Windows.

## The Problem

Compiling ROS2 on Windows has always been painful — it requires a complex toolchain setup (`rosdep`, `colcon`, Python dependencies, environment variables, etc.). Many developers simply give up or switch to WSL2, which has its own limitations for hardware access and GUI applications.

## Ros2Simple — Pre-built ROS2 Humble Windows SDK

**Repository:** GitHub - huming516520/Ros2Simple: Pre-built ROS2 Humble Windows x64 SDK — Build ROS2 apps with CMake directly, no native toolchain required. Supports Linux & ARM-Linux cross-compilation for industrial deployment. · GitHub

A pre-compiled ROS2 Humble SDK for Windows x64 that allows you to build ROS2 applications using **standard CMake** — no ROS2 native toolchain, no `colcon`, no `rosdep`.

### Key features:

- Based on **ROS2 Humble**, Windows x64 Debug build

- Build with **CMake directly** — just point to the SDK path

- No ROS2 toolchain installation required

- Can be **cross-compiled to Linux and ARM-Linux** for industrial deployment

- Includes a working publisher/subscriber demo

### Quick build:

```bash

cd demo && mkdir build && cd build

cmake ..

cmake --build . --config Debug

demo_pubsub.exe pub

```

## Rviz2Windows — Standalone RViz2 for Windows

**Repository:** GitHub - huming516520/Rviz2Windows: Standalone RViz2 for Windows — Double-click to run, no ROS2 installation needed. Built from ROS2 Humble source. Includes all dependencies (Qt5, Ogre3D). · GitHub

A standalone RViz2 that runs on Windows by **double-clicking `rviz2.exe`** — no ROS2 installation, no environment configuration, no dependencies to install.

### Key features:

- Built from **ROS2 Humble** source

- **Zero configuration** — double-click to run

- All dependencies bundled (Qt5, Ogre3D, plugins)

- Works with ROS2 nodes built from Ros2Simple or any DDS-compatible node

### Demo screenshots:

**Publisher / Subscriber running on Windows:**

![pub_sub](https://raw.githubusercontent.com/huming516520/Ros2Simple/main/ScreenShot_2026-06-07_210651_552.png)

**RViz2 running standalone on Windows:**

![rviz2](https://raw.githubusercontent.com/huming516520/Rviz2Windows/main/ScreenShot_2026-06-07_210751_398.png)

## Use Cases

- **Industrial deployment**: Build ROS2 applications on Windows-based industrial PCs

- **Education**: Students can start learning ROS2 without struggling with environment setup

- **Rapid prototyping**: Quick iteration without the overhead of the full ROS2 toolchain

- **Cross-platform products**: Compile the same codebase to Linux / ARM-Linux for production

## License

Both projects are based on ROS2 Humble source code and comply with the original ROS2 licenses (Apache License 2.0 / BSD License). All third-party libraries retain their original licenses.

## Feedback Welcome

I’d love to hear your feedback, suggestions, or issues. Feel free to open GitHub issues or reach out.

If you have commercial needs (custom builds, consulting, industrial deployment, training), you can contact me at: **huming516520@gmail.com**

Thanks!

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/pre-built-ros2-humble-windows-sdk-standalone-rviz2-no-toolchain-required/55340

ROS Discourse General: CS student diving into ROS 2 and Gazebo, looking for feedback and community

Hello everyone, I’m new here and wanted to introduce myself and share my current focus.

I’m currently finishing up my CS degree and along the way stumbled into robotics software and simulation. I recently installed Ubuntu 24.04, added ROS2 Jazzy, and Gazebo Harmonic, and I’m teaching myself Gazebo from scratch to start running physics and navigation simulations. I kind of stumbled on this middleware from just tinkering in linux, and this field is just fascinating to me!

Since I don’t have a physical hardware lab right now, my work is 100% focused on software, coding, and mastering these simulation environments. I am treating my desktop as my entire proving ground.

I’m the only one in my current physical circle working on this stack, so I’m looking for a community to bounce ideas off of, share my progress, and learn from people who actually know what they’re looking at.

Looking forward to learning from you all. Any pointers on avoiding rookie mistakes with ROS 2 or Gazebo would be heavily appreciated, and maybe some tips on useful tools or concepts to learn.

3 posts - 2 participants

Read full topic

https://discourse.openrobotics.org/t/cs-student-diving-into-ros-2-and-gazebo-looking-for-feedback-and-community/55335

ROS Discourse General: Building a Rust tool for zero-allocation Serial Autobauding & Protocol Detection

Problem:
Integrating unknown hardware into the companion computer (a custom 1D LiDAR for altitude hold, telemetry radio from decades ago, and a mysterious motor controller) involves lots of frustration trying to find the baud rate manually. The autobaud solutions that I have come across rely on human-readable ASCII and are therefore of no use in determining the baud rate of binary streams.

Solution (autobaud-core):
I’m developing a cross-platform Rust-based autobaud detector as a library and CLI daemon that can be called programmatically to detect baud rates without user interaction.

As opposed to cycling through possible rates blindfolded, my solution involves a two-tier strategy based on:

Hardware Framing Validation: For Linux systems, I use the nix crate to access termios PARMRK flags. In case the baud rate is incorrect, the software detects it almost immediately at the hardware framing level (the electrical framing is faulty), and it does not waste CPU cycles running the loop.
Zero-Allocation Matching Heuristics: With the electrical framing established, it proceeds to examine the raw &[u8] byte buffer for protocol-specific “magic” bytes (MAVLink v2 = 0xFD, TFmini LiDAR = 0x59 0x59, NMEA = $GPGGA).

I have used ROS2 myself but I am a beginner but could come up with this idea from my experience with working with drones and sensor fusion

Questions from me:

Validation: Would such a CLI utility be a real-world thing used when adding new physical devices?
ROS 2 Integration: I would like to see some integration beyond a command-line utility here. Could we wrap this into a ROS 2 node, e.g., “Dynamic Serial Bridge” that autobauds the connection, determines the protocol, and immediately starts publishing the raw binary data on a certain topic for parsing further?
Protocols missing in action: Besides the aforementioned MAVLink, NMEA, ASCII protocols, what other raw serial protocols are common when interfacing between hardware and ROS?

3 posts - 2 participants

Read full topic

https://discourse.openrobotics.org/t/building-a-rust-tool-for-zero-allocation-serial-autobauding-protocol-detection/55228

ROS Discourse General: 2026 State of Cloud Robotics Survey

Hello, robotics developers!

If you’re using cloud robotics at all, we (the Cloud Robotics Working Group) would appreciate you taking the time to fill out a survey.

We’re gathering data on how cloud robotics is used, what it’s doing well, and what still needs improving.

The survey link is here: https://forms.gle/WxXiRcCkRxxJLUop9

Responses will be published anonymously. Thank you!

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/2026-state-of-cloud-robotics-survey/55226

ROS Discourse General: Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI

Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI

0.Preface

This project provides an Isaac Lab-based teleoperation and demonstration collection workflow for the AgileX Piper robotic arm. It is designed for developers working on Embodied AI, robot learning, and imitation learning pipelines using simulation environments.

The workflow supports:

keyboard-based robot teleoperation
cube stacking manipulation tasks
trajectory recording and replay
imitation learning dataset generation
ROS2-compatible robotic development
Isaac Lab manipulation environments

Project Has Already Completed

employment of the Piper robotic arm inside NVIDIA Isaac Lab
keyboard teleoperation for robot manipulation
cube stacking task environment configuration
URDF-to-USD robot asset conversion
gripper and actuator parameter configuration
observation pipeline customization for Piper grippers
human demonstration data collection workflow
Robomimic-compatible imitation learning configuration
replay support for collected robot trajectories

Recommended Environment

Operating System: Ubuntu 22.04 or compatible Linux distribution
Simulation Framework: NVIDIA Isaac Lab
ROS Version: ROS 2 Humble
Python Version: Python 3.10+

Reference & Resource

Official IsaacLab Tutorial: Welcome to Isaac Lab! — Isaac Lab Documentation
IsaacLab: GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub)
Project Repository: GitHub - szyzp/IsaacLab_Data_Collection · GitHub
Piper Robotic Arm Platform

The following section provides how developers can rapidly build a complete robotic manipulation and data collection workflow, steps are shown below:

Pipeline Overview
Keyboard Input
↓
Robot Teleoperation
↓
Isaac Lab Simulation
↓
Trajectory Recording
↓
Dataset Generation
↓
Imitation Learning / RL Training

Step 1.Create an Isaac Lab External Project

Before building the teleoperation and data collection workflow, first create an external Isaac Lab project environment.

1.1 Create a New Isaac Lab External Project

1.1.1 Initialize the Project

Run the following command inside the Isaac Lab root directory.

Make sure Isaac Lab has already been installed and the isaaclab Conda environment is activated.

conda activate isaaclab
./isaaclab.sh --new

1.1.2 Recommended Project Configuration

Setting	Recommended Value
Project Type	`External`
Project Path	`/home/agilex/isaac-cosmos/`
Project Name	`agx_teleop`
Workflow	`Manager-based\|single-agent`
RL Library	optional
Project Path	`/home/agilex/isaac-cosmos/`

Configuration are shown above

The initial project structure of agx_teleop is as follows:

├── pyproject.toml
├── README.md
├── scripts
│   ├── list_envs.py
│   ├── random_agent.py
│   ├── sb3
│   │   ├── play.py
│   │   └── train.py
│   └── zero_agent.py
└── source
    └── agx_teleop
        ├── agx_teleop
        │   ├── __init__.py
        │   ├── tasks
        │   │   ├── __init__.py
        │   │   └── manager_based
        │   │       ├── agx_teleop
        │   │       │   ├── agents
        │   │       │   │   ├── __init__.py
        │   │       │   │   └── sb3_ppo_cfg.yaml
        │   │       │   ├── agx_teleop_env_cfg.py
        │   │       │   ├── __init__.py
        │   │       │   └── mdp
        │   │       │       ├── __init__.py
        │   │       │       └── rewards.py
        │   │       └── __init__.py
        │   └── ui_extension_example.py
        ├── agx_teleop.egg-info
        │   ├── dependency_links.txt
        │   ├── not-zip-safe
        │   ├── PKG-INFO
        │   ├── requires.txt
        │   ├── SOURCES.txt
        │   └── top_level.txt
        ├── config
        │   └── extension.toml
        ├── docs
        │   └── CHANGELOG.rst
        ├── pyproject.toml
        └── setup.py

1.1.3 Install External Projects

Running the following commands:

cd ../agx_teleop
python -m pip install -e source/agx_teleop

1.2 Configure the PiPER USD Asset

1.2.1 Prepare the URDF

Clone the official AgileX PiPER URDF repository:

cd agx_teleop
mkdir -p source/agx_teleop/agx_description
cd source/agx_teleop/agx_description
git clone https://github.com/agilexrobotics/agx_arm_urdf.git

Then merge:

piper_with_gripper_description.xacro
piper_description.urdf
into a unified:
piper_gripper.urdf

Complete PiPER URDF file content:

<?xml version="1.0" encoding="utf-8"?>
<!-- This URDF was automatically created by SolidWorks to URDF Exporter! Originally created by Stephen Brawner (brawner@gmail.com) 
     Commit Version: 1.6.0-4-g7f85cfe  Build Version: 1.6.7995.38578
     For more information, please see http://wiki.ros.org/sw_urdf_exporter -->
<robot name="piper">
    <link name="world"/>
    <joint name="world_to_base_link" type="fixed">
        <parent link="world"/>
        <child link="base_link"/>
        <origin xyz="0 0 0" rpy="0 0 0"/>
    </joint>
    <link name="base_link">
        <inertial>
            <origin xyz="-0.00473641164191482 2.56829134630247E-05 0.041451518036016" rpy="0 0 0"/>
            <mass value="1.02" />
            <inertia ixx="0.00267433" ixy="-0.00000073" ixz="-0.00017389" iyy="0.00282612" iyz="0.0000004" izz="0.00089624"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/base_link.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/base_link.stl"/>
            </geometry>
        </collision>
    </link>
    <link name="link1">
        <inertial>
            <origin xyz="0.00032 -0.00041 -0.00348" rpy="0 0 0"/>
            <mass value="0.71"/>
            <inertia ixx="0.00050027" ixy="0.00000078" ixz="0.00000626" iyy="0.00040879" iyz="0.00000462" izz="0.00045802"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link1.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link1.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint1" type="revolute">
        <origin xyz="0 0 0.123" rpy="0 0 0"/>
        <parent link="base_link"/>
        <child link="link1"/>
        <axis xyz="0 0 1"/>
        <limit lower="-2.6179938" upper="2.6179938" effort="100" velocity="5"/>
    </joint>
    <link name="link2">
        <inertial>
            <origin xyz="0.21852 -0.00861 0.00126" rpy="0 0 0"/>
            <mass value="1.16"/>
            <inertia ixx="0.00111915" ixy="0.00182916" ixz="0.00019233" iyy="0.06763251" iyz="0.00000767" izz="0.067559"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link2.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link2.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint2" type="revolute">
        <origin xyz="0 0 0" rpy="1.5707963 -0.1357866 -3.1415926"/>
        <parent link="link1"/>
        <child link="link2"/>
        <axis xyz="0 0 1"/>
        <limit lower="0" upper="3.1415926" effort="100" velocity="5"/>
    </joint>
    <link name="link3">
        <inertial>
            <origin xyz="-0.01999 -0.14808 -0.00053" rpy="0 0 0"/>
            <mass value="0.5"/>
            <inertia ixx="0.01347608" ixy="0.00161034" ixz="0.00000066" iyy="0.00046377" iyz="0.00000162" izz="0.01363321"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link3.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link3.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint3" type="revolute">
        <origin xyz="0.28503 0 0" rpy="0 0 -1.7938494"/>
        <parent link="link2"/>
        <child link="link3"/>
        <axis xyz="0 0 1"/>
        <limit lower="-2.9670597" upper="0" effort="100" velocity="5"/>
    </joint>
    <link name="link4">
        <inertial>
            <origin xyz="0.00013 -0.00076 -0.00394" rpy="0 0 0"/>
            <mass value="0.38"/>
            <inertia ixx="0.00017844" ixy="0.00000062" ixz="0.00000067" iyy="0.00018914" iyz="0.00000408" izz="0.00014613"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link4.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link4.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint4" type="revolute">
        <origin xyz="-0.02198 -0.25075 0" rpy="1.5707963 0 0"/>
        <parent link="link3"/>
        <child link="link4"/>
        <axis xyz="0 0 1"/>
        <limit lower="-1.7453292" upper="1.7453292" effort="100" velocity="5"/>
    </joint>
    <link name="link5">
        <inertial>
            <origin xyz="0.00018 -0.06104 -0.00227" rpy="0 0 0"/>
            <mass value="0.383"/>
            <inertia ixx="0.00172318" ixy="0.00000259" ixz="0.00000010" iyy="0.00017936" iyz="0.00000990" izz="0.00171172"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link5.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link5.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint5" type="revolute">
        <origin xyz="0 0 0" rpy="-1.5707963 0 0"/>
        <parent link="link4"/>
        <child link="link5"/>
        <axis xyz="0 0 1"/>
        <limit lower="-1.2217304" upper="1.2217304" effort="100" velocity="5"/>
    </joint>
    <link name="link6">
        <inertial>
            <origin xyz="-0.00003 0.00006 -0.002" rpy="0 0 0"/>
            <mass value="0.007"/>
            <inertia ixx="0.00152112" ixy="0.00000202" ixz="0.00000261" iyy="0.00138626" iyz="0.00000002" izz="0.00031152"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/link6.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/link6.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="joint6" type="revolute">
        <origin xyz="8.8259E-05 -0.091 0" rpy="1.5707963 0 0"/>
        <parent link="link5"/>
        <child link="link6"/>
        <axis xyz="0 0 1"/>
        <limit lower="-2.0943951" upper="2.0943951" effort="100" velocity="5"/>
    </joint>

    <!-- Gripper components -->
    <link name="gripper_base">
        <inertial>
            <origin xyz="-0.000183807162235591 8.05033155577911E-05 0.0321436689908876" rpy="0 0 0"/>
            <mass value="0.45"/>
            <inertia ixx="0.00092934" ixy="0.00000034" ixz="-0.00000738" iyy="0.00071447" iyz="0.00000005" izz="0.00039442"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/gripper_base.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/gripper_base.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="gripper_base_joint" type="fixed">
        <origin xyz="0 0 0" rpy="0 0 0"/>
        <parent link="link6"/>
        <child link="gripper_base"/>
    </joint>
    <link name="gripper_link1">
        <inertial>
            <origin xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025" rpy="0 0 0"/>
            <mass value="0.025"/>
            <inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/gripper_link1.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/gripper_link1.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="gripper_joint1" type="prismatic">
        <origin xyz="0 0 0.1358" rpy="1.5707963 0 0"/>
        <parent link="gripper_base"/>
        <child link="gripper_link1"/>
        <axis xyz="0 0 1"/>
        <limit lower="0" upper="0.05" effort="10" velocity="3"/>
    </joint>
    <link name="gripper_link2">
        <inertial>
            <origin xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025" rpy="0 0 0"/>
            <mass value="0.025"/>
            <inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
        </inertial>
        <visual>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/dae/gripper_link2.dae"/>
            </geometry>
        </visual>
        <collision>
            <origin xyz="0 0 0" rpy="0 0 0"/>
            <geometry>
                <mesh filename="../meshes/gripper_link2.stl"/>
            </geometry>
        </collision>
    </link>
    <joint name="gripper_joint2" type="prismatic">
        <origin xyz="0 0 0.1358" rpy="1.5707963 0 -3.1415926"/>
        <parent link="gripper_base"/>
        <child link="gripper_link2"/>
        <axis xyz="0 0 -1"/>
        <limit lower="-0.05" upper="0" effort="10" velocity="3"/>
    </joint>
</robot>

1.2.2 Convert URDF to USD

Isaac Lab uses USD assets for simulation.

Convert the Piper URDF into a USD robot asset:

python ../IsaacLab/scripts/tools/convert_urdf.py \
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/piper/urdf/piper_gripper.urdf \
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/piper/usd/piper.usd \
--fix-base

Step 2.Configure the PiPER Robot in Isaac Lab

2.1 Create the Robot Asset Configuration

cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/assets

2.1.1 Create the Asset Package

Create the assets/__init__.py file:

"""Package containing robot configurations."""
import os
import toml

##
# Configuration for different assets.
##

# Conveniences to other module directories via relative paths
AGX_TELEOP_EXT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))
"""Path to the source agx_teleop directory."""

AGX_TELEOP_DESCRIPTION_DIR = os.path.join(AGX_TELEOP_EXT_DIR, "agx_description")
"""Path to the agx_description directory."""

# Robot description assets
AGX_TELEOP_DESCRIPTION_DIR = os.path.join(
    AGX_TELEOP_EXT_DIR,
    "agx_description"
)
"""Path to the agx_description directory."""

2.1.2 Configure the PiPER Robot Asset

Create the assets/piper.py file

import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets.articulation import ArticulationCfg

from agx_teleop.assets import AGX_TELEOP_DESCRIPTION_DIR


PIPER_GRIPPER_CFG = ArticulationCfg(

    # USD robot asset
    spawn=sim_utils.UsdFileCfg(
        usd_path=f"{AGX_TELEOP_DESCRIPTION_DIR}/agx_arm_urdf/piper/usd/piper.usd",

        # Rigid body physics settings
        rigid_props=sim_utils.RigidBodyPropertiesCfg(
            disable_gravity=False,
            max_depenetration_velocity=5.0,
        ),

        # Articulation solver settings
        articulation_props=sim_utils.ArticulationRootPropertiesCfg(
            enabled_self_collisions=True,
            solver_position_iteration_count=8,
            solver_velocity_iteration_count=0,
        ),
    ),

    # Initial robot pose
    init_state=ArticulationCfg.InitialStateCfg(
        joint_pos={
            "joint1": 0.0,
            "joint2": 1.57,
            "joint3": -1.57,
            "joint4": 0.0,
            "joint5": 1.22,
            "joint6": 0.0,
            "gripper_joint1": 0.0,
            "gripper_joint2": 0.0,
        },
        pos=(0.0, 0.0, 0.0),
    ),

    # Actuator configuration
    actuators={

        # Shoulder joints
        "piper_shoulder": ImplicitActuatorCfg(
            joint_names_expr=["joint[1-3]"],
            effort_limit_sim=100.0,
            velocity_limit_sim=5.0,
            stiffness=80.0,
            damping=20.0,
        ),

        # Forearm joints
        "piper_forearm": ImplicitActuatorCfg(
            joint_names_expr=["joint[4-6]"],
            effort_limit_sim=100.0,
            velocity_limit_sim=5.0,
            stiffness=40.0,
            damping=10.0,
        ),

        # Parallel gripper
        "gripper": ImplicitActuatorCfg(
            joint_names_expr=[
                "gripper_joint1",
                "gripper_joint2"
            ],
            effort_limit_sim=22,
            velocity_limit_sim=1.5,
            stiffness=800.0,
            damping=20.0,
        ),
    },

    # Soft joint limit scaling
    soft_joint_pos_limit_factor=0.9,
)

"""Configuration for the Piper robot with parallel gripper."""

PIPER_GRIPPER_HIGH_PD_CFG = PIPER_GRIPPER_CFG.copy()
PIPER_GRIPPER_HIGH_PD_CFG.spawn.rigid_props.disable_gravity = True
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_shoulder"].stiffness = 400.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_shoulder"].damping = 80.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_forearm"].stiffness = 200.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_forearm"].damping = 40.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["gripper"].velocity_limit_sim = 0.5
"""Configuration of Piper with gripper robot with stiffer PD control.

This configuration is useful for task-space control using differential IK.
"""

Step 3.Build the Manipulation Environment

3.1 Project Structure Setup

The following structure is modified from the official IsaacLab stack task isaaclab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack. We will create a specialized directory for the PiPER robot under the agx_teleop project.

Create Directories

cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/config/piper/agents/robomimic
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/mdp

Initialize Files

cd source/agx_teleop/agx_teleop/tasks/manager_based/

touch manipulation/__init__.py \
      manipulation/stack/__init__.py \
      manipulation/stack/stack_env_cfg.py \
      manipulation/stack/mdp/__init__.py \
      manipulation/stack/mdp/observations.py \
      manipulation/stack/mdp/piper_stack_events.py \
      manipulation/stack/mdp/terminations.py \
      manipulation/stack/config/__init__.py \
      manipulation/stack/config/piper/__init__.py \
      manipulation/stack/config/piper/agents/__init__.py \
      manipulation/stack/config/piper/agents/robomimic/bc_rnn_low_dim.json \
      manipulation/stack/config/piper/piper_stack_ik_rel_env_cfg.py \
      manipulation/stack/config/piper/piper_stack_joint_pos_env_cfg.py

3.2 Core File Modifications

3.2.1 Standard IsaacLab Porting

The following files are ported directly from the IsaacLab Franka stack example without significant modification. They define the base MDP (Markov Decision Process) settings and the Robomimic imitation learning configuration.

stack/stack_env_cfg.py: Defines the scene (table, lights, ground) and observation groups.
stack/mdp/init.py: Handles module exports.
stack/config/piper/agents/robomimic/bc_rnn_low_dim.json: Standard BC-RNN configuration for low-dimensional observations.

Copy the corresponding code from IsaacLab without modification :

/Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/stack_env_cfg.py
/Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/__init__.py
/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/agents/robomimic/bc_rnn_low_dim.json

3.2.2 Event Customization (mdp/piper_stack_events.py)

Copy the corresponding code from Issaclab directly, without any modifying :

Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/mdp/franka_stack_events.py

3.2.3 Adapting Observations & Terminations for Piper

Copy the corresponding code of isaaclab and modify the part related to gripper_open_val. The Franka panda gripper control uses a single joint value, while the piper gripper control uses two joint values, which need to be adjusted to match the piper gripper.

Key Logic Adjustment (Lines 320-343 & 380-407):

stack/mdp/observations.py

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from __future__ import annotations

from typing import TYPE_CHECKING, Literal

import torch

import isaaclab.utils.math as math_utils
from isaaclab.assets import Articulation, RigidObject, RigidObjectCollection
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformer

if TYPE_CHECKING:
    from isaaclab.envs import ManagerBasedRLEnv

def cube_positions_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
    """The position of the cubes in the world frame."""
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]

    return torch.cat((cube_1.data.root_pos_w, cube_2.data.root_pos_w, cube_3.data.root_pos_w), dim=1)

def instance_randomize_cube_positions_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
    """The position of the cubes in the world frame."""
    if not hasattr(env, "rigid_objects_in_focus"):
        return torch.full((env.num_envs, 9), fill_value=-1)

    cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
    cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
    cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]

    cube_1_pos_w = []
    cube_2_pos_w = []
    cube_3_pos_w = []
    for env_id in range(env.num_envs):
        cube_1_pos_w.append(cube_1.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][0], :3])
        cube_2_pos_w.append(cube_2.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][1], :3])
        cube_3_pos_w.append(cube_3.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][2], :3])
    cube_1_pos_w = torch.stack(cube_1_pos_w)
    cube_2_pos_w = torch.stack(cube_2_pos_w)
    cube_3_pos_w = torch.stack(cube_3_pos_w)

    return torch.cat((cube_1_pos_w, cube_2_pos_w, cube_3_pos_w), dim=1)

def cube_orientations_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
):
    """The orientation of the cubes in the world frame."""
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]

    return torch.cat((cube_1.data.root_quat_w, cube_2.data.root_quat_w, cube_3.data.root_quat_w), dim=1)

def instance_randomize_cube_orientations_in_world_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
    """The orientation of the cubes in the world frame."""
    if not hasattr(env, "rigid_objects_in_focus"):
        return torch.full((env.num_envs, 9), fill_value=-1)

    cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
    cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
    cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]

    cube_1_quat_w = []
    cube_2_quat_w = []
    cube_3_quat_w = []
    for env_id in range(env.num_envs):
        cube_1_quat_w.append(cube_1.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][0], :4])
        cube_2_quat_w.append(cube_2.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][1], :4])
        cube_3_quat_w.append(cube_3.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][2], :4])
    cube_1_quat_w = torch.stack(cube_1_quat_w)
    cube_2_quat_w = torch.stack(cube_2_quat_w)
    cube_3_quat_w = torch.stack(cube_3_quat_w)

    return torch.cat((cube_1_quat_w, cube_2_quat_w, cube_3_quat_w), dim=1)

def object_obs(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
):
    """
    Object observations (in world frame):
        cube_1 pos,
        cube_1 quat,
        cube_2 pos,
        cube_2 quat,
        cube_3 pos,
        cube_3 quat,
        gripper to cube_1,
        gripper to cube_2,
        gripper to cube_3,
        cube_1 to cube_2,
        cube_2 to cube_3,
        cube_1 to cube_3,
    """
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]

    cube_1_pos_w = cube_1.data.root_pos_w
    cube_1_quat_w = cube_1.data.root_quat_w

    cube_2_pos_w = cube_2.data.root_pos_w
    cube_2_quat_w = cube_2.data.root_quat_w

    cube_3_pos_w = cube_3.data.root_pos_w
    cube_3_quat_w = cube_3.data.root_quat_w

    ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    gripper_to_cube_1 = cube_1_pos_w - ee_pos_w
    gripper_to_cube_2 = cube_2_pos_w - ee_pos_w
    gripper_to_cube_3 = cube_3_pos_w - ee_pos_w

    cube_1_to_2 = cube_1_pos_w - cube_2_pos_w
    cube_2_to_3 = cube_2_pos_w - cube_3_pos_w
    cube_1_to_3 = cube_1_pos_w - cube_3_pos_w

    return torch.cat(
        (
            cube_1_pos_w - env.scene.env_origins,
            cube_1_quat_w,
            cube_2_pos_w - env.scene.env_origins,
            cube_2_quat_w,
            cube_3_pos_w - env.scene.env_origins,
            cube_3_quat_w,
            gripper_to_cube_1,
            gripper_to_cube_2,
            gripper_to_cube_3,
            cube_1_to_2,
            cube_2_to_3,
            cube_1_to_3,
        ),
        dim=1,
    )

def instance_randomize_object_obs(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
):
    """
    Object observations (in world frame):
        cube_1 pos,
        cube_1 quat,
        cube_2 pos,
        cube_2 quat,
        cube_3 pos,
        cube_3 quat,
        gripper to cube_1,
        gripper to cube_2,
        gripper to cube_3,
        cube_1 to cube_2,
        cube_2 to cube_3,
        cube_1 to cube_3,
    """
    if not hasattr(env, "rigid_objects_in_focus"):
        return torch.full((env.num_envs, 9), fill_value=-1)

    cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
    cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
    cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]

    cube_1_pos_w = []
    cube_2_pos_w = []
    cube_3_pos_w = []
    cube_1_quat_w = []
    cube_2_quat_w = []
    cube_3_quat_w = []
    for env_id in range(env.num_envs):
        cube_1_pos_w.append(cube_1.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][0], :3])
        cube_2_pos_w.append(cube_2.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][1], :3])
        cube_3_pos_w.append(cube_3.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][2], :3])
        cube_1_quat_w.append(cube_1.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][0], :4])
        cube_2_quat_w.append(cube_2.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][1], :4])
        cube_3_quat_w.append(cube_3.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][2], :4])
    cube_1_pos_w = torch.stack(cube_1_pos_w)
    cube_2_pos_w = torch.stack(cube_2_pos_w)
    cube_3_pos_w = torch.stack(cube_3_pos_w)
    cube_1_quat_w = torch.stack(cube_1_quat_w)
    cube_2_quat_w = torch.stack(cube_2_quat_w)
    cube_3_quat_w = torch.stack(cube_3_quat_w)

    ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    gripper_to_cube_1 = cube_1_pos_w - ee_pos_w
    gripper_to_cube_2 = cube_2_pos_w - ee_pos_w
    gripper_to_cube_3 = cube_3_pos_w - ee_pos_w

    cube_1_to_2 = cube_1_pos_w - cube_2_pos_w
    cube_2_to_3 = cube_2_pos_w - cube_3_pos_w
    cube_1_to_3 = cube_1_pos_w - cube_3_pos_w

    return torch.cat(
        (
            cube_1_pos_w - env.scene.env_origins,
            cube_1_quat_w,
            cube_2_pos_w - env.scene.env_origins,
            cube_2_quat_w,
            cube_3_pos_w - env.scene.env_origins,
            cube_3_quat_w,
            gripper_to_cube_1,
            gripper_to_cube_2,
            gripper_to_cube_3,
            cube_1_to_2,
            cube_2_to_3,
            cube_1_to_3,
        ),
        dim=1,
    )

def ee_frame_pos(env: ManagerBasedRLEnv, ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")) -> torch.Tensor:
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    ee_frame_pos = ee_frame.data.target_pos_w[:, 0, :] - env.scene.env_origins[:, 0:3]

    return ee_frame_pos

def ee_frame_quat(env: ManagerBasedRLEnv, ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")) -> torch.Tensor:
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    ee_frame_quat = ee_frame.data.target_quat_w[:, 0, :]

    return ee_frame_quat

def gripper_pos(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
) -> torch.Tensor:
    """
    Obtain the versatile gripper position of both Gripper and Suction Cup.
    """
    robot: Articulation = env.scene[robot_cfg.name]

    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        # Handle multiple surface grippers by concatenating their states
        gripper_states = []
        for gripper_name, surface_gripper in env.scene.surface_grippers.items():
            gripper_states.append(surface_gripper.state.view(-1, 1))

        if len(gripper_states) == 1:
            return gripper_states[0]
        else:
            return torch.cat(gripper_states, dim=1)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Observation gripper_pos only support parallel gripper for now"
            finger_joint_1 = robot.data.joint_pos[:, gripper_joint_ids[0]].clone().unsqueeze(1)
            finger_joint_2 = -1 * robot.data.joint_pos[:, gripper_joint_ids[1]].clone().unsqueeze(1)
            return torch.cat((finger_joint_1, finger_joint_2), dim=1)
        else:
            raise NotImplementedError("[Error] Cannot find gripper_joint_names in the environment config")

def object_grasped(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg,
    ee_frame_cfg: SceneEntityCfg,
    object_cfg: SceneEntityCfg,
    diff_threshold: float = 0.06,
) -> torch.Tensor:
    """Check if an object is grasped by the specified robot."""

    robot: Articulation = env.scene[robot_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    object: RigidObject = env.scene[object_cfg.name]

    object_pos = object.data.root_pos_w
    end_effector_pos = ee_frame.data.target_pos_w[:, 0, :]
    pose_diff = torch.linalg.vector_norm(object_pos - end_effector_pos, dim=1)

    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        surface_gripper = env.scene.surface_grippers["surface_gripper"]
        suction_cup_status = surface_gripper.state.view(-1, 1)  # 1: closed, 0: closing, -1: open
        suction_cup_is_closed = (suction_cup_status == 1).to(torch.float32)
        grasped = torch.logical_and(suction_cup_is_closed, pose_diff < diff_threshold)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Observations only support parallel gripper for now"

            # Support per-joint open values (list/tuple) or single scalar (applied to all joints)
            if isinstance(env.cfg.gripper_open_val, (list, tuple)):
                open_val_0 = env.cfg.gripper_open_val[0]
                open_val_1 = env.cfg.gripper_open_val[1]
            else:
                open_val_0 = env.cfg.gripper_open_val
                open_val_1 = env.cfg.gripper_open_val

            grasped = torch.logical_and(
                pose_diff < diff_threshold,
                torch.abs(
                    robot.data.joint_pos[:, gripper_joint_ids[0]]
                    - torch.tensor(open_val_0, dtype=torch.float32).to(env.device)
                )
                > env.cfg.gripper_threshold,
            )
            grasped = torch.logical_and(
                grasped,
                torch.abs(
                    robot.data.joint_pos[:, gripper_joint_ids[1]]
                    - torch.tensor(open_val_1, dtype=torch.float32).to(env.device)
                )
                > env.cfg.gripper_threshold,
            )

    return grasped

def object_stacked(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg,
    upper_object_cfg: SceneEntityCfg,
    lower_object_cfg: SceneEntityCfg,
    xy_threshold: float = 0.05,
    height_threshold: float = 0.005,
    height_diff: float = 0.0468,
) -> torch.Tensor:
    """Check if an object is stacked by the specified robot."""

    robot: Articulation = env.scene[robot_cfg.name]
    upper_object: RigidObject = env.scene[upper_object_cfg.name]
    lower_object: RigidObject = env.scene[lower_object_cfg.name]

    pos_diff = upper_object.data.root_pos_w - lower_object.data.root_pos_w
    height_dist = torch.linalg.vector_norm(pos_diff[:, 2:], dim=1)
    xy_dist = torch.linalg.vector_norm(pos_diff[:, :2], dim=1)

    stacked = torch.logical_and(xy_dist < xy_threshold, (height_dist - height_diff) < height_threshold)

    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        surface_gripper = env.scene.surface_grippers["surface_gripper"]
        suction_cup_status = surface_gripper.state.view(-1, 1)  # 1: closed, 0: closing, -1: open
        suction_cup_is_open = (suction_cup_status == -1).to(torch.float32)
        stacked = torch.logical_and(suction_cup_is_open, stacked)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Observations only support parallel gripper for now"
            
            # Support per-joint open values (list/tuple) or single scalar (applied to all joints)
            if isinstance(env.cfg.gripper_open_val, (list, tuple)):
                open_val_0 = env.cfg.gripper_open_val[0]
                open_val_1 = env.cfg.gripper_open_val[1]
            else:
                open_val_0 = env.cfg.gripper_open_val
                open_val_1 = env.cfg.gripper_open_val

            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[0]],
                    torch.tensor(open_val_0, dtype=torch.float32).to(env.device),
                    atol=1e-4,
                    rtol=1e-4,
                ),
                stacked,
            )
            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[1]],
                    torch.tensor(open_val_1, dtype=torch.float32).to(env.device),
                    atol=1e-4,
                    rtol=1e-4,
                ),
                stacked,
            )
        else:
            raise ValueError("No gripper_joint_names found in environment config")

    return stacked

def cube_poses_in_base_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
    return_key: Literal["pos", "quat", None] = None,
) -> torch.Tensor:
    """The position and orientation of the cubes in the robot base frame."""

    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]

    pos_cube_1_world = cube_1.data.root_pos_w
    pos_cube_2_world = cube_2.data.root_pos_w
    pos_cube_3_world = cube_3.data.root_pos_w

    quat_cube_1_world = cube_1.data.root_quat_w
    quat_cube_2_world = cube_2.data.root_quat_w
    quat_cube_3_world = cube_3.data.root_quat_w

    robot: Articulation = env.scene[robot_cfg.name]
    root_pos_w = robot.data.root_pos_w
    root_quat_w = robot.data.root_quat_w

    pos_cube_1_base, quat_cube_1_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, pos_cube_1_world, quat_cube_1_world
    )
    pos_cube_2_base, quat_cube_2_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, pos_cube_2_world, quat_cube_2_world
    )
    pos_cube_3_base, quat_cube_3_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, pos_cube_3_world, quat_cube_3_world
    )

    pos_cubes_base = torch.cat((pos_cube_1_base, pos_cube_2_base, pos_cube_3_base), dim=1)
    quat_cubes_base = torch.cat((quat_cube_1_base, quat_cube_2_base, quat_cube_3_base), dim=1)

    if return_key == "pos":
        return pos_cubes_base
    elif return_key == "quat":
        return quat_cubes_base
    else:
        return torch.cat((pos_cubes_base, quat_cubes_base), dim=1)

def object_abs_obs_in_base_frame(
    env: ManagerBasedRLEnv,
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
):
    """
    Object Abs observations (in base frame): remove the relative observations,
    and add abs gripper pos and quat in robot base frame
        cube_1 pos,
        cube_1 quat,
        cube_2 pos,
        cube_2 quat,
        cube_3 pos,
        cube_3 quat,
        gripper pos,
        gripper quat,
    """
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]
    cube_3: RigidObject = env.scene[cube_3_cfg.name]
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    robot: Articulation = env.scene[robot_cfg.name]

    root_pos_w = robot.data.root_pos_w
    root_quat_w = robot.data.root_quat_w

    cube_1_pos_w = cube_1.data.root_pos_w
    cube_1_quat_w = cube_1.data.root_quat_w

    cube_2_pos_w = cube_2.data.root_pos_w
    cube_2_quat_w = cube_2.data.root_quat_w

    cube_3_pos_w = cube_3.data.root_pos_w
    cube_3_quat_w = cube_3.data.root_quat_w

    pos_cube_1_base, quat_cube_1_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, cube_1_pos_w, cube_1_quat_w
    )
    pos_cube_2_base, quat_cube_2_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, cube_2_pos_w, cube_2_quat_w
    )
    pos_cube_3_base, quat_cube_3_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, cube_3_pos_w, cube_3_quat_w
    )

    ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    ee_quat_w = ee_frame.data.target_quat_w[:, 0, :]
    ee_pos_base, ee_quat_base = math_utils.subtract_frame_transforms(root_pos_w, root_quat_w, ee_pos_w, ee_quat_w)

    return torch.cat(
        (
            pos_cube_1_base,
            quat_cube_1_base,
            pos_cube_2_base,
            quat_cube_2_base,
            pos_cube_3_base,
            quat_cube_3_base,
            ee_pos_base,
            ee_quat_base,
        ),
        dim=1,
    )

def ee_frame_pose_in_base_frame(
    env: ManagerBasedRLEnv,
    ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
    return_key: Literal["pos", "quat", None] = None,
) -> torch.Tensor:
    """
    The end effector pose in the robot base frame.
    """
    ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
    ee_frame_pos_w = ee_frame.data.target_pos_w[:, 0, :]
    ee_frame_quat_w = ee_frame.data.target_quat_w[:, 0, :]

    robot: Articulation = env.scene[robot_cfg.name]
    root_pos_w = robot.data.root_pos_w
    root_quat_w = robot.data.root_quat_w
    ee_pos_in_base, ee_quat_in_base = math_utils.subtract_frame_transforms(
        root_pos_w, root_quat_w, ee_frame_pos_w, ee_frame_quat_w
    )

    if return_key == "pos":
        return ee_pos_in_base
    elif return_key == "quat":
        return ee_quat_in_base
    else:
        return torch.cat((ee_pos_in_base, ee_quat_in_base), dim=1)

Similarly, the cubes_stacked termination condition is updated to ensure the success criteria check both gripper joints before concluding the task.

Below is the complete code, with relevant modifications in Lines 81-109.
stack/mdp/terminations.py

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

"""Common functions that can be used to activate certain terminations for the lift task.

The functions can be passed to the :class:`isaaclab.managers.TerminationTermCfg` object to enable
the termination introduced by the function.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import torch
from isaaclab.assets import Articulation, RigidObject
from isaaclab.managers import SceneEntityCfg

if TYPE_CHECKING:
    from isaaclab.envs import ManagerBasedRLEnv

def cubes_stacked(
    env: ManagerBasedRLEnv,
    robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
    cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
    cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
    cube_3_cfg: SceneEntityCfg | None = SceneEntityCfg("cube_3"),
    xy_threshold: float = 0.04,
    height_threshold: float = 0.005,
    height_diff: float = 0.0468,
    atol: float = 0.0001,
    rtol: float = 0.0001,
) -> torch.Tensor:
    robot: Articulation = env.scene[robot_cfg.name]
    cube_1: RigidObject = env.scene[cube_1_cfg.name]
    cube_2: RigidObject = env.scene[cube_2_cfg.name]

    pos_diff_c12 = cube_1.data.root_pos_w - cube_2.data.root_pos_w

    # Compute cube position difference in x-y plane
    xy_dist_c12 = torch.norm(pos_diff_c12[:, :2], dim=1)

    # Compute cube height difference
    h_dist_c12 = torch.norm(pos_diff_c12[:, 2:], dim=1)

    # Check cube positions
    stacked = xy_dist_c12 < xy_threshold
    stacked = torch.logical_and(h_dist_c12 - height_diff < height_threshold, stacked)
    stacked = torch.logical_and(pos_diff_c12[:, 2] < 0.0, stacked)

    if cube_3_cfg is not None:
        cube_3: RigidObject = env.scene[cube_3_cfg.name]
        pos_diff_c23 = cube_2.data.root_pos_w - cube_3.data.root_pos_w

        # Compute cube position difference in x-y plane
        xy_dist_c23 = torch.norm(pos_diff_c23[:, :2], dim=1)

        # Compute cube height difference
        h_dist_c23 = torch.norm(pos_diff_c23[:, 2:], dim=1)

        # Check cube positions
        stacked = torch.logical_and(xy_dist_c23 < xy_threshold, stacked)
        stacked = torch.logical_and(h_dist_c23 - height_diff < height_threshold, stacked)
        stacked = torch.logical_and(pos_diff_c23[:, 2] < 0.0, stacked)

    # Check gripper positions
    if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
        surface_gripper = env.scene.surface_grippers["surface_gripper"]
        suction_cup_status = surface_gripper.state.view(-1)  # 1: closed, 0: closing, -1: open
        suction_cup_is_open = (suction_cup_status == -1).to(torch.float32)
        stacked = torch.logical_and(suction_cup_is_open, stacked)

    else:
        if hasattr(env.cfg, "gripper_joint_names"):
            gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
            assert len(gripper_joint_ids) == 2, "Terminations only support parallel gripper for now"

            # Support per-joint open values (list/tuple) or single scalar (applied to all joints)
            if isinstance(env.cfg.gripper_open_val, (list, tuple)):
                open_val_0 = env.cfg.gripper_open_val[0]
                open_val_1 = env.cfg.gripper_open_val[1]
            else:
                open_val_0 = env.cfg.gripper_open_val
                open_val_1 = env.cfg.gripper_open_val

            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[0]],
                    torch.tensor(open_val_0, dtype=torch.float32).to(env.device),
                    atol=atol,
                    rtol=rtol,
                ),
                stacked,
            )
            stacked = torch.logical_and(
                torch.isclose(
                    robot.data.joint_pos[:, gripper_joint_ids[1]],
                    torch.tensor(open_val_1, dtype=torch.float32).to(env.device),
                    atol=atol,
                    rtol=rtol,
                ),
                stacked,
            )
        else:
            raise ValueError("No gripper_joint_names found in environment config")

    return stacked

3.2.4 Piper Joint Position Configuration (`piper_stack_joint_pos_env_cfg.py`)

This file maps the Piper asset to the environment.

Major changes include:

Default Pose: Adjusted to [0.0, 1.57, -1.57, 0.0, 1.22, 0.0, 0.0, 0.0] (6 ARM joints + 2 Gripper joints).
Gripper Command Mapping:
- Open: gripper_joint1: 0.05, gripper_joint2: -0.05
- Close: 0.0 for both.
End-Effector (EE) Offset: Set to [0.0, 0.0, 0.14] to account for the Piper gripper’s length from the base link to the Tool Center Point (TCP).

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from isaaclab.assets import RigidObjectCfg
from isaaclab.managers import EventTermCfg as EventTerm
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformerCfg
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg
from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR

from agx_teleop.tasks.manager_based.manipulation.stack import mdp
from agx_teleop.tasks.manager_based.manipulation.stack.mdp import piper_stack_events
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import StackEnvCfg

##
# Pre-defined configs
##
from isaaclab.markers.config import FRAME_MARKER_CFG  # isort: skip
from agx_teleop.assets.piper import PIPER_GRIPPER_CFG  # isort: skip

@configclass
class EventCfg:
    """Configuration for events."""

    init_piper_arm_pose = EventTerm(
        func=piper_stack_events.set_default_joint_pose,
        mode="reset",
        params={
            "default_pose": [0.0, 1.57, -1.57, 0.0, 1.22, 0.0, 0.0, 0.0],   # 6机械臂+2夹爪，夹爪关闭
        },
    )

    randomize_piper_joint_state = EventTerm(
        func=piper_stack_events.randomize_joint_by_gaussian_offset,
        mode="reset",
        params={
            "mean": 0.0,
            "std": 0.02,
            "asset_cfg": SceneEntityCfg("robot"),
        },
    )

    randomize_cube_positions = EventTerm(
        func=piper_stack_events.randomize_object_pose,
        mode="reset",
        params={
            "pose_range": {"x": (0.3, 0.5), "y": (-0.10, 0.10), "z": (0.0203, 0.0203), "yaw": (-1.0, 1, 0)},
            "min_separation": 0.12,
            "asset_cfgs": [SceneEntityCfg("cube_1"), SceneEntityCfg("cube_2"), SceneEntityCfg("cube_3")],
        },
    )

@configclass
class PiperCubeStackEnvCfg(StackEnvCfg):
    """Configuration for the Piper Cube Stack Environment."""

    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        # Set events
        self.events = EventCfg()

        # Set Piper as robot
        self.scene.robot = PIPER_GRIPPER_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
        self.scene.robot.spawn.semantic_tags = [("class", "robot")]

        # Add semantics to table
        self.scene.table.spawn.semantic_tags = [("class", "table")]

        # Add semantics to ground
        self.scene.plane.semantic_tags = [("class", "ground")]

        # Set actions for the specific robot type (piper)
        self.actions.arm_action = mdp.JointPositionActionCfg(
            asset_name="robot", joint_names=["joint.*"], scale=0.5, use_default_offset=True
        )
        self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
            asset_name="robot",
            joint_names=["gripper_joint.*"],
            open_command_expr={
                "gripper_joint1": 0.05,
                "gripper_joint2": -0.05,    
            },
            close_command_expr={
                "gripper_joint1": 0.0,
                "gripper_joint2": 0.0,
            },
        )
        # utilities for gripper status check
        self.gripper_joint_names = ["gripper_joint.*"]
        self.gripper_open_val = [0.05, -0.05]
        self.gripper_threshold = 0.005

        # Rigid body properties of each cube
        cube_properties = RigidBodyPropertiesCfg(
            solver_position_iteration_count=16,
            solver_velocity_iteration_count=1,
            max_angular_velocity=1000.0,
            max_linear_velocity=1000.0,
            max_depenetration_velocity=5.0,
            disable_gravity=False,
        )

        # Set each stacking cube deterministically
        self.scene.cube_1 = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Cube_1",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0.0, 0.0203], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/blue_block.usd",
                scale=(1.0, 1.0, 1.0),
                rigid_props=cube_properties,
                semantic_tags=[("class", "cube_1")],
            ),
        )
        self.scene.cube_2 = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Cube_2",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.55, 0.05, 0.0203], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/red_block.usd",
                scale=(1.0, 1.0, 1.0),
                rigid_props=cube_properties,
                semantic_tags=[("class", "cube_2")],
            ),
        )
        self.scene.cube_3 = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Cube_3",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.60, -0.1, 0.0203], rot=[1, 0, 0, 0]),
            spawn=UsdFileCfg(
                usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/green_block.usd",
                scale=(1.0, 1.0, 1.0),
                rigid_props=cube_properties,
                semantic_tags=[("class", "cube_3")],
            ),
        )

        # Listens to the required transforms
        marker_cfg = FRAME_MARKER_CFG.copy()
        marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
        marker_cfg.prim_path = "/Visuals/FrameTransformer"
        self.scene.ee_frame = FrameTransformerCfg(
            prim_path="{ENV_REGEX_NS}/Robot/base_link",
            debug_vis=False,
            visualizer_cfg=marker_cfg,
            target_frames=[
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_base",
                    name="end_effector",
                    offset=OffsetCfg(
                        pos=[0.0, 0.0, 0.14],   # gripper_base 到 TCP 的距离
                    ),
                ),
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_link1",
                    name="tool_rightfinger",
                    offset=OffsetCfg(
                        pos=(0.0, 0.0, 0.0),  
                    ),
                ),
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/gripper_link2",
                    name="tool_leftfinger",
                    offset=OffsetCfg(
                        pos=(0.0, 0.0, 0.0), 
                    ),
                ),
            ],
        )

3.2.5 Inverse Kinematics Configuration (`piper_stack_ik_rel_env_cfg.py`)

This file configures the Differential IK controller. We use a high-gain PD controller (PIPER_GRIPPER_HIGH_PD_CFG) to ensure precise tracking during teleoperation or inference.

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from isaaclab.controllers.differential_ik_cfg import DifferentialIKControllerCfg
from isaaclab.devices.device_base import DeviceBase, DevicesCfg
from isaaclab.devices.keyboard import Se3KeyboardCfg
from isaaclab.devices.openxr.openxr_device import OpenXRDeviceCfg
from isaaclab.devices.openxr.retargeters.manipulator.gripper_retargeter import GripperRetargeterCfg
from isaaclab.devices.openxr.retargeters.manipulator.se3_rel_retargeter import Se3RelRetargeterCfg
from isaaclab.envs.mdp.actions.actions_cfg import DifferentialInverseKinematicsActionCfg
from isaaclab.managers import SceneEntityCfg
from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.utils import configclass

from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import mdp

from . import piper_stack_joint_pos_env_cfg

##
# Pre-defined configs
##
from agx_teleop.assets.piper import PIPER_GRIPPER_HIGH_PD_CFG  # isort: skip

@configclass
class PiperCubeStackEnvCfg(piper_stack_joint_pos_env_cfg.PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        # Set Piper as robot
        # We switch here to a stiffer PD controller for IK tracking to be better.
        self.scene.robot = PIPER_GRIPPER_HIGH_PD_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

        # Set actions for the specific robot type (piper)
        self.actions.arm_action = DifferentialInverseKinematicsActionCfg(
            asset_name="robot",
            joint_names=["joint.*"],
            body_name="gripper_base",
            controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),
            scale=0.5,
            body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.14]),
        )

        self.teleop_devices = DevicesCfg(
            devices={
                "handtracking": OpenXRDeviceCfg(
                    retargeters=[
                        Se3RelRetargeterCfg(
                            bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT,
                            zero_out_xy_rotation=True,
                            use_wrist_rotation=False,
                            use_wrist_position=True,
                            delta_pos_scale_factor=10.0,
                            delta_rot_scale_factor=10.0,
                            sim_device=self.sim.device,
                        ),
                        GripperRetargeterCfg(
                            bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT, sim_device=self.sim.device
                        ),
                    ],
                    sim_device=self.sim.device,
                    xr_cfg=self.xr,
                ),
                "keyboard": Se3KeyboardCfg(
                    pos_sensitivity=0.05,
                    rot_sensitivity=0.05,
                    sim_device=self.sim.device,
                ),
            }
        )

@configclass
class PiperCubeStackRedGreenEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={"cube_1_cfg": SceneEntityCfg("cube_2"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
        )

@configclass
class PiperCubeStackRedGreenBlueEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={
                "cube_1_cfg": SceneEntityCfg("cube_2"),
                "cube_2_cfg": SceneEntityCfg("cube_3"),
                "cube_3_cfg": SceneEntityCfg("cube_1"),
            },
        )

@configclass
class PiperCubeStackBlueGreenEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={"cube_1_cfg": SceneEntityCfg("cube_1"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
        )

@configclass
class PiperCubeStackBlueGreenRedEnvCfg(PiperCubeStackEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()

        self.terminations.success = DoneTerm(
            func=mdp.cubes_stacked,
            params={
                "cube_1_cfg": SceneEntityCfg("cube_1"),
                "cube_2_cfg": SceneEntityCfg("cube_3"),
                "cube_3_cfg": SceneEntityCfg("cube_2"),
            },
        )

3.2.6 Environment Registration (`stack/config/piper/init.py`)

Finally, register the task in Gymnasium so it can be launched via the IsaacLab runner:

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
import gymnasium as gym

# from isaaclab_tasks.manager_based.manipulation.stack.config.franka import agents
from agx_teleop.tasks.manager_based.manipulation.stack.config.piper import agents

##
# Register Gym environments.
##

gym.register(
    id="Isaac-Stack-Cube-Piper-IK-Rel-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    kwargs={
        "env_cfg_entry_point": f"{__name__}.piper_stack_ik_rel_env_cfg:PiperCubeStackEnvCfg",
        "robomimic_bc_cfg_entry_point": f"{agents.__name__}:robomimic/bc_rnn_low_dim.json",
    },
    disable_env_checker=True,
)

Step 4.Teleoperation & Data Collection

4.1 Teleoperation Configuration

Create the tool directory and script files within the agx_teleop plugin:

cd agx_teleop
mkdir -p scripts/tools
touch scripts/tools/record_demos.py \
      scripts/tools/repaly_demos.py

Copy the corresponding IsaacLab source code and modify the imports for tasks. Required sections are listed below:

from isaaclab.envs import DirectRLEnvCfg, ManagerBasedRLEnvCfg
from isaaclab.envs.mdp.recorders.recorders_cfg import ActionStateRecorderManagerCfg
from isaaclab.envs.ui import EmptyWindow
from isaaclab.managers import DatasetExportMode

# import isaaclab_tasks  # noqa: F401
import agx_teleop.tasks  # noqa: F401
from isaaclab_tasks.utils.parse_cfg import parse_env_cfg

scripts/tools/repaly_demos.py:

if args_cli.enable_pinocchio:
    import isaaclab_tasks.manager_based.locomanipulation.pick_place  # noqa: F401
    import isaaclab_tasks.manager_based.manipulation.pick_place  # noqa: F401

# import isaaclab_tasks  # noqa: F401
import agx_teleop.tasks  # noqa: F401
from isaaclab_tasks.utils.parse_cfg import parse_env_cfg

4.2 Teleoperation Configuration

# First, create the dataset directory
mkdir -p datasets

# Command for teleoperated data collection; keyboard teleoperation used here. Available devices: spacemouse, keyboard, handtracking
python scripts/tools/record_demos.py \
    --task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
    --device cpu --teleop_device keyboard \
    --dataset_file ./datasets/dataset.hdf5 \
    --num_demos 10

# Replay the collected dataset
python scripts/tools/replay_demos.py \
    --task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
    --device cpu --dataset_file ./datasets/dataset.hdf5

Keyboard Teleoperation Method:

Keyboard Controller for SE(3): Se3Keyboard
   Reset all commands: R
   Toggle gripper (open/close): K
   Move arm along x-axis: W/S
   Move arm along y-axis: A/D
   Move arm along z-axis: Q/E
   Rotate arm along x-axis: Z/X
   Rotate arm along y-axis: T/G
   Rotate arm along z-axis: C/V

piperteleop

Q & A

Q1: Why is the End-Effector (EE) offset set to [0.0, 0.0, 0.14] in the IK configuration?

A: This offset accounts for the physical length of the PiPER gripper. It shifts the Tool Center Point (TCP) from the gripper_base link to the actual center of the fingertips. Without this body_offset, the Inverse Kinematics (IK) controller would attempt to align the robot’s “wrist” with the target cube, causing the physical fingers to overshoot and collide with the objects.

Q2: How does the workflow ensure the quality and validity of the recorded .hdf5 datasets?
A: The workflow employs a two-layer validation system:

Automated Filtering: The record_demos.py script uses DatasetExportMode.EXPORT_SUCCEEDED_ONLY, meaning only episodes that trigger the success termination criteria (cubes stacked for a set number of steps) are saved.
Manual Verification: The replay_demos.py script allows developers to re-simulate recorded actions to confirm that the recorded trajectories consistently result in a successful task outcome before starting training.

Q3: What is the reasoning behind recommending --device cpu during the teleoperation recording phase?

A: Teleoperation is highly sensitive to input latency. When recording demonstrations, the GPU is often heavily taxed by real-time rendering and physics calculations. Running the simulation logic on the CPU can reduce “input lag” and jitter, providing the human operator with a smoother control experience, which leads to higher-quality, more natural trajectory data for imitation learning.

Q4:In piper_stack_ik_rel_env_cfg.py, why choose the relative pose mode with use_relative_mode=True?
A: The Relative Mode is more intuitive for human teleoperation: the input from the keyboard or joystick is mapped to “increments of the current end position” rather than “absolute coordinates”. During training, relative control can also reduce the difficulty for the model to learn spatial features, as it focuses on motion trends rather than fixed global coordinates, which helps improve the model’s generalization ability across different environmental positions.

Q5: Why is the --fix-base parameter required during the URDF to USD conversion process?
A: PiPER robotic arms are usually fixed on workbenches. In Isaac Sim/Isaac Lab, if the robot base is not fixed (Fix), the robotic arm will displace or tip over in the virtual space due to reaction forces when performing high-dynamic actions. The --fix-base parameter rigidly connects the base_link in the URDF to the global coordinate system (World Frame) of the simulation world, ensuring the stability of the robotic arm base.

Have Question?

If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/isaac-lab-teleoperation-and-data-collection-workflow-piper-robotic-arm-for-embodied-ai/55223

ROS Discourse General: Built a flight recorder for ROS 2 robots — looking for brutal feedback

Hey ROS community,

We’ve been building something called **BlackBox** — think of it like a flight data recorder for your ROS 2 robots.

The idea: every time your robot does something unexpected in the field, you shouldn’t have to dig through raw logs hoping you can reproduce it. BlackBox lets you record **episodes** (a bounded window of what your robot was doing), tag failures, monitor `/diagnostics`, and see fleet-wide patterns over time — without major changes to your existing ROS 2 setup.

We’re genuinely at the stage where we want to know if we’re solving the right problem.

**How you can connect:**

- **ROS 2 node** — the primary integration, works today

- **REST API** — for non-ROS systems or custom pipelines

- **RabbitMQ & MQTT** — for event-driven and IoT setups *(under active development)*

If your stack doesn’t speak ROS 2 natively, the API/MQTT path is coming — and knowing what people actually need here would help us prioritize.

**Try it without signing up first:**

- Live sandbox with real demo data: BlackBox Robotics — Failure Intelligence for Robotics | ROS 2 Data Platform

- Analyze view: BlackBox Robotics — Failure Intelligence for Robotics | ROS 2 Data Platform

**Or create a free account and connect your own robot:**

Sign up at https://bbrobotics.in — it takes 2 minutes, no credit card. You can hook up a ROS 2 node, upload an episode, and see how it looks on your actual data. That’s probably the most useful way to poke holes in it.

**What we want from you:**

- Is this a problem you actually have in your deployments?

- What’s obviously missing that would make this useful?

- What would make you never use something like this?

- If you’ve dealt with hard-to-reproduce field failures, how are you handling it today?

- Would MQTT/RabbitMQ support change anything for your setup?

We’d rather hear “this is pointless because X” now than after 6 more months of building. Roasts, edge cases, and skeptical takes all welcome.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/built-a-flight-recorder-for-ros-2-robots-looking-for-brutal-feedback/55210

ROS Discourse General: Show ROS: BAGEL – A zero-install, browser-native 3D visualizer and scrubber for MCAP, db3, and ROS1 bags

Hi everyone,

I wanted to share a project I’ve been working on called BAGEL (BAG ExpLoration). It’s an open-source, fully static web application designed to let you inspect, scrub, and visualize ROS 1 and ROS 2 bag files locally in your browser with zero installation, zero configuration, and no cloud uploads.

Live App: https://bagel-ros2.vercel.app

GitHub Repository: GitHub - Hussain004/BAGEL: ROS Bag Visualizer · GitHub

Demo Playlist: https://youtube.com/playlist?list=PLHZhJb0NUk4Jv26hmARxgLa_dVw6Ab5j8&si=zSZQnYMwOmESiOgq

Why build another visualizer?

While tools like RViz2 are great for live debugging, standing up a full ROS environment just to quickly sanity-check a data log on a laptop or to share results with a non-ROS collaborator can be friction-heavy.

BAGEL is built entirely on a client-side architecture. Your data never leaves your machine. It utilizes web workers and streaming chunk decoders to handle heavy robotics telemetry data directly in the browser thread.

Key Features

Multi-Format Ingestion: Seamless drag-and-drop or file-selection for ROS2 `.mcap` (Humble/Jazzy compatibility), ROS2 `.db3` (SQLite), and legacy ROS1 `.bag` files.
TF-Aware 3D Panel (Three.js): Decodes and visualizes `sensor_msgs/PointCloud2`, `sensor_msgs/LaserScan`, `geometry_msgs/PoseStamped`, `nav_msgs/OccupancyGrid` SLAM maps, and `visualization_msgs/MarkerArray`. It natively handles action semantics (`ADD`, `MODIFY`, `DELETEALL`) and lifetime fade-outs.
Geographical Grounding: The trajectory panel parses `sensor_msgs/NavSatFix` and projects the coordinates using an opt-in OpenStreetMap tile underlay with an in-memory LRU cache.
Remote URL Streaming via HTTP Range: You can paste a public URL (S3 bucket, GitHub release asset, etc.). For MCAP and ROS1 bags, the tool only streams the precise chunks you scrub through, allowing a multi-gigabyte log to open in under a second.
Custom Schema On-the-Fly Injection: Since `.db3` files do not embed message schemas, anything outside the standard packages usually fails to decode. BAGEL adds a runtime paste flow backed by `@foxglove/rosmsg` validation and `l@foxglovecalSto@foxgloveage` persistence, allowing you to paste your custom `.msg` definitions once and decode proprietary planner or vendor packages instantly.
Blazing-Fast Time-Series: Uses `uPlot` to instantly graph any numerical primitive from any message payload with zero layout lag.

Technical Stack

The frontend is built using React 19, TypeScript, and Vite. Heavy parsing workloads are offloaded to worke@mca@mcaps:

@mcap/c@foxglov@foxglovere & @foxglove/rosbag@mcap for low-level@mc@foxglove@foxgloveparsing.
@foxglove sql.js (WebAssembly) to@foxglovequery unindexed `.db3` tables in memory.
seek-bzip & lz4js running in web workers to handle compressed ROS1 chunks natively without blocking UI rendering loops.

Feedback Welcome!

This is a fully open-source effort aimed at making data sharing and quick field log triage as low-friction as possible.

I would love to hear feedback from the community regarding performance benchmarks, missing primitive types you rely on, or general UX enhancements. Feel free to open issues or submit PRs on the repository!

6 posts - 4 participants

Read full topic

https://discourse.openrobotics.org/t/show-ros-bagel-a-zero-install-browser-native-3d-visualizer-and-scrubber-for-mcap-db3-and-ros1-bags/55178

ROS Discourse General: [Release] LinkForge v1.4.0: A Programmable Robot Description Engine (URDF/SRDF)

Hi ROS Community,

We’re excited to announce LinkForge v1.4.0, a major architectural update that transforms LinkForge into a programmable Intermediate Representation (IR) for robot descriptions.

If you’ve ever debugged a crash in Gazebo caused by a bad inertia tensor, or spent hours manually resolving namespace collisions in URDFs and MoveIt 2 SRDFs when attaching a manipulator to a mobile base, this is built for you.

Key Features for ROS Developers:

Lossless URDF/XACRO Parsing: Preserves unknown metadata/tags, allowing seamless round-trips.
Automated SRDF & MoveIt 2: Programmatically compose subsystems and LinkForge will automatically merge planning groups and self-collision matrices into a unified SRDF.
ROS 2 Control & Gazebo: Auto-generates ros2_control hardware interface tags based on high-level Python definitions.
Physical Linter: Calculates rigorous inertia tensors from geometry and catches non-physical masses before you launch your simulation.

We’d love the community’s feedback on how well it integrates with your existing ROS 1 / ROS 2 workflows!

GitHub: LinkForge
Pypi: pip install linkforge-core

3 posts - 2 participants

Read full topic

https://discourse.openrobotics.org/t/release-linkforge-v1-4-0-a-programmable-robot-description-engine-urdf-srdf/55171

ROS Discourse General: Introducing Vulcanexus HRI: Human-Robot Interaction for ROS 2

Introducing Vulcanexus HRI: Human-Robot Interaction for ROS 2

As robots become increasingly present in workplaces, public environments and everyday services, the ability to perceive, communicate and interact naturally with people is becoming more important than ever.

We are happy to introduce Vulcanexus HRI, a new framework within the Vulcanexus ecosystem designed to help developers integrate Human-Robot Interaction capabilities into ROS 2 applications.

Vulcanexus HRI provides a modular set of tools for perception, communication and interaction, including:

Face detection
Human pose recognition
Emotion recognition
Speech-to-text
Text-to-speech
Visual interaction components
Shared HRI message definitions

The framework is designed to work seamlessly within existing Vulcanexus and ROS 2 systems, allowing developers to enrich robotic applications with human-related information without redesigning the rest of their architecture.

The different modules can be used independently or combined depending on the application requirements, making Vulcanexus HRI suitable both for experimentation and for more advanced interactive robotic systems.

Documentation and Getting Started guide: 8. Vulcanexus HRI Overview -

We have also prepared a short video showing VulcanBot interacting using Vulcanexus HRI capabilities.

We would love to hear feedback from the ROS community and discuss possible use cases, integrations and future improvements.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/introducing-vulcanexus-hri-human-robot-interaction-for-ros-2/55130

ROS Discourse General: Embodied Intelligence Upgraded: Self-Improving Robot Policies via RISE World Model Reasoning

Embodied Intelligence Upgraded: Self-Improving Robot Policies via RISE World Model Reasoning

Vision-Language-Action (VLA) models struggle with contact-rich tasks such as dynamic sorting and flexible packing, where tiny execution deviations often result in failure. Traditional real-world reinforcement learning (RL) faces barriers to scaling: high hardware costs, manual environment resets, and slow serial interactions.

The RISE framework (RSS 2026), developed by CUHK MMLab, HKU OpenDriveLab, Tsinghua University and so on, solves this with an “imagination-based self-evolution” paradigm. By training in simulated “imaginary space” instead of the physical world, it eliminates costly trial-and-error and achieves a 95% success rate on complex manipulation tasks using the AgileX Piper robotic arm.

References & Links

Project Page: https://opendrivelab.com/RISE/

Paper Link: https://arxiv.org/pdf/2602.11075

Hardware Used: AgileX PiPER 6-DoF Robotic Arm

Project HomePage：RISE: Self-Improving Robot Policy with Compositional World Model

Open Source: GitHub - OpenDriveLab/RISE: [RSS 2026] Code for RISE: Self-Improving Robot Policy with Compositional World Model · GitHub

RISE(1) (2) (1) (1) (1)

1. Real-World Robot Learning Still Struggles to Scale

Modern Vision-Language-Action (VLA) models can perform basic robotic manipulation through imitation learning (IL), but they still struggle with contact-rich tasks involving dynamic objects, deformable materials, and bimanual coordination. Even small execution errors can lead to task failure.

While reinforcement learning (RL) offers a path toward autonomous robot learning, real-world training remains limited by:

high hardware costs
low training efficiency
manual environment resetting
safety and reliability risks

Researchers have long relied on simulation and world models to improve scalability, but challenges such as the sim-to-real gap, unstable action generation, and slow robot planning continue to limit real-world deployment in embodied AI and autonomous robotics.

2.Building Self-Improving Robot Policies with Compositional World Models

GIF1

The core idea behind RISE is simple: instead of relying entirely on expensive real-world robot training, robots improve themselves inside an imagined environment powered by compositional world models.

RISE separates robot learning into two key components:

1.Controllable Dynamics Model

A fast and controllable world model predicts future robot interactions and manipulation outcomes.

Built on the Genie Envisioner video diffusion model
Generates multi-view future robot trajectories within seconds
Uses lightweight action encoders to ensure actions remain physically consistent and controllable
Pretrained on large-scale robot datasets such as Agibot World and Galaxea for realistic robot manipulation prediction

GIF2

2.Progress Value Model

A value prediction model continuously evaluates robot behavior during manipulation tasks.

Built on the π0.5 Vision-Language-Action (VLA) framework
Combines progress regression and temporal-difference (TD) learning
Detects subtle manipulation failures such as object slipping or unstable contact
Outputs real-time advantage scores for autonomous robot policy optimization

Together, these components enable scalable self-improving robot learning for embodied AI, contact-rich manipulation, and sim-to-real robotic systems.

3.Closed-Loop Self-Improving Robot Learning in Imagined Environments

By using the AgileX PiPER Robot Arm, RISE enables autonomous robot policy improvement entirely within imagined environments, reducing the need for large-scale real-world trial-and-error training.

The self-improving robot learning pipeline consists of three stages:

Policy Warm-Up

The robot policy is initialized using a small amount of offline robot data, including demonstrations and successful or failed manipulation rollouts, allowing the system to learn basic robot manipulation skills.

Imagined Rollout

The robot policy generates actions, while the compositional world model predicts future robot interactions and manipulation trajectories. At the same time, the value model evaluates action quality and estimates real-time advantage values.

Policy Optimization

High-advantage robot actions are reinforced, while low-quality behaviors are gradually filtered out through iterative policy optimization, enabling continuous self-improving robot learning.

GIF3

RISE performs the entire optimization process in virtual environments without requiring repeated real-world robot interaction. During inference, the world model is no longer involved, meaning the system introduces no additional runtime computation cost for real-world robot deployment.

4.Three Challenging Robot Manipulation Tasks, Significant Performance Gains

RISE was evaluated on three high-difficulty real-world robot manipulation tasks, including dynamic brick sorting, flexible bag packing, and precision box assembly, significantly outperforming existing imitation learning (IL) and reinforcement learning (RL) baselines.

Benchmark Results

Benchmark evaluation on the AgileX PiPER Robot Arm indicates a substantial improvement in manipulation success rate, as shown in the results below.

Dynamic Brick Sorting

Success rate improved from 50% → 85%

GIF4

Flexible Bag Packing

Success rate improved from 40% → 85%

GIF5

Precision Box Assembly

Success rate improved from 60% → 95%

GIF6

Compared with online reinforcement learning methods such as PPO and DSRL, RISE demonstrated significantly more stable robot policy optimization without training collapse.

Compared with offline RL approaches such as RECAP, RISE continuously expands robot training distributions through imagined world-model rollouts, greatly improving generalization and reducing overfitting in contact-rich manipulation tasks.

5.Key Design Choices Behind RISE

Ablation studies show that every core module in RISE plays a critical role in stable robot learning and contact-rich manipulation performance.

Key Findings

Removing dynamics model pretraining reduced dynamic sorting accuracy by 32%
Removing the task-centric batch strategy decreased overall task success by 30%
Removing temporal-difference (TD) learning from the value model weakened failure detection and reduced success rates by 35%
An offline-to-online data ratio of 0.6 achieved the best balance between robot policy stability and autonomous exploration

6.Toward Scalable and Low-Cost Self-Improving Robot Learning

RISE demonstrates that a well-trained compositional world model can directly serve as an online reinforcement learning environment for real-world robot manipulation.

This brings three major advantages for embodied AI and autonomous robotics:

1. Lower Training Cost

RISE shifts expensive real-world robot trial-and-error into scalable computation, making high-performance robot policy learning more accessible for smaller robotics teams.

2. Higher Training Efficiency

Unlike traditional real-world RL, imagined robot interactions can run in parallel, dramatically accelerating robot learning and policy optimization.

3. Safer Robot Learning

By performing large-scale trial-and-error inside virtual environments, RISE reduces physical risks and prevents damage to real robotic systems during training.

At the same time, several challenges still remain:

small sim-to-real inconsistencies in rare manipulation scenarios
manual tuning of offline-to-online data ratios
high computational cost for large-scale world model training

Future research will likely focus on uncertainty-aware world models, physics-constrained robot prediction, and more efficient embodied AI training pipelines, enabling robots to solve increasingly complex real-world manipulation tasks through autonomous imagined learning.

Have Question?

If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/embodied-intelligence-upgraded-self-improving-robot-policies-via-rise-world-model-reasoning/55125

ROS Discourse General: Part 2: Preparing for State of Cloud Robotics Survey | Cloud Robotics WG Meeting 2026-06-01

Please come and join us for this coming meeting at Mon, Jun 1, 2026 4:00 PM UTC→Mon, Jun 1, 2026 5:00 PM UTC, where we plan to continue writing the survey questions for a new State of Cloud Robotics survey. The last survey was in 2024 (see https://cloudroboticshub.github.io/survey), and we’d like to refresh the results as of this year. We made good progress last session in updating our old questions and introducing new ones, so this coming session will be further refining the questions and grouping them in a logical way.

If you’re interested in watching the previous session, the meeting recording is available on YouTube.

The meeting link for next meeting is here, and you can sign up to our calendar or our Google Group for meeting notifications or keep an eye on the Cloud Robotics Hub.

Hopefully we will see you there!

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/part-2-preparing-for-state-of-cloud-robotics-survey-cloud-robotics-wg-meeting-2026-06-01/55114

ROS Discourse General: Connext Robotics Toolkit for ROS Lyrical Luth is now available

Hi ROS 2 Community,

To go with the Lyrical Luth release, RTI has updated the Connext Robotics Toolkit with Connext 7.7 LTS. The toolkit provides a simple single-step Debian installation of Lyrical and Connext DDS RMW on Ubuntu Linux, now supporting both amd64 and arm64 including Raspberry Pi and NVIDIA Orin platforms. It remains free-of-charge for prototyping, research, non-commercial and academic use.

Connext 7.7 brings several enhancements for ROS users looking to move from prototype to production:

Improved ROSâ€“DDS interoperability. New Topic and Type Aliasing automatically maps between ROS-mangled and conventional DDS names, making it far easier to integrate ROS 2 systems with native DDS applications based on standards like AUTOSAR, GVA and UMAA, without any code changes or naming gateways.

Improved performance, scalability and security. Faster discovery with less network overhead, reduced memory footprint, and enhanced security capabilities.

To learn more, see installation instructions and try the new aliasing feature hands-on, visit the full announcement here: https://community.rti.com/forum-topic/connext-77-ros-lyrical-luth-now-available

Feel free to reach out if you have any questions.

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/connext-robotics-toolkit-for-ros-lyrical-luth-is-now-available/55110

ROS Discourse General: New Synthetic Datasets for Industrial Bin Picking

Quick video promo: https://youtu.be/yYms177Rx3w?si=qH2vG1ggwfYdtumr
Datasets are available on Kaggle: Telekinesis | Kaggle
Telekinesis Agentic Skill library: https://docs.telekinesis.ai/

1 post - 1 participant

Read full topic

https://discourse.openrobotics.org/t/new-synthetic-datasets-for-industrial-bin-picking/55096

ROS Discourse General: ICRA 2026 in Vienna Open Thread

ICRA 2026 in Vienna Open Thread

Hi Everyone,

Now that the Lyrical release and Open Hardware Summit are behind me (ask @KimMcG or @adityakamath about their experiences at OHS ) I want to take a minute to talk about ICRA 2026 in Vienna.

@mrpollo and I have scrambled to put together not one, but two open source robotics meetups at ICRA 2026:

Our ROS meetup for ICRA will be on Tuesday Tue, Jun 2, 2026 4:00 PM UTC (6pm local time) and you can RSVP for that meetup here.
Our PX4 meetup for ICRA will be held on Thursday, Thu, Jun 4, 2026 5:00 PM UTC (7pm local time) and you can register for that event here.

Both @mrpollo and I will be at ICRA all week so feel free to message us if you want to meet and chat about what you are working on.

If you or your colleagues happen to be presenting a ROS-related talk at ICRA feel free to shamelessly self promote it in the thread below. We want to hear about what you are working on!

p.s. My suitcase is full of Lyrical stickers fresh from the printer. Iâ€™ll be handing them out so come and find me.

3 posts - 2 participants

Read full topic

https://discourse.openrobotics.org/t/icra-2026-in-vienna-open-thread/55091

ROS Discourse General: 🚀 Introducing Hiroz - by ZettaScale

Hiroz (High-performance Interoperable Robotics on Zenoh) is a project by ZettaScale Technology, the company supporting Eclipse Zenoh — and the new name for what was previously known as ROS-Z.

Hiroz is a robotics stack built on top of Zenoh, implemented in pure Rust. The core idea is to provide full ROS 2 compatibility while enabling roboticists to go beyond what the ROS architecture allows — with less coupling to ROS internals, faster iteration, and a foundation for enriching ROS itself.

Relationship with ROS 2 and rmw_zenoh

Hiroz is not a replacement for rmw_zenoh. It is a distinct layer that sits on top of Zenoh directly:

rmw_zenoh is a Tier-1 RMW implementation based on Zenoh and using it with specific key expressions and encoding (CDR). It can be used as a shared library via the ROS 2 rcl, rclcpp, rclpy and rcl-rs API
Hiroz is a Zenoh-native platform — not bound to the RMW interface, but still interoperable with ROS 2 by using the same key expressions and encoding

Both can coexist and communicate over the same Zenoh infrastructure, meaning you can mix your regular ROS 2 Nodes with enhanced Hiroz Nodes in a same system.

Features extending beyond ROS

Multiple serialization formats: CDR (ROS-compatible), Protobuf, and more to come
Multiple language bindings: Rust (native), Python, and Go (Go bindings sponsored by Dexory)
Zero-copy and GPU memory support: Zenoh’s buffer-aware pub/sub enables CPU and CUDA memory backends coming soon
Full-Rust stack: memory safety and predictable performance by design; fits naturally into AI-assisted development workflows
Framework interoperability: communicate between ROS 2 nodes and non-ROS systems over the same Zenoh infrastructure

For more details, see the full feature comparison page

Supported ROS 2 distributions

Distribution	Status
Humble	Supported
Jazzy	Supported
Kilted	Supported
Lyrical	Very soon

Links

Repository: https://github.com/ZettaScaleLabs/hiroz
Documentation: https://zettascalelabs.github.io/hiroz

Feedback, questions, and contributions are welcome. If you have been following ROS-Z, everything carries over — just under the new name.

At ZettaScale, we are committed to keep supporting rmw_zenoh and also to evolving Hiroz in close collaboration with partners and customers. If you have specific needs — performance, interoperability, custom transports, new language bindings — feel free to reach out.

— The ZettaScale team

5 posts - 3 participants

Read full topic

https://discourse.openrobotics.org/t/introducing-hiroz-by-zettascale/55081

ROS Discourse General: ROS Made Easier Now

ArcForge made a GUI for ROS2 for Windows and Mac that actively checks the sketch against detectable hardware.

6 posts - 4 participants

Read full topic

https://discourse.openrobotics.org/t/ros-made-easier-now/55076

Planet ROS

Planet ROS - http://planet.ros.org

ROS Discourse General: ROS2 multi-machine with WSL2 + Linux (NVIDIA Jetson Orin Nano)

ROS Discourse General: STS3215 pan/tilt + LD19: a no-SDK 3D scanning module for ROS 2 Jazzy

ROS Discourse General: Isaac Lab Teleoperation & Data Collection 2.0: Automated Block Stacking Dataset Generation with AgileX PiPER and NERO

0.Introduction

What’s New in this Tutorial ?

1.Systerm Architecture

2.Implementation Steps

Step 1.Automated Block Stacking Controller

Step 2.Differential IK-Based Control

Step 3.Trajectory Smoothing with PCHIP

Step 4.Automatic Success Verification

Step 5.Dataset Export

Step6. Running the Pipeline

3.Why This Matters

FAQ

Have Question?

ROS Discourse General: Invitation to African Robotics Summit 2026 - June 25, 2026

ROS Discourse General: EpisodeVault: open source tool to debug why your LeRobot model regressed

Four commands

What the diff looks like

The blame command

Compatibility

Install

ROS Discourse General: What's the most annoying part of your ROS 2 workflow?

ROS Discourse General: Logging and Observability Guide Discussion | Cloud Robotics WG Meeting 2026-06-15

ROS Discourse General: Context & Hardware Setup

The Problem

Current Configuration & Attempts

Questions for the Community

ROS Discourse General: Isaac Lab Teleoperation & Data Collection: Controlling the AgileX NERO 7-DoF Robotic Arm

0. Preface

1.Environment & Hardware Configuration

1.1 Generate the Unified URDF File

1.2 Compile URDF to USD Format

1.3 Create the Asset Configuration File

2.Task Environment Configuration

2.1 Configure Directory Tree Structure

3. Implement Configurations

3.1 bc_rnn_low_dim.json (Robomimic Behavior Cloning Profile)

3.2 nero_stack_joint_pos_env_cfg.py (Joint Position Environment)

3.3 Teleoperation, Data Collection & Replay

3.3.1 stack/config/nero/nero_stack_ik_rel_env_cfg.py

3.3.2 stack/config/nero/init.py**

4. Collect Demonstration Data via Teleoperation

FAQ

ROS Discourse General: Pre-built ROS2 Humble Windows SDK + Standalone RViz2 (No Toolchain Required)

ROS Discourse General: CS student diving into ROS 2 and Gazebo, looking for feedback and community

ROS Discourse General: Building a Rust tool for zero-allocation Serial Autobauding & Protocol Detection

ROS Discourse General: 2026 State of Cloud Robotics Survey

ROS Discourse General: Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI

Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI

0.Preface

Step 1.Create an Isaac Lab External Project

1.1 Create a New Isaac Lab External Project

1.1.1 Initialize the Project

1.1.2 Recommended Project Configuration

1.1.3 Install External Projects

1.2 Configure the PiPER USD Asset

1.2.1 Prepare the URDF

1.2.2 Convert URDF to USD

Step 2.Configure the PiPER Robot in Isaac Lab

2.1 Create the Robot Asset Configuration

2.1.1 Create the Asset Package

2.1.2 Configure the PiPER Robot Asset

Step 3.Build the Manipulation Environment

3.1 Project Structure Setup

3.2 Core File Modifications

3.2.1 Standard IsaacLab Porting

3.2.2 Event Customization (mdp/piper_stack_events.py)

3.2.3 Adapting Observations & Terminations for Piper

3.2.4 Piper Joint Position Configuration (piper_stack_joint_pos_env_cfg.py)

3.2.5 Inverse Kinematics Configuration (piper_stack_ik_rel_env_cfg.py)

3.2.6 Environment Registration (stack/config/piper/__init__.py)

Step 4.Teleoperation & Data Collection

4.1 Teleoperation Configuration

4.2 Teleoperation Configuration

Q & A

Have Question?

3.1 `bc_rnn_low_dim.json` (Robomimic Behavior Cloning Profile)

3.2 `nero_stack_joint_pos_env_cfg.py` (Joint Position Environment)

3.2.4 Piper Joint Position Configuration (`piper_stack_joint_pos_env_cfg.py`)

3.2.5 Inverse Kinematics Configuration (`piper_stack_ik_rel_env_cfg.py`)

3.2.6 Environment Registration (`stack/config/piper/init.py`)