Planet ROS
Planet ROS - http://planet.ros.org
Planet ROS - http://planet.ros.org![[WWW]](./rostheme/img/moin-www.png "[WWW]"){kind=small}
http://planet.ros.org
ROS Discourse General: ZeroDDS: a pure-Rust RMW for ROS 2 (rc.3), built against 349 real ROS↔DDS pain reports
Full disclosure up front: I work on ZeroDDS, so this is our project, not a neutral review. Sharing it here because it’s specifically aimed at problems this community has been reporting for years, and we’d genuinely like the feedback.
What it is: an alternative RMW for ROS 2, with the whole DDS stack underneath written from scratch in Rust (memory-safe, Apache-2.0, no per-seat license). It speaks native RTPS 2.5 and interoperates with your existing setup: ROS 2 over `rmw_zerodds` talks to ROS 2 over `rmw_cyclonedds` bidirectionally (20/20 in our pub/sub + service tests). It’s a drop-in `RMW_IMPLEMENTATION`, not a fork of your graph.
We didn’t guess at the pain, we catalogued it. Before writing “ROS 2 support” anywhere, we collected 349 real-world ROS↔DDS pain reports from this forum, GitHub and elsewhere, and engineered against the recurring ones:
-
Silent QoS / type mismatches — the classic where two endpoints just never match and nothing is logged. Fixed the keyed/no-key entity-id case, and shipped `zerodds-ros2-shim doctor`, a discovery-independent check that flags the most common WiFi misconfig (multicast-free with no unicast peers) as a hard error instead of a silent no-match.
-
WiFi discovery loss — a single SPDP beacon dropped during 802.11 power-save leaves you at `participants=0`. We send an initial-announcement burst (10×200 ms until matched), like FastDDS’ `initial_announcements`. Verified on a real WiFi rig, not just loopback.
-
Large fleets / WAN — a `multi_robot()` profile: unicast-only discovery, no multicast storm.
-
DDS-Security / SROS2. ZeroDDS consumes SROS2 enclave files (identity/permissions CA, cert, key, governance, permissions) and fails hard on a broken config (no silent downgrade). Caveat for ROS-users: today you point it at the enclave via a ZERODDS_SECURITY_DIR env var, not the standard ROS_SECURITY_KEYSTORE / ROS_SECURITY_ENABLE variables yet. Wiring those up so ros2 security works unmodified is still open.
-
Introspection — `ros2 topic info -v` resolves endpoint GUIDs to node names.
Works with rclpy and rclcpp.
New in rc.3: zero-copy shared memory now works on Windows too (via `CreateFileMapping` + `LockFileEx`), tested on a real Windows runner.
Next (rc.4): a continuous multi-distro live `ros2` smoke as a CI gate across Humble, Iron and Jazzy, so interop is proven on every commit rather than by hand. The `rmw_zerodds` surface itself is code-complete (pub/sub, services, wait-sets, loaned messages, REP-2009 type hash + endpoint info; nothing left `UNSUPPORTED`).
Honest about maturity: it’s a release candidate, not years-hardened. On performance we’re on par with the established stacks across real workloads, not uniformly ahead, and we don’t claim to beat RT-tuned commercial configs. An OMG Vendor-ID is filed but not yet assigned. We are not safety-certified.
Try it:
-
build the rmw layer and run `RMW_IMPLEMENTATION=rmw_zerodds_cpp ros2 …`
-
repo + docs: GitHub - zero-objects/zero-dds: OSS DDS implementation in Rust · GitHub · https://zerodds.de
If you hit a mismatch with your Cyclone/Fast-DDS nodes, or a QoS combination that doesn’t behave, that’s exactly the feedback we want. Tell us where it breaks.
(For the DDS/non-ROS folks who wander in: the same codebase also covers XTypes, DDS-Security, RPC, bridges, seven language bindings, and a full CORBA 3.3 ORB. Happy to go into any of it, but I’ll keep this post ROS-focused.)
1 post - 1 participant
https://discourse.openrobotics.org/t/zerodds-a-pure-rust-rmw-for-ros-2-rc-3-built-against-349-real-ros-dds-pain-reports/55581
ROS Discourse General: A VS Code extension for quickly running ROS2 launch files, Python nodes, and C++ nodes
ROS2 Quick Runner
This is a VSCode extension I recently wrote, and I’ve been using it lately
A VS Code extension for quickly running ROS2 launch files, Python nodes, and C++ nodes.
Features
1. Launch ROS2 Files
Right-click any .launch.py file and select “ros2 launch” to:
- Automatically find the ROS2 workspace
- Source the workspace’s
install/setup.bash - Execute
ros2 launch <package_name> <launch_file>
2. Run ROS2 Nodes
Right-click any .py or .cpp file and select “ros2 run” to:
- Automatically find the ROS2 workspace and package name
- Source the workspace’s
install/setup.bash - Execute
ros2 run <package_name> <node_name>
3. Get Workspace Name
Right-click any file and select “ros2 source” to:
- Display the workspace name
- Source the workspace in a new terminal
4. Build ROS2 Workspace
Right-click any folder in your ROS2 workspace and select “colcon build” to:
- Automatically find the ROS2 workspace root
- Open a terminal at the workspace root
- Execute
colcon build
Smart path detection:
- Right-click
xxx_ws/→ build inxxx_ws/ - Right-click
xxx_ws/src/→ build inxxx_ws/ - Right-click any sub-folder (e.g.
xxx_ws/src/pkg_a) → build inxxx_ws/
How It Works
The extension automatically:
- Finds the ROS2 workspace by searching for directories whose
src/subdirectory contains at least one package withpackage.xml - Extracts the package name by parsing the
package.xmlfile in the package directory - Executes commands in a new VS Code terminal
Usage
- Search for
ros2-quick-runnerin VS Code extensions and install it - Open your ROS2 project in VS Code
- Right-click on a file in the explorer:
.launch.pyfiles → “ros2 launch”.pyor.cppfiles → “ros2 run”- Any file → “ros2 source”
- Any folder → “colcon build”
Commands
| Command | Description |
|---|---|
ros2-quick-runner.ros2launch |
Launch a .launch.py file |
ros2-quick-runner.ros2run |
Run a Python or C++ node |
ros2-quick-runner.getWorkspaceName |
Get and source the workspace |
ros2-quick-runner.colconBuild |
Build the ROS2 workspace |
Requirements
- VS Code 1.80.0 or higher
- ROS2 installed (e.g., ROS 2 Humble)
- A compiled ROS2 workspace with
install/directory
Release Notes
For detailed changelog, please see: vscode-ros2-quick-runner/CHANGELOG.md at main · Knighthood2001/vscode-ros2-quick-runner · GitHub
0.0.3
- Initial release
- Support for ROS2 launch files, Python nodes, and C++ nodes
- Automatic workspace detection
- Automatic package name extraction
1 post - 1 participant
https://discourse.openrobotics.org/t/a-vs-code-extension-for-quickly-running-ros2-launch-files-python-nodes-and-c-nodes/55563
ROS Discourse General: ROS 2 Robot V1.0.0 - A Modern GUI for ROS 2 Workspace Management (Like GitHub Desktop for ROS)
Hi everyone,
“ROS is used to build robots, but isn’t it time for ROS to have its own robot?�
I’m excited to share a tool I’ve been working on to make ROS 2 development a bit smoother: Ros2 Robot.
As developers, we spend a lot of time typing repetitive terminal commands just to manage packages, source workspaces, and inspect topics. I wanted to build a tool that handles the boilerplate so we can focus on actual robotics innovation. It also serves as a great stepping stone for newcomers who find the ROS 2 CLI overwhelming.
What is it?
Ros2 Robot is a PyQt/PySide6 based GUI that acts like “GitHub Desktop� for your ROS 2 projects.
Key Features in V1.0.0:
- Create packages, rebuild, and source them all with one click.
- Run and manage nodes all from one place.
- Inspect topics in real-time.
- Manage ROS bags effortlessly.
- Build launch files by adding blocks.
- Visualize your URDF files and control your joints.
- Open your favorite ROS Plugins directly from the interface.
- Educational: understand what each feature does in the background CLI.
- Supports Native Linux (Tested On Ubuntu 22.04/24.04) and Windows with WSL 2!
Installed with ��� ����, and launched with ��� �������.
GitHub Repository: GitHub - saheraalreqeb/Ros2_Robot: Ros2 Robot is a simple GUI designed to help you manage your ROS 2 projects a. Think of it as GitHub Desktop, but for ROS 2. It eliminates those repetitive, soul-crushing terminal commands so you can actually focus on coding and development. · GitHub
(I’ll be posting a full video tutorial later today showing it in action!)
I’m releasing this as V1.0.0 because the core features are stable, but this is just the beginning. I would love for the community to test it out. If you encounter bugs, have ideas for new features, or want to contribute code, please head over to the GitHub repo and open an issue or a PR.
Let me know what you think!
2 posts - 1 participant
https://discourse.openrobotics.org/t/ros-2-robot-v1-0-0-a-modern-gui-for-ros-2-workspace-management-like-github-desktop-for-ros/55555
ROS Discourse General: Seeking feedback: Is dependency blocking during ROS development a significant pain point?
Seeking Feedback from Robotics Software Engineers / ROS Developers
Hello everyone,
I hope you are doing well.
I am a BE Computer Science student with a strong interest in robotics software. I am currently researching a potential software product and, before investing significant time into building it, I would like to validate whether the problem I am trying to solve is a genuine challenge faced by robotics engineers.
From my research, I have observed that robotics engineers are often blocked because another part of the robot is not yet available or is temporarily broken. For example:
- A navigation engineer cannot continue because localization is not ready.
- A manipulation engineer cannot continue because MoveIt is unavailable or malfunctioning.
- A docking engineer cannot test their feature because the navigation stack is incomplete.
To continue development, engineers often write temporary Python scripts, use ROS CLI commands, or create custom test nodes just to simulate missing robot components.
The idea I am exploring is a ROS-native platform that allows engineers to discover, call, mock, save, and reuse ROS 2 Actions and Services through a visual interface, helping them continue development without waiting for missing dependencies.
Before building anything, I would sincerely appreciate your honest opinion.
- Have you experienced dependency-blocking situations like these in your projects?
- How do you currently work around these problems?
- If this problem exists, how painful is it in day-to-day development?
- Do you think a tool like this would provide meaningful value, or am I solving the wrong problem?
- Is there any existing tool that already solves this problem well?
- Is there anything important that I might be overlooking?
I genuinely welcome honest criticism, even if your opinion is that this problem is not worth solving. Constructive feedback would help me avoid spending months building something that isn’t valuable to the robotics community.
If anyone is willing to discuss this in more detail, I would be sincerely grateful for even 10–15 minutes of your time through a direct message, phone call, or Google Meet at your convenience.
Thank you very much for taking the time to read this post. I truly appreciate any feedback, suggestions, or guidance you are willing to share.
is this good and worth posting
2 posts - 2 participants
https://discourse.openrobotics.org/t/seeking-feedback-is-dependency-blocking-during-ros-development-a-significant-pain-point/55553
ROS Discourse General: Has anyone tried conformal prediction for sensor gating in a nav stack?
Been thinking about this for a while.
The chi-squared gate in most localization stacks assumes Gaussian noise. When when we look outdoors… it rarely is. Like doing multipath near buildings, or under tree canopy, and even around field equipment.
I came across this conformal prediction (Angelopoulos & Bates 2022). The idea is: instead of assuming a distribution, you basically test each new measurement against your own empirical data. It states coverage guarantees hold regardless of noise shape.
Has anyone tried something like this in a nav stack? And honestly… is GPS covariance mismatch painful enough in real deployments that it’s worth a proper fix, or does tuning R get you 90% of the way there?
2 posts - 2 participants
https://discourse.openrobotics.org/t/has-anyone-tried-conformal-prediction-for-sensor-gating-in-a-nav-stack/55539
ROS Discourse General: ROS2 on STM32 serial to SBC
Hi all,
I’m building a small ROS 2 robot with an STM32 microcontroller and an SBC running a Zenoh router (rmw_zenoh). I want to keep the MCU side Zenoh-native with Zephyr RTOS, and I’m trying to figure out the best communication stack between the STM32 and the rest of the graph.
I’ve been looking at three options:
- micro-ROS (classic) — well documented, but I’m not sure whether I’d still need zenoh-bridge-dds on the SBC side given I’m already running rmw_zenoh there
- Zenoh-Pico with rmw_zenoh_pico (eSOL) — conceptually a perfect fit, but the repo seems to only target Linux/RPi so far, and I’m not sure how much work porting to STM32 + Zephyr would be
- Pico-ROS — appeals to me architecturally (native Zenoh, no micro-ROS overhead), but documentation is very sparse and I couldn’t find real-world STM32 examples
I’m not necessarily looking for a solution that works out of the box — I’m happy to do integration work. What I’m really after is whether anyone has actually tried any of these on STM32 + Zephyr, what pain points they hit, and whether there’s a better path I’m missing.
Any experience or pointers would be greatly appreciated!
4 posts - 4 participants
https://discourse.openrobotics.org/t/ros2-on-stm32-serial-to-sbc/55510
ROS Industrial: Bringing the ROS Community Together: Meetups in Heilbronn and Karlsruhe
Beyond the large annual conferences, some of the most valuable moments in the ROS ecosystem happen at a smaller scale, when local developers, researchers, and companies gather in one room for an afternoon of talks, demos, and conversation. Over the spring, the robotics community had two such occasions: the first-ever ROS Meetup in Heilbronn in March, followed by the second ROS Meetup in Karlsruhe in May. Both events reflected a healthy, growing grassroots scene around ROS 2, and both reaffirmed why face-to-face exchange remains so important to the open-source robotics community.
A First for Heilbronn
On 23 March 2026, the robotics community came together at the TUM Campus Heilbronn (Bildungscampus) for the first ROS Meetup ever hosted in the city. The event was organized jointly by Neobotix GmbH, TUM Campus Heilbronn, and Fraunhofer IPA, and brought together developers, researchers, and students for an afternoon of talks, lab tours, and networking over a shared lunch at the Bildungscampus.
The afternoon opened with an introduction from Neobotix GmbH before moving into a full technical program. Denis Stogl of b-robotized presented recent improvements to real-time control in ros2_control, focusing on coordinating multiple robots, an area where reliable, deterministic control is essential for industrial adoption. Robert Wilbrandt of the FZI Forschungszentrum Informatik for Information Technology introduced a reusable, MoveIt-based manipulation planning component designed to simplify the integration of motion planning across projects. Vishnuprasad Prachandabhanu and Sanjeev Kumar from Fraunhofer IPA followed with a talk on developing ROS 2 controllers for the Unitree G1 humanoid platform.
The lineup also reached into learning, general-purpose robotics, and safety. Pauline Steffel, a PhD student in the AImotion department, presented work toward a configurable and reusable reinforcement-learning training infrastructure for autonomous mobile robots in ROS 2. Tobias Weyer of TNG Technology Consulting shared a broader perspective on general-purpose robotics with ROS, and Zhen Zhang of TUM Campus Heilbronn closed the talks with research on safe, LLM-controlled robots that provide formal guarantees through reachability analysis.
The presentations were complemented by hands-on demonstrations. Attendees were given an inside look at the research underway in the TUM Cyber-Physical Systems labs, and Neobotix showed several of its professional-grade mobile robots running live. As is often the case, the networking session afterward proved just as valuable as the talks themselves, providing the kind of informal, face-to-face exchange that turns into future collaboration.
Organizing a community event from scratch is no small undertaking, and the success of this first edition owed much to the team at Neobotix together with TUM Campus Heilbronn's Cyber-Physical Systems group, including Prof. Amr Alanwar, Zhen Zhang, and Hadi Elnemr, and the support staff who made the venue and logistics work. ROS-Industrial Europe and Fraunhofer IPA were glad to collaborate on bringing the meetup to life. For a first event, it set an encouraging precedent and showed clear appetite for a recurring gathering in the region.
A Second Round in Karlsruhe
Two months later, on 21 May 2026, the community reconvened at the FZI Forschungszentrum Informatik House of Living Labs in Karlsruhe for the second ROS Meetup hosted there. With six presentations spanning research and industry, the afternoon offered a broad cross-section of how ROS is being applied today, from manipulation and motion planning to medical robotics and industrial machine tools.
Robert Wilbrandt of the FZI Forschungszentrum Informatik for Information Technology opened the technical program with an overview of how ROS sits at the core of FZI's robotics projects. He highlighted several open-source packages the center maintains, including vdb_mapping for long-term, large-scale 3D mapping and navigation, behavior-tree-based approaches for the automatic, LLM-driven generation of assembly and disassembly programs. Dr. Jennifer Bühler and Dr. Denis Stogl of the b-robotized group followed with their approach to the Intrinsic Industrial AI Challenge using HIL-SERL (Human-in-the-Loop Sample-Efficient Reinforcement Learning), a method that combines a small number of demonstrations with targeted human interventions during training so that manipulation tasks can be learned with far less task-specific data than many current approaches.
The motion-planning thread continued with Sebastian Jahr of the ZEISS Group, who introduced OInK, an optimal inverse kinematics solver built on Roboplan, an emerging library based on Pinocchio. OInK is a QP-based differential IK solver written in C++ that computes joint commands in real time while tracking multiple objectives and respecting constraints and safety barriers. Dr.-Ing. Marius Siegfarth and Javier Moviglia from the Mannheim Institute for Intelligent Systems in Medicine (Medical Faculty Mannheim of Heidelberg University) then turned the focus to medical robotics, showing how they use ROS to develop robotic prototypes for operating-room automation, integrate them with imaging modalities such as CT and MRI, and connect devices within the OR environment.
Rounding out the program, Matthias Mayr of Mayr Robotics presented a Cartesian impedance control stack for torque-controlled manipulators. Built with ros2_control, it enables the compliant interaction that many real-world manipulation tasks require and generalizes across platforms such as the Franka Research 3 and the KUKA iiwa. Finally, Matthias Marquart (ISW, University of Stuttgart) and Benjamin Kaiser (ISG – Industrielle Steuerungstechnik GmbH) bridged ROS and industrial CNC. Motivated by use cases such as robotic timber milling, they showed a hybrid architecture in which a TwinCAT CNC triggers a MoveIt planner for collision-free path planning and then executes the resulting path back on the CNC, combining industrial reliability with the flexibility of the ROS ecosystem, before looking ahead to native integration of the ISG-Kernel SDK into ROS.
The meetup wrapped up with a tour of FZI's robotics labs, including a showcase of the center's custom-developed legged robots, followed by more networking.
Why These Events Matter
Taken together, the Heilbronn and Karlsruhe meetups illustrate something the ROS-Industrial community has long believed: real-world progress depends as much on people connecting as on code being written. The talks spanned the full breadth of the field, from low-level real-time control and inverse kinematics to reinforcement learning, medical automation, and CNC integration, while the demos and lab tours grounded those ideas in working hardware.
A sincere thank you goes to everyone who made these gatherings possible: the organizers at Neobotix, TUM Campus Heilbronn, FZI Forschungszentrum Informatik, and Fraunhofer IPA, the speakers who shared their work, and the many attendees who brought their curiosity and ideas. With two successful meetups behind us, we look forward to seeing this regional community continue to grow, and to many more afternoons of talks, demos, and good conversation.
https://rosindustrial.org/news/2026/6/15/bringing-the-ros-community-together-meetups-in-heilbronn-and-karlsruhe
ROS Discourse General: Looking for a repository within ROS2 to support
Hello guys, I have history with C, python and have worked with ROS2 and ROS2 controls, for Control Theory stuff. I would like to be involved in supporting a repository or a stable project. Kindly recommend a starting point. You can also check my github to see if I woul be suitable for your project at KevinKipkorir254.
2 posts - 2 participants
https://discourse.openrobotics.org/t/looking-for-a-repository-within-ros2-to-support/55489
ROS Discourse General: ROS2 Studio Update
ROS2-STUDIO update— new Bag Recorder features!
Just pushed some improvements to the bag recorder in ROS2-STUDIO:
Storage format selection — choose between sqlite3 and mcap directly from the GUI
Timed recording — set a duration (in seconds) and the recorder auto-stops when done
Batch/split recording— split bags into chunks every N seconds (--max-bag-duration style)
All options are configurable from the UI without touching the terminal.
Feedback and contributions welcome!
1 post - 1 participant
https://discourse.openrobotics.org/t/ros2-studio-update/55482
ROS Discourse General: LSEP v0.2 — now a buildable ROS 2 reference implementation: typed lsep_msgs + managed lifecycle node, green CI
Hi everyone,
v0.2.0 now ships a runnable ROS 2 reference implementation — a package you can colcon build in a couple of minutes, with typed messages, a managed lifecycle node, and green CI. Before anything gets locked towards a 1.0, I’d like people who run real robots to poke holes in the architecture.
What’s in v0.2
1. Typed interfaces (lsep_msgs) lsep_msgs/Signal carries the state twice: as a uint8 enum for machines and as a state_name string for debugging via ros2 topic echo. Undefined physics are encoded as NaN instead of magic numbers (e.g. TTC when closing velocity is zero). Light, Sound, and Motion are separate message types — currently nested inside Signal on a single topic. The win today is type reuse and zero parsing overhead versus our previous JSON-in-String transport; whether modalities should get their own topics is an open question I’d like your input on (see below).
2. Managed lifecycle node The core lsep_node is an rclpy.lifecycle.Node following the ROS 2 managed-nodes design, so a safety supervisor can orchestrate configure/activate/deactivate deterministically and keep the signaling layer isolated from the navigation stack.
3. Engine v2.1 — two blind spots in my original state machine, fixed
- Dwell-based de-escalation: a calmer raw state must remain stable for
dwell_de_escalation_s(default 1.5 s) before the machine steps down. Escalation stays immediate; THREAT (< 0.5 s TTC) bypasses everything. - Input watchdog: if the sensor stream dies or occludes for longer than
input_timeout_s, the reported state degrades toLOW_CONFinstead of confidently latching the last danger state on dead sensors.
4. Deprecation policy The old JSON string on /lsep/state is kept as a deprecated mirror behind a publish_json parameter, so nothing existing breaks during migration.
Evidence
CI builds the workspace in a ros:jazzy container on every push, runs the 9/9 unit tests (the engine is testable without a ROS installation), and runs a headless smoke-test where a simulated human approaches, lingers, and retreats. The CI job hard-fails unless the observed state ladder actually reaches THREAT and then de-escalates — so green CI proves the temporal behavior, not just compilation. Ordered ladder from the smoke-test (two full cycles):
INTEGRITY → AWARENESS → INTENT → CARE → CRITICAL → THREAT → AWARENESS → IDLE
→ AWARENESS → INTENT → CARE → CRITICAL → THREAT → AWARENESS
└─ dwell-based de-escalation, 2x ─┘
The THREAT -> AWARENESS step is the one that matters: it shows the dwell timer stepping the robot down only after the danger has genuinely cleared, rather than oscillating.
What I’d genuinely like review on
- State representation:
uint8enum + mirror string in one message — sane, or would you model this differently? - NaN convention:
NaNinfloat32fields for undefined physics — acceptable, or do you prefer explicit validity bools? - QoS: what profile would you expect for a safety-adjacent signaling topic at 10 Hz — reliable vs. best effort, what depth?
- Topic layout: one
Signaltopic with nested modalities (current), or per-modality topics (lsep/light, …) so e.g. an LED driver subscribes only to what it renders? - Naming/namespacing conventions you’d want before this touches real fleets.
Known limits (honestly)
The TTC computation is currently 1D (distance / closing_velocity) — a placeholder; 2D time-to-intercept with trajectory prediction is on the roadmap. No hardware-in-the-loop testing yet — validation so far is the ros:jazzy CI build plus the simulated human, not a physical robot. This is a reference implementation, not certified production safety code.
Repo & 5 minute quick start: GitHub - NemanjaGalic/LSEP: Open protocol for standardized human-robot communication — 9 states, 3 modalities, 1 grammar. Physics-based. EU AI Act ready. · GitHub Release: Release ROS 2 Reference Implementation v0.2.0 · NemanjaGalic/LSEP · GitHub
Thanks for any time you spend tearing this apart.
— Nemanja
3 posts - 2 participants
https://discourse.openrobotics.org/t/lsep-v0-2-now-a-buildable-ros-2-reference-implementation-typed-lsep-msgs-managed-lifecycle-node-green-ci/55466
ROS Discourse General: Measure serialized topic bandwidth and serialization times using ros_babel_fish_tools
Greetings fellow roboticists ,
I’ve just added something to ros_babel_fish which could be useful to you as well.
I wanted to know how much bandwidth the serialized message on a topic consumes and how much latency the serialization adds.
So, ros_babel_fish_tools got a new tool: stats
While the existing echo allows subscribing to any topic and outputs the message content as JSON or yaml, the new stats also subscribes to any topic, measures the received rate, message latency (if the message has a header and requires synchronized clocks), the deserialization time from serialized message to ROS message, and the bandwidth (based on the size of the serialized message).
Additionally, you can check how compression of the serialized message would affect the bandwidth and latency.
For example, for our filtered point cloud, it makes quite a difference, but for our compressed wide-angle image, it is wasted time.
Thank you for reading to the end, here’s your image:
1 post - 1 participant
https://discourse.openrobotics.org/t/measure-serialized-topic-bandwidth-and-serialization-times-using-ros-babel-fish-tools/55462
ROS Discourse General: ROS2 multi-machine with WSL2 + Linux (NVIDIA Jetson Orin Nano)
Hi everyone,
I am trying setting up a ROS2 multi-machine communication between WSL2 (Ubuntu 22.04) running on a local Windows11 machine and a Linux-native platform, i.e., NVIDIA Jetson Orin Nano. My objective is generating data on my WSL2 machine, and process them using the Jetson.
However, I am experiencing issues related to the fact that the two machines cannot see the respective topics / nodes. They are both connected to the same network, and I have activated the mirroring mode networking in WSL2 using the flag networkingMode=mirrored. I have also tried using Cyclone and FastDDS using custom configuration files, but nothing worked so far.
I was thus wondering if anybody has succeded in setting up a proper ROS2 communication between WSL2 and a native Linux machine. If you need any further information, please do not hesitate to ask.
Thank you in advance!
Simone
4 posts - 2 participants
https://discourse.openrobotics.org/t/ros2-multi-machine-with-wsl2-linux-nvidia-jetson-orin-nano/55452
ROS Discourse General: STS3215 pan/tilt + LD19: a no-SDK 3D scanning module for ROS 2 Jazzy
I put together a small ROS 2 subsystem that turns a 2-DOF pan/tilt platform and a cheap 2D LiDAR into a stop-and-capture 3D scanner, and figured it might be useful to someone else here.
The setup: two Feetech STS3215 serial-bus servos aim an LDROBOT LD19. A node sweeps the platform and an assembler stacks the 2D scans into a `PointCloud2` using the live TF tree. There’s an optional MQTT bridge so an external controller (in my case a microcontroller mission queue on a rover) can trigger scans and get a completion handshake back.
It’s a *complete* project — it even includes a fix to the LiDAR driver (upstream `ldlidar_stl_ros2` won’t build on recent GCC/glibc; the patched fork is linked below). It talks to the rover over a well-defined set of MQTT messages, but every command also has an equivalent ROS 2 topic, so if you want a pure ROS 2 setup you just don’t launch the bridge. (Personally I love the MQTT side — it lets me drive the whole thing from a tablet.)
No vendor SDK — the Feetech STS/SMS half-duplex protocol is implemented directly over pyserial, including handling the URT-1 adapter’s habit of echoing every TX byte back on the RX line (the kind of thing that eats an evening if you don’t know it’s coming). The assembler is driver-agnostic: it consumes standard `sensor_msgs/LaserScan` on `/scan`, so any conformant 2D LiDAR should work.
It’s running on an RK3588 today and is built to go headless on a Pi 5.
This is the first piece I’m open-sourcing from a larger autonomous rover project, GPL-3.0. I’d genuinely welcome feedback — particularly from anyone who’s done multi-LiDAR or TF-timing work, since the scan-to-TF synchronization was the fussiest part to get right. But it does work! Happy to answer questions about any of it.
- Patched LiDAR driver: GitHub - aa2mz/ldlidar_stl_ros2: LDROBOT DTOF LiDAR ROS2 Package · GitHub
5 posts - 3 participants
https://discourse.openrobotics.org/t/sts3215-pan-tilt-ld19-a-no-sdk-3d-scanning-module-for-ros-2-jazzy/55447
ROS Discourse General: Isaac Lab Teleoperation & Data Collection 2.0: Automated Block Stacking Dataset Generation with AgileX PiPER and NERO
0.Introduction
In our previous article, we introduced a complete teleoperation and demonstration collection pipeline for the AgileX NERO and PiPER robotic arms in Isaac Lab.
The project demonstrated how to:
-
Import custom robot assets into Isaac Lab
-
Configure Differential IK control
-
Perform keyboard-based teleoperation
-
Record demonstrations into HDF5 datasets
-
Replay trajectories for verification
Previous article:
Isaac Lab Teleoperation & Data Collection: Controlling the AgileX NERO 7-DoF Robotic Arm
Repository: The full source code is available in the repository.
While teleoperation is an effective approach for collecting expert demonstrations, scaling datasets through manual operation quickly becomes time-consuming and difficult to maintain consistently.
In this follow-up work, we extend the original pipeline with an automated block stacking data generation framework that can autonomously complete manipulation tasks and continuously export successful demonstrations.
What’s New in this Tutorial ?
Compared with the previous teleoperation workflow, this version introduces:
Automated trajectory generation
Autonomous block stacking execution
Differential IK-based task completion
PCHIP trajectory smoothing
Automatic success detection
Continuous dataset generation
Zero human intervention during collection
The result is a scalable demonstration generation pipeline suitable for robot learning workflows.
1.Systerm Architecture
The overall workflow is shown below:
Cube Position Detection
↓
Trajectory Planner
↓
IK Stack Controller
↓
Differential IK (Isaac Lab)
↓
Robot Motion
↓
Recorder Manager
↓
HDF5 Dataset Export
Unlike the teleoperation workflow, no human operator is involved once the simulation starts.
The controller generates Cartesian target motions, while Isaac Lab’s built-in Differential IK module handles inverse kinematics computation internally.
2.Implementation Steps
Step 1.Automated Block Stacking Controller
A new script was introduced:
scripts/tools/record_ik_stack.py
The controller executes a complete stacking sequence:
Approach Cube
↓
Descend
↓
Grasp
↓
Lift
↓
Move To Target
↓
Place
↓
Retreat
Instead of commanding joint angles directly, the controller outputs Cartesian pose increments:
action = [ Δx,Δy,Δz,Δrx,Δry,Δrz,gripper ]
This design allows the same controller logic to be reused across different robot configurations.
The newly introduced block stacking controller is shown below.
sequence = [
(self._shift_z(src, self.Z_PRE), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(src, self.Z_LIFT), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PRE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_OPEN_VAL, 20),
(self._shift_z(place_pos, self.Z_RETREAT), self.GRIP_OPEN_VAL, 30),
]
Code shown below to realize this step
def plan(self) -> None:
"""plan the full pick-and-place trajectory."""
cube_height = self.cubes[0].data.root_pos_w[:, 2:3].clone(2.0
all_waypoints: list[torch.Tensor] = []
all_gripper_vals: list[float] = []
all_durations: list[int] = []
base_pos = self.cubes[0].data.root_pos_w.clone()
for i in range(1, len(self.cubes)):
src = self.cubes[i].data.root_pos_w.clone()
place_pos = self._shift_z(base_pos, cube_height)
sequence = [
(self._shift_z(src, self.Z_PRE), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(src, self.Z_LIFT), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PRE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_OPEN_VAL, 20),
(self._shift_z(place_pos, self.Z_RETREAT), self.GRIP_OPEN_VAL, 30),
]
for wp, gv, dur in sequence:
all_waypoints.append(wp.squeeze(0))
all_gripper_vals.append(gv)
all_durations.append(dur)
all_durations[-1] += 30
base_pos = self._shift_z(base_pos, cube_height)
# Generate smooth trajectory
self._path = self._generate_smooth_trajectory(all_waypoinall_durations)
# Expand gripper timeline
self._gripper_timeline = []
for gv, dur in zip(all_gripper_vals, all_durations):
self._gripper_timeline.extend([gv] * dur)
self._total_steps = self._path.shape[0]
self._gripper_timeline = self._gripper_timeline[:s_total_steps]
self._step_idx = 0
self._post_traj_wait_count = 0
self._last_gripper_val = self.GRIP_OPEN_VAL
print(f"[IKStackController] Trajectory planned: {s_total_steps} steps")
Step 2.Differential IK-Based Control
Rather than solving inverse kinematics explicitly, the controller outputs Cartesian pose increments and delegates IK computation to Isaac Lab’s built-in Differential IK controller.
def compute_action(self, env: gym.Env) -> torch.Tensor:
"""Return the action for the current step.
Returns:
torch.Tensor of shape ``(1, 7)``:
``[Δx, Δy, Δz, Δrx, Δry, Δrz, gripper_binary]``
"""
# trajectory exhausted → hold position
if self._step_idx >= self._total_steps:
return torch.tensor(
[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, self_last_gripper_val]],
device=env.device,
)
# delta position = target − current
target_pos = self._path[self._step_idx] # [1, 3]
current_pos = self.ee_frame.data.target_pos_w[:, 0, :] # [1,3]
delta_pos = target_pos - current_pos # [1, 3]
# delta rotation = zero (keep orientation)
delta_rot = torch.zeros(1, 3, device=env.device)
# gripper binary command
gv = self._gripper_timeline[self._step_idx]
grip_tensor = torch.tensor([[gv]], device=env.device)
action = torch.cat([delta_pos, delta_rot, grip_tensor],dim=-1) # (1, 7)
self._step_idx += 1
self._last_gripper_val = gv
return action
Step 3.Trajectory Smoothing with PCHIP
One challenge during autonomous execution is trajectory discontinuity between waypoints.
To improve motion quality, the controller uses Hermite-based interpolation with PCHIP-style tangents to generate smooth Cartesian trajectories.
@staticmethod
def _generate_smooth_trajectory(
waypoints: list[torch.Tensor], durations: list[int]
) -> torch.Tensor:
"""Interpolate waypoints with PCHIP-style tangents.
Args:
waypoints: list of ``(3,)`` tensors.
durations: steps per segment (``len == len(waypoints) -1``).
Returns:
``torch.Tensor`` of shape ``(Total, 1, 3)``.
"""
n = len(waypoints)
wp = torch.stack(waypoints) # [N, 3]
tangents = torch.zeros_like(wp)
if n > 1:
tangents[0] = wp[1] - wp[0]
tangents[-1] = wp[-1] - wp[-2]
for i in range(1, n - 1):
tangents[i] = (wp[i + 1] - wp[i - 1]) / 2.0
segments = []
for i in range(n - 1):
p0, p1 = wp[i], wp[i + 1]
m0, m1 = tangents[i], tangents[i + 1]
steps = durations[i]
if torch.norm(p1 - p0) < 1e-4:
seg = p0.unsqueeze(0).repeat(steps, 1)
else:
s = torch.linspace(0, 1, steps, device=p0.device)unsqueeze(-1)
h00 = 2 * s**3 - 3 * s**2 + 1
h10 = s**3 - 2 * s**2 + s
h01 = -2 * s**3 + 3 * s**2
h11 = s**3 - s**2
seg = h00 * p0 + h10 * m0 + h01 * p1 + h11 * m1
segments.append(seg)
return torch.cat(segments, dim=0).unsqueeze(1) # [Total, 1, 3]
Step 4.Automatic Success Verification
To ensure dataset quality, demonstrations are exported only after the stacking task is successfully completed.
The workflow continuously evaluates the stacking success condition:
Failed trajectories are automatically discarded.
def process_success_condition(env: gym.Env, success_term: object | None, success_step_count: int) -> tuple[int, bool]:
if success_term is None:
return success_step_count, False
if bool(success_term.func(env, **success_term.params)[0]):
success_step_count += 1
if success_step_count >= args_cli.num_success_steps:
env.recorder_manager.record_pre_reset([0], force_export_or_skip=False)
env.recorder_manager.set_success_to_episodes(
[0], torch.tensor([[True]], dtype=torch.bool, device=env.device)
)
env.recorder_manager.export_episodes([0])
print("Success condition met! Recording completed.")
return success_step_count, True
else:
success_step_count = 0
return success_step_count, False
Step 5.Dataset Export
Successful demonstrations are exported automatically using Isaac Lab’s Recorder Manager.
Output format:
ik_dataset.hdf5
The exported datasets contain:
-
Observations
-
Actions
-
Episode Metadata
-
Success Labels
and can be directly integrated into:
-
Robomimic
-
LeRobot
-
ACT
-
Diffusion Policy
-
OpenVLA
with minimal conversion effort.
Step6. Running the Pipeline
- PiPER
python scripts/tools/record_ik_stack.py \
--task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
--device cuda \
--dataset_file ./datasets/ik_dataset.hdf5 \
--num_demos 10
- NERO
python scripts/tools/record_ik_stack.py \
--task Isaac-Stack-Cube-Nero-IK-Rel-v0 \
--device cuda \
--dataset_file ./datasets/ik_dataset.hdf5 \
--num_demos 10
3.Why This Matters
This work moves the Isaac Lab workflow from:
Human Demonstration Collection
towards
Automated Demonstration Generation
which is an important step for building large-scale robot learning datasets.
Rather than spending hours manually collecting demonstrations, developers can now generate consistent trajectories automatically and focus on downstream policy training.
FAQ
Q1: Is this solution fully compatible with all Sogot desktop robotic arms?
This solution natively supports the two mainstream Sogot desktop robotic arm models: PiPER and NERO. It features dedicated optimizations for joint limits, movement speed, gripper parameters, and workspace of these two models. You can switch between models directly via corresponding task commands. It is not compatible with other arm models out of the box, but compatibility can be achieved by fine-tuning motion offset parameters and trajectory step lengths.
Q2: How to resolve environment mismatch errors during runtime?
First, confirm you have deployed the IsaacLab base environment with CUDA support, and verify matching versions for PyTorch, gymnasium, and Pinocchio dependencies.
Next, ensure the task parameter in your run command matches your Sogot arm model — PiPER and NERO task commands cannot be interchanged.
Finally, check that GPU acceleration is enabled. This project requires CUDA acceleration for robotic arm inverse kinematics (IK) solving and trajectory computation; running on CPU will throw runtime errors immediately.
Q3: Can the collected datasets be directly used for training physical Sogot robotic arms?
Yes. The datasets output by this solution replicate the physical characteristics of real Sogot arms across motion workspace, control logic, and movement trajectories. Simulation data aligns closely with real hardware control logic, so no extra format conversion is required. The datasets can be directly applied to imitation learning training for physical PiPER and NERO arms, effectively cutting the high cost and operational risks of real-world data sampling.
Q4: How to customize data sample volume and sampling frequency?
Flexible configuration is available via runtime command arguments:
-
num_demos: Sets the total number of successful demonstration samples to collect; set to 0 for infinite loop sampling.
-
step_hz: Adjusts the simulation sampling frequency (default: 30 Hz). Tune this value based on your Sogot arm’s movement speed to match different operation paces.
Q5: What causes stuttering trajectories or stacking failures on the robotic arm?
This issue most often stems from two root causes:
CUDA acceleration is not active, leading to delayed IK solving and choppy trajectory calculations.
Mismatch between simulation environment frame rate and robotic arm motion step length.
Troubleshooting steps: First check your GPU operational status. Then fine-tune PCHIP trajectory interpolation step length and Z-axis offset parameters to meet precise operation requirements for Sogot robotic arms.
Have Question?
If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.
1 post - 1 participant
https://discourse.openrobotics.org/t/isaac-lab-teleoperation-data-collection-2-0-automated-block-stacking-dataset-generation-with-agilex-piper-and-nero/55439
ROS Discourse General: Invitation to African Robotics Summit 2026 - June 25, 2026
Dear Friends and Colleagues,
On behalf of the African Robotics Network (AfROB), we are delighted to invite you to the AfROB Virtual Summit, taking place on Thursday, June 25, 2026, at 2:00 - 5:00 PM Central African Time (CAT).
This special gathering will mark the launch of our State of Robotics in Africa report and provide an opportunity for our community to reflect, connect, and contribute to shaping the future of robotics and AI across Africa.
Please find the flyer attached, and you are warmly welcome to register and learn more at afrob.org/summit.
Thank you.
1 post - 1 participant
https://discourse.openrobotics.org/t/invitation-to-african-robotics-summit-2026-june-25-2026/55432
ROS Discourse General: EpisodeVault: open source tool to debug why your LeRobot model regressed
Been building with LeRobot v3 and kept hitting the same wall: retrain a policy, it gets worse, no clear idea why. DVC tells you files changed. MLflow tells you which run. Nobody tells you which tasks dropped or which episodes degraded between dataset versions.
Built a small open source library to fill that gap.
Four commands
episodevault track ./my_dataset
episodevault commit -m "added kitchen episodes"
episodevault diff v1.0 v2.0
episodevault blame model_v3
What the diff looks like
Ran this against two real LeRobot datasets:
Dataset diff: v1.0 → v2.0
────────────────────────────────────────────────────
Episodes added: +0
Episodes removed: -7
Distribution shift:
factory_pick 2 → 6 ↑ 200% ⚠️
kitchen_grasp 4 → 1 ↓ 75% ⚠️
Quality metrics:
avg episode length: 3.7s → 3.0s ↓
success_rate: 0.88 → 0.38 ↓
camera_sync_score: 1.00 → 1.00 →
Likely regression cause:
'kitchen_grasp' episodes dropped 75% (4 → 1). Restore from prior
version if this task is in your eval benchmark.
The blame command
One line in your training script:
import episodevault as ev
ev.log_training_run(model_version="model_v3", dataset_version="v2.0")
Then later:
episodevault blame model_v3
Traces the model back to the exact dataset version that trained it and shows the diff automatically.
Compatibility
Tested against four real datasets from the Hub:
| Robot | Dataset | Episodes | Frames | Parse time |
|---|---|---|---|---|
| aloha | aloha_static_pro_pencil | 25 | 8,750 | 0.35s |
| aloha | aloha_mobile_shrimp | 18 | 67,500 | 0.38s |
| so100 | svla_so100_stacking | 56 | 22,956 | 0.63s |
| aloha | aloha_mobile_cabinet | 85 | 127,500 | 2.73s |
Install
pip install episodevault
Python 3.10+. Works on any local LeRobot v3 dataset.
GitHub: Rohan-Prabhakar/EpisodeVault
If you have a dataset where this breaks or gives a wrong regression hint, open an issue, that’s the most useful feedback right now.
2 posts - 1 participant
https://discourse.openrobotics.org/t/episodevault-open-source-tool-to-debug-why-your-lerobot-model-regressed/55401
ROS Discourse General: What's the most annoying part of your ROS 2 workflow?
Been spending more time in ROS 2 lately and getting a feel for the daily grind.
For folks working on real robots, what part of going from code change to seeing it run drives you the most nuts? And have you found anything that actually makes it better?
8 posts - 4 participants
https://discourse.openrobotics.org/t/whats-the-most-annoying-part-of-your-ros-2-workflow/55371
ROS Discourse General: Logging and Observability Guide Discussion | Cloud Robotics WG Meeting 2026-06-15
Please come and join us for this coming meeting at Mon, Jun 15, 2026 4:00 PM UTC→Mon, Jun 15, 2026 5:00 PM UTC, where we will be discussing Logging & Observability tips and best practices. The group has been researching in this area for the past several months, so we are preparing to assemble a guide, then invite previous guest speakers to review it before publishing.
Last session, we continued preparing our questions for the State of Cloud Robotics Survey 2026. This survey has now been released and is available here (all developers using the cloud in any way are invited to respond!), so once the survey has received a few more submissions, we will be analysing and discussing the results. If you’re interested in watching the previous session, the meeting recording is available on YouTube.
The meeting link for next meeting is here, and you can sign up to our calendar or our Google Group for meeting notifications or keep an eye on the Cloud Robotics Hub.
Hopefully we will see you there!
1 post - 1 participant
https://discourse.openrobotics.org/t/logging-and-observability-guide-discussion-cloud-robotics-wg-meeting-2026-06-15/55370
ROS Discourse General: Context & Hardware Setup
I am developing an Autonomous Mobile Robot (AMR) designed to operate inside a dairy cattle barn. The locomotion system suffers from high wheel slippage due to the environment’s muddy, uneven, and unpaved terrain.
My current sensor suite consists of:
-
Wheel Encoders (Differential drive odometry)
-
IMU (Providing angular velocity and absolute orientation)
-
Ultra-Wideband (UWB) Positioning System (1 Tag on the robot, multiple Anchors around the barn)
-
(Upcoming) Range sensors for obstacle avoidance (cattle detection).
For state estimation, I am using the robot_localization package running a Dual EKF setup to broadcast the map -> odom -> base_footprint transform tree.
The Problem
I am experiencing significant localization inaccuracies, specifically with the robot’s heading (yaw) and absolute positioning.
-
Heading / Yaw Errors: The estimated orientation is frequently wrong or drifts rapidly. The barn structure contains dense steel pillars (headlocks and support frames) which I suspect are causing massive magnetic disturbances to the IMU’s magnetometer.
-
UWB Inaccuracy & Jumps: Initially, the UWB update rate was 1 Hz, which caused huge position jumps. I managed to optimize the anchor-tag communication to achieve around 10 Hz. While the frequency issue is resolved, I still observe positional errors between 10 cm to 40 cm, which drastically degrade especially when the robot approaches the steel pillars or when cattle walk between the tag and the anchors.
Current Configuration & Attempts
-
The first EKF instance fuses wheel odometry and IMU (angular velocity) to output continuous
odom -> base_footprint. -
The second EKF instance fuses the
odomdata and absolute UWB positions to outputmap -> odom. -
I have noticed that high multipath reflections and Non-Line-of-Sight (NLOS) conditions caused by steel structures and livestock are heavily corrupting the UWB ranges.
Questions for the Community
Given the harsh environment (muddy terrain, heavy metal interference, dynamic obstacles like cows), I would highly appreciate any advice or architectural suggestions:
-
How should I handle the IMU in a high-metal environment? Should I completely disable the magnetometer fusion in
robot_localizationand rely solely ondifferential_driveodometry + IMU angular velocity (z-axis gyro) for heading, even if it drifts over time? -
How can I mitigate UWB multipath/NLOS errors in EKF? Are there proven outlier rejection methods or thresholding techniques within
robot_localization(like using thecovariancematrix dynamically ormahalanobis_distancefiltering) to ignore bad UWB fixes near the steel pillars? -
Alternative Fusion Strategies: Has anyone successfully implemented robust UWB-based localization in similar agricultural setups? Would moving towards an Unscented Kalman Filter (UKF) or a factor-graph-based optimization approach (like Ceres or GTSAM) yield better results than standard Dual EKF?
Thank you in advance for your insights!
My ekf configuration :
odometry_ekf_filter_node:
ros__parameters:
sensor_timeout: 0.1
# The frequency, in Hz, at which the filter will output a position estimate. Note that the filter will not begin
# computation until it receives at least one message from one of the inputs. It will then run continuously at the
# frequency specified here, regardless of whether it receives more measurements. Defaults to 30 if unspecified.
frequency: 30.0
permit_corrected_publication: false
# ekf_localization_node and ukf_localization_node both use a 3D omnidirectional motion model. If this parameter is
# set to true, no 3D information will be used in your state estimate. Use this if you are operating in a planar
# environment and want to ignore the effect of small variations in the ground plane that might otherwise be detected
# by, for example, an IMU. Defaults to false if unspecified.
two_d_mode: true
# Use this parameter to provide an offset to the transform generated by ekf_localization_node. This can be used for
# future dating the transform, which is required for interaction with some other packages. Defaults to 0.0 if
# unspecified.
transform_time_offset: 0.0
# Use this parameter to provide specify how long the tf listener should wait for a transform to become available.
# Defaults to 0.0 if unspecified.
transform_timeout: 0.05
# If you're having trouble, try setting this to true, and then echo the /diagnostics_agg topic to see if the node is
# unhappy with any settings or data.
print_diagnostics: false
# Whether to publish the acceleration state. Defaults to false if unspecified.
publish_acceleration: false
# Whether to broadcast the transformation over the /tf topic. Defaults to true if unspecified.
publish_tf: true
# initial_state: [2.0, -8.0, 0.0,
# 0.0, 0.0, 0.0,
# 0.0, 0.0, 0.0,
# 0.0, 0.0, 0.0,
# 0.0, 0.0, 0.0]
# 1. Set the map_frame, odom_frame, and base_link frames to the appropriate frame names for your system.
# 1a. If your system does not have a map_frame, just remove it, and make sure "world_frame" is set to the value of odom_frame.
# 2. If you are fusing continuous position data such as wheel encoder odometry, visual odometry, or IMU data, set "world_frame"
# to your odom_frame value. This is the default behavior for robot_localization's state estimation nodes.
# 3. If you are fusing global absolute position data that is subject to discrete jumps (e.g., GPS or position updates from landmark
# observations) then:
# 3a. Set your "world_frame" to your map_frame value
# 3b. MAKE SURE something else is generating the odom->base_link transform. Note that this can even be another state estimation node
# from robot_localization! However, that instance should *not* fuse the global data.
#map_frame: map # Defaults to "map" if unspecified
odom_frame: odom # Defaults to "odom" if unspecified
base_link_frame: base_footprint # Defaults to "base_link" if unspecified
world_frame: odom # Defaults to the value of odom_frame if unspecified
odom0: /wheel_odom
# x, y pozisyon differential modda fuse ediliyor.
# Twist (vx) kapalı - twist verisine güvenilmiyor.
# differential: true -> mutlak pose değil, adım adım delta alınıyor,
# böylece birikimli odometri hatası direkt EKF'e yansımaz.
odom0_config: [true, true, false,
false, false, false,
false, false, false,
false, false, false,
false, false, false]
odom0_differential: true
odom0_relative: false
odom0_queue_size: 10
odom0_pose_rejection_threshold: 5.0
odom0_twist_rejection_threshold: 5.0
imu0: /witmotion/imu
imu0_config: [false, false, false,
false, false, false, # absolute yaw KAPALI — motor EMI manyetometreyi bozuyor, gradual drift yaratıyor
false, false, false,
false, false, true, # yaw_rate (jiroskop) — SLAM map→odom global yaw'ı düzeltiyor
false, false, false]
imu0_differential: false
imu0_relative: false
imu0_queue_size: 10
# EMI koruması: 10.0'dan 3.0'a düşürüldü.
# Motorlardan kaynaklanan magnetik spike'lar bu eşiği aşarsa elenir.
# Eğer yaw sürekli reddediliyorsa (diagnostics'te görünür) değeri
# biraz artır (4.0-5.0) ya da imu0_config[5]'i false yap.
imu0_pose_rejection_threshold: 3.0
imu0_twist_rejection_threshold: 5.0
imu0_linear_acceleration_rejection_threshold: 0.8
imu0_remove_gravitational_acceleration: true
ekf_filter_node:
ros__parameters:
frequency: 10.0
sensor_timeout: 0.5
two_d_mode: true # Set false for 3D navigation (drones, etc.)
transform_time_offset: 0.0
transform_timeout: 0.0
print_diagnostics: true
debug: false
publish_tf: false
map_frame: map
odom_frame: odom
base_link_frame: base_footprint
world_frame: map
# [x, y, z, roll, pitch, yaw, vx, vy, vz, vroll, vpitch, vyaw, ax, ay, az]
# initial_state: [2.55, -7.4, 0.0,
# 0.0, 0.0, 0.0,
# 0.0, 0.0, 0.0,
# 0.0, 0.0, 0.0,
# 0.0, 0.0, 0.0]
# -----------------------------------------------------------------------
# Sensor 0: Wheel Odometry
# -----------------------------------------------------------------------
# Fuse: vx, vy, vyaw (velocities from wheel encoders)
# [x, y, z, roll, pitch, yaw, vx, vy, vz, vroll, vpitch, vyaw, ax, ay, az]
odom0: /wheel_odom
# x, y pozisyon differential modda fuse ediliyor.
# Twist (vx) kapalı - twist verisine güvenilmiyor.
# differential: true -> mutlak pose değil, adım adım delta alınıyor,
# böylece birikimli odometri hatası direkt EKF'e yansımaz.
odom0_config: [true, true, false,
false, false, false,
false, false, false,
false, false, false,
false, false, false]
odom0_differential: true
odom0_relative: false
odom0_queue_size: 10
odom0_pose_rejection_threshold: 5.0
odom0_twist_rejection_threshold: 5.0
imu0: /witmotion/imu
imu0_config: [false, false, false,
false, false, false, # absolute yaw KAPALI — motor EMI manyetometreyi bozuyor, gradual drift yaratıyor
false, false, false,
false, false, true, # yaw_rate (jiroskop) — SLAM map→odom global yaw'ı düzeltiyor
false, false, false]
imu0_differential: false
imu0_relative: false
imu0_queue_size: 10
# EMI koruması: 10.0'dan 3.0'a düşürüldü.
# Motorlardan kaynaklanan magnetik spike'lar bu eşiği aşarsa elenir.
# Eğer yaw sürekli reddediliyorsa (diagnostics'te görünür) değeri
# biraz artır (4.0-5.0) ya da imu0_config[5]'i false yap.
imu0_pose_rejection_threshold: 3.0
imu0_twist_rejection_threshold: 0.8
imu0_linear_acceleration_rejection_threshold: 0.8
imu0_remove_gravitational_acceleration: true
# -----------------------------------------------------------------------
# Sensor 1: UWB Trilateration Position
# -----------------------------------------------------------------------
# Fuse: x, y, z (absolute position from trilateration)
# [x, y, z, roll, pitch, yaw, vx, vy, vz, vroll, vpitch, vyaw, ax, ay, az]
pose0: /uwb/pose
pose0_config: [true, true, false,
false, false, false,
false, false, false,
false, false, false,
false, false, false]
pose0_differential: false
pose0_queue_size: 10
pose0_rejection_threshold: 1.0 # Reject UWB jumps > 2m (NLOS protection)
1 post - 1 participant
https://discourse.openrobotics.org/t/context-hardware-setup/55356
ROS Discourse General: Isaac Lab Teleoperation & Data Collection: Controlling the AgileX NERO 7-DoF Robotic Arm
0. Preface
In this tutorial, we walk through the step-by-step implementation of keyboard teleoperation and demonstration data collection for the AgileX NERO (7-DoF) and PiPER robotic arms within NVIDIA Isaac Lab.
By the end of this guide, you will be able to control the robotic arm in a simulated environment to perform a block-stacking (Pick and Place) task, collect human demonstration data (HDF5), and replay the dataset for imitation learning pipeline verification.
Key Capabilities Supported
-
Multimodal Teleoperation: Direct keyboard control for end-effector SE(3) pose alignment.
-
Pick-and-Place (Stacking) Task: Multi-block stacking simulation environment.
-
Demonstration Collection: Seamless HDF5 data logging compatible with Robomimic.
-
Trajectory Replay: Instant validation of collected human demonstrations.
-
Cross-Hardware Support: Out-of-the-box support for AgileX NERO (7-DoF) and PiPER (6-DoF) arms.
Project Open-Source Repo
1.Environment & Hardware Configuration
1.1 Generate the Unified URDF File
To import the NERO arm into Omniverse, we must first merge its separate XACRO description files into a single, clean URDF.
Navigate to your local repository directory:
source/agx_teleop/agx_description/agx_arm_urdf/nero
Merge nero_description.urdf, nero_with_gripper_flange_description.xacro, and nero_with_gripper_description.xacro into a single file named nero_gripper.urdf. Ensure all visual/collision mesh paths pointing to local ROS packages are resolved to relative directory paths (e.g., ../meshes/).
Here is the complete nero_gripper.urdf:
<?xml version="1.0" ?>
<robot name="agx_arm">
<link name="world"/>
<joint name="world_to_base_link" type="fixed">
<parent link="world"/>
<child link="base_link"/>
<origin rpy="0 0 3.14" xyz="0 0 0"/>
</joint>
<link name="base_link">
<inertial>
<origin rpy="0 0 0" xyz="-0.00319465997 -0.00005467608 0.04321758463"/>
<mass value="1.06458435"/>
<inertia ixx="0.00102659855152" ixy="0.00000186219753" ixz="-0.00000295298037" iyy="0.00114399299508" iyz="-0.00000078763492" izz="0.00090872933022"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/base_link.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/base_link.stl"/>
</geometry>
</collision>
</link>
<link name="link1">
<inertial>
<origin rpy="0 0 0" xyz="-0.00076810578 -0.00005718031 -0.00686372765"/>
<mass value="0.76710504"/>
<inertia ixx="0.00066689512233" ixy="0.00000273938306" ixz="0.00001463284987" iyy="0.00054501235592" iyz="0.00000183168344" izz="0.00051509465737"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link1.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link1.stl"/>
</geometry>
</collision>
</link>
<joint name="joint1" type="revolute">
<origin rpy="0 0 0" xyz="0 0 0.138"/>
<parent link="base_link"/>
<child link="link1"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-2.70526" upper="2.70526" velocity="5"/>
</joint>
<link name="link2">
<inertial>
<origin rpy="0 0 0" xyz="0.00079686657 -0.07890325930 0.00111849422"/>
<mass value="0.69381908"/>
<inertia ixx="0.00201093857656" ixy="-0.00002399426785" ixz="-0.00000072824714" iyy="0.00063544232555" iyz="0.00005831284343" izz="0.00181943989593"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link2.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link2.stl"/>
</geometry>
</collision>
</link>
<joint name="joint2" type="revolute">
<origin rpy="1.5707963 3.1415926 0" xyz="0 0 0"/>
<parent link="link1"/>
<child link="link2"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-1.74" upper="1.74" velocity="5"/>
</joint>
<link name="link3">
<inertial>
<origin rpy="0 0 0" xyz="0.00014513363 -0.00121378766 -0.04861136795"/>
<mass value="0.67246544"/>
<inertia ixx="0.00343229961910" ixy="-0.00000026976739" ixz="0.00000305078107" iyy="0.00338662526897" iyz="-0.00003550220027" izz="0.00028853579710"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link3.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link3.stl"/>
</geometry>
</collision>
</link>
<joint name="joint3" type="revolute">
<origin rpy="-1.5707963 0 3.1415926926" xyz="0 -0.31 0"/>
<parent link="link2"/>
<child link="link3"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-2.75" upper="2.75" velocity="5"/>
</joint>
<link name="link4">
<inertial>
<origin rpy="0 0 0" xyz="-0.00031334623 -0.05675192422 0.00076056133"/>
<mass value="0.48451414"/>
<inertia ixx="0.00050386693505" ixy="-0.00000529675237" ixz="0.00000006134740" iyy="0.00023502068202" iyz="0.00001815932566" izz="0.00040149523118"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link4.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link4.stl"/>
</geometry>
</collision>
</link>
<joint name="joint4" type="revolute">
<origin rpy="-1.5707963 0 3.1415926926" xyz="0 0 0"/>
<parent link="link3"/>
<child link="link4"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-1.01" upper="2.14" velocity="5"/>
</joint>
<link name="link5">
<inertial>
<origin rpy="0 0 0" xyz="0.00265591357 0.00201195753 -0.11620855434"/>
<mass value="0.58368405"/>
<inertia ixx="0.00089537686723" ixy="0.00000648218232" ixz="-0.00001463346849" iyy="0.00085866122972" iyz="-0.00004035185637" izz="0.00021774946532"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link5.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link5.stl"/>
</geometry>
</collision>
</link>
<joint name="joint5" type="revolute">
<origin rpy="1.5707963 -1.5707963 0" xyz="0 -0.27001 0"/>
<parent link="link4"/>
<child link="link5"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-2.75" upper="2.75" velocity="5"/>
</joint>
<link name="link6">
<inertial>
<origin rpy="0 0 0" xyz="0.00009501505 0.00107831174 -0.00019110990"/>
<mass value="0.3140406"/>
<inertia ixx="0.00006027211661" ixy="-0.00000024650903" ixz="0.00000015701294" iyy="0.00004699514792" iyz="0.00000004170875" izz="0.00005756771792"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link6.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link6.stl"/>
</geometry>
</collision>
</link>
<joint name="joint6" type="revolute">
<origin rpy="1.5707963 -1.5707963 0" xyz="0 0 0"/>
<parent link="link5"/>
<child link="link6"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-0.73" upper="0.95" velocity="5"/>
</joint>
<link name="link7">
<inertial>
<origin rpy="0 0 0" xyz="-0.00014000 -0.00010000 -0.00275000"/>
<mass value="0.000001"/>
<inertia ixx="1.1000000000000001e-07" ixy="0" ixz="0" iyy="1.1000000000000001e-07" iyz="0" izz="1.9999999999999999e-07"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link7.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/link7.stl"/>
</geometry>
</collision>
</link>
<joint name="joint7" type="revolute">
<origin rpy="1.5707963 0 0" xyz="0 -0.0235 0"/>
<parent link="link6"/>
<child link="link7"/>
<axis xyz="0 0 1"/>
<limit effort="100" lower="-1.5707963" upper="1.5707963" velocity="5"/>
</joint>
<link name="gripper_flange">
<inertial>
<origin rpy="0 0 0" xyz="0.0 0.0 -0.00032"/>
<mass value="0.04771096"/>
<inertia ixx="2.697e-05" ixy="0" ixz="0" iyy="4.311e-05" iyz="0" izz="2.479e-05"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_flange.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_flange.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_flange_joint" type="fixed">
<origin rpy="-1.5708 0 -1.5708" xyz="0.032 0 -0.0235"/>
<parent link="link7"/>
<child link="gripper_flange"/>
</joint>
<link name="gripper_base">
<inertial>
<origin rpy="0 0 0" xyz="-0.000183807162235591 8.05033155577911E-05 0.0321436689908876"/>
<mass value="0.45"/>
<inertia ixx="0.00092934" ixy="0.00000034" ixz="-0.00000738" iyy="0.00071447" iyz="0.00000005" izz="0.00039442"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_base.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_base.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_base_joint" type="fixed">
<origin rpy="0 0 0" xyz="0 0 0.0055"/>
<parent link="gripper_flange"/>
<child link="gripper_base"/>
</joint>
<link name="gripper_link1">
<inertial>
<origin rpy="0 0 0" xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025"/>
<mass value="0.025"/>
<inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_link1.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_link1.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_joint1" type="prismatic">
<origin rpy="1.5707963 0 3.1415926" xyz="0 0 0.1358"/>
<parent link="gripper_base"/>
<child link="gripper_link1"/>
<axis xyz="0 0 1"/>
<limit effort="10" lower="0" upper="0.05" velocity="3"/>
</joint>
<link name="gripper_link2">
<inertial>
<origin rpy="0 0 0" xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025"/>
<mass value="0.025"/>
<inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
</inertial>
<visual>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_link2.dae"/>
</geometry>
</visual>
<collision>
<origin rpy="0 0 0" xyz="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_link2.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_joint2" type="prismatic">
<origin rpy="1.5707963 0 0" xyz="0 0 0.1358"/>
<parent link="gripper_base"/>
<child link="gripper_link2"/>
<axis xyz="0 0 -1"/>
<limit effort="10" lower="-0.05" upper="0" velocity="3"/>
</joint>
</robot>
1.2 Compile URDF to USD Format
Convert the generated URDF into an NVIDIA USD (Universal Scene Description) file using Isaac Lab’s built-in exporter utility:
python ../IsaacLab/scripts/tools/convert_urdf.py
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/nero/urdf/nero_gripper.urdf
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/nero/usd/nero.usd
--fix-base
1.3 Create the Asset Configuration File
Create assets/nero.py to define the simulated joints, limits, PD controller gains, and stiffness/damping coefficients.
import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets.articulation import ArticulationCfg
from agx_teleop.assets import AGX_TELEOP_DESCRIPTION_DIR
NERO_GRIPPER_CFG = ArticulationCfg(
spawn=sim_utils.UsdFileCfg(
usd_path=f"{AGX_TELEOP_DESCRIPTION_DIR}/agx_arm_urdf/nero/usd/nero.usd",
rigid_props=sim_utils.RigidBodyPropertiesCfg(
disable_gravity=False,
max_depenetration_velocity=5.0,
),
articulation_props=sim_utils.ArticulationRootPropertiesCfg(
enabled_self_collisions=False, solver_position_iteration_count=8, solver_velocity_iteration_count=0
),
),
init_state=ArticulationCfg.InitialStateCfg(
rot=(1.0, 0.0, 0.0, 0.0),
joint_pos={
"joint1": 0.0,
"joint2": 0.0,
"joint3": 0.0,
"joint4": 1.22,
"joint5": 0.0,
"joint6": 0.0,
"joint7": 1.31,
"gripper_joint1": 0.05,
"gripper_joint2": -0.05
},
joint_vel={".*": 0.0},
),
actuators={
"arm": ImplicitActuatorCfg(
joint_names_expr=["joint.*"],
effort_limit=25.0,
velocity_limit=1.5,
stiffness={
"joint1": 200.0,
"joint2": 170.0,
"joint3": 120.0,
"joint4": 80.0,
"joint5": 50.0,
"joint6": 20.0,
"joint7": 10.0
},
damping={
"joint1": 100.0,
"joint2": 60.0,
"joint3": 70.0,
"joint4": 24.0,
"joint5": 20.0,
"joint6": 10.0,
"joint7": 5,
},
),
"gripper": ImplicitActuatorCfg(
joint_names_expr=\["gripper_joint1","gripper_joint2"\],
effort_limit_sim=22,
velocity_limit_sim=1.5,
stiffness=800.0,
damping=20.0,
),
},
soft_joint_pos_limit_factor=0.9,
)
NERO_GRIPPER_HIGH_PD_CFG = NERO_GRIPPER_CFG.copy()
NERO_GRIPPER_HIGH_PD_CFG.spawn.rigid_props.disable_gravity = True
2.Task Environment Configuration
2.1 Configure Directory Tree Structure
Setup your task configurations under the standard Isaac Lab layout:
Adapted from the official Isaac Lab stacking task implementation: isaaclab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack
cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/config/nero/agents/robomimic
Create the required task files:
cd source/agx_teleop/agx_teleop/tasks/manager_based/
touch manipulation/stack/config/nero/__init__.py
manipulation/stack/config/nero/agents/__init__.py
manipulation/stack/config/nero/agents/robomimic/bc_rnn_low_dim.json
manipulation/stack/config/nero/nero_stack_ik_rel_env_cfg.py
manipulation/stack/config/nero/nero_stack_joint_pos_env_cfg.py
3. Implement Configurations
3.1 bc_rnn_low_dim.json (Robomimic Behavior Cloning Profile)
This file defines policy network structures (LSTM with Gaussian Mixture Models) for downstream imitation learning. Copy your target JSON config as-is without extra modifications:
{
"algo_name": "bc",
"experiment": {
"name": "bc_rnn_low_dim_franka_stack",
"validate": false,
"logging": { "terminal_output_to_txt": true, "log_tb": true },
"save": {
"enabled": true,
"every_n_seconds": null,
"every_n_epochs": 100,
"epochs": \[\],
"on_best_validation": false,
"on_best_rollout_return": false,
"on_best_rollout_success_rate": true
},
"epoch_every_n_steps": 100,
"env": null,
"additional_envs": null,
"render": false,
"render_video": false,
"rollout": { "enabled": false }
},
"train": {
"data": null,
"num_data_workers": 4,
"hdf5_cache_mode": "all",
"hdf5_use_swmr": true,
"hdf5_normalize_obs": false,
"hdf5_filter_key": null,
"hdf5_validation_filter_key": null,
"seq_length": 10,
"dataset_keys": [ "actions" ],
"goal_mode": null,
"cuda": true,
"batch_size": 100,
"num_epochs": 2000,
"seed": 101
},
"algo": {
"optim_params": {
"policy": {
"optimizer_type": "adam",
"learning_rate": {
"initial": 0.001,
"decay_factor": 0.1,
"epoch_schedule": [],
"scheduler_type": "multistep"
},
"regularization": { "L2": 0.0 }
}
},
"loss": { "l2_weight": 1.0, "l1_weight": 0.0, "cos_weight": 0.0 },
"actor_layer_dims": [],
"gmm": {
"enabled": true,
"num_modes": 5,
"min_std": 0.0001,
"std_activation": "softplus",
"low_noise_eval": true
},
"rnn": {
"enabled": true,
"horizon": 10,
"hidden_dim": 400,
"rnn_type": "LSTM",
"num_layers": 2,
"open_loop": false,
"kwargs": { "bidirectional": false }
}
},
"observation": {
"modalities": {
"obs": {
"low_dim": [ "eef_pos", "eef_quat", "gripper_pos", "object" ],
"rgb": [], "depth": [], "scan": []
}
}
}
}
3.2 nero_stack_joint_pos_env_cfg.py (Joint Position Environment)
Sets up state configurations, randomizing object spawn spaces and assigning physical bounds to NERO.
from isaaclab.assets import RigidObjectCfg
from isaaclab.managers import EventTermCfg as EventTerm
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformerCfg
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg
from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
from agx_teleop.tasks.manager_based.manipulation.stack import mdp
from agx_teleop.tasks.manager_based.manipulation.stack.mdp import piper_stack_events
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import StackEnvCfg
from isaaclab.markers.config import FRAME_MARKER_CFG
from agx_teleop.assets.nero import NERO_GRIPPER_CFG
@configclass
class EventCfg:
init_nero_arm_pose = EventTerm(
func=piper_stack_events.set_default_joint_pose,
mode="reset",
params={
"default_pose": [0.0, 0.0, 0.0, 1.22, 0.0, 0.0, 1.31, 0.05, -0.05],
},
)
randomize_nero_joint_state = EventTerm(
func=piper_stack_events.randomize_joint_by_gaussian_offset,
mode="reset",
params={
"mean": 0.0,
"std": 0.02,
"asset_cfg": SceneEntityCfg("robot"),
},
)
randomize_cube_positions = EventTerm(
func=piper_stack_events.randomize_object_pose,
mode="reset",
params={
"pose_range": {"x": (0.3, 0.5), "y": (-0.10, 0.10), "z": (0.0203, 0.0203), "yaw": (-1.0, 1, 0)},
"min_separation": 0.12,
"asset_cfgs": [SceneEntityCfg("cube_1"), SceneEntityCfg("cube_2"), SceneEntityCfg("cube_3")],
},
)
@configclass
class NeroCubeStackEnvCfg(StackEnvCfg):
def __post_init__(self):
super().__post_init__()
self.events = EventCfg()
self.scene.robot = NERO_GRIPPER_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
self.scene.robot.spawn.semantic_tags = [("class", "robot")]
self.scene.table.spawn.semantic_tags = [("class", "table")]
self.scene.plane.semantic_tags = [("class", "ground")]
self.actions.arm_action = mdp.JointPositionActionCfg(
asset_name="robot", joint_names=["joint.*"], scale=0.5, use_default_offset=True
)
self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
asset_name="robot",
joint_names=["gripper_joint.*"],
open_command_expr={"gripper_joint1": 0.05, "gripper_joint2": -0.05},
close_command_expr={"gripper_joint1": 0.0, "gripper_joint2": 0.0},
)
self.gripper_joint_names = ["gripper_joint.*"]
self.gripper_open_val = [0.05, -0.05]
self.gripper_threshold = 0.005
cube_properties = RigidBodyPropertiesCfg(
solver_position_iteration_count=16,
solver_velocity_iteration_count=1,
max_angular_velocity=1000.0,
max_linear_velocity=1000.0,
max_depenetration_velocity=5.0,
disable_gravity=False,
)
self.scene.cube_1 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_1",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0.0, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/blue_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_1")],
),
)
self.scene.cube_2 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_2",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.55, 0.05, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/red_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_2")],
),
)
self.scene.cube_3 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_3",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.60, -0.1, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/green_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_3")],
),
)
marker_cfg = FRAME_MARKER_CFG.copy()
marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
marker_cfg.prim_path = "/Visuals/FrameTransformer"
self.scene.ee_frame = FrameTransformerCfg(
prim_path="{ENV_REGEX_NS}/Robot/world",
debug_vis=False,
visualizer_cfg=marker_cfg,
target_frames=[
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_base",
name="end_effector",
offset=OffsetCfg(pos=[0.0, 0.0, 0.125\]),
),
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_link1",
name="tool_rightfinger",
offset=OffsetCfg(pos=(0.0, 0.0, 0.0)),
),
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_link2",
name="tool_leftfinger",
offset=OffsetCfg(pos=(0.0, 0.0, 0.0)),
),
],
)
3.3 Teleoperation, Data Collection & Replay
3.3.1 stack/config/nero/nero_stack_ik_rel_env_cfg.py
Copy the corresponding Isaac Lab code and modify the following:
-
Update imports and naming
-
Modify the end-effector offset
Complete code for nero_stack_ik_rel_env_cfg.py:
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from isaaclab.controllers.differential_ik_cfg import DifferentialIKControllerCfg
from isaaclab.devices.device_base import DeviceBase, DevicesCfg
from isaaclab.devices.keyboard import Se3KeyboardCfg
from isaaclab.devices.openxr.openxr_device import OpenXRDeviceCfg
from isaaclab.devices.openxr.retargeters.manipulator.gripper_retargeter import GripperRetargeterCfg
from isaaclab.devices.openxr.retargeters.manipulator.se3_rel_retargeter import Se3RelRetargeterCfg
from isaaclab.envs.mdp.actions.actions_cfg import DifferentialInverseKinematicsActionCfg
from isaaclab.managers import SceneEntityCfg
from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.utils import configclass
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import mdp
from . import nero_stack_joint_pos_env_cfg
##
# Pre-defined configs
##
from agx_teleop.assets.nero import NERO_GRIPPER_HIGH_PD_CFG # isort: skip
@configclass
class NeroCubeStackEnvCfg(nero_stack_joint_pos_env_cfg.NeroCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
# Set Nero as robot
# We switch here to a stiffer PD controller for IK tracking to be better.
self.scene.robot = NERO_GRIPPER_HIGH_PD_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
# Set actions for the specific robot type (nero)
self.actions.arm_action = DifferentialInverseKinematicsActionCfg(
asset_name="robot",
joint_names=["joint.*"],
body_name="gripper_base",
controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),
scale=0.5,
body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.125]),
)
self.teleop_devices = DevicesCfg(
devices={
"handtracking": OpenXRDeviceCfg(
retargeters=[
Se3RelRetargeterCfg(
bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT,
zero_out_xy_rotation=True,
use_wrist_rotation=False,
use_wrist_position=True,
delta_pos_scale_factor=10.0,
delta_rot_scale_factor=10.0,
sim_device=self.sim.device,
),
GripperRetargeterCfg(
bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT, sim_device=self.sim.device
),
],
sim_device=self.sim.device,
xr_cfg=self.xr,
),
"keyboard": Se3KeyboardCfg(
pos_sensitivity=0.05,
rot_sensitivity=0.05,
sim_device=self.sim.device,
),
}
)
@configclass
class NeroCubeStackRedGreenEnvCfg(NeroCubeStackEnvCfg):
def __post_init__(self):
\# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={"cube_1_cfg": SceneEntityCfg("cube_2"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
)
@configclass
class NeroCubeStackRedGreenBlueEnvCfg(NeroCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={
"cube_1_cfg": SceneEntityCfg("cube_2"),
"cube_2_cfg": SceneEntityCfg("cube_3"),
"cube_3_cfg": SceneEntityCfg("cube_1"),
},
)
@configclass
class NeroCubeStackBlueGreenEnvCfg(NeroCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={"cube_1_cfg": SceneEntityCfg("cube_1"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
)
@configclass
class NeroCubeStackBlueGreenRedEnvCfg(NeroCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={
"cube_1_cfg": SceneEntityCfg("cube_1"),
"cube_2_cfg": SceneEntityCfg("cube_3"),
"cube_3_cfg": SceneEntityCfg("cube_2"),
},
)
3.3.2 stack/config/nero/init.py**
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause
import gymnasium as gym
from isaaclab_tasks.manager_based.manipulation.stack.config.franka import agents
from agx_teleop.tasks.manager_based.manipulation.stack.config.nero import agents
\##
# Register Gym environments.
##
gym.register(
id="Isaac-Stack-Cube-Nero-IK-Rel-v0",
entry_point="isaaclab.envs:ManagerBasedRLEnv",
kwargs={
"env_cfg_entry_point": f"{__name__}.nero_stack_ik_rel_env_cfg:NeroCubeStackEnvCfg",
"robomimic_bc_cfg_entry_point": f"{agents.__name__}:robomimic/bc_rnn_low_dim.json",
},
disable_env_checker=True,
)
4. Collect Demonstration Data via Teleoperation
# First, create a dataset directory
mkdir -p datasets
# Record demonstrations via teleoperation.
# This example uses keyboard teleoperation.
# Available devices: spacemouse, keyboard, handtracking
python scripts/tools/record_demos.py
--task Isaac-Stack-Cube-Nero-IK-Rel-v0
--device cpu --teleop_device keyboard
--dataset_file ./datasets/nero_dataset.hdf5
--num_demos 10
# Replay the recorded demonstrations
python scripts/tools/replay_demos.py
--task Isaac-Stack-Cube-Nero-IK-Rel-v0
--device cpu --dataset_file ./datasets/nero_dataset.hdf5
Keyboard teleoperation controls:
Keyboard Controller for SE(3): Se3Keyboard
Reset all commands: R
Toggle gripper (open/close): K
Move arm along x-axis: W/S
Move arm along y-axis: A/D
Move arm along z-axis: Q/E
Rotate arm along x-axis: Z/X
Rotate arm along y-axis: T/G
Rotate arm along z-axis: C/V
FAQ
Q1: How do I adapt this pipeline from the 7-DoF NERO arm to the 6-DoF PiPER arm (or any other custom manipulator)?
To swap the robot asset, you need to update three core configuration files in your task directory:
1.Asset Definition (assets/piper.py): Define the new USD file path, update the default joint positions (joint_pos) to match a 6-DoF configuration, and re-tune the stiffness/damping parameters for the 6 active joints.
2.Action Configuration (piper_stack_joint_pos_env_cfg.py): Modify the joint regular expression to match the PiPER joint names (e.g., changing expression targets from 7 joints to 6). Update the gripper joint names and binary open/close limit values.
3.Differential IK Config (piper_stack_ik_rel_env_cfg.py):Update the body_name parameter to point to the correct end-effector link (e.g., link6 or gripper_base) and adjust the body_offset to match the physical dimension of the PiPER gripper flange.
Q2: The robotic gripper slips or fails to lift the blocks during teleoperation. How do I improve grap stability?
Slipping or unstable grasping in Isaac Lab is usually a physics-tuning issue rather than an algorithmic one. Try the following adjustments:
-
Increase Gripper Stiffness: In your asset configuration (assets/nero.py or piper.py), increase the gripper actuator stiffness (e.g., to 800.0 or 1000.0) and effort_limit to ensure the gripper applies enough physical force.
-
Verify Collision Geometries: Open your converted .usd asset in Isaac Sim and check that the collision meshes for the gripper fingers are set to “Convex Decomposition” (or Simplfied Box) rather than a loose bounding box.
-
Adjust Material Friction: Add a high-friction physics material (e.g., static friction coefficient.
Q3: Are the collected HDF5 datasets directly compatible with popular imitation learning codebases like ACT or Diffusion Policy?
Yes, with minor formatting adjustments. The HDF5 files generated by our record_demos.py script closely follow the standard Robomimic data structure, containing hierarchically organized datasets:
-
data/demo_x/obs/eef_pos & eef_quat (End-effector pose)
-
data/demo_x/obs/joint_values (Joint angles)
-
data/demo_x/actions (User control inputs)
To feed this data into ACT (Action Chunking with Transformers) or Diffusion Policy, you can use Robomimic’s conversion scripts (convert_robomimic_to_numpy.py) or write a simple Python wrapper to map the HDF5 dictionary keys to the observation/action expectations of your specific network.
Q4: The simulation occasionally crashes or the robot “explodes” (joints jitter violently) when starting teleoperation. How do I fix this?
“Exploding” physics usually happen when the control gains (PD) are too aggressive, or the solver parameters are too coarse.
- Increase Solver Iterations: In your environment configuration, increase the solver iterations on the robot asset:
articulation_props=sim_utils.ArticulationRootPropertiesCfg(
solver_position_iteration_count=16, # Increase from 8 to 16
solver_velocity_iteration_count=4 # Increase from 0 to 4
)
-
Disable Self-Collisions: Ensure that adjacent links on the arm do not have overlapping collision meshes by setting enabled_self_collisions=False.
-
Lower Stiffness: If the arm vibrates violently upon contact, slightly lower the joint stiffness in your actuator configuration and add proportional damping (ideal ratio is roughly 10:1 to 20:1 for stiffness to damping).
1 post - 1 participant
https://discourse.openrobotics.org/t/isaac-lab-teleoperation-data-collection-controlling-the-agilex-nero-7-dof-robotic-arm/55348
ROS Discourse General: Pre-built ROS2 Humble Windows SDK + Standalone RViz2 (No Toolchain Required)
# Pre-built ROS2 Humble Windows SDK + Standalone RViz2 (No Toolchain Required)
Hi everyone,
I’d like to share two projects that may help developers working with ROS2 on Windows.
## The Problem
Compiling ROS2 on Windows has always been painful — it requires a complex toolchain setup (`rosdep`, `colcon`, Python dependencies, environment variables, etc.). Many developers simply give up or switch to WSL2, which has its own limitations for hardware access and GUI applications.
## Ros2Simple — Pre-built ROS2 Humble Windows SDK
A pre-compiled ROS2 Humble SDK for Windows x64 that allows you to build ROS2 applications using **standard CMake** — no ROS2 native toolchain, no `colcon`, no `rosdep`.
### Key features:
- Based on **ROS2 Humble**, Windows x64 Debug build
- Build with **CMake directly** — just point to the SDK path
- No ROS2 toolchain installation required
- Can be **cross-compiled to Linux and ARM-Linux** for industrial deployment
- Includes a working publisher/subscriber demo
### Quick build:
```bash
cd demo && mkdir build && cd build
cmake ..
cmake --build . --config Debug
demo_pubsub.exe pub
```
## Rviz2Windows — Standalone RViz2 for Windows
A standalone RViz2 that runs on Windows by **double-clicking `rviz2.exe`** — no ROS2 installation, no environment configuration, no dependencies to install.
### Key features:
- Built from **ROS2 Humble** source
- **Zero configuration** — double-click to run
- All dependencies bundled (Qt5, Ogre3D, plugins)
- Works with ROS2 nodes built from Ros2Simple or any DDS-compatible node
### Demo screenshots:
**Publisher / Subscriber running on Windows:**
**RViz2 running standalone on Windows:**
## Use Cases
- **Industrial deployment**: Build ROS2 applications on Windows-based industrial PCs
- **Education**: Students can start learning ROS2 without struggling with environment setup
- **Rapid prototyping**: Quick iteration without the overhead of the full ROS2 toolchain
- **Cross-platform products**: Compile the same codebase to Linux / ARM-Linux for production
## License
Both projects are based on ROS2 Humble source code and comply with the original ROS2 licenses (Apache License 2.0 / BSD License). All third-party libraries retain their original licenses.
## Feedback Welcome
I’d love to hear your feedback, suggestions, or issues. Feel free to open GitHub issues or reach out.
If you have commercial needs (custom builds, consulting, industrial deployment, training), you can contact me at: **huming516520@gmail.com**
Thanks!
1 post - 1 participant
https://discourse.openrobotics.org/t/pre-built-ros2-humble-windows-sdk-standalone-rviz2-no-toolchain-required/55340
ROS Discourse General: CS student diving into ROS 2 and Gazebo, looking for feedback and community
Hello everyone, I’m new here and wanted to introduce myself and share my current focus.
I’m currently finishing up my CS degree and along the way stumbled into robotics software and simulation. I recently installed Ubuntu 24.04, added ROS2 Jazzy, and Gazebo Harmonic, and I’m teaching myself Gazebo from scratch to start running physics and navigation simulations. I kind of stumbled on this middleware from just tinkering in linux, and this field is just fascinating to me!
Since I don’t have a physical hardware lab right now, my work is 100% focused on software, coding, and mastering these simulation environments. I am treating my desktop as my entire proving ground.
I’m the only one in my current physical circle working on this stack, so I’m looking for a community to bounce ideas off of, share my progress, and learn from people who actually know what they’re looking at.
Looking forward to learning from you all. Any pointers on avoiding rookie mistakes with ROS 2 or Gazebo would be heavily appreciated, and maybe some tips on useful tools or concepts to learn.
3 posts - 2 participants
https://discourse.openrobotics.org/t/cs-student-diving-into-ros-2-and-gazebo-looking-for-feedback-and-community/55335
ROS Discourse General: Building a Rust tool for zero-allocation Serial Autobauding & Protocol Detection
Problem:
Integrating unknown hardware into the companion computer (a custom 1D LiDAR for altitude hold, telemetry radio from decades ago, and a mysterious motor controller) involves lots of frustration trying to find the baud rate manually. The autobaud solutions that I have come across rely on human-readable ASCII and are therefore of no use in determining the baud rate of binary streams.
Solution (autobaud-core):
I’m developing a cross-platform Rust-based autobaud detector as a library and CLI daemon that can be called programmatically to detect baud rates without user interaction.
As opposed to cycling through possible rates blindfolded, my solution involves a two-tier strategy based on:
- Hardware Framing Validation: For Linux systems, I use the nix crate to access termios PARMRK flags. In case the baud rate is incorrect, the software detects it almost immediately at the hardware framing level (the electrical framing is faulty), and it does not waste CPU cycles running the loop.
- Zero-Allocation Matching Heuristics: With the electrical framing established, it proceeds to examine the raw &[u8] byte buffer for protocol-specific “magic” bytes (MAVLink v2 = 0xFD, TFmini LiDAR = 0x59 0x59, NMEA = $GPGGA).
I have used ROS2 myself but I am a beginner but could come up with this idea from my experience with working with drones and sensor fusion
Questions from me:
- Validation: Would such a CLI utility be a real-world thing used when adding new physical devices?
- ROS 2 Integration: I would like to see some integration beyond a command-line utility here. Could we wrap this into a ROS 2 node, e.g., “Dynamic Serial Bridge” that autobauds the connection, determines the protocol, and immediately starts publishing the raw binary data on a certain topic for parsing further?
- Protocols missing in action: Besides the aforementioned MAVLink, NMEA, ASCII protocols, what other raw serial protocols are common when interfacing between hardware and ROS?
3 posts - 2 participants
https://discourse.openrobotics.org/t/building-a-rust-tool-for-zero-allocation-serial-autobauding-protocol-detection/55228
ROS Discourse General: 2026 State of Cloud Robotics Survey
Hello, robotics developers!
If you’re using cloud robotics at all, we (the Cloud Robotics Working Group) would appreciate you taking the time to fill out a survey.
We’re gathering data on how cloud robotics is used, what it’s doing well, and what still needs improving.
The survey link is here: https://forms.gle/WxXiRcCkRxxJLUop9
Responses will be published anonymously. Thank you!
1 post - 1 participant
https://discourse.openrobotics.org/t/2026-state-of-cloud-robotics-survey/55226
ROS Discourse General: Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI
Isaac Lab Teleoperation and Data Collection Workflow | Piper Robotic Arm for Embodied AI
0.Preface
This project provides an Isaac Lab-based teleoperation and demonstration collection workflow for the AgileX Piper robotic arm. It is designed for developers working on Embodied AI, robot learning, and imitation learning pipelines using simulation environments.
The workflow supports:
- keyboard-based robot teleoperation
- cube stacking manipulation tasks
- trajectory recording and replay
- imitation learning dataset generation
- ROS2-compatible robotic development
- Isaac Lab manipulation environments
Project Has Already Completed
- employment of the Piper robotic arm inside NVIDIA Isaac Lab
- keyboard teleoperation for robot manipulation
- cube stacking task environment configuration
- URDF-to-USD robot asset conversion
- gripper and actuator parameter configuration
- observation pipeline customization for Piper grippers
- human demonstration data collection workflow
- Robomimic-compatible imitation learning configuration
- replay support for collected robot trajectories
Recommended Environment
- Operating System: Ubuntu 22.04 or compatible Linux distribution
- Simulation Framework: NVIDIA Isaac Lab
- ROS Version: ROS 2 Humble
- Python Version: Python 3.10+
Reference & Resource
-
Official IsaacLab Tutorial: Welcome to Isaac Lab! — Isaac Lab Documentation
-
IsaacLab: GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub)
-
Project Repository: GitHub - szyzp/IsaacLab_Data_Collection · GitHub
The following section provides how developers can rapidly build a complete robotic manipulation and data collection workflow, steps are shown below:
Pipeline Overview
Keyboard Input
↓
Robot Teleoperation
↓
Isaac Lab Simulation
↓
Trajectory Recording
↓
Dataset Generation
↓
Imitation Learning / RL Training
Step 1.Create an Isaac Lab External Project
Before building the teleoperation and data collection workflow, first create an external Isaac Lab project environment.
1.1 Create a New Isaac Lab External Project
1.1.1 Initialize the Project
Run the following command inside the Isaac Lab root directory.
Make sure Isaac Lab has already been installed and the isaaclab Conda environment is activated.
conda activate isaaclab
./isaaclab.sh --new
1.1.2 Recommended Project Configuration
| Setting | Recommended Value |
|---|---|
| Project Type | External |
| Project Path | /home/agilex/isaac-cosmos/ |
| Project Name | agx_teleop |
| Workflow | Manager-based|single-agent |
| RL Library | optional |
| Project Path | /home/agilex/isaac-cosmos/ |
Configuration are shown above
The initial project structure of agx_teleop is as follows:
├── pyproject.toml
├── README.md
├── scripts
│ ├── list_envs.py
│ ├── random_agent.py
│ ├── sb3
│ │ ├── play.py
│ │ └── train.py
│ └── zero_agent.py
└── source
└── agx_teleop
├── agx_teleop
│ ├── __init__.py
│ ├── tasks
│ │ ├── __init__.py
│ │ └── manager_based
│ │ ├── agx_teleop
│ │ │ ├── agents
│ │ │ │ ├── __init__.py
│ │ │ │ └── sb3_ppo_cfg.yaml
│ │ │ ├── agx_teleop_env_cfg.py
│ │ │ ├── __init__.py
│ │ │ └── mdp
│ │ │ ├── __init__.py
│ │ │ └── rewards.py
│ │ └── __init__.py
│ └── ui_extension_example.py
├── agx_teleop.egg-info
│ ├── dependency_links.txt
│ ├── not-zip-safe
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── config
│ └── extension.toml
├── docs
│ └── CHANGELOG.rst
├── pyproject.toml
└── setup.py
1.1.3 Install External Projects
Running the following commands:
cd ../agx_teleop
python -m pip install -e source/agx_teleop
1.2 Configure the PiPER USD Asset
1.2.1 Prepare the URDF
Clone the official AgileX PiPER URDF repository:
cd agx_teleop
mkdir -p source/agx_teleop/agx_description
cd source/agx_teleop/agx_description
git clone https://github.com/agilexrobotics/agx_arm_urdf.git
Then merge:
piper_with_gripper_description.xacropiper_description.urdf
into a unified:piper_gripper.urdf
Complete PiPER URDF file content:
<?xml version="1.0" encoding="utf-8"?>
<!-- This URDF was automatically created by SolidWorks to URDF Exporter! Originally created by Stephen Brawner (brawner@gmail.com)
Commit Version: 1.6.0-4-g7f85cfe Build Version: 1.6.7995.38578
For more information, please see http://wiki.ros.org/sw_urdf_exporter -->
<robot name="piper">
<link name="world"/>
<joint name="world_to_base_link" type="fixed">
<parent link="world"/>
<child link="base_link"/>
<origin xyz="0 0 0" rpy="0 0 0"/>
</joint>
<link name="base_link">
<inertial>
<origin xyz="-0.00473641164191482 2.56829134630247E-05 0.041451518036016" rpy="0 0 0"/>
<mass value="1.02" />
<inertia ixx="0.00267433" ixy="-0.00000073" ixz="-0.00017389" iyy="0.00282612" iyz="0.0000004" izz="0.00089624"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/base_link.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/base_link.stl"/>
</geometry>
</collision>
</link>
<link name="link1">
<inertial>
<origin xyz="0.00032 -0.00041 -0.00348" rpy="0 0 0"/>
<mass value="0.71"/>
<inertia ixx="0.00050027" ixy="0.00000078" ixz="0.00000626" iyy="0.00040879" iyz="0.00000462" izz="0.00045802"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link1.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/link1.stl"/>
</geometry>
</collision>
</link>
<joint name="joint1" type="revolute">
<origin xyz="0 0 0.123" rpy="0 0 0"/>
<parent link="base_link"/>
<child link="link1"/>
<axis xyz="0 0 1"/>
<limit lower="-2.6179938" upper="2.6179938" effort="100" velocity="5"/>
</joint>
<link name="link2">
<inertial>
<origin xyz="0.21852 -0.00861 0.00126" rpy="0 0 0"/>
<mass value="1.16"/>
<inertia ixx="0.00111915" ixy="0.00182916" ixz="0.00019233" iyy="0.06763251" iyz="0.00000767" izz="0.067559"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link2.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/link2.stl"/>
</geometry>
</collision>
</link>
<joint name="joint2" type="revolute">
<origin xyz="0 0 0" rpy="1.5707963 -0.1357866 -3.1415926"/>
<parent link="link1"/>
<child link="link2"/>
<axis xyz="0 0 1"/>
<limit lower="0" upper="3.1415926" effort="100" velocity="5"/>
</joint>
<link name="link3">
<inertial>
<origin xyz="-0.01999 -0.14808 -0.00053" rpy="0 0 0"/>
<mass value="0.5"/>
<inertia ixx="0.01347608" ixy="0.00161034" ixz="0.00000066" iyy="0.00046377" iyz="0.00000162" izz="0.01363321"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link3.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/link3.stl"/>
</geometry>
</collision>
</link>
<joint name="joint3" type="revolute">
<origin xyz="0.28503 0 0" rpy="0 0 -1.7938494"/>
<parent link="link2"/>
<child link="link3"/>
<axis xyz="0 0 1"/>
<limit lower="-2.9670597" upper="0" effort="100" velocity="5"/>
</joint>
<link name="link4">
<inertial>
<origin xyz="0.00013 -0.00076 -0.00394" rpy="0 0 0"/>
<mass value="0.38"/>
<inertia ixx="0.00017844" ixy="0.00000062" ixz="0.00000067" iyy="0.00018914" iyz="0.00000408" izz="0.00014613"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link4.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/link4.stl"/>
</geometry>
</collision>
</link>
<joint name="joint4" type="revolute">
<origin xyz="-0.02198 -0.25075 0" rpy="1.5707963 0 0"/>
<parent link="link3"/>
<child link="link4"/>
<axis xyz="0 0 1"/>
<limit lower="-1.7453292" upper="1.7453292" effort="100" velocity="5"/>
</joint>
<link name="link5">
<inertial>
<origin xyz="0.00018 -0.06104 -0.00227" rpy="0 0 0"/>
<mass value="0.383"/>
<inertia ixx="0.00172318" ixy="0.00000259" ixz="0.00000010" iyy="0.00017936" iyz="0.00000990" izz="0.00171172"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link5.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/link5.stl"/>
</geometry>
</collision>
</link>
<joint name="joint5" type="revolute">
<origin xyz="0 0 0" rpy="-1.5707963 0 0"/>
<parent link="link4"/>
<child link="link5"/>
<axis xyz="0 0 1"/>
<limit lower="-1.2217304" upper="1.2217304" effort="100" velocity="5"/>
</joint>
<link name="link6">
<inertial>
<origin xyz="-0.00003 0.00006 -0.002" rpy="0 0 0"/>
<mass value="0.007"/>
<inertia ixx="0.00152112" ixy="0.00000202" ixz="0.00000261" iyy="0.00138626" iyz="0.00000002" izz="0.00031152"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/link6.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/link6.stl"/>
</geometry>
</collision>
</link>
<joint name="joint6" type="revolute">
<origin xyz="8.8259E-05 -0.091 0" rpy="1.5707963 0 0"/>
<parent link="link5"/>
<child link="link6"/>
<axis xyz="0 0 1"/>
<limit lower="-2.0943951" upper="2.0943951" effort="100" velocity="5"/>
</joint>
<!-- Gripper components -->
<link name="gripper_base">
<inertial>
<origin xyz="-0.000183807162235591 8.05033155577911E-05 0.0321436689908876" rpy="0 0 0"/>
<mass value="0.45"/>
<inertia ixx="0.00092934" ixy="0.00000034" ixz="-0.00000738" iyy="0.00071447" iyz="0.00000005" izz="0.00039442"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_base.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_base.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_base_joint" type="fixed">
<origin xyz="0 0 0" rpy="0 0 0"/>
<parent link="link6"/>
<child link="gripper_base"/>
</joint>
<link name="gripper_link1">
<inertial>
<origin xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025" rpy="0 0 0"/>
<mass value="0.025"/>
<inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_link1.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_link1.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_joint1" type="prismatic">
<origin xyz="0 0 0.1358" rpy="1.5707963 0 0"/>
<parent link="gripper_base"/>
<child link="gripper_link1"/>
<axis xyz="0 0 1"/>
<limit lower="0" upper="0.05" effort="10" velocity="3"/>
</joint>
<link name="gripper_link2">
<inertial>
<origin xyz="0.00065123185041968 -0.0491929869131989 0.00972258769184025" rpy="0 0 0"/>
<mass value="0.025"/>
<inertia ixx="0.00007371" ixy="-0.00000113" ixz="0.00000021" iyy="0.00000781" iyz="-0.00001372" izz="0.0000747"/>
</inertial>
<visual>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/dae/gripper_link2.dae"/>
</geometry>
</visual>
<collision>
<origin xyz="0 0 0" rpy="0 0 0"/>
<geometry>
<mesh filename="../meshes/gripper_link2.stl"/>
</geometry>
</collision>
</link>
<joint name="gripper_joint2" type="prismatic">
<origin xyz="0 0 0.1358" rpy="1.5707963 0 -3.1415926"/>
<parent link="gripper_base"/>
<child link="gripper_link2"/>
<axis xyz="0 0 -1"/>
<limit lower="-0.05" upper="0" effort="10" velocity="3"/>
</joint>
</robot>
1.2.2 Convert URDF to USD
Isaac Lab uses USD assets for simulation.
Convert the Piper URDF into a USD robot asset:
python ../IsaacLab/scripts/tools/convert_urdf.py \
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/piper/urdf/piper_gripper.urdf \
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/piper/usd/piper.usd \
--fix-base
Step 2.Configure the PiPER Robot in Isaac Lab
2.1 Create the Robot Asset Configuration
cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/assets
2.1.1 Create the Asset Package
Create the assets/__init__.py file:
"""Package containing robot configurations."""
import os
import toml
##
# Configuration for different assets.
##
# Conveniences to other module directories via relative paths
AGX_TELEOP_EXT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))
"""Path to the source agx_teleop directory."""
AGX_TELEOP_DESCRIPTION_DIR = os.path.join(AGX_TELEOP_EXT_DIR, "agx_description")
"""Path to the agx_description directory."""
# Robot description assets
AGX_TELEOP_DESCRIPTION_DIR = os.path.join(
AGX_TELEOP_EXT_DIR,
"agx_description"
)
"""Path to the agx_description directory."""
2.1.2 Configure the PiPER Robot Asset
Create the assets/piper.py file
import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets.articulation import ArticulationCfg
from agx_teleop.assets import AGX_TELEOP_DESCRIPTION_DIR
PIPER_GRIPPER_CFG = ArticulationCfg(
# USD robot asset
spawn=sim_utils.UsdFileCfg(
usd_path=f"{AGX_TELEOP_DESCRIPTION_DIR}/agx_arm_urdf/piper/usd/piper.usd",
# Rigid body physics settings
rigid_props=sim_utils.RigidBodyPropertiesCfg(
disable_gravity=False,
max_depenetration_velocity=5.0,
),
# Articulation solver settings
articulation_props=sim_utils.ArticulationRootPropertiesCfg(
enabled_self_collisions=True,
solver_position_iteration_count=8,
solver_velocity_iteration_count=0,
),
),
# Initial robot pose
init_state=ArticulationCfg.InitialStateCfg(
joint_pos={
"joint1": 0.0,
"joint2": 1.57,
"joint3": -1.57,
"joint4": 0.0,
"joint5": 1.22,
"joint6": 0.0,
"gripper_joint1": 0.0,
"gripper_joint2": 0.0,
},
pos=(0.0, 0.0, 0.0),
),
# Actuator configuration
actuators={
# Shoulder joints
"piper_shoulder": ImplicitActuatorCfg(
joint_names_expr=["joint[1-3]"],
effort_limit_sim=100.0,
velocity_limit_sim=5.0,
stiffness=80.0,
damping=20.0,
),
# Forearm joints
"piper_forearm": ImplicitActuatorCfg(
joint_names_expr=["joint[4-6]"],
effort_limit_sim=100.0,
velocity_limit_sim=5.0,
stiffness=40.0,
damping=10.0,
),
# Parallel gripper
"gripper": ImplicitActuatorCfg(
joint_names_expr=[
"gripper_joint1",
"gripper_joint2"
],
effort_limit_sim=22,
velocity_limit_sim=1.5,
stiffness=800.0,
damping=20.0,
),
},
# Soft joint limit scaling
soft_joint_pos_limit_factor=0.9,
)
"""Configuration for the Piper robot with parallel gripper."""
PIPER_GRIPPER_HIGH_PD_CFG = PIPER_GRIPPER_CFG.copy()
PIPER_GRIPPER_HIGH_PD_CFG.spawn.rigid_props.disable_gravity = True
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_shoulder"].stiffness = 400.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_shoulder"].damping = 80.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_forearm"].stiffness = 200.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["piper_forearm"].damping = 40.0
PIPER_GRIPPER_HIGH_PD_CFG.actuators["gripper"].velocity_limit_sim = 0.5
"""Configuration of Piper with gripper robot with stiffer PD control.
This configuration is useful for task-space control using differential IK.
"""
Step 3.Build the Manipulation Environment
3.1 Project Structure Setup
The following structure is modified from the official IsaacLab stack task isaaclab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack. We will create a specialized directory for the PiPER robot under the agx_teleop project.
Create Directories
cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/config/piper/agents/robomimic
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/mdp
Initialize Files
cd source/agx_teleop/agx_teleop/tasks/manager_based/
touch manipulation/__init__.py \
manipulation/stack/__init__.py \
manipulation/stack/stack_env_cfg.py \
manipulation/stack/mdp/__init__.py \
manipulation/stack/mdp/observations.py \
manipulation/stack/mdp/piper_stack_events.py \
manipulation/stack/mdp/terminations.py \
manipulation/stack/config/__init__.py \
manipulation/stack/config/piper/__init__.py \
manipulation/stack/config/piper/agents/__init__.py \
manipulation/stack/config/piper/agents/robomimic/bc_rnn_low_dim.json \
manipulation/stack/config/piper/piper_stack_ik_rel_env_cfg.py \
manipulation/stack/config/piper/piper_stack_joint_pos_env_cfg.py
3.2 Core File Modifications
3.2.1 Standard IsaacLab Porting
The following files are ported directly from the IsaacLab Franka stack example without significant modification. They define the base MDP (Markov Decision Process) settings and the Robomimic imitation learning configuration.
- stack/stack_env_cfg.py: Defines the scene (table, lights, ground) and observation groups.
- stack/mdp/init.py: Handles module exports.
- stack/config/piper/agents/robomimic/bc_rnn_low_dim.json: Standard BC-RNN configuration for low-dimensional observations.
Copy the corresponding code from IsaacLab without modification :
/Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/stack_env_cfg.py
/Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/__init__.py
/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/agents/robomimic/bc_rnn_low_dim.json
3.2.2 Event Customization (mdp/piper_stack_events.py)
Copy the corresponding code from Issaclab directly, without any modifying :
Isaac/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/mdp/franka_stack_events.py
3.2.3 Adapting Observations & Terminations for Piper
Copy the corresponding code of isaaclab and modify the part related to gripper_open_val. The Franka panda gripper control uses a single joint value, while the piper gripper control uses two joint values, which need to be adjusted to match the piper gripper.
Key Logic Adjustment (Lines 320-343 & 380-407):
stack/mdp/observations.py
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from __future__ import annotations
from typing import TYPE_CHECKING, Literal
import torch
import isaaclab.utils.math as math_utils
from isaaclab.assets import Articulation, RigidObject, RigidObjectCollection
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformer
if TYPE_CHECKING:
from isaaclab.envs import ManagerBasedRLEnv
def cube_positions_in_world_frame(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
"""The position of the cubes in the world frame."""
cube_1: RigidObject = env.scene[cube_1_cfg.name]
cube_2: RigidObject = env.scene[cube_2_cfg.name]
cube_3: RigidObject = env.scene[cube_3_cfg.name]
return torch.cat((cube_1.data.root_pos_w, cube_2.data.root_pos_w, cube_3.data.root_pos_w), dim=1)
def instance_randomize_cube_positions_in_world_frame(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
"""The position of the cubes in the world frame."""
if not hasattr(env, "rigid_objects_in_focus"):
return torch.full((env.num_envs, 9), fill_value=-1)
cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]
cube_1_pos_w = []
cube_2_pos_w = []
cube_3_pos_w = []
for env_id in range(env.num_envs):
cube_1_pos_w.append(cube_1.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][0], :3])
cube_2_pos_w.append(cube_2.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][1], :3])
cube_3_pos_w.append(cube_3.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][2], :3])
cube_1_pos_w = torch.stack(cube_1_pos_w)
cube_2_pos_w = torch.stack(cube_2_pos_w)
cube_3_pos_w = torch.stack(cube_3_pos_w)
return torch.cat((cube_1_pos_w, cube_2_pos_w, cube_3_pos_w), dim=1)
def cube_orientations_in_world_frame(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
):
"""The orientation of the cubes in the world frame."""
cube_1: RigidObject = env.scene[cube_1_cfg.name]
cube_2: RigidObject = env.scene[cube_2_cfg.name]
cube_3: RigidObject = env.scene[cube_3_cfg.name]
return torch.cat((cube_1.data.root_quat_w, cube_2.data.root_quat_w, cube_3.data.root_quat_w), dim=1)
def instance_randomize_cube_orientations_in_world_frame(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
) -> torch.Tensor:
"""The orientation of the cubes in the world frame."""
if not hasattr(env, "rigid_objects_in_focus"):
return torch.full((env.num_envs, 9), fill_value=-1)
cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]
cube_1_quat_w = []
cube_2_quat_w = []
cube_3_quat_w = []
for env_id in range(env.num_envs):
cube_1_quat_w.append(cube_1.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][0], :4])
cube_2_quat_w.append(cube_2.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][1], :4])
cube_3_quat_w.append(cube_3.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][2], :4])
cube_1_quat_w = torch.stack(cube_1_quat_w)
cube_2_quat_w = torch.stack(cube_2_quat_w)
cube_3_quat_w = torch.stack(cube_3_quat_w)
return torch.cat((cube_1_quat_w, cube_2_quat_w, cube_3_quat_w), dim=1)
def object_obs(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
):
"""
Object observations (in world frame):
cube_1 pos,
cube_1 quat,
cube_2 pos,
cube_2 quat,
cube_3 pos,
cube_3 quat,
gripper to cube_1,
gripper to cube_2,
gripper to cube_3,
cube_1 to cube_2,
cube_2 to cube_3,
cube_1 to cube_3,
"""
cube_1: RigidObject = env.scene[cube_1_cfg.name]
cube_2: RigidObject = env.scene[cube_2_cfg.name]
cube_3: RigidObject = env.scene[cube_3_cfg.name]
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
cube_1_pos_w = cube_1.data.root_pos_w
cube_1_quat_w = cube_1.data.root_quat_w
cube_2_pos_w = cube_2.data.root_pos_w
cube_2_quat_w = cube_2.data.root_quat_w
cube_3_pos_w = cube_3.data.root_pos_w
cube_3_quat_w = cube_3.data.root_quat_w
ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
gripper_to_cube_1 = cube_1_pos_w - ee_pos_w
gripper_to_cube_2 = cube_2_pos_w - ee_pos_w
gripper_to_cube_3 = cube_3_pos_w - ee_pos_w
cube_1_to_2 = cube_1_pos_w - cube_2_pos_w
cube_2_to_3 = cube_2_pos_w - cube_3_pos_w
cube_1_to_3 = cube_1_pos_w - cube_3_pos_w
return torch.cat(
(
cube_1_pos_w - env.scene.env_origins,
cube_1_quat_w,
cube_2_pos_w - env.scene.env_origins,
cube_2_quat_w,
cube_3_pos_w - env.scene.env_origins,
cube_3_quat_w,
gripper_to_cube_1,
gripper_to_cube_2,
gripper_to_cube_3,
cube_1_to_2,
cube_2_to_3,
cube_1_to_3,
),
dim=1,
)
def instance_randomize_object_obs(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
):
"""
Object observations (in world frame):
cube_1 pos,
cube_1 quat,
cube_2 pos,
cube_2 quat,
cube_3 pos,
cube_3 quat,
gripper to cube_1,
gripper to cube_2,
gripper to cube_3,
cube_1 to cube_2,
cube_2 to cube_3,
cube_1 to cube_3,
"""
if not hasattr(env, "rigid_objects_in_focus"):
return torch.full((env.num_envs, 9), fill_value=-1)
cube_1: RigidObjectCollection = env.scene[cube_1_cfg.name]
cube_2: RigidObjectCollection = env.scene[cube_2_cfg.name]
cube_3: RigidObjectCollection = env.scene[cube_3_cfg.name]
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
cube_1_pos_w = []
cube_2_pos_w = []
cube_3_pos_w = []
cube_1_quat_w = []
cube_2_quat_w = []
cube_3_quat_w = []
for env_id in range(env.num_envs):
cube_1_pos_w.append(cube_1.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][0], :3])
cube_2_pos_w.append(cube_2.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][1], :3])
cube_3_pos_w.append(cube_3.data.object_pos_w[env_id, env.rigid_objects_in_focus[env_id][2], :3])
cube_1_quat_w.append(cube_1.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][0], :4])
cube_2_quat_w.append(cube_2.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][1], :4])
cube_3_quat_w.append(cube_3.data.object_quat_w[env_id, env.rigid_objects_in_focus[env_id][2], :4])
cube_1_pos_w = torch.stack(cube_1_pos_w)
cube_2_pos_w = torch.stack(cube_2_pos_w)
cube_3_pos_w = torch.stack(cube_3_pos_w)
cube_1_quat_w = torch.stack(cube_1_quat_w)
cube_2_quat_w = torch.stack(cube_2_quat_w)
cube_3_quat_w = torch.stack(cube_3_quat_w)
ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
gripper_to_cube_1 = cube_1_pos_w - ee_pos_w
gripper_to_cube_2 = cube_2_pos_w - ee_pos_w
gripper_to_cube_3 = cube_3_pos_w - ee_pos_w
cube_1_to_2 = cube_1_pos_w - cube_2_pos_w
cube_2_to_3 = cube_2_pos_w - cube_3_pos_w
cube_1_to_3 = cube_1_pos_w - cube_3_pos_w
return torch.cat(
(
cube_1_pos_w - env.scene.env_origins,
cube_1_quat_w,
cube_2_pos_w - env.scene.env_origins,
cube_2_quat_w,
cube_3_pos_w - env.scene.env_origins,
cube_3_quat_w,
gripper_to_cube_1,
gripper_to_cube_2,
gripper_to_cube_3,
cube_1_to_2,
cube_2_to_3,
cube_1_to_3,
),
dim=1,
)
def ee_frame_pos(env: ManagerBasedRLEnv, ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")) -> torch.Tensor:
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
ee_frame_pos = ee_frame.data.target_pos_w[:, 0, :] - env.scene.env_origins[:, 0:3]
return ee_frame_pos
def ee_frame_quat(env: ManagerBasedRLEnv, ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame")) -> torch.Tensor:
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
ee_frame_quat = ee_frame.data.target_quat_w[:, 0, :]
return ee_frame_quat
def gripper_pos(
env: ManagerBasedRLEnv,
robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
) -> torch.Tensor:
"""
Obtain the versatile gripper position of both Gripper and Suction Cup.
"""
robot: Articulation = env.scene[robot_cfg.name]
if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
# Handle multiple surface grippers by concatenating their states
gripper_states = []
for gripper_name, surface_gripper in env.scene.surface_grippers.items():
gripper_states.append(surface_gripper.state.view(-1, 1))
if len(gripper_states) == 1:
return gripper_states[0]
else:
return torch.cat(gripper_states, dim=1)
else:
if hasattr(env.cfg, "gripper_joint_names"):
gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
assert len(gripper_joint_ids) == 2, "Observation gripper_pos only support parallel gripper for now"
finger_joint_1 = robot.data.joint_pos[:, gripper_joint_ids[0]].clone().unsqueeze(1)
finger_joint_2 = -1 * robot.data.joint_pos[:, gripper_joint_ids[1]].clone().unsqueeze(1)
return torch.cat((finger_joint_1, finger_joint_2), dim=1)
else:
raise NotImplementedError("[Error] Cannot find gripper_joint_names in the environment config")
def object_grasped(
env: ManagerBasedRLEnv,
robot_cfg: SceneEntityCfg,
ee_frame_cfg: SceneEntityCfg,
object_cfg: SceneEntityCfg,
diff_threshold: float = 0.06,
) -> torch.Tensor:
"""Check if an object is grasped by the specified robot."""
robot: Articulation = env.scene[robot_cfg.name]
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
object: RigidObject = env.scene[object_cfg.name]
object_pos = object.data.root_pos_w
end_effector_pos = ee_frame.data.target_pos_w[:, 0, :]
pose_diff = torch.linalg.vector_norm(object_pos - end_effector_pos, dim=1)
if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
surface_gripper = env.scene.surface_grippers["surface_gripper"]
suction_cup_status = surface_gripper.state.view(-1, 1) # 1: closed, 0: closing, -1: open
suction_cup_is_closed = (suction_cup_status == 1).to(torch.float32)
grasped = torch.logical_and(suction_cup_is_closed, pose_diff < diff_threshold)
else:
if hasattr(env.cfg, "gripper_joint_names"):
gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
assert len(gripper_joint_ids) == 2, "Observations only support parallel gripper for now"
# Support per-joint open values (list/tuple) or single scalar (applied to all joints)
if isinstance(env.cfg.gripper_open_val, (list, tuple)):
open_val_0 = env.cfg.gripper_open_val[0]
open_val_1 = env.cfg.gripper_open_val[1]
else:
open_val_0 = env.cfg.gripper_open_val
open_val_1 = env.cfg.gripper_open_val
grasped = torch.logical_and(
pose_diff < diff_threshold,
torch.abs(
robot.data.joint_pos[:, gripper_joint_ids[0]]
- torch.tensor(open_val_0, dtype=torch.float32).to(env.device)
)
> env.cfg.gripper_threshold,
)
grasped = torch.logical_and(
grasped,
torch.abs(
robot.data.joint_pos[:, gripper_joint_ids[1]]
- torch.tensor(open_val_1, dtype=torch.float32).to(env.device)
)
> env.cfg.gripper_threshold,
)
return grasped
def object_stacked(
env: ManagerBasedRLEnv,
robot_cfg: SceneEntityCfg,
upper_object_cfg: SceneEntityCfg,
lower_object_cfg: SceneEntityCfg,
xy_threshold: float = 0.05,
height_threshold: float = 0.005,
height_diff: float = 0.0468,
) -> torch.Tensor:
"""Check if an object is stacked by the specified robot."""
robot: Articulation = env.scene[robot_cfg.name]
upper_object: RigidObject = env.scene[upper_object_cfg.name]
lower_object: RigidObject = env.scene[lower_object_cfg.name]
pos_diff = upper_object.data.root_pos_w - lower_object.data.root_pos_w
height_dist = torch.linalg.vector_norm(pos_diff[:, 2:], dim=1)
xy_dist = torch.linalg.vector_norm(pos_diff[:, :2], dim=1)
stacked = torch.logical_and(xy_dist < xy_threshold, (height_dist - height_diff) < height_threshold)
if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
surface_gripper = env.scene.surface_grippers["surface_gripper"]
suction_cup_status = surface_gripper.state.view(-1, 1) # 1: closed, 0: closing, -1: open
suction_cup_is_open = (suction_cup_status == -1).to(torch.float32)
stacked = torch.logical_and(suction_cup_is_open, stacked)
else:
if hasattr(env.cfg, "gripper_joint_names"):
gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
assert len(gripper_joint_ids) == 2, "Observations only support parallel gripper for now"
# Support per-joint open values (list/tuple) or single scalar (applied to all joints)
if isinstance(env.cfg.gripper_open_val, (list, tuple)):
open_val_0 = env.cfg.gripper_open_val[0]
open_val_1 = env.cfg.gripper_open_val[1]
else:
open_val_0 = env.cfg.gripper_open_val
open_val_1 = env.cfg.gripper_open_val
stacked = torch.logical_and(
torch.isclose(
robot.data.joint_pos[:, gripper_joint_ids[0]],
torch.tensor(open_val_0, dtype=torch.float32).to(env.device),
atol=1e-4,
rtol=1e-4,
),
stacked,
)
stacked = torch.logical_and(
torch.isclose(
robot.data.joint_pos[:, gripper_joint_ids[1]],
torch.tensor(open_val_1, dtype=torch.float32).to(env.device),
atol=1e-4,
rtol=1e-4,
),
stacked,
)
else:
raise ValueError("No gripper_joint_names found in environment config")
return stacked
def cube_poses_in_base_frame(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
return_key: Literal["pos", "quat", None] = None,
) -> torch.Tensor:
"""The position and orientation of the cubes in the robot base frame."""
cube_1: RigidObject = env.scene[cube_1_cfg.name]
cube_2: RigidObject = env.scene[cube_2_cfg.name]
cube_3: RigidObject = env.scene[cube_3_cfg.name]
pos_cube_1_world = cube_1.data.root_pos_w
pos_cube_2_world = cube_2.data.root_pos_w
pos_cube_3_world = cube_3.data.root_pos_w
quat_cube_1_world = cube_1.data.root_quat_w
quat_cube_2_world = cube_2.data.root_quat_w
quat_cube_3_world = cube_3.data.root_quat_w
robot: Articulation = env.scene[robot_cfg.name]
root_pos_w = robot.data.root_pos_w
root_quat_w = robot.data.root_quat_w
pos_cube_1_base, quat_cube_1_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, pos_cube_1_world, quat_cube_1_world
)
pos_cube_2_base, quat_cube_2_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, pos_cube_2_world, quat_cube_2_world
)
pos_cube_3_base, quat_cube_3_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, pos_cube_3_world, quat_cube_3_world
)
pos_cubes_base = torch.cat((pos_cube_1_base, pos_cube_2_base, pos_cube_3_base), dim=1)
quat_cubes_base = torch.cat((quat_cube_1_base, quat_cube_2_base, quat_cube_3_base), dim=1)
if return_key == "pos":
return pos_cubes_base
elif return_key == "quat":
return quat_cubes_base
else:
return torch.cat((pos_cubes_base, quat_cubes_base), dim=1)
def object_abs_obs_in_base_frame(
env: ManagerBasedRLEnv,
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg = SceneEntityCfg("cube_3"),
ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
):
"""
Object Abs observations (in base frame): remove the relative observations,
and add abs gripper pos and quat in robot base frame
cube_1 pos,
cube_1 quat,
cube_2 pos,
cube_2 quat,
cube_3 pos,
cube_3 quat,
gripper pos,
gripper quat,
"""
cube_1: RigidObject = env.scene[cube_1_cfg.name]
cube_2: RigidObject = env.scene[cube_2_cfg.name]
cube_3: RigidObject = env.scene[cube_3_cfg.name]
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
robot: Articulation = env.scene[robot_cfg.name]
root_pos_w = robot.data.root_pos_w
root_quat_w = robot.data.root_quat_w
cube_1_pos_w = cube_1.data.root_pos_w
cube_1_quat_w = cube_1.data.root_quat_w
cube_2_pos_w = cube_2.data.root_pos_w
cube_2_quat_w = cube_2.data.root_quat_w
cube_3_pos_w = cube_3.data.root_pos_w
cube_3_quat_w = cube_3.data.root_quat_w
pos_cube_1_base, quat_cube_1_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, cube_1_pos_w, cube_1_quat_w
)
pos_cube_2_base, quat_cube_2_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, cube_2_pos_w, cube_2_quat_w
)
pos_cube_3_base, quat_cube_3_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, cube_3_pos_w, cube_3_quat_w
)
ee_pos_w = ee_frame.data.target_pos_w[:, 0, :]
ee_quat_w = ee_frame.data.target_quat_w[:, 0, :]
ee_pos_base, ee_quat_base = math_utils.subtract_frame_transforms(root_pos_w, root_quat_w, ee_pos_w, ee_quat_w)
return torch.cat(
(
pos_cube_1_base,
quat_cube_1_base,
pos_cube_2_base,
quat_cube_2_base,
pos_cube_3_base,
quat_cube_3_base,
ee_pos_base,
ee_quat_base,
),
dim=1,
)
def ee_frame_pose_in_base_frame(
env: ManagerBasedRLEnv,
ee_frame_cfg: SceneEntityCfg = SceneEntityCfg("ee_frame"),
robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
return_key: Literal["pos", "quat", None] = None,
) -> torch.Tensor:
"""
The end effector pose in the robot base frame.
"""
ee_frame: FrameTransformer = env.scene[ee_frame_cfg.name]
ee_frame_pos_w = ee_frame.data.target_pos_w[:, 0, :]
ee_frame_quat_w = ee_frame.data.target_quat_w[:, 0, :]
robot: Articulation = env.scene[robot_cfg.name]
root_pos_w = robot.data.root_pos_w
root_quat_w = robot.data.root_quat_w
ee_pos_in_base, ee_quat_in_base = math_utils.subtract_frame_transforms(
root_pos_w, root_quat_w, ee_frame_pos_w, ee_frame_quat_w
)
if return_key == "pos":
return ee_pos_in_base
elif return_key == "quat":
return ee_quat_in_base
else:
return torch.cat((ee_pos_in_base, ee_quat_in_base), dim=1)
Similarly, the cubes_stacked termination condition is updated to ensure the success criteria check both gripper joints before concluding the task.
Below is the complete code, with relevant modifications in Lines 81-109.
stack/mdp/terminations.py
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
"""Common functions that can be used to activate certain terminations for the lift task.
The functions can be passed to the :class:`isaaclab.managers.TerminationTermCfg` object to enable
the termination introduced by the function.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import torch
from isaaclab.assets import Articulation, RigidObject
from isaaclab.managers import SceneEntityCfg
if TYPE_CHECKING:
from isaaclab.envs import ManagerBasedRLEnv
def cubes_stacked(
env: ManagerBasedRLEnv,
robot_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
cube_1_cfg: SceneEntityCfg = SceneEntityCfg("cube_1"),
cube_2_cfg: SceneEntityCfg = SceneEntityCfg("cube_2"),
cube_3_cfg: SceneEntityCfg | None = SceneEntityCfg("cube_3"),
xy_threshold: float = 0.04,
height_threshold: float = 0.005,
height_diff: float = 0.0468,
atol: float = 0.0001,
rtol: float = 0.0001,
) -> torch.Tensor:
robot: Articulation = env.scene[robot_cfg.name]
cube_1: RigidObject = env.scene[cube_1_cfg.name]
cube_2: RigidObject = env.scene[cube_2_cfg.name]
pos_diff_c12 = cube_1.data.root_pos_w - cube_2.data.root_pos_w
# Compute cube position difference in x-y plane
xy_dist_c12 = torch.norm(pos_diff_c12[:, :2], dim=1)
# Compute cube height difference
h_dist_c12 = torch.norm(pos_diff_c12[:, 2:], dim=1)
# Check cube positions
stacked = xy_dist_c12 < xy_threshold
stacked = torch.logical_and(h_dist_c12 - height_diff < height_threshold, stacked)
stacked = torch.logical_and(pos_diff_c12[:, 2] < 0.0, stacked)
if cube_3_cfg is not None:
cube_3: RigidObject = env.scene[cube_3_cfg.name]
pos_diff_c23 = cube_2.data.root_pos_w - cube_3.data.root_pos_w
# Compute cube position difference in x-y plane
xy_dist_c23 = torch.norm(pos_diff_c23[:, :2], dim=1)
# Compute cube height difference
h_dist_c23 = torch.norm(pos_diff_c23[:, 2:], dim=1)
# Check cube positions
stacked = torch.logical_and(xy_dist_c23 < xy_threshold, stacked)
stacked = torch.logical_and(h_dist_c23 - height_diff < height_threshold, stacked)
stacked = torch.logical_and(pos_diff_c23[:, 2] < 0.0, stacked)
# Check gripper positions
if hasattr(env.scene, "surface_grippers") and len(env.scene.surface_grippers) > 0:
surface_gripper = env.scene.surface_grippers["surface_gripper"]
suction_cup_status = surface_gripper.state.view(-1) # 1: closed, 0: closing, -1: open
suction_cup_is_open = (suction_cup_status == -1).to(torch.float32)
stacked = torch.logical_and(suction_cup_is_open, stacked)
else:
if hasattr(env.cfg, "gripper_joint_names"):
gripper_joint_ids, _ = robot.find_joints(env.cfg.gripper_joint_names)
assert len(gripper_joint_ids) == 2, "Terminations only support parallel gripper for now"
# Support per-joint open values (list/tuple) or single scalar (applied to all joints)
if isinstance(env.cfg.gripper_open_val, (list, tuple)):
open_val_0 = env.cfg.gripper_open_val[0]
open_val_1 = env.cfg.gripper_open_val[1]
else:
open_val_0 = env.cfg.gripper_open_val
open_val_1 = env.cfg.gripper_open_val
stacked = torch.logical_and(
torch.isclose(
robot.data.joint_pos[:, gripper_joint_ids[0]],
torch.tensor(open_val_0, dtype=torch.float32).to(env.device),
atol=atol,
rtol=rtol,
),
stacked,
)
stacked = torch.logical_and(
torch.isclose(
robot.data.joint_pos[:, gripper_joint_ids[1]],
torch.tensor(open_val_1, dtype=torch.float32).to(env.device),
atol=atol,
rtol=rtol,
),
stacked,
)
else:
raise ValueError("No gripper_joint_names found in environment config")
return stacked
3.2.4 Piper Joint Position Configuration (piper_stack_joint_pos_env_cfg.py)
This file maps the Piper asset to the environment.
Major changes include:
- Default Pose: Adjusted to [0.0, 1.57, -1.57, 0.0, 1.22, 0.0, 0.0, 0.0] (6 ARM joints + 2 Gripper joints).
- Gripper Command Mapping:
- Open: gripper_joint1: 0.05, gripper_joint2: -0.05
- Close: 0.0 for both.
- End-Effector (EE) Offset: Set to [0.0, 0.0, 0.14] to account for the Piper gripper’s length from the base link to the Tool Center Point (TCP).
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from isaaclab.assets import RigidObjectCfg
from isaaclab.managers import EventTermCfg as EventTerm
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformerCfg
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg
from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
from agx_teleop.tasks.manager_based.manipulation.stack import mdp
from agx_teleop.tasks.manager_based.manipulation.stack.mdp import piper_stack_events
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import StackEnvCfg
##
# Pre-defined configs
##
from isaaclab.markers.config import FRAME_MARKER_CFG # isort: skip
from agx_teleop.assets.piper import PIPER_GRIPPER_CFG # isort: skip
@configclass
class EventCfg:
"""Configuration for events."""
init_piper_arm_pose = EventTerm(
func=piper_stack_events.set_default_joint_pose,
mode="reset",
params={
"default_pose": [0.0, 1.57, -1.57, 0.0, 1.22, 0.0, 0.0, 0.0], # 6机械臂+2夹爪，夹爪关闭
},
)
randomize_piper_joint_state = EventTerm(
func=piper_stack_events.randomize_joint_by_gaussian_offset,
mode="reset",
params={
"mean": 0.0,
"std": 0.02,
"asset_cfg": SceneEntityCfg("robot"),
},
)
randomize_cube_positions = EventTerm(
func=piper_stack_events.randomize_object_pose,
mode="reset",
params={
"pose_range": {"x": (0.3, 0.5), "y": (-0.10, 0.10), "z": (0.0203, 0.0203), "yaw": (-1.0, 1, 0)},
"min_separation": 0.12,
"asset_cfgs": [SceneEntityCfg("cube_1"), SceneEntityCfg("cube_2"), SceneEntityCfg("cube_3")],
},
)
@configclass
class PiperCubeStackEnvCfg(StackEnvCfg):
"""Configuration for the Piper Cube Stack Environment."""
def __post_init__(self):
# post init of parent
super().__post_init__()
# Set events
self.events = EventCfg()
# Set Piper as robot
self.scene.robot = PIPER_GRIPPER_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
self.scene.robot.spawn.semantic_tags = [("class", "robot")]
# Add semantics to table
self.scene.table.spawn.semantic_tags = [("class", "table")]
# Add semantics to ground
self.scene.plane.semantic_tags = [("class", "ground")]
# Set actions for the specific robot type (piper)
self.actions.arm_action = mdp.JointPositionActionCfg(
asset_name="robot", joint_names=["joint.*"], scale=0.5, use_default_offset=True
)
self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
asset_name="robot",
joint_names=["gripper_joint.*"],
open_command_expr={
"gripper_joint1": 0.05,
"gripper_joint2": -0.05,
},
close_command_expr={
"gripper_joint1": 0.0,
"gripper_joint2": 0.0,
},
)
# utilities for gripper status check
self.gripper_joint_names = ["gripper_joint.*"]
self.gripper_open_val = [0.05, -0.05]
self.gripper_threshold = 0.005
# Rigid body properties of each cube
cube_properties = RigidBodyPropertiesCfg(
solver_position_iteration_count=16,
solver_velocity_iteration_count=1,
max_angular_velocity=1000.0,
max_linear_velocity=1000.0,
max_depenetration_velocity=5.0,
disable_gravity=False,
)
# Set each stacking cube deterministically
self.scene.cube_1 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_1",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0.0, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/blue_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_1")],
),
)
self.scene.cube_2 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_2",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.55, 0.05, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/red_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_2")],
),
)
self.scene.cube_3 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_3",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.60, -0.1, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/green_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_3")],
),
)
# Listens to the required transforms
marker_cfg = FRAME_MARKER_CFG.copy()
marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
marker_cfg.prim_path = "/Visuals/FrameTransformer"
self.scene.ee_frame = FrameTransformerCfg(
prim_path="{ENV_REGEX_NS}/Robot/base_link",
debug_vis=False,
visualizer_cfg=marker_cfg,
target_frames=[
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_base",
name="end_effector",
offset=OffsetCfg(
pos=[0.0, 0.0, 0.14], # gripper_base 到 TCP 的距离
),
),
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_link1",
name="tool_rightfinger",
offset=OffsetCfg(
pos=(0.0, 0.0, 0.0),
),
),
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_link2",
name="tool_leftfinger",
offset=OffsetCfg(
pos=(0.0, 0.0, 0.0),
),
),
],
)
3.2.5 Inverse Kinematics Configuration (piper_stack_ik_rel_env_cfg.py)
This file configures the Differential IK controller. We use a high-gain PD controller (PIPER_GRIPPER_HIGH_PD_CFG) to ensure precise tracking during teleoperation or inference.
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from isaaclab.controllers.differential_ik_cfg import DifferentialIKControllerCfg
from isaaclab.devices.device_base import DeviceBase, DevicesCfg
from isaaclab.devices.keyboard import Se3KeyboardCfg
from isaaclab.devices.openxr.openxr_device import OpenXRDeviceCfg
from isaaclab.devices.openxr.retargeters.manipulator.gripper_retargeter import GripperRetargeterCfg
from isaaclab.devices.openxr.retargeters.manipulator.se3_rel_retargeter import Se3RelRetargeterCfg
from isaaclab.envs.mdp.actions.actions_cfg import DifferentialInverseKinematicsActionCfg
from isaaclab.managers import SceneEntityCfg
from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.utils import configclass
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import mdp
from . import piper_stack_joint_pos_env_cfg
##
# Pre-defined configs
##
from agx_teleop.assets.piper import PIPER_GRIPPER_HIGH_PD_CFG # isort: skip
@configclass
class PiperCubeStackEnvCfg(piper_stack_joint_pos_env_cfg.PiperCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
# Set Piper as robot
# We switch here to a stiffer PD controller for IK tracking to be better.
self.scene.robot = PIPER_GRIPPER_HIGH_PD_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
# Set actions for the specific robot type (piper)
self.actions.arm_action = DifferentialInverseKinematicsActionCfg(
asset_name="robot",
joint_names=["joint.*"],
body_name="gripper_base",
controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),
scale=0.5,
body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.14]),
)
self.teleop_devices = DevicesCfg(
devices={
"handtracking": OpenXRDeviceCfg(
retargeters=[
Se3RelRetargeterCfg(
bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT,
zero_out_xy_rotation=True,
use_wrist_rotation=False,
use_wrist_position=True,
delta_pos_scale_factor=10.0,
delta_rot_scale_factor=10.0,
sim_device=self.sim.device,
),
GripperRetargeterCfg(
bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT, sim_device=self.sim.device
),
],
sim_device=self.sim.device,
xr_cfg=self.xr,
),
"keyboard": Se3KeyboardCfg(
pos_sensitivity=0.05,
rot_sensitivity=0.05,
sim_device=self.sim.device,
),
}
)
@configclass
class PiperCubeStackRedGreenEnvCfg(PiperCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={"cube_1_cfg": SceneEntityCfg("cube_2"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
)
@configclass
class PiperCubeStackRedGreenBlueEnvCfg(PiperCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={
"cube_1_cfg": SceneEntityCfg("cube_2"),
"cube_2_cfg": SceneEntityCfg("cube_3"),
"cube_3_cfg": SceneEntityCfg("cube_1"),
},
)
@configclass
class PiperCubeStackBlueGreenEnvCfg(PiperCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={"cube_1_cfg": SceneEntityCfg("cube_1"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
)
@configclass
class PiperCubeStackBlueGreenRedEnvCfg(PiperCubeStackEnvCfg):
def __post_init__(self):
# post init of parent
super().__post_init__()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={
"cube_1_cfg": SceneEntityCfg("cube_1"),
"cube_2_cfg": SceneEntityCfg("cube_3"),
"cube_3_cfg": SceneEntityCfg("cube_2"),
},
)
3.2.6 Environment Registration (stack/config/piper/__init__.py)
Finally, register the task in Gymnasium so it can be launched via the IsaacLab runner:
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
import gymnasium as gym
# from isaaclab_tasks.manager_based.manipulation.stack.config.franka import agents
from agx_teleop.tasks.manager_based.manipulation.stack.config.piper import agents
##
# Register Gym environments.
##
gym.register(
id="Isaac-Stack-Cube-Piper-IK-Rel-v0",
entry_point="isaaclab.envs:ManagerBasedRLEnv",
kwargs={
"env_cfg_entry_point": f"{__name__}.piper_stack_ik_rel_env_cfg:PiperCubeStackEnvCfg",
"robomimic_bc_cfg_entry_point": f"{agents.__name__}:robomimic/bc_rnn_low_dim.json",
},
disable_env_checker=True,
)
Step 4.Teleoperation & Data Collection
4.1 Teleoperation Configuration
Create the tool directory and script files within the agx_teleop plugin:
cd agx_teleop
mkdir -p scripts/tools
touch scripts/tools/record_demos.py \
scripts/tools/repaly_demos.py
Copy the corresponding IsaacLab source code and modify the imports for tasks. Required sections are listed below:
from isaaclab.envs import DirectRLEnvCfg, ManagerBasedRLEnvCfg
from isaaclab.envs.mdp.recorders.recorders_cfg import ActionStateRecorderManagerCfg
from isaaclab.envs.ui import EmptyWindow
from isaaclab.managers import DatasetExportMode
# import isaaclab_tasks # noqa: F401
import agx_teleop.tasks # noqa: F401
from isaaclab_tasks.utils.parse_cfg import parse_env_cfg
scripts/tools/repaly_demos.py:
if args_cli.enable_pinocchio:
import isaaclab_tasks.manager_based.locomanipulation.pick_place # noqa: F401
import isaaclab_tasks.manager_based.manipulation.pick_place # noqa: F401
# import isaaclab_tasks # noqa: F401
import agx_teleop.tasks # noqa: F401
from isaaclab_tasks.utils.parse_cfg import parse_env_cfg
4.2 Teleoperation Configuration
# First, create the dataset directory
mkdir -p datasets
# Command for teleoperated data collection; keyboard teleoperation used here. Available devices: spacemouse, keyboard, handtracking
python scripts/tools/record_demos.py \
--task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
--device cpu --teleop_device keyboard \
--dataset_file ./datasets/dataset.hdf5 \
--num_demos 10
# Replay the collected dataset
python scripts/tools/replay_demos.py \
--task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
--device cpu --dataset_file ./datasets/dataset.hdf5
Keyboard Teleoperation Method:
Keyboard Controller for SE(3): Se3Keyboard
Reset all commands: R
Toggle gripper (open/close): K
Move arm along x-axis: W/S
Move arm along y-axis: A/D
Move arm along z-axis: Q/E
Rotate arm along x-axis: Z/X
Rotate arm along y-axis: T/G
Rotate arm along z-axis: C/V
Q & A
Q1: Why is the End-Effector (EE) offset set to [0.0, 0.0, 0.14] in the IK configuration?
A: This offset accounts for the physical length of the PiPER gripper. It shifts the Tool Center Point (TCP) from the gripper_base link to the actual center of the fingertips. Without this body_offset, the Inverse Kinematics (IK) controller would attempt to align the robot’s “wrist” with the target cube, causing the physical fingers to overshoot and collide with the objects.
Q2: How does the workflow ensure the quality and validity of the recorded .hdf5 datasets?
A: The workflow employs a two-layer validation system:
- Automated Filtering: The
record_demos.pyscript uses DatasetExportMode.EXPORT_SUCCEEDED_ONLY, meaning only episodes that trigger the success termination criteria (cubes stacked for a set number of steps) are saved. - Manual Verification: The
replay_demos.pyscript allows developers to re-simulate recorded actions to confirm that the recorded trajectories consistently result in a successful task outcome before starting training.
Q3: What is the reasoning behind recommending --device cpu during the teleoperation recording phase?
A: Teleoperation is highly sensitive to input latency. When recording demonstrations, the GPU is often heavily taxed by real-time rendering and physics calculations. Running the simulation logic on the CPU can reduce “input lag” and jitter, providing the human operator with a smoother control experience, which leads to higher-quality, more natural trajectory data for imitation learning.
Q4:In piper_stack_ik_rel_env_cfg.py, why choose the relative pose mode with use_relative_mode=True?
A: The Relative Mode is more intuitive for human teleoperation: the input from the keyboard or joystick is mapped to “increments of the current end position” rather than “absolute coordinates”. During training, relative control can also reduce the difficulty for the model to learn spatial features, as it focuses on motion trends rather than fixed global coordinates, which helps improve the model’s generalization ability across different environmental positions.
Q5: Why is the --fix-base parameter required during the URDF to USD conversion process?
A: PiPER robotic arms are usually fixed on workbenches. In Isaac Sim/Isaac Lab, if the robot base is not fixed (Fix), the robotic arm will displace or tip over in the virtual space due to reaction forces when performing high-dynamic actions. The --fix-base parameter rigidly connects the base_link in the URDF to the global coordinate system (World Frame) of the simulation world, ensuring the stability of the robotic arm base.
Have Question?
If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.
1 post - 1 participant
https://discourse.openrobotics.org/t/isaac-lab-teleoperation-and-data-collection-workflow-piper-robotic-arm-for-embodied-ai/55223