# rl_training **Repository Path**: coder_faust/rl_training ## Basic Information - **Project Name**: rl_training - **Description**: No description available - **Primary Language**: Unknown - **License**: BSD-3-Clause - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2026-11-02 - **Last Updated**: 2026-11-02 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # rl_training [![IsaacSim](https://img.shields.io/badge/IsaacSim-5.1.0-silver.svg)](https://docs.omniverse.nvidia.com/isaacsim/latest/overview.html) [![Isaac Lab](https://img.shields.io/badge/IsaacLab-2.3.2-silver)](https://isaac-sim.github.io/IsaacLab) [![RSL-RL](https://img.shields.io/badge/RSL--RL-5.0.1-silver)](https://github.com/leggedrobotics/rsl_rl) [![Discord](https://img.shields.io/badge/-Discord-5865F2?style=flat&logo=Discord&logoColor=white)](https://discord.gg/gdM9mQutC8) [![Python](https://img.shields.io/badge/python-3.11-blue.svg)](https://docs.python.org/3/whatsnew/3.11.html) [![Linux platform](https://img.shields.io/badge/platform-linux--64-orange.svg)](https://releases.ubuntu.com/22.04/) [![License](https://img.shields.io/badge/license-BSD%203--Clause-blue.svg)](https://opensource.org/license/bsd-3-clause) ## Overview **rl_training** is a RL training library for deeprobotics robots, based on IsaacLab. The table below lists all available environments: | Robot Model | Environment Name (ID) | Screenshot | Tutorial Videos | |---------------------|------------------------------------------------------------|------------|-----------------| | [Deeprobotics Lite3](https://www.deeprobotics.cn/robot/index/product1.html) | Rough-Deeprobotics-Lite3-v0 | Lite3 | [Bilibili](https://b23.tv/UoIqsFn) / [YouTube](https://youtube.com/playlist?list=PLy9YHJvMnjO0X4tx_NTWugTUMJXUrOgFH&si=pjUGF5PbFf3tGLFz) | | [Deeprobotics M20](https://www.deeprobotics.cn/robot/index/m20.html) | Rough-Deeprobotics-M20-v0 | deeprobotics_m20 | [Bilibili](https://b23.tv/CSEyYvw) / [YouTube](https://youtu.be/5VKMahWWYM4?si=ctktKv-1FmfcdbeZ) | | [Deeprobotics M20S](https://www.deeprobotics.cn/robot/index/m20.html) | Rough-Deeprobotics-M20S-v0 | deeprobotics_m20s | — | | [Deeprobotics DR02](https://www.deeprobotics.cn/robot/index/dr02.html) | Amp-Flat-Deeprobotics-DR02-v0 | deeprobotics_dr02 | [Bilibili](https://b23.tv/658wI0Q) / [YouTube](https://youtu.be/xmptqd-LDUs?si=APUTRh6i8KmLGrmD) | | [Deeprobotics S10](https://www.deeprobotics.cn/robot/index/lynxs10.html) | Rough-Deeprobotics-S10-v0 | deeprobotics_s10 | — | > [!NOTE] > If you want to deploy policies in mujoco or real robots, please use the corresponding deploy repo in [Deep Robotics Github Center](https://github.com/DeepRoboticsLab). ## Contribution Everyone is welcome to contribute to this repo. If you discover a bug or optimize our training config, just submit a pull request and we will look into it. ## Installation - Install **Isaac Sim 5.1.0 and Isaac Lab 2.3.2 with Python 3.11** using the [installation guide](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html). These are the versions tested with this repository; select these versions when following the guide. The commands below assume Linux and Bash with an activated Isaac Lab Conda environment. - Clone this repository separately from the Isaac Lab installation (i.e. outside the `IsaacLab` directory): ```bash git clone --recurse-submodules https://github.com/DeepRoboticsLab/rl_training.git cd rl_training # Also repairs an existing clone made without --recurse-submodules. git submodule update --init --recursive ``` - Activate your Isaac Lab environment (replace `deep-robotics-humanoid` with your environment name), then install the library. This also installs **RSL-RL 5.0.1** and **PyBullet 3.2.7**, required by DR02 AMP training. ```bash conda activate deep-robotics-humanoid python -m pip install -e source/rl_training ``` - Configure the environment's C++ runtime before launching Isaac Sim: ```bash python scripts/tools/setup_conda_runtime.py conda deactivate conda activate deep-robotics-humanoid ``` The script checks that Conda's `libstdc++.so.6` provides `CXXABI_1.3.15` and installs activation/deactivation hooks in that environment. This prevents Isaac Sim from loading an older system runtime that breaks Conda's ICU/SQLite imports. It preserves any existing `LD_PRELOAD` value and restores it on deactivation. Re-running the script is safe. If it reports an outdated or missing runtime, run `conda install -c conda-forge "libstdcxx-ng>=15"`, then retry the script. System libraries are unchanged. - Verify that the extension is correctly installed by running the following command to print all the available environments in the extension: ```bash python scripts/tools/list_envs.py ``` - Verify DR02 with a short training run before starting a full run: ```bash python scripts/reinforcement_learning/rsl_rl/train.py \ --task=Amp-Flat-Deeprobotics-DR02-v0 --headless --num_envs=16 --max_iterations=1 ``` It should complete one learning iteration and save `model_0.pt` under `logs/rsl_rl/dr02_amp//`.
Setup as Omniverse Extension (Optional, click to expand) We provide an example UI extension that will load upon enabling your extension defined in `source/rl_training/rl_training/ui_extension_example.py`. To enable your extension, follow these steps: 1. **Add the search path of your repository** to the extension manager: - Navigate to the extension manager using `Window` -> `Extensions`. - Click on the **Hamburger Icon** (☰), then go to `Settings`. - In the `Extension Search Paths`, enter the absolute path to `rl_trainingb/source` - If not already present, in the `Extension Search Paths`, enter the path that leads to Isaac Lab's extension directory directory (`IsaacLab/source`) - Click on the **Hamburger Icon** (☰), then click `Refresh`. 2. **Search and enable your extension**: - Find your extension under the `Third Party` category. - Toggle it to enable your extension.
## Try examples Deeprobotics Lite3: ```bash # Train python scripts/reinforcement_learning/rsl_rl/train.py --task=Rough-Deeprobotics-Lite3-v0 --headless # Play python scripts/reinforcement_learning/rsl_rl/play.py --task=Rough-Deeprobotics-Lite3-v0 --num_envs=10 ``` Deeprobotics M20: ```bash # Train python scripts/reinforcement_learning/rsl_rl/train.py --task=Rough-Deeprobotics-M20-v0 --headless # Play python scripts/reinforcement_learning/rsl_rl/play.py --task=Rough-Deeprobotics-M20-v0 --num_envs=10 ``` Deeprobotics M20S: ```bash # Train python scripts/reinforcement_learning/rsl_rl/train.py --task=Rough-Deeprobotics-M20S-v0 --headless # Play python scripts/reinforcement_learning/rsl_rl/play.py --task=Rough-Deeprobotics-M20S-v0 --num_envs=10 ``` Deeprobotics S10: ```bash # Train python scripts/reinforcement_learning/rsl_rl/train.py --task=Rough-Deeprobotics-S10-v0 --headless # Play python scripts/reinforcement_learning/rsl_rl/play.py --task=Rough-Deeprobotics-S10-v0 --num_envs=10 ``` Deeprobotics DR02: ```bash # Train python scripts/reinforcement_learning/rsl_rl/train.py --task=Amp-Flat-Deeprobotics-DR02-v0 --headless # Play python scripts/reinforcement_learning/rsl_rl/play.py --task=Amp-Flat-Deeprobotics-DR02-v0 --num_envs=10 ``` > [!NOTE] > If you want to control a **SINGLE ROBOT** with the keyboard during playback, add `--keyboard` at the end of the play script. > > ``` > Key bindings: > ====================== ========================= ======================== > Command Key (+ve axis) Key (-ve axis) > ====================== ========================= ======================== > Move along x-axis Numpad 8 / Arrow Up Numpad 2 / Arrow Down > Move along y-axis Numpad 4 / Arrow Right Numpad 6 / Arrow Left > Rotate along z-axis Numpad 7 / Z Numpad 9 / X > ====================== ========================= ======================== > ``` * Record video of a trained agent (requires installing `ffmpeg`), add `--video --video_length 200` * Play/Train with 32 environments, add `--num_envs 32` * Play on specific folder or checkpoint, add `--load_run run_folder_name --checkpoint model.pt` * Resume training from folder or checkpoint, add `--resume --load_run run_folder_name --checkpoint model.pt` ## Multi-gpu acceleration * To train with multiple GPUs, use the following command, where --nproc_per_node represents the number of available GPUs: ```bash python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task= --headless python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task=Rough-Deeprobotics-Lite3-v0 --headless --distributed --num_envs=2048 ``` * Note: each gpu will have the same number of envs specified in the config, to use the previous total number of envs, devide it by the number of gpus. * To scale up training beyond multiple GPUs on a single machine, it is also possible to train across multiple nodes. To train across multiple nodes/machines, it is required to launch an individual process on each node. For the master node, use the following command, where --nproc_per_node represents the number of available GPUs, and --nnodes represents the number of nodes: ```bash python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/rsl_rl/train.py --task= --headless --distributed ``` Note that the port (`5555`) can be replaced with any other available port. For non-master nodes, use the following command, replacing `--node_rank` with the index of each machine: ```bash python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/rsl_rl/train.py --task= --headless --distributed ``` ## Tensorboard To view tensorboard, run: ```bash tensorboard --logdir=logs ``` ## Export Policy to ONNX (without Isaac Sim) Export a trained checkpoint to ONNX directly from the `.pt` file — no Isaac Sim or environment setup required: ```bash # Lite3 python scripts/tools/export_onnx_fast.py \ --checkpoint_path logs/rsl_rl/deeprobotics_lite3_rough//model_5000.pt \ --robot lite3 \ --output_path exported/lite3_policy.onnx # M20 python scripts/tools/export_onnx_fast.py \ --checkpoint_path logs/rsl_rl/deeprobotics_m20_rough//model_5000.pt \ --robot m20 \ --output_path exported/m20_policy.onnx ``` Robot metadata (joint names, stiffness/damping, default positions, action scales) is embedded in the ONNX file as model properties. Add `--no_metadata` to skip this. ## Compare Training Runs Diff the `agent.yaml` and `env.yaml` configs between two runs (saved automatically to `params/` by `train.py`): ```bash python scripts/tools/compare_runs.py \ logs/rsl_rl/deeprobotics_lite3_rough/ \ logs/rsl_rl/deeprobotics_lite3_rough/ ``` ## Troubleshooting ### DR02 startup errors - **`CXXABI_1.3.15 not found`, `omni.kit has no attribute test`, or `cannot import name tests`:** the latter errors can cascade from the C++ runtime failure. Run `python scripts/tools/setup_conda_runtime.py` in your activated Conda environment, then deactivate/reactivate it and restart training. An already running Python process cannot pick up the new runtime. - **Missing `DR02-pro.urdf` or `DR02-pro_fix_joints.urdf`:** run `git submodule update --init --recursive` from the repository root. - **Missing `pybullet_utils` or incompatible RSL-RL:** re-run `python -m pip install -e source/rl_training` in the environment used for training. The package declares the tested versions. The training scripts use the repository's config converter for RSL-RL 5, so the missing Isaac Lab `handle_deprecated_rsl_rl_cfg` helper is not required. Some startup warnings remain with the tested setup. A missing viewport is expected in headless mode. The DR02 URDF contains fixed sensor links without inertia; the importer assigns small inertias and adjusts joint axes. These messages do not prevent training, but changing the robot's inertial properties should be based on measured model data. CPU powersave and GPU peer-to-peer messages concern machine performance, not Python imports. ### Pylance Missing Indexing of Extensions In some VsCode versions, the indexing of part of the extensions is missing. In this case, add the path to your extension in `.vscode/settings.json` under the key `"python.analysis.extraPaths"`. **Note: Replace `` with your own IsaacLab path.** ```json { "python.languageServer": "Pylance", "python.analysis.extraPaths": [ "${workspaceFolder}/source/rl_training", "//source/isaaclab", "//source/isaaclab_assets", "//source/isaaclab_mimic", "//source/isaaclab_rl", "//source/isaaclab_tasks", ] } ``` ### Clean USD Caches Temporary USD files are generated in `/tmp/IsaacLab/usd_{date}_{time}_{random}` during simulation runs. These files can consume significant disk space and can be cleaned by: ```bash rm -rf /tmp/IsaacLab/usd_* ``` ## Acknowledgements The project uses some code from the following open-source code repositories: - [fan-ziqi/robot_lab](https://github.com/fan-ziqi/robot_lab)