Isaac Teleop + GR00T N1.7 + LeRobot v0.6: Full Collect, Fine-tune & Deploy Pipeline

On July 6, 2026, NVIDIA and Hugging Face jointly announced what the robotics community had been waiting for: a fully closed robot learning loop in a single ecosystem. Isaac Teleop for data collection, LeRobot v0.6 for training, GR00T N1.7 for inference — all connected seamlessly, with no manual format conversion and no glue code.

This is a hands-on walkthrough of that complete pipeline.

The Big Picture: Why July 2026 Is a Turning Point

Before diving into each step, let's understand why this matters.

The old workflow: To train a manipulation policy you had to:

1. Collect teleop data with some tool (UMI, a custom ROS2 node, ACT teleop)
2. Convert to HDF5 or RLDS format
3. Write a custom data loader for your training framework
4. Train the model
5. Write a separate inference node
6. Deploy to the robot — often rewriting from scratch

Each step was its own mini-project. A team could spend 2–4 weeks just building the pipeline before training a single model.

The new workflow: Isaac Teleop exports directly to LeRobot dataset format. LeRobot v0.6 reads that dataset, fine-tunes GR00T N1.7, and lerobot-rollout deploys the policy to the same robot. Inputs and outputs share a unified format. This is the "closed loop" that LeRobot v0.6's release title — "Imagine, Evaluate, Improve" — is about: fast iteration because you never rebuild the pipeline between runs.

Isaac Teleop data collection exports directly to LeRobot dataset format — source: NVIDIA/HuggingFace blog

GR00T N1.7: Quick Architecture Summary

For an in-depth breakdown, see GR00T N1.7 and EgoScale: Fine-tune from Zero to Deploy. Here's the essential context:

Action Cascade architecture (3B parameters):

• System 2 — VLM: Cosmos-Reason2-2B (Qwen3-VL based) processes camera images and natural-language task instructions, producing high-level action tokens. This is the "thinking" layer — task decomposition, multi-step planning.
• System 1 — Diffusion Transformer: A 32-layer DiT takes the VLM's action tokens plus live robot state and denoises them into precise motor commands. This is the "reflex" layer — fast, fine-grained control.

EgoScale pre-training: 20,854 hours of human egocentric video (first-person head camera + wrist camera), covering 20+ task categories from manufacturing, healthcare, and retail to home environments. This is why N1.7 generalizes so well from very few fine-tuning demos — it has already "seen" human hands manipulating objects for tens of thousands of hours.

LIBERO benchmark: 96.5% average across four task suites (up from 87% with N1.5).

Available on HuggingFace: nvidia/GR00T-N1.7-3B and nvidia/GR00T-H-N1.7 (humanoid whole-body variant).

Step 0: Hardware and Software Requirements

Minimum hardware for training:

• GPU: RTX 4090 (24 GB VRAM) for single-GPU training with small batch size
• Recommended GPU: A100 40 GB or H100 80 GB
• RAM: 64 GB+
• Disk: 100 GB+ for datasets and checkpoints
• Robot arm: SO-101 follower + SO-101 leader (leader-follower setup) or SO-100

Software:

• Ubuntu 22.04 or 24.04 (Linux required — CUDA training is not supported on Windows)
• Python 3.12
• CUDA 12.x
• uv (package manager — significantly faster than pip)
• ffmpeg (video processing in datasets)

Step 1: Install the Environment

LeRobot v0.6 uses uv instead of pip for dependency management. The advantages: proper virtual environment isolation, fast installs, and no dependency conflicts.

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc  # or restart terminal

# Install ffmpeg
sudo apt update && sudo apt install -y ffmpeg

# Clone LeRobot
git clone https://github.com/huggingface/lerobot.git
cd lerobot

# Create a Python 3.12 virtual environment
uv venv --python 3.12
source .venv/bin/activate

# Install LeRobot with required extras:
# groot: GR00T model support
# training: training dependencies
# feetech: SO-101 servo driver
# viz: visualization tools
uv pip install -e ".[groot,training,feetech,viz]"

Note: The feetech extra is only needed for SO-101. For other arms (WidowX, Panda), substitute the appropriate extra. Use uv pip install -e ".[groot,training,viz]" for a hardware-agnostic setup.

Verify installation:

python -c "import lerobot; import gr00t; print('OK')"

Step 2: Set Up Isaac Teleop for Data Collection

Isaac Teleop is NVIDIA's teleoperation framework. It supports several input devices:

• SO-101 Leader arm (recommended): Leader-follower setup where your hand moves the leader arm and the follower arm mirrors it. Highest quality data because motion maps naturally to robot kinematics.
• SpaceMouse (3Dconnexion Compact/Wireless): 6-DoF control, easier to set up but typically produces less smooth data (10–20% lower policy performance).
• XR Controller (Meta Quest, Vision Pro): Immersive; good for whole-body humanoid teleoperation.

SO-101 Leader-Follower Setup (recommended for manipulation)

# Install Isaac Teleop (same virtualenv as LeRobot)
uv pip install isaac-teleop

# Identify serial port
ls /dev/ttyUSB*  # usually /dev/ttyUSB0 or /dev/ttyUSB1

# One-time calibration (run once per new setup)
uv run python -m lerobot.scripts.control_robot calibrate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyUSB0 \
  --robot.id=my_so101

# Test connection
uv run python -m lerobot.scripts.control_robot teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyUSB0

Step 3: Collect Demonstration Data

This is the most important step for policy quality. Rule of thumb: 50–100 demonstrations for a simple task (pick-and-place), 100–200 for complex tasks (stacking, peg insertion, folding).

Tips for high-quality data collection:

• Each demonstration should be 3–10 seconds
• Vary the starting position of objects (don't always place the target object in the exact same spot)
• Move slowly and deliberately — noisy data from rushing trains a noisy policy
• Include failure recovery if possible

export ROBOT_PORT=/dev/ttyUSB0
export DATASET="your-hf-username/pick-place-demo-50"

# Collect 50 demonstrations
uv run python examples.isaac_teleop_to_so101.record \
  --robot.type=so101_follower \
  --robot.port=$ROBOT_PORT \
  --dataset.repo_id=$DATASET \
  --dataset.num_episodes=50 \
  --dataset.push_to_hub=true

--dataset.push_to_hub=true automatically uploads to the HuggingFace Hub after collection. To keep data local:

uv run python examples.isaac_teleop_to_so101.record \
  --robot.type=so101_follower \
  --robot.port=$ROBOT_PORT \
  --dataset.repo_id=$DATASET \
  --dataset.push_to_hub=false \
  --dataset.root=data/

The resulting dataset follows the standard LeRobot format:

data/
  your-hf-username/pick-place-demo-50/
    meta/
      info.json          # fps, embodiment, camera names
      tasks.jsonl        # language labels per episode
    data/
      chunk-000/
        episode_000000.parquet
        ...
    videos/
      chunk-000/
        observation.images.top/
          episode_000000.mp4
          ...

Preview the dataset:

uv run python -m lerobot.scripts.visualize_dataset \
  --repo-id $DATASET \
  --episode-index 0

Step 4: Fine-tune GR00T N1.7 with LeRobot v0.6

LeRobot v0.6 integrates GR00T N1.7 through the groot policy type. The fine-tuning command is significantly simpler than using the Isaac-GR00T repo directly:

export DATASET="your-hf-username/pick-place-demo-50"
export OUTPUT_DIR="outputs/groot-n17-pick-place"

uv run lerobot-train \
  --dataset.repo_id=$DATASET \
  --policy.type=groot \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B \
  --steps=20000 \
  --batch_size=64 \
  --output_dir=$OUTPUT_DIR

Key parameter reference:

Monitor training with Weights & Biases:

wandb login

uv run lerobot-train \
  --dataset.repo_id=$DATASET \
  --policy.type=groot \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B \
  --steps=20000 \
  --batch_size=64 \
  --output_dir=$OUTPUT_DIR \
  --wandb.enable=true \
  --wandb.project=groot-manipulation

Estimated training time:

• RTX 4090: ~4–6 hours for 20k steps at batch 32
• A100 40 GB: ~1.5–2 hours at batch 64
• H100 80 GB: ~45 minutes at batch 128

The best checkpoint is saved at $OUTPUT_DIR/checkpoints/last/ and automatically selected based on validation loss.

Alternative: Use the Isaac-GR00T Repo Directly

If you need more control (custom embodiment configs, multi-GPU FSDP training, TensorRT export), use the original repo:

git clone https://github.com/NVIDIA/Isaac-GR00T.git
cd Isaac-GR00T
conda create -n gr00t python=3.10 -y && conda activate gr00t
uv pip install -e .

CUDA_VISIBLE_DEVICES=0 uv run python gr00t/experiment/launch_finetune.py \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path /path/to/dataset \
  --embodiment-tag SO101_LEROBOT \
  --modality-config-path configs/modality_config/so101_lerobot.json \
  --num-gpus 1 \
  --output-dir outputs/gr00t-so101 \
  --max-steps 20000 \
  --global-batch-size 64

Available embodiment tags: UNITREE_G1, LIBERO_PANDA, OXE_WIDOWX, SO101_LEROBOT, and more. For custom embodiments not yet registered, see getting_started/finetune_new_embodiment.md in the repo.

Step 5: Inference and Deployment

After training, deploy to the real robot with lerobot-rollout:

export MODEL_ID="outputs/groot-n17-pick-place/checkpoints/last"

uv run lerobot-rollout \
  --policy.path=$MODEL_ID \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyUSB0

This script:

1. Loads the policy checkpoint
2. Starts camera streams
3. Runs the inference loop: camera frame → GR00T N1.7 → motor commands
4. Sends commands to the robot at 50 Hz

Accelerate inference with TensorRT (for Jetson Orin or edge GPU deployment):

# Export TensorRT engine (one-time step)
uv run python -m gr00t.export.tensorrt \
  --model-path $MODEL_ID \
  --output-path outputs/groot-tensorrt.engine \
  --batch-size 1

# Rollout with TensorRT
uv run lerobot-rollout \
  --policy.path=$MODEL_ID \
  --policy.backend=tensorrt \
  --policy.engine_path=outputs/groot-tensorrt.engine \
  --robot.type=so101_follower

TensorRT reduces inference latency from ~80 ms to ~20 ms — important for contact-rich tasks requiring fast feedback.

Fine-tuned GR00T N1.7 policy running autonomously on robot — source: NVIDIA/HuggingFace blog

Step 6: Evaluate and Iterate

LeRobot v0.6 includes built-in evaluation tools:

uv run python -m lerobot.scripts.eval \
  --policy.path=$MODEL_ID \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B \
  --robot.type=so101_follower \
  --eval.n_episodes=20 \
  --eval.max_steps=300

Output: success rate, average episode length, and failure mode breakdown (timeout / drop / wrong target).

Improvement loop:

1. Success rate < 60%: collect more demos, focus on object position diversity
2. 60–80%: increase steps (30k → 50k) or add data augmentation

3. 80% but failing on specific cases: collect targeted demos for those failure modes

4. Poor generalization (only works on seen objects): add a second camera (wrist camera)

Benchmark Results

Numbers published by NVIDIA for GR00T N1.7:

The LeRobot integration is parity-tested against the original Isaac-GR00T repo — same inputs, same outputs, same benchmark scores.

The first scaling law for robot dexterity:
Going from 1,000 to 20,000 hours of human egocentric video more than doubles task completion rate in a predictable, consistent way. This is the first time robotics has a scaling law analogous to LLMs — more data = reliably better performance.

GR00T N1.7 Bimanual Dexterous Manipulation

With a full humanoid arm setup (22 DoF), GR00T N1.7 supports finger-level control for contact-rich tasks:

GR00T N1.7 performing a bimanual task with fine-grained finger control — source: NVIDIA GR00T N1.7 blog

Validated embodiments include: Unitree G1, Bimanual YAM, AGIBot Genie 1. With nvidia/GR00T-H-N1.7 (H = humanoid whole-body), you can fine-tune for full-body loco-manipulation tasks directly.

When to Use This Pipeline (and When Not To)

Good fit:

• Robot arm manipulation: pick-and-place, sorting, simple assembly
• Small to medium datasets (50–500 demos)
• Fast iteration without writing infrastructure code
• Already have or planning to buy SO-101 (best ecosystem support)
• Need a commercial license (GR00T N1.7 is fully open commercial)

Think carefully before using:

• Tasks requiring extreme sample efficiency (< 10 demos): consider ACT or ACoT-VLA — diffusion-based VLAs like GR00T need more data than non-diffusion baselines
• Full-body bipedal loco-manipulation: needs GR00T-Sonic or a WBC stack on top
• Latency < 10 ms: even with TensorRT, GR00T DiT inference lands at ~15–20 ms
• CPU-only inference: not feasible — minimum is a mobile GPU like Jetson Orin NX 16 GB

Practical Tips from Running This Pipeline

1. Calibrate carefully before recording: One bad calibration = all 100 demos are off. Spend 15 minutes verifying calibration in teleoperate mode first.

2. Push your dataset to Hub immediately: --dataset.push_to_hub=true doubles as a backup and lets collaborators access the data. LeRobot also auto-caches Hub datasets, preventing redundant re-downloads.

3. Visualize every 10 demos: Catch problems early (gripper not closing fully, wrong camera angle) before spending hours collecting 100 demos you'll need to redo.

4. Sanity-check with --steps=5000 first: Run 5k steps to verify the dataset loads and the loss decreases before committing to a full training run. This saves 4–6 hours when there's a dataset issue.

5. Wrist camera is a game changer: If possible, attach a camera to the wrist. Policies trained with a wrist camera see the contact point directly and generalize significantly better. Isaac Teleop supports multi-camera natively.

For a deeper dive into how the LeRobot dataset format works and how to fine-tune smaller policies like G0-Tiny, see G0-Tiny and LeRobot: Fine-tune Manipulation Policy.

Conclusion

July 2026 marks the first time a robot manipulation VLA pipeline is genuinely accessible: environment setup in 10 minutes, data collection in an afternoon, overnight fine-tuning, rollout the next morning. No PhD in robotics required, no custom CUDA kernels, no bespoke data pipelines.

GR00T N1.7 + LeRobot v0.6 isn't a perfect pipeline for every use case — but it's currently the best starting point for anyone building real-world manipulation capability.

Next steps: try it on your robot, document the failure modes, and contribute your dataset to the Hub. NVIDIA is training GR00T N2.0 — and community data will be part of it.

Related Posts

• GR00T N1.7 and EgoScale: In-Depth Architecture and Fine-tuning Guide
• LeRobot Hands-on: VLA Practice on a Robot Arm from A to Z
• G0-Tiny and LeRobot: Fine-tune Manipulation Policy for Small Robot Arms