VnRoboVnRobo
AboutPricingBlogContact
🇻🇳VISign InStart learning
🇻🇳VI
VnRobo logoVnRobo logo

Learn Physical AI and experiment in your browser. Connect your fleet when you have real robots.

Product

  • Features
  • Pricing
  • Knowledge Base
  • Services

Company

  • About Us
  • Blog
  • Contact

Legal

  • Privacy Policy
  • Terms of Service

© 2026 VnRobo. All rights reserved.

Made with♥in Vietnam
VnRoboVnRobo
AboutPricingBlogContact
🇻🇳VISign InStart learning
🇻🇳VI
  1. Home
  2. Blog
  3. Dexterous Manipulation: Teaching Robot Hands
manipulationdexterousmanipulationtactile-sensingrobot-hand

Dexterous Manipulation: Teaching Robot Hands

In-hand rotation, tool use, DexGraspNet and tactile sensing — complete guide to dexterous manipulation with multi-finger robot hands.

Nguyen Anh TuanMarch 18, 20267 min readUpdated: Nov 14, 2026
Dexterous Manipulation: Teaching Robot Hands

Beyond Parallel-Jaw Gripper

In previous posts of this series, I focused mainly on parallel-jaw grippers — two fingers that open/close. They're the most common because simple, cheap, and sufficient for many tasks (pick-and-place, bin picking).

But humans have 5 fingers with 20+ DOF, enabling delicate manipulation: rotating a ball, using tools, opening a bottle with one hand. Dexterous manipulation with multi-finger robot hands is the next frontier — and one of the hardest problems in robotics.

This post covers: hardware (Allegro, LEAP, Shadow Hand), core problems (in-hand manipulation, tool use), datasets (DexGraspNet), tactile sensing, and state-of-the-art methods.

See Tactile Sensing for Manipulation for deep dive on tactile sensors.

Multi-finger robot hand — frontier of dexterous manipulation
Multi-finger robot hand — frontier of dexterous manipulation

Hardware: Popular Robot Hands

Allegro Hand

Allegro Hand from Wonik Robotics is most popular in research:

  • 16 DOF: 4 fingers x 4 joints each
  • Torque-controlled: allows precise force control
  • Price: ~15,000 USD
  • Ecosystem: ROS 2 driver, MuJoCo/Isaac Sim models available
  • Used by: Stanford IRIS, CMU, UC Berkeley

LEAP Hand

LEAP Hand from Carnegie Mellon is low-cost alternative:

  • 16 DOF: similar to Allegro but uses Dynamixel servos
  • Price: ~2,000 USD (7x cheaper than Allegro)
  • Open-source: CAD files, firmware, software all public
  • Anthropomorphic: design more human-like than Allegro
  • Limitation: lower torque, no torque sensing built-in

Shadow Dexterous Hand

Shadow Hand is gold standard — most human-like:

  • 24 DOF: 5 fingers + thumb opposition
  • Tactile sensors: BioTac fingertips (optional)
  • Price: 100,000+ USD
  • Used by: OpenAI (Rubik's Cube), Google DeepMind

Comparison

Criterion Allegro LEAP Shadow
DOF 16 16 24
Price ~15K USD ~2K USD ~100K+ USD
Torque sensing Yes No Yes
Tactile No (add-on) No BioTac (optional)
Open-source No Yes No
Sim models MuJoCo, Isaac MuJoCo MuJoCo, Isaac
Best for Research balance Education, budget Top-tier research

Core Problem: In-Hand Manipulation

In-Hand Object Rotation

In-hand rotation is the classic benchmark: robot hand holds object (cube, sphere) and rotates it to target orientation without dropping. Sounds simple but extremely hard:

  • High-dimensional action space: 16+ joints simultaneously
  • Unstable contacts: object can slip or drop anytime
  • Sim-to-real gap: friction, deformation differs between simulation and reality

OpenAI's Rubik's Cube (2019) was a milestone: Shadow Hand solved Rubik's Cube after training RL in simulation with Automatic Domain Randomization (ADR). 13,000+ years simulated experience, transferred zero-shot to real robot.

Modern approach (2024-2026): RL in Isaac Lab with 4,096+ parallel environments. Unitree uses this for their dexterous hands, and Google DeepMind uses it for in-hand manipulation research.

Tool Use

Ability to use tools distinguishes human hands. Robot dexterous manipulation aims for:

  • Hammer grasping and striking: hold hammer, strike target
  • Screwdriver: hold and twist tool
  • Scissors: coordinate 2 fingers for cutting

Current methods use keypoint-based representations: define important points on tool and hand, learn policy to align them. Stanford and UC Berkeley lead here.

DexGraspNet: Large-Scale Dexterous Grasp Dataset

DexGraspNet (Wang et al., 2023) is the largest dexterous grasping dataset:

  • 1.32 million grasps for 5,355 objects from 133+ categories
  • Generated via differentiable force closure optimization in Isaac Gym
  • Each grasp validated by physics simulation
  • For Shadow Hand (also LEAP hand version)

Using DexGraspNet

python
# Load DexGraspNet data (simplified)
import numpy as np

# Each object has ~200+ diverse grasps
grasp_data = np.load("dexgraspnet/core-bottle-1a7ba1f4c892e2da30711cdbdbc73e71.npy",
                      allow_pickle=True).item()

# Each grasp contains:
# - hand_pose: (22,) — wrist 6D + 16 joint angles
# - hand_qpos: (16,) — joint positions
# - target_qpos: (16,) — target joint positions for grasping
# - score: float — grasp quality (force closure based)

print(f"Number of grasps: {len(grasp_data['grasps'])}")
print(f"Object category: {grasp_data['category']}")

DexGraspNet 2.0

DexGraspNet 2.0 (2024) expands to:

  • Cluttered scenes: 8,270 scenes with multiple objects
  • 427 million grasp labels for LEAP hand
  • Generative model to synthesize grasps for unseen objects

Tactile Sensing: "Touch" for Robots

Why Tactile Matters for Dexterous Manipulation?

Vision shows where object is, but tactile shows what it feels like: force, slip detection, surface texture. For dexterous manipulation, tactile is essential:

  • Detect slip before object drops
  • Estimate force to avoid breaking fragile objects
  • Determine material (metal vs plastic vs fabric)

Types of Tactile Sensors

Vision-based (GelSight, DIGIT):

  • Camera + elastomer: surface deformation -> image -> contact geometry
  • High resolution (sub-mm), cheap (~300 USD/sensor)
  • Medium force resolution

Capacitive/resistive (BioTac):

  • Precise force, fast response
  • Expensive (~1,000+ USD/fingertip)
  • Low spatial resolution

Piezoelectric (tuxedo-labs):

  • Extremely fast response (microsecond)
  • Good for vibration/slip detection
  • Limited spatial resolution

DIGIT Sensor for Research

DIGIT from Meta AI is most popular in research:

python
# Read DIGIT sensor data
import digit_interface

digit = digit_interface.Digit("D00123", resolution=(320, 240))
digit.connect()

# Get frame from sensor (contact geometry)
frame = digit.get_frame()  # (240, 320, 3) RGB image

# Contact detection
is_contact = detect_contact(frame, baseline_frame)

# Force estimation (if calibrated)
force_estimate = estimate_force(frame, calibration_model)

Tactile sensing and sim-to-real transfer for dexterous manipulation
Tactile sensing and sim-to-real transfer for dexterous manipulation

State-of-the-Art Methods

RL + Sim-to-Real (Dominant Approach)

Most common method for dexterous manipulation:

  1. Setup in Isaac Lab: robot hand + object, reward function
  2. Train PPO/SAC with 4,096+ parallel environments
  3. Domain randomization: friction, mass, sensor noise, hand dimensions
  4. Deploy zero-shot to real robot
python
# Reward function for in-hand rotation (simplified)
def reward_function(env):
    # Orientation error between current and target
    rot_error = quaternion_distance(
        env.object_quat, env.target_quat
    )

    # Bonus when achieving target
    success_bonus = 10.0 if rot_error < 0.1 else 0.0

    # Penalty for dropping
    drop_penalty = -5.0 if env.object_pos[2] < 0.3 else 0.0

    # Encourage contact between finger and object
    fingertip_reward = -0.1 * fingertip_to_object_distance(env)

    return -rot_error + success_bonus + drop_penalty + fingertip_reward

Diffusion Policy for Dexterous

2025-2026 trend: use Diffusion Policy instead of RL. Advantage: collect data via human teleoperation (more natural), no reward engineering.

Stanford IRIS showed Diffusion Policy can learn in-hand reorientation from 100 human demos, achieving comparable performance to RL after thousands of hours sim training.

Teacher-Student Framework

Effective approach: train teacher policy in sim with privileged information (ground truth object pose, contact forces), then distill to student policy using only real robot info (images, joints, tactile).

code
Teacher (sim, privileged):
  Input: object pose + contact forces + joint pos
  Output: action
  Method: RL (PPO) with full state

Student (real-world compatible):
  Input: images + joint pos + tactile
  Output: action
  Method: BC from teacher demonstrations

Challenges and Future Directions

Sim-to-Real Still the Biggest Problem

Contact physics in simulation still isn't 100% accurate. MuJoCo 3.x with convex optimization contact solver is much better, but deformable objects (fabric, rope) remain open challenge.

Hardware is Still a Bottleneck

Current robot hands are:

  • Expensive: Shadow Hand 100K+ USD
  • Fragile: break easily on impact
  • Slow: response time slower than human hand
  • Limited tactile: most lack integrated tactile sensors

LEAP Hand ($2K, open-source) is reducing barriers, but needs better durability and torque.

Bi-manual Dexterous

Combining 2 robot hands (32 DOF total) multiplies complexity exponentially. This is frontier of frontier — see Part 6 for more.

Resources

  • DexGraspNet paper: arXiv:2210.02697
  • DexGraspNet 2.0: arXiv:2410.23004
  • LEAP Hand: https://leaphand.com/
  • DIGIT sensor: https://digit.ml/
  • Isaac Lab dexterous examples: github.com/isaac-sim/IsaacLab

Next in Series

  • Part 6: Bimanual Manipulation: Teaching Robots to Use Both Arms — ALOHA, Mobile ALOHA, ACT for bimanual
  • Part 7: Building Manipulation Systems with LeRobot — End-to-end deployment

Tool recommendations

VLA train/deploy stack

Train on cloud/workstation, then deploy optimized models to Jetson or the robot computer.

Cloud GPU for VLA / policy training Use for imitation learning, diffusion policies, RL, and robotics model fine-tuning. View cloud GPU → NVIDIA Jetson Orin NX / Orin Nano Edge deployment hardware for perception, logging, and optimized inference. View Jetson → Hugging Face / robotics dataset hosting Host datasets, checkpoints, and model cards for cleaner LeRobot/VLA workflows. View platform →

Related Articles

  • VLA for Manipulation: RT-2, Octo, pi0 — Part 4 of this series
  • Tactile Sensing for Manipulation — Deep dive on tactile sensors
  • Robot Grasping 101: Analytical to Learning-Based — Part 1 of this series
  • Sim-to-Real Transfer: Train in Simulation, Deploy in Reality — Domain randomization for dexterous
NT

Nguyễn Anh Tuấn

Robotics & AI Engineer. Building VnRobo — sharing knowledge about robot learning, VLA models, and automation.

Explore VnRobo

Fleet MonitoringROS 2 IntegrationAMR Solutions
manipulation-masterclass — Part 5/7
← VLA for Manipulation: RT-2, Octo, pi0 in PracticeBimanual Manipulation: Teaching Robots to Use Both Arms →

Related Posts

Tutorial
Bimanual Manipulation: Teaching Robots to Use Both Arms
bimanualmanipulationalohaPart 6
manipulation

Bimanual Manipulation: Teaching Robots to Use Both Arms

ALOHA hardware, Mobile ALOHA, ACT for bimanual tasks, data collection tips and LeRobot SO-100 dual arm — complete guide to bimanual manipulation.

3/22/20267 min read
NT
Deep Dive
Diffusion Policy in Practice: From Theory to Code
diffusion-policymanipulationddpmPart 3
manipulation

Diffusion Policy in Practice: From Theory to Code

DDPM recap, Diffusion Policy architecture (CNN vs Transformer), implementation with LeRobot, and benchmark results — complete practical guide.

3/16/20267 min read
NT
Research
VLA for Manipulation: RT-2, Octo, pi0 in Practice
vlamanipulationrt-2Part 4
manipulation

VLA for Manipulation: RT-2, Octo, pi0 in Practice

Vision-Language-Action models changing manipulation — from Google's RT-2 to pi0 by Physical Intelligence, with fine-tuning and real-world limitations.

3/6/20268 min read
NT
VnRobo logoVnRobo logo

Learn Physical AI and experiment in your browser. Connect your fleet when you have real robots.

Product

  • Features
  • Pricing
  • Knowledge Base
  • Services

Company

  • About Us
  • Blog
  • Contact

Legal

  • Privacy Policy
  • Terms of Service

© 2026 VnRobo. All rights reserved.

Made with♥in Vietnam