Opt2VLA: Force-Aware VLA for Humanoid Whole-Body Manipulation

Most state-of-the-art Vision-Language-Action (VLA) models teach robots to move to the right place — but completely ignore a critical question: how hard should the robot push? When a humanoid picks up a heavy box, slides an object along a shelf, or wipes a table, each task demands a different contact force profile. Without explicit force information, the downstream controller has to guess — and it will either be too gentle (slip, fail) or too aggressive (break the object, lose balance). This is exactly the gap that Opt2VLA from Georgia Tech closes.

Opt2VLA: Force-Aware Vision-Language-Action for Contact-Rich Humanoid Whole-Body Manipulation — Fukang Liu*, Yipu Chen*, Jaehwi Jang, Danfei Xu, Zsolt Kira, Ye Zhao — Georgia Institute of Technology, arXiv 2609.23968 (September 2026).

What "Opt2" Means

The "Opt2" prefix in Opt2VLA stands for Optimization-to — specifically, whole-body Trajectory Optimization (TO). This is the same naming convention as the team's prior work Opt2Skill (RA-L 2025). The key insight: instead of relying on human teleoperation data (which captures motion but not force), Opt2VLA uses a DDP (Differential Dynamic Programming) solver inside Crocoddyl to generate physically grounded trajectories with exact contact force labels. This is the "ground truth" for force that no human demo dataset can provide.

The Core Problem: VLA Lacks a Force Dimension

Imagine teaching someone to wipe a table by only describing the trajectory of their hand — never mentioning "press harder" or "go lighter." They would apply a random, unpredictable force level every time. This is precisely the state of today's VLA models for manipulation.

Specifically, state-of-the-art VLAs like π0 and GR00T-N1 output:

• Motion goals: 3D waypoints for end-effectors (hands, feet)
• Missing: Contact force references

The downstream whole-body controller (WBC) receives these waypoints and must infer the needed force on its own — with no grounding in the task context. For free-space manipulation this is acceptable. For contact-rich tasks (pushing, wiping, pressing), it is a critical blind spot.

Opt2VLA solves this by extending the VLA's action space to include a force reference, and training the VLA to map from language ("gently," "firmly," "strongly") to specific force levels.

Two-Level Architecture

Opt2VLA is a hierarchical system with two levels operating in sequence:

Level 1: VLA Policy (fine-tuned GR00T-N1.7)

The foundation model is GR00T-N1.7 by NVIDIA, fine-tuned with extended inputs and outputs.

Extended inputs:

• Language instruction ("Pick up the box gently")
• Egocentric RGB camera observation
• Proprioceptive state: joint positions, velocities, base pose and velocity
• NEW: Joint torque history (10 timesteps at 200 ms spacing) — an implicit force sensing signal that doesn't require expensive F/T sensors

Extended outputs:

• 3D waypoints for left hand, right hand, and feet (standard WBC interface)
• NEW: Continuous contact force reference for the primary manipulation contact

Level 2: Force-Conditioned RL Controller (FCT)

An asymmetric actor-critic RL policy that receives the VLA's predicted motion goals plus force reference and executes them as joint torques. Three variants are compared:

FCT is the proposed method. The torque supervision is privileged training signal only — it is dropped entirely at inference time.

Four-Step Training Pipeline

Step 1: Data Generation via Trajectory Optimization

Solver: Crocoddyl with DDP (Differential Dynamic Programming).

The optimal control problem is formulated with:

• Floating-base full-body dynamics of the Digit humanoid (48 kg, 30 DOF)
• Contact Jacobians and contact wrench forces
• Coulomb friction cone constraints
• Joint position, velocity, and torque limits

For each task configuration (object position, orientation, prescribed force level), DDP produces a dynamically-feasible trajectory carrying an exact force label. This is something motion capture or teleoperation cannot provide — force is computed from physics, not measured from a human operator.

Step 2: Train FCT RL Controllers

Reward components of the FCT controller (from Table IV in the paper):

The torque tracking reward is the "privileged supervision" — it guides the controller toward joint torques matching the optimized trajectory. This signal is dropped at deployment.

Step 3: Collect Simulation Dataset

Roll out trained FCT controllers in simulation across diverse task configurations. Each sample pairs:

• Visual observation (egocentric RGB)
• Language annotation describing the force level ("gently lift," "firmly push," "strongly wipe")
• Proprioceptive state + torque history
• Ground-truth motion goal + force reference (label)

Step 4: Fine-tune GR00T-N1.7

GR00T-N1.7 is fine-tuned on the simulation dataset, learning to map (vision, language, proprioception) → (motion waypoints + force reference). The authors plan to release this dataset publicly.

Hardware Demos on Digit

Opt2VLA is validated on the Digit humanoid by Agility Robotics:

• Mass: ~48 kg
• DOF: 30 total, 20 actuated joints

Three contact-rich tasks are demonstrated:

Language-Conditioned Force Levels

The robot modulates force in real time as it receives new language commands — demonstrating closed-loop language-conditioned force control on a real humanoid for the first time.

Experimental Results

Simulation (90 episodes, 3 tasks × 3 force levels)

Ablation on surface wiping task:

• MO (baseline): No meaningful force differentiation across levels
• FC: 2.4 ± 2.9 N average force error
• FCT (proposed): 1.7 ± 2.1 N average force error — ~29% improvement

Hardware (Digit, 45 trials)

Surface wiping shows the largest error because it requires the most complex force distribution (cloth deformation, surface irregularities). Even so, 7.0 N within a 12–22 N target range still produces controlled, task-completing manipulation — far better than the MO baseline which cannot differentiate force levels at all.

How to Reproduce

At the time of writing (September 2026), code and dataset are not yet released. However, you can set up the environment:

Environment Setup

# Isaac Lab for simulation
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
./isaaclab.sh --install

# Crocoddyl for trajectory optimization
pip install crocoddyl

# GR00T-N1.7 (NVIDIA)
pip install gr00t
# Or from HuggingFace: nvidia/GR00T-N1.7-3B

Step 1: Trajectory Optimization Data (Crocoddyl)

import crocoddyl
import numpy as np
import pinocchio

# Setup OCP for a contact task (e.g. wiping)
state = crocoddyl.StateMultibody(robot_model)
actuation = crocoddyl.ActuationModelFloatingBase(state)

# Contact model
contact_model = crocoddyl.ContactModelMultiple(state, actuation.nu)
contact_6d = crocoddyl.ContactModel6D(
    state,
    frame_id,
    pinocchio.SE3.Identity(),
    actuation.nu,
    np.array([0., 50.])  # baumgarte stabilization gains
)
contact_model.addContact("contact_ee", contact_6d)

# Force cost: target "strong" = 15N for wiping
force_ref = np.array([0., 0., 15.0, 0., 0., 0.])
force_residual = crocoddyl.ResidualModelContactForce(
    state, frame_id, pinocchio.Force(force_ref), 6, actuation.nu
)
force_cost = crocoddyl.CostModelResidual(state, force_residual)

# Solve with DDP
problem = crocoddyl.ShootingProblem(x0, running_models, terminal_model)
ddp = crocoddyl.SolverDDP(problem)
ddp.solve([], [], max_iters=300)

# Extract ground-truth force trajectory
optimal_forces = [
    m.differential.contacts.contacts["contact_ee"].f
    for m in ddp.problem.runningModels
]

Step 2: FCT RL Training (Isaac Lab)

# Reward function with force + privileged torque tracking
def compute_rewards(env):
    # Standard motion tracking
    pos_reward = torch.exp(-joint_pos_error.pow(2).sum(-1) / 0.25)

    # Contact force tracking — core contribution
    force_error = (measured_contact_force - reference_force).norm(dim=-1)
    force_reward = torch.exp(-force_error.pow(2) / 4.0)

    # Torque supervision from TO (privileged — training only)
    torque_error = (joint_torques - to_reference_torques).pow(2).sum(-1)
    torque_reward = torch.exp(-torque_error / 10.0)

    return {
        "pos": 3.0 * pos_reward,
        "force": 2.0 * force_reward,
        "torque_supervised": 2.0 * torque_reward,
        "smoothness": -3.0 * action_rate_penalty,
    }

Step 4: Fine-tune GR00T-N1.7

from gr00t.model import GR00TN17
from gr00t.data import ForceAwareDataset

dataset = ForceAwareDataset(
    data_dir="opt2vla_sim_dataset/",
    include_torque_history=True,
    torque_history_steps=10,
    torque_history_interval_ms=200,
)

model = GR00TN17.from_pretrained("nvidia/GR00T-N1.7-3B")

# Extend action head to predict 6-DOF force wrench
model.extend_action_space(force_dim=6)

trainer.fine_tune(model=model, dataset=dataset, epochs=50, lr=1e-4)

Inference Pipeline

def opt2vla_inference(rgb_obs, language_cmd, joint_state, torque_history):
    """
    Args:
        rgb_obs: (H, W, 3) egocentric RGB
        language_cmd: str, e.g. "Wipe the table firmly"
        joint_state: (N,) joint positions/velocities + base pose/vel
        torque_history: (10, N) torques at 200ms intervals
    Returns:
        joint_actions: (N,) torque commands for this timestep
    """
    # Level 1: VLA predicts motion goals + force reference
    with torch.no_grad():
        waypoints, force_ref = vla_model(
            rgb_obs, language_cmd, joint_state, torque_history
        )

    # Level 2: FCT controller executes with force reference
    # Note: no torque supervision signal needed at inference
    joint_actions = fct_controller(
        current_state=joint_state,
        target_waypoints=waypoints,
        target_force=force_ref,
    )
    return joint_actions

Comparison with Related Work

Opt2VLA's unique angle: combining physics (trajectory optimization) with language — the user simply says "gentle" or "strong," no need to specify a number. This is the most natural user interface among all the approaches above, and the only one demonstrated on a full humanoid platform.

Limitations and Future Directions

Current limitations:

1. Single robot tested (Digit) — performance on Unitree G1, Boston Dynamics Atlas, or standard robot arms is unknown
2. Dataset not yet released — full reproduction is not yet possible
3. Three relatively simple tasks — complex dexterous manipulation (screwing, pouring, cutting) not yet evaluated
4. Single contact point — force reference is for one primary contact; multi-contact scenarios are not addressed

Future directions:

• Extension to standard robot arms (UR5, Franka)
• Multi-contact force prediction
• Online adaptation when object properties differ from expectations
• Integration with tactile sensing for true closed-loop force feedback

Why This Is a Meaningful Step Forward

VLA for humanoids is at a stage where motion control is increasingly capable — but force control remains a critical gap. Opt2VLA is the first work to:

1. Include force reference in the VLA action space for humanoid whole-body manipulation
2. Use trajectory optimization to produce ground-truth force labels — no expensive F/T sensors needed in data collection
3. Condition force on language — users adjust force through natural language, not numeric specifications

With a dataset release coming and GR00T-N1.7 as the backbone, Opt2VLA has the potential to become a standard baseline for force-aware VLA research.

Related Posts

• FM-VLA: Force Memory for Contact-Rich VLA Manipulation
• CARE: Helping VLA Self-Correct with Failure Recovery on RoboTwin 2.0
• GR00T-N1.7 EgoScale: Fine-tuning Humanoid VLA from Egocentric Data