TouchScale

500 Hours of Human Vision and Touch for Visual–Tactile Learning

Dayou Li^1,*
Hao Wang^2,*
Qianqian Yang^3,*
Zihao Zhu^1,*
Haoquan Fang⁴
Ziyao Zeng⁵
Yan Han⁶
Zihan Wang⁷
Yan Wang⁸
Baoruo Huang⁹
Dilin Wang¹⁰
Kenji Shimada³
Yiyue Luo¹¹
Manling Li¹²
Teresa Lv¹³
Mustafa Mukadam¹¹
Rakesh Ranjan¹⁰
Ruohan Zhang⁴
Qi He⁶
Changliu Liu³
Xu Chen¹¹
Marco Pavone^4,8
Bangya Liu⁷
Jiachen Li¹⁴
Masayoshi Tomizuka¹⁵
Zhiwen Fan^1,†

¹ Texas A&M University
² Google DeepMind
³ CMU
⁴ Stanford University
⁵ Yale University
⁶ Microsoft
⁷ Overfit Lab
⁸ NVIDIA
⁹ University of Liverpool
¹⁰ Meta
¹¹ University of Washington
¹² Northwestern University
¹³ Sony
¹⁴ Georgia Tech
¹⁵ UC Berkeley

^* Equal contribution^† Corresponding author

Human vision and touch

Policy rollout

TouchScale bridges human visual–tactile data and robot skill learning.

What Does Scaling Touch Unlock?

22.5%57.5%

Robot task success

+35 percentage points across four tasks

+35.1%

Contact IoU improvement

Zero-shot transfer on EgoTactile

+43.7%

MECCANO action recognition

vs. EgoTouch · 20.16% → 28.97%

Figure 5 left: robot success rises from 22.5 to 57.5 percent as TouchScale mid-training data increases — More human visual–tactile data improves robot task success and zero-shot touch prediction. The curves show performance as the fraction of TouchScale training data increases.

Figure 6 contact IoU: zero-shot contact prediction on EgoTactile improves consistently as TouchScale training data increases from 10 to 100 percent — More human visual–tactile data improves robot task success and zero-shot touch prediction. The curves show performance as the fraction of TouchScale training data increases.

Resources

CodeGitHub DatasetHugging Face BlogOverfit Lab

Data Collection Setup

A unified wearable setup captures synchronized vision and touch.
A head-mounted RGB-D camera ① records the overall interaction, while two wrist-mounted RGB cameras ② provide detailed views of hand–object contact. Both record at 30 Hz.
Bimanual tactile gloves ③ each contain 880 sensing taxels across the five fingers and palm, with spatial resolution below 2 mm. The sequence below shows time-aligned observations before contact, during wringing, and after release.

Complete data collection figure: wearable RGB-D camera, wrist cameras, and tactile gloves, alongside synchronized RGB, depth, wrist views, and bimanual touch at three stages of cloth wringing

Scale and Task Diversity

500 hours across laboratories, kitchens, workbenches, and everyday environments.

Data Samples

Explore contact-rich interactions across everyday settings. Each video pairs an egocentric view with synchronized wrist cameras and bimanual tactile measurements, available under “All sensors.”

Real-World Robot Experiments

From human visual–tactile recordings to contact-rich robot manipulation.
We use an xArm6 arm with a BrainCo Revo 2 robotic hand, tactile sensing, and ego-view and wrist cameras. A VR headset and motion-capture gloves support demonstration collection.
TouchScale mid-training teaches the policy to predict future tactile changes before robot post-training. In the paper’s evaluation, both variants use 50 demonstrations per task and 20 evaluation trials per task.