SituHOI

SituHOI: Scalable Capture of Scene-Situated Full-Body Human-Object Interaction

Scalable, scene-aligned full-body HOI data from one moving RGB-D camera, with FlexHOI-Bench for flexible-time joint prediction.

Qifan Wang* Qihang Fang* Boao Shi Yushi Du Pei Zhou Siyan Dong Yanchao Yang†

The University of Hong Kong

* Equal contribution  ·  † Corresponding author

wqf040701@connect.hku.hk u3010402@connect.hku.hk yanchaoy@hku.hk

Demo Paper CodeSoon DatasetSoon Citation

Demo

Lift a box
Lift a box
Wash at the sink
Wash at the sink
Carry a basket
Carry a basket
Sit on the sofa
Sit on the sofa
Place a straw hat
Place a straw hat
Sit at a desk
Sit at a desk
Walk to the television
Walk to the television
Walk beside the counter
Walk beside the counter

SituHOI Dataset

Full-body human motion, dynamic objects, and room-scale geometry in a shared scene coordinate frame, paired with semantic annotations.

Quality-screened activity
~86hours
Annotated clips
18,429
Indoor rooms
16
Dynamic objects
13

Scene × human × object

3D room with human-object interactions.

Human motionObject poses

Lived-in spaces

Reconstructed indoor space
Reconstructed indoor space
Reconstructed indoor space
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions
Reconstructed indoor space
Reconstructed indoor space with human-object interactions
Reconstructed indoor space with human-object interactions

Everyday objects

Chair mesh
Box mesh
Bin mesh
Basin mesh
Large basket mesh
Stool mesh
Cooking pot mesh
Small basket mesh
Bottle mesh
Cup illustration
Hat mesh
Bag mesh
Mat mesh

Reconstruction Pipeline

From a one-time room scan and single-camera RGB-D recordings to camera poses, full-body motion, and object trajectories in a shared metric scene frame.

SituHOI construction pipeline: room scans and RGB-D interactions feed scene reconstruction, camera localization, human fitting, and object tracking, producing scene-aligned motion with language annotations.

FlexHOI-Bench

A unified benchmark on SituHOI for goal-conditioned human-object motion prediction, with flexible observation and forecast durations.

Given
  • Human-object history
  • Scene & object geometry
  • Target object pose
Predict
  • Full-body human motion
  • Intermediate object trajectory

Explore the time split

H + F ≤ 4 s

Drag the split and goal handles, use arrow keys, or choose a reported setting. The ruler measures elapsed time from the window start; the split is the paper’s t = 0. Dots mark reported splits at the current span.

Active span4.00 / 4 sFull time budget

SharedHOI results

Reported setting
Human-root ADE ↓
355.04mm
Object ADE ↓
132.92mm

± 0.66 mm across seeds

Proximity F1 ↑
58.08%

H = 0.25 s · F = 3.75 s — reported in the paper.

Paper settings H / F in seconds

† Held-out timing

Flexible-time evaluation

Evaluate the same model across varying observation and forecast durations, including unseen duration combinations.

Recording-level splits

Keep each recording within a single split, so overlapping motion windows never cross training, validation, and test sets.

Motion & coordination

Assess both trajectory accuracy and human-object coordination through position errors and wrist-object proximity.

SharedHOI

Reference method

A shared temporal predictor couples human and object motion, followed by object-translation refinement that preserves the supplied goal.

SharedHOI for FlexHOI-Bench: motion history, scene and object geometry, an object goal, and query durations are encoded by a shared transformer; human prediction and object refinement produce future motion in scene coordinates.

Citation

arXiv:2610.23980

@misc{wang2026situhoi,
  title  = {SituHOI: Scalable Capture of Scene-Situated
            Full-Body Human-Object Interaction},
  author = {Qifan Wang and Qihang Fang and Boao Shi and
            Yushi Du and Pei Zhou and Siyan Dong and Yanchao Yang},
  year   = {2026},
  eprint = {2610.23980},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url    = {https://arxiv.org/abs/2610.23980}
}