VTWM ablation — short 16-frame rollouts (auto-uploaded)

Diffusion-forcing world model rollouts. Each sample is ~2.7 s (16 raw frames @ 6 fps). Side-by-side videos show ground truth on the left, model prediction on the right. First half of each clip = history (GT), second half = autoregressively sampled future. Click a sample header to expand its videos (default-collapsed to keep the page light).

Sections

Setup comparison

Click a run name to jump to its rollout videos. ✓ = feature on, ✗ = off. Lower val_loss is better.

Rollout-latent metrics (10 val samples)

Latent-space MSE of predicted future vs ground truth. Contact/no-contact split uses tactile latent-to-reference energy (threshold 0.05).

Training loss curves & conclusions

📈 Live interactive curves: https://wandb.ai/yuxiangma/tactile-world-model-diffusion (wandb — always current; static PNG below is a snapshot)

# Training loss curves — conclusions

Regenerated from wandb (`scripts/plot_loss_curves.py`). Runs covered: **37**.

- **shift16 + delta-to-ref** are the two tactile fixes; without shift16 the tactile stream diverges.
- **cam-pose** curves look fine on val_loss but decoded LPIPS is far worse — see perceptual table.
- **stride2 (4x data)** is the biggest perceptual win; p01 trades a little val_loss for rollout consistency.
- **hybrid_flow** uses rectified flow — its val_loss is on a different scale, judge by decoded metrics.

Included runs: mm_v0, mot_vt_p01, mot_vta_p01, mv1_256_delta_shift, mv1_3v2t_delta_ref, mv1_p01, mv1_p3_delta_shift, mv1_v2_perframe, mv1_v2_stride2, mv1_v2_stride2_p01_adaptw, mv1_v2_stride2_p01_hybrid, mv1_v2_stride2_p01_hybrid_deep, mv1_v2_stride2_p01_hybrid_flow, mv1_v2_stride2_p01_hybrid_rope, mv1_v2_stride2_p01_plucker, mv1_v2_stride2_p01_tactile_only, mv1_v2_stride2_p01_v2t, mv1_v2_stride2_p01_vision_only, mv1_v2_stride2_p01rand, mv2_3v2t_campose_delta_ref, mv2_p4_delta_shift, p2_mv, p3_mv2t, p4_gate, p5_gate, vo_left, vo_middle, vo_right, vo_v0, vtwm_e2_delta_ref, vtwm_e3_delta_ref_null, vtwm_n1_shift16_delta_ref_null_prenoise, vtwm_n2_shift16_delta_ref_tactile_dropout, vtwm_p0_shift16_delta_episode_p01_ref, vtwm_s1_shift16_mmdit_mv, vtwm_s2_shift16_delta_ref, vtwm_s3_shift16_delta_ref_weight

_Not on wandb yet (curve omitted): mv1_v2_h4f12, mv1_v2_stride2_p01, p4_mv2t_extr, vtwm_e1_1v2t_mmdit_mv_

B. episode_p01 ref (rollout-consistent)

stride2 + p01

Stride2 with tactile_ref_source=episode_p01. Best perceptual quality overall (view LPIPS 0.075) despite higher latent val_loss than window. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

p01 + random no-contact ref

p01 recipe with a randomly sampled no-contact reference frame per window instead of the fixed episode_p01 frame. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

C. Architecture — MoT / hybrid

MoT V+T

Codex-designed MoT variant — per-modality QKV+MLP+adaLN with shared spatial attention pool. Functionally ≡ mmdit_mv. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

MoT V+T+Action

MoT-VT + 4 zero-init action-expert tokens joining the spatial pool. The gate never opened without a supervision signal — action tokens contributed nothing. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + p01 + hybrid + RoPE

Dual→single hybrid fusion (HunyuanVideo/FLUX style) with RoPE positional encoding. Warm-started from mv1_v2_stride2_p01. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + p01 + hybrid (dual6+single7)

Dual→single hybrid fusion, 85M params (0.84x mmdit_mv). First 6 blocks per-modality, last 7 shared-QKV with per-modality adaLN. Warm-started from mv1_v2_stride2_p01. Ties the p01 baseline — architecture is not the bottleneck. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + p01 + hybrid (deeper, depth14)

Deeper hybrid: depth=14 (+2 single-stream blocks), ~100M params (matches mmdit_mv budget). dual_depth=6 preserves warm-start. Ties the p01 baseline at 0.0159. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

D. Single-modality controls

vision-only control

p01 recipe with the tactile stream removed — visual-only lower bound (val_loss 0.0101). Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

tactile-only control

p01 recipe with the visual stream removed — tactile-only lower bound (val_loss 0.0062; tactile latents are easier to predict than visual). Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

E. Cosmos tokenizer

Cosmos v3 + causal backbone

Cosmos CV4x8x8 latents (v3 delta-space per-channel whitening) with a CAUSAL diffusion backbone (action_mode=per_frame_causal) to match the causal tokenizer + AR rollout; anchor_loss_weight=1.0. val_loss is on the Cosmos/whitened scale — NOT comparable to the VideoVAE+ runs above; judge by decoded rollout quality only. Showing 8 of 16 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

Cosmos v3 + contact-aware tactile loss

v3 + additive x0-space contact-aware tactile aux losses (contact-patch reconstruction 0.25 + false-contact suppression 0.10, alpha_bar-weighted; docs/CONTACT_AWARE_TACTILE_LOSS.md). Thresholds calibrated for Cosmos delta units (0.35/0.076/0.11 = P95/P70/P80 pixel energy). Single-variable vs v3 (full_window backbone, anchor 0.5). Judge by tactile contact MSE + false-contact rate + view LPIPS, not val_loss. Showing 8 of 16 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)