Do causal video diffusers really need dense causal attention at every layer, every denoising step?

We looked inside and found: no. Causality is separable from denoising.

Here are two surprising observations that hold across architectures, training objectives, and scales.