Sequence to sequence restoration

HYPI2VR Restoration Pipeline

Carrying an image diffusion prior into video: how the injected noise is rebalanced in frequency, warped along motion, and tied across frames.

Deg.RemovalEncoder Temporally-AwareRestoration Network VAEDecoder Input Frame 1 Restored Frame 1 Noise Injection of Frame 1 Noise EnergyRebalancing Sampled Noise Deg.RemovalEncoder Temporally-AwareRestoration Network VAEDecoder Input Frame 2 Restored Frame 2 Noise Injection of Frame 2 Noise EnergyRebalancing Flow Noise TemporalAlignment Deg.RemovalEncoder Temporally-AwareRestoration Network VAEDecoder Input Frame t Restored Frame t Noise Injection of Frame t Noise EnergyRebalancing Flow Noise TemporalAlignment Modified from Image Restoration Network Denoising Encoder Objective ℒenc = ℒmse + 0.05 ℒflow Restoration Objective minθ maxϕ ℒadv + λmse ‖Uθ(y) − x‖2 + λlpips LPIPS(VD(Uθ(y)), I) Noise-to-Noise Denoising Temporal Attention
01 / 10  ·  Image restoration backbone