SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models

In this paper, we incorporate this spectral evolution, or equivalently the evolution of the signal-to-noise ratio, into cache scheduling. SeaCache applies a Spectral-Evolution-Aware (SEA) Filter to raw diffusion features so that the distance measure better captures timestep-aware spectral residuals between timesteps.

Rather than treating all spectral components of diffusion feature equally, we design a cache metric that focuses on the signal component while downweighting the noise component. By grounding reuse decisions on discrepancies in the synthesized content, the resulting metric becomes less sensitive to high-frequency noise and encourages cache gating to respond to meaningful signal alignment rather than stochastic variation.

To validate this idea, we conduct an oracle experiment that compares cache schedules derived from raw feature distances with those derived from distances in a signal-emphasized space. In standard caching schemes, the decision to skip or compute is based on the distance between input features at consecutive timesteps. In our oracle analysis, we instead compare consecutive output features, thereby removing input-to-output approximation error and isolating the effect of spectral filtering. Specifically, we compare two criteria: one that measures distances after applying the SEA (Spectral-Evolution-Aware) filter, which downweights the noise component, and another that uses unfiltered raw outputs, as shown in below figure. The filtered criterion yields cache decisions that more closely track the full-compute trajectory, as evidenced by consistently higher PSNR. This suggests that spectrum-aware scheduling better preserves the behavior of the original model.

To design a filter that reflects spectral evolution, we formalize the change of the effective frequency band across timesteps. By deriving an optimal linear denoiser, we propose a Spectral-Evolution-Aware (SEA) filter that effectively preserves content-relevant components while suppressing noise. The visualization of the SEA filter is presented below.

Optimal linear denoising filter for different \(t\)

Noramlized SEA filter for different \(t\)

Spectrum-Aware Dynamic Caching

Directly using SEA-filtered outputs in the cache metric is not practical, since the output is only available after a full denoiser run and thus offers no speedup. We therefore seek an input-side proxy that matches the SEA-filtered output distance as closely as possible. Building on the input features, introduced in SeaCache, we compare several candidates: raw input, raw output, the polynomial fitted input used in TeaCache, and their SEA-filtered counterparts obtained by applying from the proposed method.

Below figure reports the relative \(\ell_1\) distance between consecutive timesteps for these feature choices, averaged over ten samples on FLUX and Wan2.1 1.3B. The SEA-filtered input distances closely follow the SEA-filtered output distances along the entire trajectory, while raw input and polynomial fitted input show weaker alignment, especially at early timesteps.

Rel \(\ell_1\) across the gen process on FLUX

Rel \(\ell_1\) across the gen process on Wan2.1 1.3B

Given input features \(I_t\) and \(I_{t+1}\), SeaCache first applies FFT, multiplies by the timestep-dependent SEA filters \(G_t^{\mathrm{norm}}\) and \(G_{t+1}^{\mathrm{norm}}\), and then applies iFFT to obtain spectral-evolution-aware features \(\mathcal{P}(G_t^{\mathrm{norm}}, I_t)\) and \(\mathcal{P}(G_{t+1}^{\mathrm{norm}}, I_{t+1})\). A spectrum-aware dynamic caching module measures the relative distance \(\widetilde{\Delta}_t\) between consecutive filtered features, accumulates it over timesteps, and either reuses the cached output or refreshes the denoiser when the threshold \(\delta\) is exceeded. The underlying diffusion model remains unchanged, so SeaCache acts as a plug-and-play cache policy that replaces only the distance metric.

For all experiments, generated images and videos are saved as PNG and MP4 files, respectively. For text-to-image (T2I) generation, we evaluate 200 prompts from DrawBench and generate images at 1024\(\times\)1024 resolution in FLUX. For text-to-video (T2V) generation, we use 944 prompts from VBench and generate 480p videos with 65 frames per prompt in Wan2.1.

For each configuration, we use the full-timestep output of the original model as the reference. We compute PSNR, LPIPS, and SSIM between each cached sample and its reference, and then report the average over all samples. The initial random seed is shared across our method and all baselines. We consider two cache budgets, approximately 50% and 30%.