Improving Generative Adversarial Networks with Self-Distillation

Antoni Nowinowski

Poznan University of Technology

Krzysztof Krawiec

Poznan University of Technology

NeurIPS 2026

Abstract

In modern GANs, maintaining an Exponential Moving Average (EMA) of the generator’s weights is a standard practice, as such an averaged model consistently outperforms the actively trained generator. However, the EMA generator is used for final deployment only and does not influence the training process. To address this missed opportunity, we introduce Self-Distilled GAN (SD-GAN) that employs the EMA generator as a teacher to guide the active generator (student) via perceptual loss. We prove the local asymptotic stability of SD-GAN in the simplified continuous-time Dirac-GAN setting and show that it dampens the parasitic cycling behavior that plagues conventional GANs. Empirical evaluations across established architectures and datasets demonstrate that SD-GAN improves the final image quality on several metrics (FID and random-FID in particular), stabilizes the optimization trajectory and provides additional learning guidance that is not trivially correlated with the conventional adversarial loss. It also proves effective for fine-tuning pretrained GAN models.

Self-Distilled GAN

Overview of the SD-GAN framework. The training is guided by both the conventional GAN loss and the self-distillation loss. Dotted lines indicate gradient flow; gradients from the perceptual loss do not propagate through

G_{\!\textit{EMA}}

We implement this idea by extending the traditional GAN adversarial loss function with a self-distillation loss applied to the outputs of the active generator model $G$ and the averaged model $G_{\!\textit{EMA}}$ . Specifically, $G$ and $G_{\!\textit{EMA}}$ are queried on the same random latent vector $z$ , and the images they produce are compared using an SD loss component $\mathcal{L}_{\textit{SD}}$ .

The objective function for the generator is a weighted sum of the standard adversarial loss $\mathcal{L}_{\textit{adv}}$ and $\mathcal{L}_{\textit{SD}}$ :

\mathcal{L}_{G} = \mathcal{L}_{\textit{adv}} + \alpha \cdot \mathcal{L}_{\textit{SD}}(G(z), G_{\!\textit{EMA}}(z))

where $\alpha$ controls the weight of the guidance provided by the EMA teacher.

BibTeX

If you use SD-GAN in your research, please use the following BibTeX entry.

@misc{nowinowski_improving_2026,
    title = {Improving Generative Adversarial Networks with Self-Distillation},
    author = {Nowinowski, Antoni and Krawiec, Krzysztof},
    url = {http://arxiv.org/abs/2605.08577},
    doi = {10.48550/arXiv.2605.08577},
    year = {2026}
}