When this article was first published, generative adversarial networks (GANs) had taken the world of deep learning and computer vision by storm. Introduced by Goodfellow and collaborators in 2014, the basic idea is to train two models at the same time: a generator, G, that produces samples, and a discriminator, D, that determines whether a sample came from the real dataset or from G.
The GAN framework resembles a two-player minimax game. The generator continually improves at producing realistic images, while the discriminator improves at identifying generated ones. Training alternates between the two models so that one does not get too far ahead of the other. The original GAN paper provides the formal treatment.
Image restoration
To make the idea concrete, this experiment follows Lesson 7 of fast.ai course-v3. The goal is to train a model that restores deliberately degraded images and removes simple synthetic text overlays. The experiment uses the Oxford-IIIT Pet Dataset.
- Choose a dataset of clean images.
- “Crappify” each image with synthetic text and reduced resolution.
- Pre-train a generator with a U-Net architecture to map degraded images back to their originals.
- Use the generator to produce an initial set of restored images.
- Pre-train a critic to distinguish generated images from originals.
- Train the complete GAN, alternating between generator and critic.
- Use the resulting generator to restore other degraded images.
Generating degraded training images
The degradation function adds random text or numbers, resizes an image to a lower resolution, and then scales it back up. This produces paired examples: a degraded input and its clean target.
Pre-training the generator and critic
The first step trains the U-Net generator to reproduce the clean image from its degraded counterpart using mean squared error. The model uses a ResNet-34 backbone pre-trained on ImageNet. After five epochs—about ten minutes on the Google Colab GPU available at the time—the model was already able to recover some image quality and cover most of the synthetic numbers.
The result was promising for such a short training run, but a visible quality gap remained. Mean squared error alone encouraged broadly correct reconstructions without recovering all the fine detail.
To pre-train the critic, I placed the generator outputs in one directory and the original images in another. The directory names supplied the “generated” and “real” labels used during classification training.
Training the GAN
GAN training alternates between updating the generator and the discriminator or critic. The generator receives an adversarial loss based on how successfully its output fools the critic. Mean squared error is retained to keep the generated image structurally close to its target. The critic pushes the adversarial objective in the opposite direction by learning to classify real and generated samples correctly.
Training GANs can be unstable and computationally expensive. Pre-training gave both networks a more useful starting point: the generator already produced recognisable restorations, while the critic had learned an initial distinction between originals and generated images.
Results
After roughly 80 epochs—nearly three hours on Google Colab at the time—the generator recovered much of the visible image quality and removed the simple synthetic overlays. Fine textures, particularly fur around the cats’ heads and faces, remained noticeably blurred.
Closing thoughts
This small experiment showed that GANs could be useful beyond generating images from random noise. The same broad approach could be adapted to other image-to-image tasks, including colourisation, super-resolution, and inpainting: construct an appropriate degradation process, train a generator to reverse it, and use an adversarial objective to encourage realistic output.