In this video, you'll learn about GAN improvements. You'll see how GANs have advanced over time and what methods have contributed to improving them. You might remember seeing this at the very beginning of the course. It's tweet from Ian Goodfellow, who is regarded as the creator of GANs. It's a visualization of the rate at which GANs have improved over the years. They've basically progressed from black and white images that are barely faces to much higher quality ones that you could believe are real people by 2018 and they've gotten better since. The first major improvement was to GAN training stability. You might recall from the video mode collapse. An Unstable GAN typically means that training at longer won't help it. In this case, the generator here is stuck in some local minimum, and it's only generating one thing and here it's only generating 4s, when it really should be generating a diversity of things. You've seen many ways to prevent mode collapse. But one interesting thing is to take a look at these two samples where you want this thing on the left with a lot of variation and there's high standard deviation across samples. But, and again, that has mode collapse when you look at a mini-batch of, say, three samples compared to the one that you want here on the left, you see that there is very low standard deviation, meaning that there is very little variation across these three samples. Each of these three samples from your mini-batch can be pretty indicative of how much variation there is inherent to your GAN. As a result, the standard deviation of these images can show when your generator is approaching mode collapse because its standard deviation will be low. To mitigate this, perhaps you can pass this information to the discriminator and it can then punish the generator or rather give feedback back to the generator to try and keep it from collapsing. Because now if this is given to the discriminator, instead of the discriminator just seeing one sample at a time, it will now see all of these and realize there might be a little bit of mode collapse here going on. Of course it doesn't see all those samples, it sees them quote unquote through perceiving the standard deviation, the statistic on the mini-batch. Another way researchers have improved stability in GANs is by enforcing one Lipschitz continuity when using Wasserstein loss or W loss. This prevents a model from both burning too fast and also keeping in valid W distance, which you've seen in Week 3 of this specialization, where you want to make sure the gradient of your function remains in these green triangles at every single point of this function. This is observing this point and observing a point out here would yield a different set of triangles, and you want to make sure you're function stays within those. That means it's not growing more than linearly. You've seen this with WGAN-GP, which is using Wasserstein loss, which can be abbreviated by this W and maintains one Lipschitz continuity by using the Gradient Penalty or GP here. Another method to enforce this is also to use Spectral Normalization on the weights. This is similar to a concept you might be more familiar with, known as Batch Norm or Batch Normalization. Another method is called Spectral Normalization, which in some senses they are similar to Batch Normalization, in the sense that it normalizes the weights and it's often instantiated as some layer, so Spectral Norm layer. This is another type of weight normalization like Batch Norm and it stabilizes training by keeping values within a certain range. I won't go into the details because it gets pretty mathematical on what a Spectral Norm is, but it doesn't maintain this one Lipschitz continuity. Don't worry too much about it. There are open-source implementations of this method. A final way researchers have tried to stabilize GANs is by using two techniques that you'll learn more about in the following videos in StyleGAN. The first is using a Moving Average that takes the average weights over several checkpoints. Instead of describing the weights of one generator, which would be typically Theta_i, where i equals, let's say Step 200, wherever you find the best generator checkpoint. Instead you want the average Theta, which I'll put a bar here, which will equal the sum of j to n Theta_j. Instead you want the average of all of these parameters. This gives you much smoother results and is done during training, but also can be applied when saving a model and then sampling from it. Here you can see the differences between averaging and not averaging. Without averaging, you can see that maybe that generator checkpoint was in some local minimum or it just wasn't very smooth versus using some averaging. Here, the exponential average can yield much smoother result and the differences are quite apparent here. The second technique is called Progressive Growing. It gradually trains your generator on increasing image resolutions. It can generate high-quality images. You'll learn more about this in the coming videos on StyleGAN. Now that you've learned some ways, researchers have increased GAN training stability so they can train for longer and improve. You'll see some other areas where GANs have advanced as well. Another major improvement to GANs was their capacity. Generators and discriminators have gone a lot wider and deeper, such as deep convolutional GANs, DC-GANs that you learned about earlier in Week 2. These models can be larger because of improved hardware like better GPUs in higher resolution data sets, like the FFHQ data set, which is a set of face images from Flickr that are really high resolution to a trained StyleGAN, the state of the StyleGAN. Some architectures had begun catering to these higher resolution data sets as well. The final major improvement to GANs has also been their diversity and their output. They have had increasing variation in their generated images and that's fantastic because we want to model all the possible human faces here, not just generate one face. Remember that this is also a key criterion that you now evaluate your GANs on. One of the ways diversity has been improved is by having larger data sets with more coverage because having more variety in the images being trained on, lead to the same in the output as well. Faces you see here are actually generated by StyleGAN, and StyleGAN has a few architectural changes to increase diversity that you'll see in the next couple of videos as well. I do want to point out here that training stability, and I do want to point out here that the mini-batch statistics that you saw previously using the mini-batch standard deviation can certainly help too. I do want to know that the method you saw before using mini-batch standard deviation to perhaps help with the stability and training can also help with increasing and improving diversity by avoiding more collapse and turning towards how much variation real images have. In summary, GANs have advanced largely from improved training, stability, capacity, and diversity. A stable GAN with less chance of mode class lets you train your model for longer and larger models and higher resolution images will improve the quality of your outputs. A few changes will increase the diversity in your generated images, and you'll see some of these in the following videos on StyleGAN. One of the best GANs today.
