Welcome to our notebook here on Convolutional Neural Nets. Here we're going to be using python to build out our convolutional neural nets in order to classify images using this famous CFR-10 dataset, and the CFR-10 dataset is going to be 60,000 different images each one 32 by 32 pixels and they're color images, so they will also have a certain amount of depth if you recall from lecture and each one of these different images will be one of 10 classes either airplane, automobile, bird, and so on. Now, in order to build out our convolutional nets, we're going to have to introduce new parts of carers, new functions, and new layers that we hadn't used in our prior notebook. We introduced the sequential model, we can still use that. At some point, we will have to use that dense connected layer similar to what we did with that fully connected layer. We can also use dropout in order to regularize and ensure that it doesn't overfit. We'll have our different activation layers that we can use as well, whether it's ReLU, or sigmoid, or hyperbolic, whatever it is and then we also have this flattened layer. This flattened layer will be important as we move from our convolutional layers to our dense layers, and eventually in order to make some type of prediction, we're going to need to flatten it out and then have that dense layer connected to that final prediction. Then we're also going to import this Conv2D and this MaxPooling2D which will allow us to build out our convolutional layers as well as our pooling layers that we introduced in lecture. Now to get our data, we imported earlier this cifar10 from the keras datasets, so this is actually within the keras library. We have this dataset available. We call load_data and when we load data, that'll give us two tuples, our training set, as well as our test set, and x and y values for each. Now we're going to print out the shape of our training set as well as the number of samples of both our train and test set separately. Now, there's going to be 50,000 training samples and 10,000 test samples, so we'll be training on those 50,000 training samples and then we can ultimately test on that holdout set on that 10,000. But what I want you to look to is the shape of x_train, and if you think back to what we'd been working with so far, and those were not imageData that we were working with. Here you see that we have four dimensions. The first dimension is going to be the number of rows or the numbers of samples that we're working with, and that's 50,000, and then the next one is going to be the height and width in terms of the number of pixels of our images as well as our depth. That's where we have the 32 by 32, then by three for the red, green, and blue different layers. We can look for each individual image that we have this 32 by 32 by three shape. That's just going to be a bunch of numbers from 0-255 for each one of these different colors: red, green, and blue. We can see that these actually represent actual images, so we see here that this is of class 9 if we call y_train and just 444. Then we can look at the x_train, which is going to be the actual image itself without the label and we see that it's actually going to be a real image. Now it's not a high-definition image if we think 32 by 32 pixels will not be a high-definition image, but we can tell you that we have a truck and hopefully our convolutional neural net will be able to pick up the fact that it's going to be tires and the large back and whatever other features there are that build out that truck. Now if we look at our y_train originally, we see that it's just a bunch of numbers each one representing a different category. As we discussed in lecture, we often need to take something that's categorical and maybe many different categories and turn that into a categorical variable doing one-hot encoding. Kares has functionality built in to change this output into that one-hot encoded version of the output. In order to do so, we just call keras.utils.to_categorical and we say what the set is that we want to change categorical and the number of classes in that set, which is going to be equal to 10. We run that and now if we look at that y_train that we had above which we see was nine, the new value is going to be that one-hot encoded version with zeros everywhere except for in the nine spot. Then we're going to want to make sure all of our values are flow and scale down to between zero and one, so recall that all of our different pixels will be values between zero and 255, so if we divide by 255, we ensure that all of our values are going to be between zero and one. Now, when we use this convolutional neural nets, when we create these layers, these convolutional layers we call just same as we did with dense we call conv2D. We want to ensure that we understand the different parameters that we can pass through so that we can specify exactly what convolutional layer we want to use. Thinking back the lecture, some of the important parameters that you should know, it's going to be the filters, and that's going to be the number of filters use per location, so in other words, that's going to be the depth of your output. Or the number of kernels use if you think about, again, that depth, we call an lecture we had at one point a depth of 10, and that was because we had 10 different kernels that will output a depth of, if we said 10 of10, if we set filter is equal to 10. Then we have our kernel size, or she'll be a tuple giving the height and width of the kernel used, and you can specify that height width to be different if you just pass through one number, it will assume a square, and I would say stick with squares to start, you can try playing around with other values, but those are going to be best practice for the majority of your starter material. Then we have the strides, so that's going to be how you move along those kernels, along your image, and whether you want to move it one at a time, going from left to right or two at a time going left to right, as well as up and down. The first value is going to be the stride going left to right, and then the next one's going to be up and down. Then you're going to want your input shape, which will recall we pass through and we had our dense neural network, and that was just one value here if you just recall what we pulled out in terms of the shape of a single image that's going to actually be three-dimensions. We want to ensure that it fits within that first layer that we specify to ensure that that's correct. One more thing that I want to point out that's not here is the padding. We set padding equal to valid that means that we're not having any padding, and it'll stop as soon as the right, if we're moving from left to right as soon as the rightmost part of our kernel hits the edge as we move along those strides. If we imagine that we have a six-by-six image in our kernel is five-by-five, then it will only move to the right once and then stop. Then if we set padding equal to, and we'll see this later on saying, then that will pad on some extra zeros generally speaking, to make sure it'll be just one set of zeros around, but maybe it's even it out there might be two on one side and there'll be padding with zeros. Then again, we have this flattened layer and that turns are whatever input it has into a one-dimensional vector. That will allow us, once we do that, to dry and just transition between the convolutional layers and those fully connected layers. We have here the initialization of our model using the sequential function, so we're going to be using that sequential API again in order to build out our model. We then add on our first Conv2D layer, and this is an ordering that we saw above, and we can see if we just call it shift tab it's got a couple more times, we can see that we're setting the number of filters, so the depths also going to be equal to 32. Our kernel size is going to be five-by-five, we're not going to use the default for strides, we're going to set that to two-by-two. We're also going to add on padding, so there's actually going to be padding on this layer rather than the default of leaving it as is. Then we specify the input shape, and if you recall, the shape of our x train is going to be the number of different samples we have, then the actual shape, and if we say one through, then we're just specifying the shape of a single object. We added on that convolutional layer, we then have our activation which is going to be relieved to ensure that we have that nonlinearity, we then add on another convolutional layer again, setting the number of filters equal to 32. We are going five-by-five in regards to our kernel, and recall that if we do five-by-five, then we're moving along our image, and that will continuously reduce the size of each one of the layers as we move across, especially since our strides are two-by-two. We then add on another activation layer, we can then do our max pooling, which will just take the max of a certain grid and resetting that poolside equal to two-by-two. We're going to reduce very quickly the size of our layer. We're then also going to introduce some drop-out to add a bit of regularization. We then flatten that out so that now we're working with just a one-dimensional object rather than that three-dimensional object that we were working with in the earlier layer. We can then do add-on a dense layer so that we have one fully-connected layer, again call ReLU. We have a nonlinearity. Again, call dropout for some extra regularization. Then we're going to add on that final dense layer so that our output is equal to the number of classes we have. Because ultimately, if you think about our neural network, it needs to specify, it needs to predict one of these 10 classes. Then we set our activation equal the softmax as we do when we're trying to predict amongst multiple categories. We can call the model one.summary, and we can see here the number of parameters at each step and we can see that our output shape is going to reduce each one of these steps or reduced at first to 16 to16 by 32. We kept the depth at 32 at both steps. Then we have 6 by 6 by 32. We called max pooling and that reduced it to 3 by 3 by 32. Then we had our dense layer. And you see there's a ton of parameters there and recall that these dense layers are going to have a lot more parameters and be a lot more variance than we would have with our convolutional layers. We have one more dense layer and then our final activation. We can then specify our batch size as well as the optimizers we're going to use so we're specifying we want the RMS prop optimizer with this learning rates and we can specify the decay if you recall from RMS prop. Then with that, we compile using categorical cross entropy rather than our binary cross entropy. We can specify the optimizer, metrics we want to track, and then actually fit our model with the bite-sized specifying the number of epochs. We have our validation data to see on the holdout set how it does and we can shuffle equals true and that'll just be in regards to, as we optimize, we want to shuffle our data throughout. I'm going to let this run and we'll come back once it's done running. I want to warn you this may take some time. That should have taken maybe five minutes, maybe a bit longer in order to run and we can actually see the timing for each epoch as we ran through. We have here the 15 different epochs. We can see that the loss on the training set continuously went down for each 1-15 epochs. We didn't save it as we did before so that we can access that history dictionary. But we can see what happened at each step and we can see that we're tracking that validation loss and that goes down for the first number of epochs. Then round here, we see that it starts to fluctuate or it goes down, back up, down again on that validation set looking from 1.049-1.08. Then we can see that the accuracy rather than continue going up, starts to fluctuate as well on that validation set, but it continues to increase for that training set. Now, if we wanted to do any type of prediction, we can do the same thing that we did in our last notebook by taking that model and calling. If we want the probabilities, we can just do.predict and we can call that on our x tests and rerun that and that'll give the probabilities for each one of the different classes. If we want to predict the specific class, we can just call.predict_classes as we did before and we see here that as a prediction for each class. Now if we recall if we wanted to test our accuracy, let's say, or any other metric, I would say if we want to look at the accuracy or something else that requires that actual prediction, if you recall, our y tests had been converted to this one-hot encoded version of y test so we'll have to take it back to what it was originally. In order to do that, we can just pull in NumPy and call np.argmax and that'll just say where's the maximum arguments? You have to specify a cross Axis 1. When we call that, we can get each one of the actual values. We can see from what we have here, probably predicted correctly for 388 and 517. The accuracy score should be what we have here but we can actually test this. We can import from sklearn.metrics our accuracy score from sklearn.metrics accuracy score. Then we can take that accuracy score of what we have here. These are the actual values and then our prediction that we have here. We have that same value 0.6176, since that was used as a validation set. That closes out that first exercise and in the next exercise, we will work through building out a different convolutional neural net and see if we can make any improvements on our current model. I'll see you there.
