Now let's go over tensors and variables in TensorFlow. It's time to see some code. How can we bring life to each dimension of a tensor that we learned about earlier? Recall that a tensor that N-dimensional array of data. When you create a tensor, you'll specify its shape. Occasionally, you'll not specify the shape completely. For example, the first element of the shape could be a variable, but that special case will be ignored for now. Understanding the shape of your data or oftentimes the shape that it should be is the first essential part of your machine learning flow. Here, for example, you're going to create a tf.constant(3). This is a zero rank tensor. It's just a number three, scalar. The shape, when you look at the tensor debug output, it's simply an open parenthesis, closed parenthesis, it's zero rank. To better understand why there isn't a number in this parenthesis, let's upgrade to the next level. If you passed in a bracketed lists like 3, 5, 7 to tf.constant instead, you would now be the proud owner of a one dimensional tensor, otherwise known as a vector. Now that you have that one dimensional tensor, let's think about that. They'll grow horizontally, like things on the x-axis, by three units. Nothing on the y-axis yet, so we're still in one-dimension. That's why the shape is (3), 1, 2, 3, comma, and nothing. All right, let's level up. Now, we have a matrix of numbers, or a 2D array. Take a look at the shape (2, 3). That means we have two rows and three columns of data. The first row being that original vector of 3, 5, 7, which also has three elements in length. That's where the three columns of data comes from. You can think of a matrix as essentially a stack of 1D tensors. The first tensor in the vector 3, 5, 7, the second 1D tensor that's being stacked is the vector 4, 6, 8. We've got height and we've got width. Let's get more complex. What does 3D look like? Well, it's a 2D tensor with another 2D tensor on top of it. Here, you can see that we're stacking the 3, 5, 7 matrix on the 1, 2, 3 matrix. We started with two two-by-three matrices, so our resulting shape of the 3D tensor is now (2, 2, 3). Of course, you could do the stacking and code itself instead of just counting parenthesis. Take the example here. Our x1 variable is a tf.constant constructed from a simple list [2, 3, 4]. That makes it a vector with a length of three. x2 is constructed by stacking x1 on top of x1. That makes it a two-by-three matrix. x3 is constructed by stacking four x2s on top of each other. Since each x2 is a two-by-three matrix, that makes x3 a 3D tensor with a shape of 4.2.3. x4 is constructed by stacking x3 on top of x3. That makes it two four-by-two-by-three tensors, or the final shape of 4D tensor. If you've worked with arrays of data before like NumPy, they're similar except for two points. tf.constant will produce tensors with constant values, whereas tf.variable produces tensors with variable values or ones that can be modified. Now, this will prove super useful later when we need to adjust those model weights during our training phase of our ML project. The weights can simply be a modifiable tensor array. Let's take a look at the syntax for each as you'll become a ninja with combining, slicing, and reshaping tensors as you see fit. Here's a constant tensor produced by tf.constant, of course. Remember that [3, 5, 7] in a 1D vector? It just stacked here to be that 2D matrix. Pop quiz, what's the shape of x? How many rows or stacks do you see, and then, how many columns do you see? If you said two-by-three or two rows and three columns, awesome. When you're coding it, you can also invoke tf.shape, which is quite handy in debugging. Much like you can stack tensors to get high level dimensions, you can also slice them down too. Let's look at the code for this, for y. It's slicing x. Is it slicing rows, columns, or both? The syntax is let y be the result of taking x, and take all rows. That's the column, and just the first column. Keep in mind that Python is zero index when it comes to arrays. What would the result be?Remember, we're going from 3D to 2D, so your answer should only be a single bracket list of numbers. If you said [5, 6], awesome. Again, take all rows, only the first index column. Don't worry, you'll get plenty of practice with this coming up in your lab. We've seen stacking and slicing. Let's talk about reshaping with tf.reshape. Let's use the same 2D tensor or matrix of values that is x. What's the shape again? Think rows and columns. If you said two-by-three, awesome. Now, what if I reshaped access three-by-two or three rows, two columns. What would happen? Well, essentially Python would read the input by row by row, and put numbers into the output tensor. It'll pick the first two values, put them in the first row, so you get three and five, and the next two values, seven and four, in its second row, and the last two values, six and eight, into the third row. Again, two columns, three rows. That's what reshaping does. Well, that's it for constants. Not too bad. Next up are variable tensors. The variable constructor requires an initial value for the variable, which can be a tensor of any shape and type. This initial value defines the type and the shape of the variable. After construction, the type and shape of the variable are fixed. The value can be changed using one of the assigned methods: assign, assign_add, or assign_sub. As we mentioned before, tf.variables are generally used for values that are modified during training such as, as you might guess, the model weights. Just like any tensor, variables created with.Variable can be used as inputs to your operations. Additionally, all the operators are overloaded for the tensor class are carried over to variables. TensorFlow has the ability to calculate the partial derivative of any function with respect to any variable. We know that during training, weights are updated by using the partial derivative of the loss with respect to each individual weight. To differentiate automatically, TensorFlow needs to remember what operations happened in what order during that forward pass. Then during the backward pass, TensorFlow traverses this list of operations in reverse order to compute those gradients. GradientTape is a context manager in which those partial differentiations are calculated. The functions have to be expressed within TensorFlow operations only, but since most basic operations like addition, multiplication, subtraction, are overloaded by TensorFlow Ops, this happens seamlessly. Let's say we want to compute a loss gradient. TensorFlow records all operations executed inside the context of tf.GradientTape onto a tape, then it uses that tape and the gradients associated with each recorded operation to compute the gradients of a recorded computation using that reverse mode differentiation like we mentioned. There are cases where you may want to control exactly how gradients are calculated rather than using the default. These cases can be when the default calculations are numerically unstable or you wish to cash an expensive computation from the forward pass, among other things. For such scenarios, you can use custom gradient functions to write a new operation or to modify the calculation of the differentiation.
