Hello and welcome back. Let us now describe the first image Inpainting model. That's going to base, be based on partial differential equations that we learned the last week. But again, this is going to be self contained, we're going to understand exactly what kind of partial differential equation are we going to use? When we talk about image in painting we talk about the following. We have an image. Here and we have a region. Where we want to change the content of the image. In this case it's the cable. Here. Someone's going to provide us this. Image and painting. Includes the topic where we basically have these two inputs into the system. We are not going to detect, this is a very subjective, very often a very subjective topic about what is that we want to change. Maybe somebody wants to change the shoes. In this case, we want to change this. Now, how we detect these. We talk a lot about segmentation. So we can segment out objects. And those are the regions of missing information. We are going to call them origins to be inpainted. With these two images, we want to make, basically, in this case, the cable disappear. We want to inpaint in. These regions. And the first topic we're going to discuss is how do we inpaint with rich, with information from the surrounding regions. We're not going to take information from very far away. We are only going to use information from the surrounding regions. So we get two images. The original image and an image which is a binary mask of the, representing the regions that we want to in paint. When I personally started working on Image Inpainting with my group, we visited a professional restaurenteur, a professional curator at the Minneapolis Institute of arts. And we asked this person how do you inpaint, how do you restore? And its very interesting because they were doing it the way that children draw. And here is the illustration. We have a region to be inpainted and we have the image. So, they first continue the boundaries of the regions. So, if there is a boundary here going inside the region to being painted. So then they continue the boundary inside. The same way that a child would basically draw the outlines of the objects. Then, they complete the color. So first, they continue the boundaries. Then, they propagate the colors inside the region. And then, lastly, they normally add kind of noise because paintings, and in this case it was a professional restorator of paintings don't look flawed so you have to kind of add this granularity to make it look more natural. And we are going to talk about these. First we will talk about these two steps. Then we are going to discuss a bit about this step which there are many techniques to do that. That's what. So remember, there's kind of a continuation of the geometry, basically the boundary and then filling with the colors. It's like if this is water, we are going to let the water flow into the region, but because we have the boundary, it's going to actually, the right water is going to flow into the right regions. Now, the job is to design a partial differential equation that emulates this process. And here it is. So we have basically the image outside and we have a region of missing information inside. And we want to propagate the information from outside in. The way we're going to do that is with this equation. This is the first component of imaging painting with partial differential equations. So what's happening here? Is the information that we want to propagate. I am not telling you yet what is that information.So is the information. That we want to propagate. And n is the direction that we're propagating that information. So n is the direction that we're doing the propagation. Now look what happens here. We have gradient l meaning the change of L. Inner product with N = 0. This is a change of A and it's projected, because of the inner product, into N, the direction of propagation and we want it to be equal to zero. So this is of course a vector. It's the gradient of information. So we have a vector. And we want that vector to be perpendicular to the direction that we are propagate, and we're have, so this is going to be the gradient of L, and this is going to be the direction. We want them to be basically perpendicular in the sense that we want to propagate information L. In such a way that is it does not change, it does not change in the direction of the propagation.That's exactly what it means propagating information. I move it in the direction that is constant along that direction. That's what this equation is basically telling us. That L, the information is constant in the direction of propagation, constant along these directions. Now how do we solve this with partial differential equations? Assume that the image is I. [SOUND] Now, we have this and we want to deform the image, we want to change the image, this is a partial differential equation, in such a way that at the end, we get this equal to zero. We have seen how to do that when we're talking about Euler-Lagrange equations. When we want something to be equal to zero, how do I deform the image in order to accomplish that? Let's think for a second how to do that. Remember last week. How do we do that? Very simple. We basically do this. Let me just go back one sec. We basically do this. We make the image change in time according to what we want. Now, when this doesn't change anymore, meaning that steady state, we get this equal to zero. That's the definition of steady state: no more change. Actually, the computer, when this very little change, we can stop. When this is ideally zero, we got what we wanted. L. Constant in the direction of propagation. So, this is actually a very simple trick. Every time you want to do something like A = zero, whatever is A, some information of the image, you do dI dT = A. And you run that to a steady state. That's basically what we did for the all day Lagrange. So this is the basic equation for this type of imaging painting. Now, we haven't finished. I have to tell you what's L and what's M. What's the information that we want to propagate? And in which direction do we want to propagate? And there's a lot of art here. So I'm just giving you examples. I'm teaching you the concepts. And these are examples that are already very powerful. But you can come with different examples. Now, let me start by the information. We want this propagation to be smooth. We don't want to see a big jump inside the region that we are inpainting. If we see a big jump, we're going to notice that. So we want it to be smooth. So it has to be anything that represent basically smoothness in the image, and one of the things that represent smoothness is the Laplacian of the image, the second derivative of the image represent smoothness. Now. Why not the first derivative? Wait a second, and you're going to understand why. Because the first derivative is also a representation of this bothness. Now, in which direction are we going to propagate? And now we go back to what we learned from professional restorators. They say, continue edges. Now, we know that if we have an edge, the gradient of the image is perpendicular to the edge. We learned that, that week, last week, so we are going to want to propagate along the edges, which means that we have to take the perpendicular to the gradient, so n is the perpendicular to the gradient, as written here, or we can write that n is basically the gradient. But the perpendicular to it. So that's important thing here, that we want to continue edges. And then, for that, we have to go perpendicular to the gradient. And that basically also explains why we cannot take the first derivative of L. Because if we were to put L. Equal to the first derivative, let's say the gradient. These two guys are always zero. So I'm not solving anything. Because the gradient and its perpendicular are always zero. And I wanted basically to go kind of. Constant in that direction. This is not enough. There is also another reason why, okay this is always perpendicular, it's not very good. Although remember we're taking the gradient of L. But the other reason is because we want. Two things to be smooth inside. We want to continue the grey values inside to be smooth. We also want to con, continue the boundaries to be smooth inside, so we need higher order of the realities. Not only because these two guys are perpendicular and then there's not enough information, even if we take the gradient of L, which is what actually N pending is propagating. We want to go a bit higher order so we get even more smoothness. And as I said, the Laplacian is one of the simplest measures of smoothness. So, we get smoothness. In the opposite direction or perpendicular to the gradient or on the level line's direction. And that basically gives us our equation. So, we have the change of l. And this is the normal, is the perpendicular to the gradient. What we're going to get at the end of this, when we get to steady state, when this becomes zero we basically get that the Laplacian became constant along the edges. And we have propagated the Laplacian along the edges. That's what basically this basic equation makes and once again you just discretize it in the computer and you run it and you get Image Inpainting. You get propagation from outside in. Every differential equation has to have boundary conditions. So, if this is my image and this is my region to be in painting, what do I assume here? And these boundary conditions for this type of equations, a bit of a strange equation, it has three derivatives, two because of the Laplacian, one more because of the gradient. Remember. We talked about this in the past. But the Laplacian is the second derivative of the image. In one direction plus the second derivative of the image in the other direction. And then we take the gradient of this. And we get a third derivative of considering both gray values and edges in the boundary and those are smoothly propagating inside thanks to have basically the Laplacian here and these are as we talked the level lines of the image the isopoles the regions of edges, the directions of edges. These are very simple but very, very interesting equation. Let us see some examples. So always when you do image processing, start with very simple examples to see what's happening. And here is a very simple example. White is the region. Then it's to be inpainted. And you can see here that there is a very smooth continuation. Really, really nice. A smooth continuation. It continues the boundaries nice. It, of course, continues the colors, black and grey. But even more, more important, it really completes the circle. It doesn't know that there is a circle there. It just goes in the normal direction. Perpendicular to the normal direction in the edge direction. Remember the gradient is in this direction. The perpendicular to the gradient is in this direction. So it's going and it's filling in. Inside the region. Propagating information in that direction. Really, really nice result. Of course, if it works only in artificial images it's not very good. Hopefully, it works on the, also on real data. Here is an example. What is being impainted? The letters. We wrote on top of this image. This is a nice image of New Orleans. And we draw with a basic rough and tough and boom, the letters are gone. So the regions to be painting are all these letters and we go inside and we make the letters disappear. We're propagating information from outside in. So hard to see here and now you get the very nice image where we invented the information that is covered by the letters. Based on information that is surrounding the letters. Very nice result, I think. Here is another example of basically an image that has deteriorations and degradation. The red are the images, the regions, that we are going to in paint. Sometimes, you mark regions that are a bit larger than the actual missing information. Look here. Is very thing. You do it a bit larger. It's like giving an impulse. So this is propagated from outside, in. You start from a, be far away. It helps you have to move even more smooth as we want. And here are the results. And it looks pretty good, some place you might see a tiny error here because it's covering a lot of the eye of this girl, so it's very difficult to invent information when there is nothing in the surrounding area that hints about what needs to be basically put in. But of course you get an extremely good. At least initial result looks almost excellent and the a professional restorator you see different programs may basically be able to go and repair only that tiny region if needed to. Sometimes, there's no need for that. Here is the original one. We have already seen it. A person is jumping with a cable. Now the cable is gone, and now you see a zooming region. Once again, look how nice the socks continue. Very nice continuation of this thread H because that's what we wanted, we were continuing HX that's exactly what we wanted, and that's exactly what this type of algorithm does. So, on our example, just a special effect, let's just look at what's happening here. And basically they're all gone. I'm going to just show you that again. There are people in the boat and they're all gone. We're basically in-painting what's basically been covered by the people sitting on the boat and basically they're all gone. Again, in-painting means modifying an image in a form that is then non-detectable, at least by the regular consumer. Here is another example of, we have scratches and now the scratches are gone basically, we basically removed those scratches. We inpaint in the information from the surrounding areas and again, looks very nice. It looks very natural as it was before it was all scratched. Now there is one example. And this is the last example I want to show in this video where it's actually not very hard to detect the regions that need to be inpainted. In other previous examples we, basically the user, defined somehow the regions. Now, this is an example, for example, of wireless transmission of images. Remember, images are transmitted, are restored, using JPEG. JPEG works in blocks. What happens is, if when you're transmitting an image, there's a signal drop. For example, on the cell phone. So half a block gets dropped. Now, signal drops are not very hard to detect, and basically. You, you have no signal and you detected it. Okay? So when there's a drop you can say, hey, I'm basic going to recover that block by mention painting. And that's what we are simulating here. And this is the original and all these blocks that you see here basically are dropped signals, simulation of dropped signal. You see that there's no signal, you plug that into a painting and then boom, you reconstruct your image. Let me show you that again, you basically see signal drop. And you reconstruct. So basically you may have a better cellular phone, a cellular phone that detect signal drop in images and boom does imaging painting, this is still JPEG, still the person that is sending it to you doesn't know. That the signal might drop. At the receiver end, you see a signal drop. This actually can help even compression. You can drop on purpose some of the blocks, because we know how to recover them with image in painting, and in that way, you basically improve the compression ratio. Some blocks are very easy to recover, and then you can actually not even send them or not even store them. Makes the reconstruction a bit more computationally expensive, but makes the compression possibly much higher. So this is the last example from this video. But there is one thing I haven't told you and that was on purpose just to keep you wondering a bit. I gave you the equation of. Image in painting, but here we have three colors. How do we do that? How do we go from that equation that I gave you before to colors? Remember the equation was something like IT equal gradient of ii and the I. In a product. Gradient I. Perpendicular, so. You can basically do this for each one of the channels. Lets say in red, green, blue or in any other color space. Or you can think about extending this to color images, to vectors. We know the concept of gradient for color vectors and we can basically define the Laplacian for color vectors as well. So, both possibilities exist. You can treat every color independently or you can try to use definition of vectorial fields for these, every component here Laplacian and gradient in vectorial fashion. All the examples I show you, every color is treated independently. So, this is an example of imaging pending with partial differential equations. Next video, I'm going to show you an example of energy painting with variational formulations that we learned last week. Very simple variation of formulation that will lead us to also a very nice energy painting technique based on the same colors of propagation and boundaries. I'm looking forward to seeing you in the next video. Thank you.
