Hello and welcome back. Pseudo-Coloring and colorization of a gray scale image are covered in this segment. The two topics closely related since with both a black and white image is turned into a color image. The objectives however of the two problems differ. A number of objectives drive Pseudo-Colorings, such as to place normal objects in strange colors for attention, color a normal scene to match the color sensitivity of a human viewer, exploit contrastivity, produce a natural color representation of a set of multi-spectral images of a scene, and enhance the visual quality of the image in general, and reveal details and structure in it that were not visible before. Colorization of black and white image, on the other hand, has as its objective. In most cases the recovery of color that existed in the scene but was simply not captured by the sensor. It is a technique that was made popular in colorizing old black and white movies, which were filmed when color was not available. We will discuss some basic information of both Pseudo-Coloring and colorization approaches. We will describe a specific colorization example, that of colorizing a black and white photograph of a famous painting by Matisse, revealing valuable information about the process the famous master was applying in his work. Let us then proceed with this exciting topic. Some of the motivations, along with some examples of turning a gray scale image into a color image image are discussed next. Pseudo-Coloring or false coloring is an enhancement technique of assigning colors to gray scale values. It is done primarily for human visualization and interpretation of gray scale events. More specifically some of the reasons are, we might want to place objects in strange colors for attention since humans will notice odd colored objects more than others. We want to take advantage of the color sensitivity of the human viewer. The luminous response of roads and cones the retina picks in the green region of the visible spectrum and we want to account for that. We want to exploit the contrasts sensitivity of the eye to changes in the view light. So we map normal colors of objects with fine details into shadows of blue. And we want to produce natural color presentation of multi-spectral images of a scene. And some of these images may not even be, obtained by sensors whose response is within the visible wavelength range, such as infrared and ultra-violet. I've also included here the colorization of black and white images and videos. Although, there, in many cases the objective is to reproduce the true colors that were there in existence, although, all we have available is a black and white image. By and large the colorization of black image and videos depends on initial selection of colors and therefore it's looked upon as an enhancement technique. So color is important because also humans by and large can discern thousands color shades and intensities, compared to only a few dozen or so of shades of gray. We show here the seven bands of a LANDSAT multi-spectral image. This image actually was shown earlier in the introduction of the course. So this is the blue, green, and red bands, and the other four are infrared bands. This is actually the LANDSAT image of the city of Amsterdam. Instead of looking at the bands one at a time as shown here as a, as a grayscale fashion, we can combine the bands and produce color images such as this one. These are actually from a different area. These are from the city of London. So combining the three visible bands, one obtains an image like this. The colors are kind of natural looking. The city shows here in gray. The water is blue and vegetation is in green. Combining the, 4-3-2 bands, 1- 2-3 are the visible ones. So there are two visible here and one infrared. One is able to better see the vegetation, if red is reflected differently. And here's a combination of two infrared and one visible band. And in this case, this is suitable for geological, agricultural, purposes. >> We see here a beautiful Pseudo-Color image of the Great Lakes area, taken by this MODIS, which stands for Moderate Resolution Imaging Spectroradiometer, aboard the NASA satellite, Aqua satellite. So this is Lake Michigan. And Chicago is around here. Lake Superior is up here. Huron, this one and so on. So, for this first color image, a combination of a shortwave infrared, a near infrared, and the red channel were used. And with this Pseudo-Coloring we're able to distinguish ice from snow and water from clouds. We see Lake Superior up here is completely frozen, and this was taken in the recent winter of 2014. We describe here some of our own work on colorization. This additional picture of the masterpiece by Matisse, Bathers by a River which is owned by the Art Institute of Chicago. This is an impressive painting, which measures almost four by 2.6 meters. It is considered to be one of the five most beautiful works of his career. He worked on it, on and off for almost nine years. The exact periods are shown here. Sometime around the beginning of November 1913, Matisse arranged for the photographer and dealer by the name of Druet to record the state of the painting. So the black and white photograph he took, is shown here. The artist kept working on the painting after the photograph was taken for another four years. So, the photograph is very different from the current state of the painting. So the problem at hand is to reproduce the color information of the painting back in November of 1913, as if Druet was using a color camera. Or in other words, we want to colorize the grayscale image. Colorization techniques have been utilized for movies since the 1970s. However, the requirements for painting colorization are different, since high accuracy and fidelity are needed. And the image formation model differs significantly from the natural or cartoon type of images that we use primarily for colorization. In our colorization algorithm, some initial conditions are needed. Pixels for which the color is known, and then this color is propagated through the rest of the pixels in the image. These initial conditions are typically referred to as scribbles. In finding this [UNKNOWN] conditions for the problem at hand, we found the correlation between the intensity values of the direct picture and the intensity values of the painting in its current state. We show the maximum of these correlation values. So, for this location shown here, where the correlation is, is large, we can borrow the color from the current state of the painting to utilize, to colorize the, the drab picture. In addition, microscopic holes were drilled into the painting. And here we see a cross-section of the painting at 200 times magnification. So it's, since, since Matisse was adding or scraping away, painting while, just working on this painting for nine years, it's not straightforward to identify which layer corresponds to which year. However, the curators were able to say, for example in this case, this gray layer here belongs to the 1913, November, 1913 period that, the picture, the great photograph was taken. So this additional information from this investigation, this forensics here, was utilized as well to generate scribbles, and at the end these are the scribbles we are going to use, we used in colorizing the Druet photograph. So the basic principles of the algorithm we used to colorize the Druet photograph are discussed here. So, changes in color correspond to changes in intensity and therefore based on that, if two pixels have similar intensity they will be assigned similar color. We worked in the YUV color space. We have the Y of course, the intensity of the luminance. And to solve for the two chrominance components, U and V. So in solving for U, for example, we have the value of U at location r, as shown here. Then you consider the neighborhood, such as the red window shown here. And you're solving for the intensities in these neighboring locations. So we're trying to minimize this square [UNKNOWN] term, but the, intensities are weighted by this factor which is dependent on the difference in intensity. So if this pixel and this pixel have similar intensities then this weight is going to be close to one. If the intense is similar then, that is going to be small. So by doing so we form a system of equations, linear equations, say x equals b, and solving that we just propagate the color from the known scribbles to the rest of the pixels image. So finally we see here the black and white photograph again, the Druet photograph taken November 1913, and here are the results of colorization of this photograph based on the algorithm that I just briefly described. The colorized result is smooth. He has limited color blending, and is consistent with conservation and art history research. It reaffirms that he started color counts of how the painting appeared at that time period. This work helps support research that has for the first time uncovered how Matisse begun the work as a highly chromatic, more naturalistic scene, but it changed it to ex, to explore new artistic directions on one hand, while at the same time, reflecting the graver national mood due to the first World War. Furthermore, the colorized version helped curators and art historians visualize the process by which Matisse composed and transferred this complex work to reach its final form, and also helped them to identify other works by the artist reflecting a similar process.
