In this class you will learn the basic principles and tools used to process images and videos, and how to apply them in solving practical problems of commercial and scientific interests. Digital images and videos are everywhere these days – in thousands of scientific (e.g., astronomical, bio-medical), consumer, industrial, and artistic applications. Moreover they come in a wide range of the electromagnetic spectrum - from visible light and infrared to gamma rays and beyond. The ability to process image and video signals is therefore an incredibly important skill to master for engineering/science students, software developers, and practicing scientists. Digital image and video processing continues to enable the multimedia technology revolution we are experiencing today. Some important examples of image and video processing include the removal of degradations images suffer during acquisition (e.g., removing blur from a picture of a fast moving car), and the compression and transmission of images and videos (if you watch videos online, or share photos via a social media website, you use this everyday!), for economical storage and efficient transmission. This course will cover the fundamentals of image and video processing. We will provide a mathematical framework to describe and analyze images and videos as two- and three-dimensional signals in the spatial, spatio-temporal, and frequency domains. In this class not only will you learn the theory behind fundamental processing tasks including image/video enhancement, recovery, and compression - but you will also learn how to perform these key processing tasks in practice using state-of-the-art techniques and tools. We will introduce and use a wide variety of such tools – from optimization toolboxes to statistical techniques. Emphasis on the special role sparsity plays in modern image and video processing will also be given. In all cases, example images and videos pertaining to specific application domains will be utilized.
Image Restoration
Loading...
From the course by Northwestern University
Fundamentals of Digital Image and Video Processing
724 ratings
From the lesson
Image Recovery: Part 1
In this module we study the problem of image and video recovery. Topics include: introduction to image and video recovery, image restoration, matrix-vector notation for images, inverse filtering, constrained least squares (CLS), set-theoretic restoration approaches, iterative restoration algorithms, and spatially adaptive algorithms.
Meet the Instructors
Aggelos K. KatsaggelosJoseph Cummings Professor
Department of Electrical Engineering and Computer Science
In this segment, we introduce the classical restoration problem.
According to it, there is a system between the original
and the acquired image, which introduces some type of degradation.
The system can represent the Hubble space telescope, or the turbulent atmosphere.
Or be a combination of the motion of an
object in the scene, and the motion of the camera.
When the degradation system is linear and spatially invariant.
The both the modeling of the degradation
and its removal, are, relatively speaking simplified problems.
We show the mathematical models for the systems, introducing some of the common
encountered LSI degradations, of course there's almost
always the ever present noise in the data as part of the degradation.
We show how is the severity of noise typically measured in image restoration.
As well as an objective metric used
for estimating the improvement in the restored image.
When a synthetic experiment is conducted.
Of the recovery problems we described,
such as concealment and painting, resolution.
They have a common structure, they're all inverse problems.
So, we assume there is a system, which you denote here by H, that takes
an image at the input F, and produces an image G at it's output.
And it could be a single image or multiple images or a video.
When we observe G,
and or solving for F,
it's a recovery and
inverse problem.
If only G is known, and still the objective is to recover F,
and of course H along the way, then becomes a blind recovery problem.
And the in between situation is when G is observed, and H is partially known.
So the shape may be of age is known, and but the, the [INAUDIBLE] but
the width of the [INAUDIBLE] is not known, so there's a parameter I need to estimate.
And in that case, it's referred to a semi blind recovery problem.
Although the main objective is again, to estimate
F as we'll see in a number of examples.
There are a number of parameters that are
introduced and they need to be estimated, as well.
If f and g are known, and the objective is to find h.
Then this becomes a system identification problem and if f and h are known, and.
The motion estimation problem is an inverse problem.
The disparate estimation, I just mentioned it in passing at some point, it's also an
inverse problem, and the bounded detection through
differentiation can be formulated as an inverse problem.
So inverse problems are encountered are all over the place.
It's a topic in Applied Math that he has seen a lot of activity
for the long past and it's a very rich literature.
There are numerous techniques that can be utilized to solve such inverse problems.
The adaptivity on restoration was a result of the space race
between the US and former Soviet Union in the 50s and 60s.
So for example, the 22 images produced during the
Mariner IV flight to
Mars in 64, cost them
million dollars so
this were is the case with most area
the activity comes in waves.
It's driven, for example, by mishaps, such
as the Hubble space telescope mishap, new applications,
such as the super resolution of images and
videos for these huge ultra high definition displays.
And, by new mathematical developments, and sparsity of course the one
that stands out and we'll talk about it at a later point.
Restoration has seen some not alright in the media.
There's a brother field of the assassination
of JFK, underwent a number of restorations.
And then in a number of movies such as no way out and rising sun, the whole plot
relies on the successful restoration of some surveillance video.
Let us look now at the components of a degradation restoration system.
F is the original input, image.
Is input to the degradation system H.
Noise is added to the output of a system, and this gives rise to the observation Y.
So, here's how the input looks, it's an aerial shot, and here's
the blurred version of this by H, plus the addition of noise.
So the objective of restoration is to design a system r, which will
operate on the observation y, and give us an estimate, of the original image f hat.
That will be as close as possible to
the original one, subject to the optimal criterion.
In carrying out this restoration step, we assume that we
have perfect knowledge of h this is the nonblind version.
Then we also need to know the noise statistics, and
here there's a double sided arrow, or arrows going in and out from our.
To indicate that these statistics can be updated
based on the partially restored image, for example.
And another component which is extremely critical when it comes
to restoration, is prior knowledge on f, on the original image.
And this prior knowledge can be expressed in various ways
and we'll see examples of that as we proceed here.
But it's also important that we are able to incorporate such, such
prior knowledge in a rather straight forward way into the Restoration Filter.
Wnd again there are two arrows,.
Going, towards and away from this prior knowledge box to indicate
again that this prior knowledge can be updated as the iteration progresses.
Now, the noise statistics can be given to us which is idealistic or in most cases
we measure them, let's say the noise variance
under a Gaussian assumption from the available data y.
Similarly if h is not known, then it needs to be estimated from the available data y.
This is referred with a blur identification
problem and in carrying out such a
step, it's also important that we have
some prior knowledge on the properties of h.
So you do see here all the components
that are involved in designing a Restoration system.
Again, prior knowledge is of the utmost
importance since it restrains the solution set.
And produces an estimate here of that is
as close as possible to the original image f.
As it became clear from the previously slide in carrying out the striation,
that blur is needed and knowledge estimate is needed if they're not known, then.
There the steps of blur identification,
noise estimation, and restoration that that involve.
And these three steps can be
done independently, using any technique of choice.
However, at least conceptually, it should be clear
that if their done at the same time, simultaneously.
And by that I mean that an error
in identifying the blur is taken into account when
the noise is estimated, and this error is
taken into account when the restoration is carried out.
Then one should expect improved results.
Stochastic methods, we'll talk about them a little bit of their,
an advantage of that nature, that all these steps can be.
Taken, simultaneously.
And again, repeating here, prior knowledge on all the unknowns.
H, f, and n is of the out most importance in providing a solution here,
f hat, that is, as, as close as possible to the original image we're after.
So translating the operations shown by the block diagram into an equation
here we have the observed image y i j is the result of the
application of this operator of this system h onto
the original image plus the noise that has been added.
So this is a very generic, general model
that can be found in very many different applications.
So the problem at hand is, given y, and given knowledge of H, we try to find
an estimate of f and, of course, knowing the statistics about the noise.
So this is the Restoration problem.
[SOUND]
Now, in many applications, this system can be well approximated by linear
and Spatial Invariant system and noise of course additive, and signal dependent.
And then as we know by now, in this case, the operation of
H on f is given by the familiar two dimensional convolution.
So the output, the observed image is the
convolution of the input with the impulse response
of the LSI system plus noise, and here
is the expression for the two dimensional convolution.
So in this particular incarnation, solving for F, given the impulse
response of the system, and the available data, becomes now a deconvolution.
Deconstruction
problem.
The system convolves, and the inverse problem we're solving is to
deconvolve the operation of the system, undo what the system did.
So restoration is a more general term, applies to this general equation, if
the system again introducing degradation is
LSI, then I'm solving de convulsion problem.
We show here that impulses sponsor some
of the most commonly encountered degradation systems.
So if the degradation is due to one dimensional
motion between the camera and the scene, so the object
in the scene move on a plane that is
parallel to the camera plane this is a D plane.
And then they move in a horizontal direction, then
here is the impulse response of this degradation system.
So here's this shape.
[BLANK_AUDIO]
And here we assume that l is an even number so l over two is an integer.
The extension of this is if the motion
is at an angle, not along the horizontal direction
but at an arbitrary angle theta, and that's also
something that can be handled with not major difficulty.
If the distortion is due to atmospheric turbulence, then the
long term exposure through the atmosphere is modelled like this.
This impulse is responsible for the Degradation system, so this,
this ocean here controlled by the variants of the ocean.
If it's an out of focus Degradation, then this disc,
in two dimensions, i, j, is, represents the impulse response of
the system, so it's a function of the severity of a,
the focusing is a function of the radius of this disc.
And finally the Pill-box distortion is kind of the extension
of the one demotion so it square here
from minus element two clone.
Two in both dimensions and again l in this particular case is
assumed to be an even number so that l over two is an integer.
So well be making use in some examples so some of these Degradations.
But there again can be found in the literature there rather widely used.
In the image restoration community, in assessing
the quality of the degraded image, we
utilize not the signal to noise ration,
but the blurred, signal to noise ratio instead.
So here's the model we've been using, the Degradation model, the observed image
is the convolution of F with impulse response of the system plus noise.
So if we called the Blurred Image here GIJ, then in the num,
numerator you see the variance of G bar is the mean or expected value of b.
We assume the image has m times n pixels.
Therefore, the numerator is the variance of the blurred signal
and then the denominator is the variance of the noise.
In assessing the quality of the restored
image for simulation situations where the original
image is available we use this ISNR matrix improvement in signal to noise ratio.
So in the b term log base 10 in the denominator of this fraction
we'll put the difference the square there between the original f is the original.
Which, again, in a simulation environment is available and f hat
is the restored, so the distance between the original they restored.
So, that's the error in the estimation and in the
numerator is the quantity f, the original minus y, the observation.
So this arrow squared.
So these are two quantities that we will be utilizing for the
rest of this presentation restoration, Blurred
Signal-to-Noise Ratio and improvement in Signal-to-Noise Ratio.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
