This course aims at introducing the fundamental concepts of Reinforcement Learning (RL), and develop use cases for applications of RL for option valuation, trading, and asset management. Prerequisites are the courses "Guided Tour of Machine Learning in Finance" and "Fundamentals of Machine Learning in Finance".
Fitted Q-Iteration: the Ψ-basis
Loading...
From the course by New York University Tandon School of Engineering
Reinforcement Learning in Finance
2 ratings
New York University Tandon School of Engineering
2 ratings
Course 3 of 4 in the Specialization Machine Learning and Reinforcement Learning in Finance
From the lesson
MDP model for option pricing - Reinforcement Learning approach
Meet the Instructors
Igor Halperin
So, in the last video we introduced two parameterizations for the Q function.
One in terms of matrix WT and another in terms of
vector U_W which was given by a product of matrix WT and vector phi of basis functions.
We saw that if we somehow observed
optimal actions HE star and optimal Q function values QT star,
this would show us two equations for three unknown components of vector U_W.
Now, this suggests that a right solution,
the one that would be found by the dynamic programming method if the dynamics are known,
should be in the space of solutions parameterized by
a time-dependent matrix WT as we did in our specification of the Q function.
So, we have to find a way to learn the matrix of coefficients WT.
Let's see how this can be done.
What we need to do is to rearrange terms in
our definition of the Q function to convert it into
a product of a parameter vector and
the vector that depends on both the state and the action.
So, how we do this.
We do it in several simple steps.
First, let's write explicitly the value of the Q function as
a trace of the matrix expression given here.
We write it as the element y is product of elements of matrix W with
elements of the matrix that is formed by direct product of vectors A and phi.
This gives us the second line in this expression.
We have two types of multiplication here.
First, this symbol stands for the element y is product of two matrices,
this product is also known as the Hadamard product of two matrices.
Second, this other expression has
this circle crossed product sign and this sign stands for
a matrix given by the outer product of two vectors A_t and Phi_t.
By definition, the element ij of this matrix is given by a product of A_ti and Phi_TJ.
So, so far so good,
we converted the whole expression into a trace of a product of two matrices.
But now, we can do one more thing and represent
this expression as a scalar product of two vectors.
What we have to do to this sense is to convert
both matrices entering this expression to vectors.
We can do this by concatenating columns of
matrix W and the matrix given by the outer product of A_t and Phi_t,
and these will give us the third line in this equation.
Now, we can compactly write this whole expression as
a dot product of vector W and vector psi.
And here, vector psi is obtained by concatenating in
the columns of the outer product of vectors A_t and Phi_t.
This can be viewed as a new set of basis functions that depend both on state and action.
The dependence on the action A_t is quadratic but the dependence on the state X_t
can be arbitrary and depends on a functional form
of the original basis functions Phi_N for the state space.
If we use for example cubic D splines as
basis functions as we did in the dynamic programming solution to the model.
Then the X dependence will be a local cubic.
Now, let's compare what we got with the setting of dynamic programming.
In our current Reinforcement Learning setting,
we reduce the expression for the Q function to a dot product of
vector W and the new vector of state action basis function Psi_t.
Both have a number of components given by 3M,
so that the number of unknown coefficients in this problem is 3M as you could also
easily guess just by looking at the matrix W_t which has dimension three by M. So,
we have 3M unknowns here,
and now what about the number of observed variables?
For each time step in the batch mode reinforcement learning setting,
we have 3M observables because we observe the state X_t,
the action A_t and the reward R_t for all
of N historical or Monte Carlo pass for the stock.
So, we have a 3N observables for 3M unknowns.
And if N is larger than M,
then our problem is well-posed,
we have N divided by M observations per one phi parameter in this setting.
Now, we can compare these with
the situation we had with the dynamic programming approach.
In that case, we had to find two quantities,
the optimal policy and the optimal Q function.
In our dynamic programming solution,
we expended both in M basis functions,
so we had 2M unknown coefficients in total.
This is less than 3M unknown coefficients in the Reinforcement Learning setting.
But on the other hand,
the number of observations per times slice in
a dynamic programming approach was also less than in Reinforcement Learning approach.
And this is because in the dynamic programming setting,
we only absorb the states.
So, for N Monte Carlo pass,
we only had N data points each time.
The number of observations per parameter was therefore N divided by 2M which is
actually less than the ratio of N divided
by M that we have for the Reinforcement Learning setting.
So, it appears that at least when the data for
Reinforcement Learning is generated from the dynamic programming solution,
both the reinforcement learning and dynamic programming methods
should perform about equally well.
This setting is called on-policy learning and it means that action is
required for training of
the Reinforcement Learning occasions were actually optimal actions.
It turns out that this is indeed the case and
both algorithms produce nearly identical results for such setting.
This actually gives us a simple way to benchmark
our Reinforcement Learning algorithms because we can't so far model,
when it's fully specified by means of dynamic programming,
we can compute the optimal policy from this solution,
then we can just use the absorbed actions and
the words obtained with this policy as data for
Fitted Q-iteration method and compute
the optimal policy and Q function using this method.
Because this solution is already known from dynamic programming,
we can use it to benchmark
our Reinforcement Learning solution and check the speed of convergence,
the final results, and so on.
This is actually very ensuring as we now know exactly what we expect to see,
so any problems with Fitted Q-iteration would immediately show up if found there.
If we now want to try some other Reinforcement Learning algorithm,
instead of Fitted Q-iteration.
Again, we have a good benchmark obtained with a dynamic programming approach.
Therefore, we can test
many different Reinforcement Learning algorithms with
our simple MGP model for option pricing.
Let's take a short pause here and continue in the next video with
the actual solution of the model with the Fitted Q-iteration method.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
