This course aims at introducing the fundamental concepts of Reinforcement Learning (RL), and develop use cases for applications of RL for option valuation, trading, and asset management. Prerequisites are the courses "Guided Tour of Machine Learning in Finance" and "Fundamentals of Machine Learning in Finance". Students are expected to know the lognormal process and how it can be simulated. Knowledge of option pricing is not assumed but desirable.
Batch Reinforcement Learning
Loading...
From the course by New York University Tandon School of Engineering
Reinforcement Learning in Finance
9 ratings
New York University Tandon School of Engineering
9 ratings
Course 3 of 4 in the Specialization Machine Learning and Reinforcement Learning in Finance
From the lesson
MDP model for option pricing - Reinforcement Learning approach
Meet the Instructors
Igor Halperin
In the last lesson we constructed the mark of decision process or MGP model for auction,
pricing, and hedging, and then solve it
using the Monte Carlo based Dynamic Programming approach.
To remind you of this Monte Carlo setting,
we simulated for POS for the underlying stock,
and then computed optimal policy,
and hence optimal actions.
The optimal option price was obtained as a negative of the time zero optimal Q function,
when the second argument is taken to be an optimal action.
Now we move to reinforcement learning.
As we said several times before,
reinforcement learning solves the same problem as dynamic programming,
that is it finds the optimal policy.
But unlike dynamic programming reinforcement learning does not
assume that transition probabilities and the work function are known.
Instead it relies on samples to find an optimal policy.
Now, why this approach is interesting?
It's interesting because it tries to go to the heart of the problem,
but without solving first another problem,
namely the problem of building a model of the world.
The conventional approach to Option Pricing requires
that we build the model of strong dynamic stock
by designing some stochastic process and
then calibrating it to auction and stock pricing data.
But let's note that diligent and model of the world is not our objective.
Our purpose is rather to find an optimal option price and hedge.
In other words our task is to find an optimal policy from data.
But this is clearly very different task from the task of building the model of the world.
And more to these in some cases,
the world might have very complex dynamics,
yet an optimal policy can be a very simple function.
Vladimir Vapnik the father of
support vector machines once formulated the principle that one
should avoid solving more difficult intermediate problems when solving a target problem.
Support vector machines that we discussed in
the previous course actually implement Vapnik's principle.
Now, in our case,
to price and hedge an option,
we do not have to explain the world,
but rather just need to learn to act optimally in this world.
This is our target task in Vapnik's principal.
The intermediate task would be to explain the world,
that is to build a model of the world.
According to the classical approach of quantitative finance,
we always first have to build a model of the world.
That is to formulate the law of dynamics and estimate model parameters.
This is called model calibration,
and it amounts to minimization of
some lost function between observables and model outputs.
Now depending on the model,
this might be a very resource demanding procedure.
But even after it is completed,
this is not the end yet,
as we still did not solve our main problem of auction, hedging, and pricing.
This takes another calculation,
though normally much less time consuming than
the first one because it doesn't involve optimization.
So, let's look at this traditional approach from the point of
view of the original problem of auction hedging,
which we will now view as a problem of optimal control in reinforcement learning pricing.
If we built a model of the world first,
we can apply dynamic programming to solve the problem of optimal control.
But any model introduces model specifications.
Therefore they can propagate into the quantities we actually care about most.
That is the optimal price and hedge.
So, what reinforcement learning approach does,
it focuses on this original task while
relying on data samples instead of a model of the world.
Therefore this approach implements the Vapnik rule.
Now, once we agree that the general approach of reinforcement learning
takes us straight to the ultimate goal at least conceptually,
we can discuss different particular specifications of this approach.
For example we can still have a model of the world or maybe know
some important model parameters in the search for enforcement learning.
Such approach would correspond to what is called model
based reinforcement learning as opposed to a model-free reinforcement learning.
Furthermore, there are different types of enforcement learning.
Some of them focus on direct policy search,
while some other maximize the value function as we outlined in the previous course.
We will concentrate for now on value based reinforcement learning which condenses
information it needs from the world to optimize policy into a value function.
Now, we will consider offline reinforcement learning also
known as Batch mode or simply batch reinforcement learning.
In this setting, we only have access to some historically collected data.
There is no access to real time environment,
and we also assume that no simulator of such environment is available.
Now, how our data looks like in the setting of batch reinforcement learning?
Data are given by a set of N trajectories,
and information set Ft at time t is given by information sets
of Ftn available from all seperate n trajectories.
Each set F(n) t includes
the historical values up to time t of the following tuples of values.
What is in this tuples?
We record the underlying stock price St,
which we express as a function of Xt.
The state variable hedge position at
instantaneous reward Rt and the next time value X sub t plus 1.
So Fn t is a collection of all such tuples as shown in this equation.
Infact as long as the dynamics hallmark of a collection of N trajectories of
lengths t each is equivalent to a data set of n times t, single step transitions.
We assume that such data set is available either as
a similar data or as a real historical stock price data,
combined with some artificial data that would track the performance
of a hypothetical stock and cash duplicating portfolio for a given option.
Now, neither the dynamics nor
the true reward distribution are assumed known in the reinforcement learning approach.
All that is given is a set of one-step transitions.
Now we can compare these data for reinforcement learning
with data we had for dynamic problem and solution to the model.
In a dynamic programming setting,
our only data was samples of stock prices simulated using Monte Carlo.
In the course of Beckworth recursion,
we computed instantaneously awards,
and then computed optimal actions that one should take to hedge the option.
Now let's compare this with the searching of batch reinforcement learning.
We have stock prices and the next steps
stock price which is the same data as any dynamic programming.
But instead of rewards and actions being
computed from the model using the knowledge of modal dynamics,
in batch reinforcement learning we are given sampled values of rewards and actions.
In the next video,
we will see how we can work with this data.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
