FALL
2009
AMS
556 DYNAMIC PROGRAMMING
T,Th 12:50 – 2:10, Melville Lbr
N3033
Professor Eugene A. Feinberg, 1-110
Office Hours: T, Th 9:30 – 10:30 and
by appointment.
Course information and Lecture Notes
will be available on the Blackboard.
This course is an introduction to
the theory of sequential stochastic and deterministic optimization and its
applications. Stochastic sequential (or
multistage) optimization problems are also known in the literature under the
name of Markov Decision Processes (MDPs). This course deals with finite and infinite
horizon problems, with problems with finite and infinite state spaces, with
discrete and continuous time problems, and with various applications.
MAIN
TOPICS
I. Finite horizon problems and general
principles.
Model and policy definitions.
Principle of optimality. Dynamic
programming equation. Randomization and
de-randomization.
II. Infinite horizon problems: total
reward criteria.
Positive, negative, and discounted problems. Value iteration and policy iteration
algorithms. Linear programming approach.
Problems with multiple criteria. General
convergent models. Infinite state
problems.
III. Infinite horizon problems: average
rewards per unit time.
Canonical triplets. Policy and
value iteration. Linear programming
methods. Multiple criteria problems. Blackwell optimality. Infinite state problems.
IV. Continuous
time problems. Semi-Markov
Decision Processes. Continuous-Time Jump
Markov Decision Processes.
V. Applications.
Production and service operations (inventory and queueing
control). Financial applications. Telecommunications applications. Neuro-dynamic programming.
Textbook. Lecture notes will be available on the
Blackboard.
Some important books on Dynamic
Programming/MDPs:
E. Altman, Constrained Markov
Decision Processes, Chapman&Hall/CRC,
R.E. Bellman, Dynamic Programming,
D.P. Bertsekas,
Dynamic Programming and Optimal Control. Volume I (second edition),
Athena Scientific,
D.P. Bertsekas,
Dynamic Programming and Optimal Control. Volume II, Athena Scientific,
D.P. Bertsekas
and S.E. Shreve, Stochastic Optimal Control: the Discrete Time Case,
Athena Scientific,
D.P. Bertsekas
and J.N. Tsitsiklis, Neuro-Dynamic
Programming, Athena Scientific,
C. Derman,
E.B. Dynkin
and A.A. Yushkevich, Controlled Markov Processes,
E.A. Feinberg and A. Shwartz (editors). Handbook of Markov Decision
Processes: Methods and Applications. Kluwer,
O. Hernandez-Lerma
and J.B. Lasserre, Discrete-Time Markov Control
Processes: Basic Optimality Criteria. Springer,
O. Hernandez-Lerma
and J.B. Lasserre, Further Topics on Discrete-Time
Markov Control Processes. Springer,
D.P. Heyman
and M.J. Sobel. Stochastic Models in Operations
Research. Volume II: Stochastic Optimization,
A. Hordijk,
Dynamic Programming and Markov Potential Theory, Mathematical Centre
Tracts,
R.A. Howard, Dynamic Programming
and Markov Processes, MIT Press,
L.C.M. Kallenberg,
Linear Programming and Finite Markovian Control
Problems, Mathematical Centre Tracts,
H. Mine and S. Osaki, Markov
Decision Processes,
A.B. Piunovskiy,
Optimal Control of Random Sequences in Problems with Constraints, Kluwer,
M.L. Puterman,
Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley&Sons,
L.I. Sennott,
Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley&Sons,
J. van der
Wal, Stochastic Dynamic Programming,
Mathematical Centre Tracts,
Grading policy: 30% Home work
average, 30% Midterm paper, 40% Final paper.
If you have a physical, psychological, medical, or learning disability that may impact your course work, please contact Disability Support Services at (631) 632-6748 or http://studentaffairs.stonybrook.edu/dss/. They will determine with you what accommodations are necessary and appropriate. All information and documentation is confidential.
Students who require assistance during emergency evacuation are encouraged to discuss their needs with me and Disability Support Services. For procedures and information go to the following website: http://www.sunysb.edu/ehs/fire/disabilities.shtml