Preface
This book aims to provide a mathematical but friendly introduction to the fundamental concepts, basic problems, and classic algorithms in reinforcement learning. Some essential features of this book are highlighted as follows.
* The book introduces reinforcement learning from a mathematical point of view. Hopefully, readers will not only know the procedure of an algorithm but also understand why the algorithm was designed in the first place and why it works effectively.
* The depth of the mathematics is carefully controlled to an adequate level. The mathematics is also presented in a carefully designed manner to ensure that the book is friendly to read. Readers can selectively read the materials presented in gray boxes according to their interests.
* Many illustrative examples are given to help readers understand the topics better. All the examples in this book are based on a grid world task, which is easy to understand and helpful for illustrating concepts and algorithms.
* When introducing an algorithm, the book aims to separate its core idea from complications that may be distracting. In this way, readers can easily grasp the core idea of an algorithm.
* The contents of the book are coherently organized. Each chapter is built based on the preceding chapter and lays a necessary foundation for the subsequent one.
This book is designed for senior undergraduate students, graduate students, researchers, and practitioners interested in reinforcement learning. It does not require readers to have any background in reinforcement learning because it starts by introducing the most basic concepts. If readers already have some background in reinforcement learning, I believe the book can help them understand some topics more deeply or provide different perspectives.This book, however, requires readers to have some knowledge of probability theory and linear algebra. Some basics of the required mathematics are also included in the appendix of this book.
I have been teaching a graduate-level course on reinforcement learning since 2019. I want to thank the students in my class for their feedback on my teaching. I put the draft of this book online in August 2022. Up to now, I have received valuable feedback from many readers. I want to express my gratitude to these readers. Moreover, I would like to thank my research assistant, Jialing Lv, for her excellent support in editing the book and my lecture videos; I also thank my teaching assistants, Jianan Li and Yize Mi, for their help in my teaching; also my Ph.D. student Canlun Zheng for his help in the design of a picture in the book; and my family for their wonderful support. Finally, I would like to thank the editors of this book, Mr. Sai Guo from Tsinghua University Press and Dr. Lanlan Chang from Springer Nature Press, for their great support.
I sincerely hope this book can help readers enter the exciting field of reinforcement learning smoothly.
Shiyu Zhao
May 2024