A deeper understanding of the principles of deep learning can consolidate and boost its already-spectacular empirical success. I will introduce some of the recent progress in the theory of deep learning, including some of my own work. We will discuss the core ML issues, such as optimization, generalization, and expressivity, and their rich interactions, in the contexts of supervised learning with (deep) non-linear models.