Talk (Work in Progress): Conny Knieling (Pittsburgh)

Title:

The Mystery of Generalization in Deep Learning

Abstract:

In recent years we can see a tremendous success of deep learning, especially in applications where the practical accomplishments of other algorithmic methods halted. Despite this, there does not exist a good understanding of why deep learning is so successful. The question of how deep learning is capable of generalizing, i.e., performing well on unseen data, stands at the core of this mystery of its unreasonable effectiveness. Explaining
how deep learning models generalize and how to build models that do is also especially important for their applications in science. Nevertheless, it remains an unanswered question in machine learning. An influential paper by Zhang et al. (2017) has further shown that the generalization success of deep learning contradicts conventional wisdom in computer science and statistical learning theory more generally. They proved many traditional approaches to understanding generalization in deep learning to be deficient. In response to this puzzle, several proposals to explain deep learning’s generalization capabilities have been brought forward since. In this presentation, I will compare and evaluate these proposals. I will argue that the current answers to explain the puzzling generalization behavior of deep learning do not suffice to amount to a successful explanation while
defending what we ought to expect of a successful theory of generalization. In doing so, I will also explain why the existence of this mystery has implications beyond being an open question in theoretical computer science.