Information Theory and Statistics: an overview

Daniel Commenges
Institut de Santé Publique, d'épidémiologie et de développement (ISPED), Université de Bordeaux
and
Institut National de la Santé de la Recherche Médicale (INSERM)

We give an overview of the role of information theory in statistics, and particularly in biostatistics. We recall the basic quantities in information theory; entropy, cross-entropy, conditional entropy, mutual information and  Kullback-Leibler risk. Then we examine the role of information theory in estimation theory. Then the basic quantities are extended to estimators, leading to criteria for estimator selection, such as Akaike criterion and its extensions. Finally we investigate the use of these concepts in Bayesian theory.