Statistical Performance Measures

How to select the best model from the available options?

Statistical performance measures are often used for model selection in machine learning and statistical inference. From multiple models trained with different sets of hyper-parameters and parameters, the one that gives best performance in terms of a selected performance criterion is finally adopted. Let’s understand commonly used performance metrics for model selection through an example so that we can choose one of these for our model selection.

Let’s assume that we have a pregnancy test that gives us binary results, positive (or 1), if tested individual is pregnant and negative (or 0) if tested individual is not pregnant (~pregnant). Now, we can define the following outputs for this binary test-

True positive: Test results are + when an individual is pregnant

True negative: Test results are - when an individual is not pregnant

False positive (Type I error): Test results are + when an individual is not pregnant

False negative (Type II error): Test results are - when an individual is pregnant

These four scenarios are illustrated in the following figure.

Figure 1 Four possible combinations of reality and our binary pregnancy test results

Let’s say we evaluate N randomly selected individuals for pregnancy using our test and note down the…