19. Confusion Matrix in Machine Learning
By Bernd Klein. Last modified: 05 Jul 2022.
In the previous chapters of our Machine Learning tutorial (Neural Networks with Python and Numpy and Neural Networks from Scratch ) we implemented various algorithms, but we didn't properly measure the quality of the output. The main reason was that we used very simple and small datasets to learn and test. In the chapter Neural Network: Testing with MNIST, we will work with large datasets and ten classes, so we need proper evaluations tools. We will introduce in this chapter the concepts of the confusion matrix:
A confusion matrix is a matrix (table) that can be used to measure the performance of an machine learning algorithm, usually a supervised learning one. Each row of the confusion matrix represents the instances of an actual class and each column represents the instances of a predicted class. This is the way we keep it in this chapter of our tutorial, but it can be the other way around as well, i.e. rows for predicted classes and columns for actual classes. The name confusion matrix reflects the fact that it makes it easy for us to see what kind of confusions occur in our classification algorithms. For example the algorithms should have predicted a sample as $c_i$ because the actual class is $c_i$, but the algorithm came out with $c_j$. In this case of mislabelling the element $cm[i, j]$ will be incremented by one, when the confusion matrix is constructed.
We will define methods to calculate the confusion matrix, precision and recall in the following class.
2-class Case
In a 2-class case, i.e. "negative" and "positive", the confusion matrix may look like this:
| predicted | ||||
|---|---|---|---|---|
| actual | negative | positive |
||
| negative | 11 |
0 | ||
| positive |
1 | 12 |
||
The fields of the matrix mean the following:
| predicted | ||||
|---|---|---|---|---|
| actual | negative | positive |
||
| negative | TN True Negative |
FP False Positive |
||
| positive |
FN False negative |
TP True positive |
||
We can define now some important performance measures used in machine learning:
Accuracy:
The accuracy is not always an adequate performance measure. Let us assume we have 1000 samples. 995 of these are negative and 5 are positive cases. Let us further assume we have a classifier, which classifies whatever it will be presented as negative. The accuracy will be a surprising 99.5%, even though the classifier could not recognize any positive samples.