Understanding Model Performance: Beyond Accuracy to F1 Score
A walkthrough of why accuracy alone can mislead on imbalanced datasets, and how precision, recall, and F1 score give a truer picture of model performance.
A classification model that reports 90% accuracy sounds better than one that reports 84%. But what if the higher number is hiding a model that has essentially memorized one answer and repeats it every time? This video works through exactly that trap, using a simple binary classifier built to tell dog images from cat images, and shows why accuracy by itself can point you toward the wrong model.
What accuracy actually measures
Classification is the task of sorting input data into predefined categories, whether that means labeling an image as a dog or a cat, running sentiment analysis on text, flagging tumor cells in medical imaging, or identifying sounds in an environment. Accuracy is the simplest way to score this kind of model: it is just the number of correct predictions divided by the total number of predictions. If a model gets 90 out of 100 images right, its accuracy is 90%. The problem is that this single number says nothing about what the underlying dataset looked like or how the errors were distributed.
Where accuracy breaks down
The video sets up two models, XYZ and ABC, trained on a dataset of 90 dog images and 10 cat images. XYZ reaches 90% accuracy, but it does so by labeling every single image as a dog, no matter what is actually in the picture. Because 90% of the dataset is dogs, that lazy strategy alone produces a 90% accuracy score. ABC, by contrast, actually attempts to distinguish dogs from cats and lands at 84% accuracy, correctly identifying both dog and cat images most of the time, with 16 total misclassifications. Despite the lower headline number, ABC is doing real work that XYZ is not. This kind of skewed data, where one class vastly outnumbers another, is called an imbalanced dataset, and it shows up in serious contexts too, such as detecting cancerous cells, where non-cancerous samples vastly outnumber cancerous ones in most datasets.
Reading a confusion matrix
To see the difference between the two models clearly, the video introduces the confusion matrix, which breaks predictions into four buckets: true positive (a dog correctly predicted as a dog), false negative (a dog predicted as a cat), false positive (a cat predicted as a dog), and true negative (a cat correctly predicted as a cat). For ABC, the confusion matrix shows 77 dog images and 7 cat images correctly identified, with a handful of mistakes in each direction. For XYZ, the matrix tells a starker story: 90 dog images correctly labeled, but zero cat images correctly labeled, because the model never predicts "cat" at all. The confusion matrix makes visible what the single accuracy number conceals.
Precision, recall, and F1 score
From the confusion matrix, the video moves to precision and recall, two metrics that evaluate performance per label instead of averaging everything into one score. Precision asks: of all the images the model predicted as dog, how many were actually dogs? Recall asks: of all the images that were actually dogs, how many did the model correctly identify? Both metrics range from zero to one, with one representing a perfect result. When applied to the cat label, XYZ's precision and recall both come out to zero, since it never predicts cat correctly, while ABC shows real, if imperfect, precision and recall for both labels.
F1 score combines precision and recall into a single number using their harmonic mean, giving a more balanced sense of how a model performs on a given label. In the video's example, ABC's F1 score comes out to roughly 0.84 and XYZ's to about 0.9, which on the surface still favors XYZ. But because the dataset is imbalanced toward dog images, the weighted F1 score, which accounts for how much each class contributes to the overall dataset, is the more honest comparison. There, ABC scores around 0.861 against XYZ's roughly 0.852, confirming that ABC is the stronger model once the imbalance is taken into account.
Key takeaways
- Accuracy alone can be misleading, especially on imbalanced datasets where one class dominates.
- A model can reach high accuracy simply by always predicting the majority class, without learning anything useful.
- Confusion matrices break predictions into true positives, false positives, true negatives, and false negatives for a clearer view of performance.
- Precision and recall evaluate a model's performance per label rather than as one blended number.
- F1 score is the harmonic mean of precision and recall, and the weighted F1 score adjusts for class imbalance.
- Choosing the right metric matters most in high-stakes applications, such as medical diagnosis, where missing a rare positive case has serious consequences.
Try it yourself
The code used in this walkthrough, including the confusion matrix visualizations and metric calculations, is available on GitHub for anyone who wants to reproduce the comparison between the two models. This kind of evaluation work sits at the core of Humanitarians AI's applied data science projects, and it is a useful exercise for anyone training classifiers on real-world data where classes are rarely perfectly balanced.
Full transcript(auto-generated, with timestamps)
[0:00]Hello everyone welcome to today's video where we'll be learning more about metrics used to evaluate classification models first let's start with understanding what classification is classification is a machine learning task where the goal is to categorize input data into predefined classes or categories based on the features so some of the examples of classification or classifying uh an image like if the image has a dog or a cat or even like there are audio classification where where you can convert spoken words into text and there is text classification where you can do sentiment analysis based on uh a Twitter data if it is positive sentiment or negative sentiment
[0:46]And in medical imaging where you can use classification to understand if there are tumor cells or not and even the environment sound classification if there are multiple sounds coming uh in the environment you can classify it as bird song or a dog barking Etc so normally we uh relate classification models with a metrics called accuracy if suppose we build a model we say uh the accuracy of the classification model is 97% or 95% so what does accuracy mean so accuracy is nothing but the number of uh items uh classification model classified currect corly versus the number of items it tried classification on so it gives a
[1:35]Percentage of how much correctly the classification models predicted the output let's talk about a scenario where we are designing a binary classifier to predict if an image uh is of a dog or a cat and say we have two models XY Z and ABC so what does it mean when we say a model has 90% accuracy so so when we say we have 90 % accuracy that means on a given data set 90% of the time a model predicted the output correctly whether if it a dog or a cat and if Suppose there are 100 images we have so 90% accuracy means the model predicted 90 of the images
[2:19]Correctly and what can we infer about the data set when we say we have 90% accuracy so just knowing the accuracy we we we won't be able to talk a lot about what data was used the number of dog or cat images that were used and let's go to the next question like imagine XY Z has accuracy 90% and a model ABC has 84% accuracy can we say just because XYZ has 90% accuracy which is more than ABC that XYZ is a better model than ABC is there a scenario where the accuracy metric might fail so let's take an example where we have 90 dog images
[3:02]And 10 cat images and based on the earlier example XYZ had 90% accuracy so if we classify anything that comes into XY Z as a dog image uh like no matter what uh image comes in we still get the 90% accuracy for this data set because 90% of the images are dog images so XY Z is hardcoded and let's take the example of ABC ABC after training predicts for some dog images as dog and for some cat images as correctly cat and the only problem is that it is not able to predict 16 of the images correctly hence it has 84% accuracy now with this
[3:45]Information we kind of understand that 84% is kind of much better than whatever XYZ was doing with 90% accuracy and such data set where you have a uh imbalanced ratio of cat and dog images or two different labels is called an imbalanced data set let's take a real life scenario where such things might happen where we have imbalanced data set so imagine we want to classify cancerous cell as cancerous or non-cancerous so the data set that we might have will have largely uh images of non-cancerous cell and when we are training the model if we if we don't accommodate for the imbalanced data set we might cause more harm than
[4:36]Having a useful implementation of a ml model now that we have talked about how accuracy alone might fail let's talk about what other things can we do during training so to begin with we can make sure that the data set is balanced so there are different ways in which we can make data set balance we can drop lot of data from the larer class or we can do data augumentation like if it is images we can do rotation flipping of images for the smaller class to increase those data set and even during training instead of using normal loss function we should use a weighted loss so that the
[5:16]Model training can incorporate the data imbalance and let's now see what metrics can be used if the data is imbalanced now let's take an example of a scenario where the accuracy metric might fail so in the data set we have image name actual prediction that is the ground Pro and the predictions from XY Z Model and ABC model let's visualize how the data looks like so this is the code to see how many uh dog or how many cats have been predicted by different models so if you see the first graph that is the ground truth or the actual prediction so we have 90 dog images and
[6:01]10 cat images and the XYZ model as described earlier outputs dog no matter what the input is so all the 100 would be classified as dog and in case of ABC model we have 80 dog predictions and 20 cat predictions so now let's see how many of the classifications were correct from both XY Z and ABC so if you see the graph uh there are 77 dog images that were cor correctly predicted by ABC model and seven images of c and when we see XY Z it predicted all the 90 dog images as correctly and so better way to visualize such uh outputs are using a confusion metrix
[6:52]Let's see what a confusion Matrix is now so the first is true positive imagine when we are building a binary classification dog is a true positive and not dog is the other case so in the predicted value if it is a dog and we predicted it as a dog that's when we say it is a two positive and imagine when it is a dog but the model predicted it as a cat so that's called a false negative and in similar way imagine if it is not a dog and the model predicted it as a dog so that's a false positive and for a case where it is not a dog and it
[7:37]Predicted as not at all that's called a true negative so this will be easy for us to visualize how the model is performing let's go ahead and see how the confusion metrix looks like for both ABC model and XYZ model so if you look here the confusion metrics for ABC model we have 77 dog images that were correctly predicted and seven cat images that were correctly predicted and for the other cells for a cat image it predicted as dog there are three inst such instances and in the same way for 13 of the dog images it predicted as C so that is why we have 13 on the top right side
[8:28]Let's see how the confusion metrix looks like for XY Z Model It could only predict so if you see here it could only predict 90 of the dog images correctly it didn't predict anything as C so for the classification of C if you see here as we know that the XYZ model always outputs the label as dog irrespective of whatever the input is so it could predict 90 of the dog images correctly but it labeled 10 cat images as wrong and for the cat section it is 0 0 from this we get an idea that the model is not working well for a cat label now let's look at how we can use
[9:22]This values for our metrics precision and recall are two new metrics that can be used instead of accuracy this will give us a better idea about how the model is performing so let's take Precision Precision is nothing but out of all the uh images where the model predicted image to be dog how many were actually dog that is true positive by true positive plus false positives and recall tells you how many of the relevant items per retrieve for example out of all the dog images that a model got how many were actually predicted as dogs correctly so these are precision and recall this will give us a better
[10:10]Understanding of how the classification works for each individual label instead of taking it overall so this is just the calculation of of how we can do precision and recall so repeating my uh of all the positive predictions how many were really positive that is precis and of all the real positive cases like of all the real dog images how many were predicted as a dog that is three and along with precision and recall F1 score also gives you an idea how how good a model is doing and F1 score is nothing but a Comin harmonic mean of precision and Recall now let us see how how uh the Precision and recall
[11:10]Are for the models ABC and XY so these are the Precision and recall values for model ABC so if if you see here it it it gives us an idea of how how good the Precision recall is for dog and for a cat and a good value for precision is one the Precision and recall ranges between zero and one and one means it did a perfect job and zero means it didn't work correctly so for xy3 model so if you see here the Precision and recall for cat is zero but when you compare with the Precision and recall of do for ABC model and XYZ model XYZ is doing a pretty good
[11:57]Job so you Precision recall we can understand how the model is working on individual label and again so F1 score if you see here F1 score of ABC model is 8399 and for XY Z Model it's 0.9 so since the data set is imbalanced towards a dog images we should use the weighted F1 score and when we see the weighted F1 score ABC has 861 and we Defence score of XY Z Model is 852
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53