Handling Missing Values in Machine Learning
Missing data shows up in almost every real dataset. This video breaks down why it happens and walks through the main techniques for handling it without introducing data leakage.
Real datasets are rarely complete. Values go missing because of data corruption, human error during entry, or simply because the information was never collected in the first place. Ignoring the gaps is always an option, but handling them properly is a crucial part of data preprocessing, and this video walks through why data goes missing and how to deal with it using the horse colic dataset from the UCI repository as an example.
Why data goes missing
Not all missing data is missing for the same reason, and understanding why matters for choosing the right fix. Data can be missing completely at random, missing at random, or missing not at random. The last category is often tied to bias: someone might be uncomfortable sharing their salary, for instance, which means that particular gap isn't random at all but reflects something about the person providing the data.
Checking how much data is missing
Before deciding how to handle gaps, it helps to see how many there are. Running .isna().sum() on a dataset shows the count of missing values per column. In the horse colic example, the surgery column has just one missing value, age has none, abdominal distention has 56, and nasogastric reflux pH has 247. That range matters: dropping a column with one missing value costs almost nothing, but dropping one with 247 missing values would throw away a large chunk of the dataset.
Splitting data before filling it in
One detail that's easy to overlook is sequencing: missing values should be handled only after splitting the data into training and test sets, using statistics from the training set alone. Filling gaps with a mean calculated across the full dataset, test set included, causes data leakage, since information from the test set ends up influencing values in the training set.
Deletion and mean or median imputation
The simplest approach is to drop any rows or columns that contain missing values. This works fine when there are only a few gaps relative to the size of the dataset, but it risks significant information loss otherwise. Mean or median imputation fills missing values with the mean or median of that column instead. It's easy to implement with tools like scikit-learn's SimpleImputer, but it can still cause data leakage if not handled carefully, doesn't account for the covariance between features, and is a poor fit for time series data, where it effectively introduces look-ahead bias.
KNN and regression imputation
A more sophisticated option is KNN imputation, which fills a missing value using the average of its nearest neighbors, implemented through scikit-learn's KNNImputer. Multiple research papers support this as an effective approach, though it's computationally expensive and sensitive to outliers. Regression imputation goes further still: after checking for correlation between the missing variable and other variables, scikit-learn's IterativeImputer with a linear regression estimator can predict missing values directly, effectively turning the missing-data problem into a regression task.
Categorical data and time series
For categorical columns, a common approach is filling missing values with the most frequently occurring label, though this again carries some risk of data leakage and isn't always optimal. Time series data calls for its own techniques: forward fill replaces a missing value with the last observed value before it, while backward fill replaces it with the next observed value after it.
Key takeaways
- Understanding why data is missing, whether at random, completely at random, or not at random, informs how it should be handled.
- Always split data into training and test sets before imputing, and fit any imputation strategy on the training data only.
- Dropping rows or columns is only advisable when the amount of missing data is small.
- Mean and median imputation is simple but can leak data and ignores relationships between features.
- KNN and regression imputation offer more accurate results at the cost of more computation.
- Time series data should use forward fill or backward fill rather than mean-based imputation.
Who this is for
This tutorial is aimed at anyone building machine learning models in Python who needs a practical, technique-by-technique guide to cleaning up incomplete data before training.
Full transcript(auto-generated, with timestamps)
[0:02]Data can be missing for a variety of reasons missing values in machine learning refer to the absence of data in a particular or a particular feature and or attribute in a data set these missing values can occur due to various reasons such as data corruption human error during data entry or simply because the information was not collected or available at the time of data collection you can ignore them but handling them is a crucial step of our data pre-processing it is also important why the data was missing in the first place data can be Miss Missing for three reasons missing at random missing completely at random and missing not at
[0:42]Random usually when you have data missing not at random it's because of certain biases that the user may have about sharing their data for example someone might not be very comfortable sharing their salary so that means that is why the data is not missing at random so coming uh to uh handling data there are different ways you can handle missing values uh in this example uh we will be taking uh the data set uh of horse colic data within the UCI repository and we will be doing some seeing some ways to handle missing values a way you can check missing values in a data is is doing do is na do
[1:32]Sum this will tell you that there are what what amount of missing values are there so for example surgery has one missing value age has no missing values but something like um abdominal detention has 56 missing values na naso gastral reflux pH has 247 missing values so if you delete something like that you would basically lose a lot of information however deleting this one would probably not have a lot of you won't be deleting a lot of information so how do you handle um missing values before you handle missing values you need to make sure that you're splitting your data into training and test sets and then only then use the training
[2:19]Data set for filling your missing values because for instance as you would discuss later on if you if you want to fill your data with any mean value of that particular column you don't want the test uh data sets values to be also counted during the mean that would basically lead to data leakage so the different methods to handle uh missing values is first one removing the missing column all together this method the idea is basically drop any and all rows that have missing values or missing columns this method is not preferable because because as I said it can result in significant information loss it is
[3:03]Only good if you have one or two or maybe very few missing values compared to your data the second method is mean median imputation mean median imputation the idea is to fill missing values with mean or median of that column this method can cause a data leakage and does not but does not also does not factor in the covariance between Fe features this method is also not useful for time series data because in Time series data if you use a method like this you're basically introducing look ahead and contaminating your uh assumptions basically data leakage you can use the simple imputer to impute and use the impute strategy
[3:49]Mean or median depending on what you want KNN imputer is another good way to handle your missing values uh you can uh replace your missing values um with average of the mean of the nearest neighbors to it and multiple research papers have shown this is a good way to perform or handle missing values you can use psych's K andn imputer for this the only con for this is that you know it is computationally expensive and it is sensitive to outliers lastly we can also look at regression imputation before this it is important that you check if there is a correlation between missing values and other variables we
[4:37]Can often get better guesses by regressing the missing variable on other variables so in psychic learn uh you can use iterative imputer and specify the estimator as linear regression to do regression imputation make sure you're using only those variables which have a correlation with the missing value or if if not you can also use the whole data set but try to do your linear regression properly so you are basically turning a missing data problem into a regression task finally so far I only spoke about Cate uh numerical variables because this data set was predominantly numerical or only numerical if you have categorical values you should can you fill the missing
[5:24]Value with the most frequently occurring label but again this may cause data leakage and it's not a very optimal me to do in case of Time series you can do forward fill or backward fill forward fill is when you replace the null values with the last observed value so basically if today's value is missing you replace it with yesterday's value and backward fill is uh basically uh you replace the Nan values with the next observed value so for example if you have a data set of this year and today's value is missing but you have tomorrow's value so you then you basically fill in today's value with tomorrow's value I've
[6:04]Just shown representative syntax over here on how to implement these imputations thank you so much and I hope you learned through this little tutorial
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53