Mastering Missing Values Handling with Scikit-Learn
This video shows how to detect missing values in a loan dataset and fill them in using scikit-learn's SimpleImputer, with a full pandas-based walkthrough.
Most real-world datasets arrive with gaps in them, and how you handle those gaps can make or break a model downstream. This video walks through using scikit-learn's imputation tools to detect and fill missing numerical values in a loan dataset, using pandas alongside scikit-learn's SimpleImputer class.
Loading and inspecting the data
The process starts with importing the necessary libraries, including pandas and matplotlib, then loading the dataset with pandas' Excel-reading function, working with a loan dataset for this example. A quick head check on the data shows the first several rows, revealing columns like age, time, status, and telephone, a mix of numerical and categorical data, along with both null and non-null values scattered throughout.
Finding the missing values
To see how many missing values exist in each column, the video uses a chained isnull().sum() call, which returns the total count of null values per column. Since this particular walkthrough focuses only on numerical columns, the next step is figuring out which columns actually contain numerical data. The dataset's info() function reveals each column's data type, for example showing that a column like sex holds object data (categorical, like male or female) while a column like age holds float64 data, meaning it is numerical.
Isolating the numerical columns
Rather than manually picking out numerical columns, the video uses select_dtypes with include=['float64']).columns to programmatically retrieve just the columns holding numerical data. Working only with numerical columns makes sense here because it is easier to apply mathematical operations, like computing a mean, to them than to categorical data.
Imputing with SimpleImputer
The imputation itself comes from scikit-learn's impute module, specifically the SimpleImputer class. This class supports several strategies, such as mean, median, or mode, depending on what makes sense for the data. In this walkthrough, a variable is set up with SimpleImputer configured to use the mean strategy, and fit_transform is called on the numerical values to produce the new, imputed data.
Verifying the fix
After imputing, checking isnull().sum() again on the new dataset confirms that every numerical column that previously had missing values now shows zero nulls, meaning each missing value has been replaced with the column's mean. As a specific example, the video points to a column like home expense, which after imputation shows a value like 17639 in place of what used to be a missing entry. To confirm that number really is the column's mean, calling the mean() function directly on that column returns the same 17639, verifying the imputation worked as expected.
Key takeaways
- scikit-learn's SimpleImputer class handles missing value imputation with configurable strategies like mean, median, or mode.
- select_dtypes lets you isolate numerical columns programmatically before applying numerical imputation methods.
- isnull().sum() is the standard way to check missing value counts before and after cleaning.
- The mean strategy replaces each missing numerical value with that column's average, verified directly against the column's mean() output.
- This workflow is a practical foundation for cleaning any dataset with mixed categorical and numerical missing data.
Who this is for
This video is aimed at anyone starting out with data cleaning in Python and wanting a concrete, reproducible pattern for handling missing numerical data. It is part of a Humanitarians AI series on missing value strategies that also covers categorical imputation and simply dropping missing rows or columns.
Full transcript(auto-generated, with timestamps)
[0:00]We can clean our data with the help of psyit Library so very first step is you need to import some libraries like penda c m plot live afterward you need to load the data for that I'm using this speed. read AEL function here I'm passing my file name I'm using a loone data set for this particular video so now if you want to check some particular number of rows so you simply do this doad function I'll pass the number here so in this data set these all are the column like age time status telephone and so on and you can see like there are uh numerical as well as
[0:48]Categorial data present in this data set also null and nonnull values are also there so if you want to check how many number of null values are present in the particular data set so you will simply do this is null function. sum function so you'll get U the sum of all the null values present for each column here in this particular data set so for this particular video I just only consider the numerical data column which have numerical data so if you want want to check like which column have the numerical data simply do data set do info function so you'll see like what column have what
[1:44]Data so like sex have object data like male or female age have float 64 that is your numerical data so I'll only consider those column which have the numerical data because it's easy to perform operation on the numerical data instead of categorial data so for that I just only need the column which have number values or the numerical data so for that I simply do this data set. select data type include float 64. columns so you'll get those columns here now I'm using the is psyit Library so for that I'm using from pyit learn. impute import simple imputer so what is imput imput is a
[2:42]Module here and I am importing this class simple imputer and this simple imputer class have different different function like mean medium mode whatever you want to do with your value so you can simply do with this simple imputer class so here I am considering the XI variable and I am uh using simple inputter function and I'm passing strategy as a argument here or under strategy I'm taking this mean and in AR variable I am simply fit transform those numerical values so now if you want to check the new missing values so new data set so you'll simply do so I'll simply consider this new data
[3:53]Set or in this data set I'm having all the values missing values so if you want to check those missing values so you'll simply do new data set is null function do some function you'll see like every numerical column which we have considered before now replaced with some value so you'll see now the null value is zero row here for all the column before in our existing data set the value these column have null values so now the null values is replaced with the mean value this is our new data set now you'll see home expense have 17639 that means existing value is replaced with
[4:57]The mean value if you want to check like is this the mean value or not so for that you'll simply do data set the particular column name do mean function so you'll see here the value is 17639 so this value is replaced here so by considering all these steps you can simply clean your data with this these Library please thank you
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53