Effective Strategies for Handling Missing Values: Imputing Categorical Data
This video compares pandas techniques for filling missing categorical data, including fillna, backward filling, and forward filling on a loan dataset.
Numerical missing values can be filled with a mean or median, but categorical data needs a different approach entirely. This video works through several techniques for imputing missing categorical values in a loan dataset using pandas, comparing a simple fill against backward filling and forward filling.
Loading and sizing up the data
The workflow starts by importing pandas and matplotlib, then loading the loan dataset with pandas' Excel-reading function. A head() call with a specified number of rows returns just that many rows for a quick look, and a shape check reveals the dataset contains 429 rows and 14 columns. From there, isnull().sum() shows how many null values exist in each column, revealing that fields like home expense, account time, and employ all carry some missing values.
A first pass: filling with a fixed value
The simplest fix is the fillna() function, which fills every missing value in a column with whatever value you specify. Applied to a column like home expense, any originally empty (NaN) entry gets replaced with that fixed value. The video is upfront that this approach, filling randomly with one fixed value, is not necessarily a good way to handle missing data, since it does not reflect anything meaningful about the surrounding records.
Better options: backward and forward filling
A more thoughtful approach uses backward filling, forward filling, or mode filling instead of an arbitrary constant. Before applying these, the info() function is used to review the dataset's data types, row counts, and memory usage, which helps confirm what still needs cleaning.
Backward filling works by taking a missing value and replacing it with the value from the next row. In the video's example, the home expense column has a missing value in its very first row; after backward filling, that gap is replaced with the value from the second row, in this case 140, since the fill pulls the next available value backward into the gap. Forward filling works in the opposite direction, carrying a value forward from a previous row into the following missing entry.
Applying the fill methods
Both techniques are applied using the same fillna() function, but with the method argument set to specify direction, backward filling or forward filling, rather than passing a single static value. This lets the missing data get replaced with contextually relevant values drawn from adjacent rows in the dataset, rather than an arbitrary placeholder.
Wrapping up the categorical cleaning process
By working through fillna with a fixed value, backward filling, and forward filling side by side, the video demonstrates that categorical data cleaning has more nuance than numerical imputation. Choosing between these methods depends on how the dataset is ordered and whether nearby rows are likely to carry genuinely similar values.
Key takeaways
- fillna() with a single fixed value is the simplest way to fill missing categorical data, but it is not usually the most meaningful choice.
- Backward filling replaces a missing value with the next available value in the column; forward filling carries the previous value forward.
- isnull().sum() and shape are quick ways to size up how much missing data a dataset contains before choosing a strategy.
- info() confirms column data types and row counts, which helps decide which imputation approach fits a given column.
- The right imputation method depends on context: a constant, backward fill, forward fill, or mode fill can each be appropriate depending on the data.
Who this is for
This video is useful for learners who have already handled numerical missing data and want to extend that skill to categorical columns. It is part of a Humanitarians AI series on missing value strategies that also covers scikit-learn's SimpleImputer and simply dropping missing rows or columns.
Full transcript(auto-generated, with timestamps)
[0:00]How we can clean a data in our data set if the data is in a categorial form so for that first you need to import the libraries that is penda saon met plot La now I'm loading the data set with the help of this b. read Axel here I'm using a loan data set so by using this doad function you can return only those number of rows which you have give this input inside this function that is if I want to return the 10 number of rows then I'm putting here 10 and now I'm returning the 10 number of rugs now if you want to see like how
[0:55]Many rows and columns are there in this data set so for that you simply use a DOT shape so 429 columns and 14 429 rows and 14 columns are there in this data set and if you want to check like how many null values are there present in this data set then you simply use isnull function and now if you want to calculate for particular column like how many null values are there here then you simply use this isnull function. sum function so as you see that home expense account time employ these have some null values already calculator with this function here so for the categorial for column
[1:53]Values we can or we can simply use if there are some missing values sorry if there are some missing value on the data set or you want to randomly fill some value then you simply use this function fill Na and whatever the value you want to fill you just enter here inside this function for example like in a home expense there is NN value nothing uh was there in the data set and now you see that it already replaced that n with them so fil any is basically to fill the data randomly if you want to you know randomly fill your value in the column although that is not a very good
[2:47]Way to you know fill your data so what is the right way so far if you want to fill data correct directly with some method so for that there are two method that is backward feeling and forward feeling and also mod filling now we will see like how it works backward and forward fing so as you see our in data set. info function will give you the data type how many rows null values everything you can basically check the information of that data set data type memory usage all these things how many objects are there so yeah you can simply check with this function now again doad function
[3:50]And you see the values here now again the values andn here in a home expense that is your null value you can see that so now how we can you know put and what are the Syntax for that particular not syntax but function to you know fill the data with the help of this filling method so for that you just only need to write this data set do fill Na and you simply right method is Bill Bill means your backward filling so as you see that in a home expense here the value is in a very first row NN if you see here the value is NN when
[4:38]You put backward filling then the value changed to 140 that is the value for the second row so it automatically takes that value and fill that replace that value with that value you so that is your basically backward filling method and in a forward filling method now you see that the value all now the value is replaced so that is your forward feeling so these two type of the method we use to you know of fill or inputting you know our categorial data with the help of this we basically clean our data if the data is present in a category form yeah that's it thank you
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53