Exploring Data Missing Values: Identification and Solutions
Using a loan dataset in Google Colab, this walkthrough shows how to detect, count, and visualize missing values in pandas before deciding how to handle them.
Before you can fix missing data, you have to actually see it, not just know in the abstract that some cells are empty, but know exactly which columns, how many rows, and what percentage of your dataset is affected. This walkthrough works through that process step by step in Google Colab, using a loan dataset as the working example.
Loading the data and getting oriented
The process starts with the basic tooling: importing pandas, seaborn, and matplotlib, then loading the dataset with pandas' Excel reader function. The example dataset here is a loan dataset with columns like age, time employed, status, telephone, account, and home expense, mixing categorical and numerical data. Before diagnosing missing values, it helps to get oriented: calling the head function to preview the first several rows, and checking the dataset's shape to see the total number of rows and columns you're working with. This particular dataset runs several hundred rows, so a 10-row preview is just a sanity check, not the full picture.
Detecting missing values
The core tool for spotting missing data is the isnull function, which returns the dataset in the same shape but with every cell replaced by True or False, where True means a null value is present at that position. That's useful for a small preview but unwieldy at full scale, which is why the next step chains a sum function onto it. Calling sum on the output of isnull gives you a per-column count of missing values, so you can see at a glance that, for example, time employed has two nulls, account has one, and home expense has four.
Converting counts to percentages
Raw counts are hard to interpret without context, especially across columns with very different amounts of data. The fix is to convert those counts into percentages, dividing the null count by the dataset's shape and multiplying by 100. Once that conversion is applied, every column's missing-value count reads as a percentage of that column's total, which makes it much easier to judge severity: a column missing 2% of its values is a very different problem than one missing 40%.
Getting the overall picture
Beyond per-column detail, it's worth checking the missing-value rate for the entire dataset at once. Summing the null counts across the whole dataset and dividing by the total number of cells, rows multiplied by columns, gives you a single overall percentage of missing data. In this walkthrough's example, that comes out to a specific overall null percentage across the whole loan dataset. The same approach in reverse, using a notnull function, tells you how much of your data is actually complete.
Visualizing missing values with a heatmap
Numbers are precise, but a heatmap makes patterns of missingness visible at a glance. Plotting a heatmap with rows on the horizontal axis and columns on the vertical axis, the dark portions of the graph indicate non-null values, while the lighter lines mark where nulls occur. That visual pattern can reveal things a table of percentages can't, like whether missing values cluster in specific rows or spread evenly across the dataset, which matters when you're deciding whether to drop rows, drop columns, or impute values.
Key takeaways
- Use pandas' Excel reader to load your dataset, then check head and shape to understand its size and structure before diagnosing problems.
- The isnull function flags every missing cell as True; chaining sum onto it gives you a missing-value count per column.
- Convert raw null counts to percentages by dividing by the dataset's shape, which makes severity easier to compare across columns.
- Divide total null count by total cell count (rows times columns) to get one overall missing-data percentage for the dataset.
- A heatmap of null values, built with the same libraries you already imported, turns a table of numbers into a visual pattern you can scan quickly.
Who this is for
This is a practical starting point for anyone doing data cleaning in pandas, whether you're a beginner working through your first dataset or someone who wants a quick refresher on the standard missing-value diagnostic workflow before moving on to imputation or deletion strategies.
Full transcript(auto-generated, with timestamps)
[0:00]Hi all in today's video we'll do some practical work on data cleaning part so for that you need jupyter notebook or Google collab in this particular video I'm using Google collab a very first step is you need to import some libraries that is Panda cbon met plot LIF then you need to load the data so for that pd. read axel that is the panda function to read your Excel file here I'm using a loan data set for this particular video so these all are the columns um in this particular data set age time status telephone and so on data is present here in a categorial and numerical
[0:56]Form and also some n values are present in this data set as you see on screen the data set. head function head 10 this means I'm returning only 10 rows here but the data set contain 400 or 500 rows that we'll see later so if you check the number of rows and Columns of so you can simply do this data set dot shape do shape will tell you the number of rows and column in this particular data set now if you want to check the number of null values then you simply do this is null function here so in this particular data set you see the values in the form of
[1:55]False and true so this true means that the null value is present here or if you want to calculate the number of null values then you need to do or you need to call this sum function here here I'm using this sum function so you can check the number of null value toward the particular column for example time employed have two null values and account have one home expense have four so with the help of this sum function you'll check the null value for the particular column but it's very difficult to calculate uh the values further with simple or without percentage so
[3:06]We'll first convert those null value into percentage here so for that we'll simply do data set dot shape into 100 so with the help of this function we'll simply convert our n values into decimal values or into percentage now you can see all the values are in percentage form now if you want to check the overall data n values so for that is simply do sum function do sum function so total number of value present in data set null values present in this particular data set is 10 here if you want to check the overall n values toward rows and column in your data set then you simply do this
[4:09]Function sum do sum function do sum and you can divide this with the data set do shape zero into Data set. shap 1 that is your number of rows and column into 100 and66 that that's the number of percentage for your overall data set null value if you want to check not null values then you simply put not null function do su function or those are the values which this is the number like these values are not null in your data set if you want to check the graphical representation for the same so for that uh we already imported that same on Library this the heat
[5:05]map so in a horizontal axis you see the number of uh rows and the vertically you'll check the number of columns here and the black portion in this grab indicates that these all are the non null values and the light color line indicates that this is the null value in your data set so with the help of graph you can easily identify the null values thank you
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53