Introduction to KNN and its Implementation on Iris Dataset by Kedar Ghule
KNN classifies a point by majority vote among its closest neighbors. This video explains how it works, how to choose K, and implements it on the Iris dataset.
Some machine learning algorithms work by building a mathematical model of the data. K-Nearest Neighbors takes a much more direct approach: it classifies a new point simply by looking at what's closest to it. This video introduces the algorithm and walks through a full implementation on the Iris dataset.
What KNN is
KNN, short for K-Nearest Neighbors, is a widely used supervised machine learning algorithm for classification. The core idea is straightforward: it determines the class of a data point by looking at how many points of a particular class are closest to it, then makes an educated guess based on that. Since it learns from labeled training data, it belongs to the family of supervised learning algorithms.
How the algorithm works
Given a new data point, KNN calculates the distance from that point to all its neighbors using a distance metric, typically Euclidean distance, though Manhattan distance is another option. It then selects the K nearest neighbors and, for classification tasks, takes a majority vote among them to decide the new point's class. If most of a point's nearest neighbors belong to one class, the new point is classified into that class. The choice of K matters a great deal: a small K value produces a more flexible decision boundary, which risks overfitting, while a larger K value simplifies the model.
Strengths and weaknesses
KNN's biggest strength is its simplicity: it's easy to understand and implement, and as a non-parametric algorithm, it doesn't assume anything about the underlying distribution of the data. Its weaknesses are just as notable. It can be computationally expensive on large datasets, since it requires calculating distances between the new point and every existing point. It's also known to suffer from the curse of dimensionality, and it's sensitive to both the choice of distance metric and the value of K, where a poor choice can lead to inaccurate predictions. Despite these limitations, KNN remains a valuable tool, particularly for small to medium-sized datasets where interpretability matters.
Implementing KNN on the Iris dataset
The implementation starts by loading the Iris dataset from scikit-learn's datasets module and reviewing its characteristics with a describe method. Checking for missing values and non-numerical types shows the Iris dataset is clean on both counts. A quick group-by on the class column confirms the dataset is well balanced, with 50 data points in each class. Scikit-learn's KNeighborsClassifier is imported from the neighbors module, and the first implementation sets the K value to 1 by passing n_neighbors=1. By default, KNeighborsClassifier uses Minkowski distance with a p value of 2, which is mathematically equivalent to Euclidean distance; setting p to 1 instead gives Manhattan distance. After fitting the model, its accuracy is checked with the .score() method, alongside a confusion matrix and classification report.
Finding a better K with the elbow method
Choosing K by guesswork isn't reliable, so the video uses the elbow method: testing K values from 1 to 10 and tracking the error rate for each. The error rate drops at K equal to 9, so the model is refit with that value. The result is a meaningful accuracy improvement, from roughly 0.95 with K equal to 1 up to roughly 0.96 with K equal to 9. The important caveat is that the optimal K is the one where the error rate is low, not zero; a K value that drives error to zero is usually a sign of overfitting rather than genuinely better performance.
Key takeaways
- KNN classifies a new point by majority vote among its K nearest neighbors, based on a distance metric like Euclidean or Manhattan.
- The K value controls the bias-variance tradeoff: small K risks overfitting, large K risks oversimplifying.
- KNN is computationally expensive on large datasets since it computes distances to every existing point.
- Scikit-learn's KNeighborsClassifier defaults to Minkowski distance with p=2, equivalent to Euclidean distance.
- The elbow method, testing a range of K values and tracking error rate, helps find a K that avoids both underfitting and overfitting.
Who this is for
This video is a solid starting point for anyone learning classification algorithms in machine learning who wants a clear explanation of KNN alongside a hands-on implementation using scikit-learn.
Full transcript(auto-generated, with timestamps)
[0:02]Hello everyone and welcome to today's video in this segment we'll be diving into the fascinating world of machine learning and exploring one of its fundamental algorithms K nearest neighbors commonly known as KNN so let's get started firstly what is KNN KNN is a very popular and widely used supervised machine learning classification algorithms the idea that came n n is that it determines the class of a data point by seeing how many points of particular class are closer to it then Kon essentially makes an educated guess to classify the data point it it belongs to the family of supervised learning algorithms meaning it learns from labeled training data
[0:54]Imagine we have a set of data points each belonging to a particular class or having a specific value Cann works by finding the K nearest neighbor nearest data points to the one that we want to classify or predict based on a distance metric typically ukian distance let's break down the process say we have a new data point represented by this blue dot on the graph we want to classify this point based on its proximity to its neighbors to do this we will calculate the distance between all its nearest neighbors using the chosen distance metric ukian or Manhattan then we select the K nearest neighbors here's what the where the
[1:44]Magic happens for classification tasks the algorithm takes the majority vote among the K neighbors to determine the class of the new data point in this case the red class is cat so you this point would be classified as a cat point however the blue point over here is among a lot of dog points the Green Dog points so it would be classified as a dog in other words if most of the neighbors belong to a certain class are poin is classified into that class one important aspect to consider is how do you find what is the how you make a choice of the K value the value
[2:32]Of K significantly impacts the performance of the algorithm a small K value results in a more flexible decision boundary which may lead to overfitting while a larger K value May simplify the model let's talk about the strengths and weaknesses of KNN one of its major strength is its Simplicity and ease of implementation it's a non-parametric algorithm meaning it does not make any assumptions of about the underlying data distribution however KNN also has its limitations it can be computationally expensive especially with large data sets as it requires calculating distances between the new data point and all existing data points it's also an algorithm which is notoriously known to
[3:30]From the cures of dimensionality additionally K is sensitive to the choice of the distance metric and the value of K improper selection can lead to inaccurate predictions despite these drawbacks K&N remains a valuable tool in the machine learning toolbox particularly for small to mediumsized data sets and when interpretability is crucial let us see how we can Implement KNN on the iris data set to implement KNN uh first we need to load the data set which is the iris data set from skarn data sets you can use the decr method uh to see the data set characteristics and information about this data set it like any machine learning
[4:25]Algorithm it is crucial that we clean our data first or check for missing values and non-numerical types this data set does not have any missing values or non-numerical data types so we are good we should also see if our data set is a balanced data set or not you can do this by doing DF dog Group by class. size since each class has 50 data points we can conclude that this is a very well balanced data set now let's split the data set into training and testing to Form K nearest uh neighbors we will import the K neighbors classifier class from the neighbors model of the psychic learn library for
[5:07]This implementation we will assume our K value as one we will create an instance of the K near neighbors classifier and then pass n neighbors parameter as one the N neighbors parameter indicates the K value then we will fit the model and predict it now it should be noted by default that K nearest neighbors uses minowsky distance with the P value of two as the default metric when minowsky distance has a P value of two it's basically ukian distance however you can change the value of p uh to any other value or also specify it as one which would make it Manhattan we can evaluate our model
[5:52]Using the DOT score method which will tell you the accuracy of our ml model finally we can also look look at the uh confusion Matrix as well as the classification report a point to be known over here is when choosing a k value you can also use the elbow method choosing an optimal K value can be an issue as it affects what class data point is assigned to now the optimal K value would be the one in which the error rate of your model is low and by low I do not mean zero otherwise we might have a problem of overfitting so over here let's take a k
[6:30]Value from the range 1 to 10 and see how uh we see as an our error value so we see that the error value decreased at nine so we can take our K value as nine and uh fit the KNN classifier again and see our results as you can see uh we will get a better uh result uh in terms of our accuracy as9 6 as opposed to .95 over here so that uh there you have it folks an overview of the K nearest neighbors algorithm I hope this video provided you with clear understanding of how K neighbors works and its applications in machine learning
[7:22]Stay tuned for more exciting content until next time happy learning
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53