Introduction to Logistic Regression and its Implementation on Breast Cancer Dataset
Despite the name, logistic regression is a classification algorithm, not a regression one. This video explains why, then implements it on the UCI breast cancer dataset.
The name is misleading from the start: logistic regression sounds like a regression technique, but it's actually one of the most fundamental classification algorithms in machine learning. This video works through why that naming exists, how the algorithm actually functions, and applies it to the UCI breast cancer dataset.
A classification algorithm with a regression name
Logistic regression is used when the dependent variable is categorical, most often in binary classification, where the goal is predicting which of two categories a data point belongs to rather than predicting a numerical value. Picture a patient with symptoms that could point to one of several conditions; the model predicts which condition it's most likely to be, a category, not a number. Unlike linear regression, logistic regression predicts the probability of an observation belonging to a particular class.
The sigmoid function and decision boundary
Imagine two classes of points plotted on a graph, colored blue and red. The goal is to draw a decision boundary that separates them. Logistic regression achieves this by fitting a sigmoid function to the data, which maps any input to a value between 0 and 1, representing the probability of belonging to one class. The decision boundary sits at the point where that probability equals 0.5: points above it are classified into one class, points below into the other. Training the model means finding the coefficients that minimize the error between predicted probabilities and actual class labels, typically using an optimization method like gradient descent that iteratively adjusts the coefficients to reduce the loss.
Why not just use linear regression
There are a couple of clear reasons linear regression fails for classification. First, if classes are numerically encoded, like 1 for influenza, 2 for malaria, and 3 for something else, the model implicitly assumes an order between the categories that doesn't actually exist. Second, fitting a straight line to binary classification data produces predictions that fall below 0 or above 1, which makes no sense for a probability. Logistic regression avoids both problems: its sigmoid-based equation always produces an S-shaped curve, so no matter the input value, the output is a sensible probability, and the decision boundary marks the clean cutoff for classification.
Maximum likelihood estimation
To fit a logistic regression model, a method called maximum likelihood is used. The idea is to estimate the coefficients, beta 0 and beta 1, such that plugging them into the model's logistic function produces a value close to 1 for data points whose actual class is 1, and close to 0 for data points whose actual class is 0. This is what the underlying similarity to linear regression comes from, and why "regression" still appears in the name despite it being a classification method.
Strengths, weaknesses, and implementation
Logistic regression's biggest strength is its simplicity and interpretability, which makes it a popular first choice for binary classification. Its weaknesses include assuming a linear relationship between the independent variables and the log odds of the outcome, and it performs poorly with highly imbalanced classes or strongly nonlinear relationships. The implementation uses the UCI breast cancer dataset. A quick check for missing values initially shows none, but the dataset's own documentation notes that it does contain missing values, and closer inspection reveals the bare nuclei column, stored as an object type, contains a "?" character standing in for 16 missing values. These are dropped, though filling them with the mean or median is also an option. After splitting into training and test sets and fitting the model with scikit-learn's LogisticRegression, setting a random state for reproducibility, the model reaches 96% accuracy. A confusion matrix and classification report round out the evaluation, reporting precision, the proportion of predicted malignant cases that are actually malignant, recall, the proportion of actual malignant cases correctly identified, and F1 score, the harmonic mean of the two.
Key takeaways
- Logistic regression is a classification algorithm, used to predict categories, not continuous numerical values.
- It fits a sigmoid function, producing a probability between 0 and 1, with a decision boundary at 0.5.
- Linear regression fails for classification because it can produce probabilities below 0 or above 1, and numeric class encoding implies a false order.
- The model is fit using maximum likelihood estimation to find the coefficients that best separate the classes.
- Missing values can hide behind non-standard characters, like a "?" in an otherwise numerical column, so always inspect column contents directly.
- Evaluation should go beyond accuracy to include precision, recall, and F1 score, especially for imbalanced classes.
Who this is for
This tutorial is well suited to anyone learning classification fundamentals in machine learning who wants both the conceptual reasoning behind logistic regression and a full, working example in scikit-learn.
Full transcript(auto-generated, with timestamps)
[0:06]Welcome everyone today we're going to dwell into the world of classification algorithms and shine a light on one of the most fundamental techniques logistic regression so let's Dive Right In first things first what exactly is logistic regression well despite its name logistic regression is actually a classification Alor not a regression one it's used when the typ when the dependent variable is categorical typically binary classification algorithms are algorithms which let you uh predict a category or a class rather than a numerical value for example if you're going to a doctor with a set of symptoms possibly attributed to one of the three medical conditions AB or C you're
[0:55]Predicting if the patient has one of the diseases a or C and not the numerical values but a category these problems are classification problems so despite having the name Lo regression in uh its name logistic regression is actually a classification problem unlike linear regression it predicts the probability of an observation belonging to a particular class let's break it down F imagine we have two classes represented as blue and red dots on a graph or course goal is to draw a decision boundary that separates these classes logistic regression achieves this by fitting a sigmoid function to the data this function Maps any input value to a value between 0 and
[1:43]One representing the probability of belonging to one of the classes the decision boundary is drawn at the point where probability equals .5 data points above the boundary are classified as one class while those Bel low are classified as the other class now you might be wondering how do we train a logistic regression model well it involves finding the optimal coefficients that minimize the error between the predicted probabilities and the actual class labels this typically is done using optimization algorithms like gradient descent which iteratively just the coefficients to minimize the loss function let's talk about the strengths and weaknesses of logistic regression one of its major strengths is its
[2:39]Simplicity and interpretability it's easy to understand and Implement making it a popular choice for binary classification tasks however logistic regression also has its limitations it assumes linear relationship between independent variables and log odds of the dependent variable additionally logistic regression performs poorly when the classes are highly imbalanced or when there are nonlinear relationships in the data despite these challenges logistic REM remain regression remains a valuable tool in the data scientist's Arsenal offering a simple yet effective approach to Binary classification problems now your question must be why not use linear regression well linear regression is used to predict a quantitative response but there are other reasons too
[3:31]You could fit a linear regression on a Model this in which the Y is either 1 two 3 influenza malaria or than you but the in encoding here impl implies that the machine learning model that the machine learning model that there some sort of an order which is incorrect there is more reason one more reason why uh linear regression fails for uh classification tasks below is a graph where we fit a line on binary classification data we're checking if a patient has influenza or not on some encant symptom data one major thing that is wrong with this graph is the regression L look closely at the
[4:12]Regression implies that the probability is less than zero or more than one which is wrong because probability is between 0o and one therefore we do not use a linear regression uh for classification problems here is the the equation of logistic regression which uses the sigmoid function to fit the model using the above function we used a method called maximum likelihood logistic regression will always produce the s-shaped curve as seen in the graph so irrespective of the value of x we will obtain a sensible prediction for y the decision boundary which is in green marks the cut off point for classification so if any data point is
[4:57]Below it it belongs to class zero and if any data point is above it it belongs to class one since the underlying technique for logistic regression is quite like linear regression that's why we have the regression part in the name what is maximum likelihood to fit the logistic regression model we use a method called maximum likelihood the idea behind maximum likelihood is we try to estimate the values of beta 0 and beta 1 the coefficients such that by plugging these values in the logistic fun function of our model we get a value close to one if the original observed class for that data point is one and a
[5:36]Number close to zero if the original observed class for that data point is zero let us see a practical example of logistic regression we will use the UCI ml repositories breast cancer data set which has the following attributes we can import the data set and clean it let's see if it has any m ing values seeing none we can move uh forward however if you go on the web page for the UC machine learning repository it says that it does have missing values so let's investigate closely as you can see the bare nuclei is an object so let's see its unique values there you go there's a question
[6:19]Mark over here that means it's a missing value and there are 16 missing values so we can just drop these missing values you can however choose to fill them with mean or Medan values too next let's split the data into training and test sets to perform logistic regression we will use we will import logistic regression function from SK learn. linear model it is important to set the random state so that your results can be replicated you fit the model and then you predict it you can also evaluate your logistic regression model using accuracy accuracy is the number of correct predictions upon the total number of correct predictions we see
[7:01]That our accuracy is 96% you can also use confusion Matrix which the psychic Library provides and visualize it using SNS do heat map finally we can also look at the classification report of our model here we get Precision recall and F1 score a higher Precision value indicates that model is a returning one returning more relevant results than irrelevant ones that is precision will tell us what propor portion of patients we diagnosed as having malignant tumor actually having malignant tumor recall tells us What proportion of patients that actually had malignant tumor were diagnosed by us having malignant tumor F1 score is harmonic mean of these two
[7:45]And there you have it an overview of logistic regression I hope this video provided you with a clearer understanding of how this classification works and its applications in machine learning thank you
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53