Polynomial Regression and its Implementation on Boston Housing Dataset
This tutorial explains polynomial regression as an extension of linear regression, then implements it on the Boston Housing dataset, comparing R2 and RMSE scores.
A straight regression line only works when the underlying relationship in the data is actually straight, and real-world data rarely cooperates. This tutorial walks through polynomial regression, the technique for modeling curved relationships between variables, explains how it extends ordinary linear regression, and then implements it step by step on the Boston Housing dataset to compare results directly against a plain linear model.
What polynomial regression is
Polynomial regression models the relationship between an independent variable X and a dependent variable Y as an Nth-degree polynomial rather than a straight line. Linear regression assumes linearity, which is almost always an approximation and sometimes a poor one. When data follows a curved pattern rather than a straight one, a linear regression line will visibly fail to represent it well, while a polynomial regression line can bend to follow the actual shape of the data by introducing curvature into the model.
Choosing the right degree
The central design decision in polynomial regression is choosing the degree of the polynomial, and it's a real tradeoff. A lower-degree polynomial risks underfitting, failing to capture the real complexity in the data. A higher-degree polynomial risks overfitting, capturing noise in the training data rather than the genuine underlying pattern. Cross-validation is the recommended way to find the right balance, since it evaluates a model's performance on data it hasn't seen, rather than just how well it fits the training set.
Why it's still called "linear" regression
One point that trips people up: even though the resulting regression line curves, polynomial regression is still referred to as polynomial linear regression. That's because "linearity" here refers to the coefficients, not to the X variable. The real question is whether the function can be expressed as a linear combination of its coefficients. Since the coefficients are the unknowns being solved for, and the model fits them the same way linear regression does, the underlying math stays linear even though the resulting curve is not.
Implementing it on the Boston Housing dataset
The practical demonstration uses the Boston Housing dataset, loaded through scikit-learn, to predict house prices, a continuous variable, which makes this a regression problem. Before modeling, the tutorial explores relationships in the data with a Seaborn pair plot, which shows that the target column (median value) is strongly correlated with the LSTAT and RM columns, and that both of those relationships have a slight nonlinear curve rather than being perfectly straight.
The comparison starts with a plain linear regression using scikit-learn's LinearRegression, which produces an R2 score of 0.54 and a root mean squared error of 5.78, calculated using scikit-learn's mean squared error function and taking its square root. Because the LSTAT relationship showed nonlinear curvature, the tutorial then applies polynomial regression: first creating a matrix of features raised to different powers using scikit-learn's PolynomialFeatures, using a degree of two, and then feeding those extended features into the same linear regression model. The polynomial version raises the R2 score by about 8 percentage points, a meaningful jump, and drops the RMSE to 5.23, a decrease of roughly 0.5. Both metrics point the same direction: the polynomial model fits the actual data better than the plain linear model did.
Key takeaways
- Polynomial regression models the relationship between X and Y as an Nth-degree polynomial, letting the regression line curve instead of forcing a straight line onto curved data.
- Choosing the polynomial's degree involves a tradeoff between underfitting (too low a degree) and overfitting (too high a degree); cross-validation helps find the right balance.
- Polynomial regression is still called "linear" because the model is linear in its coefficients, not in the X variable itself.
- On the Boston Housing dataset, plain linear regression scored an R2 of 0.54 and an RMSE of 5.78.
- Adding polynomial features of degree two raised the R2 score by about 8 percentage points and reduced the RMSE to 5.23, confirming the nonlinear relationship visible in the pair plot.
Who this is for
This tutorial is built for anyone learning regression techniques in Python, whether just starting with scikit-learn or already comfortable with linear regression and looking to extend that toolkit to nonlinear relationships. It walks through both the conceptual reasoning and the actual code, using NumPy, pandas, and scikit-learn, making it a solid reference for students and practitioners working through a real dataset rather than a toy example.
Full transcript(auto-generated, with timestamps)
[0:02]Welcome everyone today we are ding into the world of regression analysis and a shining Spotlight on one of its versatile techniques polom regression so let's Dive Right In first things first what is polinomial regression well it's a form of regression analysis where the relationship between the independent variable X and the dependent variable Y is modeled as an Nth Degree polom so let's look at this the red points are at data points and the blue line is a regression line linear regression has an assumption of linearity which is almost always an approximation this approximation might not be a good one at times so unlike simple linear regression which fits a
[0:56]Straight line on the data polinomial regression can capture more complex relations between variables by introducing curves into the model as you can see this regression line is not very well representative of the data a good line would be the one which follows the pattern of this curve and this data is not linear so Inc comes polinomial regression let's break it down a bit further imagine we have a data set with scattered points that doesn't always seem to follow a linear path pattern this is where polinomial regression comes into play by increasing the degree of the polinomial we can capture the curvature of the data more accurately the flexibility allows
[1:43]Us to fit curves that closely align with the underlying patterns in the data now you might be wondering how do we choose the degree of the polinomial this decision is crucial as it impacts the model's performance a lower degree polom May underfit the data failing to capture its complexity while a higher degree polinomial May overfit capturing noise in the data rather than the underlying pattern finding the right balance is key techniques like cross validation can help us determine the optimal degree by evaluating the model's performance on unseen data let us talk about strengths and weaknesses of polinomial regression one of its major strengths is its
[2:34]Flexibility in capturing nonlinear relationships in the data making it suitable for a wide range of applications however polinomial regression also has its limitations as the degree of the polinomial increases the model becomes more complex leading to potential overfitting and increased computational complexity it is important to strike a balance between model complexity and performance to ensure reliable predictions despite its challenges polinomial regression remains a valuable tool in the data scientists toolbox offering a powerful approach to modeling complex relationships in the data this is what a linear regression expression looks like and as I said earlier we need to increase the degree of the polinomial so this is what a
[3:32]Polom regression uh equation would look like if we use polinomial regression on the above data that we saw our regression line would look like this which would be more representative of the data now even after getting a curve as a line we a polinomial regression is also referred to as polinomial linear regession why is that that is because we're talking about linearity when we talking about linearity we are talking about the coefficients and not the X variable the question is can this function be expressed as a linear combination of these coefficients this is because the coefficients are unknowns here and our goal is to find the actual
[4:21]Value of the coefficients so that we can use those coefficients to plug in the X values and predict y let us look at polinomial regression in action we will use the Boston Housing data set which can be imported from sk. data sets import loore Boston a regression algorithm predicts a continuous variable we will be predicting the prices of the houses before you uh apply regression it's important that you explore the relation ships in the data set cbon pair plot helps you to explore different relationships in the graph and you can see that MV column is strongly correlated to lstat and RM columns also the relationship over here
[5:19]It's not necessarily linear it has a little curvature to it the MV versus RM graph and the MV versus L stattic graft they have a slight nonlinear variation so let's apply polinomial regression before that we will split the data into training and test split we'll also do simple linear regression so that we can compare the results of our models you can do simple linear regression by doing from SK learn. import linear model import linear regression we get a R2 score of 0.54 now let us also check the root mean squ error we get 5.78 root mean squared error can be used uh from skarn metrix
[6:16]Import mean squared error and then you to take a square root of it next since we observe that L stat has a slight nonlinear variation we can use polinomial regression and then compare this r s and r values Step One is we create a matrix of features at different powers for that we will use from SK learn. preprocessing import polinomial features step two is we integrate that into the linear regression model yes at the end we will be doing a linear regression only however that this time the linear regression will be performed on the extended features which is features raised to the power in this
[6:59]Case I have taken a degree of two this is the value of the value of the nth degree next once we have our polinomial features ready in X poly train we can Implement linear regression and then we can take the R2 score notice that our R2 score has has a significant chump an 8% chump which which means our model is doing good and we are on the right track we can also look at the rmsc value which has decreased to 5.23 so there's roughly .5 decrease in the RMC value it's not much but it clearly says that your the goodness of fit as measured by the R2 r s value is
[7:52]Better for polinomial regression and there you have it folks a glimpse into the world of polinomial regression I I hope this video provided you with a better understanding of how this technique works and its applications in data analysis stay tuned for more enlightening content and until next time happy exploring
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53