Support Vector Regression (SVR) and its Implementation on the Boston Housing Dataset
SVR adapts support vector machines for continuous prediction using an epsilon-insensitive tube around the fitted hyperplane. This video covers the theory and a Boston Housing implementation.
Most regression problems are handled with straightforward methods like linear regression, but those methods struggle when relationships in the data are nonlinear or when outliers are common. Support Vector Regression, or SVR, was built to handle exactly that kind of problem, and this video explains how it works before implementing it on the Boston Housing dataset.
Where SVR comes from
SVR was invented in the 1990s at Bell Labs by Vladimir Vapnik. Despite the name, it's a regression algorithm, specifically a supervised machine learning method used to predict continuous outcomes rather than discrete categories. It's built on the same underlying principles as SVM, or support vector machines, which are more commonly used for classification tasks.
The epsilon-insensitive tube
The central idea in SVR is a tube-like structure called the epsilon-insensitive tube. Its width is controlled by a parameter called epsilon, and it acts as a margin of error the model is allowed to have. What SVR actually cares about are the points that fall outside this tube, not the ones inside it. Points above and below the tube are called slack variables, and the goal is to minimize the sum of their distances from the tube. The points sitting exactly on the tube's margin are the support vectors, since they define the structure of the tube and give the method its name.
How SVR works
SVR requires feature scaling before it's applied. The process starts by mapping the input features into a high-dimensional feature space, which can be done using a kernel function, a variant known as kernel SVR. From there, an optimization step finds a hyperplane that best fits the data by maximizing the margin, the distance between the hyperplane and the closest support vectors. Rather than fitting the hyperplane directly to every data point, SVR fits a tube around it within the epsilon margin, and instances that fall inside that margin don't affect the model's predictions, which is why the tube is called epsilon-insensitive. Once trained, SVR predicts new data points by mapping them into the same feature space and determining their position relative to the hyperplane within the tube.
Strengths, weaknesses, and kernels
SVR is effective in high-dimensional spaces and fairly robust to overfitting, and its variety of kernel functions let it model nonlinear relationships by making them linear in a higher-dimensional space. On the downside, it's sensitive to hyperparameters, particularly the choice of kernel, the regularization parameter, and epsilon, and it can be computationally expensive on large datasets. It's also less interpretable than simple linear regression, even when it performs better. Among the available kernels, the linear kernel is the simplest and assumes a linear relationship between inputs and target. The polynomial kernel handles data with a nonlinear, polynomial relationship. The radial basis function kernel, also called the Gaussian kernel, measures similarity between points using Euclidean distance and is widely used when the shape of the relationship isn't known in advance. The sigmoid kernel is particularly suited to binary classification tasks and works similarly to logistic regression.
Implementing SVR on Boston Housing
The implementation uses the UCI Machine Learning Repository's Boston Housing dataset, imported through scikit-learn along with train_test_split and StandardScaler. Scaling has to be applied correctly: fit_transform is used on the training set, while only transform is applied to the test set, so that test data never influences the scaling parameters. The radial basis function kernel is used for this implementation, and after fitting the model, performance is evaluated using mean squared error, a standard metric for regression tasks.
Key takeaways
- SVR adapts support vector machine principles to continuous prediction using an epsilon-insensitive tube around the fitted hyperplane.
- Points within the tube margin don't affect predictions; only points outside it, plus the support vectors on the margin, shape the model.
- SVR requires feature scaling, with fit_transform on training data and transform only on test data.
- Kernel choice matters: linear, polynomial, RBF, and sigmoid kernels each suit different data shapes.
- SVR trades interpretability and computational cost for the ability to model nonlinear relationships robustly.
Who this is for
This video is aimed at learners working through core regression algorithms in machine learning who want to understand both the theory behind SVR and a concrete implementation in scikit-learn.
Full transcript(auto-generated, with timestamps)
[0:02]Support Vector regression is an algorithm that was invented in the 1990s at Bell Labs by Vladimir vapnik it is of course as the name suggests a regression algorithm but also a particularly a supervised machine learning algorithm the idea is that you use it to predict continuous outcomes rather than discrete categories it is based on the same PR principles as svm which are used for classification tasks so before we dive deep into it let's look at this diagram we have a tube like structure over here isn't it that is called an Epsilon insensitive tube the width of the tube or the value parameter Epsilon is not perpendicular
[0:58]But vertical to the tube the tube is kind of like a margin of error that we are allowing our model to have what we care about are the errors of the points outside these this Epsilon insensitive tube and not inside the error points above the uh tube and below the tube are called slack variables we want the sum of these distances that is from the tube to the slack variables to be minimized the points on the tube margin are support vectors they're dictating how this tube is created and hence they're called support vectors since they're supporting the structure of the tube some key things that I want to
[1:46]Mention are that svr needs feature scaling to be applied before uh you apply the model itself so how does Sr work as I mentioned you need to map the features into a high dimensional feature space this can you can uh use a kernel function for it too uh and that is more of Kernel svr the second step is optimization that is once the data is mapped into feature space svr tries to find a hyperplane uh which is the big black line over here that best fits the data this hyperplane is chosen such that it maximizes the margin which the distance between the which is the distance
[2:29]Between the hyperplane and the closest data points known as support vectors in svr instead of fitting hyperplane directly to the data point it fits a tube around the hyper plane within a certain margin of error known as Epsilon which is basically the Epsilon insensitive tube this Tube accounts for acceptable error in prediction the goal is to minimize this error while maximizing the margin Epsilon is a hyperparameter adding more training instances within the margin does not affect the model's predictions therefore the tube is called Epsilon insensitive prediction after training svr can predict the target variable for new data points by mapping them into the same feature space and then determining their
[3:25]Positive position relative to the hyperplane within the tube the predicted value is then adjusted based on the margin Sr seeks to find a hyper plane that best fits the data points in a continuous space svr tries to fit as many instances possible on this hyp plane or within this uh Epsilon insensitive tube while limiting margin violations so before we proceed to the implementation of svr let's look at the pros and cons of svr first of all it is very if Pro Pros first of all it's very effective in high dimensional spaces it's very robust to overfitting you can use a lot of Kernel functions uh if you
[4:11]Have data which is nonlinear uh but is linear in a higher dimensional format and building on top of that these kernel functions help you uh model nonlinear Relationships by making sure that they're linear in a higher Dimension the cons are that there's it this s is very sensitive to hyper parameters particularly the kernel that you select any regularization parameter that you select or even Epsilon as which I mentioned earlier is a hyper parameter it is also very computationally expensive for large data sets the model is less interpretable than simple linear regression but it performs better on some tasks on most tasks and as I mentioned earlier you need to scale the
[5:05]Data meaning the data needs to be standardized or normalized so to ensure that all features equally contribute to the model so that is one g of it that you need scaling to be applied to the data before you implement the model so let's look at an example we will use svr and implement it on the UCI machine learning repositories Boston Housing data set this data set can be imported uh using SK learn. data sets and uh we will also include uh inclue import things like train test split and standard scal here's the information about this data set now as I mentioned earlier you need to scale your data but scaling needs to
[5:49]Be done first only on the training set for test you need to do transform for training you need to do fit transform when you're training the model you will be using XT trained scale and when you'll be testing the model you will be using X test scale now let's talk about kernels once again in svr the kernel function plays a crucial role in mapping the input data to a higher Dimension where it can be more easily separated or approximated by a Hy plane there are various kernels that you can use the linear kernel is the simplest kernel uh which is used when input features and the target is
[6:32]Assumed to be linear so that's the normal one however there are some other ones if if your data seems to have a polom relationship you can uh which is a nonlinear relationship you can use the polinomial kernal to make it linearly separable your data linearly separable in a higher Dimension radial basis function is another one it's also called as the gaussian kernal it measures the sity between two data points based on their ukian distance in the original feature space this kernel can capture complex non linear relationships and is widely used when the nature of the data is not known a priority and finally there's the
[7:18]Sigmoid K which is particularly useful for binary classification tasks and works on the same basis the way logistic regression works anyways for this task we'll be using the radial basis function as our kernel and we just fit our svr model since svr is a regression model we can use a metric like mean squar error and below is the value as seen for the mean squared error so this is spr in a nutshell thank you so much
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53