Run this notebook: Open in Colab Open in Kaggle

Plot Svm Regression

=================================================================== Support Vector Regression (SVR) using linear and non-linear kernels

Toy example of 1D regression using linear, polynomial and RBF kernels.

Imports for Support Vector Regression (SVR) with Different Kernels

Support Vector Regression adapts the SVM framework from classification to regression by introducing an epsilon-insensitive tube around the predicted function. Points within this tube (distance less than epsilon from the prediction) incur zero loss, while points outside are penalized linearly. Only the points outside the tube become support vectors, making SVR naturally sparse – the prediction depends on a subset of training data.

Kernel choice for regression: The linear kernel fits a straight line (like ordinary linear regression but with the epsilon-tube), suitable for truly linear relationships. The RBF kernel can fit arbitrary smooth curves by summing Gaussian bumps centered at support vectors, making it the most flexible choice. The polynomial kernel fits polynomial curves of a specified degree. The C parameter controls the penalty for points outside the tube (larger C means the model tries harder to fit every point), while epsilon controls the tube width (larger epsilon means more tolerance for errors, yielding fewer support vectors and smoother predictions).

# Authors: The scikit-learn developers
# SPDX-License-Identifier: BSD-3-Clause

import matplotlib.pyplot as plt
import numpy as np

from sklearn.svm import SVR

# %%
# Generate sample data
# --------------------
X = np.sort(5 * np.random.rand(40, 1), axis=0)
y = np.sin(X).ravel()

# add noise to targets
y[::5] += 3 * (0.5 - np.random.rand(8))

# %%
# Fit regression model
# --------------------
svr_rbf = SVR(kernel="rbf", C=100, gamma=0.1, epsilon=0.1)
svr_lin = SVR(kernel="linear", C=100, gamma="auto")
svr_poly = SVR(kernel="poly", C=100, gamma="auto", degree=3, epsilon=0.1, coef0=1)

# %%
# Look at the results
# -------------------
lw = 2

svrs = [svr_rbf, svr_lin, svr_poly]
kernel_label = ["RBF", "Linear", "Polynomial"]
model_color = ["m", "c", "g"]

fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 10), sharey=True)
for ix, svr in enumerate(svrs):
    axes[ix].plot(
        X,
        svr.fit(X, y).predict(X),
        color=model_color[ix],
        lw=lw,
        label="{} model".format(kernel_label[ix]),
    )
    axes[ix].scatter(
        X[svr.support_],
        y[svr.support_],
        facecolor="none",
        edgecolor=model_color[ix],
        s=50,
        label="{} support vectors".format(kernel_label[ix]),
    )
    axes[ix].scatter(
        X[np.setdiff1d(np.arange(len(X)), svr.support_)],
        y[np.setdiff1d(np.arange(len(X)), svr.support_)],
        facecolor="none",
        edgecolor="k",
        s=50,
        label="other training data",
    )
    axes[ix].legend(
        loc="upper center",
        bbox_to_anchor=(0.5, 1.1),
        ncol=1,
        fancybox=True,
        shadow=True,
    )

fig.text(0.5, 0.04, "data", ha="center", va="center")
fig.text(0.06, 0.5, "target", ha="center", va="center", rotation="vertical")
fig.suptitle("Support Vector Regression", fontsize=14)
plt.show()