Run this notebook: Open in Colab Open in Kaggle

Plot Sgd Penalties

============== SGD: Penalties

Contours of where the penalty is equal to 1 for the three penalties L1, L2 and elastic-net.

All of the above are supported by :class:~sklearn.linear_model.SGDClassifier and :class:~sklearn.linear_model.SGDRegressor.

Imports for Visualizing L1, L2, and Elastic-Net Penalty Contours

The geometry of a regularization penalty determines the shape of its constraint region in weight space, which directly impacts what kinds of solutions the optimizer finds. Plotting the unit-level contours of each penalty in 2D weight space (w1, w2) provides powerful geometric intuition for why different penalties produce different coefficient patterns.

Geometric insight: The L1 penalty |w1| + |w2| = 1 forms a diamond (rotated square) with corners on the axes – these corners are where one coefficient is exactly zero. The L2 penalty w1^2 + w2^2 = 1 forms a circle, which has no corners, so the optimal point rarely lands exactly on an axis (coefficients shrink but do not vanish). The Elastic-Net penalty blends both, creating a shape between the diamond and circle. When the loss function’s contour (an ellipse for squared error) is tangent to the penalty contour, that tangent point is the regularized solution – and the diamond’s corners make L1 far more likely to produce sparse solutions.

# Authors: The scikit-learn developers
# SPDX-License-Identifier: BSD-3-Clause

import matplotlib.pyplot as plt
import numpy as np

l1_color = "navy"
l2_color = "c"
elastic_net_color = "darkorange"

line = np.linspace(-1.5, 1.5, 1001)
xx, yy = np.meshgrid(line, line)

l2 = xx**2 + yy**2
l1 = np.abs(xx) + np.abs(yy)
rho = 0.5
elastic_net = rho * l1 + (1 - rho) * l2

plt.figure(figsize=(10, 10), dpi=100)
ax = plt.gca()

elastic_net_contour = plt.contour(
    xx, yy, elastic_net, levels=[1], colors=elastic_net_color
)
l2_contour = plt.contour(xx, yy, l2, levels=[1], colors=l2_color)
l1_contour = plt.contour(xx, yy, l1, levels=[1], colors=l1_color)
ax.set_aspect("equal")
ax.spines["left"].set_position("center")
ax.spines["right"].set_color("none")
ax.spines["bottom"].set_position("center")
ax.spines["top"].set_color("none")

plt.clabel(
    elastic_net_contour,
    inline=1,
    fontsize=18,
    fmt={1.0: "elastic-net"},
    manual=[(-1, -1)],
)
plt.clabel(l2_contour, inline=1, fontsize=18, fmt={1.0: "L2"}, manual=[(-1, -1)])
plt.clabel(l1_contour, inline=1, fontsize=18, fmt={1.0: "L1"}, manual=[(-1, -1)])

plt.tight_layout()
plt.show()