polynomial regression torch

Daniel Parker

3 min read

·

26-02-2025

1

0

Share

Polynomial regression extends linear regression by adding polynomial terms to the model, enabling it to capture non-linear relationships in data. This guide demonstrates how to implement polynomial regression using PyTorch, a powerful deep learning framework. We'll cover the theoretical underpinnings, practical implementation, and best practices.

Understanding Polynomial Regression

Unlike linear regression, which models relationships with a straight line, polynomial regression uses higher-order polynomials to fit curves to the data. This allows for more flexibility in capturing complex patterns. The general form of a polynomial regression model is:

y = β₀ + β₁x + β₂x² + β₃x³ + ... + βₙxⁿ

where:

• y is the dependent variable
• x is the independent variable
• β₀, β₁, β₂, ..., βₙ are the coefficients to be estimated
• n is the degree of the polynomial

A higher degree polynomial (larger 'n') allows for more complex curves, but also increases the risk of overfitting. Overfitting occurs when the model fits the training data too closely, failing to generalize well to unseen data.

Implementing Polynomial Regression in PyTorch

Let's build a polynomial regression model using PyTorch. We'll use a simple dataset for demonstration purposes, but the principles can be easily extended to more complex scenarios.

First, we import necessary libraries:

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

Next, we generate some sample data with a non-linear relationship:

# Generate sample data
np.random.seed(0)
X = np.linspace(-1, 1, 100)
y = 2*X**3 + X**2 - X + 1 + np.random.normal(0, 0.2, 100) #Add some noise

# Convert to PyTorch tensors
X = torch.tensor(X, dtype=torch.float32).reshape(-1,1)
y = torch.tensor(y, dtype=torch.float32).reshape(-1,1)

Now, we define our model:

# Define the model
class PolynomialRegression(nn.Module):
    def __init__(self, degree):
        super(PolynomialRegression, self).__init__()
        self.degree = degree
        self.linear = nn.Linear(degree + 1, 1)

    def forward(self, x):
        # Create polynomial features
        polynomial_features = torch.cat([x**i for i in range(self.degree + 1)], dim=1)
        # Pass through linear layer
        output = self.linear(polynomial_features)
        return output

# Initialize model (let's use a 3rd-degree polynomial)
model = PolynomialRegression(degree=3)

We define our loss function and optimizer:

# Define loss function and optimizer
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

Then, we train the model:

# Train the model
epochs = 1000
for epoch in range(epochs):
    # Forward pass
    y_pred = model(X)
    loss = criterion(y_pred, y)

    # Backward pass and optimization
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print(f'Epoch {epoch}, Loss: {loss.item():.4f}')

Finally, we can plot the results:

# Plot the results
predicted = model(X).detach().numpy()
plt.scatter(X.numpy(), y.numpy(), label='Data')
plt.plot(X.numpy(), predicted, color='red', label='Prediction')
plt.xlabel('X')
plt.ylabel('y')
plt.legend()
plt.title('Polynomial Regression')
plt.show()

This code creates a polynomial regression model, trains it on sample data, and visualizes the results. Remember to adjust the degree, learning_rate, and epochs parameters for optimal performance.

Choosing the Degree of the Polynomial

The degree of the polynomial is a crucial hyperparameter. A higher degree can capture more complex relationships but increases the risk of overfitting. Techniques like cross-validation can help determine the optimal degree. Too low a degree will underfit, while too high a degree will overfit.

Regularization

Regularization techniques like L1 or L2 regularization can help mitigate overfitting by adding penalties to the model's coefficients. These penalties discourage the model from fitting the noise in the data. You can incorporate these into your PyTorch model by adding a regularization term to your loss function.

This comprehensive guide provides a strong foundation for implementing and understanding polynomial regression using PyTorch. Remember to experiment with different parameters and techniques to optimize your model for your specific dataset.