Math behind Simple and Multiple Linear Regression

Introduction

Linear regression is a fundamental machine learning algorithm used for predicting a continuous target variable based on one or more input features. It's a simple yet powerful method for modeling the relationship between variables. In simple linear regression, we have one independent variable, while in multiple linear regression, we have multiple independent variables.

1. Simple Linear Regression

In simple linear regression, we have only one feature (X₁), the dataset will look like:

Cost of House (Y)	Area of House(X)
150,000	1500
450,000	2000
2,000,000	10000
350,000	3000
...	...

Equation

Thus, the equation for Simple Linear Regression becomes:

Y = θ₀ + θ₁X + ϵ

Here,

• Y is the predicted value
• θ₀ is the intercept
• θ₁ is the slope
• ϵ is the error

Cost Function

The cost function (J) using the Mean Squared Error (MSE) can be defined as :

J(θ₀,θ₁)=(1/2n) * Σⁿ_i=1(Y_i - (θ₀ + θ₁X_i))²

Here,

• n is the number of data points
• Y_i is the actual value
• (θ₀ + θ₁X_i) is the predicted value for data point i

2. Multiple Linear Regression

In Multiple linear regression, we have more than one feature (X₁..X_p ), the dataset will look like:

Cost of House (Y)	Area of House(X)	Number of Rooms	Built In
150,000	1500	3	1950
450,000	2000	4	1990
2,000,000	10000	10	2019
350,000	3000	2	2010
...	...	...	...

Equation

Thus, the equation for Multiple Linear Regression becomes:

Y = θ₀ + θ₁X₁ + θ₂X₂ + .. + θ_pX_p

Cost Function

The cost function (J) using the Mean Squared Error (MSE) can be defined as :

J(θ₀,θ₁,θ₂..θ_p)=(1/2n) * Σⁿ_i=1(Y_i - (θ₀ + θ₁X_1i + θ₂X_2i +.... + θ_pX_pi))²

Here,

• p is the number of features
• (θ₀ + θ₁X_1i + θ₂X_2i +.... + θ_pX_pi) is the predicted value for data point i

3. Polynomial Regression

Polynomial regression is a powerful extension of linear regression. While linear regression models relationships using straight lines, polynomial regression can capture curved and nonlinear relationships between variables. We are using only 1 feature here, as the number of feature increases, the equation becomes more complex.

Equation

The equation for Polynomial Regression is:

Y = θ₀ + θ₁X¹₁ + θ₂X²₁ + .. + θ_nX^m₁ + ϵ

Here,

• θ₀, θ₁ ... θ_m are the coefficients to be estimated
• ϵ is the error term
• m is the degree of polynomial

Cost Function

The cost function (J) using the Mean Squared Error (MSE) remains the same :

J(θ₀,θ₁,θ₂..θ_p)=(1/2n) * Σⁿ_i=1(Y_i - θ₀ - Σ^m_j=1θ_jX_ij)²

Note The summation(Σ) is still for the number of data points along with one more Σ for the number of features

4. Lasso and Ridge Regression

Lasso

Lasso (Least Absolute Shrinkage and Selection Operator) regression is a regularization technique that can not only predict but also select important features. It works by finding the best linear equation (a combination of features with coefficients) that fits your data and predicts the target variable. However, it adds a twist: it penalizes the absolute values of the coefficients of the features.

Cost function

The equation for Lasso Regression cost function adds λ.Σⁿ_i=1​∣β_i​∣:

J(θ₀,θ₁,θ₂..θ_n)=(1/2n) * Σⁿ_i=1(Y_i - (θ₀ + θ₁X_1i + θ₂X_2i...+θ_nX_ni))² + λ.Σⁿ_j=1​∣β_j​∣

The penalty term λ.Σⁿ_i=1​∣β_i​∣ encourages some coefficients to become exactly zero, effectively performing feature selection. Lasso helps in simplifying complex models by removing less important features. This is L1 regularization as we are doing regularization of degree 1.

Ridge

Ridge regression uses L2 regularization, which adds the squares of the coefficients as a penalty term to the cost function. L2 regularization helps prevent overfitting by shrinking the coefficients towards zero but doesn't force them to become exactly zero. It retains all features in the model.

Cost Function

The equation for Ridge Regression cost function adds λ.Σⁿ_i=1​∣β²_i​∣: