Logistic Regression

Introduction

Logistic Regression is a popular machine learning algorithm used for binary classification tasks, where the goal is to predict one of two possible outcomes (e.g., yes/no, 1/0, spam/not spam). At its core, Logistic Regression models the relationship between a set of independent variables (features) and the probability of a particular outcome. It's called "logistic" because it uses the logistic function (or sigmoid function) to map any real-valued number into a value between 0 and 1. This makes it suitable for estimating probabilities.

Sigmoid Function

p = 1/ (1 + e^-y)

Here,

y = θ₀ + θ₁X₁ + .. θ₁X_n

• y is the linear combination.
• θ₀, θ₁, θ₂, ... θ_n are the coefficients.
• X₀, X₁, X₂, ... X_n are the input features.
• p is the predicted probability that the outcome is 1

Decision Boundary

Typically, a threshold (e.g., 0.5) is chosen. If p is greater than the threshold, the predicted outcome is 1; otherwise, it's 0.

Cost Function

n logistic regression, the cost function, often referred to as the log loss or cross-entropy loss, is used to measure the error between the predicted probabilities and the actual binary outcomes (0 or 1). The goal is to find the values of the coefficients that minimize this error.

J(θ) = (-1/n)Σⁿ_i=1(y_i log(p_i) + (1 - y_i) log(1 - p_i))

Here:

• J(θ) is the cost function to be minimized
• n is the number of training examples
• y_i is the actual binary outcome (0 or 1) for the iith training example
• p_i is the predicted probability that the ith example belongs to class 1

The goal during training is to find the values of θ₀, θ₁, θ₂, ... θ_n that minimize this cost function. This is typically done using optimization algorithms like gradient descent.

References:

1. Activation Functions in Neural Networks

Note: Parts of the article are developed by using ChatGPT