Chapter 1: Introduction

Logistic Regression is a variation of the Linear Regression model which can be used for classification. One could first think, why don’t we just use Linear Regression as is? Indeed, this could potentially work. The target variable would be either 1 or 0 depending on the class, and the model would predict whichever class its prediction is closer to.

Visually, the fitted curve would serve as a decision boundary. But a quick example already shows that this isn’t a viable option.

Indeed, when points are added to the dataset, even though these points should be “obvious”, they skew the decision boundary, which causes classification mistakes. So there must be a better way. Logistic Regression is the solution to this problem.

Ideally, we want our model to be more certain of its classification for points that are further away from the decision boundary. This is why Logistic Regression uses the sigmoid function, also called logit function, to model its hypothesis. The sigmoid function is defined as follows:

class Sigmoid():

    def __call__(self, x):

        return 1 / (1 + np.exp(-x))

    def gradient(self, x):

        p = self.__class__(x)

        return p * (1 - p)

The Sigmoid function takes any real input and maps it between 0 and 1, making it suitable to also represent probabilities. Like Linear Regression, Logistic Regression assumes that the target variable of a data point can be explained as a linear combination of its features. The only difference is that this linear combination is then passed into the Sigmoid function. The goal then is to learn weights such that the final output equals 1 or 0 based on which class the data point belongs to.