Chapter 6: ElasticNet regularization

Ridge regularization is an alternative to LASSO. It minimizes the L2-Norm instead of the L1-Norm, which means that the weights are maintained under control but not necessarily to zero. This offers a smoother alternative to LASSO, as the modelled function might still need to be high-dimensional in certain areas.

It is defined as follows:

class ElasticNet:

    def __init__(self, _lambda, l1_ratio=0.5):
        self.ratio = l1_ratio
        self._lambda = _lambda

    def __call__(self, theta):

        l1 = LASSO(_lambda=1)
        l2 = Ridge(_lambda=1)

        return self._lambda * (self.ratio * l1(theta) + (1 - self.ratio) * l2(theta))

    def gradient(self, theta):

        l1 = LASSO(_lambda=1)
        l2 = Ridge(_lambda=1)

        return self._lambda * (self.ratio * l1.gradient(theta) + (1 - self.ratio) * l2.gradient(theta))