Chapter 6: ElasticNet regularization
Ridge regularization is an alternative to LASSO. It minimizes the L2-Norm instead of the L1-Norm, which means that the weights are maintained under control but not necessarily to zero. This offers a smoother alternative to LASSO, as the modelled function might still need to be high-dimensional in certain areas.
It is defined as follows:
class ElasticNet: def __init__(self, _lambda, l1_ratio=0.5): self.ratio = l1_ratio self._lambda = _lambda def __call__(self, theta): l1 = LASSO(_lambda=1) l2 = Ridge(_lambda=1) return self._lambda * (self.ratio * l1(theta) + (1 - self.ratio) * l2(theta)) def gradient(self, theta): l1 = LASSO(_lambda=1) l2 = Ridge(_lambda=1) return self._lambda * (self.ratio * l1.gradient(theta) + (1 - self.ratio) * l2.gradient(theta))