LASSO penalization in generalized linear mixed models

Jacob Jonas Grytzka, TU Dortmund University

Co-authors: Paul Bürkner, TU Dortmund University; Andreas Groll, TU Dortmund University

Abstract: LASSO penalization is a well-established method for variable selection and regularization in regression modeling. However, its application in mixed models, which combine fixed and random effects, arises specific challenges related to the estimation methods used for these models. In our work, we focus on penalizing the fixed effects while leaving the random effects unaffected to preserve their hierarchical variance structures. This approach aims at eliminating irrelevant predictors and reducing model complexity without restricting the flexibility in modeling a hierarchical structure.

Instead of the classical LASSO penalty, we use a quadratic approximation, ensuring differentiability of the penalty. This modification facilitates optimization while retaining the core idea of sparse modeling. We implement our LASSO penalty in two widely applied R packages for mixed models.

Using extensive simulations, we demonstrate that the new penalty enables efficient variable selection in mixed models even in high-dimensional settings with much more predictors than observations.