relgam: Fitting reluctant generalized additive models

Introduction and motivation

tl;dr: Reluctant generalized additive modeling (RGAM) produces highly interpretable sparse models which allow non-linear relationships between the response and each individual feature. However, non-linear relationships are only included if deemed important in improving prediction performance. RGAMs working with quantitative, binary, count and survival responses and is computationally efficient.

Consider the supervised learning setting, where we have observations of features for and , along with n responses . Let denote the values of the th feature. Generalized linear models (GLMs) assume that the relationship between the response and the features is

where is a link function and is a mean-zero error term. Generalized additive models (GAMs) are a more flexible class of models, assuming the true relationship to be

where the ‘s are unknown functions to be determined by the model.

These two classes of models include all features in the model which is often undesirable, especially when we have tons of features. (We usually expect only a small fraction of features to have any influence on the response variable.) This is especially problematic with GAMs as overfitting can occur much more easily. A host of methods have arisen to create sparse GAMs, i.e. GAMs that involve only a handful of features. Earlier examples of such examples include COSSO (Lin & Zhang 2006) and SpAM (Ravikumar et al. 2007).

While providing sparsity, these methods dictated that the features included had to have a non-linear relationship with the response even if a linear relationship would have been sufficient to capture the relationship. More sophisticated methods were developed to give both sparsity and the possibility of linear or non-linear relationships between the features and response. Examples of such methods are GAMSEL (Chouldechova & Hastie 2015), SPLAM (Lou et al. 2016) and SPLAT (Petersen & Witten 2019). GAMSEL is available on R in the gamsel package (see my unofficial vignette here) and I was not able to find R packages for the other two methods.

Reluctant generalized additive models (RGAM) fall in the same class as these last group of methods. It is available on R in the relgam package. RGAMs are computationally fast and work with quantitative, binary, count and survival response variables. (To my knowledge, existing software only works for quantitative and binary variables.)

RGAM: What is it?

Reluctant generalizing additive modeling was inspired by reluctant interaction modeling (Yu et al. 2019). The idea is that

A simple example

The CRAN vignette is the best place to start learning how to fit RGAMs in practice. Below I give an example of the types of models that can come out of RGAM. (Code for this example can be found here.)

We simulate data with observations and features. Each entry in the matrix is an independent draw from the standard normal distribution, and the true response is

We fit a RGAM to this data for a sequence of values. The larger the index, the smaller the value, the less penalty imposed in the lasso in Step 3, resulting in more flexible models.

For each value, RGAM’s predictions have the form

Let . We plot the model fits for the first 30 values in the animation below. In each of the 12 panes, we see the estimated for each variable (in blue, green or red), and the true relationship in black.