While Gradient Boosting is often our go-to model, based on recent readings and personal experiments, here are five situations where it’s better to avoid using it:
When the relationship is mostly linear:
Linear or logistic regression:
Trains faster
Is more interpretable
Requires less tuning
When data is noisy and low in variability:
For noisy and sparse datasets, Gradient Boosting may not outperform simpler models like linear regression.
When extrapolation matters:
Gradient Boosting (based on decision trees) performs poorly in extrapolation. Better alternatives:
Neural Networks
Gaussian Processes
When you need a fast, nonlinear baseline:
Random Forest is a better choice when:
You need quick setup
You want to avoid overfitting
When the model is used for optimization:
The piecewise constant structure of Gradient Boosting leads to unstable gradients. Consider:
Neural Networks
Gaussian Processes
Splines
No comment