Skip to content

[source]

Gradient Boost#

Gradient Boost (GBM) is a stage-wise additive ensemble that uses a Gradient Descent boosting scheme for training boosters (Decision Trees) to correct the error residuals of a base learner.

Note

The default booster is a Regression Tree with a max height of 3.

Note

Progress monitoring and early stopping require a validation set. Use setValidationDataset() to supply one.

Interfaces: Estimator, Learner, Verbose, Ranks Features, Persistable

Data Type Compatibility: Categorical and Continuous

Parameters#

# Name Default Type Description
1 booster RegressionTree Learner The regressor used to up the error residuals of the base learner.
2 rate 0.1 float The learning rate of the ensemble i.e. the shrinkage applied to each step.
3 ratio 0.5 float The ratio of samples to subsample from the training set to train each booster.
4 epochs 1000 int The maximum number of training epochs. i.e. the number of times to iterate before terminating.
5 minChange 1e-5 float The minimum change in the training loss necessary to continue training.
6 evalInterval 3 int The number of epochs to train before evaluating the model using the validation set.
7 window 5 int The number of evaluations without improvement in the validation score to wait before considering an early stop. Set to 0 to disable early stopping.
8 metric RMSE Metric The metric used to score the generalization performance of the model during training.

Example#

use Rubix\ML\CrossValidation\Metrics\SMAPE;
use Rubix\ML\Regressors\GradientBoost;
use Rubix\ML\Regressors\RegressionTree;

$estimator = new GradientBoost(new RegressionTree(3), 0.1, 0.8, 1000, 1e-4, 3, 10, new SMAPE());

Additional Methods#

Return the loss for each epoch from the last training session.

public losses() : float[]|null

Return the progress table combining every epoch recorded during the last training session — the loss, the validation score, and the gradient norm when available — into a single ordered sequence.

public progress() : iterable

Set the dataset used to score the model during training. Once a validation dataset is set, evalInterval and window determine how often it is scored and when training stops early. Pass null to disable progress monitoring and early stopping.

public setValidationDataset(?Labeled $dataset) : void

Return the validation score for each epoch from the last training session.

public scores() : float[]|null

References#

  1. J. H. Friedman. (2001). Greedy Function Approximation: A Gradient Boosting Machine. ↩

  2. J. H. Friedman. (1999). Stochastic Gradient Boosting. ↩

  3. Y. Wei. et al. (2017). Early stopping for kernel boosting algorithms: A general analysis with localized complexities. ↩

  4. G. Ke et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. ↩