Skip to content

[source]

Logit Boost#

A stage-wise additive ensemble that uses regression trees to iteratively learn a Logistic Regression model for binary classification problems. Unlike standard Logistic Regression, Logit Boost has the ability to learn a smooth non-linear decision surface by training decision trees to follow the gradient of the cross entropy loss function. In addition, Logit Boost concentrates more effort on classifying samples that it is less certain about.

Note

Progress monitoring and early stopping require a validation set. Use setValidationDataset() to supply one.

Interfaces: Estimator, Learner, Probabilistic, Verbose, Ranks Features, Persistable

Data Type Compatibility: Depends on base learners

Parameters#

# Name Default Type Description
1 booster RegressionTree Learner The regressor used to fix up the error residuals of the base learner.
2 rate 0.1 float The learning rate of the ensemble i.e. the shrinkage applied to each step.
3 ratio 0.5 float The ratio of samples to subsample from the training set to train each booster.
4 epochs 1000 int The maximum number of training epochs. i.e. the number of times to iterate before terminating.
5 minChange 1e-5 float The minimum change in the training loss necessary to continue training.
6 evalInterval 3 int The number of epochs to train before evaluating the model using the validation set.
7 window 5 int The number of evaluations without improvement in the validation score to wait before considering an early stop. Set to 0 to disable early stopping.
8 metric F Beta Metric The metric used to score the generalization performance of the model during training.

Example#

use Rubix\ML\Classifiers\LogitBoost;
use Rubix\ML\Regressors\RegressionTree;
use Rubix\ML\CrossValidation\Metrics\FBeta;

$estimator = new LogitBoost(new RegressionTree(4), 0.1, 0.5, 1000, 1e-5, 3, 5, new FBeta());

Additional Methods#

Return the loss for each epoch from the last training session.

public losses() : float[]|null

Return the progress table combining every epoch recorded during the last training session — the loss, the validation score, and the gradient norm when available — into a single ordered sequence.

public progress() : iterable

Set the dataset used to score the model during training. Once a validation dataset is set, evalInterval and window determine how often it is scored and when training stops early. Pass null to disable progress monitoring and early stopping.

public setValidationDataset(?Labeled $dataset) : void

Return the validation score for each epoch from the last training session.

public scores() : float[]|null

References#

  1. J. H. Friedman et al. (2000). Additive Logistic Regression: A Statistical View of Boosting. ↩

  2. J. H. Friedman. (2001). Greedy Function Approximation: A Gradient Boosting Machine. ↩

  3. J. H. Friedman. (1999). Stochastic Gradient Boosting. ↩

  4. Y. Wei. et al. (2017). Early stopping for kernel boosting algorithms: A general analysis with localized complexities. ↩

  5. G. Ke et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. ↩