AdaBoost#
Short for Adaptive Boosting, this ensemble classifier can improve the performance of an otherwise weak classifier by focusing more attention on samples that are harder to classify. It builds an additive model where, at each stage, a new learner is trained and given an influence score inversely proportional to the loss it incurs at that epoch. In addition, the learner features progress monitoring which stops training when it can no longer improve the validation score.
Note
The default base learner is a Classification Tree with a max height of 1 i.e a Decision Stump.
Note
Progress monitoring and early stopping require a validation set. Use setValidationDataset() to supply one.
Interfaces: Estimator, Learner, Probabilistic, Verbose, Persistable
Data Type Compatibility: Depends on base learner
Parameters#
| # | Name | Default | Type | Description |
|---|---|---|---|---|
| 1 | base | ClassificationTree | Learner | The base weak classifier to be boosted. |
| 2 | rate | 1.0 | float | The learning rate of the ensemble i.e. the shrinkage applied to each step. |
| 3 | ratio | 0.8 | float | The ratio of samples to subsample from the training set to train each weak learner. |
| 4 | epochs | 100 | int | The maximum number of training epochs. i.e. the number of times to iterate before terminating. |
| 5 | minChange | 1e-5 | float | The minimum change in the training loss necessary to continue training. |
| 6 | evalInterval | 3 | int | The number of epochs to train before evaluating the model using the validation set. |
| 7 | window | 5 | int | The number of evaluations without improvement in the validation score to wait before considering an early stop. Set to 0 to disable early stopping. |
| 8 | metric | FBeta | Metric | The validation metric used to score the generalization performance of the model during training. |
Example#
use Rubix\ML\Classifiers\AdaBoost;
use Rubix\ML\Classifiers\ExtraTreeClassifier;
use Rubix\ML\CrossValidation\Metrics\MCC;
$estimator = new AdaBoost(new ExtraTreeClassifier(3), 0.1, 0.5, 200, 1e-3, 3, 5, new MCC());
Additional Methods#
Return the loss for each epoch from the last training session.
public losses() : float[]|null
Return the progress table combining every epoch recorded during the last training session — the loss, the validation score, and the gradient norm when available — into a single ordered sequence.
public progress() : iterable
Set the dataset used to score the model during training. Once a validation dataset is set, evalInterval and window determine how often it is scored and when training stops early. Pass null to disable progress monitoring and early stopping.
public setValidationDataset(?Labeled $dataset) : void
Return the validation score for each epoch from the last training session.
public scores() : float[]|null