Another Way of Growing Trees: XGBoost's grow_policy

Controlling tree growth in XGBoost

In addition to limiting the maximum depth of trees using a max_depth hyperparameter, there is another paradigm for controlling tree growth: finding the node where a split would result in the greatest reduction in the loss function, and splitting this node, regardless of how deep it will make the tree. This may result in a tree with one or two very deep branches, while the other branches may not have grown very far. XGBoost offers a hyperparameter called grow_policy, and setting this to lossguide results in this kind of tree growth, while the depthwise option is the default and grows trees to an indicated max_depth, as we’ve done in the chapter “Decision Trees and Random Forests,” and so far in this chapter. The lossguide grow policy is a newer option in XGBoost and mimics the behavior of LightGBM, another popular gradient boosting package.

To use the lossguide policy, it is necessary to set another hyperparameter we haven’t discussed yet, tree_method, which must be set ...