gpboost.grid_search_tune_parameters

gpboost.grid_search_tune_parameters(param_grid, train_set, gp_model=None, num_try_random=None, params=None, num_boost_round=1000, early_stopping_rounds=None, folds=None, nfold=5, metric=None, use_gp_model_for_validation=True, train_gp_model_cov_pars=True, stratified=False, shuffle=True, fobj=None, feval=None, init_model=None, feature_name='auto', categorical_feature='auto', fpreproc=None, verbose_eval=1, seed=0, callbacks=None, metrics=None, return_all_combinations=False)[source]

Function for choosing tuning parameters from a grid in a determinstic or random way using cross validation or validation data sets.

Parameters:

param_grid (dict) – Candidate parameters defining the grid over which a search is done. See https://github.com/fabsig/GPBoost/blob/master/docs/Main_parameters.rst#tuning-parameters–hyperparameters-for-the-tree-boosting-part
train_set (Dataset) – Data to be trained on.
gp_model (GPModel or None, optional (default=None)) – GPModel object for the GPBoost algorithm
num_try_random (int, optional (default=None)) – Number of random trial on parameter grid. If none, a deterministic search is done
params (dict, optional (default=None)) – Other parameters not included in param_grid.
num_boost_round (int, optional (default=1000)) – Number of boosting iterations.
early_stopping_rounds (int or None, optional (default=None)) – Activates early stopping. The metric needs to improve at least every early_stopping_rounds round(s) to continue.
folds (generator or iterator of (train_idx, test_idx) tuples, scikit-learn splitter object or None, optional (default=None)) – If generator or iterator, it should yield the train and test indices for each fold. If object, it should be one of the scikit-learn splitter classes (https://scikit-learn.org/stable/modules/classes.html#splitter-classes) and have split method. This argument has highest priority over other data split arguments.
nfold (int, optional (default=5)) – Number of folds in CV.
metric (string, list of strings or None, optional (default=None)) –
Evaluation metric to be monitored when doing parameter tuning. If not None, the metric in params will be overridden.
- Default =”test_neg_log_likelihood” if there is a GPModel
- Non-exhaustive list of supported metrics: “test_neg_log_likelihood”, “mse”, “rmse”, “mae”, “crps_gaussian”, “auc”, “average_precision”, “binary_logloss”, “binary_error”
- See https://gpboost.readthedocs.io/en/latest/Parameters.html#metric-parameters for a complete list of valid metrics
use_gp_model_for_validation (bool, optional (default=True)) – If True, the ‘gp_model’ (Gaussian process and/or random effects) is also used (in addition to the tree model) for calculating predictions on the validation data. If False, the ‘gp_model’ (random effects part) is ignored for making predictions and only the tree ensemble is used for making predictions for calculating the validation / test error.
train_gp_model_cov_pars (bool, optional (default=True)) – If True, the covariance parameters of the ‘gp_model’ (Gaussian process and/or random effects) are estimated in every boosting iterations, otherwise the ‘gp_model’ parameters are not estimated. In the latter case, you need to either estimate them beforehand or provide values via the ‘init_cov_pars’ parameter when creating the ‘gp_model’
stratified (bool, optional (default=False)) – Whether to perform stratified sampling.
shuffle (bool, optional (default=True)) – Whether to shuffle before splitting data.
fobj (callable or None, optional (default=None)) –
Customized objective function. Only for independent boosting. The GPBoost algorithm currently does not support this. Should accept two parameters: preds, train_data, and return (grad, hess).

predslist or numpy 1-D array
The predicted values.

train_dataDataset
The training dataset.

gradlist or numpy 1-D array
The value of the first order derivative (gradient) for each sample point.

hesslist or numpy 1-D array
The value of the second order derivative (Hessian) for each sample point.

For binary task, the preds is margin. For multi-class task, the preds is group by class_id first, then group by row_id. If you want to get i-th row preds in j-th class, the access way is score[j * num_data + i] and you should group grad and hess in this way as well.
feval (callable, list of callable functions or None, optional (default=None)) –
Customized evaluation function. If more than one evaluation function is provided, only the first evaluation function will be used to choose tuning parameters Each evaluation function should accept two parameters: preds, train_data, and return (eval_name, eval_result, is_higher_better) or list of such tuples.

predslist or numpy 1-D array
The predicted values.

train_dataDataset
The training dataset.

eval_namestring
The name of evaluation function (without whitespaces).

eval_resultfloat
The eval result.

is_higher_betterbool
Is eval result higher better, e.g. AUC is is_higher_better.

For binary task, the preds is probability of positive class (or margin in case of specified fobj). For multi-class task, the preds is group by class_id first, then group by row_id. If you want to get i-th row preds in j-th class, the access way is preds[j * num_data + i]. To ignore the default metric corresponding to the used objective, set metric to the string "None".
init_model (string, Booster or None, optional (default=None)) – Filename of GPBoost model or Booster instance used for continue training.
feature_name (list of strings or 'auto', optional (default="auto")) – Feature names. If ‘auto’ and data is pandas DataFrame, data columns names are used.
categorical_feature (list of strings or int, or 'auto', optional (default="auto")) – Categorical features. If list of int, interpreted as indices. If list of strings, interpreted as feature names (need to specify feature_name as well). If ‘auto’ and data is pandas DataFrame, pandas unordered categorical columns are used. All values in categorical features should be less than int32 max value (2147483647). Large values could be memory consuming. Consider using consecutive integers starting from zero. All negative values in categorical features will be treated as missing values. The output cannot be monotonically constrained with respect to a categorical feature.
fpreproc (callable or None, optional (default=None)) – Preprocessing function that takes (dtrain, dtest, params) and returns transformed versions of those.
verbose_eval (int or None, optional (default=1)) – Whether to display information on the progress of tuning parameter choice. If None or 0, verbose is of. If = 1, summary progress information is displayed for every parameter combination. If >= 2, detailed progress is displayed at every boosting stage for every parameter combination.
seed (int, optional (default=0)) – Seed used to generate folds and random grid search (passed to numpy.random.seed).
callbacks (list of callables or None, optional (default=None)) – List of callback functions that are applied at each iteration. See Callbacks in Python API for more information.
metrics (string, list of strings or None, discontinued (default=None)) – This is discontinued. Use the renamed equivalent argument ‘metric’ instead
return_all_combinations (bool, optional (default=False)) – If True, all tried parameter combinations are returned

Returns:

return – Dictionary with the best parameter combination and score The dictionary has the following format: {‘best_params’: best_params, ‘best_num_boost_round’: best_num_boost_round, ‘best_score’: best_score} If return_all_combinations is True, then the dictionary contains an additional entry ‘all_combinations’

Return type:

dict

Example

>>> # Define parameter search grid
>>> # Note: if the best combination found below is close to the bounday for a paramter, you might want to extend the corresponding range
>>> param_grid = { 'learning_rate': [0.001, 0.01, 0.1, 1, 10],
>>>   'min_data_in_leaf': [1, 10, 100, 1000],
>>>   'max_depth': [-1], # -1 means no depth limit as we tune 'num_leaves'. Can also additionally tune 'max_depth', e.g., 'max_depth': [-1, 1, 2, 3, 5, 10]
>>>   'num_leaves': 2**np.arange(1,10),
>>>   'lambda_l2': [0, 1, 10, 100],
>>>   'max_bin': [250, 500, 1000, np.min([10000,n])],
>>>   'line_search_step_length': [True, False]}
>>> other_params = {'verbose': 0} # avoid trace information when training models
>>> metric = "mse" # Define metric
>>> if likelihood in ("bernoulli_probit", "bernoulli_logit"):
>>>   metric = "binary_logloss"
>>> # Note: can also use metric = "test_neg_log_likelihood". For more options, see https://github.com/fabsig/GPBoost/blob/master/docs/Parameters.rst#metric-parameters
>>> gp_model = gpb.GPModel(group_data=group, likelihood=likelihood)
>>> data_train = gpb.Dataset(data=X, label=y)
>>> # Run parameter optimization using random grid search and 4-fold CV
>>> # Note: deterministic grid search can be done by setting 'num_try_random=None'
>>> opt_params = gpb.grid_search_tune_parameters(param_grid=param_grid, params=other_params,
>>>                                              train_set=data_train, gp_model=gp_model,
>>>                                              num_try_random=100, nfold=5,
>>>                                              num_boost_round=1000, early_stopping_rounds=20,
>>>                                              verbose_eval=1, metric=metric, seed=4)
>>> print("Best parameters: " + str(opt_params['best_params']))
>>> print("Best number of iterations: " + str(opt_params['best_iter']))
>>> print("Best score: " + str(opt_params['best_score']))
>>>
>>> # Alternatively and faster: using manually defined validation data instead of cross-validation
>>> np.random.seed(10)
>>> permute_aux = np.random.permutation(n)
>>> train_tune_idx = permute_aux[0:int(0.8 * n)] # use 20% of the data as validation data
>>> valid_tune_idx = permute_aux[int(0.8 * n):n]
>>> folds = [(train_tune_idx, valid_tune_idx)]
>>> opt_params = gpb.grid_search_tune_parameters(param_grid=param_grid, params=other_params,
>>>                                              train_set=data_train, gp_model=gp_model,
>>>                                              num_try_random=100, folds=folds,
>>>                                              num_boost_round=1000, early_stopping_rounds=20,
>>>                                              verbose_eval=1, metric=metric, seed=4)

Authors:: Fabio Sigrist