Supported Regression models¶

The package currently supports various scikit-learn objects. It also supports Gradient Boosting Regression from XGBoost and has limited support for Keras and PyTorch. Only sequential neural networks with ReLU activation function are currently supported. In Mixed Integer Formulations, we briefly outline the formulations used for the various regression models.

The versions of those packages tested with the current version (1.5.6dev0) are listed in the table Supported packages with version 1.5.6dev0.

Scikit-learn¶

The following tables list the name of the models supported, the name of the corresponding object in the Python framework, and the function that can be used to insert it in a Gurobi model.

Supported regression models of scikit-learn¶
Regression Model	Scikit-learn object	Function to insert
Ordinary Least Square	`LinearRegression`	`add_linear_regression_constr`
Partial Least Square	`PLSRegression`	`add_pls_regression_constr`
Logistic Regression [1]	`LogisticRegression`	`add_logistic_regression_constr`
Neural-network [2]	`MLPRegressor`	`add_mlp_regressor_constr`
Decision tree	`DecisionTreeRegressor`	`add_decision_tree_regressor_constr`
Gradient boosting	`GradientBoostingRegressor`	`add_gradient_boosting_regressor_constr`
Random Forest	`RandomForestRegressor`	`add_random_forest_regressor_constr`

Transformers in scikit-learn¶
Transformer	Scikit-learn object	Function to insert
StandardScaler	`StandardScaler`	`add_standard_scaler_constr`
Pipeline	`Pipeline`	`add_pipeline_constr`
PolynomialFeatures [3]	`PolynomialFeatures`	`add_polynomial_features_constr`
ColumnTransformer	`ColumnTransformer`	`add_column_transformer_constr`

Keras¶

Keras neural networks are generated either using the functional API, subclassing model or the Sequential class.

They can be formulated in a Gurobi model with the function add_keras_constr.

Currently, only two types of layers are supported:

Dense layers (possibly with relu activation),

ReLU layers with default settings.

PyTorch¶

In PyTorch, only torch.nn.Sequential objects are supported.

They can be formulated in a Gurobi model with the function add_sequential_constr.

Currently, only two types of layers are supported:

Linear layers,

ReLU layers.

ONNX¶

ONNX models for sequential multi-layer perceptrons are supported when composed of Gemm (dense) operators and Relu activations.

They can be formulated in a Gurobi model with the function add_onnx_constr.

Currently, only the following are supported:

Sequential MLPs with Gemm nodes (PyTorch/sklearn style) with default attributes (alpha=1, beta=1) and optional transB attribute,

Sequential MLPs with MatMul + Add sequences (TensorFlow/tf2onnx style),

Relu activations.

Models must be strictly sequential with no skip connections or residual blocks. Each tensor should be consumed by at most one node. Add nodes are only supported for bias addition in the MatMul + Add pattern, not for combining branches.

XGBoost¶

XGBoost’s xgboost.Booster can be formulated in a Gurobi model with the function add_xgboost_regressor_constr. The scikit-learn wrapper xgboost.XGBRegressor can be formulated using add_xgbregressor_constr.

Currently only “gbtree” boosters are supported. Note that all options of “gbtree” may not be supported. In particular, those that may result in a different prediction function. For the “objective” option, we only support the default reg:squarederror and reg:logistic. If you encounter an issue don’t hesitate to contact us.

LightGBM¶

LightGBM’s lightgbm.Booster can be formulated in a Gurobi model with the function add_lgbm_booster_constr. The scikit-learn wrapper lightgbm.sklearn.LGBMRegressor can be formulated using add_lgbmregressor_constr.

Note that all options of LightGBM may not be supported. In particular, those that may result in a different prediction function. If you encounter an issue don’t hesitate to contact us.

Footnotes