Examples ======== Stacked classifiers (naive protocol) ------------------------------------ Similar to the the example in the quick-start guide, (a naive) stacks of classifiers (or regressors) can be built like shown below. Note that you can specify the function the step should use for computation, in this case ``compute_func='predict_proba'`` to use the label probabilities as the features of the meta-classifier. .. code-block:: python x = Input() y_t = Input() y_p1 = LogisticRegression()(x, y_t, compute_func="predict_proba") y_p2 = RandomForestClassifier()(x, y_t, compute_func="predict_proba") # predict_proba returns arrays whose columns sum to one, so we drop one column drop_first_col = Lambda(lambda array: array[:, 1:]) y_p1 = drop_first_col(y_p1) y_p2 = drop_first_col(y_p2) ensemble_features = ColumnStack()([y_p1, y_p2]) y_p = ExtraTreesClassifier()(ensemble_features, y_t) model = Model(x, y_p, y_t) .. container:: toggle .. literalinclude:: ../../examples/stacked_classifiers_naive.py Stacked classifiers (standard protocol) --------------------------------------- In the naive stack above, each classifier in the 1st level will calculate the predictions for the 2nd level using the same data it used for fitting its parameters. This is prone to overfitting as the 2nd level classifier will tend to give more weight to an overfit classifier in the 1st level. To avoid this, the standard protocol recommends that, during fit, the 1st level classifiers are still trained on the original data, but instead they provide out-of-fold (OOF) predictions to the 2nd level classifier. To achieve this special behavior, we leverage the ``fit_compute_func`` API: we define a ``fit_predict`` method that does the fitting and the OOF predictions, and add it as a method of the 1st level classifiers (``LogisticRegression`` and ``RandomForestClassifier``, in the example) when making the steps. **baikal** will then detect and use this method during fit. .. code-block:: python from sklearn.model_selection import cross_val_predict def fit_predict(self, X, y): self.fit(X, y) return cross_val_predict(self, X, y, method="predict_proba") attr_dict = {"fit_predict": fit_predict} # 1st level classifiers LogisticRegression = make_step(sklearn.linear_model.LogisticRegression, attr_dict) RandomForestClassifier = make_step(sklearn.ensemble.RandomForestClassifier, attr_dict) # 2nd level classifier ExtraTreesClassifier = make_step(sklearn.ensemble.ExtraTreesClassifier) The rest of the stack is built exactly the same as in the naive example. .. container:: toggle .. literalinclude:: ../../examples/stacked_classifiers_standard.py Classifier chain ---------------- .. _ClassifierChainWikiURL: https://en.wikipedia.org/wiki/Classifier_chains .. _ClassifierChainURL: https://scikit-learn.org/stable/modules/generated/sklearn.multioutput.ClassifierChain.html#sklearn.multioutput.ClassifierChain .. _RegressorChainURL: https://scikit-learn.org/stable/modules/generated/sklearn.multioutput.RegressorChain.html#sklearn.multioutput.RegressorChain The API also lends itself for more interesting configurations, such as that of `classifier chains `__. By leveraging the API and Python's own control flow, a classifier chain model can be built as follows: .. code-block:: python x = Input() y_t = Input() order = list(range(n_targets)) random.shuffle(order) squeeze = Lambda(np.squeeze, axis=1) ys_t = Split(n_targets, axis=1)(y_t) ys_p = [] for j, k in enumerate(order): x_stacked = ColumnStack()([x, *ys_p[:j]]) ys_t[k] = squeeze(ys_t[k]) ys_p.append(LogisticRegression()(x_stacked, ys_t[k])) ys_p = [ys_p[order.index(j)] for j in range(n_targets)] y_p = ColumnStack()(ys_p) model = Model(x, y_p, y_t) Sure, scikit-learn already does have `ClassifierChain `__ and `RegressorChain `__ classes for this. But with **baikal** you could, for example, mix classifiers and regressors to predict multilabels that include both categorical and continuous labels. .. container:: toggle .. literalinclude:: ../../examples/classifier_chain.py Transformed target ------------------ You can also call steps on the targets to apply transformations on them. Note that by making the transformer a shared step, you can re-use learned parameters to apply the inverse transform later in the pipeline. .. code-block:: python transformer = QuantileTransformer(n_quantiles=300, output_distribution="normal") x = Input() y_t = Input() # QuantileTransformer requires an explicit feature dimension, hence the Lambda step y_t_trans = Lambda(np.reshape, newshape=(-1, 1))(y_t) y_t_trans = transformer(y_t_trans) y_p_trans = RidgeCV()(x, y_t_trans) y_p = transformer(y_p_trans, compute_func="inverse_transform", trainable=False) # Note that transformer is a shared step since it was called twice model = Model(x, y_p, y_t) .. container:: toggle .. literalinclude:: ../../examples/transformed_target.py Tune a model with ``GridSearchCV`` ---------------------------------- Below is an example showing how to use the scikit-learn wrapper to tune the parameters of a **baikal** model using ``GridSearchCV``. .. container:: toggle .. literalinclude:: ../../examples/gridsearchcv_sklearn_wrapper.py