Assembled Utility Functions

assembled.ensemble_evaluation.evaluate_ensemble_on_metatask(metatask, technique, technique_args, technique_name, folds_to_run=None, use_validation_data_to_train_ensemble_techniques=False, meta_train_test_split_fraction=0.5, meta_train_test_split_random_state=0, pre_fit_base_models=False, base_models_with_names=False, label_encoder=False, preprocessor=None, output_dir_path=None, store_results='sequential', save_evaluation_metadata=False, oracle=False, probability_calibration='no', predict_method='predict', return_scores=None, verbose=False, isolate_ensemble_execution=False, refit=False, store_metadata_in_fake_base_model=False)[source]

Run an ensemble technique on all folds and return the results

The current implementation builds fake base models by default such that we can evaluate methods this way.

Parameters:

  • metatask (Metatask) – The metatask on which we want to evaluate the ensemble.

  • technique (sklearn estimator like object) – The Ensemble Technique (as a sklearn like fit/predict style) model.

  • technique_args (dict) – The arguments that shall be supplied to the ensemble

  • technique_name (str) – Name of the technique used to identify the technique later on in the saved results.

  • folds_to_run (List of integers, default=None) – If None, we run the ensemble on all folds. Otherwise, we run the ensemble on all specified folds.

  • use_validation_data_to_train_ensemble_techniques (bool, default=False) – Whether to use validation data to train the ensemble techniques (through the faked base models) and use the fold’s prediction data to evaluate the ensemble technique. If True, the metataks requires validation data. If False, test predictions of a fold are split into meta_train and meta_test subsets where the meta_train subset is used to train the ensemble techniques and meta_test to evaluate the techniques.

  • meta_train_test_split_fraction (float, default=0.5) – The fraction for the meta train/test split. Only used if

  • meta_train_test_split_random_state (int, default=0) – The randomness for the meta train/test split.

  • pre_fit_base_models (bool, default=False) – Whether or not the base models need to be fitted to be passed to the ensemble technique.

  • base_models_with_names (bool, default=False) – Whether or not the base models’ list should contain the model and its name.

  • label_encoder (bool, default=False) – Whether the ensemble technique expects that a label encoder is applied to the fake models. Often required for sklearn ensemble techniques.

  • preprocessor (sklearn-like transformer, default=None) – Function used to preprocess the data for later. called fit_transform on X_train and transform on X_test. If None, the default preprocessor is ued. The default preprocessor encodes categories as ordinal numbers and fills missing values.

  • output_dir_path (str, default=None) – Path to directory where the results of the folds shall be stored. If none, we do not store anything. We assume existing files are in the correct format for store_results=”sequential” and will create a new file if it does not exit. Here, no option to purge/delete existing files is given. This is must be done in an outer scope.

  • store_results ({"sequential", "parallel"}, default="sequential") –

    How to store the results of a fold’s evaluation.

    • ”sequential”: Store the results of all evaluated methods in one unique file.

    • ”parallel”: Store each fold’s results in different files such that they can later be merged easily.

  • save_evaluation_metadata (bool, default=False) – If true, save evaluation metadata in a (separate) file. Otherwise, do not save run metadata. Currently evaluation metadata include: {“fit_time”, “predict_time”} The metadata is stored for each fold.

  • oracle (bool, default=False) – Whether the ensemble technique is an oracle. If true, we pass and call the method differently.

  • probability_calibration ({"sigmoid", "isotonic", "auto", "no"}, default="no") –

    What type of probability calibration (see https://scikit-learn.org/stable/modules/calibration.html) shall be applied to the base models:

    • ”sigmoid”: Use CalibratedClassifierCV with method=”sigmoid”

    • ”isotonic”: Use CalibratedClassifierCV with method=”isotonic”

    • ”auto”: Determine which method to use for CalibratedClassifierCV depending on the number of instances.

    • ”no”: Do not use probability calibration.

    If pre_fit_base_models is False, CalibratedClassifierCV is employed with ensemble=”False” to simulate cross_val_predictions by our Faked Base Models. If pre_fit_base_models is True, CalibratedClassifierCV is employed with cv=”prefit” beforehand such that we “replace” the base models with calibrated base models.

  • predict_method (str in {"predict", "predict_proba"}, default="predict") – Predict method used. For regression only “predict” works. For classification, predict_proba will return prediction probabilities (a.k.a. confidences).

  • return_scores (Callable, default=None) – If the evaluation shall return the scores for each fold. If not None, a metric function is expected.

  • verbose (bool, default=False) – If True, evaluation status information are logged.

  • isolate_ensemble_execution (bool, default=False) – If True, we isolate the execution of the ensemble in its own subprocess. This avoids problems with memory leakage or other problems from implementations of the ensemble. FIXME: !WARNING! Only works on Linux currently;

  • refit (bool, default=False) – Set to true if the base models have been re-fitted before predicting for test data.

  • store_metadata_in_fake_base_model (bool, default=False) – If true, we store metadata about a base model in the FakedBaseModel that is initialized. To do so, the metatasks must store metadata about a base model in a dictionary called “predictor_descriptions” where the name of a predictor maps to the desired metadata.

assembled.benchmaker.load_benchmark_details(path_to_benchmark_details)[source]
assembled.benchmaker.rebuild_benchmark(output_path_benchmark_metatask, id_to_dataset_load_function=None, openml_data=False)[source]

Rebuild the benchmark data

Parameters:

  • output_path_benchmark_metatask (str) – Path to the directory in which the benchmark_details.json and bmer_timp.zip is stored.

  • id_to_dataset_load_function (dict, default=None) – A dictionary with the keys being metatask IDs and the values being function that can be called to return the dataset.

  • openml_data (bool, default=False) – If true, id_to_dataset_load_function is not needed and will be build by default assuming that all datasets are from OpenML. In other words, if true, we fill the datasets automatically and the user does not need to worry about it. Requires Assembled-OpenML and its requirements to be installed as well (which is the case by default)