Learning

DataCleaner

class MEDimage.learning.DataCleaner.DataCleaner(df_features: DataFrame, type: str = 'continuous')

Bases: object

Class that will clean features of the csv by removing features with too many missing values, too little variation, too many missing values per sample, too little variation per sample, and imputing missing values.

__call__(cleaning_dict: Dict, imputation_method: str = 'mean', missing_cutoff_ps: float = 0.25, missing_cutoff_pf: float = 0.1, cov_cutoff: float = 0.1) → DataFrame

Applies data cleaning to the features of type.

Parameters:

• cleaning_dict (dict) – Dictionary of cleaning parameters (missing cutoffs and coefficient of variation cutoffs etc.).

• var_of_type (list, optional) – List of variable names.

• imputation_method (str) – Method of imputation. Can be “mean”, “median”, “mode” or “random”. For “random” imputation, a seed can be provided by adding the seed value after the method name, for example “random42”.

• missing_cutoff_ps (float, optional) – maximal percentage cut-offs of missing features per sample.

• missing_cutoff_pf (float, optional) – maximal percentage cut-offs of missing samples per variable.

• cov_cutoff (float, optional) – minimal coefficient of variation cut-offs over samples per variable.

Returns:

Cleaned table of features.

Return type:

pd.DataFrame

__init__(df_features: DataFrame, type: str = 'continuous')

Constructor of the class DataCleaner

Parameters:

• df_features (pd.DataFrame) – Table of features.

• type (str) – Type of variable: “continuous”, “hcategorical” or “icategorical”. Defaults to “continuous”.

cut_off_missing_per_feature(var_of_type: List[str], missing_cutoff: float = 0.1) → List[str]

Removes features with more than missing_cutoff missing patients.

Parameters:

• var_of_type (list) – List of variable names.

• missing_cutoff (float) – maximal percentage cut-offs of missing patient samples per variable.

Returns:

List of variable names that were not flagged out.

Return type:

List[str]

cut_off_missing_per_sample(var_of_type: List[str], missing_cutoff: float = 0.25) → None

Removes observations/samples with more than missing_cutoff missing features.

Parameters:

• var_of_type (List[str]) – List of variable names.

• missing_cutoff (float) – Maximum percentage cut-offs of missing features per sample. Defaults to 25%.

Returns:

None.

cut_off_variation(var_of_type: List[str], cov_cutoff: float = 0.1) → List[str]

Removes features with a coefficient of variation (cov) less than cov_cutoff.

Parameters:

• var_of_type (list) – List of variable names.

• cov_cutoff (float) – minimal coefficient of variation cut-offs over samples per variable. Defaults to 10%.

Returns:

List of variable names that were not flagged out.

Return type:

List[str]

impute_missing(var_of_type: List[str], imputation_method: str = 'mean') → None

Imputes missing values of the features of type.

Parameters:

• var_of_type (list) – List of variable names.

• imputation_method (str) – Method of imputation. Can be “mean”, “median”, “mode” or “random”. For “random” imputation, a seed can be provided by adding the seed value after the method name, for example “random42”.

Returns:

None.

Desgin experiment

class MEDimage.learning.DesignExperiment.DesignExperiment(path_study: Path, path_settings: Path, experiment_label: str)

Bases: object

__init__(path_study: Path, path_settings: Path, experiment_label: str) → None

Constructor of the class DesignExperiment.

Parameters:

• path_study (Path) – Path to the main study folder where the outcomes, learning patients and holdout patients dictionaries are found.

• path_settings (Path) – Path to the settings folder.

• experiment_label (str) – String specifying the label to attach to a given learning experiment in “path_experiments”. This label will be attached to the ml__$experiments_label$.json file as well as the learn__$experiment_label$ folder. This label is used to keep track of different experiments with different settings (e.g. radiomics, scans, machine learning algorithms, etc.).

Returns:

None

create_experiment(ml: dict | None = None) → Dict

Create the machine learning experiment dictionary, organizes each test/split information in a seperate folder.

Parameters:

ml (dict, optional) – Dictionary containing all the machine learning settings. Defaults to None.

Returns:

Dictionary containing all the organized machine learning settings.

Return type:

Dict

fill_learner_dict(path_ml_options: Path) → Path

Fills the main expirement dictionary from the settings in the different json files. This main dictionary will hold all the settings for the data processing and learning experiment.

Parameters:

path_ml_options (Path) – Path to the ml_options json file for the experiment.

Returns:

Path to the learner object.

Return type:

Path

generate_experiment()

Generate the json files containing all the options the experiment. The json files will then be used in machine learning.

generate_learner_dict() → dict

Generates a dictionary containing all the settings for the learning experiment.

Returns:

Dictionary containing all the settings for the learning experiment.

Return type:

dict

Feature set reduction

class MEDimage.learning.FSR.FSR(method: str = 'fda')

Bases: object

__init__(method: str = 'fda') → None

Feature set reduction class constructor.

Parameters:

method (str) – Method of feature set reduction. Can be “FDA”, “LASSO” or “mRMR”.

apply_fda(ml: Dict, variable_table: List, outcome_table_binary: DataFrame, logging: bool = True, path_save_logging: Path | None = None) → List

Applies false discovery avoidance method.

Parameters:

• ml (dict) – Machine learning dictionary containing the learning options.

• variable_table (List) – Table of variables.

• outcome_table_binary (pd.DataFrame) – Table of binary outcomes.

• logging (bool, optional) – If True, will save a dict that tracks features selsected for each level. Defaults to True.

• path_save_logging (Path, optional) – Path to save the logging dict. Defaults to None.

Returns:

Table of variables after feature set reduction.

Return type:

List

apply_fda_balanced(ml: Dict, variable_table: List, outcome_table_binary: DataFrame) → List

Applies false discovery avoidance method but balances the number of features on each level.

Parameters:

• ml (dict) – Machine learning dictionary containing the learning options.

• variable_table (List) – Table of variables.

• outcome_table_binary (pd.DataFrame) – Table of binary outcomes.

• logging (bool, optional) – If True, will save a dict that tracks features selsected for each level. Defaults to True.

• path_save_logging (Path, optional) – Path to save the logging dict. Defaults to None.

Returns:

Table of variables after feature set reduction.

Return type:

List

apply_fda_one_space(ml: Dict, variable_table: List, outcome_table_binary: DataFrame, del_variants: bool = True, logging_dict: Dict | None = None) → List

Applies false discovery avoidance method.

Parameters:

• ml (dict) – Machine learning dictionary containing the learning options.

• variable_table (List) – Table of variables.

• outcome_table_binary (pd.DataFrame) – Table of binary outcomes.

• del_variants (bool, optional) – If True, will delete the variants of the same feature. Defaults to True.

Returns:

Table of variables after feature set reduction.

Return type:

List

apply_fsr(ml: Dict, variable_table: List, outcome_table_binary: DataFrame, path_save_logging: Path | None = None) → List

Applies feature set reduction method.

Parameters:

• ml (dict) – Machine learning dictionary containing the learning options.

• variable_table (List) – Table of variables.

• outcome_table_binary (pd.DataFrame) – Table of binary outcomes.

Returns:

Table of variables after feature set reduction.

Return type:

List

apply_random_fsr(ml: Dict, variable_table: List) → List

Applies random feature set reduction by choosing a random number of features.

Parameters:

• ml (dict) – Machine learning dictionary containing the learning options.

• variable_table (List) – Table of variables.

• outcome_table_binary (pd.DataFrame) – Table of binary outcomes.

Returns:

Table of variables after feature set reduction.

Return type:

List

apply_random_fsr_one_space(ml: Dict, variable_table: DataFrame) → List

Normalization

class MEDimage.learning.Normalization.Normalization(method: str = 'combat', variable_table: DataFrame | None = None, covariates_df: DataFrame | None = None, institutions: list | None = None)

Bases: object

__init__(method: str = 'combat', variable_table: DataFrame | None = None, covariates_df: DataFrame | None = None, institutions: list | None = None) → None

Constructor of the Normalization class.

apply_combat(variable_table: DataFrame, covariate_df: DataFrame | None = None, institutions: list | None = None) → DataFrame

Applys ComBat Normalization method to the data. More details this link.

Parameters:

• variable_table (pd.DataFrame) –

  pandas data frame on which Combat harmonization will be applied. This table is of size N X F (Observations X Features) and has the IDs as index. Requirements for this table

  • Does not contain NaNs.

  • No feature has 0 variance.

  • All variables are continuous (For example: Radiomics variables).

• covariate_df (pd.DataFrame, optional) – N X M pandas data frame, where N must equal the number of observations in variable_table. M is the number of covariates to include in the algorithm.

• institutions (list, optional) – List of size n_observations X 1 with the different institutions.

Returns:

variable_table after Combat harmonization.

Return type:

pd.DataFrame

Radiomics Learner

class MEDimage.learning.RadiomicsLearner.RadiomicsLearner(path_study: Path, path_settings: Path, experiment_label: str)

Bases: object

__init__(path_study: Path, path_settings: Path, experiment_label: str) → None

Constructor of the class DesignExperiment.

Parameters:

• path_study (Path) – Path to the main study folder where the outcomes, learning patients and holdout patients dictionaries are found.

• path_settings (Path) – Path to the settings folder.

• experiment_label (str) – String specifying the label to attach to a given learning experiment in “path_experiments”. This label will be attached to the ml__$experiments_label$.json file as well as the learn__$experiment_label$ folder. This label is used to keep track of different experiments with different settings (e.g. radiomics, scans, machine learning algorithms, etc.).

Returns:

None

get_hold_out_set_table(ml: Dict, var_id: str, patients_id: List)

Loads and pre-processes different radiomics tables then combines them to be used for hold-out testing.

Parameters:

• ml (Dict) – The machine learning dictionary containing the information of the machine learning test.

• var_id (str) – String specifying the ID of the radiomics variable in ml. –> Ex: var1

• patients_id (List) – List of patients of the hold-out set.

Returns:

Radiomics table for the hold-out set.

Return type:

pd.DataFrame

ml_run(path_ml: Path, holdout_test: bool = True, method: str = 'auto') → None

This function runs the machine learning test for the ceated experiment.

Parameters:

• path_ml (Path) – Path to the main dictionary containing info about the ml current experiment.

• holdout_test (bool, optional) – Boolean specifying if the hold-out test should be performed.

Returns:

None.

pre_process_radiomics_table(ml: Dict, var_id: str, outcome_table_binary: DataFrame, patients_train: list) → Tuple[DataFrame, DataFrame]

For the given variable, this function loads the corresponding radiomics tables and pre-processes them (cleaning, normalization and feature set reduction).

Note

Only patients of the training/learning set should be found in the given outcome table.

Parameters:

• ml (Dict) – The machine learning dictionary containing the information of the machine learning test (parameters, options, etc.).

• var_id (str) – String specifying the ID of the radiomics variable in ml. For example: ‘var1’.

• outcome_table_binary (pd.DataFrame) – outcome table with binary labels. This table may be used to pre-process some variables with the “FDA” feature set reduction algorithm.

• patients_train (list) – List of patients to use for training.

Returns:

Two dataframes of processed radiomics tables, one for training

and one for testing (no feature set reduction).

Return type:

Tuple[pd.DataFrame, pd.DataFrame]

pre_process_variables(ml: Dict, outcome_table_binary: DataFrame) → Tuple[DataFrame, DataFrame]

Loads and pre-processes different radiomics tables from different variable types found in the ml dict.

Note

only patients of the training/learning set should be found in this outcome table.

Parameters:

• ml (Dict) – The machine learning dictionary containing the information of the machine learning test.

• outcome_table_binary (pd.DataFrame) – outcome table with binary labels. This table may be used to pre-process some variables with the “FDA” feature set reduction algorithm.

Returns:

Two dict of processed radiomics tables, one dict for training and one for

testing (no feature set reduction).

Return type:

Tuple

predict_xgb(xgb_model: XGBClassifier, variable_table: DataFrame) → float

Computes the prediction of the XGBoost model for the given variable table.

Parameters:

• xgb_model (XGBClassifier) – XGBClassifier model.

• variable_table (pd.DataFrame) – Variable table for the prediction.

Returns:

Prediction of the XGBoost model.

Return type:

float

run_experiment(holdout_test: bool = True, method: str = 'pycaret') → None

Run the machine learning experiment for each split/run

Parameters:

• holdout_test (bool, optional) – Boolean specifying if the hold-out test should be performed.

• method (str, optional) – String specifying the method to use to train the XGBoost model. - “pycaret”: Use PyCaret to train the model (automatic). - “grid_search”: Grid search with cross-validation to find the best parameters. - “random_search”: Random search with cross-validation to find the best parameters.

Returns:

None

test_xgb_model(model_dict: Dict, variable_table: DataFrame, patient_list: List) → List

Tests the XGBoost model for the given dataset patients.

Parameters:

• model_dict (Dict) – Dictionary containing info about the trained XGBoost model.

• variable_table (pd.DataFrame) – Radiomics table for the test set (should not be normalized).

• patient_list (List) – List of patients to test.

Returns:

List the model response for the training and test sets.

Return type:

List

train_xgboost_model(var_table_train: DataFrame, outcome_table_binary_train: DataFrame, var_importance_threshold: float = 0.05, optimal_threshold: float | None = None, optimization_metric: str = 'MCC', method: str = 'pycaret', use_gpu: bool = True, seed: int | None = None) → Dict

Trains an XGBoost model for the given machine learning test.

Parameters:

• var_table_train (pd.DataFrame) – Radiomics table for the training/learning set.

• outcome_table_binary_train (pd.DataFrame) – Outcome table with binary labels for the training/learning set.

• var_importance_threshold (float) – Threshold for the variable importance. Variables with importance below this threshold will be removed from the model.

• optimal_threshold (float, optional) – Optimal threshold for the XGBoost model. If not given, it will be computed using the training set.

• optimization_metric (str, optional) – String specifying the metric to use to optimize the ml model.

• method (str, optional) – String specifying the method to use to train the XGBoost model. - “pycaret”: Use PyCaret to train the model (automatic). - “grid_search”: Grid search with cross-validation to find the best parameters. - “random_search”: Random search with cross-validation to find the best parameters.

• use_gpu (bool, optional) – Boolean specifying if the GPU should be used to train the model. Default is True.

• seed (int, optional) – Integer specifying the seed to use for the random number generator.

Returns:

Dictionary containing info about the trained XGBoost model.

Return type:

Dict

Results

class MEDimage.learning.Results.Results(model_dict: dict = {}, model_id: str = '')

Bases: object

A class to analyze the results of a given machine learning experiment, including the assessment of the model’s performance,

Parameters:

• model_dict (dict, optional) – Dictionary containing the model’s parameters. Defaults to {}.

• model_id (str, optional) – ID of the model. Defaults to “”.

model_dict

Dictionary containing the model’s parameters.

Type:

dict

model_id

ID of the model.

Type:

str

results_dict

Dictionary containing the results of the model’s performance.

Type:

dict

__init__(model_dict: dict = {}, model_id: str = '') → None

Constructor of the class Results

average_results(path_results: Path, save: bool = False) → None

Averages the results (AUC, BAC, Sensitivity and Specifity) of all the runs of the same experiment, for training, testing and holdout sets.

Parameters:

• path_results (Path) – path to the folder containing the results of the experiment.

• save (bool, optional) – If True, saves the results in the same folder as the model.

Returns:

None.

get_model_performance(response: list, outcome_table: DataFrame) → None

Calculates the performance of the model :param response: List of machine learning model predictions. :type response: list :param outcome_table: Outcome table with binary labels. :type outcome_table: pd.DataFrame

Returns:

Updates the run_results attribute.

Return type:

None

get_optimal_level(path_experiments: Path, experiment: str, modalities: List, levels: List, metric: str = 'AUC_mean', p_value_test: str = 'wilcoxon', aggregate: bool = False) → None

This function plots a heatmap of the metrics values for the performance of the models in the given experiment.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• modalities (List) – List of imaging modalities to include in the plot.

• levels (List) – List of radiomics levels to include in plot. For example: [‘morph’, ‘intensity’]. You can also use list of variants to plot the best variant for each level. For example: [[‘morph’, ‘morph5’], ‘intensity’].

• metric (str, optional) – Metric to plot. Defaults to ‘AUC_mean’.

• p_value_test (str, optional) –

  Method to use to calculate the p-value. Defaults to ‘wilcoxon’. Available options:

  • ’delong’: Delong test.

  • ’ttest’: T-test.

  • ’wilcoxon’: Wilcoxon signed rank test.

  • ’bengio’: Bengio and Nadeau corrected t-test.

• aggregate (bool, optional) – If True, aggregates the results of all the splits and computes one final p-value. Only valid for the Delong test when cross-validation is used. Defaults to False.

Returns:

None.

plot_fda_analysis_heatmap(path_experiments: Path, experiment: str, levels: List, modalities: List, title: str | None = None, save: bool = False) → None

This function plots a heatmap of the percentage of stable features and final features selected by FDA for a given experiment.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• levels (List) – List of radiomics levels to include in plot. For example: [‘morph’, ‘intensity’].

• modalities (List) – List of imaging modalities to include in the plot.

• title (str, optional) – Title and name used to save the plot. Defaults to None.

• save (bool, optional) – Whether to save the plot. Defaults to False.

Returns:

None.

plot_feature_analysis(path_experiments: Path, experiment: str, levels: List, modalities: List = [], title: str | None = None, save: bool = False) → None

This function plots a pie chart of the percentage of the final features used to train the model per radiomics level.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• levels (List) – List of radiomics levels to include in plot. For example: [‘morph’, ‘intensity’].

• modalities (List, optional) – List of imaging modalities to include in the plot. Defaults to [].

• title (str, optional) – Title and name used to save the plot. Defaults to None.

• save (bool, optional) – Whether to save the plot. Defaults to False.

Returns:

None.

plot_features_importance_histogram(path_experiments: Path, experiment: str, level: str, modalities: List, sort_option: str = 'importance', title: str | None = None, save: bool = True, figsize: tuple = (12, 12)) → None

Plots a histogram of the features importance for the given experiment.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• level (str) – Radiomics level to plot. For example: ‘morph’.

• modalities (List) – List of imaging modalities to use for the plot. A plot for each modality.

• sort_option (str, optional) – Option used to sort the features. Available options: - ‘importance’: Sorts the features by importance. - ‘times_selected’: Sorts the features by the number of times they were selected across the different splits. - ‘both’: Sorts the features by importance and then by the number of times they were selected.

• title (str, optional) – Title of the plot. Defaults to None.

• save (bool, optional) – Whether to save the plot. Defaults to True.

• figsize (tuple, optional) – Size of the figure. Defaults to (12, 12).

Returns:

None. Plots the figure or saves it.

plot_heatmap(path_experiments: Path, experiment: str, modalities: List, levels: List, metric: str = 'AUC_mean', stat_extra: list = [], plot_p_values: bool = True, p_value_test: str = 'wilcoxon', aggregate: bool = False, title: str | None = None, save: bool = False, figsize: tuple = (8, 8)) → None

This function plots a heatmap of the metrics values for the performance of the models in the given experiment.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• modalities (List) – List of imaging modalities to include in the plot.

• levels (List) – List of radiomics levels to include in plot. For example: [‘morph’, ‘intensity’]. You can also use list of variants to plot the best variant for each level. For example: [[‘morph’, ‘morph5’], ‘intensity’].

• metric (str, optional) – Metric to plot. Defaults to ‘AUC_mean’.

• stat_extra (list, optional) – List of extra statistics to include in the plot. Defaults to [].

• plot_p_values (bool, optional) – If True plots the p-value of the choosen test. Defaults to True.

• p_value_test (str, optional) –

  Method to use to calculate the p-value. Defaults to ‘wilcoxon’. Available options:

  • ’delong’: Delong test.

  • ’ttest’: T-test.

  • ’wilcoxon’: Wilcoxon signed rank test.

  • ’bengio’: Bengio and Nadeau corrected t-test.

• aggregate (bool, optional) – If True, aggregates the results of all the splits and computes one final p-value. Only valid for the Delong test when cross-validation is used. Defaults to False.

• extra_xlabels (List, optional) – List of extra x-axis labels. Defaults to [].

• title (str, optional) – Title of the plot. Defaults to None.

• save (bool, optional) – Whether to save the plot. Defaults to False.

• figsize (tuple, optional) – Size of the figure. Defaults to (8, 8).

Returns:

None.

plot_lf_level_tree(path_experiments: Path, experiment: str, level: str, modalities: list, initial_width: float = 4, lines_weight: float = 1, title: str | None = None, figsize: tuple = (12, 10)) → None

Plots a tree explaining the impact of features in the linear filters radiomics complexity level.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• level (List) – Radiomics complexity level to use for the plot.

• modalities (List, optional) – List of imaging modalities to include in the plot. Defaults to [].

• initial_width (float, optional) – Initial width of the lines. Defaults to 1. For aesthetic purposes.

• lines_weight (float, optional) – Weight applied to the lines of the tree. Defaults to 2. For aesthetic purposes.

• title (str, optional) – Title and name used to save the plot. Defaults to None.

• figsize (tuple, optional) – Size of the figure. Defaults to (20, 10).

Returns:

None.

plot_original_level_tree(path_experiments: Path, experiment: str, level: str, modalities: list, initial_width: float = 4, lines_weight: float = 1, title: str | None = None, figsize: tuple = (12, 10)) → None

Plots a tree explaining the impact of features in the original radiomics complexity level.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• level (List) – Radiomics complexity level to use for the plot.

• modalities (List, optional) – List of imaging modalities to include in the plot. Defaults to [].

• initial_width (float, optional) – Initial width of the lines. Defaults to 1. For aesthetic purposes.

• lines_weight (float, optional) – Weight applied to the lines of the tree. Defaults to 2. For aesthetic purposes.

• title (str, optional) – Title and name used to save the plot. Defaults to None.

• figsize (tuple, optional) – Size of the figure. Defaults to (20, 10).

Returns:

None.

plot_radiomics_starting_percentage(path_experiments: Path, experiment: str, levels: List, modalities: List, title: str | None = None, figsize: tuple = (15, 10), save: bool = False) → None

This function plots a pie chart of the percentage of features used in experiment per radiomics level.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• levels (List) – List of radiomics levels to include in the plot.

• modalities (List) – List of imaging modalities to include in the plot.

• title (str, optional) – Title and name used to save the plot. Defaults to None.

• figsize (tuple, optional) – Size of the figure. Defaults to (15, 10).

• save (bool, optional) – Whether to save the plot. Defaults to False.

Returns:

None.

plot_tf_level_tree(path_experiments: Path, experiment: str, level: str, modalities: list, initial_width: float = 4, lines_weight: float = 1, title: str | None = None, figsize: tuple = (12, 10)) → None

Plots a tree explaining the impact of features in the textural filters radiomics complexity level.

Parameters:

• path_experiments (Path) – Path to the folder containing the experiments.

• experiment (str) – Name of the experiment to plot. Will be used to find the results.

• level (List) – Radiomics complexity level to use for the plot.

• modalities (List, optional) – List of imaging modalities to include in the plot. Defaults to [].

• initial_width (float, optional) – Initial width of the lines. Defaults to 1. For aesthetic purposes.

• lines_weight (float, optional) – Weight applied to the lines of the tree. Defaults to 2. For aesthetic purposes.

• title (str, optional) – Title and name used to save the plot. Defaults to None.

• figsize (tuple, optional) – Size of the figure. Defaults to (20, 10).

Returns:

None.

to_json(response_train: list | None = None, response_test: list | None = None, response_holdout: list | None = None, patients_train: list | None = None, patients_test: list | None = None, patients_holdout: list | None = None) → dict

Creates a dictionary with the results of the model using the class attributes.

Parameters:

• response_train (list) – List of machine learning model predictions for the training set.

• response_test (list) – List of machine learning model predictions for the test set.

• patients_train (list) – List of patients in the training set.

• patients_test (list) – List of patients in the test set.

• patients_holdout (list) – List of patients in the holdout set.

Returns:

Dictionary with the the responses of the model and the patients used for training, testing and holdout.

Return type:

Dict