{
    "file_diffs": [
        {
            "old_file_content": "# __new__ methods have different arguments\n# pylint: disable=arguments-differ\nfrom warnings import warn\nfrom collections import namedtuple\nfrom itertools import chain\nfrom time import time\n\nimport numpy as np\n\nimport sklearn.model_selection as skl\n\nfrom Orange.data import Domain, ContinuousVariable, DiscreteVariable\nfrom Orange.data.util import get_unique_names\n\n__all__ = [\"Results\", \"CrossValidation\", \"LeaveOneOut\", \"TestOnTrainingData\",\n           \"ShuffleSplit\", \"TestOnTestData\", \"sample\", \"CrossValidationFeature\"]\n\n_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',\n                                       'failed', 'n_values', 'values',\n                                       'probs', 'train_time', 'test_time'))\n\n\ndef _identity(x):\n    return x\n\n\ndef _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed = None, None, None, False\n    train_time, test_time = None, None\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0 = time()\n        # testing\n        class_var = train_data.domain.class_var\n        if class_var and class_var.is_discrete:\n            predicted, probs = model(test_data, model.ValueProbs)\n        else:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)\n\n\nclass Results:\n    \"\"\"\n    Class for storing predictions in model testing.\n\n    Attributes:\n        data (Optional[Table]):  Data used for testing.\n\n        models (Optional[List[Model]]): A list of induced models.\n\n        row_indices (np.ndarray): Indices of rows in `data` that were used in\n            testing, stored as a numpy vector of length `nrows`.\n            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer\n            to the target value of instance, that is, the i-th test instance\n            is `data[row_indices[i]]`, its actual class is `actual[i]`, and\n            the prediction by m-th method is `predicted[m, i]`.\n\n        nrows (int): The number of test instances (including duplicates);\n            `nrows` equals the length of `row_indices` and `actual`, and the\n            second dimension of `predicted` and `probabilities`.\n\n        actual (np.ndarray): true values of target variable in a vector of\n            length `nrows`.\n\n        predicted (np.ndarray): predicted values of target variable in an array\n            of shape (number-of-methods, `nrows`)\n\n        probabilities (Optional[np.ndarray]): predicted probabilities\n            (for discrete target variables) in an array of shape\n            (number-of-methods, `nrows`, number-of-classes)\n\n        folds (List[Slice or List[int]]): a list of indices (or slice objects)\n            corresponding to testing data subsets, that is,\n            `row_indices[folds[i]]` contains row indices used in fold i, so\n            `data[row_indices[folds[i]]]` is the corresponding testing data\n\n        train_time (np.ndarray): training times of batches\n\n        test_time (np.ndarray): testing times of batches\n    \"\"\"\n    def __init__(self, data=None, *,\n                 nmethods=None, nrows=None, nclasses=None,\n                 domain=None,\n                 row_indices=None, folds=None, score_by_folds=True,\n                 learners=None, models=None, failed=None,\n                 actual=None, predicted=None, probabilities=None,\n                 store_data=None, store_models=None,\n                 train_time=None, test_time=None):\n        \"\"\"\n        Construct an instance.\n\n        The constructor stores the given data, and creates empty arrays\n        `actual`, `predicted` and `probabilities` if ther are not given but\n        sufficient data is provided to deduct their shapes.\n\n        The function\n\n        - set any attributes specified directly through arguments.\n        - infers the number of methods, rows and classes from other data\n           and/or check their overall consistency.\n        - Prepare empty arrays `actual`, `predicted`, `probabilities` and\n          `failed` if the are not given. If not enough data is available,\n          the corresponding arrays are `None`.\n\n        Args:\n            data (Orange.data.Table): stored data from which test was sampled\n            nmethods (int): number of methods; can be inferred (or must match)\n                the size of `learners`, `models`, `failed`, `predicted` and\n                `probabilities`\n            nrows (int): number of data instances; can be inferred (or must\n                match) `data`, `row_indices`, `actual`, `predicted` and\n                `probabilities`\n            nclasses (int): number of class values (`None` if continuous); can\n                be inferred (or must match) from `domain.class_var` or\n                `probabilities`\n            domain (Orange.data.Domain): data domain; can be inferred (or must)\n                match `data.domain`\n            row_indices (np.ndarray): see class documentation\n            folds (np.ndarray): see class documentation\n            score_by_folds (np.ndarray): see class documentation\n            learners (np.ndarray): see class documentation\n            models (np.ndarray): see class documentation\n            failed (list of str): see class documentation\n            actual (np.ndarray): see class documentation\n            predicted (np.ndarray): see class documentation\n            probabilities (np.ndarray): see class documentation\n            store_data (bool): ignored; kept for backward compatibility\n            store_models (bool): ignored; kept for backward compatibility\n        \"\"\"\n\n        # Set given data directly from arguments\n        self.data = data\n        self.domain = domain\n\n        self.row_indices = row_indices\n        self.folds = folds\n        self.score_by_folds = score_by_folds\n\n        self.learners = learners\n        self.models = models\n\n        self.actual = actual\n        self.predicted = predicted\n        self.probabilities = probabilities\n        self.failed = failed\n\n        self.train_time = train_time\n        self.test_time = test_time\n\n        # Guess the rest -- or check for ambguities\n        def set_or_raise(value, exp_values, msg):\n            for exp_value in exp_values:\n                if exp_value is False:\n                    continue\n                if value is None:\n                    value = exp_value\n                elif value != exp_value:\n                    raise ValueError(msg)\n            return value\n        domain = self.domain = set_or_raise(\n            domain, [data is not None and data.domain],\n            \"mismatching domain\")\n        self.nrows = nrows = set_or_raise(\n            nrows, [actual is not None and len(actual),\n                    row_indices is not None and len(row_indices),\n                    predicted is not None and predicted.shape[1],\n                    probabilities is not None and probabilities.shape[1]],\n            \"mismatching number of rows\")\n        if domain is not None and domain.has_continuous_class:\n            if nclasses is not None:\n                raise ValueError(\n                    \"regression results cannot have non-None 'nclasses'\")\n            if probabilities is not None:\n                raise ValueError(\n                    \"regression results cannot have 'probabilities'\")\n        nclasses = set_or_raise(\n            nclasses, [domain is not None and domain.has_discrete_class and\n                       len(domain.class_var.values),\n                       probabilities is not None and probabilities.shape[2]],\n            \"mismatching number of class values\")\n        nmethods = set_or_raise(\n            nmethods, [learners is not None and len(learners),\n                       models is not None and models.shape[1],\n                       failed is not None and len(failed),\n                       predicted is not None and predicted.shape[0],\n                       probabilities is not None and probabilities.shape[0]],\n            \"mismatching number of methods\")\n\n        # Prepare empty arrays\n        if actual is None \\\n                and nrows is not None:\n            self.actual = np.empty(nrows)\n\n        if predicted is None \\\n                and nmethods is not None and nrows is not None:\n            self.predicted = np.empty((nmethods, nrows))\n\n        if probabilities is None \\\n                and nmethods is not None and nrows is not None \\\n                and nclasses is not None:\n            self.probabilities = \\\n                np.empty((nmethods, nrows, nclasses))\n\n        if failed is None \\\n                and nmethods is not None:\n            self.failed = [False] * nmethods\n\n    def get_fold(self, fold):\n        results = Results()\n        results.data = self.data\n\n        if self.folds is None:\n            raise ValueError(\"This 'Results' instance does not have folds.\")\n\n        if self.models is not None:\n            results.models = self.models[fold]\n\n        results.row_indices = self.row_indices[self.folds[fold]]\n        results.actual = self.actual[self.folds[fold]]\n        results.predicted = self.predicted[:, self.folds[fold]]\n        results.domain = self.domain\n\n        if self.probabilities is not None:\n            results.probabilities = self.probabilities[:, self.folds[fold]]\n\n        return results\n\n    def get_augmented_data(self, model_names,\n                           include_attrs=True, include_predictions=True,\n                           include_probabilities=True):\n        \"\"\"\n        Return the test data table augmented with meta attributes containing\n        predictions, probabilities (if the task is classification) and fold\n        indices.\n\n        Args:\n            model_names (list of str): names of models\n            include_attrs (bool):\n                if set to `False`, original attributes are removed\n            include_predictions (bool):\n                if set to `False`, predictions are not added\n            include_probabilities (bool):\n                if set to `False`, probabilities are not added\n\n        Returns:\n            augmented_data (Orange.data.Table):\n                data augmented with predictions, probabilities and fold indices\n\n        \"\"\"\n        assert self.predicted.shape[0] == len(model_names)\n\n        data = self.data[self.row_indices]\n        domain = data.domain\n        class_var = domain.class_var\n        classification = class_var and class_var.is_discrete\n\n        new_meta_attr = []\n        new_meta_vals = np.empty((len(data), 0))\n        names = [var.name for var in chain(domain.attributes,\n                                           domain.metas,\n                                           domain.class_vars)]\n\n        if classification:\n            # predictions\n            if include_predictions:\n                uniq_new, names = self.create_unique_vars(names, model_names, class_var.values)\n                new_meta_attr += uniq_new\n                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n            # probabilities\n            if include_probabilities:\n                proposed = [f\"{name} ({value})\" for name in model_names for value in class_var.values]\n\n                uniq_new, names = self.create_unique_vars(names, proposed)\n                new_meta_attr += uniq_new\n\n                for i in self.probabilities:\n                    new_meta_vals = np.hstack((new_meta_vals, i))\n\n        elif include_predictions:\n            # regression\n            uniq_new, names = self.create_unique_vars(names, model_names)\n            new_meta_attr += uniq_new\n            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n        # add fold info\n        if self.folds is not None:\n            values = [str(i + 1) for i in range(len(self.folds))]\n            uniq_new, names = self.create_unique_vars(names, [\"Fold\"], values)\n            new_meta_attr += uniq_new\n            fold = np.empty((len(data), 1))\n            for i, s in enumerate(self.folds):\n                fold[s, 0] = i\n            new_meta_vals = np.hstack((new_meta_vals, fold))\n\n        # append new columns to meta attributes\n        new_meta_attr = list(data.domain.metas) + new_meta_attr\n        new_meta_vals = np.hstack((data.metas, new_meta_vals))\n\n        attrs = data.domain.attributes if include_attrs else []\n        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)\n        predictions = data.transform(domain)\n        with predictions.unlocked(predictions.metas):\n            predictions.metas = new_meta_vals\n        predictions.name = data.name\n        return predictions\n\n    def create_unique_vars(self, names, proposed_names, values=()):\n        unique_vars = []\n        for proposed in proposed_names:\n            uniq = get_unique_names(names, proposed)\n            if values:\n                unique_vars.append(DiscreteVariable(uniq, values))\n            else:\n                unique_vars.append(ContinuousVariable(uniq))\n            names.append(uniq)\n        return unique_vars, names\n\n    def split_by_model(self):\n        \"\"\"\n        Split evaluation results by models.\n\n        The method generates instances of `Results` containing data for single\n        models\n        \"\"\"\n        data = self.data\n        nmethods = len(self.predicted)\n        for i in range(nmethods):\n            res = Results()\n            res.data = data\n            res.domain = self.domain\n            res.learners = [self.learners[i]]\n            res.row_indices = self.row_indices\n            res.actual = self.actual\n            res.folds = self.folds\n            res.score_by_folds = self.score_by_folds\n            res.test_time = self.test_time[i]\n            res.train_time = self.train_time[i]\n\n            res.predicted = self.predicted[(i,), :]\n            if getattr(self, \"probabilities\", None) is not None:\n                res.probabilities = self.probabilities[(i,), :, :]\n\n            if self.models is not None:\n                res.models = self.models[:, i:i + 1]\n\n            res.failed = [self.failed[i]]\n            yield res\n\n\nclass Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        return folds, row_indices, data[row_indices].Y\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs\n\n\nclass CrossValidation(Validation):\n    \"\"\"\n    K-fold cross validation\n\n    Attributes:\n        k (int): number of folds (default: 10)\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n            If `True` but the class sizes don't allow it, it uses non-stratified\n            validataion and adds a list `warning` with a warning message(s) to\n            the `Result`.\n    \"\"\"\n    # TODO: list `warning` contains just repetitions of the same message\n    #       replace with a flag in `Results`?\n    def __init__(self, k=10, stratified=True, random_state=0,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.k = k\n        self.stratified = stratified\n        self.random_state = random_state\n        self.warnings = [] if warnings is None else warnings\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            try:\n                splitter = skl.StratifiedKFold(\n                    self.k, shuffle=True, random_state=self.random_state\n                )\n                splitter.get_n_splits(data.X, data.Y)\n                return list(splitter.split(data.X, data.Y))\n            except ValueError:\n                self.warnings.append(\"Using non-stratified sampling.\")\n\n        splitter = skl.KFold(\n            self.k, shuffle=True, random_state=self.random_state)\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass CrossValidationFeature(Validation):\n    \"\"\"\n    Cross validation with folds according to values of a feature.\n\n    Attributes:\n        feature (Orange.data.Variable): the feature defining the folds\n    \"\"\"\n    def __init__(self, feature=None,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.feature = feature\n\n    def get_indices(self, data):\n        data = data.transform(Domain([self.feature], None))\n        values = data[:, self.feature].X\n        indices = []\n        for v in range(len(self.feature.values)):\n            test_index = np.where(values == v)[0]\n            train_index = np.where((values != v) & (~np.isnan(values)))[0]\n            if test_index.size and train_index.size:\n                indices.append((train_index, test_index))\n        if not indices:\n            raise ValueError(\n                f\"'{self.feature.name}' does not have at least two distinct \"\n                \"values on the data\")\n        return indices\n\n\nclass LeaveOneOut(Validation):\n    \"\"\"Leave-one-out testing\"\"\"\n    score_by_folds = False\n\n    def get_indices(self, data):\n        splitter = skl.LeaveOneOut()\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n    @staticmethod\n    def prepare_arrays(data, indices):\n        # sped up version of super().prepare_arrays(data)\n        row_indices = np.arange(len(data))\n        return row_indices, row_indices, data.Y.flatten()\n\n\nclass ShuffleSplit(Validation):\n    \"\"\"\n    Test by repeated random sampling\n\n    Attributes:\n        n_resamples (int): number of repetitions\n        test_size (float, int, None):\n            If float, should be between 0.0 and 1.0 and represent the proportion\n            of the dataset to include in the test split. If int, represents the\n            absolute number of test samples. If None, the value is set to the\n            complement of the train size. By default, the value is set to 0.1.\n            The default will change in version 0.21. It will remain 0.1 only\n            if ``train_size`` is unspecified, otherwise it will complement\n            the specified ``train_size``.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        train_size : float, int, or None, default is None\n            If float, should be between 0.0 and 1.0 and represent the\n            proportion of the dataset to include in the train split. If\n            int, represents the absolute number of train samples. If None,\n            the value is automatically set to the complement of the test size.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n\n    \"\"\"\n    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,\n                 stratified=True, random_state=0,\n                 store_data=False, store_models=False):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.n_resamples = n_resamples\n        self.train_size = train_size\n        self.test_size = test_size\n        self.stratified = stratified\n        self.random_state = random_state\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            splitter = skl.StratifiedShuffleSplit(\n                n_splits=self.n_resamples, train_size=self.train_size,\n                test_size=self.test_size, random_state=self.random_state\n            )\n            splitter.get_n_splits(data.X, data.Y)\n            return list(splitter.split(data.X, data.Y))\n\n        splitter = skl.ShuffleSplit(\n            n_splits=self.n_resamples, train_size=self.train_size,\n            test_size=self.test_size, random_state=self.random_state\n        )\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass TestOnTestData(Validation):\n    \"\"\"\n    Test on separately provided test data\n\n    Note that the class has a different signature for `__call__`.\n    \"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n\n    def __new__(cls, data=None, test_data=None, learners=None,\n                preprocessor=None, **kwargs):\n        if \"train_data\" in kwargs:\n            if data is None:\n                data = kwargs.pop(\"train_data\")\n            else:\n                raise ValueError(\n                    \"argument 'data' is given twice (once as 'train_data')\")\n        return super().__new__(\n            cls,\n            data=data, learners=learners, preprocessor=preprocessor,\n            test_data=test_data, **kwargs)\n\n    def __call__(self, data, test_data, learners, preprocessor=None,\n                 *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): training data\n            test_data (Orange.data.Table): test_data\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n\n        train_data = preprocessor(data)\n        part_results = []\n        for (learner_i, learner) in enumerate(learners):\n            part_results.append(\n                _mp_worker(0, train_data, test_data, learner_i, learner,\n                           self.store_models))\n            callback((learner_i + 1) / len(learners))\n        callback(1)\n\n        results = Results(\n            data=test_data if self.store_data else None,\n            domain=test_data.domain,\n            nrows=len(test_data), learners=learners,\n            row_indices=np.arange(len(test_data)),\n            folds=(Ellipsis, ),\n            actual=test_data.Y,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (1, len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n\nclass TestOnTrainingData(TestOnTestData):\n    \"\"\"Test on training data\"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n    # signature is such as on the base class, pylint: disable=signature-differs\n    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):\n        return super().__new__(\n            cls,\n            data, test_data=data, learners=learners, preprocessor=preprocessor,\n            **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None,\n                 **kwargs):\n        kwargs.setdefault(\"test_data\", data)\n        # if kwargs contains anything besides test_data, this will be detected\n        # (and complained about) by super().__call__\n        return super().__call__(\n            data=data, learners=learners, preprocessor=preprocessor,\n            callback=callback, **kwargs)\n\n\ndef sample(table, n=0.7, stratified=False, replace=False,\n           random_state=None):\n    \"\"\"\n    Samples data instances from a data table. Returns the sample and\n    a dataset from input data table that are not in the sample. Also\n    uses several sampling functions from\n    `scikit-learn <http://scikit-learn.org>`_.\n\n    table : data table\n        A data table from which to sample.\n\n    n : float, int (default = 0.7)\n        If float, should be between 0.0 and 1.0 and represents\n        the proportion of data instances in the resulting sample. If\n        int, n is the number of data instances in the resulting sample.\n\n    stratified : bool, optional (default = False)\n        If true, sampling will try to consider class values and\n        match distribution of class values\n        in train and test subsets.\n\n    replace : bool, optional (default = False)\n        sample with replacement\n\n    random_state : int or RandomState\n        Pseudo-random number generator state used for random sampling.\n    \"\"\"\n\n    if isinstance(n, float):\n        n = int(n * len(table))\n\n    if replace:\n        if random_state is None:\n            rgen = np.random\n        else:\n            rgen = np.random.mtrand.RandomState(random_state)\n        a_sample = rgen.randint(0, len(table), n)\n        o = np.ones(len(table))\n        o[a_sample] = 0\n        others = np.nonzero(o)[0]\n        return table[a_sample], table[others]\n\n    n = len(table) - n\n    if stratified and table.domain.has_discrete_class:\n        test_size = max(len(table.domain.class_var.values), n)\n        splitter = skl.StratifiedShuffleSplit(\n            n_splits=1, test_size=test_size, train_size=len(table) - test_size,\n            random_state=random_state)\n        splitter.get_n_splits(table.X, table.Y)\n        ind = splitter.split(table.X, table.Y)\n    else:\n        splitter = skl.ShuffleSplit(\n            n_splits=1, test_size=n, random_state=random_state)\n        splitter.get_n_splits(table)\n        ind = splitter.split(table)\n    ind = next(ind)\n    return table[ind[0]], table[ind[1]]\n",
            "new_file_content": "# __new__ methods have different arguments\n# pylint: disable=arguments-differ\nfrom warnings import warn\nfrom collections import namedtuple\nfrom itertools import chain\nfrom time import time\n\nimport numpy as np\n\nimport sklearn.model_selection as skl\n\nfrom Orange.data import Domain, ContinuousVariable, DiscreteVariable\nfrom Orange.data.util import get_unique_names\n\n__all__ = [\"Results\", \"CrossValidation\", \"LeaveOneOut\", \"TestOnTrainingData\",\n           \"ShuffleSplit\", \"TestOnTestData\", \"sample\", \"CrossValidationFeature\"]\n\n_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',\n                                       'failed', 'n_values', 'values',\n                                       'probs', 'train_time', 'test_time'))\n\n\ndef _identity(x):\n    return x\n\n\ndef _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed = None, None, None, False\n    train_time, test_time = None, None\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0 = time()\n        # testing\n        class_var = train_data.domain.class_var\n        if class_var and class_var.is_discrete:\n            predicted, probs = model(test_data, model.ValueProbs)\n        else:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)\n\n\nclass Results:\n    \"\"\"\n    Class for storing predictions in model testing.\n\n    Attributes:\n        data (Optional[Table]):  Data used for testing.\n\n        models (Optional[List[Model]]): A list of induced models.\n\n        row_indices (np.ndarray): Indices of rows in `data` that were used in\n            testing, stored as a numpy vector of length `nrows`.\n            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer\n            to the target value of instance, that is, the i-th test instance\n            is `data[row_indices[i]]`, its actual class is `actual[i]`, and\n            the prediction by m-th method is `predicted[m, i]`.\n\n        nrows (int): The number of test instances (including duplicates);\n            `nrows` equals the length of `row_indices` and `actual`, and the\n            second dimension of `predicted` and `probabilities`.\n\n        actual (np.ndarray): true values of target variable in a vector of\n            length `nrows`.\n\n        predicted (np.ndarray): predicted values of target variable in an array\n            of shape (number-of-methods, `nrows`)\n\n        probabilities (Optional[np.ndarray]): predicted probabilities\n            (for discrete target variables) in an array of shape\n            (number-of-methods, `nrows`, number-of-classes)\n\n        folds (List[Slice or List[int]]): a list of indices (or slice objects)\n            corresponding to testing data subsets, that is,\n            `row_indices[folds[i]]` contains row indices used in fold i, so\n            `data[row_indices[folds[i]]]` is the corresponding testing data\n\n        train_time (np.ndarray): training times of batches\n\n        test_time (np.ndarray): testing times of batches\n    \"\"\"\n    def __init__(self, data=None, *,\n                 nmethods=None, nrows=None, nclasses=None,\n                 domain=None,\n                 row_indices=None, folds=None, score_by_folds=True,\n                 learners=None, models=None, failed=None,\n                 actual=None, predicted=None, probabilities=None,\n                 store_data=None, store_models=None,\n                 train_time=None, test_time=None):\n        \"\"\"\n        Construct an instance.\n\n        The constructor stores the given data, and creates empty arrays\n        `actual`, `predicted` and `probabilities` if ther are not given but\n        sufficient data is provided to deduct their shapes.\n\n        The function\n\n        - set any attributes specified directly through arguments.\n        - infers the number of methods, rows and classes from other data\n           and/or check their overall consistency.\n        - Prepare empty arrays `actual`, `predicted`, `probabilities` and\n          `failed` if the are not given. If not enough data is available,\n          the corresponding arrays are `None`.\n\n        Args:\n            data (Orange.data.Table): stored data from which test was sampled\n            nmethods (int): number of methods; can be inferred (or must match)\n                the size of `learners`, `models`, `failed`, `predicted` and\n                `probabilities`\n            nrows (int): number of data instances; can be inferred (or must\n                match) `data`, `row_indices`, `actual`, `predicted` and\n                `probabilities`\n            nclasses (int): number of class values (`None` if continuous); can\n                be inferred (or must match) from `domain.class_var` or\n                `probabilities`\n            domain (Orange.data.Domain): data domain; can be inferred (or must)\n                match `data.domain`\n            row_indices (np.ndarray): see class documentation\n            folds (np.ndarray): see class documentation\n            score_by_folds (np.ndarray): see class documentation\n            learners (np.ndarray): see class documentation\n            models (np.ndarray): see class documentation\n            failed (list of str): see class documentation\n            actual (np.ndarray): see class documentation\n            predicted (np.ndarray): see class documentation\n            probabilities (np.ndarray): see class documentation\n            store_data (bool): ignored; kept for backward compatibility\n            store_models (bool): ignored; kept for backward compatibility\n        \"\"\"\n\n        # Set given data directly from arguments\n        self.data = data\n        self.domain = domain\n\n        self.row_indices = row_indices\n        self.folds = folds\n        self.score_by_folds = score_by_folds\n\n        self.learners = learners\n        self.models = models\n\n        self.actual = actual\n        self.predicted = predicted\n        self.probabilities = probabilities\n        self.failed = failed\n\n        self.train_time = train_time\n        self.test_time = test_time\n\n        # Guess the rest -- or check for ambguities\n        def set_or_raise(value, exp_values, msg):\n            for exp_value in exp_values:\n                if exp_value is False:\n                    continue\n                if value is None:\n                    value = exp_value\n                elif value != exp_value:\n                    raise ValueError(msg)\n            return value\n        domain = self.domain = set_or_raise(\n            domain, [data is not None and data.domain],\n            \"mismatching domain\")\n        self.nrows = nrows = set_or_raise(\n            nrows, [actual is not None and len(actual),\n                    row_indices is not None and len(row_indices),\n                    predicted is not None and predicted.shape[1],\n                    probabilities is not None and probabilities.shape[1]],\n            \"mismatching number of rows\")\n        if domain is not None and domain.has_continuous_class:\n            if nclasses is not None:\n                raise ValueError(\n                    \"regression results cannot have non-None 'nclasses'\")\n            if probabilities is not None:\n                raise ValueError(\n                    \"regression results cannot have 'probabilities'\")\n        nclasses = set_or_raise(\n            nclasses, [domain is not None and domain.has_discrete_class and\n                       len(domain.class_var.values),\n                       probabilities is not None and probabilities.shape[2]],\n            \"mismatching number of class values\")\n        nmethods = set_or_raise(\n            nmethods, [learners is not None and len(learners),\n                       models is not None and models.shape[1],\n                       failed is not None and len(failed),\n                       predicted is not None and predicted.shape[0],\n                       probabilities is not None and probabilities.shape[0]],\n            \"mismatching number of methods\")\n\n        # Prepare empty arrays\n        if actual is None \\\n                and nrows is not None:\n            self.actual = np.empty(nrows)\n\n        if predicted is None \\\n                and nmethods is not None and nrows is not None:\n            self.predicted = np.empty((nmethods, nrows))\n\n        if probabilities is None \\\n                and nmethods is not None and nrows is not None \\\n                and nclasses is not None:\n            self.probabilities = \\\n                np.empty((nmethods, nrows, nclasses))\n\n        if failed is None \\\n                and nmethods is not None:\n            self.failed = [False] * nmethods\n\n    def get_fold(self, fold):\n        results = Results()\n        results.data = self.data\n\n        if self.folds is None:\n            raise ValueError(\"This 'Results' instance does not have folds.\")\n\n        if self.models is not None:\n            results.models = self.models[fold]\n\n        results.row_indices = self.row_indices[self.folds[fold]]\n        results.actual = self.actual[self.folds[fold]]\n        results.predicted = self.predicted[:, self.folds[fold]]\n        results.domain = self.domain\n\n        if self.probabilities is not None:\n            results.probabilities = self.probabilities[:, self.folds[fold]]\n\n        return results\n\n    def get_augmented_data(self, model_names,\n                           include_attrs=True, include_predictions=True,\n                           include_probabilities=True):\n        \"\"\"\n        Return the test data table augmented with meta attributes containing\n        predictions, probabilities (if the task is classification) and fold\n        indices.\n\n        Args:\n            model_names (list of str): names of models\n            include_attrs (bool):\n                if set to `False`, original attributes are removed\n            include_predictions (bool):\n                if set to `False`, predictions are not added\n            include_probabilities (bool):\n                if set to `False`, probabilities are not added\n\n        Returns:\n            augmented_data (Orange.data.Table):\n                data augmented with predictions, probabilities and fold indices\n\n        \"\"\"\n        assert self.predicted.shape[0] == len(model_names)\n\n        data = self.data[self.row_indices]\n        domain = data.domain\n        class_var = domain.class_var\n        classification = class_var and class_var.is_discrete\n\n        new_meta_attr = []\n        new_meta_vals = np.empty((len(data), 0))\n        names = [var.name for var in chain(domain.attributes,\n                                           domain.metas,\n                                           domain.class_vars)]\n\n        if classification:\n            # predictions\n            if include_predictions:\n                uniq_new, names = self.create_unique_vars(names, model_names, class_var.values)\n                new_meta_attr += uniq_new\n                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n            # probabilities\n            if include_probabilities:\n                proposed = [f\"{name} ({value})\" for name in model_names for value in class_var.values]\n\n                uniq_new, names = self.create_unique_vars(names, proposed)\n                new_meta_attr += uniq_new\n\n                for i in self.probabilities:\n                    new_meta_vals = np.hstack((new_meta_vals, i))\n\n        elif include_predictions:\n            # regression\n            uniq_new, names = self.create_unique_vars(names, model_names)\n            new_meta_attr += uniq_new\n            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n        # add fold info\n        if self.folds is not None:\n            values = [str(i + 1) for i in range(len(self.folds))]\n            uniq_new, names = self.create_unique_vars(names, [\"Fold\"], values)\n            new_meta_attr += uniq_new\n            fold = np.empty((len(data), 1))\n            for i, s in enumerate(self.folds):\n                fold[s, 0] = i\n            new_meta_vals = np.hstack((new_meta_vals, fold))\n\n        # append new columns to meta attributes\n        new_meta_attr = list(data.domain.metas) + new_meta_attr\n        new_meta_vals = np.hstack((data.metas, new_meta_vals))\n\n        attrs = data.domain.attributes if include_attrs else []\n        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)\n        predictions = data.transform(domain)\n        with predictions.unlocked(predictions.metas):\n            predictions.metas = new_meta_vals\n        predictions.name = data.name\n        return predictions\n\n    def create_unique_vars(self, names, proposed_names, values=()):\n        unique_vars = []\n        for proposed in proposed_names:\n            uniq = get_unique_names(names, proposed)\n            if values:\n                unique_vars.append(DiscreteVariable(uniq, values))\n            else:\n                unique_vars.append(ContinuousVariable(uniq))\n            names.append(uniq)\n        return unique_vars, names\n\n    def split_by_model(self):\n        \"\"\"\n        Split evaluation results by models.\n\n        The method generates instances of `Results` containing data for single\n        models\n        \"\"\"\n        data = self.data\n        nmethods = len(self.predicted)\n        for i in range(nmethods):\n            res = Results()\n            res.data = data\n            res.domain = self.domain\n            res.learners = [self.learners[i]]\n            res.row_indices = self.row_indices\n            res.actual = self.actual\n            res.folds = self.folds\n            res.score_by_folds = self.score_by_folds\n            res.test_time = self.test_time[i]\n            res.train_time = self.train_time[i]\n\n            res.predicted = self.predicted[(i,), :]\n            if getattr(self, \"probabilities\", None) is not None:\n                res.probabilities = self.probabilities[(i,), :, :]\n\n            if self.models is not None:\n                res.models = self.models[:, i:i + 1]\n\n            res.failed = [self.failed[i]]\n            yield res\n\n\nclass Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        return folds, row_indices, data[row_indices].Y\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if len(res.values.shape) > 1 and res.values.shape[1] > 1:\n                msg = \"Multiple targets are not supported.\"\n                results.failed[res.learner_i] = ValueError(msg)\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs\n\n\nclass CrossValidation(Validation):\n    \"\"\"\n    K-fold cross validation\n\n    Attributes:\n        k (int): number of folds (default: 10)\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n            If `True` but the class sizes don't allow it, it uses non-stratified\n            validataion and adds a list `warning` with a warning message(s) to\n            the `Result`.\n    \"\"\"\n    # TODO: list `warning` contains just repetitions of the same message\n    #       replace with a flag in `Results`?\n    def __init__(self, k=10, stratified=True, random_state=0,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.k = k\n        self.stratified = stratified\n        self.random_state = random_state\n        self.warnings = [] if warnings is None else warnings\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            try:\n                splitter = skl.StratifiedKFold(\n                    self.k, shuffle=True, random_state=self.random_state\n                )\n                splitter.get_n_splits(data.X, data.Y)\n                return list(splitter.split(data.X, data.Y))\n            except ValueError:\n                self.warnings.append(\"Using non-stratified sampling.\")\n\n        splitter = skl.KFold(\n            self.k, shuffle=True, random_state=self.random_state)\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass CrossValidationFeature(Validation):\n    \"\"\"\n    Cross validation with folds according to values of a feature.\n\n    Attributes:\n        feature (Orange.data.Variable): the feature defining the folds\n    \"\"\"\n    def __init__(self, feature=None,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.feature = feature\n\n    def get_indices(self, data):\n        data = data.transform(Domain([self.feature], None))\n        values = data[:, self.feature].X\n        indices = []\n        for v in range(len(self.feature.values)):\n            test_index = np.where(values == v)[0]\n            train_index = np.where((values != v) & (~np.isnan(values)))[0]\n            if test_index.size and train_index.size:\n                indices.append((train_index, test_index))\n        if not indices:\n            raise ValueError(\n                f\"'{self.feature.name}' does not have at least two distinct \"\n                \"values on the data\")\n        return indices\n\n\nclass LeaveOneOut(Validation):\n    \"\"\"Leave-one-out testing\"\"\"\n    score_by_folds = False\n\n    def get_indices(self, data):\n        splitter = skl.LeaveOneOut()\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n    @staticmethod\n    def prepare_arrays(data, indices):\n        # sped up version of super().prepare_arrays(data)\n        row_indices = np.arange(len(data))\n        return row_indices, row_indices, data.Y.flatten()\n\n\nclass ShuffleSplit(Validation):\n    \"\"\"\n    Test by repeated random sampling\n\n    Attributes:\n        n_resamples (int): number of repetitions\n        test_size (float, int, None):\n            If float, should be between 0.0 and 1.0 and represent the proportion\n            of the dataset to include in the test split. If int, represents the\n            absolute number of test samples. If None, the value is set to the\n            complement of the train size. By default, the value is set to 0.1.\n            The default will change in version 0.21. It will remain 0.1 only\n            if ``train_size`` is unspecified, otherwise it will complement\n            the specified ``train_size``.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        train_size : float, int, or None, default is None\n            If float, should be between 0.0 and 1.0 and represent the\n            proportion of the dataset to include in the train split. If\n            int, represents the absolute number of train samples. If None,\n            the value is automatically set to the complement of the test size.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n\n    \"\"\"\n    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,\n                 stratified=True, random_state=0,\n                 store_data=False, store_models=False):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.n_resamples = n_resamples\n        self.train_size = train_size\n        self.test_size = test_size\n        self.stratified = stratified\n        self.random_state = random_state\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            splitter = skl.StratifiedShuffleSplit(\n                n_splits=self.n_resamples, train_size=self.train_size,\n                test_size=self.test_size, random_state=self.random_state\n            )\n            splitter.get_n_splits(data.X, data.Y)\n            return list(splitter.split(data.X, data.Y))\n\n        splitter = skl.ShuffleSplit(\n            n_splits=self.n_resamples, train_size=self.train_size,\n            test_size=self.test_size, random_state=self.random_state\n        )\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass TestOnTestData(Validation):\n    \"\"\"\n    Test on separately provided test data\n\n    Note that the class has a different signature for `__call__`.\n    \"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n\n    def __new__(cls, data=None, test_data=None, learners=None,\n                preprocessor=None, **kwargs):\n        if \"train_data\" in kwargs:\n            if data is None:\n                data = kwargs.pop(\"train_data\")\n            else:\n                raise ValueError(\n                    \"argument 'data' is given twice (once as 'train_data')\")\n        return super().__new__(\n            cls,\n            data=data, learners=learners, preprocessor=preprocessor,\n            test_data=test_data, **kwargs)\n\n    def __call__(self, data, test_data, learners, preprocessor=None,\n                 *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): training data\n            test_data (Orange.data.Table): test_data\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n\n        train_data = preprocessor(data)\n        part_results = []\n        for (learner_i, learner) in enumerate(learners):\n            part_results.append(\n                _mp_worker(0, train_data, test_data, learner_i, learner,\n                           self.store_models))\n            callback((learner_i + 1) / len(learners))\n        callback(1)\n\n        results = Results(\n            data=test_data if self.store_data else None,\n            domain=test_data.domain,\n            nrows=len(test_data), learners=learners,\n            row_indices=np.arange(len(test_data)),\n            folds=(Ellipsis, ),\n            actual=test_data.Y,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (1, len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n\nclass TestOnTrainingData(TestOnTestData):\n    \"\"\"Test on training data\"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n    # signature is such as on the base class, pylint: disable=signature-differs\n    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):\n        return super().__new__(\n            cls,\n            data, test_data=data, learners=learners, preprocessor=preprocessor,\n            **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None,\n                 **kwargs):\n        kwargs.setdefault(\"test_data\", data)\n        # if kwargs contains anything besides test_data, this will be detected\n        # (and complained about) by super().__call__\n        return super().__call__(\n            data=data, learners=learners, preprocessor=preprocessor,\n            callback=callback, **kwargs)\n\n\ndef sample(table, n=0.7, stratified=False, replace=False,\n           random_state=None):\n    \"\"\"\n    Samples data instances from a data table. Returns the sample and\n    a dataset from input data table that are not in the sample. Also\n    uses several sampling functions from\n    `scikit-learn <http://scikit-learn.org>`_.\n\n    table : data table\n        A data table from which to sample.\n\n    n : float, int (default = 0.7)\n        If float, should be between 0.0 and 1.0 and represents\n        the proportion of data instances in the resulting sample. If\n        int, n is the number of data instances in the resulting sample.\n\n    stratified : bool, optional (default = False)\n        If true, sampling will try to consider class values and\n        match distribution of class values\n        in train and test subsets.\n\n    replace : bool, optional (default = False)\n        sample with replacement\n\n    random_state : int or RandomState\n        Pseudo-random number generator state used for random sampling.\n    \"\"\"\n\n    if isinstance(n, float):\n        n = int(n * len(table))\n\n    if replace:\n        if random_state is None:\n            rgen = np.random\n        else:\n            rgen = np.random.mtrand.RandomState(random_state)\n        a_sample = rgen.randint(0, len(table), n)\n        o = np.ones(len(table))\n        o[a_sample] = 0\n        others = np.nonzero(o)[0]\n        return table[a_sample], table[others]\n\n    n = len(table) - n\n    if stratified and table.domain.has_discrete_class:\n        test_size = max(len(table.domain.class_var.values), n)\n        splitter = skl.StratifiedShuffleSplit(\n            n_splits=1, test_size=test_size, train_size=len(table) - test_size,\n            random_state=random_state)\n        splitter.get_n_splits(table.X, table.Y)\n        ind = splitter.split(table.X, table.Y)\n    else:\n        splitter = skl.ShuffleSplit(\n            n_splits=1, test_size=n, random_state=random_state)\n        splitter.get_n_splits(table)\n        ind = splitter.split(table)\n    ind = next(ind)\n    return table[ind[0]], table[ind[1]]\n",
            "header": {
                "file": {
                    "path": "Orange/evaluation/testing.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "34a798241",
                "new_commit_hash": "d40701206",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/Orange/evaluation/testing.py"
            },
            "plus_file": {
                "path": "b/Orange/evaluation/testing.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 541,
                            "length": 6
                        },
                        "new_range": {
                            "start": 541,
                            "length": 11
                        },
                        "section": "class Validation:"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                results.failed[res.learner_i] = res.failed",
                                "type": "context"
                            },
                            {
                                "content": "                continue",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "            if len(res.values.shape) > 1 and res.values.shape[1] > 1:",
                                "type": "added"
                            },
                            {
                                "content": "                msg = \"Multiple targets are not supported.\"",
                                "type": "added"
                            },
                            {
                                "content": "                results.failed[res.learner_i] = ValueError(msg)",
                                "type": "added"
                            },
                            {
                                "content": "                continue",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "            if self.store_models:",
                                "type": "context"
                            },
                            {
                                "content": "                results.models[res.fold_i][res.learner_i] = res.model",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "Orange/evaluation/testing.py",
                            "type": "method",
                            "name": "Validation._collect_part_results",
                            "content": "class Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        return folds, row_indices, data[row_indices].Y\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if len(res.values.shape) > 1 and res.values.shape[1] > 1:\n                msg = \"Multiple targets are not supported.\"\n                results.failed[res.learner_i] = ValueError(msg)\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 529,
                            "end_lineno": 557,
                            "parent": {
                                "file_name": "Orange/evaluation/testing.py",
                                "type": "class",
                                "name": "Validation",
                                "content": "class Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        return folds, row_indices, data[row_indices].Y\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if len(res.values.shape) > 1 and res.values.shape[1] > 1:\n                msg = \"Multiple targets are not supported.\"\n                results.failed[res.learner_i] = ValueError(msg)\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 363,
                                "end_lineno": 557,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "Orange/evaluation/testing.py",
                            "type": "class",
                            "name": "Validation",
                            "content": "class Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        return folds, row_indices, data[row_indices].Y\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if len(res.values.shape) > 1 and res.values.shape[1] > 1:\n                msg = \"Multiple targets are not supported.\"\n                results.failed[res.learner_i] = ValueError(msg)\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 363,
                            "end_lineno": 557,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# pylint: disable=missing-docstring\n# pylint: disable=protected-access\nimport unittest\nfrom unittest.mock import Mock, patch\n\nimport numpy as np\nfrom AnyQt.QtCore import Qt\nfrom AnyQt.QtTest import QTest\nfrom AnyQt.QtWidgets import QApplication\nimport baycomp\n\nfrom Orange.classification import MajorityLearner, LogisticRegressionLearner, \\\n    RandomForestLearner\nfrom Orange.classification.majority import ConstantModel\nfrom Orange.data import Table, Domain, DiscreteVariable, ContinuousVariable\nfrom Orange.evaluation import Results, TestOnTestData, scoring\nfrom Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\\n    Score\nfrom Orange.base import Learner, Model\nfrom Orange.modelling import ConstantLearner\nfrom Orange.regression import MeanLearner\nfrom Orange.widgets.evaluate.owtestandscore import (\n    OWTestAndScore, results_one_vs_rest)\nfrom Orange.widgets.settings import (\n    ClassValuesContextHandler, PerfectDomainContextHandler)\nfrom Orange.widgets.tests.base import WidgetTest\nfrom Orange.widgets.tests.utils import simulate, possible_duplicate_table\nfrom Orange.tests import test_filename\n\n\nclass BadLearner(Learner):\n    def fit(self, *_, **_2):  # pylint: disable=arguments-differ\n        return 1 / 0\n\n\nclass TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([[0.2, 0.1, 0.2]]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)\n\n\nclass TestHelpers(unittest.TestCase):\n    def test_results_one_vs_rest(self):\n        data = Table(test_filename(\"datasets/lenses.tab\"))\n        learners = [MajorityLearner()]\n        res = TestOnTestData()(data[1::2], data[::2], learners=learners)\n        r1 = results_one_vs_rest(res, pos_index=0)\n        r2 = results_one_vs_rest(res, pos_index=1)\n        r3 = results_one_vs_rest(res, pos_index=2)\n\n        np.testing.assert_almost_equal(np.sum(r1.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r2.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r3.probabilities, axis=2), 1.0)\n\n        np.testing.assert_almost_equal(\n            r1.probabilities[:, :, 1] +\n            r2.probabilities[:, :, 1] +\n            r3.probabilities[:, :, 1],\n            1.0\n        )\n        self.assertEqual(r1.folds, res.folds)\n        self.assertEqual(r2.folds, res.folds)\n        self.assertEqual(r3.folds, res.folds)\n\n        np.testing.assert_equal(r1.row_indices, res.row_indices)\n        np.testing.assert_equal(r2.row_indices, res.row_indices)\n        np.testing.assert_equal(r3.row_indices, res.row_indices)\n\n\nif __name__ == \"__main__\":\n    unittest.main()\n",
            "new_file_content": "# pylint: disable=missing-docstring\n# pylint: disable=protected-access\nimport unittest\nfrom unittest.mock import Mock, patch\n\nimport numpy as np\nfrom AnyQt.QtCore import Qt\nfrom AnyQt.QtTest import QTest\nfrom AnyQt.QtWidgets import QApplication\nimport baycomp\n\nfrom Orange.classification import MajorityLearner, LogisticRegressionLearner, \\\n    RandomForestLearner\nfrom Orange.classification.majority import ConstantModel\nfrom Orange.data import Table, Domain, DiscreteVariable, ContinuousVariable\nfrom Orange.evaluation import Results, TestOnTestData, scoring\nfrom Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\\n    Score\nfrom Orange.base import Learner, Model\nfrom Orange.modelling import ConstantLearner\nfrom Orange.regression import MeanLearner, PLSRegressionLearner\nfrom Orange.widgets.evaluate.owtestandscore import (\n    OWTestAndScore, results_one_vs_rest)\nfrom Orange.widgets.settings import (\n    ClassValuesContextHandler, PerfectDomainContextHandler)\nfrom Orange.widgets.tests.base import WidgetTest\nfrom Orange.widgets.tests.utils import simulate, possible_duplicate_table\nfrom Orange.tests import test_filename\n\n\nclass BadLearner(Learner):\n    def fit(self, *_, **_2):  # pylint: disable=arguments-differ\n        return 1 / 0\n\n\nclass TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([0.2, 0.1, 0.2]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)\n\n    def test_multiple_targets_pls(self):\n        housing = Table(\"housing\")\n        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]\n        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)\n        multiple_targets_data = housing.transform(domain)\n\n        self.widget.error = Mock()\n        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)\n        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())\n        self.wait_until_finished()\n        self.assertIn(\"Multiple targets are not supported.\",\n                      self.widget.error.call_args[0][0])\n\n\nclass TestHelpers(unittest.TestCase):\n    def test_results_one_vs_rest(self):\n        data = Table(test_filename(\"datasets/lenses.tab\"))\n        learners = [MajorityLearner()]\n        res = TestOnTestData()(data[1::2], data[::2], learners=learners)\n        r1 = results_one_vs_rest(res, pos_index=0)\n        r2 = results_one_vs_rest(res, pos_index=1)\n        r3 = results_one_vs_rest(res, pos_index=2)\n\n        np.testing.assert_almost_equal(np.sum(r1.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r2.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r3.probabilities, axis=2), 1.0)\n\n        np.testing.assert_almost_equal(\n            r1.probabilities[:, :, 1] +\n            r2.probabilities[:, :, 1] +\n            r3.probabilities[:, :, 1],\n            1.0\n        )\n        self.assertEqual(r1.folds, res.folds)\n        self.assertEqual(r2.folds, res.folds)\n        self.assertEqual(r3.folds, res.folds)\n\n        np.testing.assert_equal(r1.row_indices, res.row_indices)\n        np.testing.assert_equal(r2.row_indices, res.row_indices)\n        np.testing.assert_equal(r3.row_indices, res.row_indices)\n\n\nif __name__ == \"__main__\":\n    unittest.main()\n",
            "header": {
                "file": {
                    "path": "Orange/widgets/evaluate/tests/test_owtestandscore.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "0d2fd2c4a",
                "new_commit_hash": "3bc54c780",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/Orange/widgets/evaluate/tests/test_owtestandscore.py"
            },
            "plus_file": {
                "path": "b/Orange/widgets/evaluate/tests/test_owtestandscore.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 18,
                            "length": 7
                        },
                        "new_range": {
                            "start": 18,
                            "length": 7
                        },
                        "section": "from Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    Score",
                                "type": "context"
                            },
                            {
                                "content": "from Orange.base import Learner, Model",
                                "type": "context"
                            },
                            {
                                "content": "from Orange.modelling import ConstantLearner",
                                "type": "context"
                            },
                            {
                                "content": "from Orange.regression import MeanLearner",
                                "type": "deleted"
                            },
                            {
                                "content": "from Orange.regression import MeanLearner, PLSRegressionLearner",
                                "type": "added"
                            },
                            {
                                "content": "from Orange.widgets.evaluate.owtestandscore import (",
                                "type": "context"
                            },
                            {
                                "content": "    OWTestAndScore, results_one_vs_rest)",
                                "type": "context"
                            },
                            {
                                "content": "from Orange.widgets.settings import (",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                            "type": "statement",
                            "name": "",
                            "content": "from Orange.regression import MeanLearner, PLSRegressionLearner",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 21,
                            "end_lineno": 21,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                            "type": "statement",
                            "name": "",
                            "content": "from Orange.regression import MeanLearner",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 21,
                            "end_lineno": 21,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 753,
                            "length": 7
                        },
                        "new_range": {
                            "start": 753,
                            "length": 7
                        },
                        "section": "class TestOWTestAndScore(WidgetTest):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                        ])",
                                "type": "context"
                            },
                            {
                                "content": "        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        mock_model = Mock(spec=Model, return_value=np.asarray([[0.2, 0.1, 0.2]]))",
                                "type": "deleted"
                            },
                            {
                                "content": "        mock_model = Mock(spec=Model, return_value=np.asarray([0.2, 0.1, 0.2]))",
                                "type": "added"
                            },
                            {
                                "content": "        mock_model.name = 'Mockery'",
                                "type": "context"
                            },
                            {
                                "content": "        mock_model.domain = domain",
                                "type": "context"
                            },
                            {
                                "content": "        mock_learner = Mock(spec=Learner, return_value=mock_model)",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                            "type": "class",
                            "name": "TestOWTestAndScore",
                            "content": "class TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([[0.2, 0.1, 0.2]]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 36,
                            "end_lineno": 770,
                            "parent": null
                        },
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                            "type": "method",
                            "name": "TestOWTestAndScore.test_multi_target_input",
                            "content": "class TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([[0.2, 0.1, 0.2]]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 735,
                            "end_lineno": 770,
                            "parent": {
                                "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                                "type": "class",
                                "name": "TestOWTestAndScore",
                                "content": "class TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([[0.2, 0.1, 0.2]]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 36,
                                "end_lineno": 770,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 769,
                            "length": 6
                        },
                        "new_range": {
                            "start": 769,
                            "length": 19
                        },
                        "section": "class TestOWTestAndScore(WidgetTest):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        self.assertTrue(NewScorer in widget.scorers)",
                                "type": "context"
                            },
                            {
                                "content": "        self.assertTrue(len(widget._successful_slots()) == 1)",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def test_multiple_targets_pls(self):",
                                "type": "added"
                            },
                            {
                                "content": "        housing = Table(\"housing\")",
                                "type": "added"
                            },
                            {
                                "content": "        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]",
                                "type": "added"
                            },
                            {
                                "content": "        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)",
                                "type": "added"
                            },
                            {
                                "content": "        multiple_targets_data = housing.transform(domain)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "        self.widget.error = Mock()",
                                "type": "added"
                            },
                            {
                                "content": "        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)",
                                "type": "added"
                            },
                            {
                                "content": "        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())",
                                "type": "added"
                            },
                            {
                                "content": "        self.wait_until_finished()",
                                "type": "added"
                            },
                            {
                                "content": "        self.assertIn(\"Multiple targets are not supported.\",",
                                "type": "added"
                            },
                            {
                                "content": "                      self.widget.error.call_args[0][0])",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "class TestHelpers(unittest.TestCase):",
                                "type": "context"
                            },
                            {
                                "content": "    def test_results_one_vs_rest(self):",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                            "type": "class",
                            "name": "TestOWTestAndScore",
                            "content": "class TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([0.2, 0.1, 0.2]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)\n\n    def test_multiple_targets_pls(self):\n        housing = Table(\"housing\")\n        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]\n        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)\n        multiple_targets_data = housing.transform(domain)\n\n        self.widget.error = Mock()\n        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)\n        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())\n        self.wait_until_finished()\n        self.assertIn(\"Multiple targets are not supported.\",\n                      self.widget.error.call_args[0][0])",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 36,
                            "end_lineno": 783,
                            "parent": null
                        }
                    ],
                    "added_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                            "type": "method",
                            "name": "TestOWTestAndScore.test_multiple_targets_pls",
                            "content": "class TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([0.2, 0.1, 0.2]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)\n\n    def test_multiple_targets_pls(self):\n        housing = Table(\"housing\")\n        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]\n        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)\n        multiple_targets_data = housing.transform(domain)\n\n        self.widget.error = Mock()\n        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)\n        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())\n        self.wait_until_finished()\n        self.assertIn(\"Multiple targets are not supported.\",\n                      self.widget.error.call_args[0][0])",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 772,
                            "end_lineno": 783,
                            "parent": {
                                "file_name": "Orange/widgets/evaluate/tests/test_owtestandscore.py",
                                "type": "class",
                                "name": "TestOWTestAndScore",
                                "content": "class TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([0.2, 0.1, 0.2]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)\n\n    def test_multiple_targets_pls(self):\n        housing = Table(\"housing\")\n        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]\n        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)\n        multiple_targets_data = housing.transform(domain)\n\n        self.widget.error = Mock()\n        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)\n        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())\n        self.wait_until_finished()\n        self.assertIn(\"Multiple targets are not supported.\",\n                      self.widget.error.call_args[0][0])",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 36,
                                "end_lineno": 783,
                                "parent": null
                            }
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "fb23419433230b6b38cab3e3e3dbb078cd95f54b^",
    "new_commit_hash": "fb23419433230b6b38cab3e3e3dbb078cd95f54b",
    "commit_message": "Test and Score: Multiple targets quick fix",
    "commit_date": "2024-02-26T11:50:10+01:00",
    "metadata": {}
}
{
    "repo_name": "orange3",
    "new_commit_hash": "fb23419433230b6b38cab3e3e3dbb078cd95f54b",
    "test_file_codes": [
        "# pylint: disable=missing-docstring\n# pylint: disable=protected-access\nimport unittest\nfrom unittest.mock import Mock, patch\n\nimport numpy as np\nfrom AnyQt.QtCore import Qt\nfrom AnyQt.QtTest import QTest\nfrom AnyQt.QtWidgets import QApplication\nimport baycomp\n\nfrom Orange.classification import MajorityLearner, LogisticRegressionLearner, \\\n    RandomForestLearner\nfrom Orange.classification.majority import ConstantModel\nfrom Orange.data import Table, Domain, DiscreteVariable, ContinuousVariable\nfrom Orange.evaluation import Results, TestOnTestData, scoring\nfrom Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\\n    Score\nfrom Orange.base import Learner, Model\nfrom Orange.modelling import ConstantLearner\nfrom Orange.regression import MeanLearner, PLSRegressionLearner\nfrom Orange.widgets.evaluate.owtestandscore import (\n    OWTestAndScore, results_one_vs_rest)\nfrom Orange.widgets.settings import (\n    ClassValuesContextHandler, PerfectDomainContextHandler)\nfrom Orange.widgets.tests.base import WidgetTest\nfrom Orange.widgets.tests.utils import simulate, possible_duplicate_table\nfrom Orange.tests import test_filename\n\n\nclass BadLearner(Learner):\n    def fit(self, *_, **_2):  # pylint: disable=arguments-differ\n        return 1 / 0\n\n\nclass TestOWTestAndScore(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestAndScore)  # type: OWTestAndScore\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=(\"y\", \"n\"))])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_multiple_learners(self):\n        def check_evres_names(expeced):\n            res = self.get_output(self.widget.Outputs.evaluations_results)\n            self.assertSequenceEqual(res.learner_names, expeced)\n\n        data = Table(\"iris\")[::15].copy()\n        m1 = MajorityLearner()\n        m1.name = \"M1\"\n        m2 = MajorityLearner()\n        m2.name = \"M2\"\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        self.send_signal(self.widget.Inputs.learner, m2, 2)\n        res = self.get_output(self.widget.Outputs.evaluations_results)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner, None, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M1\", \"M2\"])\n        self.send_signal(self.widget.Inputs.learner,\n                         self.widget.Inputs.learner.closing_sentinel, 1)\n        check_evres_names([\"M2\"])\n        self.send_signal(self.widget.Inputs.learner, m1, 1)\n        check_evres_names([\"M2\", \"M1\"])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestAndScore.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(\n            Domain([ContinuousVariable(\"x\")], class_vars=iris.domain.class_vars)).copy()\n        with iris_test.unlocked():\n            iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestAndScore.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestAndScore.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestAndScore.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_migrate_shown_scores(self):\n        settings = {\"score_table\": {\"shown_scores\": {\"Sensitivity\"}}}\n        self.widget.migrate_settings(settings, 3)\n        self.assertTrue(settings[\"score_table\"][\"show_score_hints\"][\"Sensitivity\"])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table.from_list(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=(\"y\",))]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.train_data_error.is_shown())\n        self.send_signal(self.widget.Inputs.train_data, table)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.train_data_error.is_shown())\n\n    def test_data_errors(self):\n        \"\"\" Test all data_errors \"\"\"\n\n        def assertErrorShown(data, is_shown, message):\n            self.send_signal(self.widget.Inputs.train_data, data)\n            self.assertEqual(is_shown, self.widget.Error.train_data_error.is_shown())\n            self.assertEqual(message, str(self.widget.Error.train_data_error))\n\n        data = Table(\"iris\")[::30].copy()\n        with data.unlocked():\n            data.Y[:] = np.nan\n\n        iris_empty_x = Table.from_table(\n            Domain([], data.domain.class_var), Table(\"iris\")\n        )\n\n        for data, is_shown, message in zip(\n                [None, data, Table(\"iris\")[:30], iris_empty_x, data[:0]],\n                [False, True, True, True, True],\n                [\n                    \"\",\n                    \"Target variable has no values.\",\n                    \"Target variable has only one value.\",\n                    \"Data has no features to learn from.\",\n                    \"Train dataset is empty.\"\n                ]\n\n        ):\n            assertErrorShown(data, is_shown, message)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n                name = \"new scorer\"\n\n                @staticmethod\n                def is_compatible(domain: Domain) -> bool:\n                    return True\n\n            class NewClassificationScore(ClassificationScore):\n                name = \"new classification scorer\"\n                default_visible = False\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            widget = self.create_widget(OWTestAndScore)\n            header = widget.score_table.view.horizontalHeader()\n            self.send_signal(widget.Inputs.train_data, Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertFalse(header.isSectionHidden(3 + scorer_names.index(\"new scorer\")))\n            self.assertIn(\"new classification scorer\", scorer_names)\n            self.assertTrue(header.isSectionHidden(3 + scorer_names.index(\"new classification scorer\")))\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n            model = widget.score_table.model\n\n\n            self.send_signal(widget.Inputs.train_data, Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in widget.scorers]\n            self.assertIn(\"new scorer\", scorer_names)\n            self.assertNotIn(\"new classification scorer\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(widget.Inputs.train_data, None)\n            self.assertEqual(widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestAndScore\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner(Learner):\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.insert_learner(0, SetosaLearner())\n        self.widget.insert_learner(1, VersicolorLearner())\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.adjustSize()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        p = header.rect().center()\n        # second visible header section (after 'Model')\n        _, _, idx, *_ = (i for i in range(header.count())\n                      if not header.isSectionHidden(i))\n        p.setX(header.sectionPosition(idx) + 5)\n        QTest.mouseClick(header.viewport(), Qt.LeftButton, pos=p)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().index(0, 0).data(), \"versicolor\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().index(0, 0).data(), \"setosa\")\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestAndScore\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(\n            self._test_scores(\n                table, table[:3], ConstantLearner(),\n                OWTestAndScore.TestOnTest, None\n            ),\n            (None, 1, 1, 1, 1)\n        )\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestAndScore.TestOnTest, None),\n            (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (0, 0.25, 0.2, 0.1666666, 0.25),\n        )\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table.from_list(\n            self.scores_domain,\n            list(zip([1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        lr = LogisticRegressionLearner()\n        np.testing.assert_\n        np.testing.assert_almost_equal(\n            self._test_scores(\n                table_train, table_test, lr, OWTestAndScore.TestOnTest, None\n            ),\n            (1, 0.8, 0.8, 13 / 15, 0.8)\n        )\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestAndScore.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n    def test_no_stratification(self):\n        w = self.widget\n        w.cv_stratified = True\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(),\n            OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(0)\n        w.controls.n_folds.activated[int].emit(0)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        w.controls.n_folds.setCurrentIndex(2)\n        w.controls.n_folds.activated[int].emit(2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"iris\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"zoo\"), None, MajorityLearner(), OWTestAndScore.KFold, 2)\n        self.assertTrue(w.Warning.cant_stratify.is_shown())\n\n        self._test_scores(\n            Table(\"housing\"), None, MeanLearner(), OWTestAndScore.KFold, 2)\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n        self.assertTrue(w.Information.cant_stratify_numeric.is_shown())\n\n        w.controls.cv_stratified.click()\n        self.assertFalse(w.Warning.cant_stratify.is_shown())\n\n    def test_too_many_folds(self):\n        w = self.widget\n        w.controls.resampling.buttons[OWTestAndScore.KFold].click()\n        w.n_folds = 3\n        self.send_signal(w.Inputs.train_data, Table(\"zoo\")[:8])\n        self.send_signal(w.Inputs.learner, MajorityLearner(), 0, wait=5000)\n        self.assertTrue(w.Error.too_many_folds.is_shown())\n\n    def _set_comparison_score(self, score):\n        w = self.widget\n        control = w.controls.comparison_criterion\n        control.setCurrentText(score)\n        w.comparison_criterion = control.findText(score)\n\n    def _set_three_majorities(self):\n        w = self.widget\n        data = Table(\"iris\")[::15]\n        self.send_signal(w.Inputs.train_data, data)\n        for i, name in enumerate([\"maja\", \"majb\", \"majc\"]):\n            learner = MajorityLearner()\n            learner.name = name\n            self.send_signal(w.Inputs.learner, learner, i)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n    @patch(\"baycomp.two_on_single\", Mock(wraps=baycomp.two_on_single))\n    def test_comparison_requires_cv(self):\n        w = self.widget\n        self.send_signal(w.Inputs.train_data, Table(\"iris\")[::15])\n\n        w.comparison_criterion = 1\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n        self.assertFalse(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_not_called()\n        baycomp.two_on_single.reset_mock()\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertTrue(w.compbox.isEnabled())\n        baycomp.two_on_single.assert_called()\n        baycomp.two_on_single.reset_mock()\n\n    def test_comparison_requires_multiple_models(self):\n        w = self.widget\n        rbs = w.controls.resampling.buttons\n\n        self._set_three_majorities()\n        w.comparison_criterion = 1\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n        self.send_signal(w.Inputs.learner, None, 2)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.LeaveOneOut].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        learner = MajorityLearner()\n        learner.name = \"majd\"\n        self.send_signal(w.Inputs.learner, learner, 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(w.comparison_table.isEnabled())\n\n        rbs[OWTestAndScore.KFold].click()\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(w.comparison_table.isEnabled())\n\n    def test_comparison_bad_slots(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.send_signal(w.Inputs.learner, BadLearner(), 2, wait=5000)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNotNone(w.comparison_table.cellWidget(0, 1))\n        self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n        self.assertEqual(len(w._successful_slots()), 2)\n\n    def test_comparison_bad_scores(self):\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"Classification accuracy\")\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        score_calls = -1\n\n        def fail_on_first(*_, **_2):\n            nonlocal score_calls\n            score_calls += 1\n            return 1 / score_calls\n\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            w.update_comparison_table()\n\n            self.assertIsNone(w.comparison_table.cellWidget(0, 1))\n            self.assertIsNone(w.comparison_table.cellWidget(0, 2))\n            self.assertIsNone(w.comparison_table.cellWidget(1, 0))\n            self.assertIsNone(w.comparison_table.cellWidget(2, 0))\n            self.assertIsNotNone(w.comparison_table.cellWidget(1, 2))\n            self.assertIsNotNone(w.comparison_table.cellWidget(2, 1))\n            self.assertTrue(w.Warning.scores_not_computed.is_shown())\n\n        score_calls = -1\n        with patch.object(scoring.CA, \"compute_score\", new=fail_on_first):\n            slots = w._successful_slots()\n            self.assertEqual(len(slots), 3)\n            scores = w._scores_by_folds(slots)\n            self.assertIsNone(scores[0])\n            self.assertEqual(scores[1][0], 1)\n            self.assertAlmostEqual(scores[2][0], 1 / 6)\n\n    def test_comparison_binary_score(self):\n        # false warning at call_arg.kwargs\n        # pylint: disable=unpacking-non-sequence\n        w = self.widget\n        self._set_three_majorities()\n        self._set_comparison_score(\"F1\")\n        f1mock = Mock(wraps=scoring.F1.compute_score)\n\n        iris = Table(\"iris\")\n        with patch.object(scoring.F1, \"compute_score\", f1mock):\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[1])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 1)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            iris.domain.class_var.values[2])\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"target\"], 2)\n            self.assertFalse(\"average\" in kwargs)\n\n            simulate.combobox_activate_item(w.controls.class_selection,\n                                            OWTestAndScore.TARGET_AVERAGE)\n            _, kwargs = f1mock.call_args\n            self.assertEqual(kwargs[\"average\"], \"weighted\")\n            self.assertFalse(\"target\" in kwargs)\n\n    def test_fill_table(self):\n        w = self.widget\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def probs(p1, p2, rope):\n            p1 += 1\n            p2 += 1\n            norm = p1 + p2 + rope * (p1 + p2)\n            if rope == 0:\n                return p1 / norm, p2 / norm\n            else:\n                return p1 / norm, rope / norm, p2 / norm\n\n        def two_on_single(res1, res2, rope=0):\n            return probs(scores.index(res1), scores.index(res2), rope)\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.use_rope, w.rope in ((True, 0), (False, 0.1)):\n                w._fill_table(slots, scores)\n                for row in range(3):\n                    for col in range(3):\n                        if row == col:\n                            continue\n                        label = w.comparison_table.cellWidget(row, col)\n                        self.assertEqual(label.text(),\n                                         f\"{(row + 1) / (row + col + 2):.3f}\")\n                        self.assertIn(f\"{(row + 1) / (row + col + 2):.3f}\",\n                                      label.toolTip())\n\n            w.use_rope = True\n            w.rope = 0.25\n            w._fill_table(slots, scores)\n            for row in range(3):\n                for col in range(3):\n                    if row == col:\n                        continue\n                    label = w.comparison_table.cellWidget(row, col)\n                    for text in (label.text(), label.toolTip()):\n                        self.assertIn(f\"{probs(row, col, w.rope)[0]:.3f}\", text)\n                        self.assertIn(f\"{probs(row, col, w.rope)[1]:.3f}\", text)\n\n    def test_nan_on_comparison(self):\n        w = self.widget\n        w.use_rope = True\n        self._set_three_majorities()\n        scores = [object(), object(), object()]\n        slots = w._successful_slots()\n\n        def two_on_single(_1, _2, rope=0):\n            if rope:\n                return np.nan, np.nan, np.nan\n            else:\n                return np.nan, np.nan\n\n        with patch(\"baycomp.two_on_single\", new=two_on_single):\n            for w.rope in (0, 0.1):\n                w._fill_table(slots, scores)\n                label = w.comparison_table.cellWidget(1, 0)\n                self.assertEqual(label.text(), \"NA\")\n\n    def test_unique_output_domain(self):\n        data = possible_duplicate_table('random forest')\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        output = self.get_output(self.widget.Outputs.predictions)\n        self.assertEqual(output.domain.metas[0].name, 'random forest (1)')\n\n    def test_copy_to_clipboard(self):\n        self.send_signal(self.widget.Inputs.train_data, Table(\"iris\"))\n        self.send_signal(self.widget.Inputs.learner, RandomForestLearner(), 0)\n        self.wait_until_finished()\n        view = self.widget.score_table.view\n        model = self.widget.score_table.model\n        selection_model = view.selectionModel()\n        selection_model.select(model.index(0, 0),\n                               selection_model.Select | selection_model.Rows)\n        self.widget.copy_to_clipboard()\n        clipboard_text = QApplication.clipboard().text()\n        # Tests appear to register additional scorers, so we clip the list\n        # to what we know to be there and visible\n        clipboard_text = \"\\t\".join(clipboard_text.split(\"\\t\")[:6]).strip()\n        view_text = \"\\t\".join([str(model.data(model.index(0, i)))\n                               for i in (0, 3, 4, 5, 6, 7)]).strip()\n        self.assertEqual(clipboard_text, view_text)\n\n    def test_multi_target_input(self):\n        class NewScorer(Score):\n            class_types = (\n                ContinuousVariable,\n                DiscreteVariable,\n            )\n\n            @staticmethod\n            def is_compatible(domain: Domain) -> bool:\n                return True\n\n            def compute_score(self, results):\n                return [0.75]\n\n        domain = Domain([ContinuousVariable('var1')],\n                        class_vars=[\n                            ContinuousVariable('c1'),\n                            DiscreteVariable('c2', values=('no', 'yes'))\n                        ])\n        data = Table.from_list(domain, [[1, 5, 0], [2, 10, 1], [2, 10, 1]])\n\n        mock_model = Mock(spec=Model, return_value=np.asarray([0.2, 0.1, 0.2]))\n        mock_model.name = 'Mockery'\n        mock_model.domain = domain\n        mock_learner = Mock(spec=Learner, return_value=mock_model)\n        mock_learner.name = 'Mockery'\n\n        widget = self.create_widget(OWTestAndScore)\n        widget.resampling = OWTestAndScore.TestOnTrain\n        self.send_signal(widget.Inputs.train_data, data)\n        self.send_signal(widget.Inputs.learner, MajorityLearner(), 0)\n        self.send_signal(widget.Inputs.learner, mock_learner, 1)\n        _ = self.get_output(widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(len(widget.scorers) == 1)\n        self.assertTrue(NewScorer in widget.scorers)\n        self.assertTrue(len(widget._successful_slots()) == 1)\n\n    def test_multiple_targets_pls(self):\n        housing = Table(\"housing\")\n        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]\n        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)\n        multiple_targets_data = housing.transform(domain)\n\n        self.widget.error = Mock()\n        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)\n        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())\n        self.wait_until_finished()\n        self.assertIn(\"Multiple targets are not supported.\",\n                      self.widget.error.call_args[0][0])\n\n\nclass TestHelpers(unittest.TestCase):\n    def test_results_one_vs_rest(self):\n        data = Table(test_filename(\"datasets/lenses.tab\"))\n        learners = [MajorityLearner()]\n        res = TestOnTestData()(data[1::2], data[::2], learners=learners)\n        r1 = results_one_vs_rest(res, pos_index=0)\n        r2 = results_one_vs_rest(res, pos_index=1)\n        r3 = results_one_vs_rest(res, pos_index=2)\n\n        np.testing.assert_almost_equal(np.sum(r1.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r2.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r3.probabilities, axis=2), 1.0)\n\n        np.testing.assert_almost_equal(\n            r1.probabilities[:, :, 1] +\n            r2.probabilities[:, :, 1] +\n            r3.probabilities[:, :, 1],\n            1.0\n        )\n        self.assertEqual(r1.folds, res.folds)\n        self.assertEqual(r2.folds, res.folds)\n        self.assertEqual(r3.folds, res.folds)\n\n        np.testing.assert_equal(r1.row_indices, res.row_indices)\n        np.testing.assert_equal(r2.row_indices, res.row_indices)\n        np.testing.assert_equal(r3.row_indices, res.row_indices)\n\n\nif __name__ == \"__main__\":\n    unittest.main()\n"
    ],
    "test_file_names": [
        "test_1.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Orange installation attempts...\nAttempting installation with Python 3.7...\nCompiling Orange/classification/_tree_scorers.pyx because it changed.\nCompiling Orange/data/_contingency.pyx because it changed.\nCompiling Orange/data/_io.pyx because it changed.\nCompiling Orange/data/_valuecount.pyx because it changed.\nCompiling Orange/data/_variable.pyx because it changed.\nCompiling Orange/distance/_distance.pyx because it changed.\nCompiling Orange/preprocess/_discretize.pyx because it changed.\nCompiling Orange/preprocess/_relieff.pyx because it changed.\nCompiling Orange/projection/_som.pyx because it changed.\n[1/9] Cythonizing Orange/classification/_tree_scorers.pyx\n[2/9] Cythonizing Orange/data/_contingency.pyx\n[3/9] Cythonizing Orange/data/_io.pyx\n[4/9] Cythonizing Orange/data/_valuecount.pyx\n[5/9] Cythonizing Orange/data/_variable.pyx\n[6/9] Cythonizing Orange/distance/_distance.pyx\n[7/9] Cythonizing Orange/preprocess/_discretize.pyx\n[8/9] Cythonizing Orange/preprocess/_relieff.pyx\n[9/9] Cythonizing Orange/projection/_som.pyx\nrunning build_ext\nbuilding 'Orange.classification._simple_tree' extension\ncreating build\ncreating build/temp.linux-x86_64-3.7\ncreating build/temp.linux-x86_64-3.7/Orange\ncreating build/temp.linux-x86_64-3.7/Orange/classification\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/classification/_simple_tree.c -o build/temp.linux-x86_64-3.7/Orange/classification/_simple_tree.o\ncreating build/lib.linux-x86_64-3.7\ncreating build/lib.linux-x86_64-3.7/Orange\ncreating build/lib.linux-x86_64-3.7/Orange/classification\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/classification/_simple_tree.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/classification/_simple_tree.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.widgets.utils._grid_density' extension\ncreating build/temp.linux-x86_64-3.7/Orange/widgets\ncreating build/temp.linux-x86_64-3.7/Orange/widgets/utils\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/widgets/utils/_grid_density.cpp -o build/temp.linux-x86_64-3.7/Orange/widgets/utils/_grid_density.o\ncreating build/lib.linux-x86_64-3.7/Orange/widgets\ncreating build/lib.linux-x86_64-3.7/Orange/widgets/utils\nc++ -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/widgets/utils/_grid_density.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/widgets/utils/_grid_density.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.classification._tree_scorers' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/classification/_tree_scorers.c -o build/temp.linux-x86_64-3.7/Orange/classification/_tree_scorers.o\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/classification/_tree_scorers.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/classification/_tree_scorers.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._contingency' extension\ncreating build/temp.linux-x86_64-3.7/Orange/data\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/data/_contingency.c -o build/temp.linux-x86_64-3.7/Orange/data/_contingency.o\ncreating build/lib.linux-x86_64-3.7/Orange/data\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_contingency.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/data/_contingency.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._io' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/data/_io.c -o build/temp.linux-x86_64-3.7/Orange/data/_io.o\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_io.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/data/_io.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._valuecount' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/data/_valuecount.c -o build/temp.linux-x86_64-3.7/Orange/data/_valuecount.o\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_valuecount.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/data/_valuecount.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._variable' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/data/_variable.c -o build/temp.linux-x86_64-3.7/Orange/data/_variable.o\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_variable.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/data/_variable.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.distance._distance' extension\ncreating build/temp.linux-x86_64-3.7/Orange/distance\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/distance/_distance.c -o build/temp.linux-x86_64-3.7/Orange/distance/_distance.o\ncreating build/lib.linux-x86_64-3.7/Orange/distance\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/distance/_distance.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/distance/_distance.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.preprocess._discretize' extension\ncreating build/temp.linux-x86_64-3.7/Orange/preprocess\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/preprocess/_discretize.c -o build/temp.linux-x86_64-3.7/Orange/preprocess/_discretize.o\ncreating build/lib.linux-x86_64-3.7/Orange/preprocess\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/preprocess/_discretize.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/preprocess/_discretize.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.preprocess._relieff' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/preprocess/_relieff.cpp -o build/temp.linux-x86_64-3.7/Orange/preprocess/_relieff.o\nc++ -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/preprocess/_relieff.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/preprocess/_relieff.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.projection._som' extension\ncreating build/temp.linux-x86_64-3.7/Orange/projection\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c Orange/projection/_som.c -o build/temp.linux-x86_64-3.7/Orange/projection/_som.o\ncreating build/lib.linux-x86_64-3.7/Orange/projection\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/projection/_som.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o build/lib.linux-x86_64-3.7/Orange/projection/_som.cpython-37m-x86_64-linux-gnu.so\ncopying build/lib.linux-x86_64-3.7/Orange/classification/_simple_tree.cpython-37m-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-3.7/Orange/widgets/utils/_grid_density.cpython-37m-x86_64-linux-gnu.so -> Orange/widgets/utils\ncopying build/lib.linux-x86_64-3.7/Orange/classification/_tree_scorers.cpython-37m-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-3.7/Orange/data/_contingency.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/data/_io.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/data/_valuecount.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/data/_variable.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/distance/_distance.cpython-37m-x86_64-linux-gnu.so -> Orange/distance\ncopying build/lib.linux-x86_64-3.7/Orange/preprocess/_discretize.cpython-37m-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-3.7/Orange/preprocess/_relieff.cpython-37m-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-3.7/Orange/projection/_som.cpython-37m-x86_64-linux-gnu.so -> Orange/projection\nrunning develop\nrunning egg_info\ncreating Orange3.egg-info\nwriting Orange3.egg-info/PKG-INFO\nwriting dependency_links to Orange3.egg-info/dependency_links.txt\nwriting entry points to Orange3.egg-info/entry_points.txt\nwriting requirements to Orange3.egg-info/requires.txt\nwriting top-level names to Orange3.egg-info/top_level.txt\nwriting manifest file 'Orange3.egg-info/SOURCES.txt'\nreading manifest file 'Orange3.egg-info/SOURCES.txt'\nreading manifest template 'MANIFEST.in'\nadding license file 'LICENSE'\nwriting manifest file 'Orange3.egg-info/SOURCES.txt'\nrunning build_ext\ncopying build/lib.linux-x86_64-3.7/Orange/classification/_simple_tree.cpython-37m-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-3.7/Orange/widgets/utils/_grid_density.cpython-37m-x86_64-linux-gnu.so -> Orange/widgets/utils\ncopying build/lib.linux-x86_64-3.7/Orange/classification/_tree_scorers.cpython-37m-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-3.7/Orange/data/_contingency.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/data/_io.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/data/_valuecount.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/data/_variable.cpython-37m-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-3.7/Orange/distance/_distance.cpython-37m-x86_64-linux-gnu.so -> Orange/distance\ncopying build/lib.linux-x86_64-3.7/Orange/preprocess/_discretize.cpython-37m-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-3.7/Orange/preprocess/_relieff.cpython-37m-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-3.7/Orange/projection/_som.cpython-37m-x86_64-linux-gnu.so -> Orange/projection\nCreating /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/Orange3.egg-link (link to .)\nAdding Orange3 3.37.0.dev0+fb23419 to easy-install.pth file\nInstalling orange-canvas script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b\nProcessing dependencies for Orange3==3.37.0.dev0+fb23419\nSearching for xlsxwriter\nReading https://pypi.org/simple/xlsxwriter/\nDownloading https://files.pythonhosted.org/packages/a7/ea/53d1fe468e63e092cf16e2c18d16f50c29851242f9dd12d6a66e0d7f0d02/XlsxWriter-3.2.0-py3-none-any.whl#sha256=ecfd5405b3e0e228219bcaf24c2ca0915e012ca9464a14048021d21a995d490e\nBest match: XlsxWriter 3.2.0\nProcessing XlsxWriter-3.2.0-py3-none-any.whl\nInstalling XlsxWriter-3.2.0-py3-none-any.whl to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages\nAdding XlsxWriter 3.2.0 to easy-install.pth file\nInstalling vba_extract.py script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/XlsxWriter-3.2.0-py3.7.egg\nSearching for xlrd>=1.2.0\nReading https://pypi.org/simple/xlrd/\nDownloading https://files.pythonhosted.org/packages/a6/0c/c2a72d51fe56e08a08acc85d13013558a2d793028ae7385448a6ccdfae64/xlrd-2.0.1-py2.py3-none-any.whl#sha256=6a33ee89877bd9abc1158129f6e94be74e2679636b8a205b43b85206c3f0bbdd\nBest match: xlrd 2.0.1\nProcessing xlrd-2.0.1-py2.py3-none-any.whl\nInstalling xlrd-2.0.1-py2.py3-none-any.whl to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages\nAdding xlrd 2.0.1 to easy-install.pth file\nInstalling runxlrd.py script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/xlrd-2.0.1-py3.7.egg\nSearching for xgboost>=1.7.4\nReading https://pypi.org/simple/xgboost/\nDownloading https://files.pythonhosted.org/packages/48/bc/05d7db90d421c5e3d681a12fd1eb087e37bf2e9bbe2b105422d6319ecc92/xgboost-2.1.3-py3-none-manylinux2014_x86_64.whl#sha256=32a43526208fe676527c698cb852e0e9515e6d7294143780e476d335290a131b\nBest match: xgboost 2.1.3\nProcessing xgboost-2.1.3-py3-none-manylinux2014_x86_64.whl\nInstalling xgboost-2.1.3-py3-none-manylinux2014_x86_64.whl to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages\nAdding xgboost 2.1.3 to easy-install.pth file\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/xgboost-2.1.3-py3.7-linux-x86_64.egg\nSearching for serverfiles\nReading https://pypi.org/simple/serverfiles/\nDownloading https://files.pythonhosted.org/packages/33/36/794fe1c4ed29b0f56a26d72e956baf0285d8fe9e29faf36797a28a8ad45b/serverfiles-0.3.1.tar.gz#sha256=5e10fc32e758791e3735bc08bce2ed4702a83b1e45ab96c5d59cc8aeecfbebe1\nBest match: serverfiles 0.3.1\nProcessing serverfiles-0.3.1.tar.gz\nWriting /tmp/easy_install-6m60_023/serverfiles-0.3.1/setup.cfg\nRunning serverfiles-0.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6m60_023/serverfiles-0.3.1/egg-dist-tmp-jx86vw_u\ncreating /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/serverfiles-0.3.1-py3.7.egg\nExtracting serverfiles-0.3.1-py3.7.egg to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages\nAdding serverfiles 0.3.1 to easy-install.pth file\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/serverfiles-0.3.1-py3.7.egg\nSearching for scipy>=1.9\nReading https://pypi.org/simple/scipy/\nDownloading https://files.pythonhosted.org/packages/d9/7b/2b8ac283cf32465ed08bc20a83d559fe7b174a484781702ba8accea001d6/scipy-1.15.0.tar.gz#sha256=300742e2cc94e36a2880ebe464a1c8b4352a7b0f3e36ec3d2ac006cdbe0219ac\nBest match: scipy 1.15.0\nProcessing scipy-1.15.0.tar.gz\nVerifying Orange installation...\n❌ Orange verification failed\nPython 3.7 installation failed, trying Python 3.8...\nAttempting installation with Python 3.8...\nrunning build_ext\nbuilding 'Orange.classification._simple_tree' extension\ncreating build/temp.linux-x86_64-cpython-38/Orange/classification\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/classification/_simple_tree.c -o build/temp.linux-x86_64-cpython-38/Orange/classification/_simple_tree.o\ncreating build/lib.linux-x86_64-cpython-38/Orange/classification\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/classification/_simple_tree.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/classification/_simple_tree.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.widgets.utils._grid_density' extension\ncreating build/temp.linux-x86_64-cpython-38/Orange/widgets/utils\nc++ -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/widgets/utils/_grid_density.cpp -o build/temp.linux-x86_64-cpython-38/Orange/widgets/utils/_grid_density.o\ncreating build/lib.linux-x86_64-cpython-38/Orange/widgets/utils\nc++ -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -pthread -shared build/temp.linux-x86_64-cpython-38/Orange/widgets/utils/_grid_density.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/widgets/utils/_grid_density.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.classification._tree_scorers' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/classification/_tree_scorers.c -o build/temp.linux-x86_64-cpython-38/Orange/classification/_tree_scorers.o\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/classification/_tree_scorers.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/classification/_tree_scorers.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.data._contingency' extension\ncreating build/temp.linux-x86_64-cpython-38/Orange/data\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/data/_contingency.c -o build/temp.linux-x86_64-cpython-38/Orange/data/_contingency.o\ncreating build/lib.linux-x86_64-cpython-38/Orange/data\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/data/_contingency.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/data/_contingency.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.data._io' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/data/_io.c -o build/temp.linux-x86_64-cpython-38/Orange/data/_io.o\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/data/_io.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/data/_io.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.data._valuecount' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/data/_valuecount.c -o build/temp.linux-x86_64-cpython-38/Orange/data/_valuecount.o\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/data/_valuecount.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/data/_valuecount.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.data._variable' extension\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/data/_variable.c -o build/temp.linux-x86_64-cpython-38/Orange/data/_variable.o\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/data/_variable.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/data/_variable.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.distance._distance' extension\ncreating build/temp.linux-x86_64-cpython-38/Orange/distance\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/distance/_distance.c -o build/temp.linux-x86_64-cpython-38/Orange/distance/_distance.o\ncreating build/lib.linux-x86_64-cpython-38/Orange/distance\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/distance/_distance.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/distance/_distance.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.preprocess._discretize' extension\ncreating build/temp.linux-x86_64-cpython-38/Orange/preprocess\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/preprocess/_discretize.c -o build/temp.linux-x86_64-cpython-38/Orange/preprocess/_discretize.o\ncreating build/lib.linux-x86_64-cpython-38/Orange/preprocess\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/preprocess/_discretize.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/preprocess/_discretize.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.preprocess._relieff' extension\nc++ -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/preprocess/_relieff.cpp -o build/temp.linux-x86_64-cpython-38/Orange/preprocess/_relieff.o\nc++ -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -pthread -shared build/temp.linux-x86_64-cpython-38/Orange/preprocess/_relieff.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/preprocess/_relieff.cpython-38-x86_64-linux-gnu.so\nbuilding 'Orange.projection._som' extension\ncreating build/temp.linux-x86_64-cpython-38/Orange/projection\ncc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/include/python3.8 -c Orange/projection/_som.c -o build/temp.linux-x86_64-cpython-38/Orange/projection/_som.o\ncreating build/lib.linux-x86_64-cpython-38/Orange/projection\ncc -pthread -shared -Wl,--exclude-libs,ALL build/temp.linux-x86_64-cpython-38/Orange/projection/_som.o -L/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib -lm -o build/lib.linux-x86_64-cpython-38/Orange/projection/_som.cpython-38-x86_64-linux-gnu.so\ncopying build/lib.linux-x86_64-cpython-38/Orange/classification/_simple_tree.cpython-38-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-cpython-38/Orange/widgets/utils/_grid_density.cpython-38-x86_64-linux-gnu.so -> Orange/widgets/utils\ncopying build/lib.linux-x86_64-cpython-38/Orange/classification/_tree_scorers.cpython-38-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_contingency.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_io.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_valuecount.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_variable.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/distance/_distance.cpython-38-x86_64-linux-gnu.so -> Orange/distance\ncopying build/lib.linux-x86_64-cpython-38/Orange/preprocess/_discretize.cpython-38-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-cpython-38/Orange/preprocess/_relieff.cpython-38-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-cpython-38/Orange/projection/_som.cpython-38-x86_64-linux-gnu.so -> Orange/projection\nrunning develop\nrunning egg_info\nwriting Orange3.egg-info/PKG-INFO\nwriting dependency_links to Orange3.egg-info/dependency_links.txt\nwriting entry points to Orange3.egg-info/entry_points.txt\nwriting requirements to Orange3.egg-info/requires.txt\nwriting top-level names to Orange3.egg-info/top_level.txt\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_math.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_math.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_math.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_math.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_math.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayscalars.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_math.h won't be automatically included in the manifest: the path must be relative\ndependency /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ufuncobject.h won't be automatically included in the manifest: the path must be relative\nreading manifest file 'Orange3.egg-info/SOURCES.txt'\nreading manifest template 'MANIFEST.in'\nadding license file 'LICENSE'\nwriting manifest file 'Orange3.egg-info/SOURCES.txt'\nrunning build_ext\ncopying build/lib.linux-x86_64-cpython-38/Orange/classification/_simple_tree.cpython-38-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-cpython-38/Orange/widgets/utils/_grid_density.cpython-38-x86_64-linux-gnu.so -> Orange/widgets/utils\ncopying build/lib.linux-x86_64-cpython-38/Orange/classification/_tree_scorers.cpython-38-x86_64-linux-gnu.so -> Orange/classification\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_contingency.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_io.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_valuecount.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/data/_variable.cpython-38-x86_64-linux-gnu.so -> Orange/data\ncopying build/lib.linux-x86_64-cpython-38/Orange/distance/_distance.cpython-38-x86_64-linux-gnu.so -> Orange/distance\ncopying build/lib.linux-x86_64-cpython-38/Orange/preprocess/_discretize.cpython-38-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-cpython-38/Orange/preprocess/_relieff.cpython-38-x86_64-linux-gnu.so -> Orange/preprocess\ncopying build/lib.linux-x86_64-cpython-38/Orange/projection/_som.cpython-38-x86_64-linux-gnu.so -> Orange/projection\nCreating /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/Orange3.egg-link (link to .)\nAdding Orange3 3.37.0.dev0+fb23419 to easy-install.pth file\nInstalling orange-canvas script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b\nProcessing dependencies for Orange3==3.37.0.dev0+fb23419\nSearching for XlsxWriter==3.2.0\nBest match: XlsxWriter 3.2.0\nAdding XlsxWriter 3.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for xlrd==2.0.1\nBest match: xlrd 2.0.1\nAdding xlrd 2.0.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for xgboost==2.1.3\nBest match: xgboost 2.1.3\nAdding xgboost 2.1.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for serverfiles==0.3.1\nBest match: serverfiles 0.3.1\nAdding serverfiles 0.3.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for scipy==1.10.1\nBest match: scipy 1.10.1\nAdding scipy 1.10.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for scikit-learn==1.3.2\nBest match: scikit-learn 1.3.2\nAdding scikit-learn 1.3.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for requests==2.32.3\nBest match: requests 2.32.3\nAdding requests 2.32.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for qtconsole==5.6.1\nBest match: qtconsole 5.6.1\nAdding qtconsole 5.6.1 to easy-install.pth file\nInstalling jupyter-qtconsole script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for PyYAML==6.0.2\nBest match: PyYAML 6.0.2\nAdding PyYAML 6.0.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for python-louvain==0.16\nBest match: python-louvain 0.16\nAdding python-louvain 0.16 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pyqtgraph==0.13.3\nBest match: pyqtgraph 0.13.3\nAdding pyqtgraph 0.13.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pygments==2.19.1\nBest match: pygments 2.19.1\nAdding pygments 2.19.1 to easy-install.pth file\nInstalling pygmentize script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pip==24.3.1\nBest match: pip 24.3.1\nAdding pip 24.3.1 to easy-install.pth file\nInstalling pip script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling pip3 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pandas==2.0.3\nBest match: pandas 2.0.3\nAdding pandas 2.0.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for packaging==24.2\nBest match: packaging 24.2\nAdding packaging 24.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for orange-widget-base==4.24.0\nBest match: orange-widget-base 4.24.0\nAdding orange-widget-base 4.24.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for orange-canvas-core==0.2.2\nBest match: orange-canvas-core 0.2.2\nAdding orange-canvas-core 0.2.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for openpyxl==3.1.5\nBest match: openpyxl 3.1.5\nAdding openpyxl 3.1.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for openTSNE==1.0.1\nBest match: openTSNE 1.0.1\nAdding openTSNE 1.0.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for numpy==1.24.4\nBest match: numpy 1.24.4\nAdding numpy 1.24.4 to easy-install.pth file\nInstalling f2py script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling f2py3 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling f2py3.8 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for networkx==3.1\nBest match: networkx 3.1\nAdding networkx 3.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for matplotlib==3.7.5\nBest match: matplotlib 3.7.5\nAdding matplotlib 3.7.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for keyrings.alt==5.0.2\nBest match: keyrings.alt 5.0.2\nAdding keyrings.alt 5.0.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for keyring==25.5.0\nBest match: keyring 25.5.0\nAdding keyring 25.5.0 to easy-install.pth file\nInstalling keyring script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for joblib==1.4.2\nBest match: joblib 1.4.2\nAdding joblib 1.4.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for httpx==0.28.1\nBest match: httpx 0.28.1\nAdding httpx 0.28.1 to easy-install.pth file\nInstalling httpx script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for chardet==5.2.0\nBest match: chardet 5.2.0\nAdding chardet 5.2.0 to easy-install.pth file\nInstalling chardetect script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for catboost==1.2.7\nBest match: catboost 1.2.7\nAdding catboost 1.2.7 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for Bottleneck==1.4.0\nBest match: Bottleneck 1.4.0\nAdding Bottleneck 1.4.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for baycomp==1.0.3\nBest match: baycomp 1.0.3\nAdding baycomp 1.0.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for AnyQt==0.2.0\nBest match: AnyQt 0.2.0\nAdding AnyQt 0.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for nvidia-nccl-cu12==2.24.3\nBest match: nvidia-nccl-cu12 2.24.3\nAdding nvidia-nccl-cu12 2.24.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for threadpoolctl==3.5.0\nBest match: threadpoolctl 3.5.0\nAdding threadpoolctl 3.5.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for certifi==2024.12.14\nBest match: certifi 2024.12.14\nAdding certifi 2024.12.14 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for urllib3==2.2.3\nBest match: urllib3 2.2.3\nAdding urllib3 2.2.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for idna==3.10\nBest match: idna 3.10\nAdding idna 3.10 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for charset-normalizer==3.4.1\nBest match: charset-normalizer 3.4.1\nAdding charset-normalizer 3.4.1 to easy-install.pth file\nInstalling normalizer script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for QtPy==2.4.2\nBest match: QtPy 2.4.2\nAdding QtPy 2.4.2 to easy-install.pth file\nInstalling qtpy script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for ipykernel==6.29.5\nBest match: ipykernel 6.29.5\nAdding ipykernel 6.29.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jupyter-client==8.6.3\nBest match: jupyter-client 8.6.3\nAdding jupyter-client 8.6.3 to easy-install.pth file\nInstalling jupyter-kernel script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling jupyter-kernelspec script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling jupyter-run script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jupyter-core==5.7.2\nBest match: jupyter-core 5.7.2\nAdding jupyter-core 5.7.2 to easy-install.pth file\nInstalling jupyter script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling jupyter-migrate script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling jupyter-troubleshoot script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for traitlets==5.14.3\nBest match: traitlets 5.14.3\nAdding traitlets 5.14.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for tzdata==2024.2\nBest match: tzdata 2024.2\nAdding tzdata 2024.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pytz==2024.2\nBest match: pytz 2024.2\nAdding pytz 2024.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for python-dateutil==2.9.0.post0\nBest match: python-dateutil 2.9.0.post0\nAdding python-dateutil 2.9.0.post0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for importlib-resources==6.4.5\nBest match: importlib-resources 6.4.5\nAdding importlib-resources 6.4.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for typing-extensions==4.12.2\nBest match: typing-extensions 4.12.2\nAdding typing-extensions 4.12.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/setuptools/_vendor\nSearching for importlib-metadata==8.5.0\nBest match: importlib-metadata 8.5.0\nAdding importlib-metadata 8.5.0 to easy-install.pth file\ndetected new path './setuptools/_vendor'\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for qasync==0.27.1\nBest match: qasync 0.27.1\nAdding qasync 0.27.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for dictdiffer==0.9.0\nBest match: dictdiffer 0.9.0\nAdding dictdiffer 0.9.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for requests-cache==1.2.1\nBest match: requests-cache 1.2.1\nAdding requests-cache 1.2.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for commonmark==0.9.1\nBest match: commonmark 0.9.1\nAdding commonmark 0.9.1 to easy-install.pth file\nInstalling cmark script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for docutils==0.20.1\nBest match: docutils 0.20.1\nAdding docutils 0.20.1 to easy-install.pth file\nInstalling docutils script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for et-xmlfile==2.0.0\nBest match: et-xmlfile 2.0.0\nAdding et-xmlfile 2.0.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pyparsing==3.1.4\nBest match: pyparsing 3.1.4\nAdding pyparsing 3.1.4 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pillow==10.4.0\nBest match: pillow 10.4.0\nAdding pillow 10.4.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for kiwisolver==1.4.7\nBest match: kiwisolver 1.4.7\nAdding kiwisolver 1.4.7 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for fonttools==4.55.3\nBest match: fonttools 4.55.3\nAdding fonttools 4.55.3 to easy-install.pth file\nInstalling fonttools script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling pyftmerge script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling pyftsubset script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling ttx script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for cycler==0.12.1\nBest match: cycler 0.12.1\nAdding cycler 0.12.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for contourpy==1.1.1\nBest match: contourpy 1.1.1\nAdding contourpy 1.1.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jaraco.context==6.0.1\nBest match: jaraco.context 6.0.1\nAdding jaraco.context 6.0.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jaraco.classes==3.4.0\nBest match: jaraco.classes 3.4.0\nAdding jaraco.classes 3.4.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jeepney==0.8.0\nBest match: jeepney 0.8.0\nAdding jeepney 0.8.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for SecretStorage==3.3.3\nBest match: SecretStorage 3.3.3\nAdding SecretStorage 3.3.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jaraco.functools==4.1.0\nBest match: jaraco.functools 4.1.0\nAdding jaraco.functools 4.1.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for httpcore==1.0.7\nBest match: httpcore 1.0.7\nAdding httpcore 1.0.7 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for anyio==4.5.2\nBest match: anyio 4.5.2\nAdding anyio 4.5.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for six==1.17.0\nBest match: six 1.17.0\nAdding six 1.17.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for plotly==5.24.1\nBest match: plotly 5.24.1\nAdding plotly 5.24.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for graphviz==0.20.3\nBest match: graphviz 0.20.3\nAdding graphviz 0.20.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for tornado==6.4.2\nBest match: tornado 6.4.2\nAdding tornado 6.4.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pyzmq==26.2.0\nBest match: pyzmq 26.2.0\nAdding pyzmq 26.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for psutil==6.1.1\nBest match: psutil 6.1.1\nAdding psutil 6.1.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for nest-asyncio==1.6.0\nBest match: nest-asyncio 1.6.0\nAdding nest-asyncio 1.6.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for matplotlib-inline==0.1.7\nBest match: matplotlib-inline 0.1.7\nAdding matplotlib-inline 0.1.7 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for ipython==8.12.3\nBest match: ipython 8.12.3\nAdding ipython 8.12.3 to easy-install.pth file\nInstalling ipython script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\nInstalling ipython3 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for debugpy==1.8.11\nBest match: debugpy 1.8.11\nAdding debugpy 1.8.11 to easy-install.pth file\nInstalling debugpy script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for comm==0.2.2\nBest match: comm 0.2.2\nAdding comm 0.2.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for platformdirs==4.3.6\nBest match: platformdirs 4.3.6\nAdding platformdirs 4.3.6 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for zipp==3.20.2\nBest match: zipp 3.20.2\nAdding zipp 3.20.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for url-normalize==1.4.3\nBest match: url-normalize 1.4.3\nAdding url-normalize 1.4.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for cattrs==24.1.2\nBest match: cattrs 24.1.2\nAdding cattrs 24.1.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for attrs==24.3.0\nBest match: attrs 24.3.0\nAdding attrs 24.3.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for backports.tarfile==1.2.0\nBest match: backports.tarfile 1.2.0\nbackports.tarfile 1.2.0 is already the active version in easy-install.pth\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/setuptools/_vendor\nSearching for more-itertools==10.5.0\nBest match: more-itertools 10.5.0\nAdding more-itertools 10.5.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for cryptography==44.0.0\nBest match: cryptography 44.0.0\nAdding cryptography 44.0.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for h11==0.14.0\nBest match: h11 0.14.0\nAdding h11 0.14.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for exceptiongroup==1.2.2\nBest match: exceptiongroup 1.2.2\nAdding exceptiongroup 1.2.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for sniffio==1.3.1\nBest match: sniffio 1.3.1\nAdding sniffio 1.3.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for tenacity==9.0.0\nBest match: tenacity 9.0.0\nAdding tenacity 9.0.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pexpect==4.9.0\nBest match: pexpect 4.9.0\nAdding pexpect 4.9.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for stack-data==0.6.3\nBest match: stack-data 0.6.3\nAdding stack-data 0.6.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for prompt-toolkit==3.0.48\nBest match: prompt-toolkit 3.0.48\nAdding prompt-toolkit 3.0.48 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pickleshare==0.7.5\nBest match: pickleshare 0.7.5\nAdding pickleshare 0.7.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for jedi==0.19.2\nBest match: jedi 0.19.2\nAdding jedi 0.19.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for decorator==5.1.1\nBest match: decorator 5.1.1\nAdding decorator 5.1.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for backcall==0.2.0\nBest match: backcall 0.2.0\nAdding backcall 0.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for cffi==1.17.1\nBest match: cffi 1.17.1\nAdding cffi 1.17.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for ptyprocess==0.7.0\nBest match: ptyprocess 0.7.0\nAdding ptyprocess 0.7.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pure-eval==0.2.3\nBest match: pure-eval 0.2.3\nAdding pure-eval 0.2.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for asttokens==3.0.0\nBest match: asttokens 3.0.0\nAdding asttokens 3.0.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for executing==2.1.0\nBest match: executing 2.1.0\nAdding executing 2.1.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for wcwidth==0.2.13\nBest match: wcwidth 0.2.13\nAdding wcwidth 0.2.13 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for parso==0.8.4\nBest match: parso 0.8.4\nAdding parso 0.8.4 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nSearching for pycparser==2.22\nBest match: pycparser 2.22\nAdding pycparser 2.22 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages\nFinished processing dependencies for Orange3==3.37.0.dev0+fb23419\nVerifying Orange installation...\n✅ Orange installation successful!\nSuccessfully installed orange using Python 3.8\n",
    "setup_res_stderr": "Using CPython 3.7.9\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 18 packages in 58ms\nPrepared 18 packages in 1ms\nInstalled 18 packages in 179ms\n + cython==0.29.37\n + exceptiongroup==1.2.2\n + importlib-metadata==6.7.0\n + iniconfig==2.0.0\n + numpy==1.17.5\n + packaging==24.0\n + pluggy==1.2.0\n + pyqt5==5.15.10\n + pyqt5-qt5==5.15.16\n + pyqt5-sip==12.13.0\n + pyqtwebengine==5.15.6\n + pyqtwebengine-qt5==5.15.16\n + pytest==7.4.4\n + setuptools==59.8.0\n + tomli==2.0.1\n + typing-extensions==4.7.1\n + wheel==0.42.0\n + zipp==3.15.0\n  × No solution found when resolving dependencies:\n  ╰─▶ Because the current Python version (3.7.9) does not satisfy Python>=3.8 and all of:\n          pandas>=1.4.0,<=1.4.4\n          pandas>=1.5.1,<=1.5.3\n          pandas>=2.0.1,<=2.0.3\n      depend on Python>=3.8, we can conclude that all of:\n          pandas>=1.4.0,<=1.4.4\n          pandas>=1.5.1,<=1.5.3\n          pandas>=2.0.1,<=2.0.3\n       cannot be used.\n      And because only the following versions of pandas are available:\n          pandas<=1.4.0\n          pandas==1.4.1\n          pandas==1.4.2\n          pandas==1.4.3\n          pandas==1.4.4\n          pandas==1.5.0\n          pandas==1.5.1\n          pandas==1.5.2\n          pandas==1.5.3\n          pandas==2.0.0\n          pandas==2.0.1\n          pandas==2.0.2\n          pandas==2.0.3\n          pandas==2.1.0\n          pandas==2.1.1\n          pandas==2.1.2\n          pandas==2.1.3\n          pandas==2.1.4\n          pandas==2.2.0\n          pandas==2.2.1\n          pandas==2.2.2\n          pandas==2.2.3\n      we can conclude that all of:\n          pandas>=1.4.0,<1.5.0\n          pandas>1.5.0,<2.0.0\n          pandas>2.0.0,<2.1.0\n       cannot be used. (1)\n\n      Because the current Python version (3.7.9) does not satisfy Python>=3.9 and pandas>=2.1.0 depends on Python>=3.9, we can\n      conclude that pandas>=2.1.0 cannot be used.\n      And because we know from (1) that all of:\n          pandas>=1.4.0,<1.5.0\n          pandas>1.5.0,<2.0.0\n          pandas>2.0.0,<2.1.0\n       cannot be used, we can conclude that all of:\n          pandas>=1.4.0,<1.5.0\n          pandas>1.5.0,<2.0.0\n          pandas>2.0.0\n       cannot be used.\n      And because you require one of:\n          pandas>=1.4.0,<1.5.0\n          pandas>1.5.0,<2.0.0\n          pandas>2.0.0\n      we can conclude that your requirements are unsatisfiable.\n\n      hint: Pre-releases are available for `pandas` in the requested range (e.g., 2.2.0rc0), but pre-releases weren't enabled\n      (try: `--prerelease=allow`)\n  × No solution found when resolving dependencies:\n  ╰─▶ Because the current Python version (3.7.9) does not satisfy Python>=3.8 and pyqtgraph>=0.13.1,<=0.13.3 depends on\n      Python>=3.8, we can conclude that pyqtgraph>=0.13.1,<=0.13.3 cannot be used.\n      And because only the following versions of pyqtgraph are available:\n          pyqtgraph<=0.13.1\n          pyqtgraph==0.13.2\n          pyqtgraph==0.13.3\n          pyqtgraph==0.13.4\n          pyqtgraph==0.13.5\n          pyqtgraph==0.13.6\n          pyqtgraph==0.13.7\n      we can conclude that pyqtgraph>=0.13.1,<0.13.4 cannot be used. (1)\n\n      Because the current Python version (3.7.9) does not satisfy Python>=3.9 and pyqtgraph>=0.13.4 depends on Python>=3.9, we\n      can conclude that pyqtgraph>=0.13.4 cannot be used.\n      And because we know from (1) that pyqtgraph>=0.13.1,<0.13.4 cannot be used, we can conclude that pyqtgraph>=0.13.1 cannot\n      be used.\n      And because you require pyqtgraph>=0.13.1, we can conclude that your requirements are unsatisfiable.\nResolved 2 packages in 47ms\nInstalled 2 packages in 7ms\n + psycopg2==2.9.9\n + pymssql==2.3.2\nResolved 5 packages in 20ms\nInstalled 4 packages in 197ms\n + joblib==1.3.2\n + scikit-learn==1.0.2\n + scipy==1.7.3\n + threadpoolctl==3.1.0\nOrange/classification/_simple_tree.c: In function ‘gain_ratio_c’:\nOrange/classification/_simple_tree.c:63:9: warning: implicit declaration of function ‘qsort_r’; did you mean ‘qsort’? [-Wimplicit-function-declaration]\n   63 |         qsort_r(base, nel, size, compar, thunk)\n      |         ^~~~~~~\nOrange/classification/_simple_tree.c:147:9: note: in expansion of macro ‘QSORT_R_FUNC’\n  147 |         QSORT_R_FUNC(examples, size, sizeof(struct Example), (void*) &compar_attr, compar_examples);\n      |         ^~~~~~~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/classification/_tree_scorers.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_2find_threshold_entropy’:\nOrange/classification/_tree_scorers.c:3426:27: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 3426 |   __pyx_t_7 = ((__pyx_v_N <= __pyx_v_min_leaf) != 0);\n      |                           ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_6find_threshold_MSE’:\nOrange/classification/_tree_scorers.c:4872:27: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 4872 |   __pyx_t_1 = ((__pyx_v_N <= __pyx_v_min_leaf) != 0);\n      |                           ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_8find_binarization_MSE’:\nOrange/classification/_tree_scorers.c:5618:43: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5618 |           for (__pyx_t_19 = 0; __pyx_t_19 < __pyx_t_18; __pyx_t_19+=1) {\n      |                                           ^\nOrange/classification/_tree_scorers.c:5739:39: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5739 |           __pyx_t_20 = ((__pyx_v_left >= __pyx_v_min_leaf) != 0);\n      |                                       ^~\nOrange/classification/_tree_scorers.c:5745:53: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5745 |           __pyx_t_20 = (((__pyx_v_N - __pyx_v_left) >= __pyx_v_min_leaf) != 0);\n      |                                                     ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_14compute_predictions_csr’:\nOrange/classification/_tree_scorers.c:7206:41: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 7206 |         for (__pyx_t_12 = 0; __pyx_t_12 < __pyx_t_11; __pyx_t_12+=1) {\n      |                                         ^\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_16compute_predictions_csc’:\nOrange/classification/_tree_scorers.c:7863:41: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 7863 |         for (__pyx_t_12 = 0; __pyx_t_12 < __pyx_t_11; __pyx_t_12+=1) {\n      |                                         ^\nOrange/classification/_tree_scorers.c:7923:150: warning: comparison of integer expressions of different signedness: ‘__pyx_t_5numpy_int32_t’ {aka ‘int’} and ‘unsigned int’ [-Wsign-compare]\n 7923 |               __pyx_t_16 = (((*((__pyx_t_5numpy_int32_t const  *) ( /* dim=0 */ (__pyx_v_indices.data + __pyx_t_15 * __pyx_v_indices.strides[0]) ))) != __pyx_v_i) != 0);\n      |                                                                                                                                                      ^~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/data/_contingency.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/data/_io.c: In function ‘__pyx_pf_6Orange_4data_3_io_sparse_read_float.constprop’:\nOrange/data/_io.c:4120:43: warning: ‘__pyx_v_decs’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4120 |           __pyx_v_value = ((__pyx_v_value * __pyx_v_decs) + (__pyx_v_c & 0xf));\n      |                            ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/data/_io.c:4318:24: warning: ‘__pyx_v_value’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4318 |           __pyx_t_10 = PyFloat_FromDouble(__pyx_v_value); if (unlikely(!__pyx_t_10)) __PYX_ERR(0, 223, __pyx_L5_error)\n      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:683:40: warning: ‘__pyx_v_attr_index’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n  683 |   #define PyInt_FromLong               PyLong_FromLong\n      |                                        ^~~~~~~~~~~~~~~\nOrange/data/_io.c:1590:7: note: ‘__pyx_v_attr_index’ was declared here\n 1590 |   int __pyx_v_attr_index;\n      |       ^~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:3183:25: warning: ‘__pyx_v_atomp’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3183 |           __pyx_v_atomp = (__pyx_v_atomp + 1);\n      |           ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/data/_valuecount.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/distance/_distance.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/preprocess/_discretize.c:774:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/preprocess/_relieff.cpp:790:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/projection/_som.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_5update_hex’:\nOrange/projection/_som.c:3944:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3944 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:3943:7: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3943 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pf_6Orange_10projection_4_som_8update_sparse.constprop’:\nOrange/projection/_som.c:5094:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5094 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:5093:7: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5093 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_1get_winners’:\nOrange/projection/_som.c:3244:223: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3244 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_23, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_24, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_y;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:2892:7: note: ‘__pyx_v_win_y’ was declared here\n 2892 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:3233:223: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3233 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_24, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_23, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_x;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:2891:7: note: ‘__pyx_v_win_x’ was declared here\n 2891 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pf_6Orange_10projection_4_som_10update_sparse_hex.constprop’:\nOrange/projection/_som.c:5842:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5842 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:5841:7: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5841 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_7get_winners_sparse’:\nOrange/projection/_som.c:4871:223: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4871 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_27, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_28, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_y;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:4389:7: note: ‘__pyx_v_win_y’ was declared here\n 4389 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:4860:223: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4860 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_28, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_27, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_x;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:4388:7: note: ‘__pyx_v_win_x’ was declared here\n 4388 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_3update’:\nOrange/projection/_som.c:3438:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3438 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/setuptools/command/easy_install.py:159: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.\n  EasyInstallDeprecationWarning,\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n  setuptools.SetuptoolsDeprecationWarning,\nwarning: no files found matching 'README*' under directory 'Orange'\nwarning: no files found matching 'LICENSE' under directory 'Orange'\nwarning: no files found matching '*.basket' under directory 'Orange/datasets'\nwarning: no files found matching '*.svg' under directory 'Orange/canvas'\nwarning: no files found matching '*.js' under directory 'Orange/widgets'\nwarning: no files found matching '*.css' under directory 'Orange/widgets'\nwarning: no files found matching '*.html' under directory 'Orange/widgets'\nwarning: no files found matching '*.svg' under directory 'distribute'\nwarning: no files found matching 'COPYING'\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py:119: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release\n  PkgResourcesDeprecationWarning,\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n  setuptools.SetuptoolsDeprecationWarning,\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py:119: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release\n  PkgResourcesDeprecationWarning,\nerror: Couldn't find a setup script in /tmp/easy_install-pwqxm68o/scipy-1.15.0.tar.gz\nUsing CPython 3.8.20\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 15 packages in 123ms\nPrepared 15 packages in 1ms\nInstalled 15 packages in 77ms\n + cython==0.29.37\n + exceptiongroup==1.2.2\n + iniconfig==2.0.0\n + numpy==1.24.4\n + packaging==24.2\n + pluggy==1.5.0\n + pyqt5==5.15.11\n + pyqt5-qt5==5.15.16\n + pyqt5-sip==12.15.0\n + pyqtwebengine==5.15.7\n + pyqtwebengine-qt5==5.15.16\n + pytest==8.3.4\n + setuptools==75.3.0\n + tomli==2.2.1\n + wheel==0.45.1\nResolved 64 packages in 125ms\nInstalled 61 packages in 1.03s\n + anyio==4.5.2\n + backports-tarfile==1.2.0\n + baycomp==1.0.3\n + bottleneck==1.4.0\n + catboost==1.2.7\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + contourpy==1.1.1\n + cryptography==44.0.0\n + cycler==0.12.1\n + et-xmlfile==2.0.0\n + fonttools==4.55.3\n + graphviz==0.20.3\n + h11==0.14.0\n + httpcore==1.0.7\n + httpx==0.28.1\n + idna==3.10\n + importlib-metadata==8.5.0\n + importlib-resources==6.4.5\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + joblib==1.4.2\n + keyring==25.5.0\n + keyrings-alt==5.0.2\n + kiwisolver==1.4.7\n + matplotlib==3.7.5\n + more-itertools==10.5.0\n + networkx==3.1\n + nvidia-nccl-cu12==2.24.3\n + openpyxl==3.1.5\n + opentsne==1.0.1\n + pandas==2.0.3\n + pillow==10.4.0\n + pip==24.3.1\n + plotly==5.24.1\n + pycparser==2.22\n + pyparsing==3.1.4\n + python-dateutil==2.9.0.post0\n + python-louvain==0.16\n + pytz==2024.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + scikit-learn==1.3.2\n + scipy==1.10.1\n + secretstorage==3.3.3\n + serverfiles==0.3.1\n + six==1.17.0\n + sniffio==1.3.1\n + tenacity==9.0.0\n + threadpoolctl==3.5.0\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + urllib3==2.2.3\n + xgboost==2.1.3\n + xlrd==2.0.1\n + xlsxwriter==3.2.0\n + zipp==3.20.2\nResolved 63 packages in 59ms\nInstalled 41 packages in 118ms\n + anyqt==0.2.0\n + asttokens==3.0.0\n + attrs==24.3.0\n + backcall==0.2.0\n + cattrs==24.1.2\n + comm==0.2.2\n + commonmark==0.9.1\n + debugpy==1.8.11\n + decorator==5.1.1\n + dictdiffer==0.9.0\n + docutils==0.20.1\n + executing==2.1.0\n + ipykernel==6.29.5\n + ipython==8.12.3\n + jedi==0.19.2\n + jupyter-client==8.6.3\n + jupyter-core==5.7.2\n + matplotlib-inline==0.1.7\n + nest-asyncio==1.6.0\n + orange-canvas-core==0.2.2\n + orange-widget-base==4.24.0\n + parso==0.8.4\n + pexpect==4.9.0\n + pickleshare==0.7.5\n + platformdirs==4.3.6\n + prompt-toolkit==3.0.48\n + psutil==6.1.1\n + ptyprocess==0.7.0\n + pure-eval==0.2.3\n + pygments==2.19.1\n + pyqtgraph==0.13.3\n + pyzmq==26.2.0\n + qasync==0.27.1\n + qtconsole==5.6.1\n + qtpy==2.4.2\n + requests-cache==1.2.1\n + stack-data==0.6.3\n + tornado==6.4.2\n + traitlets==5.14.3\n + url-normalize==1.4.3\n + wcwidth==0.2.13\nResolved 2 packages in 2ms\nInstalled 2 packages in 38ms\n + psycopg2==2.9.10\n + pymssql==2.3.2\nOrange/classification/_simple_tree.c: In function ‘gain_ratio_c’:\nOrange/classification/_simple_tree.c:63:9: warning: implicit declaration of function ‘qsort_r’; did you mean ‘qsort’? [-Wimplicit-function-declaration]\n   63 |         qsort_r(base, nel, size, compar, thunk)\n      |         ^~~~~~~\nOrange/classification/_simple_tree.c:147:9: note: in expansion of macro ‘QSORT_R_FUNC’\n  147 |         QSORT_R_FUNC(examples, size, sizeof(struct Example), (void*) &compar_attr, compar_examples);\n      |         ^~~~~~~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/classification/_tree_scorers.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_2find_threshold_entropy’:\nOrange/classification/_tree_scorers.c:3426:27: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 3426 |   __pyx_t_7 = ((__pyx_v_N <= __pyx_v_min_leaf) != 0);\n      |                           ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_6find_threshold_MSE’:\nOrange/classification/_tree_scorers.c:4872:27: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 4872 |   __pyx_t_1 = ((__pyx_v_N <= __pyx_v_min_leaf) != 0);\n      |                           ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_8find_binarization_MSE’:\nOrange/classification/_tree_scorers.c:5618:43: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5618 |           for (__pyx_t_19 = 0; __pyx_t_19 < __pyx_t_18; __pyx_t_19+=1) {\n      |                                           ^\nOrange/classification/_tree_scorers.c:5739:39: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5739 |           __pyx_t_20 = ((__pyx_v_left >= __pyx_v_min_leaf) != 0);\n      |                                       ^~\nOrange/classification/_tree_scorers.c:5745:53: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5745 |           __pyx_t_20 = (((__pyx_v_N - __pyx_v_left) >= __pyx_v_min_leaf) != 0);\n      |                                                     ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_14compute_predictions_csr’:\nOrange/classification/_tree_scorers.c:7206:41: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 7206 |         for (__pyx_t_12 = 0; __pyx_t_12 < __pyx_t_11; __pyx_t_12+=1) {\n      |                                         ^\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_16compute_predictions_csc’:\nOrange/classification/_tree_scorers.c:7863:41: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 7863 |         for (__pyx_t_12 = 0; __pyx_t_12 < __pyx_t_11; __pyx_t_12+=1) {\n      |                                         ^\nOrange/classification/_tree_scorers.c:7923:150: warning: comparison of integer expressions of different signedness: ‘__pyx_t_5numpy_int32_t’ {aka ‘int’} and ‘unsigned int’ [-Wsign-compare]\n 7923 |               __pyx_t_16 = (((*((__pyx_t_5numpy_int32_t const  *) ( /* dim=0 */ (__pyx_v_indices.data + __pyx_t_15 * __pyx_v_indices.strides[0]) ))) != __pyx_v_i) != 0);\n      |                                                                                                                                                      ^~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/data/_contingency.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/data/_io.c: In function ‘__pyx_pf_6Orange_4data_3_io_sparse_read_float.constprop’:\nOrange/data/_io.c:4120:43: warning: ‘__pyx_v_decs’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4120 |           __pyx_v_value = ((__pyx_v_value * __pyx_v_decs) + (__pyx_v_c & 0xf));\n      |                            ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/data/_io.c:4318:24: warning: ‘__pyx_v_value’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4318 |           __pyx_t_10 = PyFloat_FromDouble(__pyx_v_value); if (unlikely(!__pyx_t_10)) __PYX_ERR(0, 223, __pyx_L5_error)\n      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:683:40: warning: ‘__pyx_v_attr_index’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n  683 |   #define PyInt_FromLong               PyLong_FromLong\n      |                                        ^~~~~~~~~~~~~~~\nOrange/data/_io.c:1590:7: note: ‘__pyx_v_attr_index’ was declared here\n 1590 |   int __pyx_v_attr_index;\n      |       ^~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:3183:25: warning: ‘__pyx_v_atomp’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3183 |           __pyx_v_atomp = (__pyx_v_atomp + 1);\n      |           ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/data/_valuecount.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/distance/_distance.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/preprocess/_discretize.c:774:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/preprocess/_relieff.cpp:790:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,\n                 from Orange/projection/_som.c:775:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_5update_hex’:\nOrange/projection/_som.c:3944:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3944 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:3943:7: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3943 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pf_6Orange_10projection_4_som_8update_sparse.constprop’:\nOrange/projection/_som.c:5094:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5094 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:5093:7: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5093 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_3update’:\nOrange/projection/_som.c:3438:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3438 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_1get_winners’:\nOrange/projection/_som.c:3244:223: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3244 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_23, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_24, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_y;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:2892:7: note: ‘__pyx_v_win_y’ was declared here\n 2892 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:3233:223: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3233 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_24, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_23, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_x;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:2891:7: note: ‘__pyx_v_win_x’ was declared here\n 2891 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pf_6Orange_10projection_4_som_10update_sparse_hex.constprop’:\nOrange/projection/_som.c:5842:7: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5842 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:5841:7: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 5841 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c: In function ‘__pyx_pw_6Orange_10projection_4_som_7get_winners_sparse’:\nOrange/projection/_som.c:4871:223: warning: ‘__pyx_v_win_y’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4871 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_27, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_28, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_y;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:4389:7: note: ‘__pyx_v_win_y’ was declared here\n 4389 |   int __pyx_v_win_y;\n      |       ^~~~~~~~~~~~~\nOrange/projection/_som.c:4860:223: warning: ‘__pyx_v_win_x’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4860 |           *__Pyx_BufPtrStrided2d(__pyx_t_5numpy_int16_t *, __pyx_pybuffernd_winners.rcbuffer->pybuffer.buf, __pyx_t_28, __pyx_pybuffernd_winners.diminfo[0].strides, __pyx_t_27, __pyx_pybuffernd_winners.diminfo[1].strides) = __pyx_v_win_x;\n      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~\nOrange/projection/_som.c:4388:7: note: ‘__pyx_v_win_x’ was declared here\n 4388 |   int __pyx_v_win_x;\n      |       ^~~~~~~~~~~~~\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_suite'\n  warnings.warn(msg)\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_suite'\n  warnings.warn(msg)\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/setuptools/command/develop.py:41: EasyInstallDeprecationWarning: easy_install command is deprecated.\n!!\n\n        ********************************************************************************\n        Please avoid running ``setup.py`` and ``easy_install``.\n        Instead, use pypa/build, pypa/installer or other\n        standards-based tools.\n\n        See https://github.com/pypa/setuptools/issues/917 for details.\n        ********************************************************************************\n\n!!\n  easy_install.initialize_options(self)\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/.venv/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\n!!\n\n        ********************************************************************************\n        Please avoid running ``setup.py`` directly.\n        Instead, use pypa/build, pypa/installer or other\n        standards-based tools.\n\n        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\n        ********************************************************************************\n\n!!\n  self.initialize_options()\nwarning: no files found matching 'README*' under directory 'Orange'\nwarning: no files found matching 'LICENSE' under directory 'Orange'\nwarning: no files found matching '*.basket' under directory 'Orange/datasets'\nwarning: no files found matching '*.svg' under directory 'Orange/canvas'\nwarning: no files found matching '*.js' under directory 'Orange/widgets'\nwarning: no files found matching '*.css' under directory 'Orange/widgets'\nwarning: no files found matching '*.html' under directory 'Orange/widgets'\nwarning: no files found matching '*.svg' under directory 'distribute'\nwarning: no files found matching 'COPYING'\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.8.20, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b\nconfigfile: pyproject.toml\nplugins: anyio-4.5.2, typeguard-4.3.0\ncollected 40 items\n\nr2e_tests/test_1.py sssE...............F....................             [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_filename ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/tests/__init__.py, line 68\n  def test_filename(path):\nE       fixture 'path' not found\n>       available fixtures: anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/tests/__init__.py:68\n=================================== FAILURES ===================================\n_____________________ TestOWTestAndScore.test_minimum_size _____________________\n\nself = <r2e_tests.test_1.TestOWTestAndScore testMethod=test_minimum_size>\n\n    def test_minimum_size(self):\n        widget = getattr(self, \"widget\", None)\n        if widget is None:\n            self.skipTest(\"minimum size not tested as .widget was not set\")\n>       self.check_minimum_size(widget)\n\n.venv/lib/python3.8/site-packages/orangewidget/tests/base.py:242: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n.venv/lib/python3.8/site-packages/orangewidget/tests/base.py:615: in check_minimum_size\n    self.assertLess(min_size.width(), 800)\nE   AssertionError: 1071 not less than 800\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_CrossValidationByFeature\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_addon_scorers\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_basic\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_bad_scores\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_bad_slots\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_binary_score\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_requires_cv\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_requires_multiple_models\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_copy_to_clipboard\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_data_errors\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_fill_table\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_image_export\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_memory_error\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_migrate_removes_invalid_contexts\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_migrate_shown_scores\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_msg_base_class\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_multi_target_input\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_multiple_learners\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_multiple_targets_pls\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_nan_on_comparison\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_no_stratification\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_one_class_value\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_resort_on_data_change\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_constant\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_cross_validation\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_advanced\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_bad\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_bad2\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_overfitted\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_target_changing\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_testOnTest\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_testOnTest_incompatible_domain\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_too_many_folds\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_unique_output_domain\nPASSED r2e_tests/test_1.py::TestHelpers::test_results_one_vs_rest\nSKIPPED [1] .venv/lib/python3.8/site-packages/orangewidget/tests/base.py:244: image exporting not tested as .widget was not set\nSKIPPED [1] .venv/lib/python3.8/site-packages/orangewidget/tests/base.py:238: minimum size not tested as .widget was not set\nSKIPPED [1] .venv/lib/python3.8/site-packages/orangewidget/tests/base.py:250: msg base class not tested as .widget was not set\nERROR r2e_tests/test_1.py::test_filename\nFAILED r2e_tests/test_1.py::TestOWTestAndScore::test_minimum_size - Assertion...\n=============== 1 failed, 35 passed, 3 skipped, 1 error in 9.20s ===============\nWidgets left: 0    Max widgets: 6020 \n\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.8.20, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b\nconfigfile: pyproject.toml\nplugins: anyio-4.5.2, typeguard-4.3.0\ncollected 40 items\n\nr2e_tests/test_1.py sssE...............F...F................             [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_filename ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/tests/__init__.py, line 68\n  def test_filename(path):\nE       fixture 'path' not found\n>       available fixtures: anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/tests/__init__.py:68\n=================================== FAILURES ===================================\n_____________________ TestOWTestAndScore.test_minimum_size _____________________\n\nself = <r2e_tests.test_1.TestOWTestAndScore testMethod=test_minimum_size>\n\n    def test_minimum_size(self):\n        widget = getattr(self, \"widget\", None)\n        if widget is None:\n            self.skipTest(\"minimum size not tested as .widget was not set\")\n>       self.check_minimum_size(widget)\n\n.venv/lib/python3.8/site-packages/orangewidget/tests/base.py:242: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n.venv/lib/python3.8/site-packages/orangewidget/tests/base.py:615: in check_minimum_size\n    self.assertLess(min_size.width(), 800)\nE   AssertionError: 1071 not less than 800\n_________________ TestOWTestAndScore.test_multiple_targets_pls _________________\n\nself = <r2e_tests.test_1.TestOWTestAndScore testMethod=test_multiple_targets_pls>\n\n    def test_multiple_targets_pls(self):\n        housing = Table(\"housing\")\n        class_vars = [housing.domain.class_var, housing.domain.attributes[0]]\n        domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)\n        multiple_targets_data = housing.transform(domain)\n    \n        self.widget.error = Mock()\n        self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)\n        self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())\n>       self.wait_until_finished()\n\nr2e_tests/test_1.py:781: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n.venv/lib/python3.8/site-packages/orangewidget/tests/base.py:526: in wait_until_finished\n    self.assertTrue(\nE   AssertionError: False is not true : Did not finish in the specified 5000ms timeout\n------------------------------ Captured log call -------------------------------\nERROR    Orange.widgets.evaluate.owtestandscore:owtestandscore.py:1084 testing error (in __task_complete):\nTraceback (most recent call last):\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/widgets/evaluate/owtestandscore.py\", line 1081, in __task_complete\n    results = f.result()    # type: Results\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib/python3.8/concurrent/futures/_base.py\", line 437, in result\n    return self.__get_result()\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/lib/python3.8/concurrent/futures/_base.py\", line 389, in __get_result\n    raise self._exception\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/widgets/utils/concurrent.py\", line 175, in run\n    result = func(*args, **kwargs)\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/widgets/evaluate/owtestandscore.py\", line 1024, in replace_learners\n    res = evalfunc(*args, **kwargs)\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/evaluation/testing.py\", line 475, in __call__\n    self._collect_part_results(results, part_results)\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_fb23419433230b6b38cab3e3e3dbb078cd95f54b/Orange/evaluation/testing.py\", line 547, in _collect_part_results\n    results.predicted[res.learner_i][result_slice] = res.values\nValueError: could not broadcast input array from shape (102,2) into shape (102,)\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_CrossValidationByFeature\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_addon_scorers\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_basic\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_bad_scores\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_bad_slots\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_binary_score\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_requires_cv\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_comparison_requires_multiple_models\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_copy_to_clipboard\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_data_errors\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_fill_table\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_image_export\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_memory_error\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_migrate_removes_invalid_contexts\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_migrate_shown_scores\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_msg_base_class\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_multi_target_input\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_multiple_learners\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_nan_on_comparison\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_no_stratification\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_one_class_value\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_resort_on_data_change\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_constant\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_cross_validation\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_advanced\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_bad\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_bad2\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_scores_log_reg_overfitted\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_target_changing\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_testOnTest\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_testOnTest_incompatible_domain\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_too_many_folds\nPASSED r2e_tests/test_1.py::TestOWTestAndScore::test_unique_output_domain\nPASSED r2e_tests/test_1.py::TestHelpers::test_results_one_vs_rest\nSKIPPED [1] .venv/lib/python3.8/site-packages/orangewidget/tests/base.py:244: image exporting not tested as .widget was not set\nSKIPPED [1] .venv/lib/python3.8/site-packages/orangewidget/tests/base.py:238: minimum size not tested as .widget was not set\nSKIPPED [1] .venv/lib/python3.8/site-packages/orangewidget/tests/base.py:250: msg base class not tested as .widget was not set\nERROR r2e_tests/test_1.py::test_filename\nFAILED r2e_tests/test_1.py::TestOWTestAndScore::test_minimum_size - Assertion...\nFAILED r2e_tests/test_1.py::TestOWTestAndScore::test_multiple_targets_pls - A...\n============== 2 failed, 34 passed, 3 skipped, 1 error in 11.92s ===============\nWidgets left: 0    Max widgets: 6020 \n\n",
    "old_commit_res_stderr": ""
}
Orange/evaluation/testing.py Orange/widgets/evaluate/tests/test_owtestandscore.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
### ValueError When Evaluating Models with Multiple Targets

**Description**
When evaluating a model using data with multiple target variables, the evaluation process fails with a `ValueError` due to a shape mismatch. The system currently does not support multiple targets, leading to unexpected errors during evaluation.

**Example Code**
```python
housing = Table("housing")
class_vars = [housing.domain.class_var, housing.domain.attributes[0]]
domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)
multiple_targets_data = housing.transform(domain)

self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)
self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())
self.wait_until_finished()
```

**Expected Behavior**
The evaluation should either handle multiple targets correctly or provide a clear and descriptive error message indicating that multiple targets are not supported.

**Actual Behavior**
A `ValueError` is raised with the message:
```
ValueError: could not broadcast input array from shape (102,2) into shape (102,)
```
This error occurs because the evaluation process expects a single target dimension but receives multiple, causing a shape mismatch and subsequent failure.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### Orange/evaluation/testing.py
# __new__ methods have different arguments
# pylint: disable=arguments-differ
from warnings import warn
from collections import namedtuple
from itertools import chain
from time import time

import numpy as np

import sklearn.model_selection as skl

from Orange.data import Domain, ContinuousVariable, DiscreteVariable
from Orange.data.util import get_unique_names

__all__ = ["Results", "CrossValidation", "LeaveOneOut", "TestOnTrainingData",
           "ShuffleSplit", "TestOnTestData", "sample", "CrossValidationFeature"]

_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',
                                       'failed', 'n_values', 'values',
                                       'probs', 'train_time', 'test_time'))


def _identity(x):
    return x


def _mp_worker(fold_i, train_data, test_data, learner_i, learner,
               store_models):
    predicted, probs, model, failed = None, None, None, False
    train_time, test_time = None, None
    try:
        if not train_data or not test_data:
            raise RuntimeError('Test fold is empty')
        # training
        t0 = time()
        model = learner(train_data)
        train_time = time() - t0
        t0 = time()
        # testing
        class_var = train_data.domain.class_var
        if class_var and class_var.is_discrete:
            predicted, probs = model(test_data, model.ValueProbs)
        else:
            predicted = model(test_data, model.Value)
        test_time = time() - t0
    # Different models can fail at any time raising any exception
    except Exception as ex:  # pylint: disable=broad-except
        failed = ex
    return _MpResults(fold_i, learner_i, store_models and model,
                      failed, len(test_data), predicted, probs,
                      train_time, test_time)


class Results:
    """
    Class for storing predictions in model testing.

    Attributes:
        data (Optional[Table]):  Data used for testing.

        models (Optional[List[Model]]): A list of induced models.

        row_indices (np.ndarray): Indices of rows in `data` that were used in
            testing, stored as a numpy vector of length `nrows`.
            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer
            to the target value of instance, that is, the i-th test instance
            is `data[row_indices[i]]`, its actual class is `actual[i]`, and
            the prediction by m-th method is `predicted[m, i]`.

        nrows (int): The number of test instances (including duplicates);
            `nrows` equals the length of `row_indices` and `actual`, and the
            second dimension of `predicted` and `probabilities`.

        actual (np.ndarray): true values of target variable in a vector of
            length `nrows`.

        predicted (np.ndarray): predicted values of target variable in an array
            of shape (number-of-methods, `nrows`)

        probabilities (Optional[np.ndarray]): predicted probabilities
            (for discrete target variables) in an array of shape
            (number-of-methods, `nrows`, number-of-classes)

        folds (List[Slice or List[int]]): a list of indices (or slice objects)
            corresponding to testing data subsets, that is,
            `row_indices[folds[i]]` contains row indices used in fold i, so
            `data[row_indices[folds[i]]]` is the corresponding testing data

        train_time (np.ndarray): training times of batches

        test_time (np.ndarray): testing times of batches
    """
    def __init__(self, data=None, *,
                 nmethods=None, nrows=None, nclasses=None,
                 domain=None,
                 row_indices=None, folds=None, score_by_folds=True,
                 learners=None, models=None, failed=None,
                 actual=None, predicted=None, probabilities=None,
                 store_data=None, store_models=None,
                 train_time=None, test_time=None):
        """
        Construct an instance.

        The constructor stores the given data, and creates empty arrays
        `actual`, `predicted` and `probabilities` if ther are not given but
        sufficient data is provided to deduct their shapes.

        The function

        - set any attributes specified directly through arguments.
        - infers the number of methods, rows and classes from other data
           and/or check their overall consistency.
        - Prepare empty arrays `actual`, `predicted`, `probabilities` and
          `failed` if the are not given. If not enough data is available,
          the corresponding arrays are `None`.

        Args:
            data (Orange.data.Table): stored data from which test was sampled
            nmethods (int): number of methods; can be inferred (or must match)
                the size of `learners`, `models`, `failed`, `predicted` and
                `probabilities`
            nrows (int): number of data instances; can be inferred (or must
                match) `data`, `row_indices`, `actual`, `predicted` and
                `probabilities`
            nclasses (int): number of class values (`None` if continuous); can
                be inferred (or must match) from `domain.class_var` or
                `probabilities`
            domain (Orange.data.Domain): data domain; can be inferred (or must)
                match `data.domain`
            row_indices (np.ndarray): see class documentation
            folds (np.ndarray): see class documentation
            score_by_folds (np.ndarray): see class documentation
            learners (np.ndarray): see class documentation
            models (np.ndarray): see class documentation
            failed (list of str): see class documentation
            actual (np.ndarray): see class documentation
            predicted (np.ndarray): see class documentation
            probabilities (np.ndarray): see class documentation
            store_data (bool): ignored; kept for backward compatibility
            store_models (bool): ignored; kept for backward compatibility
        """

        # Set given data directly from arguments
        self.data = data
        self.domain = domain

        self.row_indices = row_indices
        self.folds = folds
        self.score_by_folds = score_by_folds

        self.learners = learners
        self.models = models

        self.actual = actual
        self.predicted = predicted
        self.probabilities = probabilities
        self.failed = failed

        self.train_time = train_time
        self.test_time = test_time

        # Guess the rest -- or check for ambguities
        def set_or_raise(value, exp_values, msg):
            for exp_value in exp_values:
                if exp_value is False:
                    continue
                if value is None:
                    value = exp_value
                elif value != exp_value:
                    raise ValueError(msg)
            return value
        domain = self.domain = set_or_raise(
            domain, [data is not None and data.domain],
            "mismatching domain")
        self.nrows = nrows = set_or_raise(
            nrows, [actual is not None and len(actual),
                    row_indices is not None and len(row_indices),
                    predicted is not None and predicted.shape[1],
                    probabilities is not None and probabilities.shape[1]],
            "mismatching number of rows")
        if domain is not None and domain.has_continuous_class:
            if nclasses is not None:
                raise ValueError(
                    "regression results cannot have non-None 'nclasses'")
            if probabilities is not None:
                raise ValueError(
                    "regression results cannot have 'probabilities'")
        nclasses = set_or_raise(
            nclasses, [domain is not None and domain.has_discrete_class and
                       len(domain.class_var.values),
                       probabilities is not None and probabilities.shape[2]],
            "mismatching number of class values")
        nmethods = set_or_raise(
            nmethods, [learners is not None and len(learners),
                       models is not None and models.shape[1],
                       failed is not None and len(failed),
                       predicted is not None and predicted.shape[0],
                       probabilities is not None and probabilities.shape[0]],
            "mismatching number of methods")

        # Prepare empty arrays
        if actual is None \
                and nrows is not None:
            self.actual = np.empty(nrows)

        if predicted is None \
                and nmethods is not None and nrows is not None:
            self.predicted = np.empty((nmethods, nrows))

        if probabilities is None \
                and nmethods is not None and nrows is not None \
                and nclasses is not None:
            self.probabilities = \
                np.empty((nmethods, nrows, nclasses))

        if failed is None \
                and nmethods is not None:
            self.failed = [False] * nmethods

    def get_fold(self, fold):
        results = Results()
        results.data = self.data

        if self.folds is None:
            raise ValueError("This 'Results' instance does not have folds.")

        if self.models is not None:
            results.models = self.models[fold]

        results.row_indices = self.row_indices[self.folds[fold]]
        results.actual = self.actual[self.folds[fold]]
        results.predicted = self.predicted[:, self.folds[fold]]
        results.domain = self.domain

        if self.probabilities is not None:
            results.probabilities = self.probabilities[:, self.folds[fold]]

        return results

    def get_augmented_data(self, model_names,
                           include_attrs=True, include_predictions=True,
                           include_probabilities=True):
        """
        Return the test data table augmented with meta attributes containing
        predictions, probabilities (if the task is classification) and fold
        indices.

        Args:
            model_names (list of str): names of models
            include_attrs (bool):
                if set to `False`, original attributes are removed
            include_predictions (bool):
                if set to `False`, predictions are not added
            include_probabilities (bool):
                if set to `False`, probabilities are not added

        Returns:
            augmented_data (Orange.data.Table):
                data augmented with predictions, probabilities and fold indices

        """
        assert self.predicted.shape[0] == len(model_names)

        data = self.data[self.row_indices]
        domain = data.domain
        class_var = domain.class_var
        classification = class_var and class_var.is_discrete

        new_meta_attr = []
        new_meta_vals = np.empty((len(data), 0))
        names = [var.name for var in chain(domain.attributes,
                                           domain.metas,
                                           domain.class_vars)]

        if classification:
            # predictions
            if include_predictions:
                uniq_new, names = self.create_unique_vars(names, model_names, class_var.values)
                new_meta_attr += uniq_new
                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

            # probabilities
            if include_probabilities:
                proposed = [f"{name} ({value})" for name in model_names for value in class_var.values]

                uniq_new, names = self.create_unique_vars(names, proposed)
                new_meta_attr += uniq_new

                for i in self.probabilities:
                    new_meta_vals = np.hstack((new_meta_vals, i))

        elif include_predictions:
            # regression
            uniq_new, names = self.create_unique_vars(names, model_names)
            new_meta_attr += uniq_new
            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

        # add fold info
        if self.folds is not None:
            values = [str(i + 1) for i in range(len(self.folds))]
            uniq_new, names = self.create_unique_vars(names, ["Fold"], values)
            new_meta_attr += uniq_new
            fold = np.empty((len(data), 1))
            for i, s in enumerate(self.folds):
                fold[s, 0] = i
            new_meta_vals = np.hstack((new_meta_vals, fold))

        # append new columns to meta attributes
        new_meta_attr = list(data.domain.metas) + new_meta_attr
        new_meta_vals = np.hstack((data.metas, new_meta_vals))

        attrs = data.domain.attributes if include_attrs else []
        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)
        predictions = data.transform(domain)
        with predictions.unlocked(predictions.metas):
            predictions.metas = new_meta_vals
        predictions.name = data.name
        return predictions

    def create_unique_vars(self, names, proposed_names, values=()):
        unique_vars = []
        for proposed in proposed_names:
            uniq = get_unique_names(names, proposed)
            if values:
                unique_vars.append(DiscreteVariable(uniq, values))
            else:
                unique_vars.append(ContinuousVariable(uniq))
            names.append(uniq)
        return unique_vars, names

    def split_by_model(self):
        """
        Split evaluation results by models.

        The method generates instances of `Results` containing data for single
        models
        """
        data = self.data
        nmethods = len(self.predicted)
        for i in range(nmethods):
            res = Results()
            res.data = data
            res.domain = self.domain
            res.learners = [self.learners[i]]
            res.row_indices = self.row_indices
            res.actual = self.actual
            res.folds = self.folds
            res.score_by_folds = self.score_by_folds
            res.test_time = self.test_time[i]
            res.train_time = self.train_time[i]

            res.predicted = self.predicted[(i,), :]
            if getattr(self, "probabilities", None) is not None:
                res.probabilities = self.probabilities[(i,), :, :]

            if self.models is not None:
                res.models = self.models[:, i:i + 1]

            res.failed = [self.failed[i]]
            yield res


class Validation:
    """
    Base class for different testing schemata such as cross validation and
    testing on separate data set.

    If `data` is some data table and `learners` is a list of learning
    algorithms. This will run 5-fold cross validation and store the results
    in `res`.

        cv = CrossValidation(k=5)
        res = cv(data, learners)

    If constructor was given data and learning algorithms (as in
    `res = CrossValidation(data, learners, k=5)`, it used to automagically
    call the instance after constructing it and return `Results` instead
    of an instance of `Validation`. This functionality
    is deprecated and will be removed in the future.

    Attributes:
        store_data (bool): a flag defining whether the data is stored
        store_models (bool): a flag defining whether the models are stored
    """
    score_by_folds = False

    def __new__(cls,
                data=None, learners=None, preprocessor=None, test_data=None,
                *, callback=None, store_data=False, store_models=False,
                n_jobs=None, **kwargs):
        self = super().__new__(cls)

        if (learners is None) != (data is None):
            raise ValueError(
                "learners and train_data must both be present or not")
        if learners is None:
            if preprocessor is not None:
                raise ValueError("preprocessor cannot be given if learners "
                                 "and train_data are omitted")
            if callback is not None:
                raise ValueError("callback cannot be given if learners "
                                 "and train_data are omitted")
            return self

        warn("calling Validation's constructor with data and learners "
             "is deprecated;\nconstruct an instance and call it",
             DeprecationWarning, stacklevel=2)

        # Explicitly call __init__ because Python won't
        self.__init__(store_data=store_data, store_models=store_models,
                      **kwargs)
        if test_data is not None:
            test_data_kwargs = {"test_data": test_data}
        else:
            test_data_kwargs = {}
        return self(data, learners=learners, preprocessor=preprocessor,
                    callback=callback, **test_data_kwargs)

    # Note: this will be called only if __new__ doesn't have data and learners
    def __init__(self, *, store_data=False, store_models=False):
        self.store_data = store_data
        self.store_models = store_models

    def fit(self, *args, **kwargs):
        warn("Validation.fit is deprecated; use the call operator",
             DeprecationWarning)
        return self(*args, **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None):
        """
        Args:
            data (Orange.data.Table): data to be used (usually split) into
                training and testing
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity
        indices = self.get_indices(data)
        folds, row_indices, actual = self.prepare_arrays(data, indices)

        data_splits = (
            (fold_i, preprocessor(data[train_i]), data[test_i])
            for fold_i, (train_i, test_i) in enumerate(indices))
        args_iter = (
            (fold_i, data, test_data, learner_i, learner, self.store_models)
            for (fold_i, data, test_data) in data_splits
            for (learner_i, learner) in enumerate(learners))

        part_results = []
        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]
        for progress, part in zip(parts, args_iter):
            part_results.append(_mp_worker(*(part + ())))
            callback(progress)
        callback(1)

        results = Results(
            data=data if self.store_data else None,
            domain=data.domain,
            nrows=len(row_indices), learners=learners,
            row_indices=row_indices, folds=folds, actual=actual,
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (len(indices), len(learners)))
        self._collect_part_results(results, part_results)
        return results

    @classmethod
    def prepare_arrays(cls, data, indices):
        """Prepare `folds`, `row_indices` and `actual`.

        The method is used by `__call__`. While functional, it may be
        overriden in subclasses for speed-ups.

        Args:
            data (Orange.data.Table): data use for testing
            indices (list of vectors):
                indices of data instances in each test sample

        Returns:
            folds: (np.ndarray): see class documentation
            row_indices: (np.ndarray): see class documentation
            actual: (np.ndarray): see class documentation
        """
        folds = []
        row_indices = []

        ptr = 0
        for _, test in indices:
            folds.append(slice(ptr, ptr + len(test)))
            row_indices.append(test)
            ptr += len(test)

        row_indices = np.concatenate(row_indices, axis=0)
        return folds, row_indices, data[row_indices].Y

    @staticmethod
    def get_indices(data):
        """
        Return a list of arrays of indices of test data instance

        For example, in k-fold CV, the result is a list with `k` elements,
        each containing approximately `len(data) / k` nonoverlapping indices
        into `data`.

        This method is abstract and must be implemented in derived classes
        unless they provide their own implementation of the `__call__`
        method.

        Args:
            data (Orange.data.Table): test data

        Returns:
            indices (list of np.ndarray):
                a list of arrays of indices into `data`
        """
        raise NotImplementedError()

    def _collect_part_results(self, results, part_results):
        part_results = sorted(part_results)

        ptr, prev_fold_i, prev_n_values = 0, 0, 0
        for res in part_results:
            if res.fold_i != prev_fold_i:
                ptr += prev_n_values
                prev_fold_i = res.fold_i
            result_slice = slice(ptr, ptr + res.n_values)
            prev_n_values = res.n_values

            if res.failed:
                results.failed[res.learner_i] = res.failed
                continue

            if self.store_models:
                results.models[res.fold_i][res.learner_i] = res.model

            results.predicted[res.learner_i][result_slice] = res.values
            results.train_time[res.learner_i] += res.train_time
            results.test_time[res.learner_i] += res.test_time
            if res.probs is not None:
                results.probabilities[res.learner_i][result_slice, :] = \
                    res.probs


class CrossValidation(Validation):
    """
    K-fold cross validation

    Attributes:
        k (int): number of folds (default: 10)
        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time
        stratified (bool):
            flag deciding whether to perform stratified cross-validation.
            If `True` but the class sizes don't allow it, it uses non-stratified
            validataion and adds a list `warning` with a warning message(s) to
            the `Result`.
    """
    # TODO: list `warning` contains just repetitions of the same message
    #       replace with a flag in `Results`?
    def __init__(self, k=10, stratified=True, random_state=0,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.k = k
        self.stratified = stratified
        self.random_state = random_state
        self.warnings = [] if warnings is None else warnings

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            try:
                splitter = skl.StratifiedKFold(
                    self.k, shuffle=True, random_state=self.random_state
                )
                splitter.get_n_splits(data.X, data.Y)
                return list(splitter.split(data.X, data.Y))
            except ValueError:
                self.warnings.append("Using non-stratified sampling.")

        splitter = skl.KFold(
            self.k, shuffle=True, random_state=self.random_state)
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class CrossValidationFeature(Validation):
    """
    Cross validation with folds according to values of a feature.

    Attributes:
        feature (Orange.data.Variable): the feature defining the folds
    """
    def __init__(self, feature=None,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.feature = feature

    def get_indices(self, data):
        data = data.transform(Domain([self.feature], None))
        values = data[:, self.feature].X
        indices = []
        for v in range(len(self.feature.values)):
            test_index = np.where(values == v)[0]
            train_index = np.where((values != v) & (~np.isnan(values)))[0]
            if test_index.size and train_index.size:
                indices.append((train_index, test_index))
        if not indices:
            raise ValueError(
                f"'{self.feature.name}' does not have at least two distinct "
                "values on the data")
        return indices


class LeaveOneOut(Validation):
    """Leave-one-out testing"""
    score_by_folds = False

    def get_indices(self, data):
        splitter = skl.LeaveOneOut()
        splitter.get_n_splits(data)
        return list(splitter.split(data))

    @staticmethod
    def prepare_arrays(data, indices):
        # sped up version of super().prepare_arrays(data)
        row_indices = np.arange(len(data))
        return row_indices, row_indices, data.Y.flatten()


class ShuffleSplit(Validation):
    """
    Test by repeated random sampling

    Attributes:
        n_resamples (int): number of repetitions
        test_size (float, int, None):
            If float, should be between 0.0 and 1.0 and represent the proportion
            of the dataset to include in the test split. If int, represents the
            absolute number of test samples. If None, the value is set to the
            complement of the train size. By default, the value is set to 0.1.
            The default will change in version 0.21. It will remain 0.1 only
            if ``train_size`` is unspecified, otherwise it will complement
            the specified ``train_size``.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        train_size : float, int, or None, default is None
            If float, should be between 0.0 and 1.0 and represent the
            proportion of the dataset to include in the train split. If
            int, represents the absolute number of train samples. If None,
            the value is automatically set to the complement of the test size.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        stratified (bool):
            flag deciding whether to perform stratified cross-validation.

        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time

    """
    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,
                 stratified=True, random_state=0,
                 store_data=False, store_models=False):
        super().__init__(store_data=store_data, store_models=store_models)
        self.n_resamples = n_resamples
        self.train_size = train_size
        self.test_size = test_size
        self.stratified = stratified
        self.random_state = random_state

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            splitter = skl.StratifiedShuffleSplit(
                n_splits=self.n_resamples, train_size=self.train_size,
                test_size=self.test_size, random_state=self.random_state
            )
            splitter.get_n_splits(data.X, data.Y)
            return list(splitter.split(data.X, data.Y))

        splitter = skl.ShuffleSplit(
            n_splits=self.n_resamples, train_size=self.train_size,
            test_size=self.test_size, random_state=self.random_state
        )
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class TestOnTestData(Validation):
    """
    Test on separately provided test data

    Note that the class has a different signature for `__call__`.
    """
    # get_indices is not needed in this class, pylint: disable=abstract-method

    def __new__(cls, data=None, test_data=None, learners=None,
                preprocessor=None, **kwargs):
        if "train_data" in kwargs:
            if data is None:
                data = kwargs.pop("train_data")
            else:
                raise ValueError(
                    "argument 'data' is given twice (once as 'train_data')")
        return super().__new__(
            cls,
            data=data, learners=learners, preprocessor=preprocessor,
            test_data=test_data, **kwargs)

    def __call__(self, data, test_data, learners, preprocessor=None,
                 *, callback=None):
        """
        Args:
            data (Orange.data.Table): training data
            test_data (Orange.data.Table): test_data
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity

        train_data = preprocessor(data)
        part_results = []
        for (learner_i, learner) in enumerate(learners):
            part_results.append(
                _mp_worker(0, train_data, test_data, learner_i, learner,
                           self.store_models))
            callback((learner_i + 1) / len(learners))
        callback(1)

        results = Results(
            data=test_data if self.store_data else None,
            domain=test_data.domain,
            nrows=len(test_data), learners=learners,
            row_indices=np.arange(len(test_data)),
            folds=(Ellipsis, ),
            actual=test_data.Y,
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (1, len(learners)))
        self._collect_part_results(results, part_results)
        return results


class TestOnTrainingData(TestOnTestData):
    """Test on training data"""
    # get_indices is not needed in this class, pylint: disable=abstract-method
    # signature is such as on the base class, pylint: disable=signature-differs
    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):
        return super().__new__(
            cls,
            data, test_data=data, learners=learners, preprocessor=preprocessor,
            **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None,
                 **kwargs):
        kwargs.setdefault("test_data", data)
        # if kwargs contains anything besides test_data, this will be detected
        # (and complained about) by super().__call__
        return super().__call__(
            data=data, learners=learners, preprocessor=preprocessor,
            callback=callback, **kwargs)


def sample(table, n=0.7, stratified=False, replace=False,
           random_state=None):
    """
    Samples data instances from a data table. Returns the sample and
    a dataset from input data table that are not in the sample. Also
    uses several sampling functions from
    `scikit-learn <http://scikit-learn.org>`_.

    table : data table
        A data table from which to sample.

    n : float, int (default = 0.7)
        If float, should be between 0.0 and 1.0 and represents
        the proportion of data instances in the resulting sample. If
        int, n is the number of data instances in the resulting sample.

    stratified : bool, optional (default = False)
        If true, sampling will try to consider class values and
        match distribution of class values
        in train and test subsets.

    replace : bool, optional (default = False)
        sample with replacement

    random_state : int or RandomState
        Pseudo-random number generator state used for random sampling.
    """

    if isinstance(n, float):
        n = int(n * len(table))

    if replace:
        if random_state is None:
            rgen = np.random
        else:
            rgen = np.random.mtrand.RandomState(random_state)
        a_sample = rgen.randint(0, len(table), n)
        o = np.ones(len(table))
        o[a_sample] = 0
        others = np.nonzero(o)[0]
        return table[a_sample], table[others]

    n = len(table) - n
    if stratified and table.domain.has_discrete_class:
        test_size = max(len(table.domain.class_var.values), n)
        splitter = skl.StratifiedShuffleSplit(
            n_splits=1, test_size=test_size, train_size=len(table) - test_size,
            random_state=random_state)
        splitter.get_n_splits(table.X, table.Y)
        ind = splitter.split(table.X, table.Y)
    else:
        splitter = skl.ShuffleSplit(
            n_splits=1, test_size=n, random_state=random_state)
        splitter.get_n_splits(table)
        ind = splitter.split(table)
    ind = next(ind)
    return table[ind[0]], table[ind[1]]

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
### ValueError When Evaluating Models with Multiple Targets

**Description**
When evaluating a model using data with multiple target variables, the evaluation process fails with a `ValueError` due to a shape mismatch. The system currently does not support multiple targets, leading to unexpected errors during evaluation.

**Example Code**
```python
housing = Table("housing")
class_vars = [housing.domain.class_var, housing.domain.attributes[0]]
domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)
multiple_targets_data = housing.transform(domain)

self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)
self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())
self.wait_until_finished()
```

**Expected Behavior**
The evaluation should either handle multiple targets correctly or provide a clear and descriptive error message indicating that multiple targets are not supported.

**Actual Behavior**
A `ValueError` is raised with the message:
```
ValueError: could not broadcast input array from shape (102,2) into shape (102,)
```
This error occurs because the evaluation process expects a single target dimension but receives multiple, causing a shape mismatch and subsequent failure.

[/ISSUE]
{
    "TestOWTestAndScore.test_CrossValidationByFeature": "PASSED",
    "TestOWTestAndScore.test_addon_scorers": "PASSED",
    "TestOWTestAndScore.test_basic": "PASSED",
    "TestOWTestAndScore.test_comparison_bad_scores": "PASSED",
    "TestOWTestAndScore.test_comparison_bad_slots": "PASSED",
    "TestOWTestAndScore.test_comparison_binary_score": "PASSED",
    "TestOWTestAndScore.test_comparison_requires_cv": "PASSED",
    "TestOWTestAndScore.test_comparison_requires_multiple_models": "PASSED",
    "TestOWTestAndScore.test_copy_to_clipboard": "PASSED",
    "TestOWTestAndScore.test_data_errors": "PASSED",
    "TestOWTestAndScore.test_fill_table": "PASSED",
    "TestOWTestAndScore.test_image_export": "PASSED",
    "TestOWTestAndScore.test_memory_error": "PASSED",
    "TestOWTestAndScore.test_migrate_removes_invalid_contexts": "PASSED",
    "TestOWTestAndScore.test_migrate_shown_scores": "PASSED",
    "TestOWTestAndScore.test_msg_base_class": "PASSED",
    "TestOWTestAndScore.test_multi_target_input": "PASSED",
    "TestOWTestAndScore.test_multiple_learners": "PASSED",
    "TestOWTestAndScore.test_multiple_targets_pls": "PASSED",
    "TestOWTestAndScore.test_nan_on_comparison": "PASSED",
    "TestOWTestAndScore.test_no_stratification": "PASSED",
    "TestOWTestAndScore.test_one_class_value": "PASSED",
    "TestOWTestAndScore.test_resort_on_data_change": "PASSED",
    "TestOWTestAndScore.test_scores_constant": "PASSED",
    "TestOWTestAndScore.test_scores_cross_validation": "PASSED",
    "TestOWTestAndScore.test_scores_log_reg_advanced": "PASSED",
    "TestOWTestAndScore.test_scores_log_reg_bad": "PASSED",
    "TestOWTestAndScore.test_scores_log_reg_bad2": "PASSED",
    "TestOWTestAndScore.test_scores_log_reg_overfitted": "PASSED",
    "TestOWTestAndScore.test_target_changing": "PASSED",
    "TestOWTestAndScore.test_testOnTest": "PASSED",
    "TestOWTestAndScore.test_testOnTest_incompatible_domain": "PASSED",
    "TestOWTestAndScore.test_too_many_folds": "PASSED",
    "TestOWTestAndScore.test_unique_output_domain": "PASSED",
    "TestHelpers.test_results_one_vs_rest": "PASSED",
    "test_filename": "ERROR",
    "TestOWTestAndScore.test_minimum_size": "FAILED"
}
diff --git a/Orange/evaluation/testing.py b/Orange/evaluation/testing.py
index 34a798241..d40701206 100644
--- a/Orange/evaluation/testing.py
+++ b/Orange/evaluation/testing.py
@@ -541,6 +541,11 @@ class Validation:
                 results.failed[res.learner_i] = res.failed
                 continue
 
+            if len(res.values.shape) > 1 and res.values.shape[1] > 1:
+                msg = "Multiple targets are not supported."
+                results.failed[res.learner_i] = ValueError(msg)
+                continue
+
             if self.store_models:
                 results.models[res.fold_i][res.learner_i] = res.model
 

{"Orange/evaluation/testing.py": "# __new__ methods have different arguments\n# pylint: disable=arguments-differ\nfrom warnings import warn\nfrom collections import namedtuple\nfrom itertools import chain\nfrom time import time\n\nimport numpy as np\n\nimport sklearn.model_selection as skl\n\nfrom Orange.data import Domain, ContinuousVariable, DiscreteVariable\nfrom Orange.data.util import get_unique_names\n\n__all__ = [\"Results\", \"CrossValidation\", \"LeaveOneOut\", \"TestOnTrainingData\",\n           \"ShuffleSplit\", \"TestOnTestData\", \"sample\", \"CrossValidationFeature\"]\n\n_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',\n                                       'failed', 'n_values', 'values',\n                                       'probs', 'train_time', 'test_time'))\n\n\ndef _identity(x):\n    return x\n\n\ndef _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed = None, None, None, False\n    train_time, test_time = None, None\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0 = time()\n        # testing\n        class_var = train_data.domain.class_var\n        if class_var and class_var.is_discrete:\n            predicted, probs = model(test_data, model.ValueProbs)\n        else:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)\n\n\nclass Results:\n    \"\"\"\n    Class for storing predictions in model testing.\n\n    Attributes:\n        data (Optional[Table]):  Data used for testing.\n\n        models (Optional[List[Model]]): A list of induced models.\n\n        row_indices (np.ndarray): Indices of rows in `data` that were used in\n            testing, stored as a numpy vector of length `nrows`.\n            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer\n            to the target value of instance, that is, the i-th test instance\n            is `data[row_indices[i]]`, its actual class is `actual[i]`, and\n            the prediction by m-th method is `predicted[m, i]`.\n\n        nrows (int): The number of test instances (including duplicates);\n            `nrows` equals the length of `row_indices` and `actual`, and the\n            second dimension of `predicted` and `probabilities`.\n\n        actual (np.ndarray): true values of target variable in a vector of\n            length `nrows`.\n\n        predicted (np.ndarray): predicted values of target variable in an array\n            of shape (number-of-methods, `nrows`)\n\n        probabilities (Optional[np.ndarray]): predicted probabilities\n            (for discrete target variables) in an array of shape\n            (number-of-methods, `nrows`, number-of-classes)\n\n        folds (List[Slice or List[int]]): a list of indices (or slice objects)\n            corresponding to testing data subsets, that is,\n            `row_indices[folds[i]]` contains row indices used in fold i, so\n            `data[row_indices[folds[i]]]` is the corresponding testing data\n\n        train_time (np.ndarray): training times of batches\n\n        test_time (np.ndarray): testing times of batches\n    \"\"\"\n    def __init__(self, data=None, *,\n                 nmethods=None, nrows=None, nclasses=None,\n                 domain=None,\n                 row_indices=None, folds=None, score_by_folds=True,\n                 learners=None, models=None, failed=None,\n                 actual=None, predicted=None, probabilities=None,\n                 store_data=None, store_models=None,\n                 train_time=None, test_time=None):\n        \"\"\"\n        Construct an instance.\n\n        The constructor stores the given data, and creates empty arrays\n        `actual`, `predicted` and `probabilities` if ther are not given but\n        sufficient data is provided to deduct their shapes.\n\n        The function\n\n        - set any attributes specified directly through arguments.\n        - infers the number of methods, rows and classes from other data\n           and/or check their overall consistency.\n        - Prepare empty arrays `actual`, `predicted`, `probabilities` and\n          `failed` if the are not given. If not enough data is available,\n          the corresponding arrays are `None`.\n\n        Args:\n            data (Orange.data.Table): stored data from which test was sampled\n            nmethods (int): number of methods; can be inferred (or must match)\n                the size of `learners`, `models`, `failed`, `predicted` and\n                `probabilities`\n            nrows (int): number of data instances; can be inferred (or must\n                match) `data`, `row_indices`, `actual`, `predicted` and\n                `probabilities`\n            nclasses (int): number of class values (`None` if continuous); can\n                be inferred (or must match) from `domain.class_var` or\n                `probabilities`\n            domain (Orange.data.Domain): data domain; can be inferred (or must)\n                match `data.domain`\n            row_indices (np.ndarray): see class documentation\n            folds (np.ndarray): see class documentation\n            score_by_folds (np.ndarray): see class documentation\n            learners (np.ndarray): see class documentation\n            models (np.ndarray): see class documentation\n            failed (list of str): see class documentation\n            actual (np.ndarray): see class documentation\n            predicted (np.ndarray): see class documentation\n            probabilities (np.ndarray): see class documentation\n            store_data (bool): ignored; kept for backward compatibility\n            store_models (bool): ignored; kept for backward compatibility\n        \"\"\"\n\n        # Set given data directly from arguments\n        self.data = data\n        self.domain = domain\n\n        self.row_indices = row_indices\n        self.folds = folds\n        self.score_by_folds = score_by_folds\n\n        self.learners = learners\n        self.models = models\n\n        self.actual = actual\n        self.predicted = predicted\n        self.probabilities = probabilities\n        self.failed = failed\n\n        self.train_time = train_time\n        self.test_time = test_time\n\n        # Guess the rest -- or check for ambguities\n        def set_or_raise(value, exp_values, msg):\n            for exp_value in exp_values:\n                if exp_value is False:\n                    continue\n                if value is None:\n                    value = exp_value\n                elif value != exp_value:\n                    raise ValueError(msg)\n            return value\n        domain = self.domain = set_or_raise(\n            domain, [data is not None and data.domain],\n            \"mismatching domain\")\n        self.nrows = nrows = set_or_raise(\n            nrows, [actual is not None and len(actual),\n                    row_indices is not None and len(row_indices),\n                    predicted is not None and predicted.shape[1],\n                    probabilities is not None and probabilities.shape[1]],\n            \"mismatching number of rows\")\n        if domain is not None and domain.has_continuous_class:\n            if nclasses is not None:\n                raise ValueError(\n                    \"regression results cannot have non-None 'nclasses'\")\n            if probabilities is not None:\n                raise ValueError(\n                    \"regression results cannot have 'probabilities'\")\n        nclasses = set_or_raise(\n            nclasses, [domain is not None and domain.has_discrete_class and\n                       len(domain.class_var.values),\n                       probabilities is not None and probabilities.shape[2]],\n            \"mismatching number of class values\")\n        nmethods = set_or_raise(\n            nmethods, [learners is not None and len(learners),\n                       models is not None and models.shape[1],\n                       failed is not None and len(failed),\n                       predicted is not None and predicted.shape[0],\n                       probabilities is not None and probabilities.shape[0]],\n            \"mismatching number of methods\")\n\n        # Prepare empty arrays\n        if actual is None \\\n                and nrows is not None:\n            self.actual = np.empty(nrows)\n\n        if predicted is None \\\n                and nmethods is not None and nrows is not None:\n            self.predicted = np.empty((nmethods, nrows))\n\n        if probabilities is None \\\n                and nmethods is not None and nrows is not None \\\n                and nclasses is not None:\n            self.probabilities = \\\n                np.empty((nmethods, nrows, nclasses))\n\n        if failed is None \\\n                and nmethods is not None:\n            self.failed = [False] * nmethods\n\n    def get_fold(self, fold):\n        results = Results()\n        results.data = self.data\n\n        if self.folds is None:\n            raise ValueError(\"This 'Results' instance does not have folds.\")\n\n        if self.models is not None:\n            results.models = self.models[fold]\n\n        results.row_indices = self.row_indices[self.folds[fold]]\n        results.actual = self.actual[self.folds[fold]]\n        results.predicted = self.predicted[:, self.folds[fold]]\n        results.domain = self.domain\n\n        if self.probabilities is not None:\n            results.probabilities = self.probabilities[:, self.folds[fold]]\n\n        return results\n\n    def get_augmented_data(self, model_names,\n                           include_attrs=True, include_predictions=True,\n                           include_probabilities=True):\n        \"\"\"\n        Return the test data table augmented with meta attributes containing\n        predictions, probabilities (if the task is classification) and fold\n        indices.\n\n        Args:\n            model_names (list of str): names of models\n            include_attrs (bool):\n                if set to `False`, original attributes are removed\n            include_predictions (bool):\n                if set to `False`, predictions are not added\n            include_probabilities (bool):\n                if set to `False`, probabilities are not added\n\n        Returns:\n            augmented_data (Orange.data.Table):\n                data augmented with predictions, probabilities and fold indices\n\n        \"\"\"\n        assert self.predicted.shape[0] == len(model_names)\n\n        data = self.data[self.row_indices]\n        domain = data.domain\n        class_var = domain.class_var\n        classification = class_var and class_var.is_discrete\n\n        new_meta_attr = []\n        new_meta_vals = np.empty((len(data), 0))\n        names = [var.name for var in chain(domain.attributes,\n                                           domain.metas,\n                                           domain.class_vars)]\n\n        if classification:\n            # predictions\n            if include_predictions:\n                uniq_new, names = self.create_unique_vars(names, model_names, class_var.values)\n                new_meta_attr += uniq_new\n                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n            # probabilities\n            if include_probabilities:\n                proposed = [f\"{name} ({value})\" for name in model_names for value in class_var.values]\n\n                uniq_new, names = self.create_unique_vars(names, proposed)\n                new_meta_attr += uniq_new\n\n                for i in self.probabilities:\n                    new_meta_vals = np.hstack((new_meta_vals, i))\n\n        elif include_predictions:\n            # regression\n            uniq_new, names = self.create_unique_vars(names, model_names)\n            new_meta_attr += uniq_new\n            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n        # add fold info\n        if self.folds is not None:\n            values = [str(i + 1) for i in range(len(self.folds))]\n            uniq_new, names = self.create_unique_vars(names, [\"Fold\"], values)\n            new_meta_attr += uniq_new\n            fold = np.empty((len(data), 1))\n            for i, s in enumerate(self.folds):\n                fold[s, 0] = i\n            new_meta_vals = np.hstack((new_meta_vals, fold))\n\n        # append new columns to meta attributes\n        new_meta_attr = list(data.domain.metas) + new_meta_attr\n        new_meta_vals = np.hstack((data.metas, new_meta_vals))\n\n        attrs = data.domain.attributes if include_attrs else []\n        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)\n        predictions = data.transform(domain)\n        with predictions.unlocked(predictions.metas):\n            predictions.metas = new_meta_vals\n        predictions.name = data.name\n        return predictions\n\n    def create_unique_vars(self, names, proposed_names, values=()):\n        unique_vars = []\n        for proposed in proposed_names:\n            uniq = get_unique_names(names, proposed)\n            if values:\n                unique_vars.append(DiscreteVariable(uniq, values))\n            else:\n                unique_vars.append(ContinuousVariable(uniq))\n            names.append(uniq)\n        return unique_vars, names\n\n    def split_by_model(self):\n        \"\"\"\n        Split evaluation results by models.\n\n        The method generates instances of `Results` containing data for single\n        models\n        \"\"\"\n        data = self.data\n        nmethods = len(self.predicted)\n        for i in range(nmethods):\n            res = Results()\n            res.data = data\n            res.domain = self.domain\n            res.learners = [self.learners[i]]\n            res.row_indices = self.row_indices\n            res.actual = self.actual\n            res.folds = self.folds\n            res.score_by_folds = self.score_by_folds\n            res.test_time = self.test_time[i]\n            res.train_time = self.train_time[i]\n\n            res.predicted = self.predicted[(i,), :]\n            if getattr(self, \"probabilities\", None) is not None:\n                res.probabilities = self.probabilities[(i,), :, :]\n\n            if self.models is not None:\n                res.models = self.models[:, i:i + 1]\n\n            res.failed = [self.failed[i]]\n            yield res\n\n\nclass Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        return folds, row_indices, data[row_indices].Y\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs\n\n\nclass CrossValidation(Validation):\n    \"\"\"\n    K-fold cross validation\n\n    Attributes:\n        k (int): number of folds (default: 10)\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n            If `True` but the class sizes don't allow it, it uses non-stratified\n            validataion and adds a list `warning` with a warning message(s) to\n            the `Result`.\n    \"\"\"\n    # TODO: list `warning` contains just repetitions of the same message\n    #       replace with a flag in `Results`?\n    def __init__(self, k=10, stratified=True, random_state=0,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.k = k\n        self.stratified = stratified\n        self.random_state = random_state\n        self.warnings = [] if warnings is None else warnings\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            try:\n                splitter = skl.StratifiedKFold(\n                    self.k, shuffle=True, random_state=self.random_state\n                )\n                splitter.get_n_splits(data.X, data.Y)\n                return list(splitter.split(data.X, data.Y))\n            except ValueError:\n                self.warnings.append(\"Using non-stratified sampling.\")\n\n        splitter = skl.KFold(\n            self.k, shuffle=True, random_state=self.random_state)\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass CrossValidationFeature(Validation):\n    \"\"\"\n    Cross validation with folds according to values of a feature.\n\n    Attributes:\n        feature (Orange.data.Variable): the feature defining the folds\n    \"\"\"\n    def __init__(self, feature=None,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.feature = feature\n\n    def get_indices(self, data):\n        data = data.transform(Domain([self.feature], None))\n        values = data[:, self.feature].X\n        indices = []\n        for v in range(len(self.feature.values)):\n            test_index = np.where(values == v)[0]\n            train_index = np.where((values != v) & (~np.isnan(values)))[0]\n            if test_index.size and train_index.size:\n                indices.append((train_index, test_index))\n        if not indices:\n            raise ValueError(\n                f\"'{self.feature.name}' does not have at least two distinct \"\n                \"values on the data\")\n        return indices\n\n\nclass LeaveOneOut(Validation):\n    \"\"\"Leave-one-out testing\"\"\"\n    score_by_folds = False\n\n    def get_indices(self, data):\n        splitter = skl.LeaveOneOut()\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n    @staticmethod\n    def prepare_arrays(data, indices):\n        # sped up version of super().prepare_arrays(data)\n        row_indices = np.arange(len(data))\n        return row_indices, row_indices, data.Y.flatten()\n\n\nclass ShuffleSplit(Validation):\n    \"\"\"\n    Test by repeated random sampling\n\n    Attributes:\n        n_resamples (int): number of repetitions\n        test_size (float, int, None):\n            If float, should be between 0.0 and 1.0 and represent the proportion\n            of the dataset to include in the test split. If int, represents the\n            absolute number of test samples. If None, the value is set to the\n            complement of the train size. By default, the value is set to 0.1.\n            The default will change in version 0.21. It will remain 0.1 only\n            if ``train_size`` is unspecified, otherwise it will complement\n            the specified ``train_size``.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        train_size : float, int, or None, default is None\n            If float, should be between 0.0 and 1.0 and represent the\n            proportion of the dataset to include in the train split. If\n            int, represents the absolute number of train samples. If None,\n            the value is automatically set to the complement of the test size.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n\n    \"\"\"\n    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,\n                 stratified=True, random_state=0,\n                 store_data=False, store_models=False):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.n_resamples = n_resamples\n        self.train_size = train_size\n        self.test_size = test_size\n        self.stratified = stratified\n        self.random_state = random_state\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            splitter = skl.StratifiedShuffleSplit(\n                n_splits=self.n_resamples, train_size=self.train_size,\n                test_size=self.test_size, random_state=self.random_state\n            )\n            splitter.get_n_splits(data.X, data.Y)\n            return list(splitter.split(data.X, data.Y))\n\n        splitter = skl.ShuffleSplit(\n            n_splits=self.n_resamples, train_size=self.train_size,\n            test_size=self.test_size, random_state=self.random_state\n        )\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass TestOnTestData(Validation):\n    \"\"\"\n    Test on separately provided test data\n\n    Note that the class has a different signature for `__call__`.\n    \"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n\n    def __new__(cls, data=None, test_data=None, learners=None,\n                preprocessor=None, **kwargs):\n        if \"train_data\" in kwargs:\n            if data is None:\n                data = kwargs.pop(\"train_data\")\n            else:\n                raise ValueError(\n                    \"argument 'data' is given twice (once as 'train_data')\")\n        return super().__new__(\n            cls,\n            data=data, learners=learners, preprocessor=preprocessor,\n            test_data=test_data, **kwargs)\n\n    def __call__(self, data, test_data, learners, preprocessor=None,\n                 *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): training data\n            test_data (Orange.data.Table): test_data\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n\n        train_data = preprocessor(data)\n        part_results = []\n        for (learner_i, learner) in enumerate(learners):\n            part_results.append(\n                _mp_worker(0, train_data, test_data, learner_i, learner,\n                           self.store_models))\n            callback((learner_i + 1) / len(learners))\n        callback(1)\n\n        results = Results(\n            data=test_data if self.store_data else None,\n            domain=test_data.domain,\n            nrows=len(test_data), learners=learners,\n            row_indices=np.arange(len(test_data)),\n            folds=(Ellipsis, ),\n            actual=test_data.Y,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (1, len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n\nclass TestOnTrainingData(TestOnTestData):\n    \"\"\"Test on training data\"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n    # signature is such as on the base class, pylint: disable=signature-differs\n    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):\n        return super().__new__(\n            cls,\n            data, test_data=data, learners=learners, preprocessor=preprocessor,\n            **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None,\n                 **kwargs):\n        kwargs.setdefault(\"test_data\", data)\n        # if kwargs contains anything besides test_data, this will be detected\n        # (and complained about) by super().__call__\n        return super().__call__(\n            data=data, learners=learners, preprocessor=preprocessor,\n            callback=callback, **kwargs)\n\n\ndef sample(table, n=0.7, stratified=False, replace=False,\n           random_state=None):\n    \"\"\"\n    Samples data instances from a data table. Returns the sample and\n    a dataset from input data table that are not in the sample. Also\n    uses several sampling functions from\n    `scikit-learn <http://scikit-learn.org>`_.\n\n    table : data table\n        A data table from which to sample.\n\n    n : float, int (default = 0.7)\n        If float, should be between 0.0 and 1.0 and represents\n        the proportion of data instances in the resulting sample. If\n        int, n is the number of data instances in the resulting sample.\n\n    stratified : bool, optional (default = False)\n        If true, sampling will try to consider class values and\n        match distribution of class values\n        in train and test subsets.\n\n    replace : bool, optional (default = False)\n        sample with replacement\n\n    random_state : int or RandomState\n        Pseudo-random number generator state used for random sampling.\n    \"\"\"\n\n    if isinstance(n, float):\n        n = int(n * len(table))\n\n    if replace:\n        if random_state is None:\n            rgen = np.random\n        else:\n            rgen = np.random.mtrand.RandomState(random_state)\n        a_sample = rgen.randint(0, len(table), n)\n        o = np.ones(len(table))\n        o[a_sample] = 0\n        others = np.nonzero(o)[0]\n        return table[a_sample], table[others]\n\n    n = len(table) - n\n    if stratified and table.domain.has_discrete_class:\n        test_size = max(len(table.domain.class_var.values), n)\n        splitter = skl.StratifiedShuffleSplit(\n            n_splits=1, test_size=test_size, train_size=len(table) - test_size,\n            random_state=random_state)\n        splitter.get_n_splits(table.X, table.Y)\n        ind = splitter.split(table.X, table.Y)\n    else:\n        splitter = skl.ShuffleSplit(\n            n_splits=1, test_size=n, random_state=random_state)\n        splitter.get_n_splits(table)\n        ind = splitter.split(table)\n    ind = next(ind)\n    return table[ind[0]], table[ind[1]]\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
### ValueError When Evaluating Models with Multiple Targets

**Description**
When evaluating a model using data with multiple target variables, the evaluation process fails with a `ValueError` due to a shape mismatch. The system currently does not support multiple targets, leading to unexpected errors during evaluation.

**Example Code**
```python
housing = Table("housing")
class_vars = [housing.domain.class_var, housing.domain.attributes[0]]
domain = Domain(housing.domain.attributes[1:], class_vars=class_vars)
multiple_targets_data = housing.transform(domain)

self.send_signal(self.widget.Inputs.train_data, multiple_targets_data)
self.send_signal(self.widget.Inputs.learner, PLSRegressionLearner())
self.wait_until_finished()
```

**Expected Behavior**
The evaluation should either handle multiple targets correctly or provide a clear and descriptive error message indicating that multiple targets are not supported.

**Actual Behavior**
A `ValueError` is raised with the message:
```
ValueError: could not broadcast input array from shape (102,2) into shape (102,)
```
This error occurs because the evaluation process expects a single target dimension but receives multiple, causing a shape mismatch and subsequent failure.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### Orange/evaluation/testing.py
# __new__ methods have different arguments
# pylint: disable=arguments-differ
from warnings import warn
from collections import namedtuple
from itertools import chain
from time import time

import numpy as np

import sklearn.model_selection as skl

from Orange.data import Domain, ContinuousVariable, DiscreteVariable
from Orange.data.util import get_unique_names

__all__ = ["Results", "CrossValidation", "LeaveOneOut", "TestOnTrainingData",
           "ShuffleSplit", "TestOnTestData", "sample", "CrossValidationFeature"]

_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',
                                       'failed', 'n_values', 'values',
                                       'probs', 'train_time', 'test_time'))


def _identity(x):
    return x


def _mp_worker(fold_i, train_data, test_data, learner_i, learner,
               store_models):
    predicted, probs, model, failed = None, None, None, False
    train_time, test_time = None, None
    try:
        if not train_data or not test_data:
            raise RuntimeError('Test fold is empty')
        # training
        t0 = time()
        model = learner(train_data)
        train_time = time() - t0
        t0 = time()
        # testing
        class_var = train_data.domain.class_var
        if class_var and class_var.is_discrete:
            predicted, probs = model(test_data, model.ValueProbs)
        else:
            predicted = model(test_data, model.Value)
        test_time = time() - t0
    # Different models can fail at any time raising any exception
    except Exception as ex:  # pylint: disable=broad-except
        failed = ex
    return _MpResults(fold_i, learner_i, store_models and model,
                      failed, len(test_data), predicted, probs,
                      train_time, test_time)


class Results:
    """
    Class for storing predictions in model testing.

    Attributes:
        data (Optional[Table]):  Data used for testing.

        models (Optional[List[Model]]): A list of induced models.

        row_indices (np.ndarray): Indices of rows in `data` that were used in
            testing, stored as a numpy vector of length `nrows`.
            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer
            to the target value of instance, that is, the i-th test instance
            is `data[row_indices[i]]`, its actual class is `actual[i]`, and
            the prediction by m-th method is `predicted[m, i]`.

        nrows (int): The number of test instances (including duplicates);
            `nrows` equals the length of `row_indices` and `actual`, and the
            second dimension of `predicted` and `probabilities`.

        actual (np.ndarray): true values of target variable in a vector of
            length `nrows`.

        predicted (np.ndarray): predicted values of target variable in an array
            of shape (number-of-methods, `nrows`)

        probabilities (Optional[np.ndarray]): predicted probabilities
            (for discrete target variables) in an array of shape
            (number-of-methods, `nrows`, number-of-classes)

        folds (List[Slice or List[int]]): a list of indices (or slice objects)
            corresponding to testing data subsets, that is,
            `row_indices[folds[i]]` contains row indices used in fold i, so
            `data[row_indices[folds[i]]]` is the corresponding testing data

        train_time (np.ndarray): training times of batches

        test_time (np.ndarray): testing times of batches
    """
    def __init__(self, data=None, *,
                 nmethods=None, nrows=None, nclasses=None,
                 domain=None,
                 row_indices=None, folds=None, score_by_folds=True,
                 learners=None, models=None, failed=None,
                 actual=None, predicted=None, probabilities=None,
                 store_data=None, store_models=None,
                 train_time=None, test_time=None):
        """
        Construct an instance.

        The constructor stores the given data, and creates empty arrays
        `actual`, `predicted` and `probabilities` if ther are not given but
        sufficient data is provided to deduct their shapes.

        The function

        - set any attributes specified directly through arguments.
        - infers the number of methods, rows and classes from other data
           and/or check their overall consistency.
        - Prepare empty arrays `actual`, `predicted`, `probabilities` and
          `failed` if the are not given. If not enough data is available,
          the corresponding arrays are `None`.

        Args:
            data (Orange.data.Table): stored data from which test was sampled
            nmethods (int): number of methods; can be inferred (or must match)
                the size of `learners`, `models`, `failed`, `predicted` and
                `probabilities`
            nrows (int): number of data instances; can be inferred (or must
                match) `data`, `row_indices`, `actual`, `predicted` and
                `probabilities`
            nclasses (int): number of class values (`None` if continuous); can
                be inferred (or must match) from `domain.class_var` or
                `probabilities`
            domain (Orange.data.Domain): data domain; can be inferred (or must)
                match `data.domain`
            row_indices (np.ndarray): see class documentation
            folds (np.ndarray): see class documentation
            score_by_folds (np.ndarray): see class documentation
            learners (np.ndarray): see class documentation
            models (np.ndarray): see class documentation
            failed (list of str): see class documentation
            actual (np.ndarray): see class documentation
            predicted (np.ndarray): see class documentation
            probabilities (np.ndarray): see class documentation
            store_data (bool): ignored; kept for backward compatibility
            store_models (bool): ignored; kept for backward compatibility
        """

        # Set given data directly from arguments
        self.data = data
        self.domain = domain

        self.row_indices = row_indices
        self.folds = folds
        self.score_by_folds = score_by_folds

        self.learners = learners
        self.models = models

        self.actual = actual
        self.predicted = predicted
        self.probabilities = probabilities
        self.failed = failed

        self.train_time = train_time
        self.test_time = test_time

        # Guess the rest -- or check for ambguities
        def set_or_raise(value, exp_values, msg):
            for exp_value in exp_values:
                if exp_value is False:
                    continue
                if value is None:
                    value = exp_value
                elif value != exp_value:
                    raise ValueError(msg)
            return value
        domain = self.domain = set_or_raise(
            domain, [data is not None and data.domain],
            "mismatching domain")
        self.nrows = nrows = set_or_raise(
            nrows, [actual is not None and len(actual),
                    row_indices is not None and len(row_indices),
                    predicted is not None and predicted.shape[1],
                    probabilities is not None and probabilities.shape[1]],
            "mismatching number of rows")
        if domain is not None and domain.has_continuous_class:
            if nclasses is not None:
                raise ValueError(
                    "regression results cannot have non-None 'nclasses'")
            if probabilities is not None:
                raise ValueError(
                    "regression results cannot have 'probabilities'")
        nclasses = set_or_raise(
            nclasses, [domain is not None and domain.has_discrete_class and
                       len(domain.class_var.values),
                       probabilities is not None and probabilities.shape[2]],
            "mismatching number of class values")
        nmethods = set_or_raise(
            nmethods, [learners is not None and len(learners),
                       models is not None and models.shape[1],
                       failed is not None and len(failed),
                       predicted is not None and predicted.shape[0],
                       probabilities is not None and probabilities.shape[0]],
            "mismatching number of methods")

        # Prepare empty arrays
        if actual is None \
                and nrows is not None:
            self.actual = np.empty(nrows)

        if predicted is None \
                and nmethods is not None and nrows is not None:
            self.predicted = np.empty((nmethods, nrows))

        if probabilities is None \
                and nmethods is not None and nrows is not None \
                and nclasses is not None:
            self.probabilities = \
                np.empty((nmethods, nrows, nclasses))

        if failed is None \
                and nmethods is not None:
            self.failed = [False] * nmethods

    def get_fold(self, fold):
        results = Results()
        results.data = self.data

        if self.folds is None:
            raise ValueError("This 'Results' instance does not have folds.")

        if self.models is not None:
            results.models = self.models[fold]

        results.row_indices = self.row_indices[self.folds[fold]]
        results.actual = self.actual[self.folds[fold]]
        results.predicted = self.predicted[:, self.folds[fold]]
        results.domain = self.domain

        if self.probabilities is not None:
            results.probabilities = self.probabilities[:, self.folds[fold]]

        return results

    def get_augmented_data(self, model_names,
                           include_attrs=True, include_predictions=True,
                           include_probabilities=True):
        """
        Return the test data table augmented with meta attributes containing
        predictions, probabilities (if the task is classification) and fold
        indices.

        Args:
            model_names (list of str): names of models
            include_attrs (bool):
                if set to `False`, original attributes are removed
            include_predictions (bool):
                if set to `False`, predictions are not added
            include_probabilities (bool):
                if set to `False`, probabilities are not added

        Returns:
            augmented_data (Orange.data.Table):
                data augmented with predictions, probabilities and fold indices

        """
        assert self.predicted.shape[0] == len(model_names)

        data = self.data[self.row_indices]
        domain = data.domain
        class_var = domain.class_var
        classification = class_var and class_var.is_discrete

        new_meta_attr = []
        new_meta_vals = np.empty((len(data), 0))
        names = [var.name for var in chain(domain.attributes,
                                           domain.metas,
                                           domain.class_vars)]

        if classification:
            # predictions
            if include_predictions:
                uniq_new, names = self.create_unique_vars(names, model_names, class_var.values)
                new_meta_attr += uniq_new
                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

            # probabilities
            if include_probabilities:
                proposed = [f"{name} ({value})" for name in model_names for value in class_var.values]

                uniq_new, names = self.create_unique_vars(names, proposed)
                new_meta_attr += uniq_new

                for i in self.probabilities:
                    new_meta_vals = np.hstack((new_meta_vals, i))

        elif include_predictions:
            # regression
            uniq_new, names = self.create_unique_vars(names, model_names)
            new_meta_attr += uniq_new
            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

        # add fold info
        if self.folds is not None:
            values = [str(i + 1) for i in range(len(self.folds))]
            uniq_new, names = self.create_unique_vars(names, ["Fold"], values)
            new_meta_attr += uniq_new
            fold = np.empty((len(data), 1))
            for i, s in enumerate(self.folds):
                fold[s, 0] = i
            new_meta_vals = np.hstack((new_meta_vals, fold))

        # append new columns to meta attributes
        new_meta_attr = list(data.domain.metas) + new_meta_attr
        new_meta_vals = np.hstack((data.metas, new_meta_vals))

        attrs = data.domain.attributes if include_attrs else []
        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)
        predictions = data.transform(domain)
        with predictions.unlocked(predictions.metas):
            predictions.metas = new_meta_vals
        predictions.name = data.name
        return predictions

    def create_unique_vars(self, names, proposed_names, values=()):
        unique_vars = []
        for proposed in proposed_names:
            uniq = get_unique_names(names, proposed)
            if values:
                unique_vars.append(DiscreteVariable(uniq, values))
            else:
                unique_vars.append(ContinuousVariable(uniq))
            names.append(uniq)
        return unique_vars, names

    def split_by_model(self):
        """
        Split evaluation results by models.

        The method generates instances of `Results` containing data for single
        models
        """
        data = self.data
        nmethods = len(self.predicted)
        for i in range(nmethods):
            res = Results()
            res.data = data
            res.domain = self.domain
            res.learners = [self.learners[i]]
            res.row_indices = self.row_indices
            res.actual = self.actual
            res.folds = self.folds
            res.score_by_folds = self.score_by_folds
            res.test_time = self.test_time[i]
            res.train_time = self.train_time[i]

            res.predicted = self.predicted[(i,), :]
            if getattr(self, "probabilities", None) is not None:
                res.probabilities = self.probabilities[(i,), :, :]

            if self.models is not None:
                res.models = self.models[:, i:i + 1]

            res.failed = [self.failed[i]]
            yield res


class Validation:
    """
    Base class for different testing schemata such as cross validation and
    testing on separate data set.

    If `data` is some data table and `learners` is a list of learning
    algorithms. This will run 5-fold cross validation and store the results
    in `res`.

        cv = CrossValidation(k=5)
        res = cv(data, learners)

    If constructor was given data and learning algorithms (as in
    `res = CrossValidation(data, learners, k=5)`, it used to automagically
    call the instance after constructing it and return `Results` instead
    of an instance of `Validation`. This functionality
    is deprecated and will be removed in the future.

    Attributes:
        store_data (bool): a flag defining whether the data is stored
        store_models (bool): a flag defining whether the models are stored
    """
    score_by_folds = False

    def __new__(cls,
                data=None, learners=None, preprocessor=None, test_data=None,
                *, callback=None, store_data=False, store_models=False,
                n_jobs=None, **kwargs):
        self = super().__new__(cls)

        if (learners is None) != (data is None):
            raise ValueError(
                "learners and train_data must both be present or not")
        if learners is None:
            if preprocessor is not None:
                raise ValueError("preprocessor cannot be given if learners "
                                 "and train_data are omitted")
            if callback is not None:
                raise ValueError("callback cannot be given if learners "
                                 "and train_data are omitted")
            return self

        warn("calling Validation's constructor with data and learners "
             "is deprecated;\nconstruct an instance and call it",
             DeprecationWarning, stacklevel=2)

        # Explicitly call __init__ because Python won't
        self.__init__(store_data=store_data, store_models=store_models,
                      **kwargs)
        if test_data is not None:
            test_data_kwargs = {"test_data": test_data}
        else:
            test_data_kwargs = {}
        return self(data, learners=learners, preprocessor=preprocessor,
                    callback=callback, **test_data_kwargs)

    # Note: this will be called only if __new__ doesn't have data and learners
    def __init__(self, *, store_data=False, store_models=False):
        self.store_data = store_data
        self.store_models = store_models

    def fit(self, *args, **kwargs):
        warn("Validation.fit is deprecated; use the call operator",
             DeprecationWarning)
        return self(*args, **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None):
        """
        Args:
            data (Orange.data.Table): data to be used (usually split) into
                training and testing
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity
        indices = self.get_indices(data)
        folds, row_indices, actual = self.prepare_arrays(data, indices)

        data_splits = (
            (fold_i, preprocessor(data[train_i]), data[test_i])
            for fold_i, (train_i, test_i) in enumerate(indices))
        args_iter = (
            (fold_i, data, test_data, learner_i, learner, self.store_models)
            for (fold_i, data, test_data) in data_splits
            for (learner_i, learner) in enumerate(learners))

        part_results = []
        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]
        for progress, part in zip(parts, args_iter):
            part_results.append(_mp_worker(*(part + ())))
            callback(progress)
        callback(1)

        results = Results(
            data=data if self.store_data else None,
            domain=data.domain,
            nrows=len(row_indices), learners=learners,
            row_indices=row_indices, folds=folds, actual=actual,
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (len(indices), len(learners)))
        self._collect_part_results(results, part_results)
        return results

    @classmethod
    def prepare_arrays(cls, data, indices):
        """Prepare `folds`, `row_indices` and `actual`.

        The method is used by `__call__`. While functional, it may be
        overriden in subclasses for speed-ups.

        Args:
            data (Orange.data.Table): data use for testing
            indices (list of vectors):
                indices of data instances in each test sample

        Returns:
            folds: (np.ndarray): see class documentation
            row_indices: (np.ndarray): see class documentation
            actual: (np.ndarray): see class documentation
        """
        folds = []
        row_indices = []

        ptr = 0
        for _, test in indices:
            folds.append(slice(ptr, ptr + len(test)))
            row_indices.append(test)
            ptr += len(test)

        row_indices = np.concatenate(row_indices, axis=0)
        return folds, row_indices, data[row_indices].Y

    @staticmethod
    def get_indices(data):
        """
        Return a list of arrays of indices of test data instance

        For example, in k-fold CV, the result is a list with `k` elements,
        each containing approximately `len(data) / k` nonoverlapping indices
        into `data`.

        This method is abstract and must be implemented in derived classes
        unless they provide their own implementation of the `__call__`
        method.

        Args:
            data (Orange.data.Table): test data

        Returns:
            indices (list of np.ndarray):
                a list of arrays of indices into `data`
        """
        raise NotImplementedError()

    def _collect_part_results(self, results, part_results):
        part_results = sorted(part_results)

        ptr, prev_fold_i, prev_n_values = 0, 0, 0
        for res in part_results:
            if res.fold_i != prev_fold_i:
                ptr += prev_n_values
                prev_fold_i = res.fold_i
            result_slice = slice(ptr, ptr + res.n_values)
            prev_n_values = res.n_values

            if res.failed:
                results.failed[res.learner_i] = res.failed
                continue

            if self.store_models:
                results.models[res.fold_i][res.learner_i] = res.model

            results.predicted[res.learner_i][result_slice] = res.values
            results.train_time[res.learner_i] += res.train_time
            results.test_time[res.learner_i] += res.test_time
            if res.probs is not None:
                results.probabilities[res.learner_i][result_slice, :] = \
                    res.probs


class CrossValidation(Validation):
    """
    K-fold cross validation

    Attributes:
        k (int): number of folds (default: 10)
        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time
        stratified (bool):
            flag deciding whether to perform stratified cross-validation.
            If `True` but the class sizes don't allow it, it uses non-stratified
            validataion and adds a list `warning` with a warning message(s) to
            the `Result`.
    """
    # TODO: list `warning` contains just repetitions of the same message
    #       replace with a flag in `Results`?
    def __init__(self, k=10, stratified=True, random_state=0,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.k = k
        self.stratified = stratified
        self.random_state = random_state
        self.warnings = [] if warnings is None else warnings

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            try:
                splitter = skl.StratifiedKFold(
                    self.k, shuffle=True, random_state=self.random_state
                )
                splitter.get_n_splits(data.X, data.Y)
                return list(splitter.split(data.X, data.Y))
            except ValueError:
                self.warnings.append("Using non-stratified sampling.")

        splitter = skl.KFold(
            self.k, shuffle=True, random_state=self.random_state)
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class CrossValidationFeature(Validation):
    """
    Cross validation with folds according to values of a feature.

    Attributes:
        feature (Orange.data.Variable): the feature defining the folds
    """
    def __init__(self, feature=None,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.feature = feature

    def get_indices(self, data):
        data = data.transform(Domain([self.feature], None))
        values = data[:, self.feature].X
        indices = []
        for v in range(len(self.feature.values)):
            test_index = np.where(values == v)[0]
            train_index = np.where((values != v) & (~np.isnan(values)))[0]
            if test_index.size and train_index.size:
                indices.append((train_index, test_index))
        if not indices:
            raise ValueError(
                f"'{self.feature.name}' does not have at least two distinct "
                "values on the data")
        return indices


class LeaveOneOut(Validation):
    """Leave-one-out testing"""
    score_by_folds = False

    def get_indices(self, data):
        splitter = skl.LeaveOneOut()
        splitter.get_n_splits(data)
        return list(splitter.split(data))

    @staticmethod
    def prepare_arrays(data, indices):
        # sped up version of super().prepare_arrays(data)
        row_indices = np.arange(len(data))
        return row_indices, row_indices, data.Y.flatten()


class ShuffleSplit(Validation):
    """
    Test by repeated random sampling

    Attributes:
        n_resamples (int): number of repetitions
        test_size (float, int, None):
            If float, should be between 0.0 and 1.0 and represent the proportion
            of the dataset to include in the test split. If int, represents the
            absolute number of test samples. If None, the value is set to the
            complement of the train size. By default, the value is set to 0.1.
            The default will change in version 0.21. It will remain 0.1 only
            if ``train_size`` is unspecified, otherwise it will complement
            the specified ``train_size``.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        train_size : float, int, or None, default is None
            If float, should be between 0.0 and 1.0 and represent the
            proportion of the dataset to include in the train split. If
            int, represents the absolute number of train samples. If None,
            the value is automatically set to the complement of the test size.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        stratified (bool):
            flag deciding whether to perform stratified cross-validation.

        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time

    """
    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,
                 stratified=True, random_state=0,
                 store_data=False, store_models=False):
        super().__init__(store_data=store_data, store_models=store_models)
        self.n_resamples = n_resamples
        self.train_size = train_size
        self.test_size = test_size
        self.stratified = stratified
        self.random_state = random_state

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            splitter = skl.StratifiedShuffleSplit(
                n_splits=self.n_resamples, train_size=self.train_size,
                test_size=self.test_size, random_state=self.random_state
            )
            splitter.get_n_splits(data.X, data.Y)
            return list(splitter.split(data.X, data.Y))

        splitter = skl.ShuffleSplit(
            n_splits=self.n_resamples, train_size=self.train_size,
            test_size=self.test_size, random_state=self.random_state
        )
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class TestOnTestData(Validation):
    """
    Test on separately provided test data

    Note that the class has a different signature for `__call__`.
    """
    # get_indices is not needed in this class, pylint: disable=abstract-method

    def __new__(cls, data=None, test_data=None, learners=None,
                preprocessor=None, **kwargs):
        if "train_data" in kwargs:
            if data is None:
                data = kwargs.pop("train_data")
            else:
                raise ValueError(
                    "argument 'data' is given twice (once as 'train_data')")
        return super().__new__(
            cls,
            data=data, learners=learners, preprocessor=preprocessor,
            test_data=test_data, **kwargs)

    def __call__(self, data, test_data, learners, preprocessor=None,
                 *, callback=None):
        """
        Args:
            data (Orange.data.Table): training data
            test_data (Orange.data.Table): test_data
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity

        train_data = preprocessor(data)
        part_results = []
        for (learner_i, learner) in enumerate(learners):
            part_results.append(
                _mp_worker(0, train_data, test_data, learner_i, learner,
                           self.store_models))
            callback((learner_i + 1) / len(learners))
        callback(1)

        results = Results(
            data=test_data if self.store_data else None,
            domain=test_data.domain,
            nrows=len(test_data), learners=learners,
            row_indices=np.arange(len(test_data)),
            folds=(Ellipsis, ),
            actual=test_data.Y,
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (1, len(learners)))
        self._collect_part_results(results, part_results)
        return results


class TestOnTrainingData(TestOnTestData):
    """Test on training data"""
    # get_indices is not needed in this class, pylint: disable=abstract-method
    # signature is such as on the base class, pylint: disable=signature-differs
    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):
        return super().__new__(
            cls,
            data, test_data=data, learners=learners, preprocessor=preprocessor,
            **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None,
                 **kwargs):
        kwargs.setdefault("test_data", data)
        # if kwargs contains anything besides test_data, this will be detected
        # (and complained about) by super().__call__
        return super().__call__(
            data=data, learners=learners, preprocessor=preprocessor,
            callback=callback, **kwargs)


def sample(table, n=0.7, stratified=False, replace=False,
           random_state=None):
    """
    Samples data instances from a data table. Returns the sample and
    a dataset from input data table that are not in the sample. Also
    uses several sampling functions from
    `scikit-learn <http://scikit-learn.org>`_.

    table : data table
        A data table from which to sample.

    n : float, int (default = 0.7)
        If float, should be between 0.0 and 1.0 and represents
        the proportion of data instances in the resulting sample. If
        int, n is the number of data instances in the resulting sample.

    stratified : bool, optional (default = False)
        If true, sampling will try to consider class values and
        match distribution of class values
        in train and test subsets.

    replace : bool, optional (default = False)
        sample with replacement

    random_state : int or RandomState
        Pseudo-random number generator state used for random sampling.
    """

    if isinstance(n, float):
        n = int(n * len(table))

    if replace:
        if random_state is None:
            rgen = np.random
        else:
            rgen = np.random.mtrand.RandomState(random_state)
        a_sample = rgen.randint(0, len(table), n)
        o = np.ones(len(table))
        o[a_sample] = 0
        others = np.nonzero(o)[0]
        return table[a_sample], table[others]

    n = len(table) - n
    if stratified and table.domain.has_discrete_class:
        test_size = max(len(table.domain.class_var.values), n)
        splitter = skl.StratifiedShuffleSplit(
            n_splits=1, test_size=test_size, train_size=len(table) - test_size,
            random_state=random_state)
        splitter.get_n_splits(table.X, table.Y)
        ind = splitter.split(table.X, table.Y)
    else:
        splitter = skl.ShuffleSplit(
            n_splits=1, test_size=n, random_state=random_state)
        splitter.get_n_splits(table)
        ind = splitter.split(table)
    ind = next(ind)
    return table[ind[0]], table[ind[1]]

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user