{
    "file_diffs": [
        {
            "old_file_content": "# __new__ methods have different arguments\n# pylint: disable=arguments-differ\nfrom warnings import warn\nfrom collections import namedtuple\nfrom time import time\n\nimport numpy as np\n\nimport sklearn.model_selection as skl\n\nfrom Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable\n\n__all__ = [\"Results\", \"CrossValidation\", \"LeaveOneOut\", \"TestOnTrainingData\",\n           \"ShuffleSplit\", \"TestOnTestData\", \"sample\", \"CrossValidationFeature\"]\n\n_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',\n                                       'failed', 'n_values', 'values',\n                                       'probs', 'train_time', 'test_time'))\n\n\ndef _identity(x):\n    return x\n\n\ndef _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed = None, None, None, False\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0  = time()\n        # testing\n        if train_data.domain.has_discrete_class:\n            predicted, probs = model(test_data, model.ValueProbs)\n        elif train_data.domain.has_continuous_class:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)\n\n\nclass Results:\n    \"\"\"\n    Class for storing predictions in model testing.\n\n    Attributes:\n        data (Optional[Table]):  Data used for testing.\n\n        models (Optional[List[Model]]): A list of induced models.\n\n        row_indices (np.ndarray): Indices of rows in `data` that were used in\n            testing, stored as a numpy vector of length `nrows`.\n            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer\n            to the target value of instance, that is, the i-th test instance\n            is `data[row_indices[i]]`, its actual class is `actual[i]`, and\n            the prediction by m-th method is `predicted[m, i]`.\n\n        nrows (int): The number of test instances (including duplicates);\n            `nrows` equals the length of `row_indices` and `actual`, and the\n            second dimension of `predicted` and `probabilities`.\n\n        actual (np.ndarray): true values of target variable in a vector of\n            length `nrows`.\n\n        predicted (np.ndarray): predicted values of target variable in an array\n            of shape (number-of-methods, `nrows`)\n\n        probabilities (Optional[np.ndarray]): predicted probabilities\n            (for discrete target variables) in an array of shape\n            (number-of-methods, `nrows`, number-of-classes)\n\n        folds (List[Slice or List[int]]): a list of indices (or slice objects)\n            corresponding to testing data subsets, that is,\n            `row_indices[folds[i]]` contains row indices used in fold i, so\n            `data[row_indices[folds[i]]]` is the corresponding testing data\n\n        train_time (np.ndarray): training times of batches\n\n        test_time (np.ndarray): testing times of batches\n    \"\"\"\n    def __init__(self, data=None, *,\n                 nmethods=None, nrows=None, nclasses=None,\n                 domain=None,\n                 row_indices=None, folds=None, score_by_folds=True,\n                 learners=None, models=None, failed=None,\n                 actual=None, predicted=None, probabilities=None,\n                 store_data=None, store_models=None,\n                 train_time=None, test_time=None):\n        \"\"\"\n        Construct an instance.\n\n        The constructor stores the given data, and creates empty arrays\n        `actual`, `predicted` and `probabilities` if ther are not given but\n        sufficient data is provided to deduct their shapes.\n\n        The function\n\n        - set any attributes specified directly through arguments.\n        - infers the number of methods, rows and classes from other data\n           and/or check their overall consistency.\n        - Prepare empty arrays `actual`, `predicted`, `probabilities` and\n          `failed` if the are not given. If not enough data is available,\n          the corresponding arrays are `None`.\n\n        Args:\n            data (Orange.data.Table): stored data from which test was sampled\n            nmethods (int): number of methods; can be inferred (or must match)\n                the size of `learners`, `models`, `failed`, `predicted` and\n                `probabilities`\n            nrows (int): number of data instances; can be inferred (or must\n                match) `data`, `row_indices`, `actual`, `predicted` and\n                `probabilities`\n            nclasses (int): number of class values (`None` if continuous); can\n                be inferred (or must match) from `domain.class_var` or\n                `probabilities`\n            domain (Orange.data.Domain): data domain; can be inferred (or must)\n                match `data.domain`\n            row_indices (np.ndarray): see class documentation\n            folds (np.ndarray): see class documentation\n            score_by_folds (np.ndarray): see class documentation\n            learners (np.ndarray): see class documentation\n            models (np.ndarray): see class documentation\n            failed (list of str): see class documentation\n            actual (np.ndarray): see class documentation\n            predicted (np.ndarray): see class documentation\n            probabilities (np.ndarray): see class documentation\n            store_data (bool): ignored; kept for backward compatibility\n            store_models (bool): ignored; kept for backward compatibility\n        \"\"\"\n\n        # Set given data directly from arguments\n        self.data = data\n        self.domain = domain\n\n        self.row_indices = row_indices\n        self.folds = folds\n        self.score_by_folds = score_by_folds\n\n        self.learners = learners\n        self.models = models\n\n        self.actual = actual\n        self.predicted = predicted\n        self.probabilities = probabilities\n        self.failed = failed\n\n        self.train_time = train_time\n        self.test_time = test_time\n\n        # Guess the rest -- or check for ambguities\n        def set_or_raise(value, exp_values, msg):\n            for exp_value in exp_values:\n                if exp_value is False:\n                    continue\n                if value is None:\n                    value = exp_value\n                elif value != exp_value:\n                    raise ValueError(msg)\n            return value\n        domain = self.domain = set_or_raise(\n            domain, [data is not None and data.domain],\n            \"mismatching domain\")\n        self.nrows = nrows = set_or_raise(\n            nrows, [actual is not None and len(actual),\n                    row_indices is not None and len(row_indices),\n                    predicted is not None and predicted.shape[1],\n                    probabilities is not None and probabilities.shape[1]],\n            \"mismatching number of rows\")\n        if domain is not None and domain.has_continuous_class:\n            if nclasses is not None:\n                raise ValueError(\n                    \"regression results cannot have non-None 'nclasses'\")\n            if probabilities is not None:\n                raise ValueError(\n                    \"regression results cannot have 'probabilities'\")\n        nclasses = set_or_raise(\n            nclasses, [domain is not None and domain.has_discrete_class and\n                       len(domain.class_var.values),\n                       probabilities is not None and probabilities.shape[2]],\n            \"mismatching number of class values\")\n        nmethods = set_or_raise(\n            nmethods, [learners is not None and len(learners),\n                       models is not None and models.shape[1],\n                       failed is not None and len(failed),\n                       predicted is not None and predicted.shape[0],\n                       probabilities is not None and probabilities.shape[0]],\n            \"mismatching number of methods\")\n\n        # Prepare empty arrays\n        if actual is None \\\n                and nrows is not None:\n            self.actual = np.empty(nrows)\n\n        if predicted is None \\\n                and nmethods is not None and nrows is not None:\n            self.predicted = np.empty((nmethods, nrows))\n\n        if probabilities is None \\\n                and nmethods is not None and nrows is not None \\\n                and nclasses is not None:\n            self.probabilities = \\\n                np.empty((nmethods, nrows, nclasses))\n\n        if failed is None \\\n                and nmethods is not None:\n            self.failed = [False] * nmethods\n\n    def get_fold(self, fold):\n        results = Results()\n        results.data = self.data\n\n        if self.folds is None:\n            raise ValueError(\"This 'Results' instance does not have folds.\")\n\n        if self.models is not None:\n            results.models = self.models[fold]\n\n        results.row_indices = self.row_indices[self.folds[fold]]\n        results.actual = self.actual[self.folds[fold]]\n        results.predicted = self.predicted[:, self.folds[fold]]\n        results.domain = self.domain\n\n        if self.probabilities is not None:\n            results.probabilities = self.probabilities[:, self.folds[fold]]\n\n        return results\n\n    def get_augmented_data(self, model_names,\n                           include_attrs=True, include_predictions=True,\n                           include_probabilities=True):\n        \"\"\"\n        Return the test data table augmented with meta attributes containing\n        predictions, probabilities (if the task is classification) and fold\n        indices.\n\n        Args:\n            model_names (list of str): names of models\n            include_attrs (bool):\n                if set to `False`, original attributes are removed\n            include_predictions (bool):\n                if set to `False`, predictions are not added\n            include_probabilities (bool):\n                if set to `False`, probabilities are not added\n\n        Returns:\n            augmented_data (Orange.data.Table):\n                data augmented with predictions, probabilities and fold indices\n\n        \"\"\"\n        assert self.predicted.shape[0] == len(model_names)\n\n        data = self.data[self.row_indices]\n        class_var = data.domain.class_var\n        classification = class_var and class_var.is_discrete\n\n        new_meta_attr = []\n        new_meta_vals = np.empty((len(data), 0))\n\n        if classification:\n            # predictions\n            if include_predictions:\n                new_meta_attr += (\n                    DiscreteVariable(name=name, values=class_var.values)\n                    for name in model_names)\n                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n            # probabilities\n            if include_probabilities:\n                for name in model_names:\n                    new_meta_attr += (\n                        ContinuousVariable(name=f\"{name} ({value})\")\n                        for value in class_var.values)\n\n                for i in self.probabilities:\n                    new_meta_vals = np.hstack((new_meta_vals, i))\n\n        elif include_predictions:\n            # regression\n            new_meta_attr += (ContinuousVariable(name=name)\n                              for name in model_names)\n            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n        # add fold info\n        if self.folds is not None:\n            new_meta_attr.append(\n                DiscreteVariable(\n                    name=\"Fold\",\n                    values=[str(i + 1) for i in range(len(self.folds))]))\n            fold = np.empty((len(data), 1))\n            for i, s in enumerate(self.folds):\n                fold[s, 0] = i\n            new_meta_vals = np.hstack((new_meta_vals, fold))\n\n        # append new columns to meta attributes\n        new_meta_attr = list(data.domain.metas) + new_meta_attr\n        new_meta_vals = np.hstack((data.metas, new_meta_vals))\n\n        attrs = data.domain.attributes if include_attrs else []\n        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)\n        predictions = data.transform(domain)\n        predictions.metas = new_meta_vals\n        predictions.name = data.name\n        return predictions\n\n    def split_by_model(self):\n        \"\"\"\n        Split evaluation results by models.\n\n        The method generates instances of `Results` containing data for single\n        models\n        \"\"\"\n        data = self.data\n        nmethods = len(self.predicted)\n        for i in range(nmethods):\n            res = Results()\n            res.data = data\n            res.domain = self.domain\n            res.learners = [self.learners[i]]\n            res.row_indices = self.row_indices\n            res.actual = self.actual\n            res.folds = self.folds\n            res.score_by_folds = self.score_by_folds\n            res.test_time = self.test_time[i]\n            res.train_time = self.train_time[i]\n\n            res.predicted = self.predicted[(i,), :]\n            if getattr(self, \"probabilities\", None) is not None:\n                res.probabilities = self.probabilities[(i,), :, :]\n\n            if self.models is not None:\n                res.models = self.models[:, i:i + 1]\n\n            res.failed = [self.failed[i]]\n            yield res\n\n\nclass Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        actual = data[row_indices].Y.ravel()\n        return folds, row_indices, actual\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs\n\n\nclass CrossValidation(Validation):\n    \"\"\"\n    K-fold cross validation\n\n    Attributes:\n        k (int): number of folds (default: 10)\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n            If `True` but the class sizes don't allow it, it uses non-stratified\n            validataion and adds a list `warning` with a warning message(s) to\n            the `Result`.\n    \"\"\"\n    # TODO: list `warning` contains just repetitions of the same message\n    #       replace with a flag in `Results`?\n    def __init__(self, k=10, stratified=True, random_state=0,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.k = k\n        self.stratified = stratified\n        self.random_state = random_state\n        self.warnings = [] if warnings is None else warnings\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            try:\n                splitter = skl.StratifiedKFold(\n                    self.k, shuffle=True, random_state=self.random_state\n                )\n                splitter.get_n_splits(data.X, data.Y)\n                return list(splitter.split(data.X, data.Y))\n            except ValueError:\n                self.warnings.append(\"Using non-stratified sampling.\")\n\n        splitter = skl.KFold(\n            self.k, shuffle=True, random_state=self.random_state)\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass CrossValidationFeature(Validation):\n    \"\"\"\n    Cross validation with folds according to values of a feature.\n\n    Attributes:\n        feature (Orange.data.Variable): the feature defining the folds\n    \"\"\"\n    def __init__(self, feature=None,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.feature = feature\n\n    def get_indices(self, data):\n        data = Table(Domain([self.feature], None), data)\n        values = data[:, self.feature].X\n        indices = []\n        for v in range(len(self.feature.values)):\n            test_index = np.where(values == v)[0]\n            train_index = np.where((values != v) & (~np.isnan(values)))[0]\n            if test_index.size and train_index.size:\n                indices.append((train_index, test_index))\n        if not indices:\n            raise ValueError(\n                f\"'{self.feature.name}' does not have at least two distinct \"\n                \"values on the data\")\n        return indices\n\n\nclass LeaveOneOut(Validation):\n    \"\"\"Leave-one-out testing\"\"\"\n    score_by_folds = False\n\n    def get_indices(self, data):\n        splitter = skl.LeaveOneOut()\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n    @staticmethod\n    def prepare_arrays(data, indices):\n        # sped up version of super().prepare_arrays(data)\n        row_indices = np.arange(len(data))\n        return row_indices, row_indices, data.Y.flatten()\n\n\nclass ShuffleSplit(Validation):\n    \"\"\"\n    Test by repeated random sampling\n\n    Attributes:\n        n_resamples (int): number of repetitions\n        test_size (float, int, None):\n            If float, should be between 0.0 and 1.0 and represent the proportion\n            of the dataset to include in the test split. If int, represents the\n            absolute number of test samples. If None, the value is set to the\n            complement of the train size. By default, the value is set to 0.1.\n            The default will change in version 0.21. It will remain 0.1 only\n            if ``train_size`` is unspecified, otherwise it will complement\n            the specified ``train_size``.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        train_size : float, int, or None, default is None\n            If float, should be between 0.0 and 1.0 and represent the\n            proportion of the dataset to include in the train split. If\n            int, represents the absolute number of train samples. If None,\n            the value is automatically set to the complement of the test size.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n\n    \"\"\"\n    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,\n                 stratified=True, random_state=0,\n                 store_data=False, store_models=False):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.n_resamples = n_resamples\n        self.train_size = train_size\n        self.test_size = test_size\n        self.stratified = stratified\n        self.random_state = random_state\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            splitter = skl.StratifiedShuffleSplit(\n                n_splits=self.n_resamples, train_size=self.train_size,\n                test_size=self.test_size, random_state=self.random_state\n            )\n            splitter.get_n_splits(data.X, data.Y)\n            return list(splitter.split(data.X, data.Y))\n\n        splitter = skl.ShuffleSplit(\n            n_splits=self.n_resamples, train_size=self.train_size,\n            test_size=self.test_size, random_state=self.random_state\n        )\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass TestOnTestData(Validation):\n    \"\"\"\n    Test on separately provided test data\n\n    Note that the class has a different signature for `__call__`.\n    \"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n\n    def __new__(cls, data=None, test_data=None, learners=None,\n                preprocessor=None, **kwargs):\n        if \"train_data\" in kwargs:\n            if data is None:\n                data = kwargs.pop(\"train_data\")\n            else:\n                raise ValueError(\n                    \"argument 'data' is given twice (once as 'train_data')\")\n        return super().__new__(\n            cls,\n            data=data, learners=learners, preprocessor=preprocessor,\n            test_data=test_data, **kwargs)\n\n    def __call__(self, data, test_data, learners, preprocessor=None,\n                 *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): training data\n            test_data (Orange.data.Table): test_data\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n\n        train_data = preprocessor(data)\n        part_results = []\n        for (learner_i, learner) in enumerate(learners):\n            part_results.append(\n                _mp_worker(0, train_data, test_data, learner_i, learner,\n                           self.store_models))\n            callback((learner_i + 1) / len(learners))\n        callback(1)\n\n        results = Results(\n            data=test_data if self.store_data else None,\n            domain=test_data.domain,\n            nrows=len(test_data), learners=learners,\n            row_indices=np.arange(len(test_data)),\n            folds=(Ellipsis, ),\n            actual=test_data.Y.ravel(),\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (1, len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n\nclass TestOnTrainingData(TestOnTestData):\n    \"\"\"Test on training data\"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n    # signature is such as on the base class, pylint: disable=signature-differs\n    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):\n        return super().__new__(\n            cls,\n            data, test_data=data, learners=learners, preprocessor=preprocessor,\n            **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None,\n                 **kwargs):\n        kwargs.setdefault(\"test_data\", data)\n        # if kwargs contains anything besides test_data, this will be detected\n        # (and complained about) by super().__call__\n        return super().__call__(\n            data=data, learners=learners, preprocessor=preprocessor,\n            callback=callback, **kwargs)\n\n\ndef sample(table, n=0.7, stratified=False, replace=False,\n           random_state=None):\n    \"\"\"\n    Samples data instances from a data table. Returns the sample and\n    a dataset from input data table that are not in the sample. Also\n    uses several sampling functions from\n    `scikit-learn <http://scikit-learn.org>`_.\n\n    table : data table\n        A data table from which to sample.\n\n    n : float, int (default = 0.7)\n        If float, should be between 0.0 and 1.0 and represents\n        the proportion of data instances in the resulting sample. If\n        int, n is the number of data instances in the resulting sample.\n\n    stratified : bool, optional (default = False)\n        If true, sampling will try to consider class values and\n        match distribution of class values\n        in train and test subsets.\n\n    replace : bool, optional (default = False)\n        sample with replacement\n\n    random_state : int or RandomState\n        Pseudo-random number generator state used for random sampling.\n    \"\"\"\n\n    if isinstance(n, float):\n        n = int(n * len(table))\n\n    if replace:\n        if random_state is None:\n            rgen = np.random\n        else:\n            rgen = np.random.mtrand.RandomState(random_state)\n        a_sample = rgen.randint(0, len(table), n)\n        o = np.ones(len(table))\n        o[a_sample] = 0\n        others = np.nonzero(o)[0]\n        return table[a_sample], table[others]\n\n    n = len(table) - n\n    if stratified and table.domain.has_discrete_class:\n        test_size = max(len(table.domain.class_var.values), n)\n        splitter = skl.StratifiedShuffleSplit(\n            n_splits=1, test_size=test_size, train_size=len(table) - test_size,\n            random_state=random_state)\n        splitter.get_n_splits(table.X, table.Y)\n        ind = splitter.split(table.X, table.Y)\n    else:\n        splitter = skl.ShuffleSplit(\n            n_splits=1, test_size=n, random_state=random_state)\n        splitter.get_n_splits(table)\n        ind = splitter.split(table)\n    ind = next(ind)\n    return table[ind[0]], table[ind[1]]\n",
            "new_file_content": "# __new__ methods have different arguments\n# pylint: disable=arguments-differ\nfrom warnings import warn\nfrom collections import namedtuple\nfrom time import time\n\nimport numpy as np\n\nimport sklearn.model_selection as skl\n\nfrom Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable\n\n__all__ = [\"Results\", \"CrossValidation\", \"LeaveOneOut\", \"TestOnTrainingData\",\n           \"ShuffleSplit\", \"TestOnTestData\", \"sample\", \"CrossValidationFeature\"]\n\n_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',\n                                       'failed', 'n_values', 'values',\n                                       'probs', 'train_time', 'test_time'))\n\n\ndef _identity(x):\n    return x\n\n\ndef _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed, train_time, test_time = None, None, None, False, None, None\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0  = time()\n        # testing\n        if train_data.domain.has_discrete_class:\n            predicted, probs = model(test_data, model.ValueProbs)\n        elif train_data.domain.has_continuous_class:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)\n\n\nclass Results:\n    \"\"\"\n    Class for storing predictions in model testing.\n\n    Attributes:\n        data (Optional[Table]):  Data used for testing.\n\n        models (Optional[List[Model]]): A list of induced models.\n\n        row_indices (np.ndarray): Indices of rows in `data` that were used in\n            testing, stored as a numpy vector of length `nrows`.\n            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer\n            to the target value of instance, that is, the i-th test instance\n            is `data[row_indices[i]]`, its actual class is `actual[i]`, and\n            the prediction by m-th method is `predicted[m, i]`.\n\n        nrows (int): The number of test instances (including duplicates);\n            `nrows` equals the length of `row_indices` and `actual`, and the\n            second dimension of `predicted` and `probabilities`.\n\n        actual (np.ndarray): true values of target variable in a vector of\n            length `nrows`.\n\n        predicted (np.ndarray): predicted values of target variable in an array\n            of shape (number-of-methods, `nrows`)\n\n        probabilities (Optional[np.ndarray]): predicted probabilities\n            (for discrete target variables) in an array of shape\n            (number-of-methods, `nrows`, number-of-classes)\n\n        folds (List[Slice or List[int]]): a list of indices (or slice objects)\n            corresponding to testing data subsets, that is,\n            `row_indices[folds[i]]` contains row indices used in fold i, so\n            `data[row_indices[folds[i]]]` is the corresponding testing data\n\n        train_time (np.ndarray): training times of batches\n\n        test_time (np.ndarray): testing times of batches\n    \"\"\"\n    def __init__(self, data=None, *,\n                 nmethods=None, nrows=None, nclasses=None,\n                 domain=None,\n                 row_indices=None, folds=None, score_by_folds=True,\n                 learners=None, models=None, failed=None,\n                 actual=None, predicted=None, probabilities=None,\n                 store_data=None, store_models=None,\n                 train_time=None, test_time=None):\n        \"\"\"\n        Construct an instance.\n\n        The constructor stores the given data, and creates empty arrays\n        `actual`, `predicted` and `probabilities` if ther are not given but\n        sufficient data is provided to deduct their shapes.\n\n        The function\n\n        - set any attributes specified directly through arguments.\n        - infers the number of methods, rows and classes from other data\n           and/or check their overall consistency.\n        - Prepare empty arrays `actual`, `predicted`, `probabilities` and\n          `failed` if the are not given. If not enough data is available,\n          the corresponding arrays are `None`.\n\n        Args:\n            data (Orange.data.Table): stored data from which test was sampled\n            nmethods (int): number of methods; can be inferred (or must match)\n                the size of `learners`, `models`, `failed`, `predicted` and\n                `probabilities`\n            nrows (int): number of data instances; can be inferred (or must\n                match) `data`, `row_indices`, `actual`, `predicted` and\n                `probabilities`\n            nclasses (int): number of class values (`None` if continuous); can\n                be inferred (or must match) from `domain.class_var` or\n                `probabilities`\n            domain (Orange.data.Domain): data domain; can be inferred (or must)\n                match `data.domain`\n            row_indices (np.ndarray): see class documentation\n            folds (np.ndarray): see class documentation\n            score_by_folds (np.ndarray): see class documentation\n            learners (np.ndarray): see class documentation\n            models (np.ndarray): see class documentation\n            failed (list of str): see class documentation\n            actual (np.ndarray): see class documentation\n            predicted (np.ndarray): see class documentation\n            probabilities (np.ndarray): see class documentation\n            store_data (bool): ignored; kept for backward compatibility\n            store_models (bool): ignored; kept for backward compatibility\n        \"\"\"\n\n        # Set given data directly from arguments\n        self.data = data\n        self.domain = domain\n\n        self.row_indices = row_indices\n        self.folds = folds\n        self.score_by_folds = score_by_folds\n\n        self.learners = learners\n        self.models = models\n\n        self.actual = actual\n        self.predicted = predicted\n        self.probabilities = probabilities\n        self.failed = failed\n\n        self.train_time = train_time\n        self.test_time = test_time\n\n        # Guess the rest -- or check for ambguities\n        def set_or_raise(value, exp_values, msg):\n            for exp_value in exp_values:\n                if exp_value is False:\n                    continue\n                if value is None:\n                    value = exp_value\n                elif value != exp_value:\n                    raise ValueError(msg)\n            return value\n        domain = self.domain = set_or_raise(\n            domain, [data is not None and data.domain],\n            \"mismatching domain\")\n        self.nrows = nrows = set_or_raise(\n            nrows, [actual is not None and len(actual),\n                    row_indices is not None and len(row_indices),\n                    predicted is not None and predicted.shape[1],\n                    probabilities is not None and probabilities.shape[1]],\n            \"mismatching number of rows\")\n        if domain is not None and domain.has_continuous_class:\n            if nclasses is not None:\n                raise ValueError(\n                    \"regression results cannot have non-None 'nclasses'\")\n            if probabilities is not None:\n                raise ValueError(\n                    \"regression results cannot have 'probabilities'\")\n        nclasses = set_or_raise(\n            nclasses, [domain is not None and domain.has_discrete_class and\n                       len(domain.class_var.values),\n                       probabilities is not None and probabilities.shape[2]],\n            \"mismatching number of class values\")\n        nmethods = set_or_raise(\n            nmethods, [learners is not None and len(learners),\n                       models is not None and models.shape[1],\n                       failed is not None and len(failed),\n                       predicted is not None and predicted.shape[0],\n                       probabilities is not None and probabilities.shape[0]],\n            \"mismatching number of methods\")\n\n        # Prepare empty arrays\n        if actual is None \\\n                and nrows is not None:\n            self.actual = np.empty(nrows)\n\n        if predicted is None \\\n                and nmethods is not None and nrows is not None:\n            self.predicted = np.empty((nmethods, nrows))\n\n        if probabilities is None \\\n                and nmethods is not None and nrows is not None \\\n                and nclasses is not None:\n            self.probabilities = \\\n                np.empty((nmethods, nrows, nclasses))\n\n        if failed is None \\\n                and nmethods is not None:\n            self.failed = [False] * nmethods\n\n    def get_fold(self, fold):\n        results = Results()\n        results.data = self.data\n\n        if self.folds is None:\n            raise ValueError(\"This 'Results' instance does not have folds.\")\n\n        if self.models is not None:\n            results.models = self.models[fold]\n\n        results.row_indices = self.row_indices[self.folds[fold]]\n        results.actual = self.actual[self.folds[fold]]\n        results.predicted = self.predicted[:, self.folds[fold]]\n        results.domain = self.domain\n\n        if self.probabilities is not None:\n            results.probabilities = self.probabilities[:, self.folds[fold]]\n\n        return results\n\n    def get_augmented_data(self, model_names,\n                           include_attrs=True, include_predictions=True,\n                           include_probabilities=True):\n        \"\"\"\n        Return the test data table augmented with meta attributes containing\n        predictions, probabilities (if the task is classification) and fold\n        indices.\n\n        Args:\n            model_names (list of str): names of models\n            include_attrs (bool):\n                if set to `False`, original attributes are removed\n            include_predictions (bool):\n                if set to `False`, predictions are not added\n            include_probabilities (bool):\n                if set to `False`, probabilities are not added\n\n        Returns:\n            augmented_data (Orange.data.Table):\n                data augmented with predictions, probabilities and fold indices\n\n        \"\"\"\n        assert self.predicted.shape[0] == len(model_names)\n\n        data = self.data[self.row_indices]\n        class_var = data.domain.class_var\n        classification = class_var and class_var.is_discrete\n\n        new_meta_attr = []\n        new_meta_vals = np.empty((len(data), 0))\n\n        if classification:\n            # predictions\n            if include_predictions:\n                new_meta_attr += (\n                    DiscreteVariable(name=name, values=class_var.values)\n                    for name in model_names)\n                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n            # probabilities\n            if include_probabilities:\n                for name in model_names:\n                    new_meta_attr += (\n                        ContinuousVariable(name=f\"{name} ({value})\")\n                        for value in class_var.values)\n\n                for i in self.probabilities:\n                    new_meta_vals = np.hstack((new_meta_vals, i))\n\n        elif include_predictions:\n            # regression\n            new_meta_attr += (ContinuousVariable(name=name)\n                              for name in model_names)\n            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n        # add fold info\n        if self.folds is not None:\n            new_meta_attr.append(\n                DiscreteVariable(\n                    name=\"Fold\",\n                    values=[str(i + 1) for i in range(len(self.folds))]))\n            fold = np.empty((len(data), 1))\n            for i, s in enumerate(self.folds):\n                fold[s, 0] = i\n            new_meta_vals = np.hstack((new_meta_vals, fold))\n\n        # append new columns to meta attributes\n        new_meta_attr = list(data.domain.metas) + new_meta_attr\n        new_meta_vals = np.hstack((data.metas, new_meta_vals))\n\n        attrs = data.domain.attributes if include_attrs else []\n        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)\n        predictions = data.transform(domain)\n        predictions.metas = new_meta_vals\n        predictions.name = data.name\n        return predictions\n\n    def split_by_model(self):\n        \"\"\"\n        Split evaluation results by models.\n\n        The method generates instances of `Results` containing data for single\n        models\n        \"\"\"\n        data = self.data\n        nmethods = len(self.predicted)\n        for i in range(nmethods):\n            res = Results()\n            res.data = data\n            res.domain = self.domain\n            res.learners = [self.learners[i]]\n            res.row_indices = self.row_indices\n            res.actual = self.actual\n            res.folds = self.folds\n            res.score_by_folds = self.score_by_folds\n            res.test_time = self.test_time[i]\n            res.train_time = self.train_time[i]\n\n            res.predicted = self.predicted[(i,), :]\n            if getattr(self, \"probabilities\", None) is not None:\n                res.probabilities = self.probabilities[(i,), :, :]\n\n            if self.models is not None:\n                res.models = self.models[:, i:i + 1]\n\n            res.failed = [self.failed[i]]\n            yield res\n\n\nclass Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        actual = data[row_indices].Y.ravel()\n        return folds, row_indices, actual\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs\n\n\nclass CrossValidation(Validation):\n    \"\"\"\n    K-fold cross validation\n\n    Attributes:\n        k (int): number of folds (default: 10)\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n            If `True` but the class sizes don't allow it, it uses non-stratified\n            validataion and adds a list `warning` with a warning message(s) to\n            the `Result`.\n    \"\"\"\n    # TODO: list `warning` contains just repetitions of the same message\n    #       replace with a flag in `Results`?\n    def __init__(self, k=10, stratified=True, random_state=0,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.k = k\n        self.stratified = stratified\n        self.random_state = random_state\n        self.warnings = [] if warnings is None else warnings\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            try:\n                splitter = skl.StratifiedKFold(\n                    self.k, shuffle=True, random_state=self.random_state\n                )\n                splitter.get_n_splits(data.X, data.Y)\n                return list(splitter.split(data.X, data.Y))\n            except ValueError:\n                self.warnings.append(\"Using non-stratified sampling.\")\n\n        splitter = skl.KFold(\n            self.k, shuffle=True, random_state=self.random_state)\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass CrossValidationFeature(Validation):\n    \"\"\"\n    Cross validation with folds according to values of a feature.\n\n    Attributes:\n        feature (Orange.data.Variable): the feature defining the folds\n    \"\"\"\n    def __init__(self, feature=None,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.feature = feature\n\n    def get_indices(self, data):\n        data = Table(Domain([self.feature], None), data)\n        values = data[:, self.feature].X\n        indices = []\n        for v in range(len(self.feature.values)):\n            test_index = np.where(values == v)[0]\n            train_index = np.where((values != v) & (~np.isnan(values)))[0]\n            if test_index.size and train_index.size:\n                indices.append((train_index, test_index))\n        if not indices:\n            raise ValueError(\n                f\"'{self.feature.name}' does not have at least two distinct \"\n                \"values on the data\")\n        return indices\n\n\nclass LeaveOneOut(Validation):\n    \"\"\"Leave-one-out testing\"\"\"\n    score_by_folds = False\n\n    def get_indices(self, data):\n        splitter = skl.LeaveOneOut()\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n    @staticmethod\n    def prepare_arrays(data, indices):\n        # sped up version of super().prepare_arrays(data)\n        row_indices = np.arange(len(data))\n        return row_indices, row_indices, data.Y.flatten()\n\n\nclass ShuffleSplit(Validation):\n    \"\"\"\n    Test by repeated random sampling\n\n    Attributes:\n        n_resamples (int): number of repetitions\n        test_size (float, int, None):\n            If float, should be between 0.0 and 1.0 and represent the proportion\n            of the dataset to include in the test split. If int, represents the\n            absolute number of test samples. If None, the value is set to the\n            complement of the train size. By default, the value is set to 0.1.\n            The default will change in version 0.21. It will remain 0.1 only\n            if ``train_size`` is unspecified, otherwise it will complement\n            the specified ``train_size``.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        train_size : float, int, or None, default is None\n            If float, should be between 0.0 and 1.0 and represent the\n            proportion of the dataset to include in the train split. If\n            int, represents the absolute number of train samples. If None,\n            the value is automatically set to the complement of the test size.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n\n    \"\"\"\n    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,\n                 stratified=True, random_state=0,\n                 store_data=False, store_models=False):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.n_resamples = n_resamples\n        self.train_size = train_size\n        self.test_size = test_size\n        self.stratified = stratified\n        self.random_state = random_state\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            splitter = skl.StratifiedShuffleSplit(\n                n_splits=self.n_resamples, train_size=self.train_size,\n                test_size=self.test_size, random_state=self.random_state\n            )\n            splitter.get_n_splits(data.X, data.Y)\n            return list(splitter.split(data.X, data.Y))\n\n        splitter = skl.ShuffleSplit(\n            n_splits=self.n_resamples, train_size=self.train_size,\n            test_size=self.test_size, random_state=self.random_state\n        )\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass TestOnTestData(Validation):\n    \"\"\"\n    Test on separately provided test data\n\n    Note that the class has a different signature for `__call__`.\n    \"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n\n    def __new__(cls, data=None, test_data=None, learners=None,\n                preprocessor=None, **kwargs):\n        if \"train_data\" in kwargs:\n            if data is None:\n                data = kwargs.pop(\"train_data\")\n            else:\n                raise ValueError(\n                    \"argument 'data' is given twice (once as 'train_data')\")\n        return super().__new__(\n            cls,\n            data=data, learners=learners, preprocessor=preprocessor,\n            test_data=test_data, **kwargs)\n\n    def __call__(self, data, test_data, learners, preprocessor=None,\n                 *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): training data\n            test_data (Orange.data.Table): test_data\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n\n        train_data = preprocessor(data)\n        part_results = []\n        for (learner_i, learner) in enumerate(learners):\n            part_results.append(\n                _mp_worker(0, train_data, test_data, learner_i, learner,\n                           self.store_models))\n            callback((learner_i + 1) / len(learners))\n        callback(1)\n\n        results = Results(\n            data=test_data if self.store_data else None,\n            domain=test_data.domain,\n            nrows=len(test_data), learners=learners,\n            row_indices=np.arange(len(test_data)),\n            folds=(Ellipsis, ),\n            actual=test_data.Y.ravel(),\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (1, len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n\nclass TestOnTrainingData(TestOnTestData):\n    \"\"\"Test on training data\"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n    # signature is such as on the base class, pylint: disable=signature-differs\n    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):\n        return super().__new__(\n            cls,\n            data, test_data=data, learners=learners, preprocessor=preprocessor,\n            **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None,\n                 **kwargs):\n        kwargs.setdefault(\"test_data\", data)\n        # if kwargs contains anything besides test_data, this will be detected\n        # (and complained about) by super().__call__\n        return super().__call__(\n            data=data, learners=learners, preprocessor=preprocessor,\n            callback=callback, **kwargs)\n\n\ndef sample(table, n=0.7, stratified=False, replace=False,\n           random_state=None):\n    \"\"\"\n    Samples data instances from a data table. Returns the sample and\n    a dataset from input data table that are not in the sample. Also\n    uses several sampling functions from\n    `scikit-learn <http://scikit-learn.org>`_.\n\n    table : data table\n        A data table from which to sample.\n\n    n : float, int (default = 0.7)\n        If float, should be between 0.0 and 1.0 and represents\n        the proportion of data instances in the resulting sample. If\n        int, n is the number of data instances in the resulting sample.\n\n    stratified : bool, optional (default = False)\n        If true, sampling will try to consider class values and\n        match distribution of class values\n        in train and test subsets.\n\n    replace : bool, optional (default = False)\n        sample with replacement\n\n    random_state : int or RandomState\n        Pseudo-random number generator state used for random sampling.\n    \"\"\"\n\n    if isinstance(n, float):\n        n = int(n * len(table))\n\n    if replace:\n        if random_state is None:\n            rgen = np.random\n        else:\n            rgen = np.random.mtrand.RandomState(random_state)\n        a_sample = rgen.randint(0, len(table), n)\n        o = np.ones(len(table))\n        o[a_sample] = 0\n        others = np.nonzero(o)[0]\n        return table[a_sample], table[others]\n\n    n = len(table) - n\n    if stratified and table.domain.has_discrete_class:\n        test_size = max(len(table.domain.class_var.values), n)\n        splitter = skl.StratifiedShuffleSplit(\n            n_splits=1, test_size=test_size, train_size=len(table) - test_size,\n            random_state=random_state)\n        splitter.get_n_splits(table.X, table.Y)\n        ind = splitter.split(table.X, table.Y)\n    else:\n        splitter = skl.ShuffleSplit(\n            n_splits=1, test_size=n, random_state=random_state)\n        splitter.get_n_splits(table)\n        ind = splitter.split(table)\n    ind = next(ind)\n    return table[ind[0]], table[ind[1]]\n",
            "header": {
                "file": {
                    "path": "Orange/evaluation/testing.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "6789b4103",
                "new_commit_hash": "9019e8899",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/Orange/evaluation/testing.py"
            },
            "plus_file": {
                "path": "b/Orange/evaluation/testing.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 24,
                            "length": 7
                        },
                        "new_range": {
                            "start": 24,
                            "length": 7
                        },
                        "section": "def _identity(x):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def _mp_worker(fold_i, train_data, test_data, learner_i, learner,",
                                "type": "context"
                            },
                            {
                                "content": "               store_models):",
                                "type": "context"
                            },
                            {
                                "content": "    predicted, probs, model, failed = None, None, None, False",
                                "type": "deleted"
                            },
                            {
                                "content": "    predicted, probs, model, failed, train_time, test_time = None, None, None, False, None, None",
                                "type": "added"
                            },
                            {
                                "content": "    try:",
                                "type": "context"
                            },
                            {
                                "content": "        if not train_data or not test_data:",
                                "type": "context"
                            },
                            {
                                "content": "            raise RuntimeError('Test fold is empty')",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "Orange/evaluation/testing.py",
                            "type": "function",
                            "name": "_mp_worker",
                            "content": "def _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed = None, None, None, False\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0  = time()\n        # testing\n        if train_data.domain.has_discrete_class:\n            predicted, probs = model(test_data, model.ValueProbs)\n        elif train_data.domain.has_continuous_class:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 25,
                            "end_lineno": 47,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# pylint: disable=missing-docstring\n# pylint: disable=protected-access\nimport unittest\n\nimport numpy as np\nfrom AnyQt.QtCore import Qt\nfrom AnyQt.QtTest import QTest\n\nfrom Orange.classification import MajorityLearner, LogisticRegressionLearner\nfrom Orange.classification.majority import ConstantModel\nfrom Orange.data import Table, Domain, DiscreteVariable, ContinuousVariable\nfrom Orange.evaluation import Results, TestOnTestData\nfrom Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\\n    Score\nfrom Orange.modelling import ConstantLearner\nfrom Orange.regression import MeanLearner\nfrom Orange.widgets.evaluate.owtestlearners import (\n    OWTestLearners, results_one_vs_rest)\nfrom Orange.widgets.evaluate.utils import BUILTIN_SCORERS_ORDER\nfrom Orange.widgets.settings import (\n    ClassValuesContextHandler, PerfectDomainContextHandler)\nfrom Orange.widgets.tests.base import WidgetTest\nfrom Orange.widgets.tests.utils import simulate\nfrom Orange.tests import test_filename\n\n\nclass TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n\nclass TestHelpers(unittest.TestCase):\n    def test_results_one_vs_rest(self):\n        data = Table(test_filename(\"datasets/lenses.tab\"))\n        learners = [MajorityLearner()]\n        res = TestOnTestData()(data[1::2], data[::2], learners=learners)\n        r1 = results_one_vs_rest(res, pos_index=0)\n        r2 = results_one_vs_rest(res, pos_index=1)\n        r3 = results_one_vs_rest(res, pos_index=2)\n\n        np.testing.assert_almost_equal(np.sum(r1.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r2.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r3.probabilities, axis=2), 1.0)\n\n        np.testing.assert_almost_equal(\n            r1.probabilities[:, :, 1] +\n            r2.probabilities[:, :, 1] +\n            r3.probabilities[:, :, 1],\n            1.0\n        )\n        self.assertEqual(r1.folds, res.folds)\n        self.assertEqual(r2.folds, res.folds)\n        self.assertEqual(r3.folds, res.folds)\n\n        np.testing.assert_equal(r1.row_indices, res.row_indices)\n        np.testing.assert_equal(r2.row_indices, res.row_indices)\n        np.testing.assert_equal(r3.row_indices, res.row_indices)\n\n\nif __name__ == \"__main__\":\n    unittest.main()\n",
            "new_file_content": "# pylint: disable=missing-docstring\n# pylint: disable=protected-access\nimport unittest\n\nimport numpy as np\nfrom AnyQt.QtCore import Qt\nfrom AnyQt.QtTest import QTest\n\nfrom Orange.classification import MajorityLearner, LogisticRegressionLearner\nfrom Orange.classification.majority import ConstantModel\nfrom Orange.data import Table, Domain, DiscreteVariable, ContinuousVariable\nfrom Orange.evaluation import Results, TestOnTestData\nfrom Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\\n    Score\nfrom Orange.modelling import ConstantLearner\nfrom Orange.regression import MeanLearner\nfrom Orange.widgets.evaluate.owtestlearners import (\n    OWTestLearners, results_one_vs_rest)\nfrom Orange.widgets.evaluate.utils import BUILTIN_SCORERS_ORDER\nfrom Orange.widgets.settings import (\n    ClassValuesContextHandler, PerfectDomainContextHandler)\nfrom Orange.widgets.tests.base import WidgetTest\nfrom Orange.widgets.tests.utils import simulate\nfrom Orange.tests import test_filename\n\n\nclass TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n\nclass TestHelpers(unittest.TestCase):\n    def test_results_one_vs_rest(self):\n        data = Table(test_filename(\"datasets/lenses.tab\"))\n        learners = [MajorityLearner()]\n        res = TestOnTestData()(data[1::2], data[::2], learners=learners)\n        r1 = results_one_vs_rest(res, pos_index=0)\n        r2 = results_one_vs_rest(res, pos_index=1)\n        r3 = results_one_vs_rest(res, pos_index=2)\n\n        np.testing.assert_almost_equal(np.sum(r1.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r2.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r3.probabilities, axis=2), 1.0)\n\n        np.testing.assert_almost_equal(\n            r1.probabilities[:, :, 1] +\n            r2.probabilities[:, :, 1] +\n            r3.probabilities[:, :, 1],\n            1.0\n        )\n        self.assertEqual(r1.folds, res.folds)\n        self.assertEqual(r2.folds, res.folds)\n        self.assertEqual(r3.folds, res.folds)\n\n        np.testing.assert_equal(r1.row_indices, res.row_indices)\n        np.testing.assert_equal(r2.row_indices, res.row_indices)\n        np.testing.assert_equal(r3.row_indices, res.row_indices)\n\n\nif __name__ == \"__main__\":\n    unittest.main()\n",
            "header": {
                "file": {
                    "path": "Orange/widgets/evaluate/tests/test_owtestlearners.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "213a2e7e0",
                "new_commit_hash": "0b58fa7a9",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/Orange/widgets/evaluate/tests/test_owtestlearners.py"
            },
            "plus_file": {
                "path": "b/Orange/widgets/evaluate/tests/test_owtestlearners.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 225,
                            "length": 16
                        },
                        "new_range": {
                            "start": 225,
                            "length": 16
                        },
                        "section": "class TestOWTestLearners(WidgetTest):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        self.send_signal(self.widget.Inputs.learner,",
                                "type": "context"
                            },
                            {
                                "content": "                         LogisticRegressionLearner(), 0, wait=5000)",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        average_auc = float(model.item(0, 1).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        average_auc = float(model.item(0, 3).text())",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")",
                                "type": "context"
                            },
                            {
                                "content": "        setosa_auc = float(model.item(0, 1).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        setosa_auc = float(model.item(0, 3).text())",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")",
                                "type": "context"
                            },
                            {
                                "content": "        versicolor_auc = float(model.item(0, 1).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        versicolor_auc = float(model.item(0, 3).text())",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")",
                                "type": "context"
                            },
                            {
                                "content": "        virginica_auc = float(model.item(0, 1).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        virginica_auc = float(model.item(0, 3).text())",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        self.assertGreater(average_auc, versicolor_auc)",
                                "type": "context"
                            },
                            {
                                "content": "        self.assertGreater(average_auc, virginica_auc)",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestlearners.py",
                            "type": "class",
                            "name": "TestOWTestLearners",
                            "content": "class TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 27,
                            "end_lineno": 382,
                            "parent": null
                        },
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestlearners.py",
                            "type": "method",
                            "name": "TestOWTestLearners.test_target_changing",
                            "content": "class TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 218,
                            "end_lineno": 243,
                            "parent": {
                                "file_name": "Orange/widgets/evaluate/tests/test_owtestlearners.py",
                                "type": "class",
                                "name": "TestOWTestLearners",
                                "content": "class TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 27,
                                "end_lineno": 382,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 287,
                            "length": 12
                        },
                        "new_range": {
                            "start": 287,
                            "length": 12
                        },
                        "section": "class TestOWTestLearners(WidgetTest):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    def _retrieve_scores(self):",
                                "type": "context"
                            },
                            {
                                "content": "        w = self.widget",
                                "type": "context"
                            },
                            {
                                "content": "        model = w.score_table.model",
                                "type": "context"
                            },
                            {
                                "content": "        auc = model.item(0, 1).text()",
                                "type": "deleted"
                            },
                            {
                                "content": "        auc = model.item(0, 3).text()",
                                "type": "added"
                            },
                            {
                                "content": "        auc = float(auc) if auc != \"\" else None",
                                "type": "context"
                            },
                            {
                                "content": "        ca = float(model.item(0, 2).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        f1 = float(model.item(0, 3).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        precision = float(model.item(0, 4).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        recall = float(model.item(0, 5).text())",
                                "type": "deleted"
                            },
                            {
                                "content": "        ca = float(model.item(0, 4).text())",
                                "type": "added"
                            },
                            {
                                "content": "        f1 = float(model.item(0, 5).text())",
                                "type": "added"
                            },
                            {
                                "content": "        precision = float(model.item(0, 6).text())",
                                "type": "added"
                            },
                            {
                                "content": "        recall = float(model.item(0, 7).text())",
                                "type": "added"
                            },
                            {
                                "content": "        return auc, ca, f1, precision, recall",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestlearners.py",
                            "type": "method",
                            "name": "TestOWTestLearners._retrieve_scores",
                            "content": "class TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 287,
                            "end_lineno": 296,
                            "parent": {
                                "file_name": "Orange/widgets/evaluate/tests/test_owtestlearners.py",
                                "type": "class",
                                "name": "TestOWTestLearners",
                                "content": "class TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 27,
                                "end_lineno": 382,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "Orange/widgets/evaluate/tests/test_owtestlearners.py",
                            "type": "class",
                            "name": "TestOWTestLearners",
                            "content": "class TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 1).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 1).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 1).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 2).text())\n        f1 = float(model.item(0, 3).text())\n        precision = float(model.item(0, 4).text())\n        recall = float(model.item(0, 5).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 27,
                            "end_lineno": 382,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "977b6d85fab32793cf881979cd7af581eb146b3d^",
    "new_commit_hash": "977b6d85fab32793cf881979cd7af581eb146b3d",
    "commit_message": "Fix tests",
    "commit_date": "2019-08-05T09:35:41+02:00",
    "metadata": {}
}
{
    "repo_name": "orange3",
    "new_commit_hash": "977b6d85fab32793cf881979cd7af581eb146b3d",
    "test_file_codes": [
        "# pylint: disable=missing-docstring\n# pylint: disable=protected-access\nimport unittest\n\nimport numpy as np\nfrom AnyQt.QtCore import Qt\nfrom AnyQt.QtTest import QTest\n\nfrom Orange.classification import MajorityLearner, LogisticRegressionLearner\nfrom Orange.classification.majority import ConstantModel\nfrom Orange.data import Table, Domain, DiscreteVariable, ContinuousVariable\nfrom Orange.evaluation import Results, TestOnTestData\nfrom Orange.evaluation.scoring import ClassificationScore, RegressionScore, \\\n    Score\nfrom Orange.modelling import ConstantLearner\nfrom Orange.regression import MeanLearner\nfrom Orange.widgets.evaluate.owtestlearners import (\n    OWTestLearners, results_one_vs_rest)\nfrom Orange.widgets.evaluate.utils import BUILTIN_SCORERS_ORDER\nfrom Orange.widgets.settings import (\n    ClassValuesContextHandler, PerfectDomainContextHandler)\nfrom Orange.widgets.tests.base import WidgetTest\nfrom Orange.widgets.tests.utils import simulate\nfrom Orange.tests import test_filename\n\n\nclass TestOWTestLearners(WidgetTest):\n    def setUp(self):\n        super().setUp()\n        self.widget = self.create_widget(OWTestLearners)  # type: OWTestLearners\n\n        self.scores_domain = Domain(\n            [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n            [DiscreteVariable(\"c\", values=[\"y\", \"n\"])])\n\n        self.scores_table_values = [[1, 1, 1.23, 23.8], [1., 2., 3., 4.]]\n\n    def tearDown(self):\n        self.widget.onDeleteWidget()\n        super().tearDown()\n\n    def test_basic(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n        self.assertIsNotNone(res.probabilities)\n\n        self.send_signal(self.widget.Inputs.learner, None, 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsNone(res)\n\n        data = Table(\"housing\")[::10]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MeanLearner(), 0)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertIsInstance(res, Results)\n        self.assertIsNotNone(res.domain)\n        self.assertIsNotNone(res.data)\n\n    def test_more_learners(self):\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 1)\n        res = self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        np.testing.assert_equal(res.probabilities[0], res.probabilities[1])\n\n    def test_testOnTest(self):\n        data = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.widget.resampling = OWTestLearners.TestOnTest\n        self.send_signal(self.widget.Inputs.test_data, data)\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\n\n    def test_CrossValidationByFeature(self):\n        data = Table(\"iris\")\n        attrs = data.domain.attributes\n        domain = Domain(attrs[:-1], attrs[-1], data.domain.class_vars)\n        data_with_disc_metas = Table.from_table(domain, data)\n        rb = self.widget.controls.resampling.buttons[OWTestLearners.FeatureFold]\n\n        self.send_signal(self.widget.Inputs.learner, ConstantLearner(), 0)\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(rb.isEnabled())\n        self.assertFalse(self.widget.features_combo.isEnabled())\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, data_with_disc_metas)\n        self.assertTrue(rb.isEnabled())\n        rb.click()\n        self.assertEqual(self.widget.resampling, OWTestLearners.FeatureFold)\n        self.assertTrue(self.widget.features_combo.isEnabled())\n        self.assertEqual(self.widget.features_combo.currentText(), \"iris\")\n        self.assertEqual(len(self.widget.features_combo.model()), 1)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n\n        self.send_signal(self.widget.Inputs.train_data, None)\n        self.assertFalse(rb.isEnabled())\n        self.assertEqual(self.widget.resampling, OWTestLearners.KFold)\n        self.assertFalse(self.widget.features_combo.isEnabled())\n\n    def test_migrate_removes_invalid_contexts(self):\n        context_invalid = ClassValuesContextHandler().new_context([0, 1, 2])\n        context_valid = PerfectDomainContextHandler().new_context(*[[]] * 4)\n        settings = {'context_settings': [context_invalid, context_valid]}\n        self.widget.migrate_settings(settings, 2)\n        self.assertEqual(settings['context_settings'], [context_valid])\n\n    def test_memory_error(self):\n        \"\"\"\n        Handling memory error.\n        GH-2316\n        \"\"\"\n        data = Table(\"iris\")[::15]\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.assertFalse(self.widget.Error.memory_error.is_shown())\n\n        with unittest.mock.patch(\n                \"Orange.evaluation.testing.Results.get_augmented_data\",\n                side_effect=MemoryError):\n            self.send_signal(self.widget.Inputs.learner, MajorityLearner(), 0, wait=5000)\n            self.assertTrue(self.widget.Error.memory_error.is_shown())\n\n    def test_one_class_value(self):\n        \"\"\"\n        Data with a class with one value causes widget to crash when that value\n        is selected.\n        GH-2351\n        \"\"\"\n        table = Table(\n            Domain(\n                [ContinuousVariable(\"a\"), ContinuousVariable(\"b\")],\n                [DiscreteVariable(\"c\", values=[\"y\"])]),\n            list(zip(\n                [42.48, 16.84, 15.23, 23.8],\n                [1., 2., 3., 4.],\n                \"yyyy\"))\n        )\n        self.widget.n_folds = 0\n        self.assertFalse(self.widget.Error.only_one_class_var_value.is_shown())\n        self.send_signal(\"Data\", table)\n        self.send_signal(\"Learner\", MajorityLearner(), 0, wait=1000)\n        self.assertTrue(self.widget.Error.only_one_class_var_value.is_shown())\n\n    def test_nan_class(self):\n        \"\"\"\n        Do not crash on a data with only nan class values.\n        GH-2751\n        \"\"\"\n        def assertErrorShown(data, is_shown):\n            self.send_signal(\"Data\", data)\n            self.assertEqual(is_shown, self.widget.Error.no_class_values.is_shown())\n\n        data = Table(\"iris\")[::30]\n        data.Y[:] = np.nan\n\n        for data, is_shown in zip([None, data, Table(\"iris\")[:30]], [False, True, False]):\n            assertErrorShown(data, is_shown)\n\n    def test_addon_scorers(self):\n        try:\n            # These classes are registered, pylint: disable=unused-variable\n            class NewScore(Score):\n                class_types = (DiscreteVariable, ContinuousVariable)\n\n            class NewClassificationScore(ClassificationScore):\n                pass\n\n            class NewRegressionScore(RegressionScore):\n                pass\n\n            builtins = BUILTIN_SCORERS_ORDER\n            self.send_signal(\"Data\", Table(\"iris\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[DiscreteVariable])]),\n                builtins[DiscreteVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertIn(\"NewClassificationScore\", scorer_names)\n            self.assertNotIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", Table(\"housing\"))\n            scorer_names = [scorer.name for scorer in self.widget.scorers]\n            self.assertEqual(\n                tuple(scorer_names[:len(builtins[ContinuousVariable])]),\n                builtins[ContinuousVariable])\n            self.assertIn(\"NewScore\", scorer_names)\n            self.assertNotIn(\"NewClassificationScore\", scorer_names)\n            self.assertIn(\"NewRegressionScore\", scorer_names)\n\n            self.send_signal(\"Data\", None)\n            self.assertEqual(self.widget.scorers, [])\n        finally:\n            del Score.registry[\"NewScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewClassificationScore\"]  # pylint: disable=no-member\n            del Score.registry[\"NewRegressionScore\"]  # pylint: disable=no-member\n\n    def test_target_changing(self):\n        data = Table(\"iris\")\n        w = self.widget  #: OWTestLearners\n        model = w.score_table.model\n\n        w.n_folds = 2\n        self.send_signal(self.widget.Inputs.train_data, data)\n        self.send_signal(self.widget.Inputs.learner,\n                         LogisticRegressionLearner(), 0, wait=5000)\n\n        average_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-setosa\")\n        setosa_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-versicolor\")\n        versicolor_auc = float(model.item(0, 3).text())\n\n        simulate.combobox_activate_item(w.controls.class_selection, \"Iris-virginica\")\n        virginica_auc = float(model.item(0, 3).text())\n\n        self.assertGreater(average_auc, versicolor_auc)\n        self.assertGreater(average_auc, virginica_auc)\n        self.assertLess(average_auc, setosa_auc)\n        self.assertGreater(setosa_auc, versicolor_auc)\n        self.assertGreater(setosa_auc, virginica_auc)\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n\n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n\n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n\n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n\n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n\n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n\n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n        self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\n\n        self.send_signal(self.widget.Inputs.test_data, versicolor, wait=5000)\n        self.assertEqual(view.model().item(0, 0).text(), \"SetosaLearner\")\n\n        self.widget.hide()\n\n    def _retrieve_scores(self):\n        w = self.widget\n        model = w.score_table.model\n        auc = model.item(0, 3).text()\n        auc = float(auc) if auc != \"\" else None\n        ca = float(model.item(0, 4).text())\n        f1 = float(model.item(0, 5).text())\n        precision = float(model.item(0, 6).text())\n        recall = float(model.item(0, 7).text())\n        return auc, ca, f1, precision, recall\n\n    def _test_scores(self, train_data, test_data, learner, sampling, n_folds):\n        w = self.widget  #: OWTestLearners\n        w.controls.resampling.buttons[sampling].click()\n        if n_folds is not None:\n            w.n_folds = n_folds\n\n        self.send_signal(self.widget.Inputs.train_data, train_data)\n        if test_data is not None:\n            self.send_signal(self.widget.Inputs.test_data, test_data)\n        self.send_signal(self.widget.Inputs.learner, learner, 0, wait=5000)\n        return self._retrieve_scores()\n\n    def test_scores_constant_all_same(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyy\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, ConstantLearner(), OWTestLearners.TestOnTest, None),\n                              (None, 1, 1, 1, 1))\n\n    def test_scores_log_reg_overfitted(self):\n        table = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table, table, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (1, 1, 1, 1, 1))\n\n    def test_scores_log_reg_bad(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"nnny\")]))\n        )\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*self.scores_table_values + [list(\"yyyn\")]))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_bad2(self):\n        table_train = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"nnyy\")]))))\n        table_test = Table(\n            self.scores_domain,\n            list(zip(*(self.scores_table_values + [list(\"yynn\")]))))\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0, 0, 0, 0, 0))\n\n    def test_scores_log_reg_advanced(self):\n        table_train = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yyynn\"))\n        )\n        table_test = Table(\n            self.scores_domain, list(zip(\n                [1, 1, 1.23, 23.8, 5.], [1., 2., 3., 4., 3.], \"yynnn\"))\n        )\n\n        self.assertTupleEqual(self._test_scores(\n            table_train, table_test, LogisticRegressionLearner(),\n            OWTestLearners.TestOnTest, None),\n                              (0.667, 0.8, 0.8, 0.867, 0.8))\n\n    def test_scores_cross_validation(self):\n        \"\"\"\n        Test more than two classes and cross-validation\n        \"\"\"\n        self.assertTrue(\n            all(x >= y for x, y in zip(\n                self._test_scores(\n                    Table(\"iris\")[::15], None, LogisticRegressionLearner(),\n                    OWTestLearners.KFold, 0),\n                (0.8, 0.5, 0.5, 0.5, 0.5))))\n\n\nclass TestHelpers(unittest.TestCase):\n    def test_results_one_vs_rest(self):\n        data = Table(test_filename(\"datasets/lenses.tab\"))\n        learners = [MajorityLearner()]\n        res = TestOnTestData()(data[1::2], data[::2], learners=learners)\n        r1 = results_one_vs_rest(res, pos_index=0)\n        r2 = results_one_vs_rest(res, pos_index=1)\n        r3 = results_one_vs_rest(res, pos_index=2)\n\n        np.testing.assert_almost_equal(np.sum(r1.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r2.probabilities, axis=2), 1.0)\n        np.testing.assert_almost_equal(np.sum(r3.probabilities, axis=2), 1.0)\n\n        np.testing.assert_almost_equal(\n            r1.probabilities[:, :, 1] +\n            r2.probabilities[:, :, 1] +\n            r3.probabilities[:, :, 1],\n            1.0\n        )\n        self.assertEqual(r1.folds, res.folds)\n        self.assertEqual(r2.folds, res.folds)\n        self.assertEqual(r3.folds, res.folds)\n\n        np.testing.assert_equal(r1.row_indices, res.row_indices)\n        np.testing.assert_equal(r2.row_indices, res.row_indices)\n        np.testing.assert_equal(r3.row_indices, res.row_indices)\n\n\nif __name__ == \"__main__\":\n    unittest.main()\n"
    ],
    "test_file_names": [
        "test_1.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Orange installation attempts...\nAttempting installation with Python 3.7...\nWarning: 'keywords' should be a list, got type 'tuple'\nWarning: 'classifiers' should be a list, got type 'tuple'\nWarning: 'keywords' should be a list, got type 'tuple'\nWarning: 'classifiers' should be a list, got type 'tuple'\nrunning build_ext\nrunning build_src\nbuild_src\nbuilding extension \"Orange.classification._simple_tree\" sources\nbuilding extension \"Orange.classification._tree_scorers\" sources\nbuilding extension \"Orange.data._valuecount\" sources\nbuilding extension \"Orange.data._contingency\" sources\nbuilding extension \"Orange.data._io\" sources\nbuilding extension \"Orange.data._variable\" sources\nbuilding extension \"Orange.distance._distance\" sources\nbuilding extension \"Orange.preprocess._discretize\" sources\nbuilding extension \"Orange.preprocess._relieff\" sources\nbuilding extension \"Orange.widgets.utils._grid_density\" sources\nbuild_src: building npy-pkg config files\ncustomize UnixCCompiler\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating /tmp/tmp7ju5i2g4/tmp\ncreating /tmp/tmp7ju5i2g4/tmp/tmp7ju5i2g4\ncompile options: '-MMD -MF /tmp/tmp7ju5i2g4/file.c.d -c'\ncc: /tmp/tmp7ju5i2g4/file.c\ncustomize UnixCCompiler using build_ext\ncustomize UnixCCompiler\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating /tmp/tmpfg5ba223/tmp\ncreating /tmp/tmpfg5ba223/tmp/tmpfg5ba223\ncompile options: '-MMD -MF /tmp/tmpfg5ba223/file.c.d -c'\ncc: /tmp/tmpfg5ba223/file.c\ncustomize UnixCCompiler using build_ext\nbuilding 'Orange.classification._simple_tree' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating build/temp.linux-x86_64-3.7/Orange\ncreating build/temp.linux-x86_64-3.7/Orange/classification\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/classification/_simple_tree.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/classification/_simple_tree.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/classification/_simple_tree.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.classification._tree_scorers' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/classification/_tree_scorers.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/classification/_tree_scorers.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/classification/_tree_scorers.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._valuecount' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating build/temp.linux-x86_64-3.7/Orange/data\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/data/_valuecount.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_valuecount.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/data/_valuecount.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._contingency' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/data/_contingency.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_contingency.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/data/_contingency.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._io' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/data/_io.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_io.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/data/_io.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.data._variable' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/data/_variable.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/data/_variable.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/data/_variable.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.distance._distance' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating build/temp.linux-x86_64-3.7/Orange/distance\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/distance/_distance.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/distance/_distance.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/distance/_distance.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.preprocess._discretize' extension\ncompiling C sources\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating build/temp.linux-x86_64-3.7/Orange/preprocess\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\ncc: Orange/preprocess/_discretize.c\ncc -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/preprocess/_discretize.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/preprocess/_discretize.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.preprocess._relieff' extension\ncompiling C++ sources\nC compiler: c++ -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\nc++: Orange/preprocess/_relieff.cpp\nc++ -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/preprocess/_relieff.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/preprocess/_relieff.cpython-37m-x86_64-linux-gnu.so\nbuilding 'Orange.widgets.utils._grid_density' extension\ncompiling C++ sources\nC compiler: c++ -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating build/temp.linux-x86_64-3.7/Orange/widgets\ncreating build/temp.linux-x86_64-3.7/Orange/widgets/utils\ncompile options: '-I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include -I/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/include -I/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/include/python3.7m -c'\nc++: Orange/widgets/utils/_grid_density.cpp\nc++ -pthread -shared -L/tools/deps/lib build/temp.linux-x86_64-3.7/Orange/widgets/utils/_grid_density.o -L/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib -lm -lpython3.7m -o Orange/widgets/utils/_grid_density.cpython-37m-x86_64-linux-gnu.so\nWarning: 'keywords' should be a list, got type 'tuple'\nWarning: 'classifiers' should be a list, got type 'tuple'\nWarning: 'keywords' should be a list, got type 'tuple'\nWarning: 'classifiers' should be a list, got type 'tuple'\nrunning develop\nrunning build_scripts\nrunning egg_info\nrunning build_src\nbuild_src\nbuilding extension \"Orange.classification._simple_tree\" sources\nbuilding extension \"Orange.classification._tree_scorers\" sources\nbuilding extension \"Orange.data._valuecount\" sources\nbuilding extension \"Orange.data._contingency\" sources\nbuilding extension \"Orange.data._io\" sources\nbuilding extension \"Orange.data._variable\" sources\nbuilding extension \"Orange.distance._distance\" sources\nbuilding extension \"Orange.preprocess._discretize\" sources\nbuilding extension \"Orange.preprocess._relieff\" sources\nbuilding extension \"Orange.widgets.utils._grid_density\" sources\nbuild_src: building npy-pkg config files\ncreating Orange3.egg-info\nwriting Orange3.egg-info/PKG-INFO\nwriting dependency_links to Orange3.egg-info/dependency_links.txt\nwriting entry points to Orange3.egg-info/entry_points.txt\nwriting requirements to Orange3.egg-info/requires.txt\nwriting top-level names to Orange3.egg-info/top_level.txt\nwriting manifest file 'Orange3.egg-info/SOURCES.txt'\nreading manifest file 'Orange3.egg-info/SOURCES.txt'\nreading manifest template 'MANIFEST.in'\nwarning: no files found matching 'README*' under directory 'Orange'\nwarning: no files found matching 'LICENSE' under directory 'Orange'\nwarning: no files found matching '*.basket' under directory 'Orange/datasets'\nwarning: no files found matching '*.js' under directory 'Orange/widgets'\nwarning: no files found matching '*.css' under directory 'Orange/widgets'\nwarning: no files found matching '*.html' under directory 'Orange/widgets'\nwarning: no files found matching 'COPYING'\nadding license file 'LICENSE'\nwriting manifest file 'Orange3.egg-info/SOURCES.txt'\nrunning build_ext\ncustomize UnixCCompiler\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating /tmp/tmpcq4gjof2/tmp\ncreating /tmp/tmpcq4gjof2/tmp/tmpcq4gjof2\ncompile options: '-MMD -MF /tmp/tmpcq4gjof2/file.c.d -c'\ncc: /tmp/tmpcq4gjof2/file.c\ncustomize UnixCCompiler using build_ext\ncustomize UnixCCompiler\nC compiler: cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC\n\ncreating /tmp/tmp11ixejvj/tmp\ncreating /tmp/tmp11ixejvj/tmp/tmp11ixejvj\ncompile options: '-MMD -MF /tmp/tmp11ixejvj/file.c.d -c'\ncc: /tmp/tmp11ixejvj/file.c\ncustomize UnixCCompiler using build_ext\nCreating /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/Orange3.egg-link (link to .)\nAdding Orange3 3.23.0.dev0+977b6d8 to easy-install.pth file\nInstalling orange-canvas script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nInstalled /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d\nProcessing dependencies for Orange3==3.23.0.dev0+977b6d8\nSearching for XlsxWriter==3.2.0\nBest match: XlsxWriter 3.2.0\nAdding XlsxWriter 3.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for xlrd==2.0.1\nBest match: xlrd 2.0.1\nAdding xlrd 2.0.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for setuptools==59.8.0\nBest match: setuptools 59.8.0\nAdding setuptools 59.8.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for serverfiles==0.3.1\nBest match: serverfiles 0.3.1\nAdding serverfiles 0.3.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for scipy==1.7.3\nBest match: scipy 1.7.3\nAdding scipy 1.7.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for scikit-learn==1.0.2\nBest match: scikit-learn 1.0.2\nAdding scikit-learn 1.0.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for requests==2.31.0\nBest match: requests 2.31.0\nAdding requests 2.31.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for python-louvain==0.16\nBest match: python-louvain 0.16\nAdding python-louvain 0.16 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for pyqtgraph==0.12.4\nBest match: pyqtgraph 0.12.4\nAdding pyqtgraph 0.12.4 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for pip==24.0\nBest match: pip 24.0\nAdding pip 24.0 to easy-install.pth file\nInstalling pip script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling pip3 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling pip3.10 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for pandas==1.3.5\nBest match: pandas 1.3.5\nAdding pandas 1.3.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for orange-widget-base==4.0.1\nBest match: orange-widget-base 4.0.1\nAdding orange-widget-base 4.0.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for orange-canvas-core==0.1.35\nBest match: orange-canvas-core 0.1.35\nAdding orange-canvas-core 0.1.35 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for openTSNE==1.0.1\nBest match: openTSNE 1.0.1\nAdding openTSNE 1.0.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for numpy==1.17.5\nBest match: numpy 1.17.5\nAdding numpy 1.17.5 to easy-install.pth file\nInstalling f2py script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling f2py3 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling f2py3.7 script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for networkx==2.6.3\nBest match: networkx 2.6.3\nAdding networkx 2.6.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for matplotlib==3.5.3\nBest match: matplotlib 3.5.3\nAdding matplotlib 3.5.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for keyrings.alt==4.2.0\nBest match: keyrings.alt 4.2.0\nAdding keyrings.alt 4.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for keyring==24.1.1\nBest match: keyring 24.1.1\nAdding keyring 24.1.1 to easy-install.pth file\nInstalling keyring script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for joblib==1.3.2\nBest match: joblib 1.3.2\nAdding joblib 1.3.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for docutils==0.20.1\nBest match: docutils 0.20.1\nAdding docutils 0.20.1 to easy-install.pth file\nInstalling docutils script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for chardet==5.2.0\nBest match: chardet 5.2.0\nAdding chardet 5.2.0 to easy-install.pth file\nInstalling chardetect script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for Bottleneck==1.4.0\nBest match: Bottleneck 1.4.0\nAdding Bottleneck 1.4.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for AnyQt==0.2.0\nBest match: AnyQt 0.2.0\nAdding AnyQt 0.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for threadpoolctl==3.1.0\nBest match: threadpoolctl 3.1.0\nAdding threadpoolctl 3.1.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for idna==3.10\nBest match: idna 3.10\nAdding idna 3.10 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for certifi==2024.12.14\nBest match: certifi 2024.12.14\nAdding certifi 2024.12.14 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for urllib3==2.0.7\nBest match: urllib3 2.0.7\nAdding urllib3 2.0.7 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for charset-normalizer==3.4.1\nBest match: charset-normalizer 3.4.1\nAdding charset-normalizer 3.4.1 to easy-install.pth file\nInstalling normalizer script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for python-dateutil==2.9.0.post0\nBest match: python-dateutil 2.9.0.post0\nAdding python-dateutil 2.9.0.post0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for pytz==2024.2\nBest match: pytz 2024.2\nAdding pytz 2024.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for dictdiffer==0.9.0\nBest match: dictdiffer 0.9.0\nAdding dictdiffer 0.9.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for requests-cache==1.1.1\nBest match: requests-cache 1.1.1\nAdding requests-cache 1.1.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for packaging==24.0\nBest match: packaging 24.0\nAdding packaging 24.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for qasync==0.25.0\nBest match: qasync 0.25.0\nAdding qasync 0.25.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for commonmark==0.9.1\nBest match: commonmark 0.9.1\nAdding commonmark 0.9.1 to easy-install.pth file\nInstalling cmark script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for importlib-metadata==6.7.0\nBest match: importlib-metadata 6.7.0\nAdding importlib-metadata 6.7.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for cycler==0.11.0\nBest match: cycler 0.11.0\nAdding cycler 0.11.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for fonttools==4.38.0\nBest match: fonttools 4.38.0\nAdding fonttools 4.38.0 to easy-install.pth file\nInstalling fonttools script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling pyftmerge script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling pyftsubset script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\nInstalling ttx script to /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/bin\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for pyparsing==3.1.4\nBest match: pyparsing 3.1.4\nAdding pyparsing 3.1.4 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for Pillow==9.5.0\nBest match: Pillow 9.5.0\nAdding Pillow 9.5.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for kiwisolver==1.4.5\nBest match: kiwisolver 1.4.5\nAdding kiwisolver 1.4.5 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for jaraco.classes==3.2.3\nBest match: jaraco.classes 3.2.3\nAdding jaraco.classes 3.2.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for importlib-resources==5.12.0\nBest match: importlib-resources 5.12.0\nAdding importlib-resources 5.12.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for SecretStorage==3.3.3\nBest match: SecretStorage 3.3.3\nAdding SecretStorage 3.3.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for jeepney==0.8.0\nBest match: jeepney 0.8.0\nAdding jeepney 0.8.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for six==1.17.0\nBest match: six 1.17.0\nAdding six 1.17.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for platformdirs==4.0.0\nBest match: platformdirs 4.0.0\nAdding platformdirs 4.0.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for url-normalize==1.4.3\nBest match: url-normalize 1.4.3\nAdding url-normalize 1.4.3 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for attrs==24.2.0\nBest match: attrs 24.2.0\nAdding attrs 24.2.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for cattrs==23.1.2\nBest match: cattrs 23.1.2\nAdding cattrs 23.1.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for typing-extensions==4.7.1\nBest match: typing-extensions 4.7.1\nAdding typing-extensions 4.7.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for zipp==3.15.0\nBest match: zipp 3.15.0\nAdding zipp 3.15.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for more-itertools==9.1.0\nBest match: more-itertools 9.1.0\nAdding more-itertools 9.1.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for cryptography==44.0.0\nBest match: cryptography 44.0.0\nAdding cryptography 44.0.0 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for exceptiongroup==1.2.2\nBest match: exceptiongroup 1.2.2\nAdding exceptiongroup 1.2.2 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for cffi==1.15.1\nBest match: cffi 1.15.1\nAdding cffi 1.15.1 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nSearching for pycparser==2.21\nBest match: pycparser 2.21\nAdding pycparser 2.21 to easy-install.pth file\n\nUsing /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages\nFinished processing dependencies for Orange3==3.23.0.dev0+977b6d8\nVerifying Orange installation...\n✅ Orange installation successful!\nSuccessfully installed orange using Python 3.7\n",
    "setup_res_stderr": "Using CPython 3.7.9\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 18 packages in 149ms\nPrepared 18 packages in 54ms\nInstalled 18 packages in 781ms\n + cython==0.29.37\n + exceptiongroup==1.2.2\n + importlib-metadata==6.7.0\n + iniconfig==2.0.0\n + numpy==1.17.5\n + packaging==24.0\n + pluggy==1.2.0\n + pyqt5==5.15.10\n + pyqt5-qt5==5.15.16\n + pyqt5-sip==12.13.0\n + pyqtwebengine==5.15.6\n + pyqtwebengine-qt5==5.15.16\n + pytest==7.4.4\n + setuptools==59.8.0\n + tomli==2.0.1\n + typing-extensions==4.7.1\n + wheel==0.42.0\n + zipp==3.15.0\nResolved 37 packages in 235ms\nInstalled 32 packages in 378ms\n + bottleneck==1.4.0\n + certifi==2024.12.14\n + cffi==1.15.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + cryptography==44.0.0\n + idna==3.10\n + importlib-resources==5.12.0\n + jaraco-classes==3.2.3\n + jeepney==0.8.0\n + joblib==1.3.2\n + keyring==24.1.1\n + keyrings-alt==4.2.0\n + more-itertools==9.1.0\n + networkx==2.6.3\n + opentsne==1.0.1\n + pandas==1.3.5\n + pip==24.0\n + pycparser==2.21\n + python-dateutil==2.9.0.post0\n + python-louvain==0.16\n + pytz==2024.2\n + requests==2.31.0\n + scikit-learn==1.0.2\n + scipy==1.7.3\n + secretstorage==3.3.3\n + serverfiles==0.3.1\n + six==1.17.0\n + threadpoolctl==3.1.0\n + urllib3==2.0.7\n + xlrd==2.0.1\n + xlsxwriter==3.2.0\nResolved 34 packages in 171ms\nInstalled 19 packages in 79ms\n + anyqt==0.2.0\n + attrs==24.2.0\n + cattrs==23.1.2\n + commonmark==0.9.1\n + cycler==0.11.0\n + dictdiffer==0.9.0\n + docutils==0.20.1\n + fonttools==4.38.0\n + kiwisolver==1.4.5\n + matplotlib==3.5.3\n + orange-canvas-core==0.1.35\n + orange-widget-base==4.0.1\n + pillow==9.5.0\n + platformdirs==4.0.0\n + pyparsing==3.1.4\n + pyqtgraph==0.12.4\n + qasync==0.25.0\n + requests-cache==1.1.1\n + url-normalize==1.4.3\nResolved 2 packages in 3ms\nInstalled 2 packages in 6ms\n + psycopg2==2.9.9\n + pymssql==2.3.2\nAudited 1 package in 1ms\nAudited 2 packages in 25ms\nOrange/classification/_simple_tree.c: In function ‘gain_ratio_c’:\nOrange/classification/_simple_tree.c:63:9: warning: implicit declaration of function ‘qsort_r’; did you mean ‘qsort’? [-Wimplicit-function-declaration]\n   63 |         qsort_r(base, nel, size, compar, thunk)\n      |         ^~~~~~~\nOrange/classification/_simple_tree.c:147:9: note: in expansion of macro ‘QSORT_R_FUNC’\n  147 |         QSORT_R_FUNC(examples, size, sizeof(struct Example), (void*) &compar_attr, compar_examples);\n      |         ^~~~~~~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/classification/_tree_scorers.c:567:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_2find_threshold_entropy’:\nOrange/classification/_tree_scorers.c:3115:27: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 3115 |   __pyx_t_7 = ((__pyx_v_N <= __pyx_v_min_leaf) != 0);\n      |                           ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_6find_threshold_MSE’:\nOrange/classification/_tree_scorers.c:4619:27: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 4619 |   __pyx_t_1 = ((__pyx_v_N <= __pyx_v_min_leaf) != 0);\n      |                           ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_8find_binarization_MSE’:\nOrange/classification/_tree_scorers.c:5396:43: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5396 |           for (__pyx_t_22 = 0; __pyx_t_22 < __pyx_t_21; __pyx_t_22+=1) {\n      |                                           ^\nOrange/classification/_tree_scorers.c:5517:39: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5517 |           __pyx_t_27 = ((__pyx_v_left >= __pyx_v_min_leaf) != 0);\n      |                                       ^~\nOrange/classification/_tree_scorers.c:5523:53: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 5523 |           __pyx_t_27 = (((__pyx_v_N - __pyx_v_left) >= __pyx_v_min_leaf) != 0);\n      |                                                     ^~\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_14compute_predictions_csr’:\nOrange/classification/_tree_scorers.c:7051:41: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 7051 |         for (__pyx_t_11 = 0; __pyx_t_11 < __pyx_t_10; __pyx_t_11+=1) {\n      |                                         ^\nOrange/classification/_tree_scorers.c: In function ‘__pyx_pf_6Orange_14classification_13_tree_scorers_16compute_predictions_csc’:\nOrange/classification/_tree_scorers.c:7741:41: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]\n 7741 |         for (__pyx_t_11 = 0; __pyx_t_11 < __pyx_t_10; __pyx_t_11+=1) {\n      |                                         ^\nOrange/classification/_tree_scorers.c:7801:143: warning: comparison of integer expressions of different signedness: ‘__pyx_t_5numpy_int32_t’ {aka ‘int’} and ‘unsigned int’ [-Wsign-compare]\n 7801 |               __pyx_t_17 = (((*((__pyx_t_5numpy_int32_t *) ( /* dim=0 */ (__pyx_v_indices.data + __pyx_t_18 * __pyx_v_indices.strides[0]) ))) != __pyx_v_i) != 0);\n      |                                                                                                                                               ^~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/data/_valuecount.c:567:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/data/_contingency.c:567:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nOrange/data/_io.c: In function ‘__pyx_pf_6Orange_4data_3_io_sparse_read_float.constprop’:\nOrange/data/_io.c:3881:24: warning: ‘__pyx_v_decs’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 3881 |           __pyx_v_decs = (__pyx_v_decs / 10.0);\n      |           ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:4070:24: warning: ‘__pyx_v_value’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 4070 |           __pyx_t_10 = PyFloat_FromDouble(__pyx_v_value); if (unlikely(!__pyx_t_10)) __PYX_ERR(0, 221, __pyx_L5_error)\n      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:480:40: warning: ‘__pyx_v_attr_index’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n  480 |   #define PyInt_FromLong               PyLong_FromLong\n      |                                        ^~~~~~~~~~~~~~~\nOrange/data/_io.c:1301:7: note: ‘__pyx_v_attr_index’ was declared here\n 1301 |   int __pyx_v_attr_index;\n      |       ^~~~~~~~~~~~~~~~~~\nOrange/data/_io.c:2935:25: warning: ‘__pyx_v_atomp’ may be used uninitialized in this function [-Wmaybe-uninitialized]\n 2935 |           __pyx_v_atomp = (__pyx_v_atomp + 1);\n      |           ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/distance/_distance.c:567:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/preprocess/_discretize.c:567:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\nIn file included from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,\n                 from /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,\n                 from Orange/preprocess/_relieff.cpp:581:\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning \"Using deprecated NumPy API, disable it with \" \"#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION\" [-Wcpp]\n   17 | #warning \"Using deprecated NumPy API, disable it with \" \\\n      |  ^~~~~~~\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n  setuptools.SetuptoolsDeprecationWarning,\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/setuptools/command/easy_install.py:159: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.\n  EasyInstallDeprecationWarning,\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n  setuptools.SetuptoolsDeprecationWarning,\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/.venv/lib/python3.7/site-packages/setuptools/command/egg_info.py:628: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.\nPlease extend command classes from setuptools instead of distutils.\n  SetuptoolsDeprecationWarning\n",
    "new_commit_res_code": 139,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d\ncollected 24 items\n\nr2e_tests/test_1.py ssE.....F....F..........                             [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_filename ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/tests/__init__.py, line 65\n  def test_filename(path):\nE       fixture 'path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/tests/__init__.py:65\n=================================== FAILURES ===================================\n_____________________ TestOWTestLearners.test_minimum_size _____________________\n\nself = <r2e_tests.test_1.TestOWTestLearners testMethod=test_minimum_size>\n\n    def test_minimum_size(self):\n        widget = getattr(self, \"widget\", None)\n        if widget is None:\n            self.skipTest(\"minimum size not tested as .widget was not set\")\n>       self.check_minimum_size(widget)\n\n.venv/lib/python3.7/site-packages/orangewidget/tests/base.py:90: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n.venv/lib/python3.7/site-packages/orangewidget/tests/base.py:378: in check_minimum_size\n    self.assertLess(min_size.width(), 800)\nE   AssertionError: 885 not less than 800\n________________ TestOWTestLearners.test_resort_on_data_change _________________\n\nself = <r2e_tests.test_1.TestOWTestLearners testMethod=test_resort_on_data_change>\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n    \n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n    \n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n    \n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n    \n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n    \n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n    \n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n>       self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\nE       AssertionError: 'SetosaLearner' != 'VersicolorLearner'\nE       - SetosaLearner\nE       + VersicolorLearner\n\nr2e_tests/test_1.py:280: AssertionError\n----------------------------- Captured stderr call -----------------------------\nThis plugin does not support propagateSizeHints()\nThis plugin does not support propagateSizeHints()\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_CrossValidationByFeature\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_addon_scorers\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_basic\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_memory_error\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_migrate_removes_invalid_contexts\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_more_learners\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_msg_base_class\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_nan_class\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_one_class_value\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_constant_all_same\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_cross_validation\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_advanced\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_bad\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_bad2\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_overfitted\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_target_changing\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_testOnTest\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_testOnTest_incompatible_domain\nPASSED r2e_tests/test_1.py::TestHelpers::test_results_one_vs_rest\nSKIPPED [1] .venv/lib/python3.7/site-packages/orangewidget/tests/base.py:86: minimum size not tested as .widget was not set\nSKIPPED [1] .venv/lib/python3.7/site-packages/orangewidget/tests/base.py:92: msg base class not tested as .widget was not set\nERROR r2e_tests/test_1.py::test_filename\nFAILED r2e_tests/test_1.py::TestOWTestLearners::test_minimum_size - Assertion...\nFAILED r2e_tests/test_1.py::TestOWTestLearners::test_resort_on_data_change - ...\n=============== 2 failed, 19 passed, 2 skipped, 1 error in 3.71s ===============\nSegmentation fault (core dumped)\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 139,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d\ncollected 24 items\n\nr2e_tests/test_1.py ssE.....F....F........F.                             [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_filename ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/tests/__init__.py, line 65\n  def test_filename(path):\nE       fixture 'path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/tests/__init__.py:65\n=================================== FAILURES ===================================\n_____________________ TestOWTestLearners.test_minimum_size _____________________\n\nself = <r2e_tests.test_1.TestOWTestLearners testMethod=test_minimum_size>\n\n    def test_minimum_size(self):\n        widget = getattr(self, \"widget\", None)\n        if widget is None:\n            self.skipTest(\"minimum size not tested as .widget was not set\")\n>       self.check_minimum_size(widget)\n\n.venv/lib/python3.7/site-packages/orangewidget/tests/base.py:90: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n.venv/lib/python3.7/site-packages/orangewidget/tests/base.py:378: in check_minimum_size\n    self.assertLess(min_size.width(), 800)\nE   AssertionError: 885 not less than 800\n________________ TestOWTestLearners.test_resort_on_data_change _________________\n\nself = <r2e_tests.test_1.TestOWTestLearners testMethod=test_resort_on_data_change>\n\n    def test_resort_on_data_change(self):\n        iris = Table(\"iris\")\n        # one example is included from the other class\n        # to keep F1 from complaining\n        setosa = iris[:51]\n        versicolor = iris[49:100]\n    \n        class SetosaLearner:\n            def __call__(self, data):\n                model = ConstantModel([1., 0, 0])\n                model.domain = iris.domain\n                return model\n    \n        class VersicolorLearner:\n            def __call__(self, data):\n                model = ConstantModel([0, 1., 0])\n                model.domain = iris.domain\n                return model\n    \n        # this is done manually to avoid multiple computations\n        self.widget.resampling = 5\n        self.widget.set_train_data(iris)\n        self.widget.set_learner(SetosaLearner(), 1)\n        self.widget.set_learner(VersicolorLearner(), 2)\n    \n        self.send_signal(self.widget.Inputs.test_data, setosa, wait=5000)\n    \n        self.widget.show()\n        view = self.widget.score_table.view\n        header = view.horizontalHeader()\n        QTest.mouseClick(header.viewport(), Qt.LeftButton)\n    \n        # Ensure that the click on header caused an ascending sort\n        # Ascending sort means that wrong model should be listed first\n        self.assertEqual(header.sortIndicatorOrder(), Qt.AscendingOrder)\n>       self.assertEqual(view.model().item(0, 0).text(), \"VersicolorLearner\")\nE       AssertionError: 'SetosaLearner' != 'VersicolorLearner'\nE       - SetosaLearner\nE       + VersicolorLearner\n\nr2e_tests/test_1.py:280: AssertionError\n----------------------------- Captured stderr call -----------------------------\nThis plugin does not support propagateSizeHints()\nThis plugin does not support propagateSizeHints()\n____________ TestOWTestLearners.test_testOnTest_incompatible_domain ____________\n\nself = <r2e_tests.test_1.TestOWTestLearners testMethod=test_testOnTest_incompatible_domain>\n\n    def test_testOnTest_incompatible_domain(self):\n        iris = Table(\"iris\")\n        self.send_signal(self.widget.Inputs.train_data, iris)\n        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n        self.assertFalse(self.widget.Error.test_data_incompatible.is_shown())\n        self.widget.resampling = OWTestLearners.TestOnTest\n        # test data with the same class (otherwise the widget shows a different error)\n        # and a non-nan X\n        iris_test = iris.transform(Domain([ContinuousVariable()],\n                                          class_vars=iris.domain.class_vars))\n        iris_test.X[:, 0] = 1\n        self.send_signal(self.widget.Inputs.test_data, iris_test)\n        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)\n>       self.assertTrue(self.widget.Error.test_data_incompatible.is_shown())\nE       AssertionError: False is not true\n\nr2e_tests/test_1.py:93: AssertionError\n------------------------------ Captured log call -------------------------------\nERROR    Orange.widgets.evaluate.owtestlearners:owtestlearners.py:851 testing error (in __task_complete):\nTraceback (most recent call last):\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/widgets/evaluate/owtestlearners.py\", line 847, in __task_complete\n    results = result.result()    # type: Results\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/concurrent/futures/_base.py\", line 428, in result\n    return self.__get_result()\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/concurrent/futures/_base.py\", line 384, in __get_result\n    raise self._exception\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/widgets/utils/concurrent.py\", line 175, in run\n    result = func(*args, **kwargs)\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/widgets/evaluate/owtestlearners.py\", line 780, in replace_learners\n    res = evalfunc(*args, **kwargs)\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/evaluation/testing.py\", line 727, in __call__\n    self.store_models))\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/orange3_977b6d85fab32793cf881979cd7af581eb146b3d/Orange/evaluation/testing.py\", line 47, in _mp_worker\n    train_time, test_time)\nUnboundLocalError: local variable 'test_time' referenced before assignment\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_CrossValidationByFeature\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_addon_scorers\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_basic\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_memory_error\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_migrate_removes_invalid_contexts\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_more_learners\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_msg_base_class\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_nan_class\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_one_class_value\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_constant_all_same\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_cross_validation\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_advanced\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_bad\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_bad2\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_scores_log_reg_overfitted\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_target_changing\nPASSED r2e_tests/test_1.py::TestOWTestLearners::test_testOnTest\nPASSED r2e_tests/test_1.py::TestHelpers::test_results_one_vs_rest\nSKIPPED [1] .venv/lib/python3.7/site-packages/orangewidget/tests/base.py:86: minimum size not tested as .widget was not set\nSKIPPED [1] .venv/lib/python3.7/site-packages/orangewidget/tests/base.py:92: msg base class not tested as .widget was not set\nERROR r2e_tests/test_1.py::test_filename\nFAILED r2e_tests/test_1.py::TestOWTestLearners::test_minimum_size - Assertion...\nFAILED r2e_tests/test_1.py::TestOWTestLearners::test_resort_on_data_change - ...\nFAILED r2e_tests/test_1.py::TestOWTestLearners::test_testOnTest_incompatible_domain\n=============== 3 failed, 18 passed, 2 skipped, 1 error in 3.36s ===============\nSegmentation fault (core dumped)\n",
    "old_commit_res_stderr": ""
}
Orange/evaluation/testing.py Orange/widgets/evaluate/tests/test_owtestlearners.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Segmentation Fault Occurs During Test Execution

**Description:**
When running the test suite for the evaluation module, the process crashes unexpectedly with a segmentation fault. This abrupt termination prevents all tests from completing and hinders the identification of other potential issues.

**Example Code:**
The crash occurs during the execution of evaluation tests. Below is a simplified version of the test setup that leads to the segmentation fault:

```python
class TestOWTestLearners(WidgetTest):

    def test_testOnTest_incompatible_domain(self):
        iris = Table('iris')
        self.send_signal(self.widget.Inputs.train_data, iris)
        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)
        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)
        self.widget.resampling = OWTestLearners.TestOnTest
        iris_test = iris.transform(Domain([ContinuousVariable()], class_vars=iris.domain.class_vars))
        iris_test.X[:, 0] = 1
        self.send_signal(self.widget.Inputs.test_data, iris_test)
        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)
```

**Expected Behavior:**
All tests should execute smoothly without crashing, allowing each test case to run to completion and report accurate results.

**Actual Behavior:**
During the execution of the tests, a segmentation fault occurs, terminating the process abruptly. The following error message is displayed:

```
Segmentation fault (core dumped)
```

This crash prevents the remaining tests from running and makes it difficult to identify other underlying issues within the test suite.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### Orange/evaluation/testing.py
# __new__ methods have different arguments
# pylint: disable=arguments-differ
from warnings import warn
from collections import namedtuple
from time import time

import numpy as np

import sklearn.model_selection as skl

from Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable

__all__ = ["Results", "CrossValidation", "LeaveOneOut", "TestOnTrainingData",
           "ShuffleSplit", "TestOnTestData", "sample", "CrossValidationFeature"]

_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',
                                       'failed', 'n_values', 'values',
                                       'probs', 'train_time', 'test_time'))


def _identity(x):
    return x


def _mp_worker(fold_i, train_data, test_data, learner_i, learner,
               store_models):
    predicted, probs, model, failed = None, None, None, False
    try:
        if not train_data or not test_data:
            raise RuntimeError('Test fold is empty')
        # training
        t0 = time()
        model = learner(train_data)
        train_time = time() - t0
        t0  = time()
        # testing
        if train_data.domain.has_discrete_class:
            predicted, probs = model(test_data, model.ValueProbs)
        elif train_data.domain.has_continuous_class:
            predicted = model(test_data, model.Value)
        test_time = time() - t0
    # Different models can fail at any time raising any exception
    except Exception as ex:  # pylint: disable=broad-except
        failed = ex
    return _MpResults(fold_i, learner_i, store_models and model,
                      failed, len(test_data), predicted, probs,
                      train_time, test_time)


class Results:
    """
    Class for storing predictions in model testing.

    Attributes:
        data (Optional[Table]):  Data used for testing.

        models (Optional[List[Model]]): A list of induced models.

        row_indices (np.ndarray): Indices of rows in `data` that were used in
            testing, stored as a numpy vector of length `nrows`.
            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer
            to the target value of instance, that is, the i-th test instance
            is `data[row_indices[i]]`, its actual class is `actual[i]`, and
            the prediction by m-th method is `predicted[m, i]`.

        nrows (int): The number of test instances (including duplicates);
            `nrows` equals the length of `row_indices` and `actual`, and the
            second dimension of `predicted` and `probabilities`.

        actual (np.ndarray): true values of target variable in a vector of
            length `nrows`.

        predicted (np.ndarray): predicted values of target variable in an array
            of shape (number-of-methods, `nrows`)

        probabilities (Optional[np.ndarray]): predicted probabilities
            (for discrete target variables) in an array of shape
            (number-of-methods, `nrows`, number-of-classes)

        folds (List[Slice or List[int]]): a list of indices (or slice objects)
            corresponding to testing data subsets, that is,
            `row_indices[folds[i]]` contains row indices used in fold i, so
            `data[row_indices[folds[i]]]` is the corresponding testing data

        train_time (np.ndarray): training times of batches

        test_time (np.ndarray): testing times of batches
    """
    def __init__(self, data=None, *,
                 nmethods=None, nrows=None, nclasses=None,
                 domain=None,
                 row_indices=None, folds=None, score_by_folds=True,
                 learners=None, models=None, failed=None,
                 actual=None, predicted=None, probabilities=None,
                 store_data=None, store_models=None,
                 train_time=None, test_time=None):
        """
        Construct an instance.

        The constructor stores the given data, and creates empty arrays
        `actual`, `predicted` and `probabilities` if ther are not given but
        sufficient data is provided to deduct their shapes.

        The function

        - set any attributes specified directly through arguments.
        - infers the number of methods, rows and classes from other data
           and/or check their overall consistency.
        - Prepare empty arrays `actual`, `predicted`, `probabilities` and
          `failed` if the are not given. If not enough data is available,
          the corresponding arrays are `None`.

        Args:
            data (Orange.data.Table): stored data from which test was sampled
            nmethods (int): number of methods; can be inferred (or must match)
                the size of `learners`, `models`, `failed`, `predicted` and
                `probabilities`
            nrows (int): number of data instances; can be inferred (or must
                match) `data`, `row_indices`, `actual`, `predicted` and
                `probabilities`
            nclasses (int): number of class values (`None` if continuous); can
                be inferred (or must match) from `domain.class_var` or
                `probabilities`
            domain (Orange.data.Domain): data domain; can be inferred (or must)
                match `data.domain`
            row_indices (np.ndarray): see class documentation
            folds (np.ndarray): see class documentation
            score_by_folds (np.ndarray): see class documentation
            learners (np.ndarray): see class documentation
            models (np.ndarray): see class documentation
            failed (list of str): see class documentation
            actual (np.ndarray): see class documentation
            predicted (np.ndarray): see class documentation
            probabilities (np.ndarray): see class documentation
            store_data (bool): ignored; kept for backward compatibility
            store_models (bool): ignored; kept for backward compatibility
        """

        # Set given data directly from arguments
        self.data = data
        self.domain = domain

        self.row_indices = row_indices
        self.folds = folds
        self.score_by_folds = score_by_folds

        self.learners = learners
        self.models = models

        self.actual = actual
        self.predicted = predicted
        self.probabilities = probabilities
        self.failed = failed

        self.train_time = train_time
        self.test_time = test_time

        # Guess the rest -- or check for ambguities
        def set_or_raise(value, exp_values, msg):
            for exp_value in exp_values:
                if exp_value is False:
                    continue
                if value is None:
                    value = exp_value
                elif value != exp_value:
                    raise ValueError(msg)
            return value
        domain = self.domain = set_or_raise(
            domain, [data is not None and data.domain],
            "mismatching domain")
        self.nrows = nrows = set_or_raise(
            nrows, [actual is not None and len(actual),
                    row_indices is not None and len(row_indices),
                    predicted is not None and predicted.shape[1],
                    probabilities is not None and probabilities.shape[1]],
            "mismatching number of rows")
        if domain is not None and domain.has_continuous_class:
            if nclasses is not None:
                raise ValueError(
                    "regression results cannot have non-None 'nclasses'")
            if probabilities is not None:
                raise ValueError(
                    "regression results cannot have 'probabilities'")
        nclasses = set_or_raise(
            nclasses, [domain is not None and domain.has_discrete_class and
                       len(domain.class_var.values),
                       probabilities is not None and probabilities.shape[2]],
            "mismatching number of class values")
        nmethods = set_or_raise(
            nmethods, [learners is not None and len(learners),
                       models is not None and models.shape[1],
                       failed is not None and len(failed),
                       predicted is not None and predicted.shape[0],
                       probabilities is not None and probabilities.shape[0]],
            "mismatching number of methods")

        # Prepare empty arrays
        if actual is None \
                and nrows is not None:
            self.actual = np.empty(nrows)

        if predicted is None \
                and nmethods is not None and nrows is not None:
            self.predicted = np.empty((nmethods, nrows))

        if probabilities is None \
                and nmethods is not None and nrows is not None \
                and nclasses is not None:
            self.probabilities = \
                np.empty((nmethods, nrows, nclasses))

        if failed is None \
                and nmethods is not None:
            self.failed = [False] * nmethods

    def get_fold(self, fold):
        results = Results()
        results.data = self.data

        if self.folds is None:
            raise ValueError("This 'Results' instance does not have folds.")

        if self.models is not None:
            results.models = self.models[fold]

        results.row_indices = self.row_indices[self.folds[fold]]
        results.actual = self.actual[self.folds[fold]]
        results.predicted = self.predicted[:, self.folds[fold]]
        results.domain = self.domain

        if self.probabilities is not None:
            results.probabilities = self.probabilities[:, self.folds[fold]]

        return results

    def get_augmented_data(self, model_names,
                           include_attrs=True, include_predictions=True,
                           include_probabilities=True):
        """
        Return the test data table augmented with meta attributes containing
        predictions, probabilities (if the task is classification) and fold
        indices.

        Args:
            model_names (list of str): names of models
            include_attrs (bool):
                if set to `False`, original attributes are removed
            include_predictions (bool):
                if set to `False`, predictions are not added
            include_probabilities (bool):
                if set to `False`, probabilities are not added

        Returns:
            augmented_data (Orange.data.Table):
                data augmented with predictions, probabilities and fold indices

        """
        assert self.predicted.shape[0] == len(model_names)

        data = self.data[self.row_indices]
        class_var = data.domain.class_var
        classification = class_var and class_var.is_discrete

        new_meta_attr = []
        new_meta_vals = np.empty((len(data), 0))

        if classification:
            # predictions
            if include_predictions:
                new_meta_attr += (
                    DiscreteVariable(name=name, values=class_var.values)
                    for name in model_names)
                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

            # probabilities
            if include_probabilities:
                for name in model_names:
                    new_meta_attr += (
                        ContinuousVariable(name=f"{name} ({value})")
                        for value in class_var.values)

                for i in self.probabilities:
                    new_meta_vals = np.hstack((new_meta_vals, i))

        elif include_predictions:
            # regression
            new_meta_attr += (ContinuousVariable(name=name)
                              for name in model_names)
            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

        # add fold info
        if self.folds is not None:
            new_meta_attr.append(
                DiscreteVariable(
                    name="Fold",
                    values=[str(i + 1) for i in range(len(self.folds))]))
            fold = np.empty((len(data), 1))
            for i, s in enumerate(self.folds):
                fold[s, 0] = i
            new_meta_vals = np.hstack((new_meta_vals, fold))

        # append new columns to meta attributes
        new_meta_attr = list(data.domain.metas) + new_meta_attr
        new_meta_vals = np.hstack((data.metas, new_meta_vals))

        attrs = data.domain.attributes if include_attrs else []
        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)
        predictions = data.transform(domain)
        predictions.metas = new_meta_vals
        predictions.name = data.name
        return predictions

    def split_by_model(self):
        """
        Split evaluation results by models.

        The method generates instances of `Results` containing data for single
        models
        """
        data = self.data
        nmethods = len(self.predicted)
        for i in range(nmethods):
            res = Results()
            res.data = data
            res.domain = self.domain
            res.learners = [self.learners[i]]
            res.row_indices = self.row_indices
            res.actual = self.actual
            res.folds = self.folds
            res.score_by_folds = self.score_by_folds
            res.test_time = self.test_time[i]
            res.train_time = self.train_time[i]

            res.predicted = self.predicted[(i,), :]
            if getattr(self, "probabilities", None) is not None:
                res.probabilities = self.probabilities[(i,), :, :]

            if self.models is not None:
                res.models = self.models[:, i:i + 1]

            res.failed = [self.failed[i]]
            yield res


class Validation:
    """
    Base class for different testing schemata such as cross validation and
    testing on separate data set.

    If `data` is some data table and `learners` is a list of learning
    algorithms. This will run 5-fold cross validation and store the results
    in `res`.

        cv = CrossValidation(k=5)
        res = cv(data, learners)

    If constructor was given data and learning algorithms (as in
    `res = CrossValidation(data, learners, k=5)`, it used to automagically
    call the instance after constructing it and return `Results` instead
    of an instance of `Validation`. This functionality
    is deprecated and will be removed in the future.

    Attributes:
        store_data (bool): a flag defining whether the data is stored
        store_models (bool): a flag defining whether the models are stored
    """
    score_by_folds = False

    def __new__(cls,
                data=None, learners=None, preprocessor=None, test_data=None,
                *, callback=None, store_data=False, store_models=False,
                n_jobs=None, **kwargs):
        self = super().__new__(cls)

        if (learners is None) != (data is None):
            raise ValueError(
                "learners and train_data must both be present or not")
        if learners is None:
            if preprocessor is not None:
                raise ValueError("preprocessor cannot be given if learners "
                                 "and train_data are omitted")
            if callback is not None:
                raise ValueError("callback cannot be given if learners "
                                 "and train_data are omitted")
            return self

        warn("calling Validation's constructor with data and learners "
             "is deprecated;\nconstruct an instance and call it",
             DeprecationWarning, stacklevel=2)

        # Explicitly call __init__ because Python won't
        self.__init__(store_data=store_data, store_models=store_models,
                      **kwargs)
        if test_data is not None:
            test_data_kwargs = {"test_data": test_data}
        else:
            test_data_kwargs = {}
        return self(data, learners=learners, preprocessor=preprocessor,
                    callback=callback, **test_data_kwargs)

    # Note: this will be called only if __new__ doesn't have data and learners
    def __init__(self, *, store_data=False, store_models=False):
        self.store_data = store_data
        self.store_models = store_models

    def fit(self, *args, **kwargs):
        warn("Validation.fit is deprecated; use the call operator",
             DeprecationWarning)
        return self(*args, **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None):
        """
        Args:
            data (Orange.data.Table): data to be used (usually split) into
                training and testing
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity
        indices = self.get_indices(data)
        folds, row_indices, actual = self.prepare_arrays(data, indices)

        data_splits = (
            (fold_i, preprocessor(data[train_i]), data[test_i])
            for fold_i, (train_i, test_i) in enumerate(indices))
        args_iter = (
            (fold_i, data, test_data, learner_i, learner, self.store_models)
            for (fold_i, data, test_data) in data_splits
            for (learner_i, learner) in enumerate(learners))

        part_results = []
        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]
        for progress, part in zip(parts, args_iter):
            part_results.append(_mp_worker(*(part + ())))
            callback(progress)
        callback(1)

        results = Results(
            data=data if self.store_data else None,
            domain=data.domain,
            nrows=len(row_indices), learners=learners,
            row_indices=row_indices, folds=folds, actual=actual,
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (len(indices), len(learners)))
        self._collect_part_results(results, part_results)
        return results

    @classmethod
    def prepare_arrays(cls, data, indices):
        """Prepare `folds`, `row_indices` and `actual`.

        The method is used by `__call__`. While functional, it may be
        overriden in subclasses for speed-ups.

        Args:
            data (Orange.data.Table): data use for testing
            indices (list of vectors):
                indices of data instances in each test sample

        Returns:
            folds: (np.ndarray): see class documentation
            row_indices: (np.ndarray): see class documentation
            actual: (np.ndarray): see class documentation
        """
        folds = []
        row_indices = []

        ptr = 0
        for _, test in indices:
            folds.append(slice(ptr, ptr + len(test)))
            row_indices.append(test)
            ptr += len(test)

        row_indices = np.concatenate(row_indices, axis=0)
        actual = data[row_indices].Y.ravel()
        return folds, row_indices, actual

    @staticmethod
    def get_indices(data):
        """
        Return a list of arrays of indices of test data instance

        For example, in k-fold CV, the result is a list with `k` elements,
        each containing approximately `len(data) / k` nonoverlapping indices
        into `data`.

        This method is abstract and must be implemented in derived classes
        unless they provide their own implementation of the `__call__`
        method.

        Args:
            data (Orange.data.Table): test data

        Returns:
            indices (list of np.ndarray):
                a list of arrays of indices into `data`
        """
        raise NotImplementedError()

    def _collect_part_results(self, results, part_results):
        part_results = sorted(part_results)

        ptr, prev_fold_i, prev_n_values = 0, 0, 0
        for res in part_results:
            if res.fold_i != prev_fold_i:
                ptr += prev_n_values
                prev_fold_i = res.fold_i
            result_slice = slice(ptr, ptr + res.n_values)
            prev_n_values = res.n_values

            if res.failed:
                results.failed[res.learner_i] = res.failed
                continue

            if self.store_models:
                results.models[res.fold_i][res.learner_i] = res.model

            results.predicted[res.learner_i][result_slice] = res.values
            results.train_time[res.learner_i] += res.train_time
            results.test_time[res.learner_i] += res.test_time
            if res.probs is not None:
                results.probabilities[res.learner_i][result_slice, :] = \
                    res.probs


class CrossValidation(Validation):
    """
    K-fold cross validation

    Attributes:
        k (int): number of folds (default: 10)
        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time
        stratified (bool):
            flag deciding whether to perform stratified cross-validation.
            If `True` but the class sizes don't allow it, it uses non-stratified
            validataion and adds a list `warning` with a warning message(s) to
            the `Result`.
    """
    # TODO: list `warning` contains just repetitions of the same message
    #       replace with a flag in `Results`?
    def __init__(self, k=10, stratified=True, random_state=0,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.k = k
        self.stratified = stratified
        self.random_state = random_state
        self.warnings = [] if warnings is None else warnings

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            try:
                splitter = skl.StratifiedKFold(
                    self.k, shuffle=True, random_state=self.random_state
                )
                splitter.get_n_splits(data.X, data.Y)
                return list(splitter.split(data.X, data.Y))
            except ValueError:
                self.warnings.append("Using non-stratified sampling.")

        splitter = skl.KFold(
            self.k, shuffle=True, random_state=self.random_state)
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class CrossValidationFeature(Validation):
    """
    Cross validation with folds according to values of a feature.

    Attributes:
        feature (Orange.data.Variable): the feature defining the folds
    """
    def __init__(self, feature=None,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.feature = feature

    def get_indices(self, data):
        data = Table(Domain([self.feature], None), data)
        values = data[:, self.feature].X
        indices = []
        for v in range(len(self.feature.values)):
            test_index = np.where(values == v)[0]
            train_index = np.where((values != v) & (~np.isnan(values)))[0]
            if test_index.size and train_index.size:
                indices.append((train_index, test_index))
        if not indices:
            raise ValueError(
                f"'{self.feature.name}' does not have at least two distinct "
                "values on the data")
        return indices


class LeaveOneOut(Validation):
    """Leave-one-out testing"""
    score_by_folds = False

    def get_indices(self, data):
        splitter = skl.LeaveOneOut()
        splitter.get_n_splits(data)
        return list(splitter.split(data))

    @staticmethod
    def prepare_arrays(data, indices):
        # sped up version of super().prepare_arrays(data)
        row_indices = np.arange(len(data))
        return row_indices, row_indices, data.Y.flatten()


class ShuffleSplit(Validation):
    """
    Test by repeated random sampling

    Attributes:
        n_resamples (int): number of repetitions
        test_size (float, int, None):
            If float, should be between 0.0 and 1.0 and represent the proportion
            of the dataset to include in the test split. If int, represents the
            absolute number of test samples. If None, the value is set to the
            complement of the train size. By default, the value is set to 0.1.
            The default will change in version 0.21. It will remain 0.1 only
            if ``train_size`` is unspecified, otherwise it will complement
            the specified ``train_size``.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        train_size : float, int, or None, default is None
            If float, should be between 0.0 and 1.0 and represent the
            proportion of the dataset to include in the train split. If
            int, represents the absolute number of train samples. If None,
            the value is automatically set to the complement of the test size.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        stratified (bool):
            flag deciding whether to perform stratified cross-validation.

        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time

    """
    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,
                 stratified=True, random_state=0,
                 store_data=False, store_models=False):
        super().__init__(store_data=store_data, store_models=store_models)
        self.n_resamples = n_resamples
        self.train_size = train_size
        self.test_size = test_size
        self.stratified = stratified
        self.random_state = random_state

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            splitter = skl.StratifiedShuffleSplit(
                n_splits=self.n_resamples, train_size=self.train_size,
                test_size=self.test_size, random_state=self.random_state
            )
            splitter.get_n_splits(data.X, data.Y)
            return list(splitter.split(data.X, data.Y))

        splitter = skl.ShuffleSplit(
            n_splits=self.n_resamples, train_size=self.train_size,
            test_size=self.test_size, random_state=self.random_state
        )
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class TestOnTestData(Validation):
    """
    Test on separately provided test data

    Note that the class has a different signature for `__call__`.
    """
    # get_indices is not needed in this class, pylint: disable=abstract-method

    def __new__(cls, data=None, test_data=None, learners=None,
                preprocessor=None, **kwargs):
        if "train_data" in kwargs:
            if data is None:
                data = kwargs.pop("train_data")
            else:
                raise ValueError(
                    "argument 'data' is given twice (once as 'train_data')")
        return super().__new__(
            cls,
            data=data, learners=learners, preprocessor=preprocessor,
            test_data=test_data, **kwargs)

    def __call__(self, data, test_data, learners, preprocessor=None,
                 *, callback=None):
        """
        Args:
            data (Orange.data.Table): training data
            test_data (Orange.data.Table): test_data
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity

        train_data = preprocessor(data)
        part_results = []
        for (learner_i, learner) in enumerate(learners):
            part_results.append(
                _mp_worker(0, train_data, test_data, learner_i, learner,
                           self.store_models))
            callback((learner_i + 1) / len(learners))
        callback(1)

        results = Results(
            data=test_data if self.store_data else None,
            domain=test_data.domain,
            nrows=len(test_data), learners=learners,
            row_indices=np.arange(len(test_data)),
            folds=(Ellipsis, ),
            actual=test_data.Y.ravel(),
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (1, len(learners)))
        self._collect_part_results(results, part_results)
        return results


class TestOnTrainingData(TestOnTestData):
    """Test on training data"""
    # get_indices is not needed in this class, pylint: disable=abstract-method
    # signature is such as on the base class, pylint: disable=signature-differs
    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):
        return super().__new__(
            cls,
            data, test_data=data, learners=learners, preprocessor=preprocessor,
            **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None,
                 **kwargs):
        kwargs.setdefault("test_data", data)
        # if kwargs contains anything besides test_data, this will be detected
        # (and complained about) by super().__call__
        return super().__call__(
            data=data, learners=learners, preprocessor=preprocessor,
            callback=callback, **kwargs)


def sample(table, n=0.7, stratified=False, replace=False,
           random_state=None):
    """
    Samples data instances from a data table. Returns the sample and
    a dataset from input data table that are not in the sample. Also
    uses several sampling functions from
    `scikit-learn <http://scikit-learn.org>`_.

    table : data table
        A data table from which to sample.

    n : float, int (default = 0.7)
        If float, should be between 0.0 and 1.0 and represents
        the proportion of data instances in the resulting sample. If
        int, n is the number of data instances in the resulting sample.

    stratified : bool, optional (default = False)
        If true, sampling will try to consider class values and
        match distribution of class values
        in train and test subsets.

    replace : bool, optional (default = False)
        sample with replacement

    random_state : int or RandomState
        Pseudo-random number generator state used for random sampling.
    """

    if isinstance(n, float):
        n = int(n * len(table))

    if replace:
        if random_state is None:
            rgen = np.random
        else:
            rgen = np.random.mtrand.RandomState(random_state)
        a_sample = rgen.randint(0, len(table), n)
        o = np.ones(len(table))
        o[a_sample] = 0
        others = np.nonzero(o)[0]
        return table[a_sample], table[others]

    n = len(table) - n
    if stratified and table.domain.has_discrete_class:
        test_size = max(len(table.domain.class_var.values), n)
        splitter = skl.StratifiedShuffleSplit(
            n_splits=1, test_size=test_size, train_size=len(table) - test_size,
            random_state=random_state)
        splitter.get_n_splits(table.X, table.Y)
        ind = splitter.split(table.X, table.Y)
    else:
        splitter = skl.ShuffleSplit(
            n_splits=1, test_size=n, random_state=random_state)
        splitter.get_n_splits(table)
        ind = splitter.split(table)
    ind = next(ind)
    return table[ind[0]], table[ind[1]]

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** Segmentation Fault Occurs During Test Execution

**Description:**
When running the test suite for the evaluation module, the process crashes unexpectedly with a segmentation fault. This abrupt termination prevents all tests from completing and hinders the identification of other potential issues.

**Example Code:**
The crash occurs during the execution of evaluation tests. Below is a simplified version of the test setup that leads to the segmentation fault:

```python
class TestOWTestLearners(WidgetTest):

    def test_testOnTest_incompatible_domain(self):
        iris = Table('iris')
        self.send_signal(self.widget.Inputs.train_data, iris)
        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)
        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)
        self.widget.resampling = OWTestLearners.TestOnTest
        iris_test = iris.transform(Domain([ContinuousVariable()], class_vars=iris.domain.class_vars))
        iris_test.X[:, 0] = 1
        self.send_signal(self.widget.Inputs.test_data, iris_test)
        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)
```

**Expected Behavior:**
All tests should execute smoothly without crashing, allowing each test case to run to completion and report accurate results.

**Actual Behavior:**
During the execution of the tests, a segmentation fault occurs, terminating the process abruptly. The following error message is displayed:

```
Segmentation fault (core dumped)
```

This crash prevents the remaining tests from running and makes it difficult to identify other underlying issues within the test suite.

[/ISSUE]
{
    "TestOWTestLearners.test_CrossValidationByFeature": "PASSED",
    "TestOWTestLearners.test_addon_scorers": "PASSED",
    "TestOWTestLearners.test_basic": "PASSED",
    "TestOWTestLearners.test_memory_error": "PASSED",
    "TestOWTestLearners.test_migrate_removes_invalid_contexts": "PASSED",
    "TestOWTestLearners.test_more_learners": "PASSED",
    "TestOWTestLearners.test_msg_base_class": "PASSED",
    "TestOWTestLearners.test_nan_class": "PASSED",
    "TestOWTestLearners.test_one_class_value": "PASSED",
    "TestOWTestLearners.test_scores_constant_all_same": "PASSED",
    "TestOWTestLearners.test_scores_cross_validation": "PASSED",
    "TestOWTestLearners.test_scores_log_reg_advanced": "PASSED",
    "TestOWTestLearners.test_scores_log_reg_bad": "PASSED",
    "TestOWTestLearners.test_scores_log_reg_bad2": "PASSED",
    "TestOWTestLearners.test_scores_log_reg_overfitted": "PASSED",
    "TestOWTestLearners.test_target_changing": "PASSED",
    "TestOWTestLearners.test_testOnTest": "PASSED",
    "TestOWTestLearners.test_testOnTest_incompatible_domain": "PASSED",
    "TestHelpers.test_results_one_vs_rest": "PASSED",
    "test_filename": "ERROR",
    "TestOWTestLearners.test_minimum_size": "FAILED",
    "TestOWTestLearners.test_resort_on_data_change": "FAILED"
}
diff --git a/Orange/evaluation/testing.py b/Orange/evaluation/testing.py
index 6789b4103..9019e8899 100644
--- a/Orange/evaluation/testing.py
+++ b/Orange/evaluation/testing.py
@@ -24,7 +24,7 @@ def _identity(x):
 
 def _mp_worker(fold_i, train_data, test_data, learner_i, learner,
                store_models):
-    predicted, probs, model, failed = None, None, None, False
+    predicted, probs, model, failed, train_time, test_time = None, None, None, False, None, None
     try:
         if not train_data or not test_data:
             raise RuntimeError('Test fold is empty')

{"Orange/evaluation/testing.py": "# __new__ methods have different arguments\n# pylint: disable=arguments-differ\nfrom warnings import warn\nfrom collections import namedtuple\nfrom time import time\n\nimport numpy as np\n\nimport sklearn.model_selection as skl\n\nfrom Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable\n\n__all__ = [\"Results\", \"CrossValidation\", \"LeaveOneOut\", \"TestOnTrainingData\",\n           \"ShuffleSplit\", \"TestOnTestData\", \"sample\", \"CrossValidationFeature\"]\n\n_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',\n                                       'failed', 'n_values', 'values',\n                                       'probs', 'train_time', 'test_time'))\n\n\ndef _identity(x):\n    return x\n\n\ndef _mp_worker(fold_i, train_data, test_data, learner_i, learner,\n               store_models):\n    predicted, probs, model, failed = None, None, None, False\n    try:\n        if not train_data or not test_data:\n            raise RuntimeError('Test fold is empty')\n        # training\n        t0 = time()\n        model = learner(train_data)\n        train_time = time() - t0\n        t0  = time()\n        # testing\n        if train_data.domain.has_discrete_class:\n            predicted, probs = model(test_data, model.ValueProbs)\n        elif train_data.domain.has_continuous_class:\n            predicted = model(test_data, model.Value)\n        test_time = time() - t0\n    # Different models can fail at any time raising any exception\n    except Exception as ex:  # pylint: disable=broad-except\n        failed = ex\n    return _MpResults(fold_i, learner_i, store_models and model,\n                      failed, len(test_data), predicted, probs,\n                      train_time, test_time)\n\n\nclass Results:\n    \"\"\"\n    Class for storing predictions in model testing.\n\n    Attributes:\n        data (Optional[Table]):  Data used for testing.\n\n        models (Optional[List[Model]]): A list of induced models.\n\n        row_indices (np.ndarray): Indices of rows in `data` that were used in\n            testing, stored as a numpy vector of length `nrows`.\n            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer\n            to the target value of instance, that is, the i-th test instance\n            is `data[row_indices[i]]`, its actual class is `actual[i]`, and\n            the prediction by m-th method is `predicted[m, i]`.\n\n        nrows (int): The number of test instances (including duplicates);\n            `nrows` equals the length of `row_indices` and `actual`, and the\n            second dimension of `predicted` and `probabilities`.\n\n        actual (np.ndarray): true values of target variable in a vector of\n            length `nrows`.\n\n        predicted (np.ndarray): predicted values of target variable in an array\n            of shape (number-of-methods, `nrows`)\n\n        probabilities (Optional[np.ndarray]): predicted probabilities\n            (for discrete target variables) in an array of shape\n            (number-of-methods, `nrows`, number-of-classes)\n\n        folds (List[Slice or List[int]]): a list of indices (or slice objects)\n            corresponding to testing data subsets, that is,\n            `row_indices[folds[i]]` contains row indices used in fold i, so\n            `data[row_indices[folds[i]]]` is the corresponding testing data\n\n        train_time (np.ndarray): training times of batches\n\n        test_time (np.ndarray): testing times of batches\n    \"\"\"\n    def __init__(self, data=None, *,\n                 nmethods=None, nrows=None, nclasses=None,\n                 domain=None,\n                 row_indices=None, folds=None, score_by_folds=True,\n                 learners=None, models=None, failed=None,\n                 actual=None, predicted=None, probabilities=None,\n                 store_data=None, store_models=None,\n                 train_time=None, test_time=None):\n        \"\"\"\n        Construct an instance.\n\n        The constructor stores the given data, and creates empty arrays\n        `actual`, `predicted` and `probabilities` if ther are not given but\n        sufficient data is provided to deduct their shapes.\n\n        The function\n\n        - set any attributes specified directly through arguments.\n        - infers the number of methods, rows and classes from other data\n           and/or check their overall consistency.\n        - Prepare empty arrays `actual`, `predicted`, `probabilities` and\n          `failed` if the are not given. If not enough data is available,\n          the corresponding arrays are `None`.\n\n        Args:\n            data (Orange.data.Table): stored data from which test was sampled\n            nmethods (int): number of methods; can be inferred (or must match)\n                the size of `learners`, `models`, `failed`, `predicted` and\n                `probabilities`\n            nrows (int): number of data instances; can be inferred (or must\n                match) `data`, `row_indices`, `actual`, `predicted` and\n                `probabilities`\n            nclasses (int): number of class values (`None` if continuous); can\n                be inferred (or must match) from `domain.class_var` or\n                `probabilities`\n            domain (Orange.data.Domain): data domain; can be inferred (or must)\n                match `data.domain`\n            row_indices (np.ndarray): see class documentation\n            folds (np.ndarray): see class documentation\n            score_by_folds (np.ndarray): see class documentation\n            learners (np.ndarray): see class documentation\n            models (np.ndarray): see class documentation\n            failed (list of str): see class documentation\n            actual (np.ndarray): see class documentation\n            predicted (np.ndarray): see class documentation\n            probabilities (np.ndarray): see class documentation\n            store_data (bool): ignored; kept for backward compatibility\n            store_models (bool): ignored; kept for backward compatibility\n        \"\"\"\n\n        # Set given data directly from arguments\n        self.data = data\n        self.domain = domain\n\n        self.row_indices = row_indices\n        self.folds = folds\n        self.score_by_folds = score_by_folds\n\n        self.learners = learners\n        self.models = models\n\n        self.actual = actual\n        self.predicted = predicted\n        self.probabilities = probabilities\n        self.failed = failed\n\n        self.train_time = train_time\n        self.test_time = test_time\n\n        # Guess the rest -- or check for ambguities\n        def set_or_raise(value, exp_values, msg):\n            for exp_value in exp_values:\n                if exp_value is False:\n                    continue\n                if value is None:\n                    value = exp_value\n                elif value != exp_value:\n                    raise ValueError(msg)\n            return value\n        domain = self.domain = set_or_raise(\n            domain, [data is not None and data.domain],\n            \"mismatching domain\")\n        self.nrows = nrows = set_or_raise(\n            nrows, [actual is not None and len(actual),\n                    row_indices is not None and len(row_indices),\n                    predicted is not None and predicted.shape[1],\n                    probabilities is not None and probabilities.shape[1]],\n            \"mismatching number of rows\")\n        if domain is not None and domain.has_continuous_class:\n            if nclasses is not None:\n                raise ValueError(\n                    \"regression results cannot have non-None 'nclasses'\")\n            if probabilities is not None:\n                raise ValueError(\n                    \"regression results cannot have 'probabilities'\")\n        nclasses = set_or_raise(\n            nclasses, [domain is not None and domain.has_discrete_class and\n                       len(domain.class_var.values),\n                       probabilities is not None and probabilities.shape[2]],\n            \"mismatching number of class values\")\n        nmethods = set_or_raise(\n            nmethods, [learners is not None and len(learners),\n                       models is not None and models.shape[1],\n                       failed is not None and len(failed),\n                       predicted is not None and predicted.shape[0],\n                       probabilities is not None and probabilities.shape[0]],\n            \"mismatching number of methods\")\n\n        # Prepare empty arrays\n        if actual is None \\\n                and nrows is not None:\n            self.actual = np.empty(nrows)\n\n        if predicted is None \\\n                and nmethods is not None and nrows is not None:\n            self.predicted = np.empty((nmethods, nrows))\n\n        if probabilities is None \\\n                and nmethods is not None and nrows is not None \\\n                and nclasses is not None:\n            self.probabilities = \\\n                np.empty((nmethods, nrows, nclasses))\n\n        if failed is None \\\n                and nmethods is not None:\n            self.failed = [False] * nmethods\n\n    def get_fold(self, fold):\n        results = Results()\n        results.data = self.data\n\n        if self.folds is None:\n            raise ValueError(\"This 'Results' instance does not have folds.\")\n\n        if self.models is not None:\n            results.models = self.models[fold]\n\n        results.row_indices = self.row_indices[self.folds[fold]]\n        results.actual = self.actual[self.folds[fold]]\n        results.predicted = self.predicted[:, self.folds[fold]]\n        results.domain = self.domain\n\n        if self.probabilities is not None:\n            results.probabilities = self.probabilities[:, self.folds[fold]]\n\n        return results\n\n    def get_augmented_data(self, model_names,\n                           include_attrs=True, include_predictions=True,\n                           include_probabilities=True):\n        \"\"\"\n        Return the test data table augmented with meta attributes containing\n        predictions, probabilities (if the task is classification) and fold\n        indices.\n\n        Args:\n            model_names (list of str): names of models\n            include_attrs (bool):\n                if set to `False`, original attributes are removed\n            include_predictions (bool):\n                if set to `False`, predictions are not added\n            include_probabilities (bool):\n                if set to `False`, probabilities are not added\n\n        Returns:\n            augmented_data (Orange.data.Table):\n                data augmented with predictions, probabilities and fold indices\n\n        \"\"\"\n        assert self.predicted.shape[0] == len(model_names)\n\n        data = self.data[self.row_indices]\n        class_var = data.domain.class_var\n        classification = class_var and class_var.is_discrete\n\n        new_meta_attr = []\n        new_meta_vals = np.empty((len(data), 0))\n\n        if classification:\n            # predictions\n            if include_predictions:\n                new_meta_attr += (\n                    DiscreteVariable(name=name, values=class_var.values)\n                    for name in model_names)\n                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n            # probabilities\n            if include_probabilities:\n                for name in model_names:\n                    new_meta_attr += (\n                        ContinuousVariable(name=f\"{name} ({value})\")\n                        for value in class_var.values)\n\n                for i in self.probabilities:\n                    new_meta_vals = np.hstack((new_meta_vals, i))\n\n        elif include_predictions:\n            # regression\n            new_meta_attr += (ContinuousVariable(name=name)\n                              for name in model_names)\n            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))\n\n        # add fold info\n        if self.folds is not None:\n            new_meta_attr.append(\n                DiscreteVariable(\n                    name=\"Fold\",\n                    values=[str(i + 1) for i in range(len(self.folds))]))\n            fold = np.empty((len(data), 1))\n            for i, s in enumerate(self.folds):\n                fold[s, 0] = i\n            new_meta_vals = np.hstack((new_meta_vals, fold))\n\n        # append new columns to meta attributes\n        new_meta_attr = list(data.domain.metas) + new_meta_attr\n        new_meta_vals = np.hstack((data.metas, new_meta_vals))\n\n        attrs = data.domain.attributes if include_attrs else []\n        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)\n        predictions = data.transform(domain)\n        predictions.metas = new_meta_vals\n        predictions.name = data.name\n        return predictions\n\n    def split_by_model(self):\n        \"\"\"\n        Split evaluation results by models.\n\n        The method generates instances of `Results` containing data for single\n        models\n        \"\"\"\n        data = self.data\n        nmethods = len(self.predicted)\n        for i in range(nmethods):\n            res = Results()\n            res.data = data\n            res.domain = self.domain\n            res.learners = [self.learners[i]]\n            res.row_indices = self.row_indices\n            res.actual = self.actual\n            res.folds = self.folds\n            res.score_by_folds = self.score_by_folds\n            res.test_time = self.test_time[i]\n            res.train_time = self.train_time[i]\n\n            res.predicted = self.predicted[(i,), :]\n            if getattr(self, \"probabilities\", None) is not None:\n                res.probabilities = self.probabilities[(i,), :, :]\n\n            if self.models is not None:\n                res.models = self.models[:, i:i + 1]\n\n            res.failed = [self.failed[i]]\n            yield res\n\n\nclass Validation:\n    \"\"\"\n    Base class for different testing schemata such as cross validation and\n    testing on separate data set.\n\n    If `data` is some data table and `learners` is a list of learning\n    algorithms. This will run 5-fold cross validation and store the results\n    in `res`.\n\n        cv = CrossValidation(k=5)\n        res = cv(data, learners)\n\n    If constructor was given data and learning algorithms (as in\n    `res = CrossValidation(data, learners, k=5)`, it used to automagically\n    call the instance after constructing it and return `Results` instead\n    of an instance of `Validation`. This functionality\n    is deprecated and will be removed in the future.\n\n    Attributes:\n        store_data (bool): a flag defining whether the data is stored\n        store_models (bool): a flag defining whether the models are stored\n    \"\"\"\n    score_by_folds = False\n\n    def __new__(cls,\n                data=None, learners=None, preprocessor=None, test_data=None,\n                *, callback=None, store_data=False, store_models=False,\n                n_jobs=None, **kwargs):\n        self = super().__new__(cls)\n\n        if (learners is None) != (data is None):\n            raise ValueError(\n                \"learners and train_data must both be present or not\")\n        if learners is None:\n            if preprocessor is not None:\n                raise ValueError(\"preprocessor cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            if callback is not None:\n                raise ValueError(\"callback cannot be given if learners \"\n                                 \"and train_data are omitted\")\n            return self\n\n        warn(\"calling Validation's constructor with data and learners \"\n             \"is deprecated;\\nconstruct an instance and call it\",\n             DeprecationWarning, stacklevel=2)\n\n        # Explicitly call __init__ because Python won't\n        self.__init__(store_data=store_data, store_models=store_models,\n                      **kwargs)\n        if test_data is not None:\n            test_data_kwargs = {\"test_data\": test_data}\n        else:\n            test_data_kwargs = {}\n        return self(data, learners=learners, preprocessor=preprocessor,\n                    callback=callback, **test_data_kwargs)\n\n    # Note: this will be called only if __new__ doesn't have data and learners\n    def __init__(self, *, store_data=False, store_models=False):\n        self.store_data = store_data\n        self.store_models = store_models\n\n    def fit(self, *args, **kwargs):\n        warn(\"Validation.fit is deprecated; use the call operator\",\n             DeprecationWarning)\n        return self(*args, **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): data to be used (usually split) into\n                training and testing\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n        indices = self.get_indices(data)\n        folds, row_indices, actual = self.prepare_arrays(data, indices)\n\n        data_splits = (\n            (fold_i, preprocessor(data[train_i]), data[test_i])\n            for fold_i, (train_i, test_i) in enumerate(indices))\n        args_iter = (\n            (fold_i, data, test_data, learner_i, learner, self.store_models)\n            for (fold_i, data, test_data) in data_splits\n            for (learner_i, learner) in enumerate(learners))\n\n        part_results = []\n        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]\n        for progress, part in zip(parts, args_iter):\n            part_results.append(_mp_worker(*(part + ())))\n            callback(progress)\n        callback(1)\n\n        results = Results(\n            data=data if self.store_data else None,\n            domain=data.domain,\n            nrows=len(row_indices), learners=learners,\n            row_indices=row_indices, folds=folds, actual=actual,\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (len(indices), len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n    @classmethod\n    def prepare_arrays(cls, data, indices):\n        \"\"\"Prepare `folds`, `row_indices` and `actual`.\n\n        The method is used by `__call__`. While functional, it may be\n        overriden in subclasses for speed-ups.\n\n        Args:\n            data (Orange.data.Table): data use for testing\n            indices (list of vectors):\n                indices of data instances in each test sample\n\n        Returns:\n            folds: (np.ndarray): see class documentation\n            row_indices: (np.ndarray): see class documentation\n            actual: (np.ndarray): see class documentation\n        \"\"\"\n        folds = []\n        row_indices = []\n\n        ptr = 0\n        for _, test in indices:\n            folds.append(slice(ptr, ptr + len(test)))\n            row_indices.append(test)\n            ptr += len(test)\n\n        row_indices = np.concatenate(row_indices, axis=0)\n        actual = data[row_indices].Y.ravel()\n        return folds, row_indices, actual\n\n    @staticmethod\n    def get_indices(data):\n        \"\"\"\n        Return a list of arrays of indices of test data instance\n\n        For example, in k-fold CV, the result is a list with `k` elements,\n        each containing approximately `len(data) / k` nonoverlapping indices\n        into `data`.\n\n        This method is abstract and must be implemented in derived classes\n        unless they provide their own implementation of the `__call__`\n        method.\n\n        Args:\n            data (Orange.data.Table): test data\n\n        Returns:\n            indices (list of np.ndarray):\n                a list of arrays of indices into `data`\n        \"\"\"\n        raise NotImplementedError()\n\n    def _collect_part_results(self, results, part_results):\n        part_results = sorted(part_results)\n\n        ptr, prev_fold_i, prev_n_values = 0, 0, 0\n        for res in part_results:\n            if res.fold_i != prev_fold_i:\n                ptr += prev_n_values\n                prev_fold_i = res.fold_i\n            result_slice = slice(ptr, ptr + res.n_values)\n            prev_n_values = res.n_values\n\n            if res.failed:\n                results.failed[res.learner_i] = res.failed\n                continue\n\n            if self.store_models:\n                results.models[res.fold_i][res.learner_i] = res.model\n\n            results.predicted[res.learner_i][result_slice] = res.values\n            results.train_time[res.learner_i] += res.train_time\n            results.test_time[res.learner_i] += res.test_time\n            if res.probs is not None:\n                results.probabilities[res.learner_i][result_slice, :] = \\\n                    res.probs\n\n\nclass CrossValidation(Validation):\n    \"\"\"\n    K-fold cross validation\n\n    Attributes:\n        k (int): number of folds (default: 10)\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n            If `True` but the class sizes don't allow it, it uses non-stratified\n            validataion and adds a list `warning` with a warning message(s) to\n            the `Result`.\n    \"\"\"\n    # TODO: list `warning` contains just repetitions of the same message\n    #       replace with a flag in `Results`?\n    def __init__(self, k=10, stratified=True, random_state=0,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.k = k\n        self.stratified = stratified\n        self.random_state = random_state\n        self.warnings = [] if warnings is None else warnings\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            try:\n                splitter = skl.StratifiedKFold(\n                    self.k, shuffle=True, random_state=self.random_state\n                )\n                splitter.get_n_splits(data.X, data.Y)\n                return list(splitter.split(data.X, data.Y))\n            except ValueError:\n                self.warnings.append(\"Using non-stratified sampling.\")\n\n        splitter = skl.KFold(\n            self.k, shuffle=True, random_state=self.random_state)\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass CrossValidationFeature(Validation):\n    \"\"\"\n    Cross validation with folds according to values of a feature.\n\n    Attributes:\n        feature (Orange.data.Variable): the feature defining the folds\n    \"\"\"\n    def __init__(self, feature=None,\n                 store_data=False, store_models=False, warnings=None):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.feature = feature\n\n    def get_indices(self, data):\n        data = Table(Domain([self.feature], None), data)\n        values = data[:, self.feature].X\n        indices = []\n        for v in range(len(self.feature.values)):\n            test_index = np.where(values == v)[0]\n            train_index = np.where((values != v) & (~np.isnan(values)))[0]\n            if test_index.size and train_index.size:\n                indices.append((train_index, test_index))\n        if not indices:\n            raise ValueError(\n                f\"'{self.feature.name}' does not have at least two distinct \"\n                \"values on the data\")\n        return indices\n\n\nclass LeaveOneOut(Validation):\n    \"\"\"Leave-one-out testing\"\"\"\n    score_by_folds = False\n\n    def get_indices(self, data):\n        splitter = skl.LeaveOneOut()\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n    @staticmethod\n    def prepare_arrays(data, indices):\n        # sped up version of super().prepare_arrays(data)\n        row_indices = np.arange(len(data))\n        return row_indices, row_indices, data.Y.flatten()\n\n\nclass ShuffleSplit(Validation):\n    \"\"\"\n    Test by repeated random sampling\n\n    Attributes:\n        n_resamples (int): number of repetitions\n        test_size (float, int, None):\n            If float, should be between 0.0 and 1.0 and represent the proportion\n            of the dataset to include in the test split. If int, represents the\n            absolute number of test samples. If None, the value is set to the\n            complement of the train size. By default, the value is set to 0.1.\n            The default will change in version 0.21. It will remain 0.1 only\n            if ``train_size`` is unspecified, otherwise it will complement\n            the specified ``train_size``.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        train_size : float, int, or None, default is None\n            If float, should be between 0.0 and 1.0 and represent the\n            proportion of the dataset to include in the train split. If\n            int, represents the absolute number of train samples. If None,\n            the value is automatically set to the complement of the test size.\n            (from documentation of scipy.sklearn.StratifiedShuffleSplit)\n\n        stratified (bool):\n            flag deciding whether to perform stratified cross-validation.\n\n        random_state (int):\n            seed for random number generator (default: 0). If set to `None`,\n            a different seed is used each time\n\n    \"\"\"\n    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,\n                 stratified=True, random_state=0,\n                 store_data=False, store_models=False):\n        super().__init__(store_data=store_data, store_models=store_models)\n        self.n_resamples = n_resamples\n        self.train_size = train_size\n        self.test_size = test_size\n        self.stratified = stratified\n        self.random_state = random_state\n\n    def get_indices(self, data):\n        if self.stratified and data.domain.has_discrete_class:\n            splitter = skl.StratifiedShuffleSplit(\n                n_splits=self.n_resamples, train_size=self.train_size,\n                test_size=self.test_size, random_state=self.random_state\n            )\n            splitter.get_n_splits(data.X, data.Y)\n            return list(splitter.split(data.X, data.Y))\n\n        splitter = skl.ShuffleSplit(\n            n_splits=self.n_resamples, train_size=self.train_size,\n            test_size=self.test_size, random_state=self.random_state\n        )\n        splitter.get_n_splits(data)\n        return list(splitter.split(data))\n\n\nclass TestOnTestData(Validation):\n    \"\"\"\n    Test on separately provided test data\n\n    Note that the class has a different signature for `__call__`.\n    \"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n\n    def __new__(cls, data=None, test_data=None, learners=None,\n                preprocessor=None, **kwargs):\n        if \"train_data\" in kwargs:\n            if data is None:\n                data = kwargs.pop(\"train_data\")\n            else:\n                raise ValueError(\n                    \"argument 'data' is given twice (once as 'train_data')\")\n        return super().__new__(\n            cls,\n            data=data, learners=learners, preprocessor=preprocessor,\n            test_data=test_data, **kwargs)\n\n    def __call__(self, data, test_data, learners, preprocessor=None,\n                 *, callback=None):\n        \"\"\"\n        Args:\n            data (Orange.data.Table): training data\n            test_data (Orange.data.Table): test_data\n            learners (list of Orange.Learner): a list of learning algorithms\n            preprocessor (Orange.preprocess.Preprocess): preprocessor applied\n                on training data\n            callback (Callable): a function called to notify about the progress\n\n        Returns:\n            results (Result): results of testing\n        \"\"\"\n        if preprocessor is None:\n            preprocessor = _identity\n        if callback is None:\n            callback = _identity\n\n        train_data = preprocessor(data)\n        part_results = []\n        for (learner_i, learner) in enumerate(learners):\n            part_results.append(\n                _mp_worker(0, train_data, test_data, learner_i, learner,\n                           self.store_models))\n            callback((learner_i + 1) / len(learners))\n        callback(1)\n\n        results = Results(\n            data=test_data if self.store_data else None,\n            domain=test_data.domain,\n            nrows=len(test_data), learners=learners,\n            row_indices=np.arange(len(test_data)),\n            folds=(Ellipsis, ),\n            actual=test_data.Y.ravel(),\n            score_by_folds=self.score_by_folds,\n            train_time=np.zeros((len(learners),)),\n            test_time=np.zeros((len(learners),)))\n\n        if self.store_models:\n            results.models = np.tile(None, (1, len(learners)))\n        self._collect_part_results(results, part_results)\n        return results\n\n\nclass TestOnTrainingData(TestOnTestData):\n    \"\"\"Test on training data\"\"\"\n    # get_indices is not needed in this class, pylint: disable=abstract-method\n    # signature is such as on the base class, pylint: disable=signature-differs\n    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):\n        return super().__new__(\n            cls,\n            data, test_data=data, learners=learners, preprocessor=preprocessor,\n            **kwargs)\n\n    def __call__(self, data, learners, preprocessor=None, *, callback=None,\n                 **kwargs):\n        kwargs.setdefault(\"test_data\", data)\n        # if kwargs contains anything besides test_data, this will be detected\n        # (and complained about) by super().__call__\n        return super().__call__(\n            data=data, learners=learners, preprocessor=preprocessor,\n            callback=callback, **kwargs)\n\n\ndef sample(table, n=0.7, stratified=False, replace=False,\n           random_state=None):\n    \"\"\"\n    Samples data instances from a data table. Returns the sample and\n    a dataset from input data table that are not in the sample. Also\n    uses several sampling functions from\n    `scikit-learn <http://scikit-learn.org>`_.\n\n    table : data table\n        A data table from which to sample.\n\n    n : float, int (default = 0.7)\n        If float, should be between 0.0 and 1.0 and represents\n        the proportion of data instances in the resulting sample. If\n        int, n is the number of data instances in the resulting sample.\n\n    stratified : bool, optional (default = False)\n        If true, sampling will try to consider class values and\n        match distribution of class values\n        in train and test subsets.\n\n    replace : bool, optional (default = False)\n        sample with replacement\n\n    random_state : int or RandomState\n        Pseudo-random number generator state used for random sampling.\n    \"\"\"\n\n    if isinstance(n, float):\n        n = int(n * len(table))\n\n    if replace:\n        if random_state is None:\n            rgen = np.random\n        else:\n            rgen = np.random.mtrand.RandomState(random_state)\n        a_sample = rgen.randint(0, len(table), n)\n        o = np.ones(len(table))\n        o[a_sample] = 0\n        others = np.nonzero(o)[0]\n        return table[a_sample], table[others]\n\n    n = len(table) - n\n    if stratified and table.domain.has_discrete_class:\n        test_size = max(len(table.domain.class_var.values), n)\n        splitter = skl.StratifiedShuffleSplit(\n            n_splits=1, test_size=test_size, train_size=len(table) - test_size,\n            random_state=random_state)\n        splitter.get_n_splits(table.X, table.Y)\n        ind = splitter.split(table.X, table.Y)\n    else:\n        splitter = skl.ShuffleSplit(\n            n_splits=1, test_size=n, random_state=random_state)\n        splitter.get_n_splits(table)\n        ind = splitter.split(table)\n    ind = next(ind)\n    return table[ind[0]], table[ind[1]]\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Segmentation Fault Occurs During Test Execution

**Description:**
When running the test suite for the evaluation module, the process crashes unexpectedly with a segmentation fault. This abrupt termination prevents all tests from completing and hinders the identification of other potential issues.

**Example Code:**
The crash occurs during the execution of evaluation tests. Below is a simplified version of the test setup that leads to the segmentation fault:

```python
class TestOWTestLearners(WidgetTest):

    def test_testOnTest_incompatible_domain(self):
        iris = Table('iris')
        self.send_signal(self.widget.Inputs.train_data, iris)
        self.send_signal(self.widget.Inputs.learner, LogisticRegressionLearner(), 0)
        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)
        self.widget.resampling = OWTestLearners.TestOnTest
        iris_test = iris.transform(Domain([ContinuousVariable()], class_vars=iris.domain.class_vars))
        iris_test.X[:, 0] = 1
        self.send_signal(self.widget.Inputs.test_data, iris_test)
        self.get_output(self.widget.Outputs.evaluations_results, wait=5000)
```

**Expected Behavior:**
All tests should execute smoothly without crashing, allowing each test case to run to completion and report accurate results.

**Actual Behavior:**
During the execution of the tests, a segmentation fault occurs, terminating the process abruptly. The following error message is displayed:

```
Segmentation fault (core dumped)
```

This crash prevents the remaining tests from running and makes it difficult to identify other underlying issues within the test suite.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### Orange/evaluation/testing.py
# __new__ methods have different arguments
# pylint: disable=arguments-differ
from warnings import warn
from collections import namedtuple
from time import time

import numpy as np

import sklearn.model_selection as skl

from Orange.data import Table, Domain, ContinuousVariable, DiscreteVariable

__all__ = ["Results", "CrossValidation", "LeaveOneOut", "TestOnTrainingData",
           "ShuffleSplit", "TestOnTestData", "sample", "CrossValidationFeature"]

_MpResults = namedtuple('_MpResults', ('fold_i', 'learner_i', 'model',
                                       'failed', 'n_values', 'values',
                                       'probs', 'train_time', 'test_time'))


def _identity(x):
    return x


def _mp_worker(fold_i, train_data, test_data, learner_i, learner,
               store_models):
    predicted, probs, model, failed = None, None, None, False
    try:
        if not train_data or not test_data:
            raise RuntimeError('Test fold is empty')
        # training
        t0 = time()
        model = learner(train_data)
        train_time = time() - t0
        t0  = time()
        # testing
        if train_data.domain.has_discrete_class:
            predicted, probs = model(test_data, model.ValueProbs)
        elif train_data.domain.has_continuous_class:
            predicted = model(test_data, model.Value)
        test_time = time() - t0
    # Different models can fail at any time raising any exception
    except Exception as ex:  # pylint: disable=broad-except
        failed = ex
    return _MpResults(fold_i, learner_i, store_models and model,
                      failed, len(test_data), predicted, probs,
                      train_time, test_time)


class Results:
    """
    Class for storing predictions in model testing.

    Attributes:
        data (Optional[Table]):  Data used for testing.

        models (Optional[List[Model]]): A list of induced models.

        row_indices (np.ndarray): Indices of rows in `data` that were used in
            testing, stored as a numpy vector of length `nrows`.
            Values of `actual[i]`, `predicted[i]` and `probabilities[i]` refer
            to the target value of instance, that is, the i-th test instance
            is `data[row_indices[i]]`, its actual class is `actual[i]`, and
            the prediction by m-th method is `predicted[m, i]`.

        nrows (int): The number of test instances (including duplicates);
            `nrows` equals the length of `row_indices` and `actual`, and the
            second dimension of `predicted` and `probabilities`.

        actual (np.ndarray): true values of target variable in a vector of
            length `nrows`.

        predicted (np.ndarray): predicted values of target variable in an array
            of shape (number-of-methods, `nrows`)

        probabilities (Optional[np.ndarray]): predicted probabilities
            (for discrete target variables) in an array of shape
            (number-of-methods, `nrows`, number-of-classes)

        folds (List[Slice or List[int]]): a list of indices (or slice objects)
            corresponding to testing data subsets, that is,
            `row_indices[folds[i]]` contains row indices used in fold i, so
            `data[row_indices[folds[i]]]` is the corresponding testing data

        train_time (np.ndarray): training times of batches

        test_time (np.ndarray): testing times of batches
    """
    def __init__(self, data=None, *,
                 nmethods=None, nrows=None, nclasses=None,
                 domain=None,
                 row_indices=None, folds=None, score_by_folds=True,
                 learners=None, models=None, failed=None,
                 actual=None, predicted=None, probabilities=None,
                 store_data=None, store_models=None,
                 train_time=None, test_time=None):
        """
        Construct an instance.

        The constructor stores the given data, and creates empty arrays
        `actual`, `predicted` and `probabilities` if ther are not given but
        sufficient data is provided to deduct their shapes.

        The function

        - set any attributes specified directly through arguments.
        - infers the number of methods, rows and classes from other data
           and/or check their overall consistency.
        - Prepare empty arrays `actual`, `predicted`, `probabilities` and
          `failed` if the are not given. If not enough data is available,
          the corresponding arrays are `None`.

        Args:
            data (Orange.data.Table): stored data from which test was sampled
            nmethods (int): number of methods; can be inferred (or must match)
                the size of `learners`, `models`, `failed`, `predicted` and
                `probabilities`
            nrows (int): number of data instances; can be inferred (or must
                match) `data`, `row_indices`, `actual`, `predicted` and
                `probabilities`
            nclasses (int): number of class values (`None` if continuous); can
                be inferred (or must match) from `domain.class_var` or
                `probabilities`
            domain (Orange.data.Domain): data domain; can be inferred (or must)
                match `data.domain`
            row_indices (np.ndarray): see class documentation
            folds (np.ndarray): see class documentation
            score_by_folds (np.ndarray): see class documentation
            learners (np.ndarray): see class documentation
            models (np.ndarray): see class documentation
            failed (list of str): see class documentation
            actual (np.ndarray): see class documentation
            predicted (np.ndarray): see class documentation
            probabilities (np.ndarray): see class documentation
            store_data (bool): ignored; kept for backward compatibility
            store_models (bool): ignored; kept for backward compatibility
        """

        # Set given data directly from arguments
        self.data = data
        self.domain = domain

        self.row_indices = row_indices
        self.folds = folds
        self.score_by_folds = score_by_folds

        self.learners = learners
        self.models = models

        self.actual = actual
        self.predicted = predicted
        self.probabilities = probabilities
        self.failed = failed

        self.train_time = train_time
        self.test_time = test_time

        # Guess the rest -- or check for ambguities
        def set_or_raise(value, exp_values, msg):
            for exp_value in exp_values:
                if exp_value is False:
                    continue
                if value is None:
                    value = exp_value
                elif value != exp_value:
                    raise ValueError(msg)
            return value
        domain = self.domain = set_or_raise(
            domain, [data is not None and data.domain],
            "mismatching domain")
        self.nrows = nrows = set_or_raise(
            nrows, [actual is not None and len(actual),
                    row_indices is not None and len(row_indices),
                    predicted is not None and predicted.shape[1],
                    probabilities is not None and probabilities.shape[1]],
            "mismatching number of rows")
        if domain is not None and domain.has_continuous_class:
            if nclasses is not None:
                raise ValueError(
                    "regression results cannot have non-None 'nclasses'")
            if probabilities is not None:
                raise ValueError(
                    "regression results cannot have 'probabilities'")
        nclasses = set_or_raise(
            nclasses, [domain is not None and domain.has_discrete_class and
                       len(domain.class_var.values),
                       probabilities is not None and probabilities.shape[2]],
            "mismatching number of class values")
        nmethods = set_or_raise(
            nmethods, [learners is not None and len(learners),
                       models is not None and models.shape[1],
                       failed is not None and len(failed),
                       predicted is not None and predicted.shape[0],
                       probabilities is not None and probabilities.shape[0]],
            "mismatching number of methods")

        # Prepare empty arrays
        if actual is None \
                and nrows is not None:
            self.actual = np.empty(nrows)

        if predicted is None \
                and nmethods is not None and nrows is not None:
            self.predicted = np.empty((nmethods, nrows))

        if probabilities is None \
                and nmethods is not None and nrows is not None \
                and nclasses is not None:
            self.probabilities = \
                np.empty((nmethods, nrows, nclasses))

        if failed is None \
                and nmethods is not None:
            self.failed = [False] * nmethods

    def get_fold(self, fold):
        results = Results()
        results.data = self.data

        if self.folds is None:
            raise ValueError("This 'Results' instance does not have folds.")

        if self.models is not None:
            results.models = self.models[fold]

        results.row_indices = self.row_indices[self.folds[fold]]
        results.actual = self.actual[self.folds[fold]]
        results.predicted = self.predicted[:, self.folds[fold]]
        results.domain = self.domain

        if self.probabilities is not None:
            results.probabilities = self.probabilities[:, self.folds[fold]]

        return results

    def get_augmented_data(self, model_names,
                           include_attrs=True, include_predictions=True,
                           include_probabilities=True):
        """
        Return the test data table augmented with meta attributes containing
        predictions, probabilities (if the task is classification) and fold
        indices.

        Args:
            model_names (list of str): names of models
            include_attrs (bool):
                if set to `False`, original attributes are removed
            include_predictions (bool):
                if set to `False`, predictions are not added
            include_probabilities (bool):
                if set to `False`, probabilities are not added

        Returns:
            augmented_data (Orange.data.Table):
                data augmented with predictions, probabilities and fold indices

        """
        assert self.predicted.shape[0] == len(model_names)

        data = self.data[self.row_indices]
        class_var = data.domain.class_var
        classification = class_var and class_var.is_discrete

        new_meta_attr = []
        new_meta_vals = np.empty((len(data), 0))

        if classification:
            # predictions
            if include_predictions:
                new_meta_attr += (
                    DiscreteVariable(name=name, values=class_var.values)
                    for name in model_names)
                new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

            # probabilities
            if include_probabilities:
                for name in model_names:
                    new_meta_attr += (
                        ContinuousVariable(name=f"{name} ({value})")
                        for value in class_var.values)

                for i in self.probabilities:
                    new_meta_vals = np.hstack((new_meta_vals, i))

        elif include_predictions:
            # regression
            new_meta_attr += (ContinuousVariable(name=name)
                              for name in model_names)
            new_meta_vals = np.hstack((new_meta_vals, self.predicted.T))

        # add fold info
        if self.folds is not None:
            new_meta_attr.append(
                DiscreteVariable(
                    name="Fold",
                    values=[str(i + 1) for i in range(len(self.folds))]))
            fold = np.empty((len(data), 1))
            for i, s in enumerate(self.folds):
                fold[s, 0] = i
            new_meta_vals = np.hstack((new_meta_vals, fold))

        # append new columns to meta attributes
        new_meta_attr = list(data.domain.metas) + new_meta_attr
        new_meta_vals = np.hstack((data.metas, new_meta_vals))

        attrs = data.domain.attributes if include_attrs else []
        domain = Domain(attrs, data.domain.class_vars, metas=new_meta_attr)
        predictions = data.transform(domain)
        predictions.metas = new_meta_vals
        predictions.name = data.name
        return predictions

    def split_by_model(self):
        """
        Split evaluation results by models.

        The method generates instances of `Results` containing data for single
        models
        """
        data = self.data
        nmethods = len(self.predicted)
        for i in range(nmethods):
            res = Results()
            res.data = data
            res.domain = self.domain
            res.learners = [self.learners[i]]
            res.row_indices = self.row_indices
            res.actual = self.actual
            res.folds = self.folds
            res.score_by_folds = self.score_by_folds
            res.test_time = self.test_time[i]
            res.train_time = self.train_time[i]

            res.predicted = self.predicted[(i,), :]
            if getattr(self, "probabilities", None) is not None:
                res.probabilities = self.probabilities[(i,), :, :]

            if self.models is not None:
                res.models = self.models[:, i:i + 1]

            res.failed = [self.failed[i]]
            yield res


class Validation:
    """
    Base class for different testing schemata such as cross validation and
    testing on separate data set.

    If `data` is some data table and `learners` is a list of learning
    algorithms. This will run 5-fold cross validation and store the results
    in `res`.

        cv = CrossValidation(k=5)
        res = cv(data, learners)

    If constructor was given data and learning algorithms (as in
    `res = CrossValidation(data, learners, k=5)`, it used to automagically
    call the instance after constructing it and return `Results` instead
    of an instance of `Validation`. This functionality
    is deprecated and will be removed in the future.

    Attributes:
        store_data (bool): a flag defining whether the data is stored
        store_models (bool): a flag defining whether the models are stored
    """
    score_by_folds = False

    def __new__(cls,
                data=None, learners=None, preprocessor=None, test_data=None,
                *, callback=None, store_data=False, store_models=False,
                n_jobs=None, **kwargs):
        self = super().__new__(cls)

        if (learners is None) != (data is None):
            raise ValueError(
                "learners and train_data must both be present or not")
        if learners is None:
            if preprocessor is not None:
                raise ValueError("preprocessor cannot be given if learners "
                                 "and train_data are omitted")
            if callback is not None:
                raise ValueError("callback cannot be given if learners "
                                 "and train_data are omitted")
            return self

        warn("calling Validation's constructor with data and learners "
             "is deprecated;\nconstruct an instance and call it",
             DeprecationWarning, stacklevel=2)

        # Explicitly call __init__ because Python won't
        self.__init__(store_data=store_data, store_models=store_models,
                      **kwargs)
        if test_data is not None:
            test_data_kwargs = {"test_data": test_data}
        else:
            test_data_kwargs = {}
        return self(data, learners=learners, preprocessor=preprocessor,
                    callback=callback, **test_data_kwargs)

    # Note: this will be called only if __new__ doesn't have data and learners
    def __init__(self, *, store_data=False, store_models=False):
        self.store_data = store_data
        self.store_models = store_models

    def fit(self, *args, **kwargs):
        warn("Validation.fit is deprecated; use the call operator",
             DeprecationWarning)
        return self(*args, **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None):
        """
        Args:
            data (Orange.data.Table): data to be used (usually split) into
                training and testing
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity
        indices = self.get_indices(data)
        folds, row_indices, actual = self.prepare_arrays(data, indices)

        data_splits = (
            (fold_i, preprocessor(data[train_i]), data[test_i])
            for fold_i, (train_i, test_i) in enumerate(indices))
        args_iter = (
            (fold_i, data, test_data, learner_i, learner, self.store_models)
            for (fold_i, data, test_data) in data_splits
            for (learner_i, learner) in enumerate(learners))

        part_results = []
        parts = np.linspace(.0, .99, len(learners) * len(indices) + 1)[1:]
        for progress, part in zip(parts, args_iter):
            part_results.append(_mp_worker(*(part + ())))
            callback(progress)
        callback(1)

        results = Results(
            data=data if self.store_data else None,
            domain=data.domain,
            nrows=len(row_indices), learners=learners,
            row_indices=row_indices, folds=folds, actual=actual,
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (len(indices), len(learners)))
        self._collect_part_results(results, part_results)
        return results

    @classmethod
    def prepare_arrays(cls, data, indices):
        """Prepare `folds`, `row_indices` and `actual`.

        The method is used by `__call__`. While functional, it may be
        overriden in subclasses for speed-ups.

        Args:
            data (Orange.data.Table): data use for testing
            indices (list of vectors):
                indices of data instances in each test sample

        Returns:
            folds: (np.ndarray): see class documentation
            row_indices: (np.ndarray): see class documentation
            actual: (np.ndarray): see class documentation
        """
        folds = []
        row_indices = []

        ptr = 0
        for _, test in indices:
            folds.append(slice(ptr, ptr + len(test)))
            row_indices.append(test)
            ptr += len(test)

        row_indices = np.concatenate(row_indices, axis=0)
        actual = data[row_indices].Y.ravel()
        return folds, row_indices, actual

    @staticmethod
    def get_indices(data):
        """
        Return a list of arrays of indices of test data instance

        For example, in k-fold CV, the result is a list with `k` elements,
        each containing approximately `len(data) / k` nonoverlapping indices
        into `data`.

        This method is abstract and must be implemented in derived classes
        unless they provide their own implementation of the `__call__`
        method.

        Args:
            data (Orange.data.Table): test data

        Returns:
            indices (list of np.ndarray):
                a list of arrays of indices into `data`
        """
        raise NotImplementedError()

    def _collect_part_results(self, results, part_results):
        part_results = sorted(part_results)

        ptr, prev_fold_i, prev_n_values = 0, 0, 0
        for res in part_results:
            if res.fold_i != prev_fold_i:
                ptr += prev_n_values
                prev_fold_i = res.fold_i
            result_slice = slice(ptr, ptr + res.n_values)
            prev_n_values = res.n_values

            if res.failed:
                results.failed[res.learner_i] = res.failed
                continue

            if self.store_models:
                results.models[res.fold_i][res.learner_i] = res.model

            results.predicted[res.learner_i][result_slice] = res.values
            results.train_time[res.learner_i] += res.train_time
            results.test_time[res.learner_i] += res.test_time
            if res.probs is not None:
                results.probabilities[res.learner_i][result_slice, :] = \
                    res.probs


class CrossValidation(Validation):
    """
    K-fold cross validation

    Attributes:
        k (int): number of folds (default: 10)
        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time
        stratified (bool):
            flag deciding whether to perform stratified cross-validation.
            If `True` but the class sizes don't allow it, it uses non-stratified
            validataion and adds a list `warning` with a warning message(s) to
            the `Result`.
    """
    # TODO: list `warning` contains just repetitions of the same message
    #       replace with a flag in `Results`?
    def __init__(self, k=10, stratified=True, random_state=0,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.k = k
        self.stratified = stratified
        self.random_state = random_state
        self.warnings = [] if warnings is None else warnings

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            try:
                splitter = skl.StratifiedKFold(
                    self.k, shuffle=True, random_state=self.random_state
                )
                splitter.get_n_splits(data.X, data.Y)
                return list(splitter.split(data.X, data.Y))
            except ValueError:
                self.warnings.append("Using non-stratified sampling.")

        splitter = skl.KFold(
            self.k, shuffle=True, random_state=self.random_state)
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class CrossValidationFeature(Validation):
    """
    Cross validation with folds according to values of a feature.

    Attributes:
        feature (Orange.data.Variable): the feature defining the folds
    """
    def __init__(self, feature=None,
                 store_data=False, store_models=False, warnings=None):
        super().__init__(store_data=store_data, store_models=store_models)
        self.feature = feature

    def get_indices(self, data):
        data = Table(Domain([self.feature], None), data)
        values = data[:, self.feature].X
        indices = []
        for v in range(len(self.feature.values)):
            test_index = np.where(values == v)[0]
            train_index = np.where((values != v) & (~np.isnan(values)))[0]
            if test_index.size and train_index.size:
                indices.append((train_index, test_index))
        if not indices:
            raise ValueError(
                f"'{self.feature.name}' does not have at least two distinct "
                "values on the data")
        return indices


class LeaveOneOut(Validation):
    """Leave-one-out testing"""
    score_by_folds = False

    def get_indices(self, data):
        splitter = skl.LeaveOneOut()
        splitter.get_n_splits(data)
        return list(splitter.split(data))

    @staticmethod
    def prepare_arrays(data, indices):
        # sped up version of super().prepare_arrays(data)
        row_indices = np.arange(len(data))
        return row_indices, row_indices, data.Y.flatten()


class ShuffleSplit(Validation):
    """
    Test by repeated random sampling

    Attributes:
        n_resamples (int): number of repetitions
        test_size (float, int, None):
            If float, should be between 0.0 and 1.0 and represent the proportion
            of the dataset to include in the test split. If int, represents the
            absolute number of test samples. If None, the value is set to the
            complement of the train size. By default, the value is set to 0.1.
            The default will change in version 0.21. It will remain 0.1 only
            if ``train_size`` is unspecified, otherwise it will complement
            the specified ``train_size``.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        train_size : float, int, or None, default is None
            If float, should be between 0.0 and 1.0 and represent the
            proportion of the dataset to include in the train split. If
            int, represents the absolute number of train samples. If None,
            the value is automatically set to the complement of the test size.
            (from documentation of scipy.sklearn.StratifiedShuffleSplit)

        stratified (bool):
            flag deciding whether to perform stratified cross-validation.

        random_state (int):
            seed for random number generator (default: 0). If set to `None`,
            a different seed is used each time

    """
    def __init__(self, n_resamples=10, train_size=None, test_size=0.1,
                 stratified=True, random_state=0,
                 store_data=False, store_models=False):
        super().__init__(store_data=store_data, store_models=store_models)
        self.n_resamples = n_resamples
        self.train_size = train_size
        self.test_size = test_size
        self.stratified = stratified
        self.random_state = random_state

    def get_indices(self, data):
        if self.stratified and data.domain.has_discrete_class:
            splitter = skl.StratifiedShuffleSplit(
                n_splits=self.n_resamples, train_size=self.train_size,
                test_size=self.test_size, random_state=self.random_state
            )
            splitter.get_n_splits(data.X, data.Y)
            return list(splitter.split(data.X, data.Y))

        splitter = skl.ShuffleSplit(
            n_splits=self.n_resamples, train_size=self.train_size,
            test_size=self.test_size, random_state=self.random_state
        )
        splitter.get_n_splits(data)
        return list(splitter.split(data))


class TestOnTestData(Validation):
    """
    Test on separately provided test data

    Note that the class has a different signature for `__call__`.
    """
    # get_indices is not needed in this class, pylint: disable=abstract-method

    def __new__(cls, data=None, test_data=None, learners=None,
                preprocessor=None, **kwargs):
        if "train_data" in kwargs:
            if data is None:
                data = kwargs.pop("train_data")
            else:
                raise ValueError(
                    "argument 'data' is given twice (once as 'train_data')")
        return super().__new__(
            cls,
            data=data, learners=learners, preprocessor=preprocessor,
            test_data=test_data, **kwargs)

    def __call__(self, data, test_data, learners, preprocessor=None,
                 *, callback=None):
        """
        Args:
            data (Orange.data.Table): training data
            test_data (Orange.data.Table): test_data
            learners (list of Orange.Learner): a list of learning algorithms
            preprocessor (Orange.preprocess.Preprocess): preprocessor applied
                on training data
            callback (Callable): a function called to notify about the progress

        Returns:
            results (Result): results of testing
        """
        if preprocessor is None:
            preprocessor = _identity
        if callback is None:
            callback = _identity

        train_data = preprocessor(data)
        part_results = []
        for (learner_i, learner) in enumerate(learners):
            part_results.append(
                _mp_worker(0, train_data, test_data, learner_i, learner,
                           self.store_models))
            callback((learner_i + 1) / len(learners))
        callback(1)

        results = Results(
            data=test_data if self.store_data else None,
            domain=test_data.domain,
            nrows=len(test_data), learners=learners,
            row_indices=np.arange(len(test_data)),
            folds=(Ellipsis, ),
            actual=test_data.Y.ravel(),
            score_by_folds=self.score_by_folds,
            train_time=np.zeros((len(learners),)),
            test_time=np.zeros((len(learners),)))

        if self.store_models:
            results.models = np.tile(None, (1, len(learners)))
        self._collect_part_results(results, part_results)
        return results


class TestOnTrainingData(TestOnTestData):
    """Test on training data"""
    # get_indices is not needed in this class, pylint: disable=abstract-method
    # signature is such as on the base class, pylint: disable=signature-differs
    def __new__(cls, data=None, learners=None, preprocessor=None, **kwargs):
        return super().__new__(
            cls,
            data, test_data=data, learners=learners, preprocessor=preprocessor,
            **kwargs)

    def __call__(self, data, learners, preprocessor=None, *, callback=None,
                 **kwargs):
        kwargs.setdefault("test_data", data)
        # if kwargs contains anything besides test_data, this will be detected
        # (and complained about) by super().__call__
        return super().__call__(
            data=data, learners=learners, preprocessor=preprocessor,
            callback=callback, **kwargs)


def sample(table, n=0.7, stratified=False, replace=False,
           random_state=None):
    """
    Samples data instances from a data table. Returns the sample and
    a dataset from input data table that are not in the sample. Also
    uses several sampling functions from
    `scikit-learn <http://scikit-learn.org>`_.

    table : data table
        A data table from which to sample.

    n : float, int (default = 0.7)
        If float, should be between 0.0 and 1.0 and represents
        the proportion of data instances in the resulting sample. If
        int, n is the number of data instances in the resulting sample.

    stratified : bool, optional (default = False)
        If true, sampling will try to consider class values and
        match distribution of class values
        in train and test subsets.

    replace : bool, optional (default = False)
        sample with replacement

    random_state : int or RandomState
        Pseudo-random number generator state used for random sampling.
    """

    if isinstance(n, float):
        n = int(n * len(table))

    if replace:
        if random_state is None:
            rgen = np.random
        else:
            rgen = np.random.mtrand.RandomState(random_state)
        a_sample = rgen.randint(0, len(table), n)
        o = np.ones(len(table))
        o[a_sample] = 0
        others = np.nonzero(o)[0]
        return table[a_sample], table[others]

    n = len(table) - n
    if stratified and table.domain.has_discrete_class:
        test_size = max(len(table.domain.class_var.values), n)
        splitter = skl.StratifiedShuffleSplit(
            n_splits=1, test_size=test_size, train_size=len(table) - test_size,
            random_state=random_state)
        splitter.get_n_splits(table.X, table.Y)
        ind = splitter.split(table.X, table.Y)
    else:
        splitter = skl.ShuffleSplit(
            n_splits=1, test_size=n, random_state=random_state)
        splitter.get_n_splits(table)
        ind = splitter.split(table)
    ind = next(ind)
    return table[ind[0]], table[ind[1]]

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user