{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface for dataset (component) pushing\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom collections import OrderedDict\nimport logging\nimport re\nfrom tempfile import TemporaryFile\n\nfrom datalad.interface.base import (\n    Interface,\n    build_doc,\n)\nfrom datalad.interface.common_opts import (\n    jobs_opt,\n    recursion_limit,\n    recursion_flag,\n)\nfrom datalad.interface.utils import (\n    eval_results,\n)\nfrom datalad.interface.results import annexjson2result\nfrom datalad.log import log_progress\nfrom datalad.support.annexrepo import (\n    AnnexJsonProtocol,\n    AnnexRepo,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import (\n    EnsureStr,\n    EnsureNone,\n    EnsureChoice,\n)\nfrom datalad.support.exceptions import CommandError\nfrom datalad.utils import (\n    Path,\n    ensure_list,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    EnsureDataset,\n    datasetmethod,\n    require_dataset,\n    resolve_path,\n)\nfrom datalad.core.local.diff import diff_dataset\n\n\nlgr = logging.getLogger('datalad.core.distributed.push')\n\n\n@build_doc\nclass Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    @staticmethod\n    def custom_result_summary_renderer(results):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]\n\n\ndef _datasets_since_(dataset, since, paths, recursive, recursion_limit):\n    \"\"\"Generator\"\"\"\n    # rely on diff() reporting sequentially across datasets\n    cur_ds = None\n    ds_res = None\n    for res in diff_dataset(\n            dataset=dataset,\n            fr=since,\n            # we never touch unsaved content\n            to='HEAD',\n            constant_refs=False,\n            path=paths,\n            # we need to know what is around locally to be able\n            # to report something that should have been pushed\n            # but could not, because we don't have a copy.\n            # however, getting this info here is needlessly\n            # expensive, we will do it at the latest possible stage\n            # in _push_data()\n            annex=None,\n            recursive=recursive,\n            recursion_limit=recursion_limit,\n            # make it as fast as possible\n            eval_file_type=False,\n            # TODO?: expose order as an option for diff and push\n            # since in some cases breadth-first would be sufficient\n            # and result in \"taking action faster\"\n            reporting_order='bottom-up'\n    ):\n        if res.get('action', None) != 'diff':\n            # we don't care right now\n            continue\n        if res.get('status', None) != 'ok':\n            # we cannot handle this situation, report it in panic\n            raise RuntimeError(\n                'Cannot handle non-OK diff result: {}'.format(res))\n        parentds = res.get('parentds', None)\n        if not parentds:\n            raise RuntimeError(\n                'Cannot handle diff result without a parent dataset '\n                'property: {}'.format(res))\n        if res.get('type', None) == 'dataset':\n            # a subdataset record in another dataset\n            # this could be here, because\n            # - this dataset was explicitly requested by path\n            #   -> should get a dedicated dataset record -- even without recursion\n            # - a path within an existing subdataset was given\n            # - a path within an non-existing subdataset was given\n            #   locally or not)\n            #   -> it should be ignored, but should not cause the branch in the\n            #   superdataset not to be pushed, if this was the only change\n            p = Path(res['path'])\n            # was given as an explicit path argument\n            if any(arg == p for arg in paths) and \\\n                    not GitRepo.is_valid_repo(res['path']):\n                raise ValueError(\n                    'Cannot publish subdataset, not present: {}'.format(res['path']))\n\n        if parentds != cur_ds:\n            if ds_res:\n                # we switch to another dataset, yield this one so outside\n                # code can start processing immediately\n                yield (cur_ds, ds_res)\n            # clean start\n            ds_res = []\n            cur_ds = parentds\n        ds_res.append({\n            k: v for k, v in res.items()\n            if k in (\n                # let's keep 'state' in for now, it would make it possible\n                # to implement a \"sync\"-type push downstream that actually\n                # pulls 'deleted' files\n                'state',\n                # 'file' to copy-to, and subdataset records to possibly\n                # act on\n                'type',\n                # essential\n                'path')\n        })\n\n    # if we have something left to report, do it\n    # importantly do not test for ds_res, even if we had only seen subdataset\n    # records to be changes, we would still want to push the git branches\n    if cur_ds:\n        yield (cur_ds, ds_res)\n\n\ndef _push(dspath, content, target, data, force, jobs, res_kwargs, pbars,\n          got_path_arg=False):\n    force_git_push = force in ('all', 'gitpush')\n\n    # nothing recursive in here, we only need a repo to work with\n    ds = Dataset(dspath)\n    repo = ds.repo\n\n    res_kwargs.update(type='dataset', path=dspath)\n\n    # content will be unique for every push (even on the same dataset)\n    pbar_id = 'push-{}-{}'.format(target, id(content))\n    # register for final orderly take down\n    pbars[pbar_id] = ds\n    log_progress(\n        lgr.info, pbar_id,\n        'Determine push target',\n        unit=' Steps',\n        label='Push',\n        total=4,\n    )\n    # pristine input arg\n    _target = target\n    # verified or auto-detected\n    target = None\n    if not _target:\n        try:\n            # let Git figure out what needs doing\n            # we will reuse the result further down again, so nothing is wasted\n            wannabe_gitpush = repo.push(remote=None, git_options=['--dry-run'])\n            # we did not get an explicit push target, get it from Git\n            target = set(p.get('remote', None) for p in wannabe_gitpush)\n            # handle case where a pushinfo record did not have a 'remote'\n            # property -- should not happen, but be robust\n            target.discard(None)\n        except Exception as e:\n            lgr.debug(\n                'Dry-run push to determine default push target failed, '\n                'assume no configuration: %s', e)\n            target = set()\n        if not len(target):\n            yield dict(\n                res_kwargs,\n                status='impossible',\n                message='No push target given, and none could be '\n                        'auto-detected, please specify via --to',\n            )\n            return\n        elif len(target) > 1:\n            # dunno if this can ever happen, but if it does, report\n            # nicely\n            yield dict(\n                res_kwargs,\n                status='error',\n                message=(\n                    'No push target given, '\n                    'multiple candidates auto-detected: %s',\n                    list(target),\n                )\n            )\n            return\n        else:\n            # can only be a single one at this point\n            target = target.pop()\n\n    if not target:\n        if _target not in repo.get_remotes():\n            yield dict(\n                res_kwargs,\n                status='error',\n                message=(\n                    \"Unknown target sibling '%s'.\", _target))\n            return\n        target = _target\n\n    log_progress(\n        lgr.info, pbar_id, \"Push refspecs\",\n        label=\"Push to '{}'\".format(target), update=1, total=4)\n\n    # define config var name for potential publication dependencies\n    depvar = 'remote.{}.datalad-publish-depends'.format(target)\n    # list of remotes that are publication dependencies for the\n    # target remote\n    publish_depends = ensure_list(ds.config.get(depvar, []))\n    if publish_depends:\n        lgr.debug(\"Discovered publication dependencies for '%s': %s'\",\n                  target, publish_depends)\n\n    # cache repo type\n    is_annex_repo = isinstance(ds.repo, AnnexRepo)\n\n    # TODO prevent this when `target` is a special remote\n    # (possibly redo) a push attempt to figure out what needs pushing\n    # do this on the main target only, and apply the result to all\n    # dependencies\n    try:\n        if _target:\n            # only do it when an explicit target was given, otherwise\n            # we can reuse the result from the auto-probing above\n            wannabe_gitpush = repo.push(\n                remote=target,\n                git_options=['--dry-run'])\n    except Exception as e:\n        lgr.debug(\n            'Dry-run push to check push configuration failed, '\n            'assume no configuration: %s', e)\n        wannabe_gitpush = []\n    refspecs2push = [\n        # if an upstream branch is set, go with it\n        p['from_ref']\n        if ds.config.get(\n            # refs come in as refs/heads/<branchname>\n            # need to cut the prefix\n            'branch.{}.remote'.format(p['from_ref'][11:]),\n            None) == target and ds.config.get(\n                'branch.{}.merge'.format(p['from_ref'][11:]),\n                None)\n        # if not, define target refspec explicitly to avoid having to\n        # set an upstream branch, which would happen implicitly from\n        # a users POV, and may also be hard to decide when publication\n        # dependencies are present\n        else '{}:{}'.format(p['from_ref'], p['to_ref'])\n        for p in wannabe_gitpush\n        # TODO: what if a publication dependency doesn't have it yet\n        # should we not attempt to push, because the main target has it?\n        if 'uptodate' not in p['operations'] and (\n            # cannot think of a scenario where we would want to push a\n            # managed branch directly, instead of the corresponding branch\n            'refs/heads/adjusted' not in p['from_ref'])\n    ]\n    # TODO this is not right with managed branches\n    active_branch = repo.get_active_branch()\n    if active_branch and is_annex_repo:\n        # we could face a managed branch, in which case we need to\n        # determine the actual one and make sure it is sync'ed with the\n        # managed one, and push that one instead. following methods can\n        # be called unconditionally\n        repo.localsync(managed_only=True)\n        active_branch = repo.get_corresponding_branch(\n            active_branch) or active_branch\n\n    if not refspecs2push and not active_branch:\n        # nothing was set up for push, and we have no active branch\n        # this is a weird one, let's confess and stop here\n        # I don't think we need to support such a scenario\n        if not active_branch:\n            yield dict(\n                res_kwargs,\n                status='impossible',\n                message=\n                'There is no active branch, cannot determine remote '\n                'branch'\n            )\n            return\n\n    # make sure that we always push the active branch (the context for the\n    # potential path arguments) and the annex branch -- because we claim\n    # to know better than any git config\n    must_have_branches = [active_branch] if active_branch else []\n    if is_annex_repo:\n        must_have_branches.append('git-annex')\n    for branch in must_have_branches:\n        _append_branch_to_refspec_if_needed(ds, refspecs2push, branch)\n\n    # we know what to push and where, now dependency processing first\n    for r in publish_depends:\n        # simply make a call to this function again, all the same, but\n        # target is different\n        yield from _push(\n            dspath,\n            content,\n            # to this particular dependency\n            r,\n            data,\n            force,\n            jobs,\n            res_kwargs.copy(),\n            pbars,\n            got_path_arg=got_path_arg,\n        )\n\n    # and lastly the primary push target\n    target_is_git_remote = repo.config.get(\n        'remote.{}.url'.format(target), None) is not None\n\n    # git-annex data copy\n    #\n    if is_annex_repo:\n        if data != \"nothing\":\n            log_progress(\n                lgr.info, pbar_id, \"Transfer data\",\n                label=\"Transfer data to '{}'\".format(target), update=2, total=4)\n            yield from _push_data(\n                ds,\n                target,\n                content,\n                data,\n                force,\n                jobs,\n                res_kwargs.copy(),\n                got_path_arg=got_path_arg,\n            )\n        else:\n            lgr.debug(\"Data transfer to '%s' disabled by argument\", target)\n    else:\n        lgr.debug(\"No data transfer: %s is not a git annex repository\", repo)\n\n    if not target_is_git_remote:\n        # there is nothing that we need to push or sync with on the git-side\n        # of things with this remote\n        return\n\n    log_progress(\n        lgr.info, pbar_id, \"Update availability information\",\n        label=\"Update availability for '{}'\".format(target), update=3, total=4)\n\n    # TODO fetch is only needed if anything was actually transferred. Collect this\n    # info and make the following conditional on it\n\n    # after file transfer the remote might have different commits to\n    # the annex branch. They have to be merged locally, otherwise a\n    # push of it further down will fail\n    try:\n        # fetch remote, let annex sync them locally, so that the push\n        # later on works.\n        # We have to fetch via the push url (if there is any),\n        # not a pull url.\n        # The latter might be dumb and without the execution of a\n        # post-update hook we might not be able to retrieve the\n        # server-side git-annex branch updates (and git-annex does\n        # not trigger the hook on copy), but we know we have\n        # full access via the push url -- we have just used it to copy.\n        lgr.debug(\"Fetch 'git-annex' branch updates from '%s'\", target)\n        fetch_cmd = ['fetch', target, 'git-annex']\n        pushurl = repo.config.get(\n            'remote.{}.pushurl'.format(target), None)\n        if pushurl:\n            # for some reason overwriting remote.{target}.url\n            # does not have any effect...\n            fetch_cmd = [\n                '-c',\n                'url.{}.insteadof={}'.format(\n                    pushurl,\n                    repo.config.get(\n                        'remote.{}.url'.format(target), None)\n                )\n            ] + fetch_cmd\n            lgr.debug(\n                \"Sync local annex branch from pushurl after remote \"\n                'availability update.')\n        repo.call_git(fetch_cmd)\n        # If no CommandError was raised, it means that remote has git-annex\n        # but local repo might not be an annex yet. Since there is nothing to \"sync\"\n        # from us, we just skip localsync without mutating repo into an AnnexRepo\n        if is_annex_repo:\n            repo.localsync(target)\n    except CommandError as e:\n        # it is OK if the remote doesn't have a git-annex branch yet\n        # (e.g. fresh repo)\n        # TODO is this possible? we just copied? Maybe check if anything\n        # was actually copied?\n        if \"fatal: couldn't find remote ref git-annex\" not in e.stderr.lower():\n            raise\n        lgr.debug('Remote does not have a git-annex branch: %s', e)\n\n    if not refspecs2push:\n        lgr.debug('No refspecs found that need to be pushed')\n        return\n\n    # and push all relevant branches, plus the git-annex branch to announce\n    # local availability info too\n    yield from _push_refspecs(\n        repo,\n        target,\n        refspecs2push,\n        force_git_push,\n        res_kwargs.copy(),\n    )\n\n\ndef _append_branch_to_refspec_if_needed(ds, refspecs, branch):\n    # try to anticipate any flavor of an idea of a branch ending up in a refspec\n    looks_like_that_branch = re.compile(\n        r'((^|.*:)refs/heads/|.*:|^){}$'.format(branch))\n    if all(not looks_like_that_branch.match(r) for r in refspecs):\n        refspecs.append(\n            branch\n            if ds.config.get('branch.{}.merge'.format(branch), None)\n            else '{branch}:{branch}'.format(branch=branch)\n        )\n\n\ndef _push_refspecs(repo, target, refspecs, force_git_push, res_kwargs):\n    push_res = repo.push(\n        remote=target,\n        refspec=refspecs,\n        git_options=['--force'] if force_git_push else None,\n    )\n    # TODO maybe compress into a single message whenever everything is\n    # OK?\n    for pr in push_res:\n        ops = pr['operations']\n        status = (\n            'error'\n            if any(o in ops for o in (\n                'error', 'no-match', 'rejected', 'remote-rejected',\n                'remote-failure'))\n            else 'notneeded'\n            if 'uptodate' in pr['operations']\n            else 'ok'\n            if any(o in ops for o in (\n                'new-tag', 'new-branch', 'forced-update', 'fast-forward'))\n            # no really a good fit, but we have tested all relevant\n            # operations above, so in some sense this condition should be\n            # impossible to achieve\n            else 'impossible'\n        )\n        refspec = '{}:{}'.format(pr['from_ref'], pr['to_ref'])\n        yield dict(\n            res_kwargs,\n            status=status,\n            target=pr['remote'],\n            refspec=refspec,\n            operations=ops,\n            hints=pr.get('hints', None),\n            # seems like a good idea to pass on Git's native message\n            # TODO maybe implement a dedicated result renderer, instead\n            # of duplicating information only so that the default one\n            # can make sense at all\n            message='{}->{}:{} {}'.format(\n                pr['from_ref'],\n                pr['remote'],\n                pr['to_ref'],\n                pr['note']),\n        )\n\n\ndef _push_data(ds, target, content, data, force, jobs, res_kwargs,\n               got_path_arg=False):\n    if ds.config.getbool('remote.{}'.format(target), 'annex-ignore', False):\n        lgr.debug(\n            \"Target '%s' is set to annex-ignore, exclude from data-push.\",\n            target,\n        )\n        return\n\n    ds_repo = ds.repo\n\n    res_kwargs['target'] = target\n    if not ds.config.get('.'.join(('remote', target, 'annex-uuid')), None):\n        # this remote either isn't an annex,\n        # or hasn't been properly initialized\n        # given that there was no annex-ignore, let's try to init it\n        # see https://github.com/datalad/datalad/issues/5143 for the story\n        ds_repo.localsync(target)\n\n        if not ds.config.get('.'.join(('remote', target, 'annex-uuid')), None):\n            # still nothing\n            # rather than barfing tons of messages for each file, do one\n            # for the entire dataset\n            yield dict(\n                res_kwargs,\n                action='copy',\n                status='impossible'\n                if force in ('all', 'checkdatapresent')\n                else 'notneeded',\n                message=(\n                    \"Target '%s' does not appear to be an annex remote\",\n                    target)\n            )\n            return\n\n    # it really looks like we will transfer files, get info on what annex\n    # has in store\n    # paths must be recoded to a dataset REPO root (in case of a symlinked\n    # location\n    annex_info_init = \\\n        {ds_repo.pathobj / Path(c['path']).relative_to(ds.pathobj): c\n         for c in content} if ds.pathobj != ds_repo.pathobj else \\\n        {Path(c['path']): c for c in content}\n    content = ds.repo.get_content_annexinfo(\n        # paths are taken from `annex_info_init`\n        paths=None,\n        init=annex_info_init,\n        ref='HEAD',\n        # this is an expensive operation that is only needed\n        # to perform a warning below, and for more accurate\n        # progress reporting (exclude unavailable content).\n        # limit to cases with explicit paths provided\n        eval_availability=True if got_path_arg else False,\n    )\n    # figure out which of the reported content (after evaluating\n    # `since` and `path` arguments needs transport\n    to_transfer = [\n        c\n        for c in content.values()\n        # by force\n        if ((force in ('all', 'checkdatapresent') or\n             # or by modification report\n             c.get('state', None) not in ('clean', 'deleted'))\n            # only consider annex'ed files\n            and 'key' in c\n        )\n    ]\n    if got_path_arg:\n        for c in [c for c in to_transfer if not c.get('has_content', False)]:\n            yield dict(\n                res_kwargs,\n                type=c['type'],\n                path=c['path'],\n                action='copy',\n                status='impossible',\n                message='Slated for transport, but no content present',\n            )\n\n    cmd = ['copy', '--batch', '-z', '--to', target]\n\n    if jobs:\n        cmd.extend(['--jobs', str(jobs)])\n\n    # Since we got here - we already have some  data != \"nothing\"\n    if (data == 'auto') or \\\n        (\n            (data == 'auto-if-wanted') and\n            ds_repo.get_preferred_content('wanted', target)\n        ):\n        lgr.debug(\"Invoking copy --auto\")\n        cmd.append('--auto')\n\n    if force not in ('all', 'checkdatapresent'):\n        # if we force, we do not trust local knowledge and do the checks\n        cmd.append('--fast')\n\n    lgr.debug(\"Push data from %s to '%s'\", ds, target)\n\n    # input has type=dataset, but now it is about files\n    res_kwargs.pop('type', None)\n\n    # A set and an OrderedDict is used to track files pointing to the\n    # same key.  The set could be dropped, using a single dictionary\n    # that has an entry for every seen key and a (likely empty) list\n    # of redundant files, but that would mean looping over potentially\n    # many keys to yield likely few if any notneeded results.\n    seen_keys = set()\n    repkey_paths = OrderedDict()\n\n    # produce final path list. use knowledge that annex command will\n    # run in the root of the dataset and compact paths to be relative\n    # to this location\n    # XXX must not be a SpooledTemporaryFile -- dunno why, but doesn't work\n    # otherwise\n    with TemporaryFile() as file_list:\n        nbytes = 0\n        for c in to_transfer:\n            key = c['key']\n            if key in seen_keys:\n                repkey_paths.setdefault(key, []).append(c['path'])\n            else:\n                file_list.write(\n                    bytes(Path(c['path']).relative_to(ds.pathobj)))\n                file_list.write(b'\\0')\n                nbytes += c['bytesize']\n                seen_keys.add(key)\n        lgr.debug('Counted %d bytes of annex data to transfer',\n                  nbytes)\n\n        # rewind stdin buffer\n        file_list.seek(0)\n\n        # and go\n        res = ds_repo._call_annex_records(\n            cmd,\n            stdin=file_list,\n            progress=True,\n            # tailor the progress protocol with the total number of files\n            # to be transferred\n            total_nbytes=nbytes)\n        for j in res:\n            yield annexjson2result(j, ds, type='file', **res_kwargs)\n\n    for annex_key, paths in repkey_paths.items():\n        for path in paths:\n            yield dict(\n                res_kwargs, action='copy', type='file', status='notneeded',\n                path=path, annexkey=annex_key,\n                message='Another file points to the same key')\n    return\n\n\ndef _get_corresponding_remote_state(repo, to):\n    since = None\n    # for managed branches we cannot assume a matching one at the remote end\n    # instead we target the corresponding branch\n    active_branch = repo.get_corresponding_branch() or repo.get_active_branch()\n\n    if to:\n        # XXX here we assume one to one mapping of names from local branches\n        # to the remote\n        since = '%s/%s' % (to, active_branch)\n    else:\n        # take tracking remote for the active branch\n        tracked_remote, tracked_refspec = repo.get_tracking_branch()\n        if tracked_remote:\n            if tracked_refspec.startswith('refs/heads/'):\n                tracked_refspec = tracked_refspec[len('refs/heads/'):]\n            #to = tracked_remote\n            since = '%s/%s' % (tracked_remote, tracked_refspec)\n    return since\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface for dataset (component) pushing\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom collections import OrderedDict\nimport logging\nimport re\nfrom tempfile import TemporaryFile\n\nfrom datalad.interface.base import (\n    Interface,\n    build_doc,\n)\nfrom datalad.interface.common_opts import (\n    jobs_opt,\n    recursion_limit,\n    recursion_flag,\n)\nfrom datalad.interface.utils import (\n    eval_results,\n    render_action_summary,\n)\nfrom datalad.interface.results import annexjson2result\nfrom datalad.log import log_progress\nfrom datalad.support.annexrepo import (\n    AnnexJsonProtocol,\n    AnnexRepo,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import (\n    EnsureStr,\n    EnsureNone,\n    EnsureChoice,\n)\nfrom datalad.support.exceptions import CommandError\nfrom datalad.utils import (\n    Path,\n    ensure_list,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    EnsureDataset,\n    datasetmethod,\n    require_dataset,\n    resolve_path,\n)\nfrom datalad.core.local.diff import diff_dataset\n\n\nlgr = logging.getLogger('datalad.core.distributed.push')\n\n\n@build_doc\nclass Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    custom_result_summary_renderer_pass_summary = True\n\n    @staticmethod\n    def custom_result_summary_renderer(results, action_summary):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]\n\n        render_action_summary(action_summary)\n\n\ndef _datasets_since_(dataset, since, paths, recursive, recursion_limit):\n    \"\"\"Generator\"\"\"\n    # rely on diff() reporting sequentially across datasets\n    cur_ds = None\n    ds_res = None\n    for res in diff_dataset(\n            dataset=dataset,\n            fr=since,\n            # we never touch unsaved content\n            to='HEAD',\n            constant_refs=False,\n            path=paths,\n            # we need to know what is around locally to be able\n            # to report something that should have been pushed\n            # but could not, because we don't have a copy.\n            # however, getting this info here is needlessly\n            # expensive, we will do it at the latest possible stage\n            # in _push_data()\n            annex=None,\n            recursive=recursive,\n            recursion_limit=recursion_limit,\n            # make it as fast as possible\n            eval_file_type=False,\n            # TODO?: expose order as an option for diff and push\n            # since in some cases breadth-first would be sufficient\n            # and result in \"taking action faster\"\n            reporting_order='bottom-up'\n    ):\n        if res.get('action', None) != 'diff':\n            # we don't care right now\n            continue\n        if res.get('status', None) != 'ok':\n            # we cannot handle this situation, report it in panic\n            raise RuntimeError(\n                'Cannot handle non-OK diff result: {}'.format(res))\n        parentds = res.get('parentds', None)\n        if not parentds:\n            raise RuntimeError(\n                'Cannot handle diff result without a parent dataset '\n                'property: {}'.format(res))\n        if res.get('type', None) == 'dataset':\n            # a subdataset record in another dataset\n            # this could be here, because\n            # - this dataset was explicitly requested by path\n            #   -> should get a dedicated dataset record -- even without recursion\n            # - a path within an existing subdataset was given\n            # - a path within an non-existing subdataset was given\n            #   locally or not)\n            #   -> it should be ignored, but should not cause the branch in the\n            #   superdataset not to be pushed, if this was the only change\n            p = Path(res['path'])\n            # was given as an explicit path argument\n            if any(arg == p for arg in paths) and \\\n                    not GitRepo.is_valid_repo(res['path']):\n                raise ValueError(\n                    'Cannot publish subdataset, not present: {}'.format(res['path']))\n\n        if parentds != cur_ds:\n            if ds_res:\n                # we switch to another dataset, yield this one so outside\n                # code can start processing immediately\n                yield (cur_ds, ds_res)\n            # clean start\n            ds_res = []\n            cur_ds = parentds\n        ds_res.append({\n            k: v for k, v in res.items()\n            if k in (\n                # let's keep 'state' in for now, it would make it possible\n                # to implement a \"sync\"-type push downstream that actually\n                # pulls 'deleted' files\n                'state',\n                # 'file' to copy-to, and subdataset records to possibly\n                # act on\n                'type',\n                # essential\n                'path')\n        })\n\n    # if we have something left to report, do it\n    # importantly do not test for ds_res, even if we had only seen subdataset\n    # records to be changes, we would still want to push the git branches\n    if cur_ds:\n        yield (cur_ds, ds_res)\n\n\ndef _push(dspath, content, target, data, force, jobs, res_kwargs, pbars,\n          got_path_arg=False):\n    force_git_push = force in ('all', 'gitpush')\n\n    # nothing recursive in here, we only need a repo to work with\n    ds = Dataset(dspath)\n    repo = ds.repo\n\n    res_kwargs.update(type='dataset', path=dspath)\n\n    # content will be unique for every push (even on the same dataset)\n    pbar_id = 'push-{}-{}'.format(target, id(content))\n    # register for final orderly take down\n    pbars[pbar_id] = ds\n    log_progress(\n        lgr.info, pbar_id,\n        'Determine push target',\n        unit=' Steps',\n        label='Push',\n        total=4,\n    )\n    # pristine input arg\n    _target = target\n    # verified or auto-detected\n    target = None\n    if not _target:\n        try:\n            # let Git figure out what needs doing\n            # we will reuse the result further down again, so nothing is wasted\n            wannabe_gitpush = repo.push(remote=None, git_options=['--dry-run'])\n            # we did not get an explicit push target, get it from Git\n            target = set(p.get('remote', None) for p in wannabe_gitpush)\n            # handle case where a pushinfo record did not have a 'remote'\n            # property -- should not happen, but be robust\n            target.discard(None)\n        except Exception as e:\n            lgr.debug(\n                'Dry-run push to determine default push target failed, '\n                'assume no configuration: %s', e)\n            target = set()\n        if not len(target):\n            yield dict(\n                res_kwargs,\n                status='impossible',\n                message='No push target given, and none could be '\n                        'auto-detected, please specify via --to',\n            )\n            return\n        elif len(target) > 1:\n            # dunno if this can ever happen, but if it does, report\n            # nicely\n            yield dict(\n                res_kwargs,\n                status='error',\n                message=(\n                    'No push target given, '\n                    'multiple candidates auto-detected: %s',\n                    list(target),\n                )\n            )\n            return\n        else:\n            # can only be a single one at this point\n            target = target.pop()\n\n    if not target:\n        if _target not in repo.get_remotes():\n            yield dict(\n                res_kwargs,\n                status='error',\n                message=(\n                    \"Unknown target sibling '%s'.\", _target))\n            return\n        target = _target\n\n    log_progress(\n        lgr.info, pbar_id, \"Push refspecs\",\n        label=\"Push to '{}'\".format(target), update=1, total=4)\n\n    # define config var name for potential publication dependencies\n    depvar = 'remote.{}.datalad-publish-depends'.format(target)\n    # list of remotes that are publication dependencies for the\n    # target remote\n    publish_depends = ensure_list(ds.config.get(depvar, []))\n    if publish_depends:\n        lgr.debug(\"Discovered publication dependencies for '%s': %s'\",\n                  target, publish_depends)\n\n    # cache repo type\n    is_annex_repo = isinstance(ds.repo, AnnexRepo)\n\n    # TODO prevent this when `target` is a special remote\n    # (possibly redo) a push attempt to figure out what needs pushing\n    # do this on the main target only, and apply the result to all\n    # dependencies\n    try:\n        if _target:\n            # only do it when an explicit target was given, otherwise\n            # we can reuse the result from the auto-probing above\n            wannabe_gitpush = repo.push(\n                remote=target,\n                git_options=['--dry-run'])\n    except Exception as e:\n        lgr.debug(\n            'Dry-run push to check push configuration failed, '\n            'assume no configuration: %s', e)\n        wannabe_gitpush = []\n    refspecs2push = [\n        # if an upstream branch is set, go with it\n        p['from_ref']\n        if ds.config.get(\n            # refs come in as refs/heads/<branchname>\n            # need to cut the prefix\n            'branch.{}.remote'.format(p['from_ref'][11:]),\n            None) == target and ds.config.get(\n                'branch.{}.merge'.format(p['from_ref'][11:]),\n                None)\n        # if not, define target refspec explicitly to avoid having to\n        # set an upstream branch, which would happen implicitly from\n        # a users POV, and may also be hard to decide when publication\n        # dependencies are present\n        else '{}:{}'.format(p['from_ref'], p['to_ref'])\n        for p in wannabe_gitpush\n        # TODO: what if a publication dependency doesn't have it yet\n        # should we not attempt to push, because the main target has it?\n        if 'uptodate' not in p['operations'] and (\n            # cannot think of a scenario where we would want to push a\n            # managed branch directly, instead of the corresponding branch\n            'refs/heads/adjusted' not in p['from_ref'])\n    ]\n    # TODO this is not right with managed branches\n    active_branch = repo.get_active_branch()\n    if active_branch and is_annex_repo:\n        # we could face a managed branch, in which case we need to\n        # determine the actual one and make sure it is sync'ed with the\n        # managed one, and push that one instead. following methods can\n        # be called unconditionally\n        repo.localsync(managed_only=True)\n        active_branch = repo.get_corresponding_branch(\n            active_branch) or active_branch\n\n    if not refspecs2push and not active_branch:\n        # nothing was set up for push, and we have no active branch\n        # this is a weird one, let's confess and stop here\n        # I don't think we need to support such a scenario\n        if not active_branch:\n            yield dict(\n                res_kwargs,\n                status='impossible',\n                message=\n                'There is no active branch, cannot determine remote '\n                'branch'\n            )\n            return\n\n    # make sure that we always push the active branch (the context for the\n    # potential path arguments) and the annex branch -- because we claim\n    # to know better than any git config\n    must_have_branches = [active_branch] if active_branch else []\n    if is_annex_repo:\n        must_have_branches.append('git-annex')\n    for branch in must_have_branches:\n        _append_branch_to_refspec_if_needed(ds, refspecs2push, branch)\n\n    # we know what to push and where, now dependency processing first\n    for r in publish_depends:\n        # simply make a call to this function again, all the same, but\n        # target is different\n        yield from _push(\n            dspath,\n            content,\n            # to this particular dependency\n            r,\n            data,\n            force,\n            jobs,\n            res_kwargs.copy(),\n            pbars,\n            got_path_arg=got_path_arg,\n        )\n\n    # and lastly the primary push target\n    target_is_git_remote = repo.config.get(\n        'remote.{}.url'.format(target), None) is not None\n\n    # git-annex data copy\n    #\n    if is_annex_repo:\n        if data != \"nothing\":\n            log_progress(\n                lgr.info, pbar_id, \"Transfer data\",\n                label=\"Transfer data to '{}'\".format(target), update=2, total=4)\n            yield from _push_data(\n                ds,\n                target,\n                content,\n                data,\n                force,\n                jobs,\n                res_kwargs.copy(),\n                got_path_arg=got_path_arg,\n            )\n        else:\n            lgr.debug(\"Data transfer to '%s' disabled by argument\", target)\n    else:\n        lgr.debug(\"No data transfer: %s is not a git annex repository\", repo)\n\n    if not target_is_git_remote:\n        # there is nothing that we need to push or sync with on the git-side\n        # of things with this remote\n        return\n\n    log_progress(\n        lgr.info, pbar_id, \"Update availability information\",\n        label=\"Update availability for '{}'\".format(target), update=3, total=4)\n\n    # TODO fetch is only needed if anything was actually transferred. Collect this\n    # info and make the following conditional on it\n\n    # after file transfer the remote might have different commits to\n    # the annex branch. They have to be merged locally, otherwise a\n    # push of it further down will fail\n    try:\n        # fetch remote, let annex sync them locally, so that the push\n        # later on works.\n        # We have to fetch via the push url (if there is any),\n        # not a pull url.\n        # The latter might be dumb and without the execution of a\n        # post-update hook we might not be able to retrieve the\n        # server-side git-annex branch updates (and git-annex does\n        # not trigger the hook on copy), but we know we have\n        # full access via the push url -- we have just used it to copy.\n        lgr.debug(\"Fetch 'git-annex' branch updates from '%s'\", target)\n        fetch_cmd = ['fetch', target, 'git-annex']\n        pushurl = repo.config.get(\n            'remote.{}.pushurl'.format(target), None)\n        if pushurl:\n            # for some reason overwriting remote.{target}.url\n            # does not have any effect...\n            fetch_cmd = [\n                '-c',\n                'url.{}.insteadof={}'.format(\n                    pushurl,\n                    repo.config.get(\n                        'remote.{}.url'.format(target), None)\n                )\n            ] + fetch_cmd\n            lgr.debug(\n                \"Sync local annex branch from pushurl after remote \"\n                'availability update.')\n        repo.call_git(fetch_cmd)\n        # If no CommandError was raised, it means that remote has git-annex\n        # but local repo might not be an annex yet. Since there is nothing to \"sync\"\n        # from us, we just skip localsync without mutating repo into an AnnexRepo\n        if is_annex_repo:\n            repo.localsync(target)\n    except CommandError as e:\n        # it is OK if the remote doesn't have a git-annex branch yet\n        # (e.g. fresh repo)\n        # TODO is this possible? we just copied? Maybe check if anything\n        # was actually copied?\n        if \"fatal: couldn't find remote ref git-annex\" not in e.stderr.lower():\n            raise\n        lgr.debug('Remote does not have a git-annex branch: %s', e)\n\n    if not refspecs2push:\n        lgr.debug('No refspecs found that need to be pushed')\n        return\n\n    # and push all relevant branches, plus the git-annex branch to announce\n    # local availability info too\n    yield from _push_refspecs(\n        repo,\n        target,\n        refspecs2push,\n        force_git_push,\n        res_kwargs.copy(),\n    )\n\n\ndef _append_branch_to_refspec_if_needed(ds, refspecs, branch):\n    # try to anticipate any flavor of an idea of a branch ending up in a refspec\n    looks_like_that_branch = re.compile(\n        r'((^|.*:)refs/heads/|.*:|^){}$'.format(branch))\n    if all(not looks_like_that_branch.match(r) for r in refspecs):\n        refspecs.append(\n            branch\n            if ds.config.get('branch.{}.merge'.format(branch), None)\n            else '{branch}:{branch}'.format(branch=branch)\n        )\n\n\ndef _push_refspecs(repo, target, refspecs, force_git_push, res_kwargs):\n    push_res = repo.push(\n        remote=target,\n        refspec=refspecs,\n        git_options=['--force'] if force_git_push else None,\n    )\n    # TODO maybe compress into a single message whenever everything is\n    # OK?\n    for pr in push_res:\n        ops = pr['operations']\n        status = (\n            'error'\n            if any(o in ops for o in (\n                'error', 'no-match', 'rejected', 'remote-rejected',\n                'remote-failure'))\n            else 'notneeded'\n            if 'uptodate' in pr['operations']\n            else 'ok'\n            if any(o in ops for o in (\n                'new-tag', 'new-branch', 'forced-update', 'fast-forward'))\n            # no really a good fit, but we have tested all relevant\n            # operations above, so in some sense this condition should be\n            # impossible to achieve\n            else 'impossible'\n        )\n        refspec = '{}:{}'.format(pr['from_ref'], pr['to_ref'])\n        yield dict(\n            res_kwargs,\n            status=status,\n            target=pr['remote'],\n            refspec=refspec,\n            operations=ops,\n            hints=pr.get('hints', None),\n            # seems like a good idea to pass on Git's native message\n            # TODO maybe implement a dedicated result renderer, instead\n            # of duplicating information only so that the default one\n            # can make sense at all\n            message='{}->{}:{} {}'.format(\n                pr['from_ref'],\n                pr['remote'],\n                pr['to_ref'],\n                pr['note']),\n        )\n\n\ndef _push_data(ds, target, content, data, force, jobs, res_kwargs,\n               got_path_arg=False):\n    if ds.config.getbool('remote.{}'.format(target), 'annex-ignore', False):\n        lgr.debug(\n            \"Target '%s' is set to annex-ignore, exclude from data-push.\",\n            target,\n        )\n        return\n\n    ds_repo = ds.repo\n\n    res_kwargs['target'] = target\n    if not ds.config.get('.'.join(('remote', target, 'annex-uuid')), None):\n        # this remote either isn't an annex,\n        # or hasn't been properly initialized\n        # given that there was no annex-ignore, let's try to init it\n        # see https://github.com/datalad/datalad/issues/5143 for the story\n        ds_repo.localsync(target)\n\n        if not ds.config.get('.'.join(('remote', target, 'annex-uuid')), None):\n            # still nothing\n            # rather than barfing tons of messages for each file, do one\n            # for the entire dataset\n            yield dict(\n                res_kwargs,\n                action='copy',\n                status='impossible'\n                if force in ('all', 'checkdatapresent')\n                else 'notneeded',\n                message=(\n                    \"Target '%s' does not appear to be an annex remote\",\n                    target)\n            )\n            return\n\n    # it really looks like we will transfer files, get info on what annex\n    # has in store\n    # paths must be recoded to a dataset REPO root (in case of a symlinked\n    # location\n    annex_info_init = \\\n        {ds_repo.pathobj / Path(c['path']).relative_to(ds.pathobj): c\n         for c in content} if ds.pathobj != ds_repo.pathobj else \\\n        {Path(c['path']): c for c in content}\n    content = ds.repo.get_content_annexinfo(\n        # paths are taken from `annex_info_init`\n        paths=None,\n        init=annex_info_init,\n        ref='HEAD',\n        # this is an expensive operation that is only needed\n        # to perform a warning below, and for more accurate\n        # progress reporting (exclude unavailable content).\n        # limit to cases with explicit paths provided\n        eval_availability=True if got_path_arg else False,\n    )\n    # figure out which of the reported content (after evaluating\n    # `since` and `path` arguments needs transport\n    to_transfer = [\n        c\n        for c in content.values()\n        # by force\n        if ((force in ('all', 'checkdatapresent') or\n             # or by modification report\n             c.get('state', None) not in ('clean', 'deleted'))\n            # only consider annex'ed files\n            and 'key' in c\n        )\n    ]\n    if got_path_arg:\n        for c in [c for c in to_transfer if not c.get('has_content', False)]:\n            yield dict(\n                res_kwargs,\n                type=c['type'],\n                path=c['path'],\n                action='copy',\n                status='impossible',\n                message='Slated for transport, but no content present',\n            )\n\n    cmd = ['copy', '--batch', '-z', '--to', target]\n\n    if jobs:\n        cmd.extend(['--jobs', str(jobs)])\n\n    # Since we got here - we already have some  data != \"nothing\"\n    if (data == 'auto') or \\\n        (\n            (data == 'auto-if-wanted') and\n            ds_repo.get_preferred_content('wanted', target)\n        ):\n        lgr.debug(\"Invoking copy --auto\")\n        cmd.append('--auto')\n\n    if force not in ('all', 'checkdatapresent'):\n        # if we force, we do not trust local knowledge and do the checks\n        cmd.append('--fast')\n\n    lgr.debug(\"Push data from %s to '%s'\", ds, target)\n\n    # input has type=dataset, but now it is about files\n    res_kwargs.pop('type', None)\n\n    # A set and an OrderedDict is used to track files pointing to the\n    # same key.  The set could be dropped, using a single dictionary\n    # that has an entry for every seen key and a (likely empty) list\n    # of redundant files, but that would mean looping over potentially\n    # many keys to yield likely few if any notneeded results.\n    seen_keys = set()\n    repkey_paths = OrderedDict()\n\n    # produce final path list. use knowledge that annex command will\n    # run in the root of the dataset and compact paths to be relative\n    # to this location\n    # XXX must not be a SpooledTemporaryFile -- dunno why, but doesn't work\n    # otherwise\n    with TemporaryFile() as file_list:\n        nbytes = 0\n        for c in to_transfer:\n            key = c['key']\n            if key in seen_keys:\n                repkey_paths.setdefault(key, []).append(c['path'])\n            else:\n                file_list.write(\n                    bytes(Path(c['path']).relative_to(ds.pathobj)))\n                file_list.write(b'\\0')\n                nbytes += c['bytesize']\n                seen_keys.add(key)\n        lgr.debug('Counted %d bytes of annex data to transfer',\n                  nbytes)\n\n        # rewind stdin buffer\n        file_list.seek(0)\n\n        # and go\n        res = ds_repo._call_annex_records(\n            cmd,\n            stdin=file_list,\n            progress=True,\n            # tailor the progress protocol with the total number of files\n            # to be transferred\n            total_nbytes=nbytes)\n        for j in res:\n            yield annexjson2result(j, ds, type='file', **res_kwargs)\n\n    for annex_key, paths in repkey_paths.items():\n        for path in paths:\n            yield dict(\n                res_kwargs, action='copy', type='file', status='notneeded',\n                path=path, annexkey=annex_key,\n                message='Another file points to the same key')\n    return\n\n\ndef _get_corresponding_remote_state(repo, to):\n    since = None\n    # for managed branches we cannot assume a matching one at the remote end\n    # instead we target the corresponding branch\n    active_branch = repo.get_corresponding_branch() or repo.get_active_branch()\n\n    if to:\n        # XXX here we assume one to one mapping of names from local branches\n        # to the remote\n        since = '%s/%s' % (to, active_branch)\n    else:\n        # take tracking remote for the active branch\n        tracked_remote, tracked_refspec = repo.get_tracking_branch()\n        if tracked_remote:\n            if tracked_refspec.startswith('refs/heads/'):\n                tracked_refspec = tracked_refspec[len('refs/heads/'):]\n            #to = tracked_remote\n            since = '%s/%s' % (tracked_remote, tracked_refspec)\n    return since\n",
            "header": {
                "file": {
                    "path": "datalad/core/distributed/push.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "147fe7104",
                "new_commit_hash": "5af9c6e67",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/core/distributed/push.py"
            },
            "plus_file": {
                "path": "b/datalad/core/distributed/push.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 28,
                            "length": 6
                        },
                        "new_range": {
                            "start": 28,
                            "length": 7
                        },
                        "section": "from datalad.interface.common_opts import ("
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.interface.utils import (",
                                "type": "context"
                            },
                            {
                                "content": "    eval_results,",
                                "type": "context"
                            },
                            {
                                "content": "    render_action_summary,",
                                "type": "added"
                            },
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.interface.results import annexjson2result",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.log import log_progress",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/core/distributed/push.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.interface.utils import (\n    eval_results,\n    render_action_summary,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 29,
                            "end_lineno": 32,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 273,
                            "length": 8
                        },
                        "new_range": {
                            "start": 274,
                            "length": 10
                        },
                        "section": "class Push(Interface):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                path=ds.path,",
                                "type": "context"
                            },
                            {
                                "content": "            )",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    custom_result_summary_renderer_pass_summary = True",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    @staticmethod",
                                "type": "context"
                            },
                            {
                                "content": "    def custom_result_summary_renderer(results):  # pragma: more cover",
                                "type": "deleted"
                            },
                            {
                                "content": "    def custom_result_summary_renderer(results, action_summary):  # pragma: more cover",
                                "type": "added"
                            },
                            {
                                "content": "        # report on any hints at the end",
                                "type": "context"
                            },
                            {
                                "content": "        # get all unique hints",
                                "type": "context"
                            },
                            {
                                "content": "        hints = set([r.get('hints', None) for r in results])",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/core/distributed/push.py",
                            "type": "method",
                            "name": "Push.custom_result_summary_renderer",
                            "content": "class Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    @staticmethod\n    def custom_result_summary_renderer(results):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 277,
                            "end_lineno": 291,
                            "parent": {
                                "file_name": "datalad/core/distributed/push.py",
                                "type": "class",
                                "name": "Push",
                                "content": "class Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    @staticmethod\n    def custom_result_summary_renderer(results):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 65,
                                "end_lineno": 291,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/core/distributed/push.py",
                            "type": "class",
                            "name": "Push",
                            "content": "class Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    @staticmethod\n    def custom_result_summary_renderer(results):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 65,
                            "end_lineno": 291,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 290,
                            "length": 6
                        },
                        "new_range": {
                            "start": 293,
                            "length": 8
                        },
                        "section": "class Push(Interface):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))",
                                "type": "context"
                            },
                            {
                                "content": "                for id, hint in enumerate(hints)]",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        render_action_summary(action_summary)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def _datasets_since_(dataset, since, paths, recursive, recursion_limit):",
                                "type": "context"
                            },
                            {
                                "content": "    \"\"\"Generator\"\"\"",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/core/distributed/push.py",
                            "type": "method",
                            "name": "Push.custom_result_summary_renderer",
                            "content": "class Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    custom_result_summary_renderer_pass_summary = True\n\n    @staticmethod\n    def custom_result_summary_renderer(results, action_summary):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]\n\n        render_action_summary(action_summary)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 280,
                            "end_lineno": 296,
                            "parent": {
                                "file_name": "datalad/core/distributed/push.py",
                                "type": "class",
                                "name": "Push",
                                "content": "class Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    custom_result_summary_renderer_pass_summary = True\n\n    @staticmethod\n    def custom_result_summary_renderer(results, action_summary):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]\n\n        render_action_summary(action_summary)",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 66,
                                "end_lineno": 296,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/core/distributed/push.py",
                            "type": "class",
                            "name": "Push",
                            "content": "class Push(Interface):\n    \"\"\"Push a dataset to a known :term:`sibling`.\n\n    This makes a saved state of a dataset available to a sibling or special\n    remote data store of a dataset. Any target sibling must already exist and\n    be known to the dataset.\n\n    || REFLOW >>\n    By default, all files tracked in the last saved state (of the current\n    branch) will be copied to the target location. Optionally, it is possible\n    to limit a push to changes relative to a particular point in the version\n    history of a dataset (e.g. a release tag) using the\n    [CMD: --since CMD][PY: since PY] option in conjunction with the\n    specification of a reference dataset. In recursive mode subdatasets will also be\n    evaluated, and only those subdatasets are pushed where a change was\n    recorded that is reflected in the current state of the top-level reference\n    dataset.\n    << REFLOW ||\n\n    .. note::\n      Power-user info: This command uses :command:`git push`, and :command:`git\n      annex copy` to push a dataset. Publication targets are either configured\n      remote Git repositories, or git-annex special remotes (if they support\n      data upload).\n    \"\"\"\n\n    # TODO add examples\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to push\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        to=Parameter(\n            args=(\"--to\",),\n            metavar='SIBLING',\n            doc=\"\"\"name of the target sibling. If no name is given an attempt is\n            made to identify the target based on the dataset's configuration\n            (i.e. a configured tracking branch, or a single sibling that is\n            configured for push)\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        since=Parameter(\n            args=(\"--since\",),\n            constraints=EnsureStr() | EnsureNone(),\n            doc=\"\"\"specifies commit-ish (tag, shasum, etc.) from which to look for\n            changes to decide whether pushing is necessary.\n            If '^' is given, the last state of the current branch at the sibling\n            is taken as a starting point.\"\"\"),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            doc=\"\"\"path to contrain a push to. If given, only\n            data or changes for those paths are considered for a push.\"\"\",\n            nargs='*',\n            constraints=EnsureStr() | EnsureNone()),\n        data=Parameter(\n            args=(\"--data\",),\n            doc=\"\"\"what to do with (annex'ed) data. 'anything' would cause\n            transfer of all annexed content, 'nothing' would avoid call to\n            `git annex copy` altogether. 'auto' would use 'git annex copy' with\n            '--auto' thus transferring only data which would satisfy \"wanted\"\n            or \"numcopies\" settings for the remote (thus \"nothing\" otherwise).\n            'auto-if-wanted' would enable '--auto' mode only if there is a \n            \"wanted\" setting for the remote, and transfer 'anything' otherwise.\n            \"\"\",\n            constraints=EnsureChoice(\n                'anything', 'nothing', 'auto', 'auto-if-wanted')),\n        force=Parameter(\n            # multi-mode option https://github.com/datalad/datalad/issues/3414\n            args=(\"-f\", \"--force\",),\n            doc=\"\"\"force particular operations, possibly overruling safety\n            protections or optimizations: use --force with git-push ('gitpush');\n            do not use --fast with git-annex copy ('checkdatapresent');\n            combine all force modes ('all').\"\"\",\n            constraints=EnsureChoice(\n                'all', 'gitpush', 'checkdatapresent', None)),\n        recursive=recursion_flag,\n        recursion_limit=recursion_limit,\n        jobs=jobs_opt,\n    )\n\n    # Desired features:\n    # - let Git do it's thing (push multiple configured refs without the need\n    #                          to specific anything on the command line\n    #   - compilication: we need publication dependencies (i.e. publish what\n    #     would be published by Git to a different remote first, hence we\n    #     cannot simply watch Git do it, and later act on it.)\n    #   - https://github.com/datalad/datalad/issues/1284\n    #   - https://github.com/datalad/datalad/issues/4006\n    # - make differences between remotes and various types of special remotes\n    #   opaque\n    #   - https://github.com/datalad/datalad/issues/3127\n    # - informative and comprehensive (error) reporting\n    #   - https://github.com/datalad/datalad/issues/2000\n    #   - https://github.com/datalad/datalad/issues/1682\n    #   - https://github.com/datalad/datalad/issues/2029\n    #   - https://github.com/datalad/datalad/issues/2855\n    #   - https://github.com/datalad/datalad/issues/3412\n    #   - https://github.com/datalad/datalad/issues/3424\n    # - ensure robust behavior in multi-lateral push scenarios (updating\n    #   a dataset that was updated by a 3rd-party after the last known\n    #   fetched change\n    #   - https://github.com/datalad/datalad/issues/2636\n    # - should NOT mimic `publish` and that it mixes `create-sibling` and\n    #   `push` into a single operation. This would fold the complexity\n    #   of all possible ways a local dataset hierarchy could possibly\n    #   connected to remote ends into this command. It would be lost battle\n    #   from the start.\n    #   - not tackle: https://github.com/datalad/datalad/issues/2186\n    # - maintain standard setup, and not reflect procedural aspects\n    #   onto the resulting outcomes\n    #   - https://github.com/datalad/datalad/issues/2001\n    # - do a straight push, nothing like 'sync'. If a remote has something that\n    #   needs merging first, fail and let users update. Any diff we are missing\n    #   locally can impact decision making via --since and friends.\n\n    @staticmethod\n    @datasetmethod(name='push')\n    @eval_results\n    def __call__(\n            path=None,\n            dataset=None,\n            to=None,\n            since=None,\n            data='auto-if-wanted',\n            force=None,\n            recursive=False,\n            recursion_limit=None,\n            jobs=None):\n        # push uses '^' to annotate the previous pushed committish, and None for default\n        # behavior. '' was/is (to be deprecated) used in `publish`. Alert user about the mistake\n        if since == '':\n            raise ValueError(\"'since' should point to commitish or use '^'.\")\n        # we resolve here, because we need to perform inspection on what was given\n        # as an input argument further down\n        paths = [resolve_path(p, dataset) for p in ensure_list(path)]\n\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='pushing')\n        ds_repo = ds.repo\n\n        res_kwargs = dict(\n            action='publish',\n            refds=ds.path,\n            logger=lgr,\n        )\n\n        get_remote_kwargs = {'exclude_special_remotes': False} \\\n            if isinstance(ds_repo, AnnexRepo) else {}\n        if to and to not in ds_repo.get_remotes(**get_remote_kwargs):\n            # get again for proper error:\n            sr = ds_repo.get_remotes(**get_remote_kwargs)\n            # yield an error result instead of raising a ValueError,\n            # to enable the use case of pushing to a target that\n            # a superdataset doesn't know, but some subdatasets to\n            # (in combination with '--on-failure ignore')\n            yield dict(\n                res_kwargs,\n                status='error',\n                path=ds.path,\n                message=\"Unknown push target '{}'. {}\".format(\n                    to,\n                    'Known targets: {}.'.format(', '.join(repr(s) for s in sr))\n                    if sr\n                    else 'No targets configured in dataset.'))\n            return\n\n        if since == '^':\n            # figure out state of remote branch and set `since`\n            since = _get_corresponding_remote_state(ds_repo, to)\n            if not since:\n                lgr.info(\n                    \"No tracked remote for active branch, \"\n                    \"detection of last pushed state not in effect.\")\n        elif since:\n            # will blow with ValueError if unusable\n            ds_repo.get_hexsha(since)\n\n\n        # obtain a generator for information on the datasets to process\n        # idea is to turn the `paths` argument into per-dataset\n        # content listings that can be acted upon\n        ds_spec = _datasets_since_(\n            # important to pass unchanged dataset arg\n            dataset,\n            since,\n            paths,\n            recursive,\n            recursion_limit)\n\n        # instead of a loop, this could all be done in parallel\n        matched_anything = False\n        for dspath, dsrecords in ds_spec:\n            matched_anything = True\n            lgr.debug('Attempt push of Dataset at %s', dspath)\n            pbars = {}\n            yield from _push(\n                dspath, dsrecords, to, data, force, jobs, res_kwargs.copy(), pbars,\n                got_path_arg=True if path else False)\n            # take down progress bars for this dataset\n            for i, ds in pbars.items():\n                log_progress(lgr.info, i, 'Finished push of %s', ds)\n        if not matched_anything:\n            yield dict(\n                res_kwargs,\n                status='notneeded',\n                message='Given constraints did not match any changes to publish',\n                type='dataset',\n                path=ds.path,\n            )\n\n    custom_result_summary_renderer_pass_summary = True\n\n    @staticmethod\n    def custom_result_summary_renderer(results, action_summary):  # pragma: more cover\n        # report on any hints at the end\n        # get all unique hints\n        hints = set([r.get('hints', None) for r in results])\n        hints = [hint for hint in hints if hint is not None]\n        if hints:\n            from datalad.ui import ui\n            from datalad.support import ansi_colors\n            intro = ansi_colors.color_word(\n                \"Potential hints to solve encountered errors: \",\n                ansi_colors.YELLOW)\n            ui.message(intro)\n            [ui.message(\"{}: {}\".format(\n                ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))\n                for id, hint in enumerate(hints)]\n\n        render_action_summary(action_summary)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 66,
                            "end_lineno": 296,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# -*- coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test push\n\n\"\"\"\n\nimport os\nimport logging\n\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.support.exceptions import (\n    IncompleteResultsError,\n    InsufficientArgumentsError,\n)\nfrom datalad.tests.utils import (\n    assert_false,\n    assert_in,\n    assert_in_results,\n    assert_not_in,\n    assert_not_in_results,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_status,\n    DEFAULT_BRANCH,\n    DEFAULT_REMOTE,\n    eq_,\n    known_failure_githubci_osx,\n    known_failure_githubci_win,\n    neq_,\n    ok_,\n    ok_file_has_content,\n    serve_path_via_http,\n    skip_if_adjusted_branch,\n    skip_if_on_windows,\n    skip_ssh,\n    slow,\n    swallow_logs,\n    with_tempfile,\n    with_tree,\n    SkipTest,\n)\nfrom datalad.utils import (\n    Path,\n    chpwd,\n    path_startswith,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.core.distributed.clone import Clone\nfrom datalad.core.distributed.push import Push\nfrom datalad.support.network import get_local_file_url\n\nDEFAULT_REFSPEC = \"refs/heads/{0}:refs/heads/{0}\".format(DEFAULT_BRANCH)\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_call(origin, tdir):\n    ds = Dataset(origin).create()\n    # no target\n    assert_status('impossible', ds.push(on_failure='ignore'))\n    # no dataset\n    with chpwd(tdir):\n        assert_raises(InsufficientArgumentsError, Push.__call__)\n    # dataset, but outside path\n    assert_raises(IncompleteResultsError, ds.push, path=tdir)\n    # given a path constraint that doesn't match anything, will cause\n    # nothing to be done\n    assert_status('notneeded', ds.push(path=ds.pathobj / 'nothere'))\n\n    # unavailable subdataset\n    dummy_sub = ds.create('sub')\n    dummy_sub.uninstall()\n    assert_in('sub', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    # now an explicit call to publish the unavailable subdataset\n    assert_raises(ValueError, ds.push, 'sub')\n\n    target = mk_push_target(ds, 'target', tdir, annex=True)\n    # revision that doesn't exist\n    assert_raises(\n        ValueError,\n        ds.push, to='target', since='09320957509720437523')\n\n    # If a publish() user accidentally passes since='', which push() spells as\n    # since='^', the call is aborted.\n    assert_raises(\n        ValueError,\n        ds.push, to='target', since='')\n\n\ndef mk_push_target(ds, name, path, annex=True, bare=True):\n    # life could be simple, but nothing is simple on windows\n    #src.create_sibling(dst_path, name='target')\n    if annex:\n        if bare:\n            target = GitRepo(path=path, bare=True, create=True)\n            # cannot use call_annex()\n            target.call_git(['annex', 'init'])\n        else:\n            target = AnnexRepo(path, init=True, create=True)\n            if not target.is_managed_branch():\n                # for managed branches we need more fireworks->below\n                target.config.set(\n                    'receive.denyCurrentBranch', 'updateInstead',\n                    where='local')\n    else:\n        target = GitRepo(path=path, bare=bare, create=True)\n    ds.siblings('add', name=name, url=path, result_renderer=None)\n    if annex and not bare and target.is_managed_branch():\n        # maximum complication\n        # the target repo already has a commit that is unrelated\n        # to the source repo, because it has built a reference\n        # commit for the managed branch.\n        # the only sane approach is to let git-annex establish a shared\n        # history\n        ds.repo.call_annex(['sync'])\n        ds.repo.call_annex(['sync', '--cleanup'])\n    return target\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef check_push(annex, src_path, dst_path):\n    # prepare src\n    src = Dataset(src_path).create(annex=annex)\n    src_repo = src.repo\n    # push should not add branches to the local dataset\n    orig_branches = src_repo.get_branches()\n    assert_not_in('synced/' + DEFAULT_BRANCH, orig_branches)\n\n    res = src.push(on_failure='ignore')\n    assert_result_count(res, 1)\n    assert_in_results(\n        res, status='impossible',\n        message='No push target given, and none could be auto-detected, '\n        'please specify via --to')\n    eq_(orig_branches, src_repo.get_branches())\n    # target sibling\n    target = mk_push_target(src, 'target', dst_path, annex=annex)\n    eq_(orig_branches, src_repo.get_branches())\n\n    res = src.push(to=\"target\")\n    eq_(orig_branches, src_repo.get_branches())\n    assert_result_count(res, 2 if annex else 1)\n    assert_in_results(\n        res,\n        action='publish', status='ok', target='target',\n        refspec=DEFAULT_REFSPEC,\n        operations=['new-branch'])\n\n    assert_repo_status(src_repo, annex=annex)\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # configure a default merge/upstream target\n    src.config.set('branch.{}.remote'.format(DEFAULT_BRANCH),\n                   'target', where='local')\n    src.config.set('branch.{}.merge'.format(DEFAULT_BRANCH),\n                   DEFAULT_BRANCH, where='local')\n\n    # don't fail when doing it again, no explicit target specification\n    # needed anymore\n    res = src.push()\n    eq_(orig_branches, src_repo.get_branches())\n    # and nothing is pushed\n    assert_status('notneeded', res)\n\n    assert_repo_status(src_repo, annex=annex)\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # some modification:\n    (src.pathobj / 'test_mod_file').write_text(\"Some additional stuff.\")\n    src.save(to_git=True, message=\"Modified.\")\n    (src.pathobj / 'test_mod_annex_file').write_text(\"Heavy stuff.\")\n    src.save(to_git=not annex, message=\"Modified again.\")\n    assert_repo_status(src_repo, annex=annex)\n\n    # we could say since='HEAD~2' to make things fast, or we are lazy\n    # and say since='^' to indicate the state of the tracking remote\n    # which is the same, because we made to commits since the last push.\n    res = src.push(to='target', since=\"^\", jobs=2)\n    assert_in_results(\n        res,\n        action='publish', status='ok', target='target',\n        refspec=DEFAULT_REFSPEC,\n        # we get to see what happened\n        operations=['fast-forward'])\n    if annex:\n        # we got to see the copy result for the annexed files\n        assert_in_results(\n            res,\n            action='copy',\n            status='ok',\n            path=str(src.pathobj / 'test_mod_annex_file'))\n        # we published, so we can drop and reobtain\n        ok_(src_repo.file_has_content('test_mod_annex_file'))\n        src_repo.drop('test_mod_annex_file')\n        ok_(not src_repo.file_has_content('test_mod_annex_file'))\n        src_repo.get('test_mod_annex_file')\n        ok_(src_repo.file_has_content('test_mod_annex_file'))\n        ok_file_has_content(\n            src_repo.pathobj / 'test_mod_annex_file',\n            'Heavy stuff.')\n\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n    if not (annex and src_repo.is_managed_branch()):\n        # the following doesn't make sense in managed branches, because\n        # a commit that could be amended is no longer the last commit\n        # of a branch after a sync has happened (which did happen\n        # during the last push above\n\n        # amend and change commit msg in order to test for force push:\n        src_repo.commit(\"amended\", options=['--amend'])\n        # push should be rejected (non-fast-forward):\n        res = src.push(to='target', since='HEAD~2', on_failure='ignore')\n        # fails before even touching the annex branch\n        assert_in_results(\n            res,\n            action='publish', status='error', target='target',\n            refspec=DEFAULT_REFSPEC,\n            operations=['rejected', 'error'])\n        # push with force=True works:\n        res = src.push(to='target', since='HEAD~2', force='gitpush')\n        assert_in_results(\n            res,\n            action='publish', status='ok', target='target',\n            refspec=DEFAULT_REFSPEC,\n            operations=['forced-update'])\n        eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n            list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # we do not have more branches than we had in the beginning\n    # in particular no 'synced/<default branch>'\n    eq_(orig_branches, src_repo.get_branches())\n\n\ndef test_push():\n    yield check_push, False\n    yield check_push, True\n\n\ndef check_datasets_order(res, order='bottom-up'):\n    \"\"\"Check that all type=dataset records not violating the expected order\n\n    it is somewhat weak test, i.e. records could be produced so we\n    do not detect that order is violated, e.g. a/b c/d would satisfy\n    either although they might be neither depth nor breadth wise.  But\n    this test would allow to catch obvious violations like a, a/b, a\n    \"\"\"\n    prev = None\n    for r in res:\n        if r.get('type') != 'dataset':\n            continue\n        if prev and r['path'] != prev:\n            if order == 'bottom-up':\n                assert_false(path_startswith(r['path'], prev))\n            elif order == 'top-down':\n                assert_false(path_startswith(prev, r['path']))\n            else:\n                raise ValueError(order)\n        prev = r['path']\n\n\n@slow  # 33sec on Yarik's laptop\n@with_tempfile\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True, suffix='sub')\n@with_tempfile(mkdir=True, suffix='subnoannex')\n@with_tempfile(mkdir=True, suffix='subsub')\ndef test_push_recursive(\n        origin_path, src_path, dst_top, dst_sub, dst_subnoannex, dst_subsub):\n    # dataset with two submodules and one subsubmodule\n    origin = Dataset(origin_path).create()\n    origin_subm1 = origin.create('sub m')\n    origin_subm1.create('subsub m')\n    origin.create('subm noannex', annex=False)\n    origin.save()\n    assert_repo_status(origin.path)\n    # prepare src as a fresh clone with all subdatasets checkout out recursively\n    # running on a clone should make the test scenario more different than\n    # test_push(), even for the pieces that should be identical\n    top = Clone.__call__(source=origin.path, path=src_path)\n    subs = top.get('.', recursive=True, get_data=False, result_xfm='datasets')\n    # order for '.' should not be relied upon, so sort by path\n    sub, subsub, subnoannex = sorted(subs, key=lambda ds: ds.path)\n\n    target_top = mk_push_target(top, 'target', dst_top, annex=True)\n    # subdatasets have no remote yet, so recursive publishing should fail:\n    res = top.push(to=\"target\", recursive=True, on_failure='ignore')\n    check_datasets_order(res)\n    assert_in_results(\n        res, path=top.path, type='dataset',\n        refspec=DEFAULT_REFSPEC,\n        operations=['new-branch'], action='publish', status='ok',\n        target='target')\n    for d in (sub, subsub, subnoannex):\n        assert_in_results(\n            res, status='error', type='dataset', path=d.path,\n            message=(\"Unknown target sibling '%s'.\",\n                     'target'))\n    # now fix that and set up targets for the submodules\n    target_sub = mk_push_target(sub, 'target', dst_sub, annex=True)\n    target_subnoannex = mk_push_target(\n        subnoannex, 'target', dst_subnoannex, annex=False)\n    target_subsub = mk_push_target(subsub, 'target', dst_subsub, annex=True)\n\n    # and same push call as above\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    # topds skipped\n    assert_in_results(\n        res, path=top.path, type='dataset',\n        action='publish', status='notneeded', target='target')\n    # the rest pushed\n    for d in (sub, subsub, subnoannex):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # all corresponding branches match across all datasets\n    for s, d in zip((top, sub, subnoannex, subsub),\n                    (target_top, target_sub, target_subnoannex,\n                     target_subsub)):\n        eq_(list(s.repo.get_branch_commits_(DEFAULT_BRANCH)),\n            list(d.get_branch_commits_(DEFAULT_BRANCH)))\n        if s != subnoannex:\n            eq_(list(s.repo.get_branch_commits_(\"git-annex\")),\n                list(d.get_branch_commits_(\"git-annex\")))\n\n    # rerun should not result in further pushes of the default branch\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    assert_not_in_results(\n        res, status='ok', refspec=DEFAULT_REFSPEC)\n    assert_in_results(\n        res, status='notneeded', refspec=DEFAULT_REFSPEC)\n\n    # now annex a file in subsub\n    test_copy_file = subsub.pathobj / 'test_mod_annex_file'\n    test_copy_file.write_text(\"Heavy stuff.\")\n    # save all the way up\n    assert_status(\n        ('ok', 'notneeded'),\n        top.save(message='subsub got something', recursive=True))\n    assert_repo_status(top.path)\n    # publish straight up, should be smart by default\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    # we see 3 out of 4 datasets pushed (sub noannex was left unchanged)\n    for d in (top, sub, subsub):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # file content copied too\n    assert_in_results(\n        res,\n        action='copy',\n        status='ok',\n        path=str(test_copy_file))\n    # verify it is accessible, drop and bring back\n    assert_status('ok', top.drop(str(test_copy_file)))\n    ok_(not subsub.repo.file_has_content('test_mod_annex_file'))\n    top.get(test_copy_file)\n    ok_file_has_content(test_copy_file, 'Heavy stuff.')\n\n    # make two modification\n    (sub.pathobj / 'test_mod_annex_file').write_text('annex')\n    (subnoannex.pathobj / 'test_mod_file').write_text('git')\n    # save separately\n    top.save(sub.pathobj, message='annexadd', recursive=True)\n    top.save(subnoannex.pathobj, message='gitadd', recursive=True)\n    # now only publish the latter one\n    res = top.push(to=\"target\", since=DEFAULT_BRANCH + '~1', recursive=True)\n    # nothing copied, no reports on the other modification\n    assert_not_in_results(res, action='copy')\n    assert_not_in_results(res, path=sub.path)\n    for d in (top, subnoannex):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # an unconditional push should now pick up the remaining changes\n    res = top.push(to=\"target\", recursive=True)\n    assert_in_results(\n        res,\n        action='copy',\n        status='ok',\n        path=str(sub.pathobj / 'test_mod_annex_file'))\n    assert_in_results(\n        res, status='ok', type='dataset', path=sub.path,\n        refspec=DEFAULT_REFSPEC)\n    for d in (top, subnoannex, subsub):\n        assert_in_results(\n            res, status='notneeded', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n\n    # if noannex target gets some annex, we still should not fail to push\n    target_subnoannex.call_git(['annex', 'init'])\n    # just to ensure that we do need something to push\n    (subnoannex.pathobj / \"newfile\").write_text(\"content\")\n    subnoannex.save()\n    res = subnoannex.push(to=\"target\")\n    assert_in_results(res, status='ok', type='dataset')\n\n\n@slow  # 12sec on Yarik's laptop\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_push_subds_no_recursion(src_path, dst_top, dst_sub, dst_subsub):\n    # dataset with one submodule and one subsubmodule\n    top = Dataset(src_path).create()\n    sub = top.create('sub m')\n    test_file = sub.pathobj / 'subdir' / 'test_file'\n    test_file.parent.mkdir()\n    test_file.write_text('some')\n    subsub = sub.create(sub.pathobj / 'subdir' / 'subsub m')\n    top.save(recursive=True)\n    assert_repo_status(top.path)\n    target_top = mk_push_target(top, 'target', dst_top, annex=True)\n    target_sub = mk_push_target(sub, 'target', dst_sub, annex=True)\n    target_subsub = mk_push_target(subsub, 'target', dst_subsub, annex=True)\n    # now publish, but NO recursion, instead give the parent dir of\n    # both a subdataset and a file in the middle subdataset\n    res = top.push(\n        to='target',\n        # give relative to top dataset to elevate the difficulty a little\n        path=str(test_file.relative_to(top.pathobj).parent))\n    assert_status('ok', res)\n    assert_in_results(res, action='publish', type='dataset', path=top.path)\n    assert_in_results(res, action='publish', type='dataset', path=sub.path)\n    assert_in_results(res, action='copy', type='file', path=str(test_file))\n    # the lowest-level subdataset isn't touched\n    assert_not_in_results(\n        res, action='publish', type='dataset', path=subsub.path)\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_force_checkdatapresent(srcpath, dstpath):\n    src = Dataset(srcpath).create()\n    target = mk_push_target(src, 'target', dstpath, annex=True, bare=True)\n    (src.pathobj / 'test_mod_annex_file').write_text(\"Heavy stuff.\")\n    src.save(to_git=False, message=\"New annex file\")\n    assert_repo_status(src.path, annex=True)\n    whereis_prior = src.repo.whereis(files=['test_mod_annex_file'])[0]\n\n    res = src.push(to='target', data='nothing')\n    # nothing reported to be copied\n    assert_not_in_results(res, action='copy')\n    # we got the git-push nevertheless\n    eq_(src.repo.get_hexsha(DEFAULT_BRANCH), target.get_hexsha(DEFAULT_BRANCH))\n    # nothing moved\n    eq_(whereis_prior, src.repo.whereis(files=['test_mod_annex_file'])[0])\n\n    # now a push without forced no-transfer\n    # we do not give since, so the non-transfered file is picked up\n    # and transferred\n    res = src.push(to='target', force=None)\n    # no branch change, done before\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec=DEFAULT_REFSPEC)\n    # but availability update\n    assert_in_results(res, action='publish', status='ok',\n                      refspec='refs/heads/git-annex:refs/heads/git-annex')\n    assert_in_results(res, status='ok',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy')\n    # whereis info reflects the change\n    ok_(len(whereis_prior) < len(\n        src.repo.whereis(files=['test_mod_annex_file'])[0]))\n\n    # do it yet again will do nothing, because all is up-to-date\n    assert_status('notneeded', src.push(to='target', force=None))\n    # an explicit reference point doesn't change that\n    assert_status('notneeded',\n                  src.push(to='target', force=None, since='HEAD~1'))\n\n    # now force data transfer\n    res = src.push(to='target', force='checkdatapresent')\n    # no branch change, done before\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec=DEFAULT_REFSPEC)\n    # no availability update\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec='refs/heads/git-annex:refs/heads/git-annex')\n    # but data transfer\n    assert_in_results(res, status='ok',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy')\n\n    # force data transfer, but data isn't available\n    src.repo.drop('test_mod_annex_file')\n    res = src.push(to='target', path='.', force='checkdatapresent', on_failure='ignore')\n    assert_in_results(res, status='impossible',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy',\n                      message='Slated for transport, but no content present')\n\n\n@skip_if_on_windows  # https://github.com/datalad/datalad/issues/4278\n@with_tempfile(mkdir=True)\n@with_tree(tree={'ria-layout-version': '1\\n'})\ndef test_ria_push(srcpath, dstpath):\n    # complex test involving a git remote, a special remote, and a\n    # publication dependency\n    src = Dataset(srcpath).create()\n    testfile = src.pathobj / 'test_mod_annex_file'\n    testfile.write_text(\"Heavy stuff.\")\n    src.save()\n    assert_status(\n        'ok',\n        src.create_sibling_ria(\n            \"ria+{}\".format(get_local_file_url(dstpath, compatibility='git')),\n            \"datastore\"))\n    res = src.push(to='datastore')\n    assert_in_results(\n        res, action='publish', target='datastore', status='ok',\n        refspec=DEFAULT_REFSPEC)\n    assert_in_results(\n        res, action='publish', target='datastore', status='ok',\n        refspec='refs/heads/git-annex:refs/heads/git-annex')\n    assert_in_results(\n        res, action='copy', target='datastore-storage', status='ok',\n        path=str(testfile))\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_gh1426(origin_path, target_path):\n    # set up a pair of repos, one the published copy of the other\n    origin = Dataset(origin_path).create()\n    target = mk_push_target(\n        origin, 'target', target_path, annex=True, bare=False)\n    origin.push(to='target')\n    assert_repo_status(origin.path)\n    assert_repo_status(target.path)\n    eq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n        target.get_hexsha(DEFAULT_BRANCH))\n\n    # gist of #1426 is that a newly added subdataset does not cause the\n    # superdataset to get published\n    origin.create('sub')\n    assert_repo_status(origin.path)\n    neq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n         target.get_hexsha(DEFAULT_BRANCH))\n    # now push\n    res = origin.push(to='target')\n    assert_result_count(\n        res, 1, status='ok', type='dataset', path=origin.path,\n        action='publish', target='target', operations=['fast-forward'])\n    eq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n        target.get_hexsha(DEFAULT_BRANCH))\n\n\n@skip_if_adjusted_branch  # gh-4075\n@skip_if_on_windows  # create_sibling incompatible with win servers\n@skip_ssh\n@with_tree(tree={'1': '123'})\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_publish_target_url(src, desttop, desturl):\n    # https://github.com/datalad/datalad/issues/1762\n    ds = Dataset(src).create(force=True)\n    ds.save('1')\n    ds.create_sibling('ssh://datalad-test:%s/subdir' % desttop,\n                      name='target',\n                      target_url=desturl + 'subdir/.git')\n    results = ds.push(to='target')\n    assert results\n    ok_file_has_content(Path(desttop, 'subdir', '1'), '123')\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile()\n@with_tempfile()\ndef test_gh1763(src, target1, target2):\n    # this test is very similar to test_publish_depends, but more\n    # comprehensible, and directly tests issue 1763\n    src = Dataset(src).create(force=True)\n    target1 = mk_push_target(src, 'target1', target1, bare=False)\n    target2 = mk_push_target(src, 'target2', target2, bare=False)\n    src.siblings('configure', name='target2', publish_depends='target1',\n                 result_renderer=None)\n    # a file to annex\n    (src.pathobj / 'probe1').write_text('probe1')\n    src.save('probe1', to_git=False)\n    # make sure the probe is annexed, not straight in Git\n    assert_in('probe1', src.repo.get_annexed_files(with_content_only=True))\n    # publish to target2, must handle dependency\n    src.push(to='target2')\n    for target in (target1, target2):\n        # with a managed branch we are pushing into the corresponding branch\n        # and do not see a change in the worktree\n        if not target.is_managed_branch():\n            # direct test for what is in the checkout\n            assert_in(\n                'probe1',\n                target.get_annexed_files(with_content_only=True))\n        # ensure git-annex knows this target has the file\n        assert_in(target.config.get('annex.uuid'), src.repo.whereis(['probe1'])[0])\n\n\n@with_tempfile()\n@with_tempfile()\ndef test_gh1811(srcpath, clonepath):\n    orig = Dataset(srcpath).create()\n    (orig.pathobj / 'some').write_text('some')\n    orig.save()\n    clone = Clone.__call__(source=orig.path, path=clonepath)\n    (clone.pathobj / 'somemore').write_text('somemore')\n    clone.save()\n    clone.repo.call_git(['checkout', 'HEAD~1'])\n    res = clone.push(to=DEFAULT_REMOTE, on_failure='ignore')\n    assert_result_count(res, 1)\n    assert_result_count(\n        res, 1,\n        path=clone.path, type='dataset', action='publish',\n        status='impossible',\n        message='There is no active branch, cannot determine remote '\n                'branch',\n    )\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@with_tempfile()\n@with_tempfile()\ndef test_push_wanted(srcpath, dstpath):\n    src = Dataset(srcpath).create()\n    (src.pathobj / 'data.0').write_text('0')\n    (src.pathobj / 'secure.1').write_text('1')\n    (src.pathobj / 'secure.2').write_text('2')\n    src.save()\n\n    # Dropping a file to mimic a case of simply not having it locally (thus not\n    # to be \"pushed\")\n    src.drop('secure.2', check=False)\n\n    # Annotate sensitive content, actual value \"verysecure\" does not matter in\n    # this example\n    src.repo.set_metadata(\n        add={'distribution-restrictions': 'verysecure'},\n        files=['secure.1', 'secure.2'])\n\n    src.create_sibling(\n        dstpath,\n        annex_wanted=\"not metadata=distribution-restrictions=*\",\n        name='target',\n    )\n    # check that wanted is obeyed, since set in sibling configuration\n    res = src.push(to='target')\n    assert_in_results(\n        res, action='copy', path=str(src.pathobj / 'data.0'), status='ok')\n    for p in ('secure.1', 'secure.2'):\n        assert_not_in_results(res, path=str(src.pathobj / p))\n    assert_status('notneeded', src.push(to='target'))\n\n    # check the target to really make sure\n    dst = Dataset(dstpath)\n    # normal file, yes\n    eq_((dst.pathobj / 'data.0').read_text(), '0')\n    # secure file, no\n    if dst.repo.is_managed_branch():\n        neq_((dst.pathobj / 'secure.1').read_text(), '1')\n    else:\n        assert_raises(FileNotFoundError, (dst.pathobj / 'secure.1').read_text)\n\n    # reset wanted config, which must enable push of secure file\n    src.repo.set_preferred_content('wanted', '', remote='target')\n    res = src.push(to='target')\n    assert_in_results(res, path=str(src.pathobj / 'secure.1'))\n    eq_((dst.pathobj / 'secure.1').read_text(), '1')\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@slow  # 10sec on Yarik's laptop\n@with_tempfile(mkdir=True)\ndef test_auto_data_transfer(path):\n    path = Path(path)\n    ds_a = Dataset(path / \"a\").create()\n    (ds_a.pathobj / \"foo.dat\").write_text(\"foo\")\n    ds_a.save()\n\n    # Should be the default, but just in case.\n    ds_a.repo.config.set(\"annex.numcopies\", \"1\", where=\"local\")\n    ds_a.create_sibling(str(path / \"b\"), name=\"b\")\n\n    # With numcopies=1, no data is copied with data=\"auto\".\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_not_in_results(res, action=\"copy\")\n\n    # Even when a file is explicitly given.\n    res = ds_a.push(to=\"b\", path=\"foo.dat\", data=\"auto\", since=None)\n    assert_not_in_results(res, action=\"copy\")\n\n    # numcopies=2 changes that.\n    ds_a.repo.config.set(\"annex.numcopies\", \"2\", where=\"local\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_in_results(\n        res, action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"foo.dat\"))\n\n    # --since= limits the files considered by --auto.\n    (ds_a.pathobj / \"bar.dat\").write_text(\"bar\")\n    ds_a.save()\n    (ds_a.pathobj / \"baz.dat\").write_text(\"baz\")\n    ds_a.save()\n    res = ds_a.push(to=\"b\", data=\"auto\", since=\"HEAD~1\")\n    assert_not_in_results(\n        res,\n        action=\"copy\", path=str(ds_a.pathobj / \"bar.dat\"))\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"baz.dat\"))\n\n    # --auto also considers preferred content.\n    ds_a.repo.config.unset(\"annex.numcopies\", where=\"local\")\n    ds_a.repo.set_preferred_content(\"wanted\", \"nothing\", remote=\"b\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_not_in_results(\n        res,\n        action=\"copy\", path=str(ds_a.pathobj / \"bar.dat\"))\n\n    ds_a.repo.set_preferred_content(\"wanted\", \"anything\", remote=\"b\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"bar.dat\"))\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@slow  # 16sec on Yarik's laptop\n@with_tempfile(mkdir=True)\ndef test_auto_if_wanted_data_transfer_path_restriction(path):\n    path = Path(path)\n    ds_a = Dataset(path / \"a\").create()\n    ds_a_sub0 = ds_a.create(\"sub0\")\n    ds_a_sub1 = ds_a.create(\"sub1\")\n\n    for ds in [ds_a, ds_a_sub0, ds_a_sub1]:\n        (ds.pathobj / \"sec.dat\").write_text(\"sec\")\n        (ds.pathobj / \"reg.dat\").write_text(\"reg\")\n    ds_a.save(recursive=True)\n\n    ds_a.create_sibling(str(path / \"b\"), name=\"b\",\n                        annex_wanted=\"not metadata=distribution-restrictions=*\",\n                        recursive=True)\n    for ds in [ds_a, ds_a_sub0, ds_a_sub1]:\n        ds.repo.set_metadata(add={\"distribution-restrictions\": \"doesntmatter\"},\n                             files=[\"sec.dat\"])\n\n    # wanted-triggered --auto can be restricted to subdataset...\n    res = ds_a.push(to=\"b\", path=\"sub0\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"reg.dat\"))\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub0.pathobj / \"reg.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub0.pathobj / \"sec.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub1.pathobj / \"reg.dat\"))\n\n    # ... and to a wanted file.\n    res = ds_a.push(to=\"b\", path=\"reg.dat\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"reg.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub1.pathobj / \"reg.dat\"))\n\n    # But asking to transfer a file does not do it if the remote has a\n    # wanted setting and doesn't want it.\n    res = ds_a.push(to=\"b\", path=\"sec.dat\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"sec.dat\"))\n\n    res = ds_a.push(to=\"b\", path=\"sec.dat\", data=\"anything\", recursive=True)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"sec.dat\"))\n\n\n@with_tempfile(mkdir=True)\ndef test_push_git_annex_branch_when_no_data(path):\n    path = Path(path)\n    ds = Dataset(path / \"a\").create()\n    target = mk_push_target(ds, \"target\", str(path / \"target\"),\n                            annex=False, bare=True)\n    (ds.pathobj / \"f0\").write_text(\"0\")\n    ds.save()\n    ds.push(to=\"target\", data=\"nothing\")\n    assert_in(\"git-annex\",\n              {d[\"refname:strip=2\"]\n               for d in target.for_each_ref_(fields=\"refname:strip=2\")})\n\n\n@known_failure_githubci_osx\n@with_tree(tree={\"ds\": {\"f0\": \"0\", \"f1\": \"0\", \"f2\": \"0\",\n                        \"f3\": \"1\",\n                        \"f4\": \"2\", \"f5\": \"2\"}})\ndef test_push_git_annex_branch_many_paths_same_data(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create(force=True)\n    ds.save()\n    mk_push_target(ds, \"target\", str(path / \"target\"),\n                   annex=True, bare=False)\n    nbytes = sum(ds.repo.get_content_annexinfo(paths=[f])[f][\"bytesize\"]\n                 for f in [ds.repo.pathobj / \"f0\",\n                           ds.repo.pathobj / \"f3\",\n                           ds.repo.pathobj / \"f4\"])\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        res = ds.push(to=\"target\")\n    assert_in(\"{} bytes of annex data\".format(nbytes), cml.out)\n    # 3 files point to content already covered by another file.\n    assert_result_count(res, 3,\n                        action=\"copy\", type=\"file\", status=\"notneeded\")\n\n\n@known_failure_githubci_osx\n@with_tree(tree={\"ds\": {\"f0\": \"0\"}})\ndef test_push_matching(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create(force=True)\n    ds.config.set('push.default', 'matching', where='local')\n    ds.save()\n    remote_ds = mk_push_target(ds, 'local', str(path / 'dssibling'),\n                               annex=True, bare=False)\n    # that fact that the next one even runs makes sure that we are in a better\n    # place than https://github.com/datalad/datalad/issues/4888\n    ds.push(to='local')\n    # and we pushed the commit in the current branch\n    eq_(remote_ds.get_hexsha(DEFAULT_BRANCH),\n        ds.repo.get_hexsha(DEFAULT_BRANCH))\n\n\n@known_failure_githubci_win  # https://github.com/datalad/datalad/issues/5271\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):\n    if 'DATALAD_SEED' in os.environ:\n        # we are using create-sibling-ria via the cmdline in here\n        # this will create random UUIDs for datasets\n        # however, given a fixed seed each call to this command will start\n        # with the same RNG seed, hence yield the same UUID on the same\n        # machine -- leading to a collision\n        raise SkipTest(\n            'Test incompatible with fixed random number generator seed')\n    # the aim here is this high-level test a std create-push-clone cycle for a\n    # dataset with a subdataset, with the goal to ensure that correct branches\n    # and commits are tracked, regardless of platform behavior and condition\n    # of individual clones. Nothing fancy, just that the defaults behave in\n    # sensible ways\n    from datalad.cmd import WitlessRunner as Runner\n    run = Runner().run\n\n    # create original nested dataset\n    with chpwd(origpath):\n        run(['datalad', 'create', 'super'])\n        run(['datalad', 'create', '-d', 'super', str(Path('super', 'sub'))])\n\n    # verify essential linkage properties\n    orig_super = Dataset(Path(origpath, 'super'))\n    orig_sub = Dataset(orig_super.pathobj / 'sub')\n\n    (orig_super.pathobj / 'file1.txt').write_text('some1')\n    (orig_sub.pathobj / 'file2.txt').write_text('some1')\n    with chpwd(orig_super.path):\n        run(['datalad', 'save', '--recursive'])\n\n    # TODO not yet reported clean with adjusted branches\n    #assert_repo_status(orig_super.path)\n\n    # the \"true\" branch that sub is on, and the gitsha of the HEAD commit of it\n    orig_sub_corr_branch = \\\n        orig_sub.repo.get_corresponding_branch() or orig_sub.repo.get_active_branch()\n    orig_sub_corr_commit = orig_sub.repo.get_hexsha(orig_sub_corr_branch)\n\n    # make sure the super trackes this commit\n    assert_in_results(\n        orig_super.subdatasets(),\n        path=orig_sub.path,\n        gitshasum=orig_sub_corr_commit,\n        # TODO it should also track the branch name\n        # Attempted: https://github.com/datalad/datalad/pull/3817\n        # But reverted: https://github.com/datalad/datalad/pull/4375\n    )\n\n    # publish to a store, to get into a platform-agnostic state\n    # (i.e. no impact of an annex-init of any kind)\n    store_url = 'ria+' + get_local_file_url(storepath)\n    with chpwd(orig_super.path):\n        run(['datalad', 'create-sibling-ria', '--recursive',\n             '-s', 'store', store_url])\n        run(['datalad', 'push', '--recursive', '--to', 'store'])\n\n    # we are using the 'store' sibling's URL, which should be a plain path\n    store_super = AnnexRepo(orig_super.siblings(name='store')[0]['url'], init=False)\n    store_sub = AnnexRepo(orig_sub.siblings(name='store')[0]['url'], init=False)\n\n    # both datasets in the store only carry the real branches, and nothing\n    # adjusted\n    for r in (store_super, store_sub):\n        eq_(set(r.get_branches()), set([orig_sub_corr_branch, 'git-annex']))\n\n    # and reobtain from a store\n    cloneurl = 'ria+' + get_local_file_url(str(storepath), compatibility='git')\n    with chpwd(clonepath):\n        run(['datalad', 'clone', cloneurl + '#' + orig_super.id, 'super'])\n        run(['datalad', '-C', 'super', 'get', '--recursive', '.'])\n\n    # verify that nothing has changed as a result of a push/clone cycle\n    clone_super = Dataset(Path(clonepath, 'super'))\n    clone_sub = Dataset(clone_super.pathobj / 'sub')\n    assert_in_results(\n        clone_super.subdatasets(),\n        path=clone_sub.path,\n        gitshasum=orig_sub_corr_commit,\n    )\n\n    for ds1, ds2, f in ((orig_super, clone_super, 'file1.txt'),\n                        (orig_sub, clone_sub, 'file2.txt')):\n        eq_((ds1.pathobj / f).read_text(), (ds2.pathobj / f).read_text())\n\n    # get status info that does not recursive into subdatasets, i.e. not\n    # looking for uncommitted changes\n    # we should see no modification reported\n    assert_not_in_results(\n        clone_super.status(eval_subdataset_state='commit'),\n        state='modified')\n    # and now the same for a more expensive full status\n    assert_not_in_results(\n        clone_super.status(recursive=True),\n        state='modified')\n",
            "new_file_content": "# -*- coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test push\n\n\"\"\"\n\nimport os\nimport logging\n\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.support.exceptions import (\n    IncompleteResultsError,\n    InsufficientArgumentsError,\n)\nfrom datalad.tests.utils import (\n    assert_false,\n    assert_in,\n    assert_in_results,\n    assert_not_in,\n    assert_not_in_results,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_status,\n    DEFAULT_BRANCH,\n    DEFAULT_REMOTE,\n    eq_,\n    known_failure_githubci_osx,\n    known_failure_githubci_win,\n    neq_,\n    ok_,\n    ok_file_has_content,\n    serve_path_via_http,\n    skip_if_adjusted_branch,\n    skip_if_on_windows,\n    skip_ssh,\n    slow,\n    swallow_logs,\n    with_tempfile,\n    with_tree,\n    SkipTest,\n)\nfrom datalad.utils import (\n    Path,\n    chpwd,\n    path_startswith,\n    swallow_outputs,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.core.distributed.clone import Clone\nfrom datalad.core.distributed.push import Push\nfrom datalad.support.network import get_local_file_url\n\nDEFAULT_REFSPEC = \"refs/heads/{0}:refs/heads/{0}\".format(DEFAULT_BRANCH)\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_call(origin, tdir):\n    ds = Dataset(origin).create()\n    # no target\n    assert_status('impossible', ds.push(on_failure='ignore'))\n    # no dataset\n    with chpwd(tdir):\n        assert_raises(InsufficientArgumentsError, Push.__call__)\n    # dataset, but outside path\n    assert_raises(IncompleteResultsError, ds.push, path=tdir)\n    # given a path constraint that doesn't match anything, will cause\n    # nothing to be done\n    assert_status('notneeded', ds.push(path=ds.pathobj / 'nothere'))\n\n    # unavailable subdataset\n    dummy_sub = ds.create('sub')\n    dummy_sub.uninstall()\n    assert_in('sub', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    # now an explicit call to publish the unavailable subdataset\n    assert_raises(ValueError, ds.push, 'sub')\n\n    target = mk_push_target(ds, 'target', tdir, annex=True)\n    # revision that doesn't exist\n    assert_raises(\n        ValueError,\n        ds.push, to='target', since='09320957509720437523')\n\n    # If a publish() user accidentally passes since='', which push() spells as\n    # since='^', the call is aborted.\n    assert_raises(\n        ValueError,\n        ds.push, to='target', since='')\n\n\ndef mk_push_target(ds, name, path, annex=True, bare=True):\n    # life could be simple, but nothing is simple on windows\n    #src.create_sibling(dst_path, name='target')\n    if annex:\n        if bare:\n            target = GitRepo(path=path, bare=True, create=True)\n            # cannot use call_annex()\n            target.call_git(['annex', 'init'])\n        else:\n            target = AnnexRepo(path, init=True, create=True)\n            if not target.is_managed_branch():\n                # for managed branches we need more fireworks->below\n                target.config.set(\n                    'receive.denyCurrentBranch', 'updateInstead',\n                    where='local')\n    else:\n        target = GitRepo(path=path, bare=bare, create=True)\n    ds.siblings('add', name=name, url=path, result_renderer=None)\n    if annex and not bare and target.is_managed_branch():\n        # maximum complication\n        # the target repo already has a commit that is unrelated\n        # to the source repo, because it has built a reference\n        # commit for the managed branch.\n        # the only sane approach is to let git-annex establish a shared\n        # history\n        ds.repo.call_annex(['sync'])\n        ds.repo.call_annex(['sync', '--cleanup'])\n    return target\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef check_push(annex, src_path, dst_path):\n    # prepare src\n    src = Dataset(src_path).create(annex=annex)\n    src_repo = src.repo\n    # push should not add branches to the local dataset\n    orig_branches = src_repo.get_branches()\n    assert_not_in('synced/' + DEFAULT_BRANCH, orig_branches)\n\n    res = src.push(on_failure='ignore')\n    assert_result_count(res, 1)\n    assert_in_results(\n        res, status='impossible',\n        message='No push target given, and none could be auto-detected, '\n        'please specify via --to')\n    eq_(orig_branches, src_repo.get_branches())\n    # target sibling\n    target = mk_push_target(src, 'target', dst_path, annex=annex)\n    eq_(orig_branches, src_repo.get_branches())\n\n    res = src.push(to=\"target\")\n    eq_(orig_branches, src_repo.get_branches())\n    assert_result_count(res, 2 if annex else 1)\n    assert_in_results(\n        res,\n        action='publish', status='ok', target='target',\n        refspec=DEFAULT_REFSPEC,\n        operations=['new-branch'])\n\n    assert_repo_status(src_repo, annex=annex)\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # configure a default merge/upstream target\n    src.config.set('branch.{}.remote'.format(DEFAULT_BRANCH),\n                   'target', where='local')\n    src.config.set('branch.{}.merge'.format(DEFAULT_BRANCH),\n                   DEFAULT_BRANCH, where='local')\n\n    # don't fail when doing it again, no explicit target specification\n    # needed anymore\n    res = src.push()\n    eq_(orig_branches, src_repo.get_branches())\n    # and nothing is pushed\n    assert_status('notneeded', res)\n\n    assert_repo_status(src_repo, annex=annex)\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # some modification:\n    (src.pathobj / 'test_mod_file').write_text(\"Some additional stuff.\")\n    src.save(to_git=True, message=\"Modified.\")\n    (src.pathobj / 'test_mod_annex_file').write_text(\"Heavy stuff.\")\n    src.save(to_git=not annex, message=\"Modified again.\")\n    assert_repo_status(src_repo, annex=annex)\n\n    # we could say since='HEAD~2' to make things fast, or we are lazy\n    # and say since='^' to indicate the state of the tracking remote\n    # which is the same, because we made to commits since the last push.\n    res = src.push(to='target', since=\"^\", jobs=2)\n    assert_in_results(\n        res,\n        action='publish', status='ok', target='target',\n        refspec=DEFAULT_REFSPEC,\n        # we get to see what happened\n        operations=['fast-forward'])\n    if annex:\n        # we got to see the copy result for the annexed files\n        assert_in_results(\n            res,\n            action='copy',\n            status='ok',\n            path=str(src.pathobj / 'test_mod_annex_file'))\n        # we published, so we can drop and reobtain\n        ok_(src_repo.file_has_content('test_mod_annex_file'))\n        src_repo.drop('test_mod_annex_file')\n        ok_(not src_repo.file_has_content('test_mod_annex_file'))\n        src_repo.get('test_mod_annex_file')\n        ok_(src_repo.file_has_content('test_mod_annex_file'))\n        ok_file_has_content(\n            src_repo.pathobj / 'test_mod_annex_file',\n            'Heavy stuff.')\n\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n    if not (annex and src_repo.is_managed_branch()):\n        # the following doesn't make sense in managed branches, because\n        # a commit that could be amended is no longer the last commit\n        # of a branch after a sync has happened (which did happen\n        # during the last push above\n\n        # amend and change commit msg in order to test for force push:\n        src_repo.commit(\"amended\", options=['--amend'])\n        # push should be rejected (non-fast-forward):\n        res = src.push(to='target', since='HEAD~2', on_failure='ignore')\n        # fails before even touching the annex branch\n        assert_in_results(\n            res,\n            action='publish', status='error', target='target',\n            refspec=DEFAULT_REFSPEC,\n            operations=['rejected', 'error'])\n        # push with force=True works:\n        res = src.push(to='target', since='HEAD~2', force='gitpush')\n        assert_in_results(\n            res,\n            action='publish', status='ok', target='target',\n            refspec=DEFAULT_REFSPEC,\n            operations=['forced-update'])\n        eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n            list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # we do not have more branches than we had in the beginning\n    # in particular no 'synced/<default branch>'\n    eq_(orig_branches, src_repo.get_branches())\n\n\ndef test_push():\n    yield check_push, False\n    yield check_push, True\n\n\ndef check_datasets_order(res, order='bottom-up'):\n    \"\"\"Check that all type=dataset records not violating the expected order\n\n    it is somewhat weak test, i.e. records could be produced so we\n    do not detect that order is violated, e.g. a/b c/d would satisfy\n    either although they might be neither depth nor breadth wise.  But\n    this test would allow to catch obvious violations like a, a/b, a\n    \"\"\"\n    prev = None\n    for r in res:\n        if r.get('type') != 'dataset':\n            continue\n        if prev and r['path'] != prev:\n            if order == 'bottom-up':\n                assert_false(path_startswith(r['path'], prev))\n            elif order == 'top-down':\n                assert_false(path_startswith(prev, r['path']))\n            else:\n                raise ValueError(order)\n        prev = r['path']\n\n\n@slow  # 33sec on Yarik's laptop\n@with_tempfile\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True, suffix='sub')\n@with_tempfile(mkdir=True, suffix='subnoannex')\n@with_tempfile(mkdir=True, suffix='subsub')\ndef test_push_recursive(\n        origin_path, src_path, dst_top, dst_sub, dst_subnoannex, dst_subsub):\n    # dataset with two submodules and one subsubmodule\n    origin = Dataset(origin_path).create()\n    origin_subm1 = origin.create('sub m')\n    origin_subm1.create('subsub m')\n    origin.create('subm noannex', annex=False)\n    origin.save()\n    assert_repo_status(origin.path)\n    # prepare src as a fresh clone with all subdatasets checkout out recursively\n    # running on a clone should make the test scenario more different than\n    # test_push(), even for the pieces that should be identical\n    top = Clone.__call__(source=origin.path, path=src_path)\n    subs = top.get('.', recursive=True, get_data=False, result_xfm='datasets')\n    # order for '.' should not be relied upon, so sort by path\n    sub, subsub, subnoannex = sorted(subs, key=lambda ds: ds.path)\n\n    target_top = mk_push_target(top, 'target', dst_top, annex=True)\n    # subdatasets have no remote yet, so recursive publishing should fail:\n    res = top.push(to=\"target\", recursive=True, on_failure='ignore')\n    check_datasets_order(res)\n    assert_in_results(\n        res, path=top.path, type='dataset',\n        refspec=DEFAULT_REFSPEC,\n        operations=['new-branch'], action='publish', status='ok',\n        target='target')\n    for d in (sub, subsub, subnoannex):\n        assert_in_results(\n            res, status='error', type='dataset', path=d.path,\n            message=(\"Unknown target sibling '%s'.\",\n                     'target'))\n    # now fix that and set up targets for the submodules\n    target_sub = mk_push_target(sub, 'target', dst_sub, annex=True)\n    target_subnoannex = mk_push_target(\n        subnoannex, 'target', dst_subnoannex, annex=False)\n    target_subsub = mk_push_target(subsub, 'target', dst_subsub, annex=True)\n\n    # and same push call as above\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    # topds skipped\n    assert_in_results(\n        res, path=top.path, type='dataset',\n        action='publish', status='notneeded', target='target')\n    # the rest pushed\n    for d in (sub, subsub, subnoannex):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # all corresponding branches match across all datasets\n    for s, d in zip((top, sub, subnoannex, subsub),\n                    (target_top, target_sub, target_subnoannex,\n                     target_subsub)):\n        eq_(list(s.repo.get_branch_commits_(DEFAULT_BRANCH)),\n            list(d.get_branch_commits_(DEFAULT_BRANCH)))\n        if s != subnoannex:\n            eq_(list(s.repo.get_branch_commits_(\"git-annex\")),\n                list(d.get_branch_commits_(\"git-annex\")))\n\n    # rerun should not result in further pushes of the default branch\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    assert_not_in_results(\n        res, status='ok', refspec=DEFAULT_REFSPEC)\n    assert_in_results(\n        res, status='notneeded', refspec=DEFAULT_REFSPEC)\n\n    # now annex a file in subsub\n    test_copy_file = subsub.pathobj / 'test_mod_annex_file'\n    test_copy_file.write_text(\"Heavy stuff.\")\n    # save all the way up\n    assert_status(\n        ('ok', 'notneeded'),\n        top.save(message='subsub got something', recursive=True))\n    assert_repo_status(top.path)\n    # publish straight up, should be smart by default\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    # we see 3 out of 4 datasets pushed (sub noannex was left unchanged)\n    for d in (top, sub, subsub):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # file content copied too\n    assert_in_results(\n        res,\n        action='copy',\n        status='ok',\n        path=str(test_copy_file))\n    # verify it is accessible, drop and bring back\n    assert_status('ok', top.drop(str(test_copy_file)))\n    ok_(not subsub.repo.file_has_content('test_mod_annex_file'))\n    top.get(test_copy_file)\n    ok_file_has_content(test_copy_file, 'Heavy stuff.')\n\n    # make two modification\n    (sub.pathobj / 'test_mod_annex_file').write_text('annex')\n    (subnoannex.pathobj / 'test_mod_file').write_text('git')\n    # save separately\n    top.save(sub.pathobj, message='annexadd', recursive=True)\n    top.save(subnoannex.pathobj, message='gitadd', recursive=True)\n    # now only publish the latter one\n    res = top.push(to=\"target\", since=DEFAULT_BRANCH + '~1', recursive=True)\n    # nothing copied, no reports on the other modification\n    assert_not_in_results(res, action='copy')\n    assert_not_in_results(res, path=sub.path)\n    for d in (top, subnoannex):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # an unconditional push should now pick up the remaining changes\n    res = top.push(to=\"target\", recursive=True)\n    assert_in_results(\n        res,\n        action='copy',\n        status='ok',\n        path=str(sub.pathobj / 'test_mod_annex_file'))\n    assert_in_results(\n        res, status='ok', type='dataset', path=sub.path,\n        refspec=DEFAULT_REFSPEC)\n    for d in (top, subnoannex, subsub):\n        assert_in_results(\n            res, status='notneeded', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n\n    # if noannex target gets some annex, we still should not fail to push\n    target_subnoannex.call_git(['annex', 'init'])\n    # just to ensure that we do need something to push\n    (subnoannex.pathobj / \"newfile\").write_text(\"content\")\n    subnoannex.save()\n    res = subnoannex.push(to=\"target\")\n    assert_in_results(res, status='ok', type='dataset')\n\n\n@slow  # 12sec on Yarik's laptop\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_push_subds_no_recursion(src_path, dst_top, dst_sub, dst_subsub):\n    # dataset with one submodule and one subsubmodule\n    top = Dataset(src_path).create()\n    sub = top.create('sub m')\n    test_file = sub.pathobj / 'subdir' / 'test_file'\n    test_file.parent.mkdir()\n    test_file.write_text('some')\n    subsub = sub.create(sub.pathobj / 'subdir' / 'subsub m')\n    top.save(recursive=True)\n    assert_repo_status(top.path)\n    target_top = mk_push_target(top, 'target', dst_top, annex=True)\n    target_sub = mk_push_target(sub, 'target', dst_sub, annex=True)\n    target_subsub = mk_push_target(subsub, 'target', dst_subsub, annex=True)\n    # now publish, but NO recursion, instead give the parent dir of\n    # both a subdataset and a file in the middle subdataset\n    res = top.push(\n        to='target',\n        # give relative to top dataset to elevate the difficulty a little\n        path=str(test_file.relative_to(top.pathobj).parent))\n    assert_status('ok', res)\n    assert_in_results(res, action='publish', type='dataset', path=top.path)\n    assert_in_results(res, action='publish', type='dataset', path=sub.path)\n    assert_in_results(res, action='copy', type='file', path=str(test_file))\n    # the lowest-level subdataset isn't touched\n    assert_not_in_results(\n        res, action='publish', type='dataset', path=subsub.path)\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_force_checkdatapresent(srcpath, dstpath):\n    src = Dataset(srcpath).create()\n    target = mk_push_target(src, 'target', dstpath, annex=True, bare=True)\n    (src.pathobj / 'test_mod_annex_file').write_text(\"Heavy stuff.\")\n    src.save(to_git=False, message=\"New annex file\")\n    assert_repo_status(src.path, annex=True)\n    whereis_prior = src.repo.whereis(files=['test_mod_annex_file'])[0]\n\n    res = src.push(to='target', data='nothing')\n    # nothing reported to be copied\n    assert_not_in_results(res, action='copy')\n    # we got the git-push nevertheless\n    eq_(src.repo.get_hexsha(DEFAULT_BRANCH), target.get_hexsha(DEFAULT_BRANCH))\n    # nothing moved\n    eq_(whereis_prior, src.repo.whereis(files=['test_mod_annex_file'])[0])\n\n    # now a push without forced no-transfer\n    # we do not give since, so the non-transfered file is picked up\n    # and transferred\n    res = src.push(to='target', force=None)\n    # no branch change, done before\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec=DEFAULT_REFSPEC)\n    # but availability update\n    assert_in_results(res, action='publish', status='ok',\n                      refspec='refs/heads/git-annex:refs/heads/git-annex')\n    assert_in_results(res, status='ok',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy')\n    # whereis info reflects the change\n    ok_(len(whereis_prior) < len(\n        src.repo.whereis(files=['test_mod_annex_file'])[0]))\n\n    # do it yet again will do nothing, because all is up-to-date\n    assert_status('notneeded', src.push(to='target', force=None))\n    # an explicit reference point doesn't change that\n    assert_status('notneeded',\n                  src.push(to='target', force=None, since='HEAD~1'))\n\n    # now force data transfer\n    res = src.push(to='target', force='checkdatapresent')\n    # no branch change, done before\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec=DEFAULT_REFSPEC)\n    # no availability update\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec='refs/heads/git-annex:refs/heads/git-annex')\n    # but data transfer\n    assert_in_results(res, status='ok',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy')\n\n    # force data transfer, but data isn't available\n    src.repo.drop('test_mod_annex_file')\n    res = src.push(to='target', path='.', force='checkdatapresent', on_failure='ignore')\n    assert_in_results(res, status='impossible',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy',\n                      message='Slated for transport, but no content present')\n\n\n@skip_if_on_windows  # https://github.com/datalad/datalad/issues/4278\n@with_tempfile(mkdir=True)\n@with_tree(tree={'ria-layout-version': '1\\n'})\ndef test_ria_push(srcpath, dstpath):\n    # complex test involving a git remote, a special remote, and a\n    # publication dependency\n    src = Dataset(srcpath).create()\n    testfile = src.pathobj / 'test_mod_annex_file'\n    testfile.write_text(\"Heavy stuff.\")\n    src.save()\n    assert_status(\n        'ok',\n        src.create_sibling_ria(\n            \"ria+{}\".format(get_local_file_url(dstpath, compatibility='git')),\n            \"datastore\"))\n    res = src.push(to='datastore')\n    assert_in_results(\n        res, action='publish', target='datastore', status='ok',\n        refspec=DEFAULT_REFSPEC)\n    assert_in_results(\n        res, action='publish', target='datastore', status='ok',\n        refspec='refs/heads/git-annex:refs/heads/git-annex')\n    assert_in_results(\n        res, action='copy', target='datastore-storage', status='ok',\n        path=str(testfile))\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_gh1426(origin_path, target_path):\n    # set up a pair of repos, one the published copy of the other\n    origin = Dataset(origin_path).create()\n    target = mk_push_target(\n        origin, 'target', target_path, annex=True, bare=False)\n    origin.push(to='target')\n    assert_repo_status(origin.path)\n    assert_repo_status(target.path)\n    eq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n        target.get_hexsha(DEFAULT_BRANCH))\n\n    # gist of #1426 is that a newly added subdataset does not cause the\n    # superdataset to get published\n    origin.create('sub')\n    assert_repo_status(origin.path)\n    neq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n         target.get_hexsha(DEFAULT_BRANCH))\n    # now push\n    res = origin.push(to='target')\n    assert_result_count(\n        res, 1, status='ok', type='dataset', path=origin.path,\n        action='publish', target='target', operations=['fast-forward'])\n    eq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n        target.get_hexsha(DEFAULT_BRANCH))\n\n\n@skip_if_adjusted_branch  # gh-4075\n@skip_if_on_windows  # create_sibling incompatible with win servers\n@skip_ssh\n@with_tree(tree={'1': '123'})\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_publish_target_url(src, desttop, desturl):\n    # https://github.com/datalad/datalad/issues/1762\n    ds = Dataset(src).create(force=True)\n    ds.save('1')\n    ds.create_sibling('ssh://datalad-test:%s/subdir' % desttop,\n                      name='target',\n                      target_url=desturl + 'subdir/.git')\n    results = ds.push(to='target')\n    assert results\n    ok_file_has_content(Path(desttop, 'subdir', '1'), '123')\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile()\n@with_tempfile()\ndef test_gh1763(src, target1, target2):\n    # this test is very similar to test_publish_depends, but more\n    # comprehensible, and directly tests issue 1763\n    src = Dataset(src).create(force=True)\n    target1 = mk_push_target(src, 'target1', target1, bare=False)\n    target2 = mk_push_target(src, 'target2', target2, bare=False)\n    src.siblings('configure', name='target2', publish_depends='target1',\n                 result_renderer=None)\n    # a file to annex\n    (src.pathobj / 'probe1').write_text('probe1')\n    src.save('probe1', to_git=False)\n    # make sure the probe is annexed, not straight in Git\n    assert_in('probe1', src.repo.get_annexed_files(with_content_only=True))\n    # publish to target2, must handle dependency\n    src.push(to='target2')\n    for target in (target1, target2):\n        # with a managed branch we are pushing into the corresponding branch\n        # and do not see a change in the worktree\n        if not target.is_managed_branch():\n            # direct test for what is in the checkout\n            assert_in(\n                'probe1',\n                target.get_annexed_files(with_content_only=True))\n        # ensure git-annex knows this target has the file\n        assert_in(target.config.get('annex.uuid'), src.repo.whereis(['probe1'])[0])\n\n\n@with_tempfile()\n@with_tempfile()\ndef test_gh1811(srcpath, clonepath):\n    orig = Dataset(srcpath).create()\n    (orig.pathobj / 'some').write_text('some')\n    orig.save()\n    clone = Clone.__call__(source=orig.path, path=clonepath)\n    (clone.pathobj / 'somemore').write_text('somemore')\n    clone.save()\n    clone.repo.call_git(['checkout', 'HEAD~1'])\n    res = clone.push(to=DEFAULT_REMOTE, on_failure='ignore')\n    assert_result_count(res, 1)\n    assert_result_count(\n        res, 1,\n        path=clone.path, type='dataset', action='publish',\n        status='impossible',\n        message='There is no active branch, cannot determine remote '\n                'branch',\n    )\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@with_tempfile()\n@with_tempfile()\ndef test_push_wanted(srcpath, dstpath):\n    src = Dataset(srcpath).create()\n    (src.pathobj / 'data.0').write_text('0')\n    (src.pathobj / 'secure.1').write_text('1')\n    (src.pathobj / 'secure.2').write_text('2')\n    src.save()\n\n    # Dropping a file to mimic a case of simply not having it locally (thus not\n    # to be \"pushed\")\n    src.drop('secure.2', check=False)\n\n    # Annotate sensitive content, actual value \"verysecure\" does not matter in\n    # this example\n    src.repo.set_metadata(\n        add={'distribution-restrictions': 'verysecure'},\n        files=['secure.1', 'secure.2'])\n\n    src.create_sibling(\n        dstpath,\n        annex_wanted=\"not metadata=distribution-restrictions=*\",\n        name='target',\n    )\n    # check that wanted is obeyed, since set in sibling configuration\n    res = src.push(to='target')\n    assert_in_results(\n        res, action='copy', path=str(src.pathobj / 'data.0'), status='ok')\n    for p in ('secure.1', 'secure.2'):\n        assert_not_in_results(res, path=str(src.pathobj / p))\n    assert_status('notneeded', src.push(to='target'))\n\n    # check the target to really make sure\n    dst = Dataset(dstpath)\n    # normal file, yes\n    eq_((dst.pathobj / 'data.0').read_text(), '0')\n    # secure file, no\n    if dst.repo.is_managed_branch():\n        neq_((dst.pathobj / 'secure.1').read_text(), '1')\n    else:\n        assert_raises(FileNotFoundError, (dst.pathobj / 'secure.1').read_text)\n\n    # reset wanted config, which must enable push of secure file\n    src.repo.set_preferred_content('wanted', '', remote='target')\n    res = src.push(to='target')\n    assert_in_results(res, path=str(src.pathobj / 'secure.1'))\n    eq_((dst.pathobj / 'secure.1').read_text(), '1')\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@slow  # 10sec on Yarik's laptop\n@with_tempfile(mkdir=True)\ndef test_auto_data_transfer(path):\n    path = Path(path)\n    ds_a = Dataset(path / \"a\").create()\n    (ds_a.pathobj / \"foo.dat\").write_text(\"foo\")\n    ds_a.save()\n\n    # Should be the default, but just in case.\n    ds_a.repo.config.set(\"annex.numcopies\", \"1\", where=\"local\")\n    ds_a.create_sibling(str(path / \"b\"), name=\"b\")\n\n    # With numcopies=1, no data is copied with data=\"auto\".\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_not_in_results(res, action=\"copy\")\n\n    # Even when a file is explicitly given.\n    res = ds_a.push(to=\"b\", path=\"foo.dat\", data=\"auto\", since=None)\n    assert_not_in_results(res, action=\"copy\")\n\n    # numcopies=2 changes that.\n    ds_a.repo.config.set(\"annex.numcopies\", \"2\", where=\"local\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_in_results(\n        res, action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"foo.dat\"))\n\n    # --since= limits the files considered by --auto.\n    (ds_a.pathobj / \"bar.dat\").write_text(\"bar\")\n    ds_a.save()\n    (ds_a.pathobj / \"baz.dat\").write_text(\"baz\")\n    ds_a.save()\n    res = ds_a.push(to=\"b\", data=\"auto\", since=\"HEAD~1\")\n    assert_not_in_results(\n        res,\n        action=\"copy\", path=str(ds_a.pathobj / \"bar.dat\"))\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"baz.dat\"))\n\n    # --auto also considers preferred content.\n    ds_a.repo.config.unset(\"annex.numcopies\", where=\"local\")\n    ds_a.repo.set_preferred_content(\"wanted\", \"nothing\", remote=\"b\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_not_in_results(\n        res,\n        action=\"copy\", path=str(ds_a.pathobj / \"bar.dat\"))\n\n    ds_a.repo.set_preferred_content(\"wanted\", \"anything\", remote=\"b\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"bar.dat\"))\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@slow  # 16sec on Yarik's laptop\n@with_tempfile(mkdir=True)\ndef test_auto_if_wanted_data_transfer_path_restriction(path):\n    path = Path(path)\n    ds_a = Dataset(path / \"a\").create()\n    ds_a_sub0 = ds_a.create(\"sub0\")\n    ds_a_sub1 = ds_a.create(\"sub1\")\n\n    for ds in [ds_a, ds_a_sub0, ds_a_sub1]:\n        (ds.pathobj / \"sec.dat\").write_text(\"sec\")\n        (ds.pathobj / \"reg.dat\").write_text(\"reg\")\n    ds_a.save(recursive=True)\n\n    ds_a.create_sibling(str(path / \"b\"), name=\"b\",\n                        annex_wanted=\"not metadata=distribution-restrictions=*\",\n                        recursive=True)\n    for ds in [ds_a, ds_a_sub0, ds_a_sub1]:\n        ds.repo.set_metadata(add={\"distribution-restrictions\": \"doesntmatter\"},\n                             files=[\"sec.dat\"])\n\n    # wanted-triggered --auto can be restricted to subdataset...\n    res = ds_a.push(to=\"b\", path=\"sub0\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"reg.dat\"))\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub0.pathobj / \"reg.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub0.pathobj / \"sec.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub1.pathobj / \"reg.dat\"))\n\n    # ... and to a wanted file.\n    res = ds_a.push(to=\"b\", path=\"reg.dat\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"reg.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub1.pathobj / \"reg.dat\"))\n\n    # But asking to transfer a file does not do it if the remote has a\n    # wanted setting and doesn't want it.\n    res = ds_a.push(to=\"b\", path=\"sec.dat\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"sec.dat\"))\n\n    res = ds_a.push(to=\"b\", path=\"sec.dat\", data=\"anything\", recursive=True)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"sec.dat\"))\n\n\n@with_tempfile(mkdir=True)\ndef test_push_git_annex_branch_when_no_data(path):\n    path = Path(path)\n    ds = Dataset(path / \"a\").create()\n    target = mk_push_target(ds, \"target\", str(path / \"target\"),\n                            annex=False, bare=True)\n    (ds.pathobj / \"f0\").write_text(\"0\")\n    ds.save()\n    ds.push(to=\"target\", data=\"nothing\")\n    assert_in(\"git-annex\",\n              {d[\"refname:strip=2\"]\n               for d in target.for_each_ref_(fields=\"refname:strip=2\")})\n\n\n@known_failure_githubci_osx\n@with_tree(tree={\"ds\": {\"f0\": \"0\", \"f1\": \"0\", \"f2\": \"0\",\n                        \"f3\": \"1\",\n                        \"f4\": \"2\", \"f5\": \"2\"}})\ndef test_push_git_annex_branch_many_paths_same_data(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create(force=True)\n    ds.save()\n    mk_push_target(ds, \"target\", str(path / \"target\"),\n                   annex=True, bare=False)\n    nbytes = sum(ds.repo.get_content_annexinfo(paths=[f])[f][\"bytesize\"]\n                 for f in [ds.repo.pathobj / \"f0\",\n                           ds.repo.pathobj / \"f3\",\n                           ds.repo.pathobj / \"f4\"])\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        res = ds.push(to=\"target\")\n    assert_in(\"{} bytes of annex data\".format(nbytes), cml.out)\n    # 3 files point to content already covered by another file.\n    assert_result_count(res, 3,\n                        action=\"copy\", type=\"file\", status=\"notneeded\")\n\n\n@known_failure_githubci_osx\n@with_tree(tree={\"ds\": {\"f0\": \"0\"}})\ndef test_push_matching(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create(force=True)\n    ds.config.set('push.default', 'matching', where='local')\n    ds.save()\n    remote_ds = mk_push_target(ds, 'local', str(path / 'dssibling'),\n                               annex=True, bare=False)\n    # that fact that the next one even runs makes sure that we are in a better\n    # place than https://github.com/datalad/datalad/issues/4888\n    ds.push(to='local')\n    # and we pushed the commit in the current branch\n    eq_(remote_ds.get_hexsha(DEFAULT_BRANCH),\n        ds.repo.get_hexsha(DEFAULT_BRANCH))\n\n\n@known_failure_githubci_win  # https://github.com/datalad/datalad/issues/5271\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):\n    if 'DATALAD_SEED' in os.environ:\n        # we are using create-sibling-ria via the cmdline in here\n        # this will create random UUIDs for datasets\n        # however, given a fixed seed each call to this command will start\n        # with the same RNG seed, hence yield the same UUID on the same\n        # machine -- leading to a collision\n        raise SkipTest(\n            'Test incompatible with fixed random number generator seed')\n    # the aim here is this high-level test a std create-push-clone cycle for a\n    # dataset with a subdataset, with the goal to ensure that correct branches\n    # and commits are tracked, regardless of platform behavior and condition\n    # of individual clones. Nothing fancy, just that the defaults behave in\n    # sensible ways\n    from datalad.cmd import WitlessRunner as Runner\n    run = Runner().run\n\n    # create original nested dataset\n    with chpwd(origpath):\n        run(['datalad', 'create', 'super'])\n        run(['datalad', 'create', '-d', 'super', str(Path('super', 'sub'))])\n\n    # verify essential linkage properties\n    orig_super = Dataset(Path(origpath, 'super'))\n    orig_sub = Dataset(orig_super.pathobj / 'sub')\n\n    (orig_super.pathobj / 'file1.txt').write_text('some1')\n    (orig_sub.pathobj / 'file2.txt').write_text('some1')\n    with chpwd(orig_super.path):\n        run(['datalad', 'save', '--recursive'])\n\n    # TODO not yet reported clean with adjusted branches\n    #assert_repo_status(orig_super.path)\n\n    # the \"true\" branch that sub is on, and the gitsha of the HEAD commit of it\n    orig_sub_corr_branch = \\\n        orig_sub.repo.get_corresponding_branch() or orig_sub.repo.get_active_branch()\n    orig_sub_corr_commit = orig_sub.repo.get_hexsha(orig_sub_corr_branch)\n\n    # make sure the super trackes this commit\n    assert_in_results(\n        orig_super.subdatasets(),\n        path=orig_sub.path,\n        gitshasum=orig_sub_corr_commit,\n        # TODO it should also track the branch name\n        # Attempted: https://github.com/datalad/datalad/pull/3817\n        # But reverted: https://github.com/datalad/datalad/pull/4375\n    )\n\n    # publish to a store, to get into a platform-agnostic state\n    # (i.e. no impact of an annex-init of any kind)\n    store_url = 'ria+' + get_local_file_url(storepath)\n    with chpwd(orig_super.path):\n        run(['datalad', 'create-sibling-ria', '--recursive',\n             '-s', 'store', store_url])\n        run(['datalad', 'push', '--recursive', '--to', 'store'])\n\n    # we are using the 'store' sibling's URL, which should be a plain path\n    store_super = AnnexRepo(orig_super.siblings(name='store')[0]['url'], init=False)\n    store_sub = AnnexRepo(orig_sub.siblings(name='store')[0]['url'], init=False)\n\n    # both datasets in the store only carry the real branches, and nothing\n    # adjusted\n    for r in (store_super, store_sub):\n        eq_(set(r.get_branches()), set([orig_sub_corr_branch, 'git-annex']))\n\n    # and reobtain from a store\n    cloneurl = 'ria+' + get_local_file_url(str(storepath), compatibility='git')\n    with chpwd(clonepath):\n        run(['datalad', 'clone', cloneurl + '#' + orig_super.id, 'super'])\n        run(['datalad', '-C', 'super', 'get', '--recursive', '.'])\n\n    # verify that nothing has changed as a result of a push/clone cycle\n    clone_super = Dataset(Path(clonepath, 'super'))\n    clone_sub = Dataset(clone_super.pathobj / 'sub')\n    assert_in_results(\n        clone_super.subdatasets(),\n        path=clone_sub.path,\n        gitshasum=orig_sub_corr_commit,\n    )\n\n    for ds1, ds2, f in ((orig_super, clone_super, 'file1.txt'),\n                        (orig_sub, clone_sub, 'file2.txt')):\n        eq_((ds1.pathobj / f).read_text(), (ds2.pathobj / f).read_text())\n\n    # get status info that does not recursive into subdatasets, i.e. not\n    # looking for uncommitted changes\n    # we should see no modification reported\n    assert_not_in_results(\n        clone_super.status(eval_subdataset_state='commit'),\n        state='modified')\n    # and now the same for a more expensive full status\n    assert_not_in_results(\n        clone_super.status(recursive=True),\n        state='modified')\n\n\n@with_tempfile\ndef test_push_custom_summary(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create()\n\n    sib = mk_push_target(ds, \"sib\", str(path / \"sib\"), bare=False, annex=False)\n    (sib.pathobj / \"f1\").write_text(\"f1\")\n    sib.save()\n\n    (ds.pathobj / \"f2\").write_text(\"f2\")\n    ds.save()\n\n    # These options are true by default and our tests usually run with a\n    # temporary home, but set them to be sure.\n    ds.config.set(\"advice.pushUpdateRejected\", \"true\", where=\"local\")\n    ds.config.set(\"advice.pushFetchFirst\", \"true\", where=\"local\")\n    with swallow_outputs() as cmo:\n        ds.push(to=\"sib\", result_renderer=\"default\", on_failure=\"ignore\")\n        assert_in(\"Potential hints to solve\", cmo.out)\n        assert_in(\"action summary:\", cmo.out)\n",
            "header": {
                "file": {
                    "path": "datalad/core/distributed/tests/test_push.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "73c256c08",
                "new_commit_hash": "a99ae9377",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/core/distributed/tests/test_push.py"
            },
            "plus_file": {
                "path": "b/datalad/core/distributed/tests/test_push.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 50,
                            "length": 6
                        },
                        "new_range": {
                            "start": 50,
                            "length": 7
                        },
                        "section": "from datalad.utils import ("
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    Path,",
                                "type": "context"
                            },
                            {
                                "content": "    chpwd,",
                                "type": "context"
                            },
                            {
                                "content": "    path_startswith,",
                                "type": "context"
                            },
                            {
                                "content": "    swallow_outputs,",
                                "type": "added"
                            },
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.support.gitrepo import GitRepo",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.support.annexrepo import AnnexRepo",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/core/distributed/tests/test_push.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.utils import (\n    Path,\n    chpwd,\n    path_startswith,\n    swallow_outputs,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 49,
                            "end_lineno": 54,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 966,
                            "length": 3
                        },
                        "new_range": {
                            "start": 967,
                            "length": 25
                        },
                        "section": "def test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_not_in_results(",
                                "type": "context"
                            },
                            {
                                "content": "        clone_super.status(recursive=True),",
                                "type": "context"
                            },
                            {
                                "content": "        state='modified')",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "@with_tempfile",
                                "type": "added"
                            },
                            {
                                "content": "def test_push_custom_summary(path):",
                                "type": "added"
                            },
                            {
                                "content": "    path = Path(path)",
                                "type": "added"
                            },
                            {
                                "content": "    ds = Dataset(path / \"ds\").create()",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    sib = mk_push_target(ds, \"sib\", str(path / \"sib\"), bare=False, annex=False)",
                                "type": "added"
                            },
                            {
                                "content": "    (sib.pathobj / \"f1\").write_text(\"f1\")",
                                "type": "added"
                            },
                            {
                                "content": "    sib.save()",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    (ds.pathobj / \"f2\").write_text(\"f2\")",
                                "type": "added"
                            },
                            {
                                "content": "    ds.save()",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    # These options are true by default and our tests usually run with a",
                                "type": "added"
                            },
                            {
                                "content": "    # temporary home, but set them to be sure.",
                                "type": "added"
                            },
                            {
                                "content": "    ds.config.set(\"advice.pushUpdateRejected\", \"true\", where=\"local\")",
                                "type": "added"
                            },
                            {
                                "content": "    ds.config.set(\"advice.pushFetchFirst\", \"true\", where=\"local\")",
                                "type": "added"
                            },
                            {
                                "content": "    with swallow_outputs() as cmo:",
                                "type": "added"
                            },
                            {
                                "content": "        ds.push(to=\"sib\", result_renderer=\"default\", on_failure=\"ignore\")",
                                "type": "added"
                            },
                            {
                                "content": "        assert_in(\"Potential hints to solve\", cmo.out)",
                                "type": "added"
                            },
                            {
                                "content": "        assert_in(\"action summary:\", cmo.out)",
                                "type": "added"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/core/distributed/tests/test_push.py",
                            "type": "function",
                            "name": "test_push_custom_summary",
                            "content": "def test_push_custom_summary(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create()\n\n    sib = mk_push_target(ds, \"sib\", str(path / \"sib\"), bare=False, annex=False)\n    (sib.pathobj / \"f1\").write_text(\"f1\")\n    sib.save()\n\n    (ds.pathobj / \"f2\").write_text(\"f2\")\n    ds.save()\n\n    # These options are true by default and our tests usually run with a\n    # temporary home, but set them to be sure.\n    ds.config.set(\"advice.pushUpdateRejected\", \"true\", where=\"local\")\n    ds.config.set(\"advice.pushFetchFirst\", \"true\", where=\"local\")\n    with swallow_outputs() as cmo:\n        ds.push(to=\"sib\", result_renderer=\"default\", on_failure=\"ignore\")\n        assert_in(\"Potential hints to solve\", cmo.out)\n        assert_in(\"action summary:\", cmo.out)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 973,
                            "end_lineno": 991,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test interface.utils\n\n\"\"\"\n\nfrom contextlib import contextmanager\nimport logging\nfrom time import sleep\nfrom os.path import (\n    exists,\n    join as opj,\n)\nfrom datalad.tests.utils import (\n    assert_dict_equal,\n    assert_equal,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_re_in,\n    assert_repo_status,\n    assert_true,\n    ok_,\n    slow,\n    with_tempfile,\n    with_tree,\n)\nfrom datalad.utils import swallow_logs\nfrom datalad.utils import swallow_outputs\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    EnsureDataset,\n)\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import (\n    EnsureKeyChoice,\n    EnsureNone,\n    EnsureStr,\n)\nfrom ..base import Interface\nfrom ..utils import (\n    discover_dataset_trace_to_targets,\n    eval_results,\n    handle_dirty_dataset,\n)\nfrom ..results import get_status_dict\nfrom datalad.interface.base import build_doc\n\n\n__docformat__ = 'restructuredtext'\nlgr = logging.getLogger('datalad.interface.tests.test_utils')\n_dirty_modes = ('fail', 'ignore', 'save-before')\n\n\ndef _check_all_clean(ds, state):\n    assert state is not None\n    for mode in _dirty_modes:\n        # nothing wrong, nothing saved\n        handle_dirty_dataset(ds, mode)\n        assert_equal(state, ds.repo.get_hexsha())\n\n\ndef _check_auto_save(ds, orig_state):\n    handle_dirty_dataset(ds, 'ignore')\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'fail')\n    handle_dirty_dataset(ds, 'save-before')\n    state = ds.repo.get_hexsha()\n    assert_not_equal(orig_state, state)\n    _check_all_clean(ds, state)\n    return state\n\n\n@with_tempfile(mkdir=True)\ndef test_dirty(path):\n    for mode in _dirty_modes:\n        # does nothing without a dataset\n        handle_dirty_dataset(None, mode)\n    # placeholder, but not yet created\n    ds = Dataset(path)\n    # unknown mode\n    assert_raises(ValueError, handle_dirty_dataset, ds, 'MADEUP')\n    # not yet created is very dirty\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'fail')\n    handle_dirty_dataset(ds, 'ignore')\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'save-before')\n    # should yield a clean repo\n    ds.create()\n    orig_state = ds.repo.get_hexsha()\n    _check_all_clean(ds, orig_state)\n    # tainted: untracked\n    with open(opj(ds.path, 'something'), 'w') as f:\n        f.write('some')\n    # we don't want to auto-add untracked files by saving (anymore)\n    assert_raises(AssertionError, _check_auto_save, ds, orig_state)\n    # tainted: staged\n    ds.repo.add('something', git=True)\n    orig_state = _check_auto_save(ds, orig_state)\n    # tainted: submodule\n    # not added to super on purpose!\n    subds = ds.create('subds')\n    _check_all_clean(subds, subds.repo.get_hexsha())\n    assert_repo_status(ds.path)\n    # subdataset must be added as a submodule!\n    assert_equal(ds.subdatasets(result_xfm='relpaths'), ['subds'])\n\n\ndemo_hierarchy = {\n    'a': {\n        'aa': {\n            'file_aa': 'file_aa'}},\n    'b': {\n        'ba': {\n            'file_ba': 'file_ba'},\n        'bb': {\n            'bba': {\n                'bbaa': {\n                    'file_bbaa': 'file_bbaa'}},\n            'file_bb': 'file_bb'}},\n    'c': {\n        'ca': {\n            'file_ca': 'file_ca'},\n        'file_c': 'file_c'},\n    'd': {\n        'da': {\n            'file_da': 'file_da'},\n        'db': {\n            'file_db': 'file_db'},\n        'file_d': 'file_d'},\n}\n\n\ndef make_demo_hierarchy_datasets(path, tree, parent=None):\n    if parent is None:\n        parent = Dataset(path).create(force=True)\n    for node, items in tree.items():\n        if isinstance(items, dict):\n            node_path = opj(path, node)\n            nodeds = parent.create(node_path, force=True)\n            make_demo_hierarchy_datasets(node_path, items, parent=nodeds)\n    return parent\n\n\n@slow  # 74.4509s\n@with_tree(demo_hierarchy)\ndef test_save_hierarchy(path):\n    # this test doesn't use API`remove` to avoid circularities\n    ds = make_demo_hierarchy_datasets(path, demo_hierarchy)\n    ds.save(recursive=True)\n    assert_repo_status(ds.path)\n    ds_bb = Dataset(opj(ds.path, 'b', 'bb'))\n    ds_bba = Dataset(opj(ds_bb.path, 'bba'))\n    ds_bbaa = Dataset(opj(ds_bba.path, 'bbaa'))\n    # introduce a change at the lowest level\n    ds_bbaa.repo.remove('file_bbaa')\n    for d in (ds, ds_bb, ds_bba, ds_bbaa):\n        ok_(d.repo.dirty)\n    # need to give file specifically, otherwise it will simply just preserve\n    # staged changes\n    ds_bb.save(path=opj(ds_bbaa.path, 'file_bbaa'))\n    # it has saved all changes in the subtrees spanned\n    # by the given datasets, but nothing else\n    for d in (ds_bb, ds_bba, ds_bbaa):\n        assert_repo_status(d.path)\n    ok_(ds.repo.dirty)\n    # now with two modified repos\n    d = Dataset(opj(ds.path, 'd'))\n    da = Dataset(opj(d.path, 'da'))\n    da.repo.remove('file_da')\n    db = Dataset(opj(d.path, 'db'))\n    db.repo.remove('file_db')\n    # generator\n    d.save(recursive=True)\n    for d in (d, da, db):\n        assert_repo_status(d.path)\n    ok_(ds.repo.dirty)\n    # and now with files all over the place and saving\n    # all the way to the root\n    aa = Dataset(opj(ds.path, 'a', 'aa'))\n    aa.repo.remove('file_aa')\n    ba = Dataset(opj(ds.path, 'b', 'ba'))\n    ba.repo.remove('file_ba')\n    bb = Dataset(opj(ds.path, 'b', 'bb'))\n    bb.repo.remove('file_bb')\n    c = Dataset(opj(ds.path, 'c'))\n    c.repo.remove('file_c')\n    ca = Dataset(opj(ds.path, 'c', 'ca'))\n    ca.repo.remove('file_ca')\n    d = Dataset(opj(ds.path, 'd'))\n    d.repo.remove('file_d')\n    ds.save(\n        # append trailing slashes to the path to indicate that we want to\n        # have the staged content in the dataset saved, rather than only the\n        # subdataset state in the respective superds.\n        path=[opj(p, '')\n               for p in (aa.path, ba.path, bb.path, c.path, ca.path, d.path)])\n\n\n# Note: class name needs to match module's name\n@build_doc\nclass TestUtils(Interface):\n    \"\"\"TestUtil's fake command\"\"\"\n\n    result_renderer = 'tailored' # overrides None default\n    return_type = 'item-or-list' # overrides 'list'\n\n    _params_ = dict(\n        number=Parameter(\n            args=(\"-n\", \"--number\",),\n            doc=\"\"\"It's a number\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"\"specify the dataset to update.  If\n            no dataset is given, an attempt is made to identify the dataset\n            based on the input and/or the current working directory\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        result_fn=Parameter(\n            args=tuple(),   # Hide this from the cmdline parser.\n            doc=\"\"\"Generate the result records with this function\n            rather than using the default logic. `number` will be\n            passed as an argument.\"\"\"),)\n\n    @staticmethod\n    @datasetmethod(name='fake_command')\n    @eval_results\n    def __call__(number, dataset=None, result_fn=None):\n        if result_fn:\n            yield from result_fn(number)\n        else:\n            for i in range(number):\n                # this dict will need to have the minimum info\n                # required by eval_results\n                yield {'path': 'some', 'status': 'ok', 'somekey': i,\n                       'action': 'off'}\n\n\ndef test_eval_results_plus_build_doc():\n\n    # test docs\n\n    # docstring was build already:\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        TestUtils().__call__(1)\n        assert_not_in(\"Building doc for\", cml.out)\n    # docstring accessible both ways:\n    doc1 = Dataset.fake_command.__doc__\n    doc2 = TestUtils().__call__.__doc__\n\n    # docstring was built from Test_Util's definition:\n    assert_equal(doc1, doc2)\n    assert_in(\"TestUtil's fake command\", doc1)\n    assert_in(\"Parameters\", doc1)\n    assert_in(\"It's a number\", doc1)\n\n    # docstring shows correct override values of defaults in eval_params\n    assert_in(\"Default: 'tailored'\", doc1)\n    assert_in(\"Default: 'item-or-list'\", doc1)\n\n    # docstring also contains eval_result's parameters:\n    assert_in(\"result_filter\", doc1)\n    assert_in(\"return_type\", doc1)\n    assert_in(\"list\", doc1)\n    assert_in(\"None\", doc1)\n    assert_in(\"return value behavior\", doc1)\n    assert_in(\"dictionary is passed\", doc1)\n\n    # test eval_results is able to determine the call, a method of which it is\n    # decorating:\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        Dataset('/does/not/matter').fake_command(3)\n        assert_in(\"Determined class of decorated function: {}\"\n                  \"\".format(TestUtils().__class__), cml.out)\n\n    # test results:\n    result = TestUtils().__call__(2)\n    assert_equal(len(list(result)), 2)\n    result = Dataset('/does/not/matter').fake_command(3)\n    assert_equal(len(list(result)), 3)\n\n    # test absent side-effect of popping eval_defaults\n    kwargs = dict(return_type='list')\n    TestUtils().__call__(2, **kwargs)\n    assert_equal(list(kwargs), ['return_type'])\n\n    # test signature:\n    from datalad.utils import getargspec\n    assert_equal(getargspec(Dataset.fake_command)[0],\n                 ['number', 'dataset', 'result_fn'])\n    assert_equal(getargspec(TestUtils.__call__)[0],\n                 ['number', 'dataset', 'result_fn'])\n\n\ndef test_result_filter():\n    # ensure baseline without filtering\n    assert_equal(\n        [r['somekey'] for r in TestUtils().__call__(4)],\n        [0, 1, 2, 3])\n    # test two functionally equivalent ways to filter results\n    # 1. Constraint-based -- filter by exception\n    #    we have a full set of AND and OR operators for this\n    # 2. custom filer function -- filter by boolean return value\n    for filt in (\n            EnsureKeyChoice('somekey', (0, 2)),\n            lambda x: x['somekey'] in (0, 2)):\n        assert_equal(\n            [r['somekey'] for r in TestUtils().__call__(\n                4,\n                result_filter=filt)],\n            [0, 2])\n        # constraint returns full dict\n        assert_dict_equal(\n            TestUtils().__call__(\n                4,\n                result_filter=filt)[-1],\n            {'action': 'off', 'path': 'some', 'status': 'ok', 'somekey': 2})\n\n    # test more sophisticated filters that actually get to see the\n    # API call's kwargs\n    def greatfilter(res, **kwargs):\n        assert_equal(kwargs.get('dataset', 'bob'), 'awesome')\n        return True\n    TestUtils().__call__(4, dataset='awesome', result_filter=greatfilter)\n\n    def sadfilter(res, **kwargs):\n        assert_equal(kwargs.get('dataset', 'bob'), None)\n        return True\n    TestUtils().__call__(4, result_filter=sadfilter)\n\n\n@with_tree({k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n@with_tempfile(mkdir=True)\ndef test_discover_ds_trace(path, otherdir):\n    ds = make_demo_hierarchy_datasets(\n        path,\n        {k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n    a = opj(ds.path, 'a')\n    aa = opj(a, 'aa')\n    d = opj(ds.path, 'd')\n    db = opj(d, 'db')\n    # we have to check whether we get the correct hierarchy, as the test\n    # subject is also involved in this\n    assert_true(exists(opj(db, 'file_db')))\n    ds.save(recursive=True)\n    assert_repo_status(ds.path)\n    # now two datasets which are not available locally, but we\n    # know about them (e.g. from metadata)\n    dba = opj(db, 'sub', 'dba')\n    dbaa = opj(dba, 'subsub', 'dbaa')\n    for input, eds, goal in (\n            ([], None, {}),\n            ([ds.path], None, {}),\n            ([otherdir], None, {}),\n            ([opj(ds.path, 'nothere')], None, {}),\n            ([opj(d, 'nothere')], None, {}),\n            ([opj(db, 'nothere')], None, {}),\n            ([a], None,\n             {ds.path: set([a])}),\n            ([aa, a], None,\n             {ds.path: set([a]), a: set([aa])}),\n            ([db], None,\n             {ds.path: set([d]), d: set([db])}),\n            ([opj(db, 'file_db')], None,\n             {ds.path: set([d]), d: set([db])}),\n            # just a regular non-existing path\n            ([dba], None, {}),\n            # but if we inject this knowledge it must come back out\n            # as the child of the closest existing dataset\n            ([dba], [dba],\n             {ds.path: set([d]), d: set([db]), db: set([dba])}),\n            # regardless of the depth\n            ([dbaa], [dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dbaa])}),\n            ([dba, dbaa], [dba, dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dba, dbaa])}),\n            # we can simply add existing and non-existing datasets to the\n            # include list get the desired result\n            ([d, dba, dbaa], [d, dba, dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dba, dbaa])}),\n    ):\n        spec = {}\n        discover_dataset_trace_to_targets(ds.path, input, [], spec, includeds=eds)\n        assert_dict_equal(spec, goal)\n\n\n@contextmanager\ndef _swallow_outputs(isatty=True):\n    with swallow_outputs() as cmo:\n        stdout = cmo.handles[0]\n        stdout.isatty = lambda: isatty\n        yield cmo\n\n\ndef test_utils_suppress_similar():\n    tu = TestUtils()\n\n    # Check suppression boundary for straight chain of similar\n    # messages.\n\n    # yield test results immediately to make test run fast\n    sleep_dur = 0.0\n\n    def n_foo(number):\n        for i in range(number):\n            yield dict(action=\"foo\",\n                       status=\"ok\",\n                       path=\"path{}\".format(i))\n            sleep(sleep_dur)\n\n\n    with _swallow_outputs() as cmo:\n        cmo.isatty = lambda: True\n        list(tu(9, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path8\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(10, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path9\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(11, result_fn=n_foo, result_renderer=\"default\"))\n        assert_not_in(\"path10\", cmo.out)\n        assert_re_in(r\"[^-0-9]1 .* suppressed\", cmo.out, match=False)\n\n    with _swallow_outputs() as cmo:\n        # for this one test yield results slightly slower than 2Hz\n        # such that we can see each individual suppression message\n        # and no get caught by the rate limiter\n        sleep_dur = 0.51\n        list(tu(13, result_fn=n_foo, result_renderer=\"default\"))\n        assert_not_in(\"path10\", cmo.out)\n        # We see an update for each result.\n        assert_re_in(r\"1 .* suppressed\", cmo.out, match=False)\n        assert_re_in(r\"2 .* suppressed\", cmo.out, match=False)\n        assert_re_in(r\"3 .* suppressed\", cmo.out, match=False)\n\n    # make tests run fast again\n    sleep_dur = 0.0\n\n    with _swallow_outputs(isatty=False) as cmo:\n        list(tu(11, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n\n    # Check a chain of similar messages, split in half by a distinct one.\n\n    def n_foo_split_by_a_bar(number):\n        half = number // 2 - 1\n        for i in range(number):\n            yield dict(action=\"foo\",\n                       status=\"ok\",\n                       path=\"path{}\".format(i))\n            if i == half:\n                yield dict(action=\"bar\",\n                           status=\"ok\",\n                           path=\"path\")\n\n    with _swallow_outputs() as cmo:\n        list(tu(20, result_fn=n_foo_split_by_a_bar, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n        assert_in(\"path19\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(21, result_fn=n_foo_split_by_a_bar, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n        assert_not_in(\"path20\", cmo.out)\n        assert_re_in(\"[^-0-9]1 .* suppressed\", cmo.out, match=False)\n\n\nclass TestUtils2(Interface):\n    # result_renderer = custom_renderer\n    _params_ = dict(\n        number=Parameter(\n            args=(\"--path\",),\n            constraints=EnsureStr() | EnsureNone()),\n    )\n    @staticmethod\n    @eval_results\n    def __call__(path=None):\n        def logger(msg, *args):\n            return msg % args\n        if path:\n            # we will be testing for path %s\n            message = (\"all good %s\", \"my friend\")\n        else:\n            message = (\"kaboom %s %s\", \"greedy\")\n        yield get_status_dict(\n            action=\"test\",\n            status=\"ok\",\n            message=message,\n            logger=logger,\n            path=path or ''\n        )\n\n\ndef test_incorrect_msg_interpolation():\n    with assert_raises(TypeError) as cme:\n        TestUtils2().__call__()\n    # this must be our custom exception\n    assert_re_in(\"Failed to render.*kaboom.*not enough arguments\", str(cme.exception))\n\n    # there should be no exception if reported in the record path contains %\n    TestUtils2().__call__(\"%eatthis\")\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test interface.utils\n\n\"\"\"\n\nfrom contextlib import contextmanager\nimport logging\nfrom time import sleep\nfrom os.path import (\n    exists,\n    join as opj,\n)\nfrom datalad.tests.utils import (\n    assert_dict_equal,\n    assert_equal,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_re_in,\n    assert_repo_status,\n    assert_result_count,\n    assert_true,\n    ok_,\n    slow,\n    with_tempfile,\n    with_tree,\n)\nfrom datalad.utils import swallow_logs\nfrom datalad.utils import swallow_outputs\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    EnsureDataset,\n)\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import (\n    EnsureKeyChoice,\n    EnsureNone,\n    EnsureStr,\n)\nfrom ..base import Interface\nfrom ..utils import (\n    discover_dataset_trace_to_targets,\n    eval_results,\n    handle_dirty_dataset,\n)\nfrom ..results import get_status_dict\nfrom datalad.interface.base import build_doc\n\n\n__docformat__ = 'restructuredtext'\nlgr = logging.getLogger('datalad.interface.tests.test_utils')\n_dirty_modes = ('fail', 'ignore', 'save-before')\n\n\ndef _check_all_clean(ds, state):\n    assert state is not None\n    for mode in _dirty_modes:\n        # nothing wrong, nothing saved\n        handle_dirty_dataset(ds, mode)\n        assert_equal(state, ds.repo.get_hexsha())\n\n\ndef _check_auto_save(ds, orig_state):\n    handle_dirty_dataset(ds, 'ignore')\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'fail')\n    handle_dirty_dataset(ds, 'save-before')\n    state = ds.repo.get_hexsha()\n    assert_not_equal(orig_state, state)\n    _check_all_clean(ds, state)\n    return state\n\n\n@with_tempfile(mkdir=True)\ndef test_dirty(path):\n    for mode in _dirty_modes:\n        # does nothing without a dataset\n        handle_dirty_dataset(None, mode)\n    # placeholder, but not yet created\n    ds = Dataset(path)\n    # unknown mode\n    assert_raises(ValueError, handle_dirty_dataset, ds, 'MADEUP')\n    # not yet created is very dirty\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'fail')\n    handle_dirty_dataset(ds, 'ignore')\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'save-before')\n    # should yield a clean repo\n    ds.create()\n    orig_state = ds.repo.get_hexsha()\n    _check_all_clean(ds, orig_state)\n    # tainted: untracked\n    with open(opj(ds.path, 'something'), 'w') as f:\n        f.write('some')\n    # we don't want to auto-add untracked files by saving (anymore)\n    assert_raises(AssertionError, _check_auto_save, ds, orig_state)\n    # tainted: staged\n    ds.repo.add('something', git=True)\n    orig_state = _check_auto_save(ds, orig_state)\n    # tainted: submodule\n    # not added to super on purpose!\n    subds = ds.create('subds')\n    _check_all_clean(subds, subds.repo.get_hexsha())\n    assert_repo_status(ds.path)\n    # subdataset must be added as a submodule!\n    assert_equal(ds.subdatasets(result_xfm='relpaths'), ['subds'])\n\n\ndemo_hierarchy = {\n    'a': {\n        'aa': {\n            'file_aa': 'file_aa'}},\n    'b': {\n        'ba': {\n            'file_ba': 'file_ba'},\n        'bb': {\n            'bba': {\n                'bbaa': {\n                    'file_bbaa': 'file_bbaa'}},\n            'file_bb': 'file_bb'}},\n    'c': {\n        'ca': {\n            'file_ca': 'file_ca'},\n        'file_c': 'file_c'},\n    'd': {\n        'da': {\n            'file_da': 'file_da'},\n        'db': {\n            'file_db': 'file_db'},\n        'file_d': 'file_d'},\n}\n\n\ndef make_demo_hierarchy_datasets(path, tree, parent=None):\n    if parent is None:\n        parent = Dataset(path).create(force=True)\n    for node, items in tree.items():\n        if isinstance(items, dict):\n            node_path = opj(path, node)\n            nodeds = parent.create(node_path, force=True)\n            make_demo_hierarchy_datasets(node_path, items, parent=nodeds)\n    return parent\n\n\n@slow  # 74.4509s\n@with_tree(demo_hierarchy)\ndef test_save_hierarchy(path):\n    # this test doesn't use API`remove` to avoid circularities\n    ds = make_demo_hierarchy_datasets(path, demo_hierarchy)\n    ds.save(recursive=True)\n    assert_repo_status(ds.path)\n    ds_bb = Dataset(opj(ds.path, 'b', 'bb'))\n    ds_bba = Dataset(opj(ds_bb.path, 'bba'))\n    ds_bbaa = Dataset(opj(ds_bba.path, 'bbaa'))\n    # introduce a change at the lowest level\n    ds_bbaa.repo.remove('file_bbaa')\n    for d in (ds, ds_bb, ds_bba, ds_bbaa):\n        ok_(d.repo.dirty)\n    # need to give file specifically, otherwise it will simply just preserve\n    # staged changes\n    ds_bb.save(path=opj(ds_bbaa.path, 'file_bbaa'))\n    # it has saved all changes in the subtrees spanned\n    # by the given datasets, but nothing else\n    for d in (ds_bb, ds_bba, ds_bbaa):\n        assert_repo_status(d.path)\n    ok_(ds.repo.dirty)\n    # now with two modified repos\n    d = Dataset(opj(ds.path, 'd'))\n    da = Dataset(opj(d.path, 'da'))\n    da.repo.remove('file_da')\n    db = Dataset(opj(d.path, 'db'))\n    db.repo.remove('file_db')\n    # generator\n    d.save(recursive=True)\n    for d in (d, da, db):\n        assert_repo_status(d.path)\n    ok_(ds.repo.dirty)\n    # and now with files all over the place and saving\n    # all the way to the root\n    aa = Dataset(opj(ds.path, 'a', 'aa'))\n    aa.repo.remove('file_aa')\n    ba = Dataset(opj(ds.path, 'b', 'ba'))\n    ba.repo.remove('file_ba')\n    bb = Dataset(opj(ds.path, 'b', 'bb'))\n    bb.repo.remove('file_bb')\n    c = Dataset(opj(ds.path, 'c'))\n    c.repo.remove('file_c')\n    ca = Dataset(opj(ds.path, 'c', 'ca'))\n    ca.repo.remove('file_ca')\n    d = Dataset(opj(ds.path, 'd'))\n    d.repo.remove('file_d')\n    ds.save(\n        # append trailing slashes to the path to indicate that we want to\n        # have the staged content in the dataset saved, rather than only the\n        # subdataset state in the respective superds.\n        path=[opj(p, '')\n               for p in (aa.path, ba.path, bb.path, c.path, ca.path, d.path)])\n\n\n# Note: class name needs to match module's name\n@build_doc\nclass TestUtils(Interface):\n    \"\"\"TestUtil's fake command\"\"\"\n\n    result_renderer = 'tailored' # overrides None default\n    return_type = 'item-or-list' # overrides 'list'\n\n    _params_ = dict(\n        number=Parameter(\n            args=(\"-n\", \"--number\",),\n            doc=\"\"\"It's a number\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"\"specify the dataset to update.  If\n            no dataset is given, an attempt is made to identify the dataset\n            based on the input and/or the current working directory\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        result_fn=Parameter(\n            args=tuple(),   # Hide this from the cmdline parser.\n            doc=\"\"\"Generate the result records with this function\n            rather than using the default logic. `number` will be\n            passed as an argument.\"\"\"),)\n\n    @staticmethod\n    @datasetmethod(name='fake_command')\n    @eval_results\n    def __call__(number, dataset=None, result_fn=None):\n        if result_fn:\n            yield from result_fn(number)\n        else:\n            for i in range(number):\n                # this dict will need to have the minimum info\n                # required by eval_results\n                yield {'path': 'some', 'status': 'ok', 'somekey': i,\n                       'action': 'off'}\n\n\ndef test_eval_results_plus_build_doc():\n\n    # test docs\n\n    # docstring was build already:\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        TestUtils().__call__(1)\n        assert_not_in(\"Building doc for\", cml.out)\n    # docstring accessible both ways:\n    doc1 = Dataset.fake_command.__doc__\n    doc2 = TestUtils().__call__.__doc__\n\n    # docstring was built from Test_Util's definition:\n    assert_equal(doc1, doc2)\n    assert_in(\"TestUtil's fake command\", doc1)\n    assert_in(\"Parameters\", doc1)\n    assert_in(\"It's a number\", doc1)\n\n    # docstring shows correct override values of defaults in eval_params\n    assert_in(\"Default: 'tailored'\", doc1)\n    assert_in(\"Default: 'item-or-list'\", doc1)\n\n    # docstring also contains eval_result's parameters:\n    assert_in(\"result_filter\", doc1)\n    assert_in(\"return_type\", doc1)\n    assert_in(\"list\", doc1)\n    assert_in(\"None\", doc1)\n    assert_in(\"return value behavior\", doc1)\n    assert_in(\"dictionary is passed\", doc1)\n\n    # test eval_results is able to determine the call, a method of which it is\n    # decorating:\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        Dataset('/does/not/matter').fake_command(3)\n        assert_in(\"Determined class of decorated function: {}\"\n                  \"\".format(TestUtils().__class__), cml.out)\n\n    # test results:\n    result = TestUtils().__call__(2)\n    assert_equal(len(list(result)), 2)\n    result = Dataset('/does/not/matter').fake_command(3)\n    assert_equal(len(list(result)), 3)\n\n    # test absent side-effect of popping eval_defaults\n    kwargs = dict(return_type='list')\n    TestUtils().__call__(2, **kwargs)\n    assert_equal(list(kwargs), ['return_type'])\n\n    # test signature:\n    from datalad.utils import getargspec\n    assert_equal(getargspec(Dataset.fake_command)[0],\n                 ['number', 'dataset', 'result_fn'])\n    assert_equal(getargspec(TestUtils.__call__)[0],\n                 ['number', 'dataset', 'result_fn'])\n\n\ndef test_result_filter():\n    # ensure baseline without filtering\n    assert_equal(\n        [r['somekey'] for r in TestUtils().__call__(4)],\n        [0, 1, 2, 3])\n    # test two functionally equivalent ways to filter results\n    # 1. Constraint-based -- filter by exception\n    #    we have a full set of AND and OR operators for this\n    # 2. custom filer function -- filter by boolean return value\n    for filt in (\n            EnsureKeyChoice('somekey', (0, 2)),\n            lambda x: x['somekey'] in (0, 2)):\n        assert_equal(\n            [r['somekey'] for r in TestUtils().__call__(\n                4,\n                result_filter=filt)],\n            [0, 2])\n        # constraint returns full dict\n        assert_dict_equal(\n            TestUtils().__call__(\n                4,\n                result_filter=filt)[-1],\n            {'action': 'off', 'path': 'some', 'status': 'ok', 'somekey': 2})\n\n    # test more sophisticated filters that actually get to see the\n    # API call's kwargs\n    def greatfilter(res, **kwargs):\n        assert_equal(kwargs.get('dataset', 'bob'), 'awesome')\n        return True\n    TestUtils().__call__(4, dataset='awesome', result_filter=greatfilter)\n\n    def sadfilter(res, **kwargs):\n        assert_equal(kwargs.get('dataset', 'bob'), None)\n        return True\n    TestUtils().__call__(4, result_filter=sadfilter)\n\n\n@with_tree({k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n@with_tempfile(mkdir=True)\ndef test_discover_ds_trace(path, otherdir):\n    ds = make_demo_hierarchy_datasets(\n        path,\n        {k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n    a = opj(ds.path, 'a')\n    aa = opj(a, 'aa')\n    d = opj(ds.path, 'd')\n    db = opj(d, 'db')\n    # we have to check whether we get the correct hierarchy, as the test\n    # subject is also involved in this\n    assert_true(exists(opj(db, 'file_db')))\n    ds.save(recursive=True)\n    assert_repo_status(ds.path)\n    # now two datasets which are not available locally, but we\n    # know about them (e.g. from metadata)\n    dba = opj(db, 'sub', 'dba')\n    dbaa = opj(dba, 'subsub', 'dbaa')\n    for input, eds, goal in (\n            ([], None, {}),\n            ([ds.path], None, {}),\n            ([otherdir], None, {}),\n            ([opj(ds.path, 'nothere')], None, {}),\n            ([opj(d, 'nothere')], None, {}),\n            ([opj(db, 'nothere')], None, {}),\n            ([a], None,\n             {ds.path: set([a])}),\n            ([aa, a], None,\n             {ds.path: set([a]), a: set([aa])}),\n            ([db], None,\n             {ds.path: set([d]), d: set([db])}),\n            ([opj(db, 'file_db')], None,\n             {ds.path: set([d]), d: set([db])}),\n            # just a regular non-existing path\n            ([dba], None, {}),\n            # but if we inject this knowledge it must come back out\n            # as the child of the closest existing dataset\n            ([dba], [dba],\n             {ds.path: set([d]), d: set([db]), db: set([dba])}),\n            # regardless of the depth\n            ([dbaa], [dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dbaa])}),\n            ([dba, dbaa], [dba, dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dba, dbaa])}),\n            # we can simply add existing and non-existing datasets to the\n            # include list get the desired result\n            ([d, dba, dbaa], [d, dba, dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dba, dbaa])}),\n    ):\n        spec = {}\n        discover_dataset_trace_to_targets(ds.path, input, [], spec, includeds=eds)\n        assert_dict_equal(spec, goal)\n\n\n@contextmanager\ndef _swallow_outputs(isatty=True):\n    with swallow_outputs() as cmo:\n        stdout = cmo.handles[0]\n        stdout.isatty = lambda: isatty\n        yield cmo\n\n\ndef test_utils_suppress_similar():\n    tu = TestUtils()\n\n    # Check suppression boundary for straight chain of similar\n    # messages.\n\n    # yield test results immediately to make test run fast\n    sleep_dur = 0.0\n\n    def n_foo(number):\n        for i in range(number):\n            yield dict(action=\"foo\",\n                       status=\"ok\",\n                       path=\"path{}\".format(i))\n            sleep(sleep_dur)\n\n\n    with _swallow_outputs() as cmo:\n        cmo.isatty = lambda: True\n        list(tu(9, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path8\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(10, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path9\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(11, result_fn=n_foo, result_renderer=\"default\"))\n        assert_not_in(\"path10\", cmo.out)\n        assert_re_in(r\"[^-0-9]1 .* suppressed\", cmo.out, match=False)\n\n    with _swallow_outputs() as cmo:\n        # for this one test yield results slightly slower than 2Hz\n        # such that we can see each individual suppression message\n        # and no get caught by the rate limiter\n        sleep_dur = 0.51\n        list(tu(13, result_fn=n_foo, result_renderer=\"default\"))\n        assert_not_in(\"path10\", cmo.out)\n        # We see an update for each result.\n        assert_re_in(r\"1 .* suppressed\", cmo.out, match=False)\n        assert_re_in(r\"2 .* suppressed\", cmo.out, match=False)\n        assert_re_in(r\"3 .* suppressed\", cmo.out, match=False)\n\n    # make tests run fast again\n    sleep_dur = 0.0\n\n    with _swallow_outputs(isatty=False) as cmo:\n        list(tu(11, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n\n    # Check a chain of similar messages, split in half by a distinct one.\n\n    def n_foo_split_by_a_bar(number):\n        half = number // 2 - 1\n        for i in range(number):\n            yield dict(action=\"foo\",\n                       status=\"ok\",\n                       path=\"path{}\".format(i))\n            if i == half:\n                yield dict(action=\"bar\",\n                           status=\"ok\",\n                           path=\"path\")\n\n    with _swallow_outputs() as cmo:\n        list(tu(20, result_fn=n_foo_split_by_a_bar, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n        assert_in(\"path19\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(21, result_fn=n_foo_split_by_a_bar, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n        assert_not_in(\"path20\", cmo.out)\n        assert_re_in(\"[^-0-9]1 .* suppressed\", cmo.out, match=False)\n\n\nclass TestUtils2(Interface):\n    # result_renderer = custom_renderer\n    _params_ = dict(\n        number=Parameter(\n            args=(\"--path\",),\n            constraints=EnsureStr() | EnsureNone()),\n    )\n    @staticmethod\n    @eval_results\n    def __call__(path=None):\n        def logger(msg, *args):\n            return msg % args\n        if path:\n            # we will be testing for path %s\n            message = (\"all good %s\", \"my friend\")\n        else:\n            message = (\"kaboom %s %s\", \"greedy\")\n        yield get_status_dict(\n            action=\"test\",\n            status=\"ok\",\n            message=message,\n            logger=logger,\n            path=path or ''\n        )\n\n\ndef test_incorrect_msg_interpolation():\n    with assert_raises(TypeError) as cme:\n        TestUtils2().__call__()\n    # this must be our custom exception\n    assert_re_in(\"Failed to render.*kaboom.*not enough arguments\", str(cme.exception))\n\n    # there should be no exception if reported in the record path contains %\n    TestUtils2().__call__(\"%eatthis\")\n\n\nclass CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")\n\n\ndef test_custom_result_summary_renderer():\n    list(CustomSummary().__call__(\"arg\"))\n    try:\n        CustomSummary.custom_result_summary_renderer_pass_summary = True\n        list(CustomSummary().__call__(\"arg\"))\n    finally:\n        del CustomSummary.custom_result_summary_renderer_pass_summary\n",
            "header": {
                "file": {
                    "path": "datalad/interface/tests/test_utils.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "7ccdb3fce",
                "new_commit_hash": "597cf37de",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/tests/test_utils.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/tests/test_utils.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 26,
                            "length": 6
                        },
                        "new_range": {
                            "start": 26,
                            "length": 7
                        },
                        "section": "from datalad.tests.utils import ("
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_raises,",
                                "type": "context"
                            },
                            {
                                "content": "    assert_re_in,",
                                "type": "context"
                            },
                            {
                                "content": "    assert_repo_status,",
                                "type": "context"
                            },
                            {
                                "content": "    assert_result_count,",
                                "type": "added"
                            },
                            {
                                "content": "    assert_true,",
                                "type": "context"
                            },
                            {
                                "content": "    ok_,",
                                "type": "context"
                            },
                            {
                                "content": "    slow,",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_utils.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.tests.utils import (\n    assert_dict_equal,\n    assert_equal,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_re_in,\n    assert_repo_status,\n    assert_result_count,\n    assert_true,\n    ok_,\n    slow,\n    with_tempfile,\n    with_tree,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 20,
                            "end_lineno": 35,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 510,
                            "length": 3
                        },
                        "new_range": {
                            "start": 511,
                            "length": 40
                        },
                        "section": "def test_incorrect_msg_interpolation():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # there should be no exception if reported in the record path contains %",
                                "type": "context"
                            },
                            {
                                "content": "    TestUtils2().__call__(\"%eatthis\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "class CustomSummary(Interface):",
                                "type": "added"
                            },
                            {
                                "content": "    result_renderer = \"tailored\"",
                                "type": "added"
                            },
                            {
                                "content": "    _params_ = dict(x=Parameter(args=(\"x\",)))",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    @staticmethod",
                                "type": "added"
                            },
                            {
                                "content": "    @eval_results",
                                "type": "added"
                            },
                            {
                                "content": "    def __call__(x):",
                                "type": "added"
                            },
                            {
                                "content": "        for action, status in [(\"test.one\", \"ok\"),",
                                "type": "added"
                            },
                            {
                                "content": "                               (\"test.two\", \"ok\"),",
                                "type": "added"
                            },
                            {
                                "content": "                               (\"test.two\", \"notneeded\"),",
                                "type": "added"
                            },
                            {
                                "content": "                               (\"test.one\", \"ok\")]:",
                                "type": "added"
                            },
                            {
                                "content": "            yield get_status_dict(action=action, status=status,",
                                "type": "added"
                            },
                            {
                                "content": "                                  message=\"message\", x=x, logger=lgr)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    @staticmethod",
                                "type": "added"
                            },
                            {
                                "content": "    def custom_result_summary_renderer(*args):",
                                "type": "added"
                            },
                            {
                                "content": "        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",",
                                "type": "added"
                            },
                            {
                                "content": "                   False):",
                                "type": "added"
                            },
                            {
                                "content": "            action_summary = args[1]",
                                "type": "added"
                            },
                            {
                                "content": "            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})",
                                "type": "added"
                            },
                            {
                                "content": "            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})",
                                "type": "added"
                            },
                            {
                                "content": "        results = args[0]",
                                "type": "added"
                            },
                            {
                                "content": "        assert_equal(len(results), 4)",
                                "type": "added"
                            },
                            {
                                "content": "        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")",
                                "type": "added"
                            },
                            {
                                "content": "        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")",
                                "type": "added"
                            },
                            {
                                "content": "        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "def test_custom_result_summary_renderer():",
                                "type": "added"
                            },
                            {
                                "content": "    list(CustomSummary().__call__(\"arg\"))",
                                "type": "added"
                            },
                            {
                                "content": "    try:",
                                "type": "added"
                            },
                            {
                                "content": "        CustomSummary.custom_result_summary_renderer_pass_summary = True",
                                "type": "added"
                            },
                            {
                                "content": "        list(CustomSummary().__call__(\"arg\"))",
                                "type": "added"
                            },
                            {
                                "content": "    finally:",
                                "type": "added"
                            },
                            {
                                "content": "        del CustomSummary.custom_result_summary_renderer_pass_summary",
                                "type": "added"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_utils.py",
                            "type": "function",
                            "name": "test_custom_result_summary_renderer",
                            "content": "def test_custom_result_summary_renderer():\n    list(CustomSummary().__call__(\"arg\"))\n    try:\n        CustomSummary.custom_result_summary_renderer_pass_summary = True\n        list(CustomSummary().__call__(\"arg\"))\n    finally:\n        del CustomSummary.custom_result_summary_renderer_pass_summary",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 544,
                            "end_lineno": 550,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/interface/tests/test_utils.py",
                            "type": "method",
                            "name": "CustomSummary.custom_result_summary_renderer",
                            "content": "class CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 531,
                            "end_lineno": 541,
                            "parent": {
                                "file_name": "datalad/interface/tests/test_utils.py",
                                "type": "class",
                                "name": "CustomSummary",
                                "content": "class CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 516,
                                "end_lineno": 541,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/interface/tests/test_utils.py",
                            "type": "method",
                            "name": "CustomSummary.__call__",
                            "content": "class CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 522,
                            "end_lineno": 528,
                            "parent": {
                                "file_name": "datalad/interface/tests/test_utils.py",
                                "type": "class",
                                "name": "CustomSummary",
                                "content": "class CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 516,
                                "end_lineno": 541,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/interface/tests/test_utils.py",
                            "type": "class",
                            "name": "CustomSummary",
                            "content": "class CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 516,
                            "end_lineno": 541,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface utility functions\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport inspect\nimport logging\nimport wrapt\nimport sys\nimport re\nfrom time import time\nfrom os import curdir\nfrom os import pardir\nfrom os import listdir\nimport os.path as op\nfrom os.path import join as opj\nfrom os.path import isdir\nfrom os.path import relpath\nfrom os.path import sep\nfrom os.path import split as psplit\nfrom itertools import chain\n\nimport json\n\n# avoid import from API to not get into circular imports\nfrom datalad.utils import with_pathsep as _with_sep  # TODO: RF whenever merge conflict is not upon us\nfrom datalad.utils import (\n    path_startswith,\n    path_is_subpath,\n    ensure_unicode,\n    getargspec,\n    get_wrapped_class,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad import cfg as dlcfg\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\n\nfrom datalad.ui import ui\nimport datalad.support.ansi_colors as ac\n\nfrom datalad.interface.base import default_logchannels\nfrom datalad.interface.base import get_allargs_as_kwargs\nfrom datalad.interface.common_opts import eval_params\nfrom datalad.interface.common_opts import eval_defaults\nfrom .results import known_result_xfms\nfrom datalad.core.local.resulthooks import (\n    get_jsonhooks_from_config,\n    match_jsonhook2result,\n    run_jsonhook,\n)\n\n\nlgr = logging.getLogger('datalad.interface.utils')\n\n\ndef cls2cmdlinename(cls):\n    \"Return the cmdline command name from an Interface class\"\n    r = re.compile(r'([a-z0-9])([A-Z])')\n    return r.sub('\\\\1-\\\\2', cls.__name__).lower()\n\n\n# TODO remove\n# only `drop` and `uninstall` are still using this\ndef handle_dirty_dataset(ds, mode, msg=None):\n    \"\"\"Detect and treat unsaved changes as instructed by `mode`\n\n    Parameters\n    ----------\n    ds : Dataset or None\n      Dataset to be inspected. Does nothing if `None`.\n    mode : {'fail', 'ignore', 'save-before'}\n      How to act upon discovering unsaved changes.\n    msg : str or None\n      Custom message to use for a potential commit.\n\n    Returns\n    -------\n    None\n    \"\"\"\n    if ds is None:\n        # nothing to be handled\n        return\n    if msg is None:\n        msg = '[DATALAD] auto-saved changes'\n\n    # make sure that all pending changes (batched annex operations, etc.)\n    # are actually reflected in Git\n    if ds.repo:\n        ds.repo.precommit()\n\n    if mode == 'ignore':\n        return\n    elif mode == 'fail':\n        if not ds.repo or ds.repo.dirty:\n            raise RuntimeError('dataset {} has unsaved changes'.format(ds))\n    elif mode == 'save-before':\n        if not ds.is_installed():\n            raise RuntimeError('dataset {} is not yet installed'.format(ds))\n        from datalad.core.local.save import Save\n        Save.__call__(dataset=ds, message=msg, updated=True)\n    else:\n        raise ValueError(\"unknown if-dirty mode '{}'\".format(mode))\n\n\ndef get_tree_roots(paths):\n    \"\"\"Return common root paths for a set of paths\n\n    This function determines the smallest set of common root\n    paths and sorts all given paths under the respective\n    root.\n\n    Returns\n    -------\n    dict\n      paths by root\n    \"\"\"\n    paths_ws = [_with_sep(p) for p in paths]\n    # sort all paths under their potential roots\n    roots = {}\n    # start from the top to get all paths down the line\n    # and collate them into as few roots as possible\n    for s in sorted(paths_ws):\n        if any([s.startswith(r) for r in roots]):\n            # this path is already covered by a known root\n            continue\n        # find all sub paths\n        subs = [p for p in paths if p.startswith(s)]\n        roots[s.rstrip(sep)] = subs\n    return roots\n\n\n# TODO remove\n# only `remove` and `uninstall` use this, the uses path `path_is_subpath`\ndef path_is_under(values, path=None):\n    \"\"\"Whether a given path is a subdirectory of any of the given test values\n\n    Parameters\n    ----------\n    values : sequence or dict\n      Paths to be tested against. This can be a dictionary in which case\n      all values from all keys will be tested against.\n    path : path or None\n      Test path. If None is given, the process' working directory is\n      used.\n\n    Returns\n    -------\n    bool\n    \"\"\"\n    if path is None:\n        from datalad.utils import getpwd\n        path = getpwd()\n    if isinstance(values, dict):\n        values = chain(*values.values())\n    path_drive, _ = op.splitdrive(path)\n    for p in values:\n        p_drive, _ = op.splitdrive(p)\n        # need to protect against unsupported use of relpath() with\n        # abspaths on windows from different drives (gh-3724)\n        if path_drive != p_drive:\n        # different drives, enough evidence for \"not under\"\n            continue\n        rpath = relpath(p, start=path)\n        if rpath == curdir \\\n                or rpath == pardir \\\n                or set(psplit(rpath)) == {pardir}:\n            # first match is enough\n            return True\n    return False\n\n\n# TODO(OPT)? YOH: from a cursory review seems like possibly an expensive function\n# whenever many paths were provided (e.g. via shell glob).\n# Might be worth testing on some usecase and py-spy'ing if notable portion\n# of time is spent.\ndef discover_dataset_trace_to_targets(basepath, targetpaths, current_trace,\n                                      spec, includeds=None):\n    \"\"\"Discover the edges and nodes in a dataset tree to given target paths\n\n    Parameters\n    ----------\n    basepath : path\n      Path to a start or top-level dataset. Really has to be a path to a\n      dataset!\n    targetpaths : list(path)\n      Any non-zero number of paths that are termination points for the\n      search algorithm. Can be paths to datasets, directories, or files\n      (and any combination thereof).\n    current_trace : list\n      For a top-level call this should probably always be `[]`\n    spec : dict\n      `content_by_ds`-style dictionary that will receive information about the\n      discovered datasets. Specifically, for each discovered dataset there\n      will be an item with its path under the key (path) of the respective\n      superdataset.\n    includeds : sequence, optional\n      Any paths given are treated as existing subdatasets, regardless of\n      whether they can be found in the filesystem. Such subdatasets will appear\n      under the key of the closest existing dataset in the `spec`.\n\n    Returns\n    -------\n    None\n      Function calls itself recursively and populates `spec` dict in-place.\n      Keys are dataset paths, values are sets of subdataset paths\n    \"\"\"\n    # convert to set for faster lookup\n    includeds = includeds if isinstance(includeds, set) else \\\n        set() if includeds is None else set(includeds)\n    # this beast walks the directory tree from a given `basepath` until\n    # it discovers any of the given `targetpaths`\n    # if it finds one, it commits any accummulated trace of visited\n    # datasets on this edge to the spec\n    valid_repo = GitRepo.is_valid_repo(basepath)\n    if valid_repo:\n        # we are passing into a new dataset, extend the dataset trace\n        current_trace = current_trace + [basepath]\n    # this edge is not done, we need to try to reach any downstream\n    # dataset\n    undiscovered_ds = set(t for t in targetpaths) # if t != basepath)\n    # whether anything in this directory matched a targetpath\n    filematch = False\n    if isdir(basepath):\n        for p in listdir(basepath):\n            p = ensure_unicode(opj(basepath, p))\n            if not isdir(p):\n                if p in targetpaths:\n                    filematch = True\n                # we cannot have anything below this one\n                continue\n            # OPT listdir might be large and we could have only few items\n            # in `targetpaths` -- so traverse only those in spec which have\n            # leading dir basepath\n            # filter targets matching this downward path\n            downward_targets = set(\n                t for t in targetpaths if path_startswith(t, p))\n            if not downward_targets:\n                continue\n            # remove the matching ones from the \"todo\" list\n            undiscovered_ds.difference_update(downward_targets)\n            # go one deeper\n            discover_dataset_trace_to_targets(\n                p, downward_targets, current_trace, spec,\n                includeds=includeds if not includeds else includeds.intersection(\n                    downward_targets))\n    undiscovered_ds = [t for t in undiscovered_ds\n                       if includeds and\n                          path_is_subpath(t, current_trace[-1]) and\n                          t in includeds]\n    if filematch or basepath in targetpaths or undiscovered_ds:\n        for i, p in enumerate(current_trace[:-1]):\n            # TODO RF prepare proper annotated path dicts\n            subds = spec.get(p, set())\n            subds.add(current_trace[i + 1])\n            spec[p] = subds\n        if undiscovered_ds:\n            spec[current_trace[-1]] = spec.get(current_trace[-1], set()).union(\n                undiscovered_ds)\n\n\ndef get_result_filter(fx):\n    \"\"\"Wrap a filter into a helper to be able to accept additional\n    arguments, if the filter doesn't support it already\"\"\"\n    _fx = fx\n    if fx and not getargspec(fx).keywords:\n        def _fx(res, **kwargs):\n            return fx(res)\n    return _fx\n\n\ndef eval_results(func):\n    \"\"\"Decorator for return value evaluation of datalad commands.\n\n    Note, this decorator is only compatible with commands that return\n    status dict sequences!\n\n    Two basic modes of operation are supported: 1) \"generator mode\" that\n    `yields` individual results, and 2) \"list mode\" that returns a sequence of\n    results. The behavior can be selected via the kwarg `return_type`.\n    Default is \"list mode\".\n\n    This decorator implements common functionality for result rendering/output,\n    error detection/handling, and logging.\n\n    Result rendering/output can be triggered via the\n    `datalad.api.result-renderer` configuration variable, or the\n    `result_renderer` keyword argument of each decorated command. Supported\n    modes are: 'default' (one line per result with action, status, path,\n    and an optional message); 'json' (one object per result, like git-annex),\n    'json_pp' (like 'json', but pretty-printed spanning multiple lines),\n    'tailored' custom output formatting provided by each command\n    class (if any).\n\n    Error detection works by inspecting the `status` item of all result\n    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'\n    will cause an IncompleteResultsError exception to be raised that carries\n    the failed actions' status dictionaries in its `failed` attribute.\n\n    Status messages will be logged automatically, by default the following\n    association of result status and log channel will be used: 'ok' (debug),\n    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger\n    instances included in the results are used to capture the origin of a\n    status report.\n\n    Parameters\n    ----------\n    func: function\n      __call__ method of a subclass of Interface,\n      i.e. a datalad command definition\n    \"\"\"\n\n    @wrapt.decorator\n    def eval_func(wrapped, instance, args, kwargs):\n        lgr.log(2, \"Entered eval_func for %s\", func)\n        # for result filters\n        # we need to produce a dict with argname/argvalue pairs for all args\n        # incl. defaults and args given as positionals\n        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)\n\n        # determine the command class associated with `wrapped`\n        wrapped_class = get_wrapped_class(wrapped)\n\n        # retrieve common options from kwargs, and fall back on the command\n        # class attributes, or general defaults if needed\n        kwargs = kwargs.copy()  # we will pop, which might cause side-effect\n        common_params = {\n            p_name: kwargs.pop(\n                # go with any explicitly given default\n                p_name,\n                # otherwise determine the command class and pull any\n                # default set in that class\n                getattr(\n                    wrapped_class,\n                    p_name,\n                    # or the common default\n                    eval_defaults[p_name]))\n            for p_name in eval_params}\n\n        # short cuts and configured setup for common options\n        return_type = common_params['return_type']\n        result_filter = get_result_filter(common_params['result_filter'])\n        # resolve string labels for transformers too\n        result_xfm = known_result_xfms.get(\n            common_params['result_xfm'],\n            # use verbatim, if not a known label\n            common_params['result_xfm'])\n        result_renderer = common_params['result_renderer']\n        # TODO remove this conditional branch entirely, done outside\n        if not result_renderer:\n            result_renderer = dlcfg.get('datalad.api.result-renderer', None)\n        # look for potential override of logging behavior\n        result_log_level = dlcfg.get('datalad.log.result-level', None)\n\n        # query cfg for defaults\n        # .is_installed and .config can be costly, so ensure we do\n        # it only once. See https://github.com/datalad/datalad/issues/3575\n        dataset_arg = allkwargs.get('dataset', None)\n        from datalad.distribution.dataset import Dataset\n        ds = dataset_arg if isinstance(dataset_arg, Dataset) \\\n            else Dataset(dataset_arg) if dataset_arg else None\n        # look for hooks\n        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)\n\n        # this internal helper function actually drives the command\n        # generator-style, it may generate an exception if desired,\n        # on incomplete results\n        def generator_func(*_args, **_kwargs):\n            # flag whether to raise an exception\n            incomplete_results = []\n            # track what actions were performed how many times\n            action_summary = {}\n\n            # if a custom summary is to be provided, collect the results\n            # of the command execution\n            results = []\n            do_custom_result_summary = result_renderer in ('tailored', 'default') \\\n                and hasattr(wrapped_class, 'custom_result_summary_renderer')\n\n            # process main results\n            for r in _process_results(\n                    # execution\n                    wrapped(*_args, **_kwargs),\n                    wrapped_class,\n                    common_params['on_failure'],\n                    # bookkeeping\n                    action_summary,\n                    incomplete_results,\n                    # communication\n                    result_renderer,\n                    result_log_level,\n                    # let renderers get to see how a command was called\n                    allkwargs):\n                for hook, spec in hooks.items():\n                    # run the hooks before we yield the result\n                    # this ensures that they are executed before\n                    # a potentially wrapper command gets to act\n                    # on them\n                    if match_jsonhook2result(hook, r, spec['match']):\n                        lgr.debug('Result %s matches hook %s', r, hook)\n                        # a hook is also a command that yields results\n                        # so yield them outside too\n                        # users need to pay attention to void infinite\n                        # loops, i.e. when a hook yields a result that\n                        # triggers that same hook again\n                        for hr in run_jsonhook(hook, spec, r, dataset_arg):\n                            # apply same logic as for main results, otherwise\n                            # any filters would only tackle the primary results\n                            # and a mixture of return values could happen\n                            if not keep_result(hr, result_filter, **allkwargs):\n                                continue\n                            hr = xfm_result(hr, result_xfm)\n                            # rationale for conditional is a few lines down\n                            if hr:\n                                yield hr\n                if not keep_result(r, result_filter, **allkwargs):\n                    continue\n                r = xfm_result(r, result_xfm)\n                # in case the result_xfm decided to not give us anything\n                # exclude it from the results. There is no particular reason\n                # to do so other than that it was established behavior when\n                # this comment was written. This will not affect any real\n                # result record\n                if r:\n                    yield r\n\n                # collect if summary is desired\n                if do_custom_result_summary:\n                    results.append(r)\n\n            # result summary before a potential exception\n            # custom first\n            if do_custom_result_summary:\n                wrapped_class.custom_result_summary_renderer(results)\n            elif result_renderer == 'default' and action_summary and \\\n                    sum(sum(s.values()) for s in action_summary.values()) > 1:\n                # give a summary in default mode, when there was more than one\n                # action performed\n                ui.message(\"action summary:\\n  {}\".format(\n                    '\\n  '.join('{} ({})'.format(\n                        act,\n                        ', '.join('{}: {}'.format(status, action_summary[act][status])\n                                  for status in sorted(action_summary[act])))\n                                for act in sorted(action_summary))))\n\n            if incomplete_results:\n                raise IncompleteResultsError(\n                    failed=incomplete_results,\n                    msg=\"Command did not complete successfully\")\n\n        if return_type == 'generator':\n            # hand over the generator\n            lgr.log(2, \"Returning generator_func from eval_func for %s\", wrapped_class)\n            return generator_func(*args, **kwargs)\n        else:\n            @wrapt.decorator\n            def return_func(wrapped_, instance_, args_, kwargs_):\n                results = wrapped_(*args_, **kwargs_)\n                if inspect.isgenerator(results):\n                    # unwind generator if there is one, this actually runs\n                    # any processing\n                    results = list(results)\n                if return_type == 'item-or-list' and \\\n                        len(results) < 2:\n                    return results[0] if results else None\n                else:\n                    return results\n            lgr.log(2, \"Returning return_func from eval_func for %s\", wrapped_class)\n            return return_func(generator_func)(*args, **kwargs)\n\n    ret = eval_func(func)\n    ret._eval_results = True\n    return ret\n\n\ndef default_result_renderer(res):\n    if res.get('status', None) != 'notneeded':\n        path = res.get('path', None)\n        if path and res.get('refds'):\n            try:\n                path = relpath(path, res['refds'])\n            except ValueError:\n                # can happen, e.g., on windows with paths from different\n                # drives. just go with the original path in this case\n                pass\n        ui.message('{action}({status}):{path}{type}{msg}'.format(\n            action=ac.color_word(\n                res.get('action', '<action-unspecified>'),\n                ac.BOLD),\n            status=ac.color_status(res.get('status', '<status-unspecified>')),\n            path=' {}'.format(path) if path else '',\n            type=' ({})'.format(\n                ac.color_word(res['type'], ac.MAGENTA)\n            ) if 'type' in res else '',\n            msg=' [{}]'.format(\n                res['message'][0] % res['message'][1:]\n                if isinstance(res['message'], tuple) else res[\n                    'message'])\n            if res.get('message', None) else ''))\n\n\ndef _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):\n    # +1 because there was the original result + nsimilar displayed.\n    n_suppressed = nsimilar - ndisplayed + 1\n    if n_suppressed > 0:\n        ts = time()\n        # rate-limit update of suppression message, with a large number\n        # of fast-paced results updating for each one can result in more\n        # CPU load than the actual processing\n        # arbitrarily go for a 2Hz update frequency -- it \"feels\" good\n        if last_ts is None or final or (ts - last_ts > 0.5):\n            ui.message('  [{} similar {} been suppressed; disable with datalad.ui.suppress-similar-results=off]'\n                       .format(n_suppressed,\n                               single_or_plural(\"message has\",\n                                                \"messages have\",\n                                                n_suppressed, False)),\n                       cr=\"\\n\" if final else \"\\r\")\n            return ts\n    return last_ts\n\n\ndef _process_results(\n        results,\n        cmd_class,\n        on_failure,\n        action_summary,\n        incomplete_results,\n        result_renderer,\n        result_log_level,\n        allkwargs):\n    # private helper pf @eval_results\n    # loop over results generated from some source and handle each\n    # of them according to the requested behavior (logging, rendering, ...)\n\n    # used to track repeated messages in the default renderer\n    last_result = None\n    last_result_ts = None\n    # which result dict keys to inspect for changes to discover repetitions\n    # of similar messages\n    repetition_keys = set(('action', 'status', 'type', 'refds'))\n    # counter for detected repetitions\n    result_repetitions = 0\n    # how many repetitions to show, before suppression kicks in\n    render_n_repetitions = \\\n        dlcfg.obtain('datalad.ui.suppress-similar-results-threshold') \\\n        if sys.stdout.isatty() \\\n        and dlcfg.obtain('datalad.ui.suppress-similar-results') \\\n        else float(\"inf\")\n\n    for res in results:\n        if not res or 'action' not in res:\n            # XXX Yarik has to no clue on how to track the origin of the\n            # record to figure out WTF, so he just skips it\n            # but MIH thinks leaving a trace of that would be good\n            lgr.debug('Drop result record without \"action\": %s', res)\n            continue\n\n        actsum = action_summary.get(res['action'], {})\n        if res['status']:\n            actsum[res['status']] = actsum.get(res['status'], 0) + 1\n            action_summary[res['action']] = actsum\n        ## log message, if there is one and a logger was given\n        msg = res.get('message', None)\n        # remove logger instance from results, as it is no longer useful\n        # after logging was done, it isn't serializable, and generally\n        # pollutes the output\n        res_lgr = res.pop('logger', None)\n        if msg and res_lgr:\n            if isinstance(res_lgr, logging.Logger):\n                # didn't get a particular log function, go with default\n                res_lgr = getattr(\n                    res_lgr,\n                    default_logchannels[res['status']]\n                    if result_log_level is None\n                    else result_log_level)\n            msg = res['message']\n            msgargs = None\n            if isinstance(msg, tuple):\n                msgargs = msg[1:]\n                msg = msg[0]\n            if 'path' in res:\n                # result path could be a path instance\n                path = str(res['path'])\n                if msgargs:\n                    # we will pass the msg for %-polation, so % should be doubled\n                    path = path.replace('%', '%%')\n                msg = '{} [{}({})]'.format(\n                    msg, res['action'], path)\n            if msgargs:\n                # support string expansion of logging to avoid runtime cost\n                try:\n                    res_lgr(msg, *msgargs)\n                except TypeError as exc:\n                    raise TypeError(\n                        \"Failed to render %r with %r from %r: %s\"\n                        % (msg, msgargs, res, exc_str(exc))\n                    )\n            else:\n                res_lgr(msg)\n\n        ## output rendering\n        # TODO RF this in a simple callable that gets passed into this function\n        if result_renderer is None or result_renderer == 'disabled':\n            pass\n        elif result_renderer == 'default':\n            trimmed_result = {k: v for k, v in res.items() if k in repetition_keys}\n            if res.get('status', None) != 'notneeded' \\\n                    and trimmed_result == last_result:\n                # this is a similar report, suppress if too many, but count it\n                result_repetitions += 1\n                if result_repetitions < render_n_repetitions:\n                    default_result_renderer(res)\n                else:\n                    last_result_ts = _display_suppressed_message(\n                        result_repetitions, render_n_repetitions, last_result_ts)\n            else:\n                # this one is new, first report on any prev. suppressed results\n                # by number, and then render this fresh one\n                last_result_ts = _display_suppressed_message(\n                    result_repetitions, render_n_repetitions, last_result_ts,\n                    final=True)\n                default_result_renderer(res)\n                result_repetitions = 0\n            last_result = trimmed_result\n        elif result_renderer in ('json', 'json_pp'):\n            ui.message(json.dumps(\n                {k: v for k, v in res.items()\n                 if k not in ('logger')},\n                sort_keys=True,\n                indent=2 if result_renderer.endswith('_pp') else None,\n                default=str))\n        elif result_renderer in ('tailored', 'default'):\n            if hasattr(cmd_class, 'custom_result_renderer'):\n                cmd_class.custom_result_renderer(res, **allkwargs)\n        elif hasattr(result_renderer, '__call__'):\n            try:\n                result_renderer(res, **allkwargs)\n            except Exception as e:\n                lgr.warning('Result rendering failed for: %s [%s]',\n                            res, exc_str(e))\n        else:\n            raise ValueError('unknown result renderer \"{}\"'.format(result_renderer))\n\n        ## error handling\n        # looks for error status, and report at the end via\n        # an exception\n        if on_failure in ('continue', 'stop') \\\n                and res['status'] in ('impossible', 'error'):\n            incomplete_results.append(res)\n            if on_failure == 'stop':\n                # first fail -> that's it\n                # raise will happen after the loop\n                break\n        yield res\n    # make sure to report on any issues that we had suppressed\n    _display_suppressed_message(\n        result_repetitions, render_n_repetitions, last_result_ts, final=True)\n\n\ndef keep_result(res, rfilter, **kwargs):\n    if not rfilter:\n        return True\n    try:\n        if not rfilter(res, **kwargs):\n            # give the slightest indication which filter was employed\n            raise ValueError(\n                'excluded by filter {} with arguments {}'.format(rfilter, kwargs))\n    except ValueError as e:\n        # make sure to report the excluded result to massively improve\n        # debugging experience\n        lgr.debug('Not reporting result (%s): %s', exc_str(e), res)\n        return False\n    return True\n\n\ndef xfm_result(res, xfm):\n    if not xfm:\n        return res\n\n    return xfm(res)\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface utility functions\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport inspect\nimport logging\nimport wrapt\nimport sys\nimport re\nfrom time import time\nfrom os import curdir\nfrom os import pardir\nfrom os import listdir\nimport os.path as op\nfrom os.path import join as opj\nfrom os.path import isdir\nfrom os.path import relpath\nfrom os.path import sep\nfrom os.path import split as psplit\nfrom itertools import chain\n\nimport json\n\n# avoid import from API to not get into circular imports\nfrom datalad.utils import with_pathsep as _with_sep  # TODO: RF whenever merge conflict is not upon us\nfrom datalad.utils import (\n    path_startswith,\n    path_is_subpath,\n    ensure_unicode,\n    getargspec,\n    get_wrapped_class,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad import cfg as dlcfg\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\n\nfrom datalad.ui import ui\nimport datalad.support.ansi_colors as ac\n\nfrom datalad.interface.base import default_logchannels\nfrom datalad.interface.base import get_allargs_as_kwargs\nfrom datalad.interface.common_opts import eval_params\nfrom datalad.interface.common_opts import eval_defaults\nfrom .results import known_result_xfms\nfrom datalad.core.local.resulthooks import (\n    get_jsonhooks_from_config,\n    match_jsonhook2result,\n    run_jsonhook,\n)\n\n\nlgr = logging.getLogger('datalad.interface.utils')\n\n\ndef cls2cmdlinename(cls):\n    \"Return the cmdline command name from an Interface class\"\n    r = re.compile(r'([a-z0-9])([A-Z])')\n    return r.sub('\\\\1-\\\\2', cls.__name__).lower()\n\n\n# TODO remove\n# only `drop` and `uninstall` are still using this\ndef handle_dirty_dataset(ds, mode, msg=None):\n    \"\"\"Detect and treat unsaved changes as instructed by `mode`\n\n    Parameters\n    ----------\n    ds : Dataset or None\n      Dataset to be inspected. Does nothing if `None`.\n    mode : {'fail', 'ignore', 'save-before'}\n      How to act upon discovering unsaved changes.\n    msg : str or None\n      Custom message to use for a potential commit.\n\n    Returns\n    -------\n    None\n    \"\"\"\n    if ds is None:\n        # nothing to be handled\n        return\n    if msg is None:\n        msg = '[DATALAD] auto-saved changes'\n\n    # make sure that all pending changes (batched annex operations, etc.)\n    # are actually reflected in Git\n    if ds.repo:\n        ds.repo.precommit()\n\n    if mode == 'ignore':\n        return\n    elif mode == 'fail':\n        if not ds.repo or ds.repo.dirty:\n            raise RuntimeError('dataset {} has unsaved changes'.format(ds))\n    elif mode == 'save-before':\n        if not ds.is_installed():\n            raise RuntimeError('dataset {} is not yet installed'.format(ds))\n        from datalad.core.local.save import Save\n        Save.__call__(dataset=ds, message=msg, updated=True)\n    else:\n        raise ValueError(\"unknown if-dirty mode '{}'\".format(mode))\n\n\ndef get_tree_roots(paths):\n    \"\"\"Return common root paths for a set of paths\n\n    This function determines the smallest set of common root\n    paths and sorts all given paths under the respective\n    root.\n\n    Returns\n    -------\n    dict\n      paths by root\n    \"\"\"\n    paths_ws = [_with_sep(p) for p in paths]\n    # sort all paths under their potential roots\n    roots = {}\n    # start from the top to get all paths down the line\n    # and collate them into as few roots as possible\n    for s in sorted(paths_ws):\n        if any([s.startswith(r) for r in roots]):\n            # this path is already covered by a known root\n            continue\n        # find all sub paths\n        subs = [p for p in paths if p.startswith(s)]\n        roots[s.rstrip(sep)] = subs\n    return roots\n\n\n# TODO remove\n# only `remove` and `uninstall` use this, the uses path `path_is_subpath`\ndef path_is_under(values, path=None):\n    \"\"\"Whether a given path is a subdirectory of any of the given test values\n\n    Parameters\n    ----------\n    values : sequence or dict\n      Paths to be tested against. This can be a dictionary in which case\n      all values from all keys will be tested against.\n    path : path or None\n      Test path. If None is given, the process' working directory is\n      used.\n\n    Returns\n    -------\n    bool\n    \"\"\"\n    if path is None:\n        from datalad.utils import getpwd\n        path = getpwd()\n    if isinstance(values, dict):\n        values = chain(*values.values())\n    path_drive, _ = op.splitdrive(path)\n    for p in values:\n        p_drive, _ = op.splitdrive(p)\n        # need to protect against unsupported use of relpath() with\n        # abspaths on windows from different drives (gh-3724)\n        if path_drive != p_drive:\n        # different drives, enough evidence for \"not under\"\n            continue\n        rpath = relpath(p, start=path)\n        if rpath == curdir \\\n                or rpath == pardir \\\n                or set(psplit(rpath)) == {pardir}:\n            # first match is enough\n            return True\n    return False\n\n\n# TODO(OPT)? YOH: from a cursory review seems like possibly an expensive function\n# whenever many paths were provided (e.g. via shell glob).\n# Might be worth testing on some usecase and py-spy'ing if notable portion\n# of time is spent.\ndef discover_dataset_trace_to_targets(basepath, targetpaths, current_trace,\n                                      spec, includeds=None):\n    \"\"\"Discover the edges and nodes in a dataset tree to given target paths\n\n    Parameters\n    ----------\n    basepath : path\n      Path to a start or top-level dataset. Really has to be a path to a\n      dataset!\n    targetpaths : list(path)\n      Any non-zero number of paths that are termination points for the\n      search algorithm. Can be paths to datasets, directories, or files\n      (and any combination thereof).\n    current_trace : list\n      For a top-level call this should probably always be `[]`\n    spec : dict\n      `content_by_ds`-style dictionary that will receive information about the\n      discovered datasets. Specifically, for each discovered dataset there\n      will be an item with its path under the key (path) of the respective\n      superdataset.\n    includeds : sequence, optional\n      Any paths given are treated as existing subdatasets, regardless of\n      whether they can be found in the filesystem. Such subdatasets will appear\n      under the key of the closest existing dataset in the `spec`.\n\n    Returns\n    -------\n    None\n      Function calls itself recursively and populates `spec` dict in-place.\n      Keys are dataset paths, values are sets of subdataset paths\n    \"\"\"\n    # convert to set for faster lookup\n    includeds = includeds if isinstance(includeds, set) else \\\n        set() if includeds is None else set(includeds)\n    # this beast walks the directory tree from a given `basepath` until\n    # it discovers any of the given `targetpaths`\n    # if it finds one, it commits any accummulated trace of visited\n    # datasets on this edge to the spec\n    valid_repo = GitRepo.is_valid_repo(basepath)\n    if valid_repo:\n        # we are passing into a new dataset, extend the dataset trace\n        current_trace = current_trace + [basepath]\n    # this edge is not done, we need to try to reach any downstream\n    # dataset\n    undiscovered_ds = set(t for t in targetpaths) # if t != basepath)\n    # whether anything in this directory matched a targetpath\n    filematch = False\n    if isdir(basepath):\n        for p in listdir(basepath):\n            p = ensure_unicode(opj(basepath, p))\n            if not isdir(p):\n                if p in targetpaths:\n                    filematch = True\n                # we cannot have anything below this one\n                continue\n            # OPT listdir might be large and we could have only few items\n            # in `targetpaths` -- so traverse only those in spec which have\n            # leading dir basepath\n            # filter targets matching this downward path\n            downward_targets = set(\n                t for t in targetpaths if path_startswith(t, p))\n            if not downward_targets:\n                continue\n            # remove the matching ones from the \"todo\" list\n            undiscovered_ds.difference_update(downward_targets)\n            # go one deeper\n            discover_dataset_trace_to_targets(\n                p, downward_targets, current_trace, spec,\n                includeds=includeds if not includeds else includeds.intersection(\n                    downward_targets))\n    undiscovered_ds = [t for t in undiscovered_ds\n                       if includeds and\n                          path_is_subpath(t, current_trace[-1]) and\n                          t in includeds]\n    if filematch or basepath in targetpaths or undiscovered_ds:\n        for i, p in enumerate(current_trace[:-1]):\n            # TODO RF prepare proper annotated path dicts\n            subds = spec.get(p, set())\n            subds.add(current_trace[i + 1])\n            spec[p] = subds\n        if undiscovered_ds:\n            spec[current_trace[-1]] = spec.get(current_trace[-1], set()).union(\n                undiscovered_ds)\n\n\ndef get_result_filter(fx):\n    \"\"\"Wrap a filter into a helper to be able to accept additional\n    arguments, if the filter doesn't support it already\"\"\"\n    _fx = fx\n    if fx and not getargspec(fx).keywords:\n        def _fx(res, **kwargs):\n            return fx(res)\n    return _fx\n\n\ndef eval_results(func):\n    \"\"\"Decorator for return value evaluation of datalad commands.\n\n    Note, this decorator is only compatible with commands that return\n    status dict sequences!\n\n    Two basic modes of operation are supported: 1) \"generator mode\" that\n    `yields` individual results, and 2) \"list mode\" that returns a sequence of\n    results. The behavior can be selected via the kwarg `return_type`.\n    Default is \"list mode\".\n\n    This decorator implements common functionality for result rendering/output,\n    error detection/handling, and logging.\n\n    Result rendering/output can be triggered via the\n    `datalad.api.result-renderer` configuration variable, or the\n    `result_renderer` keyword argument of each decorated command. Supported\n    modes are: 'default' (one line per result with action, status, path,\n    and an optional message); 'json' (one object per result, like git-annex),\n    'json_pp' (like 'json', but pretty-printed spanning multiple lines),\n    'tailored' custom output formatting provided by each command\n    class (if any).\n\n    Error detection works by inspecting the `status` item of all result\n    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'\n    will cause an IncompleteResultsError exception to be raised that carries\n    the failed actions' status dictionaries in its `failed` attribute.\n\n    Status messages will be logged automatically, by default the following\n    association of result status and log channel will be used: 'ok' (debug),\n    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger\n    instances included in the results are used to capture the origin of a\n    status report.\n\n    Parameters\n    ----------\n    func: function\n      __call__ method of a subclass of Interface,\n      i.e. a datalad command definition\n    \"\"\"\n\n    @wrapt.decorator\n    def eval_func(wrapped, instance, args, kwargs):\n        lgr.log(2, \"Entered eval_func for %s\", func)\n        # for result filters\n        # we need to produce a dict with argname/argvalue pairs for all args\n        # incl. defaults and args given as positionals\n        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)\n\n        # determine the command class associated with `wrapped`\n        wrapped_class = get_wrapped_class(wrapped)\n\n        # retrieve common options from kwargs, and fall back on the command\n        # class attributes, or general defaults if needed\n        kwargs = kwargs.copy()  # we will pop, which might cause side-effect\n        common_params = {\n            p_name: kwargs.pop(\n                # go with any explicitly given default\n                p_name,\n                # otherwise determine the command class and pull any\n                # default set in that class\n                getattr(\n                    wrapped_class,\n                    p_name,\n                    # or the common default\n                    eval_defaults[p_name]))\n            for p_name in eval_params}\n\n        # short cuts and configured setup for common options\n        return_type = common_params['return_type']\n        result_filter = get_result_filter(common_params['result_filter'])\n        # resolve string labels for transformers too\n        result_xfm = known_result_xfms.get(\n            common_params['result_xfm'],\n            # use verbatim, if not a known label\n            common_params['result_xfm'])\n        result_renderer = common_params['result_renderer']\n        # TODO remove this conditional branch entirely, done outside\n        if not result_renderer:\n            result_renderer = dlcfg.get('datalad.api.result-renderer', None)\n        # look for potential override of logging behavior\n        result_log_level = dlcfg.get('datalad.log.result-level', None)\n\n        # query cfg for defaults\n        # .is_installed and .config can be costly, so ensure we do\n        # it only once. See https://github.com/datalad/datalad/issues/3575\n        dataset_arg = allkwargs.get('dataset', None)\n        from datalad.distribution.dataset import Dataset\n        ds = dataset_arg if isinstance(dataset_arg, Dataset) \\\n            else Dataset(dataset_arg) if dataset_arg else None\n        # look for hooks\n        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)\n\n        # this internal helper function actually drives the command\n        # generator-style, it may generate an exception if desired,\n        # on incomplete results\n        def generator_func(*_args, **_kwargs):\n            # flag whether to raise an exception\n            incomplete_results = []\n            # track what actions were performed how many times\n            action_summary = {}\n\n            # if a custom summary is to be provided, collect the results\n            # of the command execution\n            results = []\n            do_custom_result_summary = result_renderer in ('tailored', 'default') \\\n                and hasattr(wrapped_class, 'custom_result_summary_renderer')\n            pass_summary = do_custom_result_summary and \\\n                getattr(wrapped_class,\n                        'custom_result_summary_renderer_pass_summary', None)\n\n            # process main results\n            for r in _process_results(\n                    # execution\n                    wrapped(*_args, **_kwargs),\n                    wrapped_class,\n                    common_params['on_failure'],\n                    # bookkeeping\n                    action_summary,\n                    incomplete_results,\n                    # communication\n                    result_renderer,\n                    result_log_level,\n                    # let renderers get to see how a command was called\n                    allkwargs):\n                for hook, spec in hooks.items():\n                    # run the hooks before we yield the result\n                    # this ensures that they are executed before\n                    # a potentially wrapper command gets to act\n                    # on them\n                    if match_jsonhook2result(hook, r, spec['match']):\n                        lgr.debug('Result %s matches hook %s', r, hook)\n                        # a hook is also a command that yields results\n                        # so yield them outside too\n                        # users need to pay attention to void infinite\n                        # loops, i.e. when a hook yields a result that\n                        # triggers that same hook again\n                        for hr in run_jsonhook(hook, spec, r, dataset_arg):\n                            # apply same logic as for main results, otherwise\n                            # any filters would only tackle the primary results\n                            # and a mixture of return values could happen\n                            if not keep_result(hr, result_filter, **allkwargs):\n                                continue\n                            hr = xfm_result(hr, result_xfm)\n                            # rationale for conditional is a few lines down\n                            if hr:\n                                yield hr\n                if not keep_result(r, result_filter, **allkwargs):\n                    continue\n                r = xfm_result(r, result_xfm)\n                # in case the result_xfm decided to not give us anything\n                # exclude it from the results. There is no particular reason\n                # to do so other than that it was established behavior when\n                # this comment was written. This will not affect any real\n                # result record\n                if r:\n                    yield r\n\n                # collect if summary is desired\n                if do_custom_result_summary:\n                    results.append(r)\n\n            # result summary before a potential exception\n            # custom first\n            if do_custom_result_summary:\n                if pass_summary:\n                    summary_args = (results, action_summary)\n                else:\n                    summary_args = (results,)\n                wrapped_class.custom_result_summary_renderer(*summary_args)\n            elif result_renderer == 'default' and action_summary and \\\n                    sum(sum(s.values()) for s in action_summary.values()) > 1:\n                # give a summary in default mode, when there was more than one\n                # action performed\n                render_action_summary(action_summary)\n\n            if incomplete_results:\n                raise IncompleteResultsError(\n                    failed=incomplete_results,\n                    msg=\"Command did not complete successfully\")\n\n        if return_type == 'generator':\n            # hand over the generator\n            lgr.log(2, \"Returning generator_func from eval_func for %s\", wrapped_class)\n            return generator_func(*args, **kwargs)\n        else:\n            @wrapt.decorator\n            def return_func(wrapped_, instance_, args_, kwargs_):\n                results = wrapped_(*args_, **kwargs_)\n                if inspect.isgenerator(results):\n                    # unwind generator if there is one, this actually runs\n                    # any processing\n                    results = list(results)\n                if return_type == 'item-or-list' and \\\n                        len(results) < 2:\n                    return results[0] if results else None\n                else:\n                    return results\n            lgr.log(2, \"Returning return_func from eval_func for %s\", wrapped_class)\n            return return_func(generator_func)(*args, **kwargs)\n\n    ret = eval_func(func)\n    ret._eval_results = True\n    return ret\n\n\ndef default_result_renderer(res):\n    if res.get('status', None) != 'notneeded':\n        path = res.get('path', None)\n        if path and res.get('refds'):\n            try:\n                path = relpath(path, res['refds'])\n            except ValueError:\n                # can happen, e.g., on windows with paths from different\n                # drives. just go with the original path in this case\n                pass\n        ui.message('{action}({status}):{path}{type}{msg}'.format(\n            action=ac.color_word(\n                res.get('action', '<action-unspecified>'),\n                ac.BOLD),\n            status=ac.color_status(res.get('status', '<status-unspecified>')),\n            path=' {}'.format(path) if path else '',\n            type=' ({})'.format(\n                ac.color_word(res['type'], ac.MAGENTA)\n            ) if 'type' in res else '',\n            msg=' [{}]'.format(\n                res['message'][0] % res['message'][1:]\n                if isinstance(res['message'], tuple) else res[\n                    'message'])\n            if res.get('message', None) else ''))\n\n\ndef render_action_summary(action_summary):\n    ui.message(\"action summary:\\n  {}\".format(\n        '\\n  '.join('{} ({})'.format(\n            act,\n            ', '.join('{}: {}'.format(status, action_summary[act][status])\n                      for status in sorted(action_summary[act])))\n                    for act in sorted(action_summary))))\n\n\n\ndef _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):\n    # +1 because there was the original result + nsimilar displayed.\n    n_suppressed = nsimilar - ndisplayed + 1\n    if n_suppressed > 0:\n        ts = time()\n        # rate-limit update of suppression message, with a large number\n        # of fast-paced results updating for each one can result in more\n        # CPU load than the actual processing\n        # arbitrarily go for a 2Hz update frequency -- it \"feels\" good\n        if last_ts is None or final or (ts - last_ts > 0.5):\n            ui.message('  [{} similar {} been suppressed; disable with datalad.ui.suppress-similar-results=off]'\n                       .format(n_suppressed,\n                               single_or_plural(\"message has\",\n                                                \"messages have\",\n                                                n_suppressed, False)),\n                       cr=\"\\n\" if final else \"\\r\")\n            return ts\n    return last_ts\n\n\ndef _process_results(\n        results,\n        cmd_class,\n        on_failure,\n        action_summary,\n        incomplete_results,\n        result_renderer,\n        result_log_level,\n        allkwargs):\n    # private helper pf @eval_results\n    # loop over results generated from some source and handle each\n    # of them according to the requested behavior (logging, rendering, ...)\n\n    # used to track repeated messages in the default renderer\n    last_result = None\n    last_result_ts = None\n    # which result dict keys to inspect for changes to discover repetitions\n    # of similar messages\n    repetition_keys = set(('action', 'status', 'type', 'refds'))\n    # counter for detected repetitions\n    result_repetitions = 0\n    # how many repetitions to show, before suppression kicks in\n    render_n_repetitions = \\\n        dlcfg.obtain('datalad.ui.suppress-similar-results-threshold') \\\n        if sys.stdout.isatty() \\\n        and dlcfg.obtain('datalad.ui.suppress-similar-results') \\\n        else float(\"inf\")\n\n    for res in results:\n        if not res or 'action' not in res:\n            # XXX Yarik has to no clue on how to track the origin of the\n            # record to figure out WTF, so he just skips it\n            # but MIH thinks leaving a trace of that would be good\n            lgr.debug('Drop result record without \"action\": %s', res)\n            continue\n\n        actsum = action_summary.get(res['action'], {})\n        if res['status']:\n            actsum[res['status']] = actsum.get(res['status'], 0) + 1\n            action_summary[res['action']] = actsum\n        ## log message, if there is one and a logger was given\n        msg = res.get('message', None)\n        # remove logger instance from results, as it is no longer useful\n        # after logging was done, it isn't serializable, and generally\n        # pollutes the output\n        res_lgr = res.pop('logger', None)\n        if msg and res_lgr:\n            if isinstance(res_lgr, logging.Logger):\n                # didn't get a particular log function, go with default\n                res_lgr = getattr(\n                    res_lgr,\n                    default_logchannels[res['status']]\n                    if result_log_level is None\n                    else result_log_level)\n            msg = res['message']\n            msgargs = None\n            if isinstance(msg, tuple):\n                msgargs = msg[1:]\n                msg = msg[0]\n            if 'path' in res:\n                # result path could be a path instance\n                path = str(res['path'])\n                if msgargs:\n                    # we will pass the msg for %-polation, so % should be doubled\n                    path = path.replace('%', '%%')\n                msg = '{} [{}({})]'.format(\n                    msg, res['action'], path)\n            if msgargs:\n                # support string expansion of logging to avoid runtime cost\n                try:\n                    res_lgr(msg, *msgargs)\n                except TypeError as exc:\n                    raise TypeError(\n                        \"Failed to render %r with %r from %r: %s\"\n                        % (msg, msgargs, res, exc_str(exc))\n                    )\n            else:\n                res_lgr(msg)\n\n        ## output rendering\n        # TODO RF this in a simple callable that gets passed into this function\n        if result_renderer is None or result_renderer == 'disabled':\n            pass\n        elif result_renderer == 'default':\n            trimmed_result = {k: v for k, v in res.items() if k in repetition_keys}\n            if res.get('status', None) != 'notneeded' \\\n                    and trimmed_result == last_result:\n                # this is a similar report, suppress if too many, but count it\n                result_repetitions += 1\n                if result_repetitions < render_n_repetitions:\n                    default_result_renderer(res)\n                else:\n                    last_result_ts = _display_suppressed_message(\n                        result_repetitions, render_n_repetitions, last_result_ts)\n            else:\n                # this one is new, first report on any prev. suppressed results\n                # by number, and then render this fresh one\n                last_result_ts = _display_suppressed_message(\n                    result_repetitions, render_n_repetitions, last_result_ts,\n                    final=True)\n                default_result_renderer(res)\n                result_repetitions = 0\n            last_result = trimmed_result\n        elif result_renderer in ('json', 'json_pp'):\n            ui.message(json.dumps(\n                {k: v for k, v in res.items()\n                 if k not in ('logger')},\n                sort_keys=True,\n                indent=2 if result_renderer.endswith('_pp') else None,\n                default=str))\n        elif result_renderer in ('tailored', 'default'):\n            if hasattr(cmd_class, 'custom_result_renderer'):\n                cmd_class.custom_result_renderer(res, **allkwargs)\n        elif hasattr(result_renderer, '__call__'):\n            try:\n                result_renderer(res, **allkwargs)\n            except Exception as e:\n                lgr.warning('Result rendering failed for: %s [%s]',\n                            res, exc_str(e))\n        else:\n            raise ValueError('unknown result renderer \"{}\"'.format(result_renderer))\n\n        ## error handling\n        # looks for error status, and report at the end via\n        # an exception\n        if on_failure in ('continue', 'stop') \\\n                and res['status'] in ('impossible', 'error'):\n            incomplete_results.append(res)\n            if on_failure == 'stop':\n                # first fail -> that's it\n                # raise will happen after the loop\n                break\n        yield res\n    # make sure to report on any issues that we had suppressed\n    _display_suppressed_message(\n        result_repetitions, render_n_repetitions, last_result_ts, final=True)\n\n\ndef keep_result(res, rfilter, **kwargs):\n    if not rfilter:\n        return True\n    try:\n        if not rfilter(res, **kwargs):\n            # give the slightest indication which filter was employed\n            raise ValueError(\n                'excluded by filter {} with arguments {}'.format(rfilter, kwargs))\n    except ValueError as e:\n        # make sure to report the excluded result to massively improve\n        # debugging experience\n        lgr.debug('Not reporting result (%s): %s', exc_str(e), res)\n        return False\n    return True\n\n\ndef xfm_result(res, xfm):\n    if not xfm:\n        return res\n\n    return xfm(res)\n",
            "header": {
                "file": {
                    "path": "datalad/interface/utils.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "25f8d07c5",
                "new_commit_hash": "02ea04381",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/utils.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/utils.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 388,
                            "length": 6
                        },
                        "new_range": {
                            "start": 388,
                            "length": 9
                        },
                        "section": "def eval_results(func):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            results = []",
                                "type": "context"
                            },
                            {
                                "content": "            do_custom_result_summary = result_renderer in ('tailored', 'default') \\",
                                "type": "context"
                            },
                            {
                                "content": "                and hasattr(wrapped_class, 'custom_result_summary_renderer')",
                                "type": "context"
                            },
                            {
                                "content": "            pass_summary = do_custom_result_summary and \\",
                                "type": "added"
                            },
                            {
                                "content": "                getattr(wrapped_class,",
                                "type": "added"
                            },
                            {
                                "content": "                        'custom_result_summary_renderer_pass_summary', None)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "            # process main results",
                                "type": "context"
                            },
                            {
                                "content": "            for r in _process_results(",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/utils.py",
                            "type": "function",
                            "name": "eval_results",
                            "content": "def eval_results(func):\n    \"\"\"Decorator for return value evaluation of datalad commands.\n\n    Note, this decorator is only compatible with commands that return\n    status dict sequences!\n\n    Two basic modes of operation are supported: 1) \"generator mode\" that\n    `yields` individual results, and 2) \"list mode\" that returns a sequence of\n    results. The behavior can be selected via the kwarg `return_type`.\n    Default is \"list mode\".\n\n    This decorator implements common functionality for result rendering/output,\n    error detection/handling, and logging.\n\n    Result rendering/output can be triggered via the\n    `datalad.api.result-renderer` configuration variable, or the\n    `result_renderer` keyword argument of each decorated command. Supported\n    modes are: 'default' (one line per result with action, status, path,\n    and an optional message); 'json' (one object per result, like git-annex),\n    'json_pp' (like 'json', but pretty-printed spanning multiple lines),\n    'tailored' custom output formatting provided by each command\n    class (if any).\n\n    Error detection works by inspecting the `status` item of all result\n    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'\n    will cause an IncompleteResultsError exception to be raised that carries\n    the failed actions' status dictionaries in its `failed` attribute.\n\n    Status messages will be logged automatically, by default the following\n    association of result status and log channel will be used: 'ok' (debug),\n    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger\n    instances included in the results are used to capture the origin of a\n    status report.\n\n    Parameters\n    ----------\n    func: function\n      __call__ method of a subclass of Interface,\n      i.e. a datalad command definition\n    \"\"\"\n\n    @wrapt.decorator\n    def eval_func(wrapped, instance, args, kwargs):\n        lgr.log(2, \"Entered eval_func for %s\", func)\n        # for result filters\n        # we need to produce a dict with argname/argvalue pairs for all args\n        # incl. defaults and args given as positionals\n        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)\n\n        # determine the command class associated with `wrapped`\n        wrapped_class = get_wrapped_class(wrapped)\n\n        # retrieve common options from kwargs, and fall back on the command\n        # class attributes, or general defaults if needed\n        kwargs = kwargs.copy()  # we will pop, which might cause side-effect\n        common_params = {\n            p_name: kwargs.pop(\n                # go with any explicitly given default\n                p_name,\n                # otherwise determine the command class and pull any\n                # default set in that class\n                getattr(\n                    wrapped_class,\n                    p_name,\n                    # or the common default\n                    eval_defaults[p_name]))\n            for p_name in eval_params}\n\n        # short cuts and configured setup for common options\n        return_type = common_params['return_type']\n        result_filter = get_result_filter(common_params['result_filter'])\n        # resolve string labels for transformers too\n        result_xfm = known_result_xfms.get(\n            common_params['result_xfm'],\n            # use verbatim, if not a known label\n            common_params['result_xfm'])\n        result_renderer = common_params['result_renderer']\n        # TODO remove this conditional branch entirely, done outside\n        if not result_renderer:\n            result_renderer = dlcfg.get('datalad.api.result-renderer', None)\n        # look for potential override of logging behavior\n        result_log_level = dlcfg.get('datalad.log.result-level', None)\n\n        # query cfg for defaults\n        # .is_installed and .config can be costly, so ensure we do\n        # it only once. See https://github.com/datalad/datalad/issues/3575\n        dataset_arg = allkwargs.get('dataset', None)\n        from datalad.distribution.dataset import Dataset\n        ds = dataset_arg if isinstance(dataset_arg, Dataset) \\\n            else Dataset(dataset_arg) if dataset_arg else None\n        # look for hooks\n        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)\n\n        # this internal helper function actually drives the command\n        # generator-style, it may generate an exception if desired,\n        # on incomplete results\n        def generator_func(*_args, **_kwargs):\n            # flag whether to raise an exception\n            incomplete_results = []\n            # track what actions were performed how many times\n            action_summary = {}\n\n            # if a custom summary is to be provided, collect the results\n            # of the command execution\n            results = []\n            do_custom_result_summary = result_renderer in ('tailored', 'default') \\\n                and hasattr(wrapped_class, 'custom_result_summary_renderer')\n            pass_summary = do_custom_result_summary and \\\n                getattr(wrapped_class,\n                        'custom_result_summary_renderer_pass_summary', None)\n\n            # process main results\n            for r in _process_results(\n                    # execution\n                    wrapped(*_args, **_kwargs),\n                    wrapped_class,\n                    common_params['on_failure'],\n                    # bookkeeping\n                    action_summary,\n                    incomplete_results,\n                    # communication\n                    result_renderer,\n                    result_log_level,\n                    # let renderers get to see how a command was called\n                    allkwargs):\n                for hook, spec in hooks.items():\n                    # run the hooks before we yield the result\n                    # this ensures that they are executed before\n                    # a potentially wrapper command gets to act\n                    # on them\n                    if match_jsonhook2result(hook, r, spec['match']):\n                        lgr.debug('Result %s matches hook %s', r, hook)\n                        # a hook is also a command that yields results\n                        # so yield them outside too\n                        # users need to pay attention to void infinite\n                        # loops, i.e. when a hook yields a result that\n                        # triggers that same hook again\n                        for hr in run_jsonhook(hook, spec, r, dataset_arg):\n                            # apply same logic as for main results, otherwise\n                            # any filters would only tackle the primary results\n                            # and a mixture of return values could happen\n                            if not keep_result(hr, result_filter, **allkwargs):\n                                continue\n                            hr = xfm_result(hr, result_xfm)\n                            # rationale for conditional is a few lines down\n                            if hr:\n                                yield hr\n                if not keep_result(r, result_filter, **allkwargs):\n                    continue\n                r = xfm_result(r, result_xfm)\n                # in case the result_xfm decided to not give us anything\n                # exclude it from the results. There is no particular reason\n                # to do so other than that it was established behavior when\n                # this comment was written. This will not affect any real\n                # result record\n                if r:\n                    yield r\n\n                # collect if summary is desired\n                if do_custom_result_summary:\n                    results.append(r)\n\n            # result summary before a potential exception\n            # custom first\n            if do_custom_result_summary:\n                if pass_summary:\n                    summary_args = (results, action_summary)\n                else:\n                    summary_args = (results,)\n                wrapped_class.custom_result_summary_renderer(*summary_args)\n            elif result_renderer == 'default' and action_summary and \\\n                    sum(sum(s.values()) for s in action_summary.values()) > 1:\n                # give a summary in default mode, when there was more than one\n                # action performed\n                render_action_summary(action_summary)\n\n            if incomplete_results:\n                raise IncompleteResultsError(\n                    failed=incomplete_results,\n                    msg=\"Command did not complete successfully\")\n\n        if return_type == 'generator':\n            # hand over the generator\n            lgr.log(2, \"Returning generator_func from eval_func for %s\", wrapped_class)\n            return generator_func(*args, **kwargs)\n        else:\n            @wrapt.decorator\n            def return_func(wrapped_, instance_, args_, kwargs_):\n                results = wrapped_(*args_, **kwargs_)\n                if inspect.isgenerator(results):\n                    # unwind generator if there is one, this actually runs\n                    # any processing\n                    results = list(results)\n                if return_type == 'item-or-list' and \\\n                        len(results) < 2:\n                    return results[0] if results else None\n                else:\n                    return results\n            lgr.log(2, \"Returning return_func from eval_func for %s\", wrapped_class)\n            return return_func(generator_func)(*args, **kwargs)\n\n    ret = eval_func(func)\n    ret._eval_results = True\n    return ret",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 284,
                            "end_lineno": 487,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 443,
                            "length": 17
                        },
                        "new_range": {
                            "start": 446,
                            "length": 16
                        },
                        "section": "def eval_results(func):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            # result summary before a potential exception",
                                "type": "context"
                            },
                            {
                                "content": "            # custom first",
                                "type": "context"
                            },
                            {
                                "content": "            if do_custom_result_summary:",
                                "type": "context"
                            },
                            {
                                "content": "                wrapped_class.custom_result_summary_renderer(results)",
                                "type": "deleted"
                            },
                            {
                                "content": "                if pass_summary:",
                                "type": "added"
                            },
                            {
                                "content": "                    summary_args = (results, action_summary)",
                                "type": "added"
                            },
                            {
                                "content": "                else:",
                                "type": "added"
                            },
                            {
                                "content": "                    summary_args = (results,)",
                                "type": "added"
                            },
                            {
                                "content": "                wrapped_class.custom_result_summary_renderer(*summary_args)",
                                "type": "added"
                            },
                            {
                                "content": "            elif result_renderer == 'default' and action_summary and \\",
                                "type": "context"
                            },
                            {
                                "content": "                    sum(sum(s.values()) for s in action_summary.values()) > 1:",
                                "type": "context"
                            },
                            {
                                "content": "                # give a summary in default mode, when there was more than one",
                                "type": "context"
                            },
                            {
                                "content": "                # action performed",
                                "type": "context"
                            },
                            {
                                "content": "                ui.message(\"action summary:\\n  {}\".format(",
                                "type": "deleted"
                            },
                            {
                                "content": "                    '\\n  '.join('{} ({})'.format(",
                                "type": "deleted"
                            },
                            {
                                "content": "                        act,",
                                "type": "deleted"
                            },
                            {
                                "content": "                        ', '.join('{}: {}'.format(status, action_summary[act][status])",
                                "type": "deleted"
                            },
                            {
                                "content": "                                  for status in sorted(action_summary[act])))",
                                "type": "deleted"
                            },
                            {
                                "content": "                                for act in sorted(action_summary))))",
                                "type": "deleted"
                            },
                            {
                                "content": "                render_action_summary(action_summary)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "            if incomplete_results:",
                                "type": "context"
                            },
                            {
                                "content": "                raise IncompleteResultsError(",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/utils.py",
                            "type": "function",
                            "name": "eval_results",
                            "content": "def eval_results(func):\n    \"\"\"Decorator for return value evaluation of datalad commands.\n\n    Note, this decorator is only compatible with commands that return\n    status dict sequences!\n\n    Two basic modes of operation are supported: 1) \"generator mode\" that\n    `yields` individual results, and 2) \"list mode\" that returns a sequence of\n    results. The behavior can be selected via the kwarg `return_type`.\n    Default is \"list mode\".\n\n    This decorator implements common functionality for result rendering/output,\n    error detection/handling, and logging.\n\n    Result rendering/output can be triggered via the\n    `datalad.api.result-renderer` configuration variable, or the\n    `result_renderer` keyword argument of each decorated command. Supported\n    modes are: 'default' (one line per result with action, status, path,\n    and an optional message); 'json' (one object per result, like git-annex),\n    'json_pp' (like 'json', but pretty-printed spanning multiple lines),\n    'tailored' custom output formatting provided by each command\n    class (if any).\n\n    Error detection works by inspecting the `status` item of all result\n    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'\n    will cause an IncompleteResultsError exception to be raised that carries\n    the failed actions' status dictionaries in its `failed` attribute.\n\n    Status messages will be logged automatically, by default the following\n    association of result status and log channel will be used: 'ok' (debug),\n    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger\n    instances included in the results are used to capture the origin of a\n    status report.\n\n    Parameters\n    ----------\n    func: function\n      __call__ method of a subclass of Interface,\n      i.e. a datalad command definition\n    \"\"\"\n\n    @wrapt.decorator\n    def eval_func(wrapped, instance, args, kwargs):\n        lgr.log(2, \"Entered eval_func for %s\", func)\n        # for result filters\n        # we need to produce a dict with argname/argvalue pairs for all args\n        # incl. defaults and args given as positionals\n        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)\n\n        # determine the command class associated with `wrapped`\n        wrapped_class = get_wrapped_class(wrapped)\n\n        # retrieve common options from kwargs, and fall back on the command\n        # class attributes, or general defaults if needed\n        kwargs = kwargs.copy()  # we will pop, which might cause side-effect\n        common_params = {\n            p_name: kwargs.pop(\n                # go with any explicitly given default\n                p_name,\n                # otherwise determine the command class and pull any\n                # default set in that class\n                getattr(\n                    wrapped_class,\n                    p_name,\n                    # or the common default\n                    eval_defaults[p_name]))\n            for p_name in eval_params}\n\n        # short cuts and configured setup for common options\n        return_type = common_params['return_type']\n        result_filter = get_result_filter(common_params['result_filter'])\n        # resolve string labels for transformers too\n        result_xfm = known_result_xfms.get(\n            common_params['result_xfm'],\n            # use verbatim, if not a known label\n            common_params['result_xfm'])\n        result_renderer = common_params['result_renderer']\n        # TODO remove this conditional branch entirely, done outside\n        if not result_renderer:\n            result_renderer = dlcfg.get('datalad.api.result-renderer', None)\n        # look for potential override of logging behavior\n        result_log_level = dlcfg.get('datalad.log.result-level', None)\n\n        # query cfg for defaults\n        # .is_installed and .config can be costly, so ensure we do\n        # it only once. See https://github.com/datalad/datalad/issues/3575\n        dataset_arg = allkwargs.get('dataset', None)\n        from datalad.distribution.dataset import Dataset\n        ds = dataset_arg if isinstance(dataset_arg, Dataset) \\\n            else Dataset(dataset_arg) if dataset_arg else None\n        # look for hooks\n        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)\n\n        # this internal helper function actually drives the command\n        # generator-style, it may generate an exception if desired,\n        # on incomplete results\n        def generator_func(*_args, **_kwargs):\n            # flag whether to raise an exception\n            incomplete_results = []\n            # track what actions were performed how many times\n            action_summary = {}\n\n            # if a custom summary is to be provided, collect the results\n            # of the command execution\n            results = []\n            do_custom_result_summary = result_renderer in ('tailored', 'default') \\\n                and hasattr(wrapped_class, 'custom_result_summary_renderer')\n\n            # process main results\n            for r in _process_results(\n                    # execution\n                    wrapped(*_args, **_kwargs),\n                    wrapped_class,\n                    common_params['on_failure'],\n                    # bookkeeping\n                    action_summary,\n                    incomplete_results,\n                    # communication\n                    result_renderer,\n                    result_log_level,\n                    # let renderers get to see how a command was called\n                    allkwargs):\n                for hook, spec in hooks.items():\n                    # run the hooks before we yield the result\n                    # this ensures that they are executed before\n                    # a potentially wrapper command gets to act\n                    # on them\n                    if match_jsonhook2result(hook, r, spec['match']):\n                        lgr.debug('Result %s matches hook %s', r, hook)\n                        # a hook is also a command that yields results\n                        # so yield them outside too\n                        # users need to pay attention to void infinite\n                        # loops, i.e. when a hook yields a result that\n                        # triggers that same hook again\n                        for hr in run_jsonhook(hook, spec, r, dataset_arg):\n                            # apply same logic as for main results, otherwise\n                            # any filters would only tackle the primary results\n                            # and a mixture of return values could happen\n                            if not keep_result(hr, result_filter, **allkwargs):\n                                continue\n                            hr = xfm_result(hr, result_xfm)\n                            # rationale for conditional is a few lines down\n                            if hr:\n                                yield hr\n                if not keep_result(r, result_filter, **allkwargs):\n                    continue\n                r = xfm_result(r, result_xfm)\n                # in case the result_xfm decided to not give us anything\n                # exclude it from the results. There is no particular reason\n                # to do so other than that it was established behavior when\n                # this comment was written. This will not affect any real\n                # result record\n                if r:\n                    yield r\n\n                # collect if summary is desired\n                if do_custom_result_summary:\n                    results.append(r)\n\n            # result summary before a potential exception\n            # custom first\n            if do_custom_result_summary:\n                wrapped_class.custom_result_summary_renderer(results)\n            elif result_renderer == 'default' and action_summary and \\\n                    sum(sum(s.values()) for s in action_summary.values()) > 1:\n                # give a summary in default mode, when there was more than one\n                # action performed\n                ui.message(\"action summary:\\n  {}\".format(\n                    '\\n  '.join('{} ({})'.format(\n                        act,\n                        ', '.join('{}: {}'.format(status, action_summary[act][status])\n                                  for status in sorted(action_summary[act])))\n                                for act in sorted(action_summary))))\n\n            if incomplete_results:\n                raise IncompleteResultsError(\n                    failed=incomplete_results,\n                    msg=\"Command did not complete successfully\")\n\n        if return_type == 'generator':\n            # hand over the generator\n            lgr.log(2, \"Returning generator_func from eval_func for %s\", wrapped_class)\n            return generator_func(*args, **kwargs)\n        else:\n            @wrapt.decorator\n            def return_func(wrapped_, instance_, args_, kwargs_):\n                results = wrapped_(*args_, **kwargs_)\n                if inspect.isgenerator(results):\n                    # unwind generator if there is one, this actually runs\n                    # any processing\n                    results = list(results)\n                if return_type == 'item-or-list' and \\\n                        len(results) < 2:\n                    return results[0] if results else None\n                else:\n                    return results\n            lgr.log(2, \"Returning return_func from eval_func for %s\", wrapped_class)\n            return return_func(generator_func)(*args, **kwargs)\n\n    ret = eval_func(func)\n    ret._eval_results = True\n    return ret",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 284,
                            "end_lineno": 485,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 511,
                            "length": 6
                        },
                        "new_range": {
                            "start": 513,
                            "length": 16
                        },
                        "section": "def default_result_renderer(res):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            if res.get('message', None) else ''))",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def render_action_summary(action_summary):",
                                "type": "added"
                            },
                            {
                                "content": "    ui.message(\"action summary:\\n  {}\".format(",
                                "type": "added"
                            },
                            {
                                "content": "        '\\n  '.join('{} ({})'.format(",
                                "type": "added"
                            },
                            {
                                "content": "            act,",
                                "type": "added"
                            },
                            {
                                "content": "            ', '.join('{}: {}'.format(status, action_summary[act][status])",
                                "type": "added"
                            },
                            {
                                "content": "                      for status in sorted(action_summary[act])))",
                                "type": "added"
                            },
                            {
                                "content": "                    for act in sorted(action_summary))))",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "def _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):",
                                "type": "context"
                            },
                            {
                                "content": "    # +1 because there was the original result + nsimilar displayed.",
                                "type": "context"
                            },
                            {
                                "content": "    n_suppressed = nsimilar - ndisplayed + 1",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/interface/utils.py",
                            "type": "function",
                            "name": "render_action_summary",
                            "content": "def render_action_summary(action_summary):\n    ui.message(\"action summary:\\n  {}\".format(\n        '\\n  '.join('{} ({})'.format(\n            act,\n            ', '.join('{}: {}'.format(status, action_summary[act][status])\n                      for status in sorted(action_summary[act])))\n                    for act in sorted(action_summary))))",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 516,
                            "end_lineno": 522,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "1c6afc45526965f755c78971af3a06c59506fe36^",
    "new_commit_hash": "1c6afc45526965f755c78971af3a06c59506fe36",
    "commit_message": "Merge pull request #5696 from kyleam/push-action-summary\n\npush: render action summary too",
    "commit_date": "2021-05-28T15:11:53+02:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "1c6afc45526965f755c78971af3a06c59506fe36",
    "test_file_codes": [
        "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test interface.utils\n\n\"\"\"\n\nfrom contextlib import contextmanager\nimport logging\nfrom time import sleep\nfrom os.path import (\n    exists,\n    join as opj,\n)\nfrom datalad.tests.utils import (\n    assert_dict_equal,\n    assert_equal,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_re_in,\n    assert_repo_status,\n    assert_result_count,\n    assert_true,\n    ok_,\n    slow,\n    with_tempfile,\n    with_tree,\n)\nfrom datalad.utils import swallow_logs\nfrom datalad.utils import swallow_outputs\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    EnsureDataset,\n)\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import (\n    EnsureKeyChoice,\n    EnsureNone,\n    EnsureStr,\n)\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import (\n    discover_dataset_trace_to_targets,\n    eval_results,\n    handle_dirty_dataset,\n)\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.base import build_doc\n\n\n__docformat__ = 'restructuredtext'\nlgr = logging.getLogger('datalad.interface.tests.test_utils')\n_dirty_modes = ('fail', 'ignore', 'save-before')\n\n\ndef _check_all_clean(ds, state):\n    assert state is not None\n    for mode in _dirty_modes:\n        # nothing wrong, nothing saved\n        handle_dirty_dataset(ds, mode)\n        assert_equal(state, ds.repo.get_hexsha())\n\n\ndef _check_auto_save(ds, orig_state):\n    handle_dirty_dataset(ds, 'ignore')\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'fail')\n    handle_dirty_dataset(ds, 'save-before')\n    state = ds.repo.get_hexsha()\n    assert_not_equal(orig_state, state)\n    _check_all_clean(ds, state)\n    return state\n\n\n@with_tempfile(mkdir=True)\ndef test_dirty(path):\n    for mode in _dirty_modes:\n        # does nothing without a dataset\n        handle_dirty_dataset(None, mode)\n    # placeholder, but not yet created\n    ds = Dataset(path)\n    # unknown mode\n    assert_raises(ValueError, handle_dirty_dataset, ds, 'MADEUP')\n    # not yet created is very dirty\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'fail')\n    handle_dirty_dataset(ds, 'ignore')\n    assert_raises(RuntimeError, handle_dirty_dataset, ds, 'save-before')\n    # should yield a clean repo\n    ds.create()\n    orig_state = ds.repo.get_hexsha()\n    _check_all_clean(ds, orig_state)\n    # tainted: untracked\n    with open(opj(ds.path, 'something'), 'w') as f:\n        f.write('some')\n    # we don't want to auto-add untracked files by saving (anymore)\n    assert_raises(AssertionError, _check_auto_save, ds, orig_state)\n    # tainted: staged\n    ds.repo.add('something', git=True)\n    orig_state = _check_auto_save(ds, orig_state)\n    # tainted: submodule\n    # not added to super on purpose!\n    subds = ds.create('subds')\n    _check_all_clean(subds, subds.repo.get_hexsha())\n    assert_repo_status(ds.path)\n    # subdataset must be added as a submodule!\n    assert_equal(ds.subdatasets(result_xfm='relpaths'), ['subds'])\n\n\ndemo_hierarchy = {\n    'a': {\n        'aa': {\n            'file_aa': 'file_aa'}},\n    'b': {\n        'ba': {\n            'file_ba': 'file_ba'},\n        'bb': {\n            'bba': {\n                'bbaa': {\n                    'file_bbaa': 'file_bbaa'}},\n            'file_bb': 'file_bb'}},\n    'c': {\n        'ca': {\n            'file_ca': 'file_ca'},\n        'file_c': 'file_c'},\n    'd': {\n        'da': {\n            'file_da': 'file_da'},\n        'db': {\n            'file_db': 'file_db'},\n        'file_d': 'file_d'},\n}\n\n\ndef make_demo_hierarchy_datasets(path, tree, parent=None):\n    if parent is None:\n        parent = Dataset(path).create(force=True)\n    for node, items in tree.items():\n        if isinstance(items, dict):\n            node_path = opj(path, node)\n            nodeds = parent.create(node_path, force=True)\n            make_demo_hierarchy_datasets(node_path, items, parent=nodeds)\n    return parent\n\n\n@slow  # 74.4509s\n@with_tree(demo_hierarchy)\ndef test_save_hierarchy(path):\n    # this test doesn't use API`remove` to avoid circularities\n    ds = make_demo_hierarchy_datasets(path, demo_hierarchy)\n    ds.save(recursive=True)\n    assert_repo_status(ds.path)\n    ds_bb = Dataset(opj(ds.path, 'b', 'bb'))\n    ds_bba = Dataset(opj(ds_bb.path, 'bba'))\n    ds_bbaa = Dataset(opj(ds_bba.path, 'bbaa'))\n    # introduce a change at the lowest level\n    ds_bbaa.repo.remove('file_bbaa')\n    for d in (ds, ds_bb, ds_bba, ds_bbaa):\n        ok_(d.repo.dirty)\n    # need to give file specifically, otherwise it will simply just preserve\n    # staged changes\n    ds_bb.save(path=opj(ds_bbaa.path, 'file_bbaa'))\n    # it has saved all changes in the subtrees spanned\n    # by the given datasets, but nothing else\n    for d in (ds_bb, ds_bba, ds_bbaa):\n        assert_repo_status(d.path)\n    ok_(ds.repo.dirty)\n    # now with two modified repos\n    d = Dataset(opj(ds.path, 'd'))\n    da = Dataset(opj(d.path, 'da'))\n    da.repo.remove('file_da')\n    db = Dataset(opj(d.path, 'db'))\n    db.repo.remove('file_db')\n    # generator\n    d.save(recursive=True)\n    for d in (d, da, db):\n        assert_repo_status(d.path)\n    ok_(ds.repo.dirty)\n    # and now with files all over the place and saving\n    # all the way to the root\n    aa = Dataset(opj(ds.path, 'a', 'aa'))\n    aa.repo.remove('file_aa')\n    ba = Dataset(opj(ds.path, 'b', 'ba'))\n    ba.repo.remove('file_ba')\n    bb = Dataset(opj(ds.path, 'b', 'bb'))\n    bb.repo.remove('file_bb')\n    c = Dataset(opj(ds.path, 'c'))\n    c.repo.remove('file_c')\n    ca = Dataset(opj(ds.path, 'c', 'ca'))\n    ca.repo.remove('file_ca')\n    d = Dataset(opj(ds.path, 'd'))\n    d.repo.remove('file_d')\n    ds.save(\n        # append trailing slashes to the path to indicate that we want to\n        # have the staged content in the dataset saved, rather than only the\n        # subdataset state in the respective superds.\n        path=[opj(p, '')\n               for p in (aa.path, ba.path, bb.path, c.path, ca.path, d.path)])\n\n\n# Note: class name needs to match module's name\n@build_doc\nclass TestUtils(Interface):\n    \"\"\"TestUtil's fake command\"\"\"\n\n    result_renderer = 'tailored' # overrides None default\n    return_type = 'item-or-list' # overrides 'list'\n\n    _params_ = dict(\n        number=Parameter(\n            args=(\"-n\", \"--number\",),\n            doc=\"\"\"It's a number\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"\"specify the dataset to update.  If\n            no dataset is given, an attempt is made to identify the dataset\n            based on the input and/or the current working directory\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        result_fn=Parameter(\n            args=tuple(),   # Hide this from the cmdline parser.\n            doc=\"\"\"Generate the result records with this function\n            rather than using the default logic. `number` will be\n            passed as an argument.\"\"\"),)\n\n    @staticmethod\n    @datasetmethod(name='fake_command')\n    @eval_results\n    def __call__(number, dataset=None, result_fn=None):\n        if result_fn:\n            yield from result_fn(number)\n        else:\n            for i in range(number):\n                # this dict will need to have the minimum info\n                # required by eval_results\n                yield {'path': 'some', 'status': 'ok', 'somekey': i,\n                       'action': 'off'}\n\n\ndef test_eval_results_plus_build_doc():\n\n    # test docs\n\n    # docstring was build already:\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        TestUtils().__call__(1)\n        assert_not_in(\"Building doc for\", cml.out)\n    # docstring accessible both ways:\n    doc1 = Dataset.fake_command.__doc__\n    doc2 = TestUtils().__call__.__doc__\n\n    # docstring was built from Test_Util's definition:\n    assert_equal(doc1, doc2)\n    assert_in(\"TestUtil's fake command\", doc1)\n    assert_in(\"Parameters\", doc1)\n    assert_in(\"It's a number\", doc1)\n\n    # docstring shows correct override values of defaults in eval_params\n    assert_in(\"Default: 'tailored'\", doc1)\n    assert_in(\"Default: 'item-or-list'\", doc1)\n\n    # docstring also contains eval_result's parameters:\n    assert_in(\"result_filter\", doc1)\n    assert_in(\"return_type\", doc1)\n    assert_in(\"list\", doc1)\n    assert_in(\"None\", doc1)\n    assert_in(\"return value behavior\", doc1)\n    assert_in(\"dictionary is passed\", doc1)\n\n    # test eval_results is able to determine the call, a method of which it is\n    # decorating:\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        Dataset('/does/not/matter').fake_command(3)\n        assert_in(\"Determined class of decorated function: {}\"\n                  \"\".format(TestUtils().__class__), cml.out)\n\n    # test results:\n    result = TestUtils().__call__(2)\n    assert_equal(len(list(result)), 2)\n    result = Dataset('/does/not/matter').fake_command(3)\n    assert_equal(len(list(result)), 3)\n\n    # test absent side-effect of popping eval_defaults\n    kwargs = dict(return_type='list')\n    TestUtils().__call__(2, **kwargs)\n    assert_equal(list(kwargs), ['return_type'])\n\n    # test signature:\n    from datalad.utils import getargspec\n    assert_equal(getargspec(Dataset.fake_command)[0],\n                 ['number', 'dataset', 'result_fn'])\n    assert_equal(getargspec(TestUtils.__call__)[0],\n                 ['number', 'dataset', 'result_fn'])\n\n\ndef test_result_filter():\n    # ensure baseline without filtering\n    assert_equal(\n        [r['somekey'] for r in TestUtils().__call__(4)],\n        [0, 1, 2, 3])\n    # test two functionally equivalent ways to filter results\n    # 1. Constraint-based -- filter by exception\n    #    we have a full set of AND and OR operators for this\n    # 2. custom filer function -- filter by boolean return value\n    for filt in (\n            EnsureKeyChoice('somekey', (0, 2)),\n            lambda x: x['somekey'] in (0, 2)):\n        assert_equal(\n            [r['somekey'] for r in TestUtils().__call__(\n                4,\n                result_filter=filt)],\n            [0, 2])\n        # constraint returns full dict\n        assert_dict_equal(\n            TestUtils().__call__(\n                4,\n                result_filter=filt)[-1],\n            {'action': 'off', 'path': 'some', 'status': 'ok', 'somekey': 2})\n\n    # test more sophisticated filters that actually get to see the\n    # API call's kwargs\n    def greatfilter(res, **kwargs):\n        assert_equal(kwargs.get('dataset', 'bob'), 'awesome')\n        return True\n    TestUtils().__call__(4, dataset='awesome', result_filter=greatfilter)\n\n    def sadfilter(res, **kwargs):\n        assert_equal(kwargs.get('dataset', 'bob'), None)\n        return True\n    TestUtils().__call__(4, result_filter=sadfilter)\n\n\n@with_tree({k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n@with_tempfile(mkdir=True)\ndef test_discover_ds_trace(path, otherdir):\n    ds = make_demo_hierarchy_datasets(\n        path,\n        {k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n    a = opj(ds.path, 'a')\n    aa = opj(a, 'aa')\n    d = opj(ds.path, 'd')\n    db = opj(d, 'db')\n    # we have to check whether we get the correct hierarchy, as the test\n    # subject is also involved in this\n    assert_true(exists(opj(db, 'file_db')))\n    ds.save(recursive=True)\n    assert_repo_status(ds.path)\n    # now two datasets which are not available locally, but we\n    # know about them (e.g. from metadata)\n    dba = opj(db, 'sub', 'dba')\n    dbaa = opj(dba, 'subsub', 'dbaa')\n    for input, eds, goal in (\n            ([], None, {}),\n            ([ds.path], None, {}),\n            ([otherdir], None, {}),\n            ([opj(ds.path, 'nothere')], None, {}),\n            ([opj(d, 'nothere')], None, {}),\n            ([opj(db, 'nothere')], None, {}),\n            ([a], None,\n             {ds.path: set([a])}),\n            ([aa, a], None,\n             {ds.path: set([a]), a: set([aa])}),\n            ([db], None,\n             {ds.path: set([d]), d: set([db])}),\n            ([opj(db, 'file_db')], None,\n             {ds.path: set([d]), d: set([db])}),\n            # just a regular non-existing path\n            ([dba], None, {}),\n            # but if we inject this knowledge it must come back out\n            # as the child of the closest existing dataset\n            ([dba], [dba],\n             {ds.path: set([d]), d: set([db]), db: set([dba])}),\n            # regardless of the depth\n            ([dbaa], [dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dbaa])}),\n            ([dba, dbaa], [dba, dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dba, dbaa])}),\n            # we can simply add existing and non-existing datasets to the\n            # include list get the desired result\n            ([d, dba, dbaa], [d, dba, dbaa],\n             {ds.path: set([d]), d: set([db]), db: set([dba, dbaa])}),\n    ):\n        spec = {}\n        discover_dataset_trace_to_targets(ds.path, input, [], spec, includeds=eds)\n        assert_dict_equal(spec, goal)\n\n\n@contextmanager\ndef _swallow_outputs(isatty=True):\n    with swallow_outputs() as cmo:\n        stdout = cmo.handles[0]\n        stdout.isatty = lambda: isatty\n        yield cmo\n\n\ndef test_utils_suppress_similar():\n    tu = TestUtils()\n\n    # Check suppression boundary for straight chain of similar\n    # messages.\n\n    # yield test results immediately to make test run fast\n    sleep_dur = 0.0\n\n    def n_foo(number):\n        for i in range(number):\n            yield dict(action=\"foo\",\n                       status=\"ok\",\n                       path=\"path{}\".format(i))\n            sleep(sleep_dur)\n\n\n    with _swallow_outputs() as cmo:\n        cmo.isatty = lambda: True\n        list(tu(9, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path8\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(10, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path9\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(11, result_fn=n_foo, result_renderer=\"default\"))\n        assert_not_in(\"path10\", cmo.out)\n        assert_re_in(r\"[^-0-9]1 .* suppressed\", cmo.out, match=False)\n\n    with _swallow_outputs() as cmo:\n        # for this one test yield results slightly slower than 2Hz\n        # such that we can see each individual suppression message\n        # and no get caught by the rate limiter\n        sleep_dur = 0.51\n        list(tu(13, result_fn=n_foo, result_renderer=\"default\"))\n        assert_not_in(\"path10\", cmo.out)\n        # We see an update for each result.\n        assert_re_in(r\"1 .* suppressed\", cmo.out, match=False)\n        assert_re_in(r\"2 .* suppressed\", cmo.out, match=False)\n        assert_re_in(r\"3 .* suppressed\", cmo.out, match=False)\n\n    # make tests run fast again\n    sleep_dur = 0.0\n\n    with _swallow_outputs(isatty=False) as cmo:\n        list(tu(11, result_fn=n_foo, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n\n    # Check a chain of similar messages, split in half by a distinct one.\n\n    def n_foo_split_by_a_bar(number):\n        half = number // 2 - 1\n        for i in range(number):\n            yield dict(action=\"foo\",\n                       status=\"ok\",\n                       path=\"path{}\".format(i))\n            if i == half:\n                yield dict(action=\"bar\",\n                           status=\"ok\",\n                           path=\"path\")\n\n    with _swallow_outputs() as cmo:\n        list(tu(20, result_fn=n_foo_split_by_a_bar, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n        assert_in(\"path19\", cmo.out)\n        assert_not_in(\"suppressed\", cmo.out)\n\n    with _swallow_outputs() as cmo:\n        list(tu(21, result_fn=n_foo_split_by_a_bar, result_renderer=\"default\"))\n        assert_in(\"path10\", cmo.out)\n        assert_not_in(\"path20\", cmo.out)\n        assert_re_in(\"[^-0-9]1 .* suppressed\", cmo.out, match=False)\n\n\nclass TestUtils2(Interface):\n    # result_renderer = custom_renderer\n    _params_ = dict(\n        number=Parameter(\n            args=(\"--path\",),\n            constraints=EnsureStr() | EnsureNone()),\n    )\n    @staticmethod\n    @eval_results\n    def __call__(path=None):\n        def logger(msg, *args):\n            return msg % args\n        if path:\n            # we will be testing for path %s\n            message = (\"all good %s\", \"my friend\")\n        else:\n            message = (\"kaboom %s %s\", \"greedy\")\n        yield get_status_dict(\n            action=\"test\",\n            status=\"ok\",\n            message=message,\n            logger=logger,\n            path=path or ''\n        )\n\n\ndef test_incorrect_msg_interpolation():\n    with assert_raises(TypeError) as cme:\n        TestUtils2().__call__()\n    # this must be our custom exception\n    assert_re_in(\"Failed to render.*kaboom.*not enough arguments\", str(cme.exception))\n\n    # there should be no exception if reported in the record path contains %\n    TestUtils2().__call__(\"%eatthis\")\n\n\nclass CustomSummary(Interface):\n    result_renderer = \"tailored\"\n    _params_ = dict(x=Parameter(args=(\"x\",)))\n\n    @staticmethod\n    @eval_results\n    def __call__(x):\n        for action, status in [(\"test.one\", \"ok\"),\n                               (\"test.two\", \"ok\"),\n                               (\"test.two\", \"notneeded\"),\n                               (\"test.one\", \"ok\")]:\n            yield get_status_dict(action=action, status=status,\n                                  message=\"message\", x=x, logger=lgr)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n            action_summary = args[1]\n            assert_equal(action_summary[\"test.one\"], {\"ok\": 2})\n            assert_equal(action_summary[\"test.two\"], {\"ok\": 1, \"notneeded\": 1})\n        results = args[0]\n        assert_equal(len(results), 4)\n        assert_result_count(results, 2, action=\"test.one\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"ok\")\n        assert_result_count(results, 1, action=\"test.two\", status=\"notneeded\")\n\n\ndef test_custom_result_summary_renderer():\n    list(CustomSummary().__call__(\"arg\"))\n    try:\n        CustomSummary.custom_result_summary_renderer_pass_summary = True\n        list(CustomSummary().__call__(\"arg\"))\n    finally:\n        del CustomSummary.custom_result_summary_renderer_pass_summary\n",
        "# -*- coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test push\n\n\"\"\"\n\nimport os\nimport logging\n\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.support.exceptions import (\n    IncompleteResultsError,\n    InsufficientArgumentsError,\n)\nfrom datalad.tests.utils import (\n    assert_false,\n    assert_in,\n    assert_in_results,\n    assert_not_in,\n    assert_not_in_results,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_status,\n    DEFAULT_BRANCH,\n    DEFAULT_REMOTE,\n    eq_,\n    known_failure_githubci_osx,\n    known_failure_githubci_win,\n    neq_,\n    ok_,\n    ok_file_has_content,\n    serve_path_via_http,\n    skip_if_adjusted_branch,\n    skip_if_on_windows,\n    skip_ssh,\n    slow,\n    swallow_logs,\n    with_tempfile,\n    with_tree,\n    SkipTest,\n)\nfrom datalad.utils import (\n    Path,\n    chpwd,\n    path_startswith,\n    swallow_outputs,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.core.distributed.clone import Clone\nfrom datalad.core.distributed.push import Push\nfrom datalad.support.network import get_local_file_url\n\nDEFAULT_REFSPEC = \"refs/heads/{0}:refs/heads/{0}\".format(DEFAULT_BRANCH)\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_call(origin, tdir):\n    ds = Dataset(origin).create()\n    # no target\n    assert_status('impossible', ds.push(on_failure='ignore'))\n    # no dataset\n    with chpwd(tdir):\n        assert_raises(InsufficientArgumentsError, Push.__call__)\n    # dataset, but outside path\n    assert_raises(IncompleteResultsError, ds.push, path=tdir)\n    # given a path constraint that doesn't match anything, will cause\n    # nothing to be done\n    assert_status('notneeded', ds.push(path=ds.pathobj / 'nothere'))\n\n    # unavailable subdataset\n    dummy_sub = ds.create('sub')\n    dummy_sub.uninstall()\n    assert_in('sub', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    # now an explicit call to publish the unavailable subdataset\n    assert_raises(ValueError, ds.push, 'sub')\n\n    target = mk_push_target(ds, 'target', tdir, annex=True)\n    # revision that doesn't exist\n    assert_raises(\n        ValueError,\n        ds.push, to='target', since='09320957509720437523')\n\n    # If a publish() user accidentally passes since='', which push() spells as\n    # since='^', the call is aborted.\n    assert_raises(\n        ValueError,\n        ds.push, to='target', since='')\n\n\ndef mk_push_target(ds, name, path, annex=True, bare=True):\n    # life could be simple, but nothing is simple on windows\n    #src.create_sibling(dst_path, name='target')\n    if annex:\n        if bare:\n            target = GitRepo(path=path, bare=True, create=True)\n            # cannot use call_annex()\n            target.call_git(['annex', 'init'])\n        else:\n            target = AnnexRepo(path, init=True, create=True)\n            if not target.is_managed_branch():\n                # for managed branches we need more fireworks->below\n                target.config.set(\n                    'receive.denyCurrentBranch', 'updateInstead',\n                    where='local')\n    else:\n        target = GitRepo(path=path, bare=bare, create=True)\n    ds.siblings('add', name=name, url=path, result_renderer=None)\n    if annex and not bare and target.is_managed_branch():\n        # maximum complication\n        # the target repo already has a commit that is unrelated\n        # to the source repo, because it has built a reference\n        # commit for the managed branch.\n        # the only sane approach is to let git-annex establish a shared\n        # history\n        ds.repo.call_annex(['sync'])\n        ds.repo.call_annex(['sync', '--cleanup'])\n    return target\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef check_push(annex, src_path, dst_path):\n    # prepare src\n    src = Dataset(src_path).create(annex=annex)\n    src_repo = src.repo\n    # push should not add branches to the local dataset\n    orig_branches = src_repo.get_branches()\n    assert_not_in('synced/' + DEFAULT_BRANCH, orig_branches)\n\n    res = src.push(on_failure='ignore')\n    assert_result_count(res, 1)\n    assert_in_results(\n        res, status='impossible',\n        message='No push target given, and none could be auto-detected, '\n        'please specify via --to')\n    eq_(orig_branches, src_repo.get_branches())\n    # target sibling\n    target = mk_push_target(src, 'target', dst_path, annex=annex)\n    eq_(orig_branches, src_repo.get_branches())\n\n    res = src.push(to=\"target\")\n    eq_(orig_branches, src_repo.get_branches())\n    assert_result_count(res, 2 if annex else 1)\n    assert_in_results(\n        res,\n        action='publish', status='ok', target='target',\n        refspec=DEFAULT_REFSPEC,\n        operations=['new-branch'])\n\n    assert_repo_status(src_repo, annex=annex)\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # configure a default merge/upstream target\n    src.config.set('branch.{}.remote'.format(DEFAULT_BRANCH),\n                   'target', where='local')\n    src.config.set('branch.{}.merge'.format(DEFAULT_BRANCH),\n                   DEFAULT_BRANCH, where='local')\n\n    # don't fail when doing it again, no explicit target specification\n    # needed anymore\n    res = src.push()\n    eq_(orig_branches, src_repo.get_branches())\n    # and nothing is pushed\n    assert_status('notneeded', res)\n\n    assert_repo_status(src_repo, annex=annex)\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # some modification:\n    (src.pathobj / 'test_mod_file').write_text(\"Some additional stuff.\")\n    src.save(to_git=True, message=\"Modified.\")\n    (src.pathobj / 'test_mod_annex_file').write_text(\"Heavy stuff.\")\n    src.save(to_git=not annex, message=\"Modified again.\")\n    assert_repo_status(src_repo, annex=annex)\n\n    # we could say since='HEAD~2' to make things fast, or we are lazy\n    # and say since='^' to indicate the state of the tracking remote\n    # which is the same, because we made to commits since the last push.\n    res = src.push(to='target', since=\"^\", jobs=2)\n    assert_in_results(\n        res,\n        action='publish', status='ok', target='target',\n        refspec=DEFAULT_REFSPEC,\n        # we get to see what happened\n        operations=['fast-forward'])\n    if annex:\n        # we got to see the copy result for the annexed files\n        assert_in_results(\n            res,\n            action='copy',\n            status='ok',\n            path=str(src.pathobj / 'test_mod_annex_file'))\n        # we published, so we can drop and reobtain\n        ok_(src_repo.file_has_content('test_mod_annex_file'))\n        src_repo.drop('test_mod_annex_file')\n        ok_(not src_repo.file_has_content('test_mod_annex_file'))\n        src_repo.get('test_mod_annex_file')\n        ok_(src_repo.file_has_content('test_mod_annex_file'))\n        ok_file_has_content(\n            src_repo.pathobj / 'test_mod_annex_file',\n            'Heavy stuff.')\n\n    eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n        list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n    if not (annex and src_repo.is_managed_branch()):\n        # the following doesn't make sense in managed branches, because\n        # a commit that could be amended is no longer the last commit\n        # of a branch after a sync has happened (which did happen\n        # during the last push above\n\n        # amend and change commit msg in order to test for force push:\n        src_repo.commit(\"amended\", options=['--amend'])\n        # push should be rejected (non-fast-forward):\n        res = src.push(to='target', since='HEAD~2', on_failure='ignore')\n        # fails before even touching the annex branch\n        assert_in_results(\n            res,\n            action='publish', status='error', target='target',\n            refspec=DEFAULT_REFSPEC,\n            operations=['rejected', 'error'])\n        # push with force=True works:\n        res = src.push(to='target', since='HEAD~2', force='gitpush')\n        assert_in_results(\n            res,\n            action='publish', status='ok', target='target',\n            refspec=DEFAULT_REFSPEC,\n            operations=['forced-update'])\n        eq_(list(target.get_branch_commits_(DEFAULT_BRANCH)),\n            list(src_repo.get_branch_commits_(DEFAULT_BRANCH)))\n\n    # we do not have more branches than we had in the beginning\n    # in particular no 'synced/<default branch>'\n    eq_(orig_branches, src_repo.get_branches())\n\n\ndef test_push():\n    yield check_push, False\n    yield check_push, True\n\n\ndef check_datasets_order(res, order='bottom-up'):\n    \"\"\"Check that all type=dataset records not violating the expected order\n\n    it is somewhat weak test, i.e. records could be produced so we\n    do not detect that order is violated, e.g. a/b c/d would satisfy\n    either although they might be neither depth nor breadth wise.  But\n    this test would allow to catch obvious violations like a, a/b, a\n    \"\"\"\n    prev = None\n    for r in res:\n        if r.get('type') != 'dataset':\n            continue\n        if prev and r['path'] != prev:\n            if order == 'bottom-up':\n                assert_false(path_startswith(r['path'], prev))\n            elif order == 'top-down':\n                assert_false(path_startswith(prev, r['path']))\n            else:\n                raise ValueError(order)\n        prev = r['path']\n\n\n@slow  # 33sec on Yarik's laptop\n@with_tempfile\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True, suffix='sub')\n@with_tempfile(mkdir=True, suffix='subnoannex')\n@with_tempfile(mkdir=True, suffix='subsub')\ndef test_push_recursive(\n        origin_path, src_path, dst_top, dst_sub, dst_subnoannex, dst_subsub):\n    # dataset with two submodules and one subsubmodule\n    origin = Dataset(origin_path).create()\n    origin_subm1 = origin.create('sub m')\n    origin_subm1.create('subsub m')\n    origin.create('subm noannex', annex=False)\n    origin.save()\n    assert_repo_status(origin.path)\n    # prepare src as a fresh clone with all subdatasets checkout out recursively\n    # running on a clone should make the test scenario more different than\n    # test_push(), even for the pieces that should be identical\n    top = Clone.__call__(source=origin.path, path=src_path)\n    subs = top.get('.', recursive=True, get_data=False, result_xfm='datasets')\n    # order for '.' should not be relied upon, so sort by path\n    sub, subsub, subnoannex = sorted(subs, key=lambda ds: ds.path)\n\n    target_top = mk_push_target(top, 'target', dst_top, annex=True)\n    # subdatasets have no remote yet, so recursive publishing should fail:\n    res = top.push(to=\"target\", recursive=True, on_failure='ignore')\n    check_datasets_order(res)\n    assert_in_results(\n        res, path=top.path, type='dataset',\n        refspec=DEFAULT_REFSPEC,\n        operations=['new-branch'], action='publish', status='ok',\n        target='target')\n    for d in (sub, subsub, subnoannex):\n        assert_in_results(\n            res, status='error', type='dataset', path=d.path,\n            message=(\"Unknown target sibling '%s'.\",\n                     'target'))\n    # now fix that and set up targets for the submodules\n    target_sub = mk_push_target(sub, 'target', dst_sub, annex=True)\n    target_subnoannex = mk_push_target(\n        subnoannex, 'target', dst_subnoannex, annex=False)\n    target_subsub = mk_push_target(subsub, 'target', dst_subsub, annex=True)\n\n    # and same push call as above\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    # topds skipped\n    assert_in_results(\n        res, path=top.path, type='dataset',\n        action='publish', status='notneeded', target='target')\n    # the rest pushed\n    for d in (sub, subsub, subnoannex):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # all corresponding branches match across all datasets\n    for s, d in zip((top, sub, subnoannex, subsub),\n                    (target_top, target_sub, target_subnoannex,\n                     target_subsub)):\n        eq_(list(s.repo.get_branch_commits_(DEFAULT_BRANCH)),\n            list(d.get_branch_commits_(DEFAULT_BRANCH)))\n        if s != subnoannex:\n            eq_(list(s.repo.get_branch_commits_(\"git-annex\")),\n                list(d.get_branch_commits_(\"git-annex\")))\n\n    # rerun should not result in further pushes of the default branch\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    assert_not_in_results(\n        res, status='ok', refspec=DEFAULT_REFSPEC)\n    assert_in_results(\n        res, status='notneeded', refspec=DEFAULT_REFSPEC)\n\n    # now annex a file in subsub\n    test_copy_file = subsub.pathobj / 'test_mod_annex_file'\n    test_copy_file.write_text(\"Heavy stuff.\")\n    # save all the way up\n    assert_status(\n        ('ok', 'notneeded'),\n        top.save(message='subsub got something', recursive=True))\n    assert_repo_status(top.path)\n    # publish straight up, should be smart by default\n    res = top.push(to=\"target\", recursive=True)\n    check_datasets_order(res)\n    # we see 3 out of 4 datasets pushed (sub noannex was left unchanged)\n    for d in (top, sub, subsub):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # file content copied too\n    assert_in_results(\n        res,\n        action='copy',\n        status='ok',\n        path=str(test_copy_file))\n    # verify it is accessible, drop and bring back\n    assert_status('ok', top.drop(str(test_copy_file)))\n    ok_(not subsub.repo.file_has_content('test_mod_annex_file'))\n    top.get(test_copy_file)\n    ok_file_has_content(test_copy_file, 'Heavy stuff.')\n\n    # make two modification\n    (sub.pathobj / 'test_mod_annex_file').write_text('annex')\n    (subnoannex.pathobj / 'test_mod_file').write_text('git')\n    # save separately\n    top.save(sub.pathobj, message='annexadd', recursive=True)\n    top.save(subnoannex.pathobj, message='gitadd', recursive=True)\n    # now only publish the latter one\n    res = top.push(to=\"target\", since=DEFAULT_BRANCH + '~1', recursive=True)\n    # nothing copied, no reports on the other modification\n    assert_not_in_results(res, action='copy')\n    assert_not_in_results(res, path=sub.path)\n    for d in (top, subnoannex):\n        assert_in_results(\n            res, status='ok', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n    # an unconditional push should now pick up the remaining changes\n    res = top.push(to=\"target\", recursive=True)\n    assert_in_results(\n        res,\n        action='copy',\n        status='ok',\n        path=str(sub.pathobj / 'test_mod_annex_file'))\n    assert_in_results(\n        res, status='ok', type='dataset', path=sub.path,\n        refspec=DEFAULT_REFSPEC)\n    for d in (top, subnoannex, subsub):\n        assert_in_results(\n            res, status='notneeded', type='dataset', path=d.path,\n            refspec=DEFAULT_REFSPEC)\n\n    # if noannex target gets some annex, we still should not fail to push\n    target_subnoannex.call_git(['annex', 'init'])\n    # just to ensure that we do need something to push\n    (subnoannex.pathobj / \"newfile\").write_text(\"content\")\n    subnoannex.save()\n    res = subnoannex.push(to=\"target\")\n    assert_in_results(res, status='ok', type='dataset')\n\n\n@slow  # 12sec on Yarik's laptop\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_push_subds_no_recursion(src_path, dst_top, dst_sub, dst_subsub):\n    # dataset with one submodule and one subsubmodule\n    top = Dataset(src_path).create()\n    sub = top.create('sub m')\n    test_file = sub.pathobj / 'subdir' / 'test_file'\n    test_file.parent.mkdir()\n    test_file.write_text('some')\n    subsub = sub.create(sub.pathobj / 'subdir' / 'subsub m')\n    top.save(recursive=True)\n    assert_repo_status(top.path)\n    target_top = mk_push_target(top, 'target', dst_top, annex=True)\n    target_sub = mk_push_target(sub, 'target', dst_sub, annex=True)\n    target_subsub = mk_push_target(subsub, 'target', dst_subsub, annex=True)\n    # now publish, but NO recursion, instead give the parent dir of\n    # both a subdataset and a file in the middle subdataset\n    res = top.push(\n        to='target',\n        # give relative to top dataset to elevate the difficulty a little\n        path=str(test_file.relative_to(top.pathobj).parent))\n    assert_status('ok', res)\n    assert_in_results(res, action='publish', type='dataset', path=top.path)\n    assert_in_results(res, action='publish', type='dataset', path=sub.path)\n    assert_in_results(res, action='copy', type='file', path=str(test_file))\n    # the lowest-level subdataset isn't touched\n    assert_not_in_results(\n        res, action='publish', type='dataset', path=subsub.path)\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_force_checkdatapresent(srcpath, dstpath):\n    src = Dataset(srcpath).create()\n    target = mk_push_target(src, 'target', dstpath, annex=True, bare=True)\n    (src.pathobj / 'test_mod_annex_file').write_text(\"Heavy stuff.\")\n    src.save(to_git=False, message=\"New annex file\")\n    assert_repo_status(src.path, annex=True)\n    whereis_prior = src.repo.whereis(files=['test_mod_annex_file'])[0]\n\n    res = src.push(to='target', data='nothing')\n    # nothing reported to be copied\n    assert_not_in_results(res, action='copy')\n    # we got the git-push nevertheless\n    eq_(src.repo.get_hexsha(DEFAULT_BRANCH), target.get_hexsha(DEFAULT_BRANCH))\n    # nothing moved\n    eq_(whereis_prior, src.repo.whereis(files=['test_mod_annex_file'])[0])\n\n    # now a push without forced no-transfer\n    # we do not give since, so the non-transfered file is picked up\n    # and transferred\n    res = src.push(to='target', force=None)\n    # no branch change, done before\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec=DEFAULT_REFSPEC)\n    # but availability update\n    assert_in_results(res, action='publish', status='ok',\n                      refspec='refs/heads/git-annex:refs/heads/git-annex')\n    assert_in_results(res, status='ok',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy')\n    # whereis info reflects the change\n    ok_(len(whereis_prior) < len(\n        src.repo.whereis(files=['test_mod_annex_file'])[0]))\n\n    # do it yet again will do nothing, because all is up-to-date\n    assert_status('notneeded', src.push(to='target', force=None))\n    # an explicit reference point doesn't change that\n    assert_status('notneeded',\n                  src.push(to='target', force=None, since='HEAD~1'))\n\n    # now force data transfer\n    res = src.push(to='target', force='checkdatapresent')\n    # no branch change, done before\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec=DEFAULT_REFSPEC)\n    # no availability update\n    assert_in_results(res, action='publish', status='notneeded',\n                      refspec='refs/heads/git-annex:refs/heads/git-annex')\n    # but data transfer\n    assert_in_results(res, status='ok',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy')\n\n    # force data transfer, but data isn't available\n    src.repo.drop('test_mod_annex_file')\n    res = src.push(to='target', path='.', force='checkdatapresent', on_failure='ignore')\n    assert_in_results(res, status='impossible',\n                      path=str(src.pathobj / 'test_mod_annex_file'),\n                      action='copy',\n                      message='Slated for transport, but no content present')\n\n\n@skip_if_on_windows  # https://github.com/datalad/datalad/issues/4278\n@with_tempfile(mkdir=True)\n@with_tree(tree={'ria-layout-version': '1\\n'})\ndef test_ria_push(srcpath, dstpath):\n    # complex test involving a git remote, a special remote, and a\n    # publication dependency\n    src = Dataset(srcpath).create()\n    testfile = src.pathobj / 'test_mod_annex_file'\n    testfile.write_text(\"Heavy stuff.\")\n    src.save()\n    assert_status(\n        'ok',\n        src.create_sibling_ria(\n            \"ria+{}\".format(get_local_file_url(dstpath, compatibility='git')),\n            \"datastore\"))\n    res = src.push(to='datastore')\n    assert_in_results(\n        res, action='publish', target='datastore', status='ok',\n        refspec=DEFAULT_REFSPEC)\n    assert_in_results(\n        res, action='publish', target='datastore', status='ok',\n        refspec='refs/heads/git-annex:refs/heads/git-annex')\n    assert_in_results(\n        res, action='copy', target='datastore-storage', status='ok',\n        path=str(testfile))\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_gh1426(origin_path, target_path):\n    # set up a pair of repos, one the published copy of the other\n    origin = Dataset(origin_path).create()\n    target = mk_push_target(\n        origin, 'target', target_path, annex=True, bare=False)\n    origin.push(to='target')\n    assert_repo_status(origin.path)\n    assert_repo_status(target.path)\n    eq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n        target.get_hexsha(DEFAULT_BRANCH))\n\n    # gist of #1426 is that a newly added subdataset does not cause the\n    # superdataset to get published\n    origin.create('sub')\n    assert_repo_status(origin.path)\n    neq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n         target.get_hexsha(DEFAULT_BRANCH))\n    # now push\n    res = origin.push(to='target')\n    assert_result_count(\n        res, 1, status='ok', type='dataset', path=origin.path,\n        action='publish', target='target', operations=['fast-forward'])\n    eq_(origin.repo.get_hexsha(DEFAULT_BRANCH),\n        target.get_hexsha(DEFAULT_BRANCH))\n\n\n@skip_if_adjusted_branch  # gh-4075\n@skip_if_on_windows  # create_sibling incompatible with win servers\n@skip_ssh\n@with_tree(tree={'1': '123'})\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_publish_target_url(src, desttop, desturl):\n    # https://github.com/datalad/datalad/issues/1762\n    ds = Dataset(src).create(force=True)\n    ds.save('1')\n    ds.create_sibling('ssh://datalad-test:%s/subdir' % desttop,\n                      name='target',\n                      target_url=desturl + 'subdir/.git')\n    results = ds.push(to='target')\n    assert results\n    ok_file_has_content(Path(desttop, 'subdir', '1'), '123')\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile()\n@with_tempfile()\ndef test_gh1763(src, target1, target2):\n    # this test is very similar to test_publish_depends, but more\n    # comprehensible, and directly tests issue 1763\n    src = Dataset(src).create(force=True)\n    target1 = mk_push_target(src, 'target1', target1, bare=False)\n    target2 = mk_push_target(src, 'target2', target2, bare=False)\n    src.siblings('configure', name='target2', publish_depends='target1',\n                 result_renderer=None)\n    # a file to annex\n    (src.pathobj / 'probe1').write_text('probe1')\n    src.save('probe1', to_git=False)\n    # make sure the probe is annexed, not straight in Git\n    assert_in('probe1', src.repo.get_annexed_files(with_content_only=True))\n    # publish to target2, must handle dependency\n    src.push(to='target2')\n    for target in (target1, target2):\n        # with a managed branch we are pushing into the corresponding branch\n        # and do not see a change in the worktree\n        if not target.is_managed_branch():\n            # direct test for what is in the checkout\n            assert_in(\n                'probe1',\n                target.get_annexed_files(with_content_only=True))\n        # ensure git-annex knows this target has the file\n        assert_in(target.config.get('annex.uuid'), src.repo.whereis(['probe1'])[0])\n\n\n@with_tempfile()\n@with_tempfile()\ndef test_gh1811(srcpath, clonepath):\n    orig = Dataset(srcpath).create()\n    (orig.pathobj / 'some').write_text('some')\n    orig.save()\n    clone = Clone.__call__(source=orig.path, path=clonepath)\n    (clone.pathobj / 'somemore').write_text('somemore')\n    clone.save()\n    clone.repo.call_git(['checkout', 'HEAD~1'])\n    res = clone.push(to=DEFAULT_REMOTE, on_failure='ignore')\n    assert_result_count(res, 1)\n    assert_result_count(\n        res, 1,\n        path=clone.path, type='dataset', action='publish',\n        status='impossible',\n        message='There is no active branch, cannot determine remote '\n                'branch',\n    )\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@with_tempfile()\n@with_tempfile()\ndef test_push_wanted(srcpath, dstpath):\n    src = Dataset(srcpath).create()\n    (src.pathobj / 'data.0').write_text('0')\n    (src.pathobj / 'secure.1').write_text('1')\n    (src.pathobj / 'secure.2').write_text('2')\n    src.save()\n\n    # Dropping a file to mimic a case of simply not having it locally (thus not\n    # to be \"pushed\")\n    src.drop('secure.2', check=False)\n\n    # Annotate sensitive content, actual value \"verysecure\" does not matter in\n    # this example\n    src.repo.set_metadata(\n        add={'distribution-restrictions': 'verysecure'},\n        files=['secure.1', 'secure.2'])\n\n    src.create_sibling(\n        dstpath,\n        annex_wanted=\"not metadata=distribution-restrictions=*\",\n        name='target',\n    )\n    # check that wanted is obeyed, since set in sibling configuration\n    res = src.push(to='target')\n    assert_in_results(\n        res, action='copy', path=str(src.pathobj / 'data.0'), status='ok')\n    for p in ('secure.1', 'secure.2'):\n        assert_not_in_results(res, path=str(src.pathobj / p))\n    assert_status('notneeded', src.push(to='target'))\n\n    # check the target to really make sure\n    dst = Dataset(dstpath)\n    # normal file, yes\n    eq_((dst.pathobj / 'data.0').read_text(), '0')\n    # secure file, no\n    if dst.repo.is_managed_branch():\n        neq_((dst.pathobj / 'secure.1').read_text(), '1')\n    else:\n        assert_raises(FileNotFoundError, (dst.pathobj / 'secure.1').read_text)\n\n    # reset wanted config, which must enable push of secure file\n    src.repo.set_preferred_content('wanted', '', remote='target')\n    res = src.push(to='target')\n    assert_in_results(res, path=str(src.pathobj / 'secure.1'))\n    eq_((dst.pathobj / 'secure.1').read_text(), '1')\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@slow  # 10sec on Yarik's laptop\n@with_tempfile(mkdir=True)\ndef test_auto_data_transfer(path):\n    path = Path(path)\n    ds_a = Dataset(path / \"a\").create()\n    (ds_a.pathobj / \"foo.dat\").write_text(\"foo\")\n    ds_a.save()\n\n    # Should be the default, but just in case.\n    ds_a.repo.config.set(\"annex.numcopies\", \"1\", where=\"local\")\n    ds_a.create_sibling(str(path / \"b\"), name=\"b\")\n\n    # With numcopies=1, no data is copied with data=\"auto\".\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_not_in_results(res, action=\"copy\")\n\n    # Even when a file is explicitly given.\n    res = ds_a.push(to=\"b\", path=\"foo.dat\", data=\"auto\", since=None)\n    assert_not_in_results(res, action=\"copy\")\n\n    # numcopies=2 changes that.\n    ds_a.repo.config.set(\"annex.numcopies\", \"2\", where=\"local\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_in_results(\n        res, action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"foo.dat\"))\n\n    # --since= limits the files considered by --auto.\n    (ds_a.pathobj / \"bar.dat\").write_text(\"bar\")\n    ds_a.save()\n    (ds_a.pathobj / \"baz.dat\").write_text(\"baz\")\n    ds_a.save()\n    res = ds_a.push(to=\"b\", data=\"auto\", since=\"HEAD~1\")\n    assert_not_in_results(\n        res,\n        action=\"copy\", path=str(ds_a.pathobj / \"bar.dat\"))\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"baz.dat\"))\n\n    # --auto also considers preferred content.\n    ds_a.repo.config.unset(\"annex.numcopies\", where=\"local\")\n    ds_a.repo.set_preferred_content(\"wanted\", \"nothing\", remote=\"b\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_not_in_results(\n        res,\n        action=\"copy\", path=str(ds_a.pathobj / \"bar.dat\"))\n\n    ds_a.repo.set_preferred_content(\"wanted\", \"anything\", remote=\"b\")\n    res = ds_a.push(to=\"b\", data=\"auto\", since=None)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"bar.dat\"))\n\n\n# FIXME: on crippled FS post-update hook enabling via create-sibling doesn't\n# work ATM\n@skip_if_adjusted_branch\n@slow  # 16sec on Yarik's laptop\n@with_tempfile(mkdir=True)\ndef test_auto_if_wanted_data_transfer_path_restriction(path):\n    path = Path(path)\n    ds_a = Dataset(path / \"a\").create()\n    ds_a_sub0 = ds_a.create(\"sub0\")\n    ds_a_sub1 = ds_a.create(\"sub1\")\n\n    for ds in [ds_a, ds_a_sub0, ds_a_sub1]:\n        (ds.pathobj / \"sec.dat\").write_text(\"sec\")\n        (ds.pathobj / \"reg.dat\").write_text(\"reg\")\n    ds_a.save(recursive=True)\n\n    ds_a.create_sibling(str(path / \"b\"), name=\"b\",\n                        annex_wanted=\"not metadata=distribution-restrictions=*\",\n                        recursive=True)\n    for ds in [ds_a, ds_a_sub0, ds_a_sub1]:\n        ds.repo.set_metadata(add={\"distribution-restrictions\": \"doesntmatter\"},\n                             files=[\"sec.dat\"])\n\n    # wanted-triggered --auto can be restricted to subdataset...\n    res = ds_a.push(to=\"b\", path=\"sub0\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"reg.dat\"))\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub0.pathobj / \"reg.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub0.pathobj / \"sec.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub1.pathobj / \"reg.dat\"))\n\n    # ... and to a wanted file.\n    res = ds_a.push(to=\"b\", path=\"reg.dat\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"reg.dat\"))\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a_sub1.pathobj / \"reg.dat\"))\n\n    # But asking to transfer a file does not do it if the remote has a\n    # wanted setting and doesn't want it.\n    res = ds_a.push(to=\"b\", path=\"sec.dat\", data=\"auto-if-wanted\",\n                    recursive=True)\n    assert_not_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"sec.dat\"))\n\n    res = ds_a.push(to=\"b\", path=\"sec.dat\", data=\"anything\", recursive=True)\n    assert_in_results(\n        res,\n        action=\"copy\", target=\"b\", status=\"ok\",\n        path=str(ds_a.pathobj / \"sec.dat\"))\n\n\n@with_tempfile(mkdir=True)\ndef test_push_git_annex_branch_when_no_data(path):\n    path = Path(path)\n    ds = Dataset(path / \"a\").create()\n    target = mk_push_target(ds, \"target\", str(path / \"target\"),\n                            annex=False, bare=True)\n    (ds.pathobj / \"f0\").write_text(\"0\")\n    ds.save()\n    ds.push(to=\"target\", data=\"nothing\")\n    assert_in(\"git-annex\",\n              {d[\"refname:strip=2\"]\n               for d in target.for_each_ref_(fields=\"refname:strip=2\")})\n\n\n@known_failure_githubci_osx\n@with_tree(tree={\"ds\": {\"f0\": \"0\", \"f1\": \"0\", \"f2\": \"0\",\n                        \"f3\": \"1\",\n                        \"f4\": \"2\", \"f5\": \"2\"}})\ndef test_push_git_annex_branch_many_paths_same_data(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create(force=True)\n    ds.save()\n    mk_push_target(ds, \"target\", str(path / \"target\"),\n                   annex=True, bare=False)\n    nbytes = sum(ds.repo.get_content_annexinfo(paths=[f])[f][\"bytesize\"]\n                 for f in [ds.repo.pathobj / \"f0\",\n                           ds.repo.pathobj / \"f3\",\n                           ds.repo.pathobj / \"f4\"])\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        res = ds.push(to=\"target\")\n    assert_in(\"{} bytes of annex data\".format(nbytes), cml.out)\n    # 3 files point to content already covered by another file.\n    assert_result_count(res, 3,\n                        action=\"copy\", type=\"file\", status=\"notneeded\")\n\n\n@known_failure_githubci_osx\n@with_tree(tree={\"ds\": {\"f0\": \"0\"}})\ndef test_push_matching(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create(force=True)\n    ds.config.set('push.default', 'matching', where='local')\n    ds.save()\n    remote_ds = mk_push_target(ds, 'local', str(path / 'dssibling'),\n                               annex=True, bare=False)\n    # that fact that the next one even runs makes sure that we are in a better\n    # place than https://github.com/datalad/datalad/issues/4888\n    ds.push(to='local')\n    # and we pushed the commit in the current branch\n    eq_(remote_ds.get_hexsha(DEFAULT_BRANCH),\n        ds.repo.get_hexsha(DEFAULT_BRANCH))\n\n\n@known_failure_githubci_win  # https://github.com/datalad/datalad/issues/5271\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):\n    if 'DATALAD_SEED' in os.environ:\n        # we are using create-sibling-ria via the cmdline in here\n        # this will create random UUIDs for datasets\n        # however, given a fixed seed each call to this command will start\n        # with the same RNG seed, hence yield the same UUID on the same\n        # machine -- leading to a collision\n        raise SkipTest(\n            'Test incompatible with fixed random number generator seed')\n    # the aim here is this high-level test a std create-push-clone cycle for a\n    # dataset with a subdataset, with the goal to ensure that correct branches\n    # and commits are tracked, regardless of platform behavior and condition\n    # of individual clones. Nothing fancy, just that the defaults behave in\n    # sensible ways\n    from datalad.cmd import WitlessRunner as Runner\n    run = Runner().run\n\n    # create original nested dataset\n    with chpwd(origpath):\n        run(['datalad', 'create', 'super'])\n        run(['datalad', 'create', '-d', 'super', str(Path('super', 'sub'))])\n\n    # verify essential linkage properties\n    orig_super = Dataset(Path(origpath, 'super'))\n    orig_sub = Dataset(orig_super.pathobj / 'sub')\n\n    (orig_super.pathobj / 'file1.txt').write_text('some1')\n    (orig_sub.pathobj / 'file2.txt').write_text('some1')\n    with chpwd(orig_super.path):\n        run(['datalad', 'save', '--recursive'])\n\n    # TODO not yet reported clean with adjusted branches\n    #assert_repo_status(orig_super.path)\n\n    # the \"true\" branch that sub is on, and the gitsha of the HEAD commit of it\n    orig_sub_corr_branch = \\\n        orig_sub.repo.get_corresponding_branch() or orig_sub.repo.get_active_branch()\n    orig_sub_corr_commit = orig_sub.repo.get_hexsha(orig_sub_corr_branch)\n\n    # make sure the super trackes this commit\n    assert_in_results(\n        orig_super.subdatasets(),\n        path=orig_sub.path,\n        gitshasum=orig_sub_corr_commit,\n        # TODO it should also track the branch name\n        # Attempted: https://github.com/datalad/datalad/pull/3817\n        # But reverted: https://github.com/datalad/datalad/pull/4375\n    )\n\n    # publish to a store, to get into a platform-agnostic state\n    # (i.e. no impact of an annex-init of any kind)\n    store_url = 'ria+' + get_local_file_url(storepath)\n    with chpwd(orig_super.path):\n        run(['datalad', 'create-sibling-ria', '--recursive',\n             '-s', 'store', store_url])\n        run(['datalad', 'push', '--recursive', '--to', 'store'])\n\n    # we are using the 'store' sibling's URL, which should be a plain path\n    store_super = AnnexRepo(orig_super.siblings(name='store')[0]['url'], init=False)\n    store_sub = AnnexRepo(orig_sub.siblings(name='store')[0]['url'], init=False)\n\n    # both datasets in the store only carry the real branches, and nothing\n    # adjusted\n    for r in (store_super, store_sub):\n        eq_(set(r.get_branches()), set([orig_sub_corr_branch, 'git-annex']))\n\n    # and reobtain from a store\n    cloneurl = 'ria+' + get_local_file_url(str(storepath), compatibility='git')\n    with chpwd(clonepath):\n        run(['datalad', 'clone', cloneurl + '#' + orig_super.id, 'super'])\n        run(['datalad', '-C', 'super', 'get', '--recursive', '.'])\n\n    # verify that nothing has changed as a result of a push/clone cycle\n    clone_super = Dataset(Path(clonepath, 'super'))\n    clone_sub = Dataset(clone_super.pathobj / 'sub')\n    assert_in_results(\n        clone_super.subdatasets(),\n        path=clone_sub.path,\n        gitshasum=orig_sub_corr_commit,\n    )\n\n    for ds1, ds2, f in ((orig_super, clone_super, 'file1.txt'),\n                        (orig_sub, clone_sub, 'file2.txt')):\n        eq_((ds1.pathobj / f).read_text(), (ds2.pathobj / f).read_text())\n\n    # get status info that does not recursive into subdatasets, i.e. not\n    # looking for uncommitted changes\n    # we should see no modification reported\n    assert_not_in_results(\n        clone_super.status(eval_subdataset_state='commit'),\n        state='modified')\n    # and now the same for a more expensive full status\n    assert_not_in_results(\n        clone_super.status(recursive=True),\n        state='modified')\n\n\n@with_tempfile\ndef test_push_custom_summary(path):\n    path = Path(path)\n    ds = Dataset(path / \"ds\").create()\n\n    sib = mk_push_target(ds, \"sib\", str(path / \"sib\"), bare=False, annex=False)\n    (sib.pathobj / \"f1\").write_text(\"f1\")\n    sib.save()\n\n    (ds.pathobj / \"f2\").write_text(\"f2\")\n    ds.save()\n\n    # These options are true by default and our tests usually run with a\n    # temporary home, but set them to be sure.\n    ds.config.set(\"advice.pushUpdateRejected\", \"true\", where=\"local\")\n    ds.config.set(\"advice.pushFetchFirst\", \"true\", where=\"local\")\n    with swallow_outputs() as cmo:\n        ds.push(to=\"sib\", result_renderer=\"default\", on_failure=\"ignore\")\n        assert_in(\"Potential hints to solve\", cmo.out)\n        assert_in(\"action summary:\", cmo.out)\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "test_2.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.14.4.dev383\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 108ms\nInstalled 25 packages in 3.42s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 62 packages in 4.09s\nPrepared 1 package in 3.82s\nInstalled 56 packages in 502ms\n + annexremote==1.6.6\n + appdirs==1.4.4\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.14.4 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.0.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 96 packages in 347ms\nInstalled 32 packages in 535ms\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 26 items\n\nr2e_tests/test_1.py FF..E...                                             [ 30%]\nr2e_tests/test_2.py ExEEEEEEEEEFFFFFEF                                   [100%]\n\n==================================== ERRORS ====================================\n___________________ ERROR at setup of test_discover_ds_trace ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_1.py, line 339\n  @with_tree({k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n  @with_tempfile(mkdir=True)\n  def test_discover_ds_trace(path, otherdir):\nE       fixture 'otherdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_1.py:339\n_____________________ ERROR at setup of test_invalid_call ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 64\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_invalid_call(origin, tdir):\nE       fixture 'origin' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:64\n____________________ ERROR at setup of test_push_recursive _____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 274\n  @slow  # 33sec on Yarik's laptop\n  @with_tempfile\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True, suffix='sub')\n  @with_tempfile(mkdir=True, suffix='subnoannex')\n  @with_tempfile(mkdir=True, suffix='subsub')\n  def test_push_recursive(\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:274\n________________ ERROR at setup of test_push_subds_no_recursion ________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 415\n  @slow  # 12sec on Yarik's laptop\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_push_subds_no_recursion(src_path, dst_top, dst_sub, dst_subsub):\nE       fixture 'src_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:415\n________________ ERROR at setup of test_force_checkdatapresent _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 448\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_force_checkdatapresent(srcpath, dstpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:448\n_______________________ ERROR at setup of test_ria_push ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 511\n  @skip_if_on_windows  # https://github.com/datalad/datalad/issues/4278\n  @with_tempfile(mkdir=True)\n  @with_tree(tree={'ria-layout-version': '1\\n'})\n  def test_ria_push(srcpath, dstpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:511\n________________________ ERROR at setup of test_gh1426 _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 538\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_gh1426(origin_path, target_path):\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:538\n__________________ ERROR at setup of test_publish_target_url ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 566\n  @skip_if_adjusted_branch  # gh-4075\n  @skip_if_on_windows  # create_sibling incompatible with win servers\n  @skip_ssh\n  @with_tree(tree={'1': '123'})\n  @with_tempfile(mkdir=True)\n  @serve_path_via_http\n  def test_publish_target_url(src, desttop, desturl):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:566\n________________________ ERROR at setup of test_gh1763 _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 584\n  @with_tempfile(mkdir=True)\n  @with_tempfile()\n  @with_tempfile()\n  def test_gh1763(src, target1, target2):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:584\n________________________ ERROR at setup of test_gh1811 _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 614\n  @with_tempfile()\n  @with_tempfile()\n  def test_gh1811(srcpath, clonepath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:614\n______________________ ERROR at setup of test_push_wanted ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 637\n  @skip_if_adjusted_branch\n  @with_tempfile()\n  @with_tempfile()\n  def test_push_wanted(srcpath, dstpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:637\n__________ ERROR at setup of test_nested_pushclone_cycle_allplatforms __________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 871\n  @known_failure_githubci_win  # https://github.com/datalad/datalad/issues/5271\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):\nE       fixture 'origpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:871\n=================================== FAILURES ===================================\n__________________________________ test_dirty __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexg8m9ynnd'}\nfilename = '/tmp/datalad_temp_test_dirtyfmgp7scm'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_dirty() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_____________________________ test_save_hierarchy ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex0t7fcq_s'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_save_hierarchy'}\nd = '/tmp/datalad_temp_tree_test_save_hierarchygibeojos'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_save_hierarchy() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:560: TypeError\n___________________________ test_auto_data_transfer ____________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexdtat6fuv'}\n_check = <function skip_if_adjusted_branch.<locals>._wrap_skip_if_adjusted_branch.<locals>._check at 0x740410aa0ee0>\n\n    @wraps(func)\n    @attr('skip_if_adjusted_branch')\n    def _wrap_skip_if_adjusted_branch(*args, **kwargs):\n        global _TESTS_ADJUSTED_TMPDIR\n        if _TESTS_ADJUSTED_TMPDIR is None:\n            @with_tempfile\n            def _check(path):\n                ds = Dataset(path).create(force=True)\n                return ds.repo.is_managed_branch()\n            _TESTS_ADJUSTED_TMPDIR = _check()\n    \n        if _TESTS_ADJUSTED_TMPDIR:\n            raise SkipTest(\"Test incompatible with adjusted branch default\")\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:1897: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexdtat6fuv'}\nfilename = '/tmp/datalad_temp_test_auto_data_transfert07mqslg'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_auto_data_transfer() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n----------------------------- Captured stderr call -----------------------------\n[INFO] Creating a new annex repo at /tmp/datalad_temp__check1bfvcgv1 \n[INFO] Scanning for unlocked files (this may take some time) \n------------------------------ Captured log call -------------------------------\nINFO     datalad.core.local.create:create.py:519 Creating a new annex repo at /tmp/datalad_temp__check1bfvcgv1\nINFO     datalad.annex:annexrepo.py:3626 Scanning for unlocked files (this may take some time)\n______________ test_auto_if_wanted_data_transfer_path_restriction ______________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexmcmyi78_'}\n\n    @wraps(func)\n    @attr('skip_if_adjusted_branch')\n    def _wrap_skip_if_adjusted_branch(*args, **kwargs):\n        global _TESTS_ADJUSTED_TMPDIR\n        if _TESTS_ADJUSTED_TMPDIR is None:\n            @with_tempfile\n            def _check(path):\n                ds = Dataset(path).create(force=True)\n                return ds.repo.is_managed_branch()\n            _TESTS_ADJUSTED_TMPDIR = _check()\n    \n        if _TESTS_ADJUSTED_TMPDIR:\n            raise SkipTest(\"Test incompatible with adjusted branch default\")\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:1897: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexmcmyi78_'}\nfilename = '/tmp/datalad_temp_test_auto_if_wanted_data_transfer_path_restriction2a7k522v'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_auto_if_wanted_data_transfer_path_restriction() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n___________________ test_push_git_annex_branch_when_no_data ____________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexwooutmhc'}\nfilename = '/tmp/datalad_temp_test_push_git_annex_branch_when_no_dataon8ql9fa'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_push_git_annex_branch_when_no_data() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_______________ test_push_git_annex_branch_many_paths_same_data ________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex9e626pf4'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_push_git_annex_branch_many_paths_same_data'}\nd = '/tmp/datalad_temp_tree_test_push_git_annex_branch_many_paths_same_datas9i6ur6i'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_push_git_annex_branch_many_paths_same_data() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:560: TypeError\n______________________________ test_push_matching ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex9mktvs2o'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_push_matching'}\nd = '/tmp/datalad_temp_tree_test_push_matchingt7p0b3fy'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_push_matching() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:560: TypeError\n___________________________ test_push_custom_summary ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexhqr2gmzk'}\nfilename = '/tmp/datalad_temp_test_push_custom_summarynduwhbfr'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_push_custom_summary() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n==================================== PASSES ====================================\n_______________________ test_eval_results_plus_build_doc _______________________\n------------------------------ Captured log call -------------------------------\nDEBUG    datalad.utils:utils.py:2545 Determined class of decorated function: <class 'r2e_tests.test_1.TestUtils'>\nDEBUG    datalad.utils:utils.py:2545 Determined class of decorated function: <class 'r2e_tests.test_1.TestUtils'>\nDEBUG    datalad.cmd:cmd.py:154 run:\n cwd=None\n cmd=['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']\nDEBUG    datalad.cmd:cmd_protocols.py:98 Process 2041093 started\nDEBUG    datalad.runner:nonasyncrunner.py:72 ReaderThread(<_io.FileIO name=17 mode='rb' closefd=True>, <queue.Queue object at 0x74041325f250>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) started\nDEBUG    datalad.runner:nonasyncrunner.py:78 ReaderThread(<_io.FileIO name=17 mode='rb' closefd=True>, <queue.Queue object at 0x74041325f250>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) exiting (stream end)\nDEBUG    datalad.runner:nonasyncrunner.py:72 ReaderThread(<_io.FileIO name=15 mode='rb' closefd=True>, <queue.Queue object at 0x74041325f250>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) started\nDEBUG    datalad.runner:nonasyncrunner.py:78 ReaderThread(<_io.FileIO name=15 mode='rb' closefd=True>, <queue.Queue object at 0x74041325f250>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) exiting (stream end)\nDEBUG    datalad.cmd:cmd_protocols.py:122 Process 2041093 exited with return code 0\n_________________________ test_utils_suppress_similar __________________________\n------------------------------ Captured log call -------------------------------\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_eval_results_plus_build_doc\nPASSED r2e_tests/test_1.py::test_result_filter\nPASSED r2e_tests/test_1.py::test_utils_suppress_similar\nPASSED r2e_tests/test_1.py::test_incorrect_msg_interpolation\nPASSED r2e_tests/test_1.py::test_custom_result_summary_renderer\nXFAIL r2e_tests/test_2.py::test_push - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_push will be ignored\nERROR r2e_tests/test_1.py::test_discover_ds_trace\nERROR r2e_tests/test_2.py::test_invalid_call\nERROR r2e_tests/test_2.py::test_push_recursive\nERROR r2e_tests/test_2.py::test_push_subds_no_recursion\nERROR r2e_tests/test_2.py::test_force_checkdatapresent\nERROR r2e_tests/test_2.py::test_ria_push\nERROR r2e_tests/test_2.py::test_gh1426\nERROR r2e_tests/test_2.py::test_publish_target_url\nERROR r2e_tests/test_2.py::test_gh1763\nERROR r2e_tests/test_2.py::test_gh1811\nERROR r2e_tests/test_2.py::test_push_wanted\nERROR r2e_tests/test_2.py::test_nested_pushclone_cycle_allplatforms\nFAILED r2e_tests/test_1.py::test_dirty - TypeError: test_dirty() got multiple...\nFAILED r2e_tests/test_1.py::test_save_hierarchy - TypeError: test_save_hierar...\nFAILED r2e_tests/test_2.py::test_auto_data_transfer - TypeError: test_auto_da...\nFAILED r2e_tests/test_2.py::test_auto_if_wanted_data_transfer_path_restriction\nFAILED r2e_tests/test_2.py::test_push_git_annex_branch_when_no_data - TypeErr...\nFAILED r2e_tests/test_2.py::test_push_git_annex_branch_many_paths_same_data\nFAILED r2e_tests/test_2.py::test_push_matching - TypeError: test_push_matchin...\nFAILED r2e_tests/test_2.py::test_push_custom_summary - TypeError: test_push_c...\n============== 8 failed, 5 passed, 1 xfailed, 12 errors in 9.93s ===============\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 26 items\n\nr2e_tests/test_1.py FF..E..F                                             [ 30%]\nr2e_tests/test_2.py ExEEEEEEEEEFFFFFEF                                   [100%]\n\n==================================== ERRORS ====================================\n___________________ ERROR at setup of test_discover_ds_trace ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_1.py, line 339\n  @with_tree({k: v for k, v in demo_hierarchy.items() if k in ['a', 'd']})\n  @with_tempfile(mkdir=True)\n  def test_discover_ds_trace(path, otherdir):\nE       fixture 'otherdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_1.py:339\n_____________________ ERROR at setup of test_invalid_call ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 64\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_invalid_call(origin, tdir):\nE       fixture 'origin' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:64\n____________________ ERROR at setup of test_push_recursive _____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 274\n  @slow  # 33sec on Yarik's laptop\n  @with_tempfile\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True, suffix='sub')\n  @with_tempfile(mkdir=True, suffix='subnoannex')\n  @with_tempfile(mkdir=True, suffix='subsub')\n  def test_push_recursive(\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:274\n________________ ERROR at setup of test_push_subds_no_recursion ________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 415\n  @slow  # 12sec on Yarik's laptop\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_push_subds_no_recursion(src_path, dst_top, dst_sub, dst_subsub):\nE       fixture 'src_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:415\n________________ ERROR at setup of test_force_checkdatapresent _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 448\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_force_checkdatapresent(srcpath, dstpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:448\n_______________________ ERROR at setup of test_ria_push ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 511\n  @skip_if_on_windows  # https://github.com/datalad/datalad/issues/4278\n  @with_tempfile(mkdir=True)\n  @with_tree(tree={'ria-layout-version': '1\\n'})\n  def test_ria_push(srcpath, dstpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:511\n________________________ ERROR at setup of test_gh1426 _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 538\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_gh1426(origin_path, target_path):\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:538\n__________________ ERROR at setup of test_publish_target_url ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 566\n  @skip_if_adjusted_branch  # gh-4075\n  @skip_if_on_windows  # create_sibling incompatible with win servers\n  @skip_ssh\n  @with_tree(tree={'1': '123'})\n  @with_tempfile(mkdir=True)\n  @serve_path_via_http\n  def test_publish_target_url(src, desttop, desturl):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:566\n________________________ ERROR at setup of test_gh1763 _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 584\n  @with_tempfile(mkdir=True)\n  @with_tempfile()\n  @with_tempfile()\n  def test_gh1763(src, target1, target2):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:584\n________________________ ERROR at setup of test_gh1811 _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 614\n  @with_tempfile()\n  @with_tempfile()\n  def test_gh1811(srcpath, clonepath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:614\n______________________ ERROR at setup of test_push_wanted ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 637\n  @skip_if_adjusted_branch\n  @with_tempfile()\n  @with_tempfile()\n  def test_push_wanted(srcpath, dstpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:637\n__________ ERROR at setup of test_nested_pushclone_cycle_allplatforms __________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py, line 871\n  @known_failure_githubci_win  # https://github.com/datalad/datalad/issues/5271\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):\nE       fixture 'origpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_1c6afc45526965f755c78971af3a06c59506fe36/r2e_tests/test_2.py:871\n=================================== FAILURES ===================================\n__________________________________ test_dirty __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexk5sgwswv'}\nfilename = '/tmp/datalad_temp_test_dirty_9_x80b7'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_dirty() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_____________________________ test_save_hierarchy ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexbwapk3dd'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_save_hierarchy'}\nd = '/tmp/datalad_temp_tree_test_save_hierarchy3928pfqd'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_save_hierarchy() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:560: TypeError\n_____________________ test_custom_result_summary_renderer ______________________\n\n    def test_custom_result_summary_renderer():\n        list(CustomSummary().__call__(\"arg\"))\n        try:\n            CustomSummary.custom_result_summary_renderer_pass_summary = True\n>           list(CustomSummary().__call__(\"arg\"))\n\nr2e_tests/test_1.py:548: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ndatalad/interface/utils.py:481: in eval_func\n    return return_func(generator_func)(*args, **kwargs)\ndatalad/interface/utils.py:474: in return_func\n    results = list(results)\ndatalad/interface/utils.py:446: in generator_func\n    wrapped_class.custom_result_summary_renderer(results)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nargs = ([{'action': 'test.one', 'message': 'message', 'status': 'ok', 'x': 'arg'}, {'action': 'test.two', 'message': 'message...ssage', 'status': 'notneeded', 'x': 'arg'}, {'action': 'test.one', 'message': 'message', 'status': 'ok', 'x': 'arg'}],)\n\n    @staticmethod\n    def custom_result_summary_renderer(*args):\n        if getattr(CustomSummary, \"custom_result_summary_renderer_pass_summary\",\n                   False):\n>           action_summary = args[1]\nE           IndexError: tuple index out of range\n\nr2e_tests/test_1.py:534: IndexError\n___________________________ test_auto_data_transfer ____________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexe5t_i007'}\n_check = <function skip_if_adjusted_branch.<locals>._wrap_skip_if_adjusted_branch.<locals>._check at 0x7c586fb784c0>\n\n    @wraps(func)\n    @attr('skip_if_adjusted_branch')\n    def _wrap_skip_if_adjusted_branch(*args, **kwargs):\n        global _TESTS_ADJUSTED_TMPDIR\n        if _TESTS_ADJUSTED_TMPDIR is None:\n            @with_tempfile\n            def _check(path):\n                ds = Dataset(path).create(force=True)\n                return ds.repo.is_managed_branch()\n            _TESTS_ADJUSTED_TMPDIR = _check()\n    \n        if _TESTS_ADJUSTED_TMPDIR:\n            raise SkipTest(\"Test incompatible with adjusted branch default\")\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:1897: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexe5t_i007'}\nfilename = '/tmp/datalad_temp_test_auto_data_transferh7nl97uu'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_auto_data_transfer() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n----------------------------- Captured stderr call -----------------------------\n[INFO] Creating a new annex repo at /tmp/datalad_temp__check9zti7v_p \n[INFO] Scanning for unlocked files (this may take some time) \n------------------------------ Captured log call -------------------------------\nINFO     datalad.core.local.create:create.py:519 Creating a new annex repo at /tmp/datalad_temp__check9zti7v_p\nINFO     datalad.annex:annexrepo.py:3626 Scanning for unlocked files (this may take some time)\n______________ test_auto_if_wanted_data_transfer_path_restriction ______________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexvf4ppu7u'}\n\n    @wraps(func)\n    @attr('skip_if_adjusted_branch')\n    def _wrap_skip_if_adjusted_branch(*args, **kwargs):\n        global _TESTS_ADJUSTED_TMPDIR\n        if _TESTS_ADJUSTED_TMPDIR is None:\n            @with_tempfile\n            def _check(path):\n                ds = Dataset(path).create(force=True)\n                return ds.repo.is_managed_branch()\n            _TESTS_ADJUSTED_TMPDIR = _check()\n    \n        if _TESTS_ADJUSTED_TMPDIR:\n            raise SkipTest(\"Test incompatible with adjusted branch default\")\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:1897: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexvf4ppu7u'}\nfilename = '/tmp/datalad_temp_test_auto_if_wanted_data_transfer_path_restriction4xycgsn8'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_auto_if_wanted_data_transfer_path_restriction() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n___________________ test_push_git_annex_branch_when_no_data ____________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexfb53bous'}\nfilename = '/tmp/datalad_temp_test_push_git_annex_branch_when_no_datavp_563aq'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_push_git_annex_branch_when_no_data() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_______________ test_push_git_annex_branch_many_paths_same_data ________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexcla0anwt'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_push_git_annex_branch_many_paths_same_data'}\nd = '/tmp/datalad_temp_tree_test_push_git_annex_branch_many_paths_same_dataloayeg2_'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_push_git_annex_branch_many_paths_same_data() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:560: TypeError\n______________________________ test_push_matching ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexi420n7qz'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_push_matching'}\nd = '/tmp/datalad_temp_tree_test_push_matchinguovn4cls'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_push_matching() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:560: TypeError\n___________________________ test_push_custom_summary ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexlvsfrujt'}\nfilename = '/tmp/datalad_temp_test_push_custom_summaryhsjshk19'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_push_custom_summary() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n==================================== PASSES ====================================\n_______________________ test_eval_results_plus_build_doc _______________________\n------------------------------ Captured log call -------------------------------\nDEBUG    datalad.utils:utils.py:2545 Determined class of decorated function: <class 'r2e_tests.test_1.TestUtils'>\nDEBUG    datalad.utils:utils.py:2545 Determined class of decorated function: <class 'r2e_tests.test_1.TestUtils'>\nDEBUG    datalad.cmd:cmd.py:154 run:\n cwd=None\n cmd=['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']\nDEBUG    datalad.cmd:cmd_protocols.py:98 Process 2054013 started\nDEBUG    datalad.runner:nonasyncrunner.py:72 ReaderThread(<_io.FileIO name=17 mode='rb' closefd=True>, <queue.Queue object at 0x7c586fdb09a0>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) started\nDEBUG    datalad.runner:nonasyncrunner.py:78 ReaderThread(<_io.FileIO name=17 mode='rb' closefd=True>, <queue.Queue object at 0x7c586fdb09a0>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) exiting (stream end)\nDEBUG    datalad.runner:nonasyncrunner.py:72 ReaderThread(<_io.FileIO name=15 mode='rb' closefd=True>, <queue.Queue object at 0x7c586fdb09a0>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) started\nDEBUG    datalad.runner:nonasyncrunner.py:78 ReaderThread(<_io.FileIO name=15 mode='rb' closefd=True>, <queue.Queue object at 0x7c586fdb09a0>, ['git', '--git-dir=', 'config', '-z', '-l', '--show-origin']) exiting (stream end)\nDEBUG    datalad.cmd:cmd_protocols.py:122 Process 2054013 exited with return code 0\n_________________________ test_utils_suppress_similar __________________________\n------------------------------ Captured log call -------------------------------\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\nINFO     datalad.ui.dialog:log.py:328 Clear progress bars\nINFO     datalad.ui.dialog:log.py:328 Refresh progress bars\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_eval_results_plus_build_doc\nPASSED r2e_tests/test_1.py::test_result_filter\nPASSED r2e_tests/test_1.py::test_utils_suppress_similar\nPASSED r2e_tests/test_1.py::test_incorrect_msg_interpolation\nXFAIL r2e_tests/test_2.py::test_push - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_push will be ignored\nERROR r2e_tests/test_1.py::test_discover_ds_trace\nERROR r2e_tests/test_2.py::test_invalid_call\nERROR r2e_tests/test_2.py::test_push_recursive\nERROR r2e_tests/test_2.py::test_push_subds_no_recursion\nERROR r2e_tests/test_2.py::test_force_checkdatapresent\nERROR r2e_tests/test_2.py::test_ria_push\nERROR r2e_tests/test_2.py::test_gh1426\nERROR r2e_tests/test_2.py::test_publish_target_url\nERROR r2e_tests/test_2.py::test_gh1763\nERROR r2e_tests/test_2.py::test_gh1811\nERROR r2e_tests/test_2.py::test_push_wanted\nERROR r2e_tests/test_2.py::test_nested_pushclone_cycle_allplatforms\nFAILED r2e_tests/test_1.py::test_dirty - TypeError: test_dirty() got multiple...\nFAILED r2e_tests/test_1.py::test_save_hierarchy - TypeError: test_save_hierar...\nFAILED r2e_tests/test_1.py::test_custom_result_summary_renderer - IndexError:...\nFAILED r2e_tests/test_2.py::test_auto_data_transfer - TypeError: test_auto_da...\nFAILED r2e_tests/test_2.py::test_auto_if_wanted_data_transfer_path_restriction\nFAILED r2e_tests/test_2.py::test_push_git_annex_branch_when_no_data - TypeErr...\nFAILED r2e_tests/test_2.py::test_push_git_annex_branch_many_paths_same_data\nFAILED r2e_tests/test_2.py::test_push_matching - TypeError: test_push_matchin...\nFAILED r2e_tests/test_2.py::test_push_custom_summary - TypeError: test_push_c...\n============== 9 failed, 4 passed, 1 xfailed, 12 errors in 7.85s ===============\n",
    "old_commit_res_stderr": ""
}
datalad/core/distributed/push.py datalad/core/distributed/tests/test_push.py datalad/interface/tests/test_utils.py datalad/interface/utils.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** `custom_result_summary_renderer` Raises IndexError When `action_summary` Is Missing

**Description:**
When executing a push action, the `custom_result_summary_renderer` function expects both `results` and `action_summary` as arguments. However, in some scenarios, it is invoked without the `action_summary`, leading to an `IndexError`.

**Example Code:**
```python
@staticmethod
def custom_result_summary_renderer(results, action_summary):  # Expected two arguments
    hints = set([r.get('hints', None) for r in results])
    # Processing logic...
```

**Issue Behavior:**
Calling `custom_result_summary_renderer` with only the `results` parameter causes the following error:
```
IndexError: tuple index out of range
```

**Expected Behavior:**
The `custom_result_summary_renderer` should handle cases where `action_summary` is not provided, allowing it to operate without raising an error.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/utils.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface utility functions

"""

__docformat__ = 'restructuredtext'

import inspect
import logging
import wrapt
import sys
import re
from time import time
from os import curdir
from os import pardir
from os import listdir
import os.path as op
from os.path import join as opj
from os.path import isdir
from os.path import relpath
from os.path import sep
from os.path import split as psplit
from itertools import chain

import json

# avoid import from API to not get into circular imports
from datalad.utils import with_pathsep as _with_sep  # TODO: RF whenever merge conflict is not upon us
from datalad.utils import (
    path_startswith,
    path_is_subpath,
    ensure_unicode,
    getargspec,
    get_wrapped_class,
)
from datalad.support.gitrepo import GitRepo
from datalad.support.exceptions import IncompleteResultsError
from datalad import cfg as dlcfg
from datalad.dochelpers import (
    exc_str,
    single_or_plural,
)

from datalad.ui import ui
import datalad.support.ansi_colors as ac

from datalad.interface.base import default_logchannels
from datalad.interface.base import get_allargs_as_kwargs
from datalad.interface.common_opts import eval_params
from datalad.interface.common_opts import eval_defaults
from .results import known_result_xfms
from datalad.core.local.resulthooks import (
    get_jsonhooks_from_config,
    match_jsonhook2result,
    run_jsonhook,
)


lgr = logging.getLogger('datalad.interface.utils')


def cls2cmdlinename(cls):
    "Return the cmdline command name from an Interface class"
    r = re.compile(r'([a-z0-9])([A-Z])')
    return r.sub('\\1-\\2', cls.__name__).lower()


# TODO remove
# only `drop` and `uninstall` are still using this
def handle_dirty_dataset(ds, mode, msg=None):
    """Detect and treat unsaved changes as instructed by `mode`

    Parameters
    ----------
    ds : Dataset or None
      Dataset to be inspected. Does nothing if `None`.
    mode : {'fail', 'ignore', 'save-before'}
      How to act upon discovering unsaved changes.
    msg : str or None
      Custom message to use for a potential commit.

    Returns
    -------
    None
    """
    if ds is None:
        # nothing to be handled
        return
    if msg is None:
        msg = '[DATALAD] auto-saved changes'

    # make sure that all pending changes (batched annex operations, etc.)
    # are actually reflected in Git
    if ds.repo:
        ds.repo.precommit()

    if mode == 'ignore':
        return
    elif mode == 'fail':
        if not ds.repo or ds.repo.dirty:
            raise RuntimeError('dataset {} has unsaved changes'.format(ds))
    elif mode == 'save-before':
        if not ds.is_installed():
            raise RuntimeError('dataset {} is not yet installed'.format(ds))
        from datalad.core.local.save import Save
        Save.__call__(dataset=ds, message=msg, updated=True)
    else:
        raise ValueError("unknown if-dirty mode '{}'".format(mode))


def get_tree_roots(paths):
    """Return common root paths for a set of paths

    This function determines the smallest set of common root
    paths and sorts all given paths under the respective
    root.

    Returns
    -------
    dict
      paths by root
    """
    paths_ws = [_with_sep(p) for p in paths]
    # sort all paths under their potential roots
    roots = {}
    # start from the top to get all paths down the line
    # and collate them into as few roots as possible
    for s in sorted(paths_ws):
        if any([s.startswith(r) for r in roots]):
            # this path is already covered by a known root
            continue
        # find all sub paths
        subs = [p for p in paths if p.startswith(s)]
        roots[s.rstrip(sep)] = subs
    return roots


# TODO remove
# only `remove` and `uninstall` use this, the uses path `path_is_subpath`
def path_is_under(values, path=None):
    """Whether a given path is a subdirectory of any of the given test values

    Parameters
    ----------
    values : sequence or dict
      Paths to be tested against. This can be a dictionary in which case
      all values from all keys will be tested against.
    path : path or None
      Test path. If None is given, the process' working directory is
      used.

    Returns
    -------
    bool
    """
    if path is None:
        from datalad.utils import getpwd
        path = getpwd()
    if isinstance(values, dict):
        values = chain(*values.values())
    path_drive, _ = op.splitdrive(path)
    for p in values:
        p_drive, _ = op.splitdrive(p)
        # need to protect against unsupported use of relpath() with
        # abspaths on windows from different drives (gh-3724)
        if path_drive != p_drive:
        # different drives, enough evidence for "not under"
            continue
        rpath = relpath(p, start=path)
        if rpath == curdir \
                or rpath == pardir \
                or set(psplit(rpath)) == {pardir}:
            # first match is enough
            return True
    return False


# TODO(OPT)? YOH: from a cursory review seems like possibly an expensive function
# whenever many paths were provided (e.g. via shell glob).
# Might be worth testing on some usecase and py-spy'ing if notable portion
# of time is spent.
def discover_dataset_trace_to_targets(basepath, targetpaths, current_trace,
                                      spec, includeds=None):
    """Discover the edges and nodes in a dataset tree to given target paths

    Parameters
    ----------
    basepath : path
      Path to a start or top-level dataset. Really has to be a path to a
      dataset!
    targetpaths : list(path)
      Any non-zero number of paths that are termination points for the
      search algorithm. Can be paths to datasets, directories, or files
      (and any combination thereof).
    current_trace : list
      For a top-level call this should probably always be `[]`
    spec : dict
      `content_by_ds`-style dictionary that will receive information about the
      discovered datasets. Specifically, for each discovered dataset there
      will be an item with its path under the key (path) of the respective
      superdataset.
    includeds : sequence, optional
      Any paths given are treated as existing subdatasets, regardless of
      whether they can be found in the filesystem. Such subdatasets will appear
      under the key of the closest existing dataset in the `spec`.

    Returns
    -------
    None
      Function calls itself recursively and populates `spec` dict in-place.
      Keys are dataset paths, values are sets of subdataset paths
    """
    # convert to set for faster lookup
    includeds = includeds if isinstance(includeds, set) else \
        set() if includeds is None else set(includeds)
    # this beast walks the directory tree from a given `basepath` until
    # it discovers any of the given `targetpaths`
    # if it finds one, it commits any accummulated trace of visited
    # datasets on this edge to the spec
    valid_repo = GitRepo.is_valid_repo(basepath)
    if valid_repo:
        # we are passing into a new dataset, extend the dataset trace
        current_trace = current_trace + [basepath]
    # this edge is not done, we need to try to reach any downstream
    # dataset
    undiscovered_ds = set(t for t in targetpaths) # if t != basepath)
    # whether anything in this directory matched a targetpath
    filematch = False
    if isdir(basepath):
        for p in listdir(basepath):
            p = ensure_unicode(opj(basepath, p))
            if not isdir(p):
                if p in targetpaths:
                    filematch = True
                # we cannot have anything below this one
                continue
            # OPT listdir might be large and we could have only few items
            # in `targetpaths` -- so traverse only those in spec which have
            # leading dir basepath
            # filter targets matching this downward path
            downward_targets = set(
                t for t in targetpaths if path_startswith(t, p))
            if not downward_targets:
                continue
            # remove the matching ones from the "todo" list
            undiscovered_ds.difference_update(downward_targets)
            # go one deeper
            discover_dataset_trace_to_targets(
                p, downward_targets, current_trace, spec,
                includeds=includeds if not includeds else includeds.intersection(
                    downward_targets))
    undiscovered_ds = [t for t in undiscovered_ds
                       if includeds and
                          path_is_subpath(t, current_trace[-1]) and
                          t in includeds]
    if filematch or basepath in targetpaths or undiscovered_ds:
        for i, p in enumerate(current_trace[:-1]):
            # TODO RF prepare proper annotated path dicts
            subds = spec.get(p, set())
            subds.add(current_trace[i + 1])
            spec[p] = subds
        if undiscovered_ds:
            spec[current_trace[-1]] = spec.get(current_trace[-1], set()).union(
                undiscovered_ds)


def get_result_filter(fx):
    """Wrap a filter into a helper to be able to accept additional
    arguments, if the filter doesn't support it already"""
    _fx = fx
    if fx and not getargspec(fx).keywords:
        def _fx(res, **kwargs):
            return fx(res)
    return _fx


def eval_results(func):
    """Decorator for return value evaluation of datalad commands.

    Note, this decorator is only compatible with commands that return
    status dict sequences!

    Two basic modes of operation are supported: 1) "generator mode" that
    `yields` individual results, and 2) "list mode" that returns a sequence of
    results. The behavior can be selected via the kwarg `return_type`.
    Default is "list mode".

    This decorator implements common functionality for result rendering/output,
    error detection/handling, and logging.

    Result rendering/output can be triggered via the
    `datalad.api.result-renderer` configuration variable, or the
    `result_renderer` keyword argument of each decorated command. Supported
    modes are: 'default' (one line per result with action, status, path,
    and an optional message); 'json' (one object per result, like git-annex),
    'json_pp' (like 'json', but pretty-printed spanning multiple lines),
    'tailored' custom output formatting provided by each command
    class (if any).

    Error detection works by inspecting the `status` item of all result
    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'
    will cause an IncompleteResultsError exception to be raised that carries
    the failed actions' status dictionaries in its `failed` attribute.

    Status messages will be logged automatically, by default the following
    association of result status and log channel will be used: 'ok' (debug),
    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger
    instances included in the results are used to capture the origin of a
    status report.

    Parameters
    ----------
    func: function
      __call__ method of a subclass of Interface,
      i.e. a datalad command definition
    """

    @wrapt.decorator
    def eval_func(wrapped, instance, args, kwargs):
        lgr.log(2, "Entered eval_func for %s", func)
        # for result filters
        # we need to produce a dict with argname/argvalue pairs for all args
        # incl. defaults and args given as positionals
        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)

        # determine the command class associated with `wrapped`
        wrapped_class = get_wrapped_class(wrapped)

        # retrieve common options from kwargs, and fall back on the command
        # class attributes, or general defaults if needed
        kwargs = kwargs.copy()  # we will pop, which might cause side-effect
        common_params = {
            p_name: kwargs.pop(
                # go with any explicitly given default
                p_name,
                # otherwise determine the command class and pull any
                # default set in that class
                getattr(
                    wrapped_class,
                    p_name,
                    # or the common default
                    eval_defaults[p_name]))
            for p_name in eval_params}

        # short cuts and configured setup for common options
        return_type = common_params['return_type']
        result_filter = get_result_filter(common_params['result_filter'])
        # resolve string labels for transformers too
        result_xfm = known_result_xfms.get(
            common_params['result_xfm'],
            # use verbatim, if not a known label
            common_params['result_xfm'])
        result_renderer = common_params['result_renderer']
        # TODO remove this conditional branch entirely, done outside
        if not result_renderer:
            result_renderer = dlcfg.get('datalad.api.result-renderer', None)
        # look for potential override of logging behavior
        result_log_level = dlcfg.get('datalad.log.result-level', None)

        # query cfg for defaults
        # .is_installed and .config can be costly, so ensure we do
        # it only once. See https://github.com/datalad/datalad/issues/3575
        dataset_arg = allkwargs.get('dataset', None)
        from datalad.distribution.dataset import Dataset
        ds = dataset_arg if isinstance(dataset_arg, Dataset) \
            else Dataset(dataset_arg) if dataset_arg else None
        # look for hooks
        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)

        # this internal helper function actually drives the command
        # generator-style, it may generate an exception if desired,
        # on incomplete results
        def generator_func(*_args, **_kwargs):
            # flag whether to raise an exception
            incomplete_results = []
            # track what actions were performed how many times
            action_summary = {}

            # if a custom summary is to be provided, collect the results
            # of the command execution
            results = []
            do_custom_result_summary = result_renderer in ('tailored', 'default') \
                and hasattr(wrapped_class, 'custom_result_summary_renderer')

            # process main results
            for r in _process_results(
                    # execution
                    wrapped(*_args, **_kwargs),
                    wrapped_class,
                    common_params['on_failure'],
                    # bookkeeping
                    action_summary,
                    incomplete_results,
                    # communication
                    result_renderer,
                    result_log_level,
                    # let renderers get to see how a command was called
                    allkwargs):
                for hook, spec in hooks.items():
                    # run the hooks before we yield the result
                    # this ensures that they are executed before
                    # a potentially wrapper command gets to act
                    # on them
                    if match_jsonhook2result(hook, r, spec['match']):
                        lgr.debug('Result %s matches hook %s', r, hook)
                        # a hook is also a command that yields results
                        # so yield them outside too
                        # users need to pay attention to void infinite
                        # loops, i.e. when a hook yields a result that
                        # triggers that same hook again
                        for hr in run_jsonhook(hook, spec, r, dataset_arg):
                            # apply same logic as for main results, otherwise
                            # any filters would only tackle the primary results
                            # and a mixture of return values could happen
                            if not keep_result(hr, result_filter, **allkwargs):
                                continue
                            hr = xfm_result(hr, result_xfm)
                            # rationale for conditional is a few lines down
                            if hr:
                                yield hr
                if not keep_result(r, result_filter, **allkwargs):
                    continue
                r = xfm_result(r, result_xfm)
                # in case the result_xfm decided to not give us anything
                # exclude it from the results. There is no particular reason
                # to do so other than that it was established behavior when
                # this comment was written. This will not affect any real
                # result record
                if r:
                    yield r

                # collect if summary is desired
                if do_custom_result_summary:
                    results.append(r)

            # result summary before a potential exception
            # custom first
            if do_custom_result_summary:
                wrapped_class.custom_result_summary_renderer(results)
            elif result_renderer == 'default' and action_summary and \
                    sum(sum(s.values()) for s in action_summary.values()) > 1:
                # give a summary in default mode, when there was more than one
                # action performed
                ui.message("action summary:\n  {}".format(
                    '\n  '.join('{} ({})'.format(
                        act,
                        ', '.join('{}: {}'.format(status, action_summary[act][status])
                                  for status in sorted(action_summary[act])))
                                for act in sorted(action_summary))))

            if incomplete_results:
                raise IncompleteResultsError(
                    failed=incomplete_results,
                    msg="Command did not complete successfully")

        if return_type == 'generator':
            # hand over the generator
            lgr.log(2, "Returning generator_func from eval_func for %s", wrapped_class)
            return generator_func(*args, **kwargs)
        else:
            @wrapt.decorator
            def return_func(wrapped_, instance_, args_, kwargs_):
                results = wrapped_(*args_, **kwargs_)
                if inspect.isgenerator(results):
                    # unwind generator if there is one, this actually runs
                    # any processing
                    results = list(results)
                if return_type == 'item-or-list' and \
                        len(results) < 2:
                    return results[0] if results else None
                else:
                    return results
            lgr.log(2, "Returning return_func from eval_func for %s", wrapped_class)
            return return_func(generator_func)(*args, **kwargs)

    ret = eval_func(func)
    ret._eval_results = True
    return ret


def default_result_renderer(res):
    if res.get('status', None) != 'notneeded':
        path = res.get('path', None)
        if path and res.get('refds'):
            try:
                path = relpath(path, res['refds'])
            except ValueError:
                # can happen, e.g., on windows with paths from different
                # drives. just go with the original path in this case
                pass
        ui.message('{action}({status}):{path}{type}{msg}'.format(
            action=ac.color_word(
                res.get('action', '<action-unspecified>'),
                ac.BOLD),
            status=ac.color_status(res.get('status', '<status-unspecified>')),
            path=' {}'.format(path) if path else '',
            type=' ({})'.format(
                ac.color_word(res['type'], ac.MAGENTA)
            ) if 'type' in res else '',
            msg=' [{}]'.format(
                res['message'][0] % res['message'][1:]
                if isinstance(res['message'], tuple) else res[
                    'message'])
            if res.get('message', None) else ''))


def _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):
    # +1 because there was the original result + nsimilar displayed.
    n_suppressed = nsimilar - ndisplayed + 1
    if n_suppressed > 0:
        ts = time()
        # rate-limit update of suppression message, with a large number
        # of fast-paced results updating for each one can result in more
        # CPU load than the actual processing
        # arbitrarily go for a 2Hz update frequency -- it "feels" good
        if last_ts is None or final or (ts - last_ts > 0.5):
            ui.message('  [{} similar {} been suppressed; disable with datalad.ui.suppress-similar-results=off]'
                       .format(n_suppressed,
                               single_or_plural("message has",
                                                "messages have",
                                                n_suppressed, False)),
                       cr="\n" if final else "\r")
            return ts
    return last_ts


def _process_results(
        results,
        cmd_class,
        on_failure,
        action_summary,
        incomplete_results,
        result_renderer,
        result_log_level,
        allkwargs):
    # private helper pf @eval_results
    # loop over results generated from some source and handle each
    # of them according to the requested behavior (logging, rendering, ...)

    # used to track repeated messages in the default renderer
    last_result = None
    last_result_ts = None
    # which result dict keys to inspect for changes to discover repetitions
    # of similar messages
    repetition_keys = set(('action', 'status', 'type', 'refds'))
    # counter for detected repetitions
    result_repetitions = 0
    # how many repetitions to show, before suppression kicks in
    render_n_repetitions = \
        dlcfg.obtain('datalad.ui.suppress-similar-results-threshold') \
        if sys.stdout.isatty() \
        and dlcfg.obtain('datalad.ui.suppress-similar-results') \
        else float("inf")

    for res in results:
        if not res or 'action' not in res:
            # XXX Yarik has to no clue on how to track the origin of the
            # record to figure out WTF, so he just skips it
            # but MIH thinks leaving a trace of that would be good
            lgr.debug('Drop result record without "action": %s', res)
            continue

        actsum = action_summary.get(res['action'], {})
        if res['status']:
            actsum[res['status']] = actsum.get(res['status'], 0) + 1
            action_summary[res['action']] = actsum
        ## log message, if there is one and a logger was given
        msg = res.get('message', None)
        # remove logger instance from results, as it is no longer useful
        # after logging was done, it isn't serializable, and generally
        # pollutes the output
        res_lgr = res.pop('logger', None)
        if msg and res_lgr:
            if isinstance(res_lgr, logging.Logger):
                # didn't get a particular log function, go with default
                res_lgr = getattr(
                    res_lgr,
                    default_logchannels[res['status']]
                    if result_log_level is None
                    else result_log_level)
            msg = res['message']
            msgargs = None
            if isinstance(msg, tuple):
                msgargs = msg[1:]
                msg = msg[0]
            if 'path' in res:
                # result path could be a path instance
                path = str(res['path'])
                if msgargs:
                    # we will pass the msg for %-polation, so % should be doubled
                    path = path.replace('%', '%%')
                msg = '{} [{}({})]'.format(
                    msg, res['action'], path)
            if msgargs:
                # support string expansion of logging to avoid runtime cost
                try:
                    res_lgr(msg, *msgargs)
                except TypeError as exc:
                    raise TypeError(
                        "Failed to render %r with %r from %r: %s"
                        % (msg, msgargs, res, exc_str(exc))
                    )
            else:
                res_lgr(msg)

        ## output rendering
        # TODO RF this in a simple callable that gets passed into this function
        if result_renderer is None or result_renderer == 'disabled':
            pass
        elif result_renderer == 'default':
            trimmed_result = {k: v for k, v in res.items() if k in repetition_keys}
            if res.get('status', None) != 'notneeded' \
                    and trimmed_result == last_result:
                # this is a similar report, suppress if too many, but count it
                result_repetitions += 1
                if result_repetitions < render_n_repetitions:
                    default_result_renderer(res)
                else:
                    last_result_ts = _display_suppressed_message(
                        result_repetitions, render_n_repetitions, last_result_ts)
            else:
                # this one is new, first report on any prev. suppressed results
                # by number, and then render this fresh one
                last_result_ts = _display_suppressed_message(
                    result_repetitions, render_n_repetitions, last_result_ts,
                    final=True)
                default_result_renderer(res)
                result_repetitions = 0
            last_result = trimmed_result
        elif result_renderer in ('json', 'json_pp'):
            ui.message(json.dumps(
                {k: v for k, v in res.items()
                 if k not in ('logger')},
                sort_keys=True,
                indent=2 if result_renderer.endswith('_pp') else None,
                default=str))
        elif result_renderer in ('tailored', 'default'):
            if hasattr(cmd_class, 'custom_result_renderer'):
                cmd_class.custom_result_renderer(res, **allkwargs)
        elif hasattr(result_renderer, '__call__'):
            try:
                result_renderer(res, **allkwargs)
            except Exception as e:
                lgr.warning('Result rendering failed for: %s [%s]',
                            res, exc_str(e))
        else:
            raise ValueError('unknown result renderer "{}"'.format(result_renderer))

        ## error handling
        # looks for error status, and report at the end via
        # an exception
        if on_failure in ('continue', 'stop') \
                and res['status'] in ('impossible', 'error'):
            incomplete_results.append(res)
            if on_failure == 'stop':
                # first fail -> that's it
                # raise will happen after the loop
                break
        yield res
    # make sure to report on any issues that we had suppressed
    _display_suppressed_message(
        result_repetitions, render_n_repetitions, last_result_ts, final=True)


def keep_result(res, rfilter, **kwargs):
    if not rfilter:
        return True
    try:
        if not rfilter(res, **kwargs):
            # give the slightest indication which filter was employed
            raise ValueError(
                'excluded by filter {} with arguments {}'.format(rfilter, kwargs))
    except ValueError as e:
        # make sure to report the excluded result to massively improve
        # debugging experience
        lgr.debug('Not reporting result (%s): %s', exc_str(e), res)
        return False
    return True


def xfm_result(res, xfm):
    if not xfm:
        return res

    return xfm(res)

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** `custom_result_summary_renderer` Raises IndexError When `action_summary` Is Missing

**Description:**
When executing a push action, the `custom_result_summary_renderer` function expects both `results` and `action_summary` as arguments. However, in some scenarios, it is invoked without the `action_summary`, leading to an `IndexError`.

**Example Code:**
```python
@staticmethod
def custom_result_summary_renderer(results, action_summary):  # Expected two arguments
    hints = set([r.get('hints', None) for r in results])
    # Processing logic...
```

**Issue Behavior:**
Calling `custom_result_summary_renderer` with only the `results` parameter causes the following error:
```
IndexError: tuple index out of range
```

**Expected Behavior:**
The `custom_result_summary_renderer` should handle cases where `action_summary` is not provided, allowing it to operate without raising an error.

[/ISSUE]
{
    "test_eval_results_plus_build_doc": "PASSED",
    "test_result_filter": "PASSED",
    "test_utils_suppress_similar": "PASSED",
    "test_incorrect_msg_interpolation": "PASSED",
    "test_custom_result_summary_renderer": "PASSED",
    "test_discover_ds_trace": "ERROR",
    "test_invalid_call": "ERROR",
    "test_push_recursive": "ERROR",
    "test_push_subds_no_recursion": "ERROR",
    "test_force_checkdatapresent": "ERROR",
    "test_ria_push": "ERROR",
    "test_gh1426": "ERROR",
    "test_publish_target_url": "ERROR",
    "test_gh1763": "ERROR",
    "test_gh1811": "ERROR",
    "test_push_wanted": "ERROR",
    "test_nested_pushclone_cycle_allplatforms": "ERROR",
    "test_dirty": "FAILED",
    "test_save_hierarchy": "FAILED",
    "test_auto_data_transfer": "FAILED",
    "test_auto_if_wanted_data_transfer_path_restriction": "FAILED",
    "test_push_git_annex_branch_when_no_data": "FAILED",
    "test_push_git_annex_branch_many_paths_same_data": "FAILED",
    "test_push_matching": "FAILED",
    "test_push_custom_summary": "FAILED"
}
diff --git a/datalad/core/distributed/push.py b/datalad/core/distributed/push.py
index 147fe7104..5af9c6e67 100644
--- a/datalad/core/distributed/push.py
+++ b/datalad/core/distributed/push.py
@@ -28,6 +28,7 @@ from datalad.interface.common_opts import (
 )
 from datalad.interface.utils import (
     eval_results,
+    render_action_summary,
 )
 from datalad.interface.results import annexjson2result
 from datalad.log import log_progress
@@ -273,8 +274,10 @@ class Push(Interface):
                 path=ds.path,
             )
 
+    custom_result_summary_renderer_pass_summary = True
+
     @staticmethod
-    def custom_result_summary_renderer(results):  # pragma: more cover
+    def custom_result_summary_renderer(results, action_summary):  # pragma: more cover
         # report on any hints at the end
         # get all unique hints
         hints = set([r.get('hints', None) for r in results])
@@ -290,6 +293,8 @@ class Push(Interface):
                 ansi_colors.color_word(id + 1, ansi_colors.YELLOW), hint))
                 for id, hint in enumerate(hints)]
 
+        render_action_summary(action_summary)
+
 
 def _datasets_since_(dataset, since, paths, recursive, recursion_limit):
     """Generator"""
diff --git a/datalad/interface/utils.py b/datalad/interface/utils.py
index 25f8d07c5..02ea04381 100644
--- a/datalad/interface/utils.py
+++ b/datalad/interface/utils.py
@@ -388,6 +388,9 @@ def eval_results(func):
             results = []
             do_custom_result_summary = result_renderer in ('tailored', 'default') \
                 and hasattr(wrapped_class, 'custom_result_summary_renderer')
+            pass_summary = do_custom_result_summary and \
+                getattr(wrapped_class,
+                        'custom_result_summary_renderer_pass_summary', None)
 
             # process main results
             for r in _process_results(
@@ -443,17 +446,16 @@ def eval_results(func):
             # result summary before a potential exception
             # custom first
             if do_custom_result_summary:
-                wrapped_class.custom_result_summary_renderer(results)
+                if pass_summary:
+                    summary_args = (results, action_summary)
+                else:
+                    summary_args = (results,)
+                wrapped_class.custom_result_summary_renderer(*summary_args)
             elif result_renderer == 'default' and action_summary and \
                     sum(sum(s.values()) for s in action_summary.values()) > 1:
                 # give a summary in default mode, when there was more than one
                 # action performed
-                ui.message("action summary:\n  {}".format(
-                    '\n  '.join('{} ({})'.format(
-                        act,
-                        ', '.join('{}: {}'.format(status, action_summary[act][status])
-                                  for status in sorted(action_summary[act])))
-                                for act in sorted(action_summary))))
+                render_action_summary(action_summary)
 
             if incomplete_results:
                 raise IncompleteResultsError(
@@ -511,6 +513,16 @@ def default_result_renderer(res):
             if res.get('message', None) else ''))
 
 
+def render_action_summary(action_summary):
+    ui.message("action summary:\n  {}".format(
+        '\n  '.join('{} ({})'.format(
+            act,
+            ', '.join('{}: {}'.format(status, action_summary[act][status])
+                      for status in sorted(action_summary[act])))
+                    for act in sorted(action_summary))))
+
+
+
 def _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):
     # +1 because there was the original result + nsimilar displayed.
     n_suppressed = nsimilar - ndisplayed + 1
 

{"datalad/interface/utils.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface utility functions\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport inspect\nimport logging\nimport wrapt\nimport sys\nimport re\nfrom time import time\nfrom os import curdir\nfrom os import pardir\nfrom os import listdir\nimport os.path as op\nfrom os.path import join as opj\nfrom os.path import isdir\nfrom os.path import relpath\nfrom os.path import sep\nfrom os.path import split as psplit\nfrom itertools import chain\n\nimport json\n\n# avoid import from API to not get into circular imports\nfrom datalad.utils import with_pathsep as _with_sep  # TODO: RF whenever merge conflict is not upon us\nfrom datalad.utils import (\n    path_startswith,\n    path_is_subpath,\n    ensure_unicode,\n    getargspec,\n    get_wrapped_class,\n)\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad import cfg as dlcfg\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\n\nfrom datalad.ui import ui\nimport datalad.support.ansi_colors as ac\n\nfrom datalad.interface.base import default_logchannels\nfrom datalad.interface.base import get_allargs_as_kwargs\nfrom datalad.interface.common_opts import eval_params\nfrom datalad.interface.common_opts import eval_defaults\nfrom .results import known_result_xfms\nfrom datalad.core.local.resulthooks import (\n    get_jsonhooks_from_config,\n    match_jsonhook2result,\n    run_jsonhook,\n)\n\n\nlgr = logging.getLogger('datalad.interface.utils')\n\n\ndef cls2cmdlinename(cls):\n    \"Return the cmdline command name from an Interface class\"\n    r = re.compile(r'([a-z0-9])([A-Z])')\n    return r.sub('\\\\1-\\\\2', cls.__name__).lower()\n\n\n# TODO remove\n# only `drop` and `uninstall` are still using this\ndef handle_dirty_dataset(ds, mode, msg=None):\n    \"\"\"Detect and treat unsaved changes as instructed by `mode`\n\n    Parameters\n    ----------\n    ds : Dataset or None\n      Dataset to be inspected. Does nothing if `None`.\n    mode : {'fail', 'ignore', 'save-before'}\n      How to act upon discovering unsaved changes.\n    msg : str or None\n      Custom message to use for a potential commit.\n\n    Returns\n    -------\n    None\n    \"\"\"\n    if ds is None:\n        # nothing to be handled\n        return\n    if msg is None:\n        msg = '[DATALAD] auto-saved changes'\n\n    # make sure that all pending changes (batched annex operations, etc.)\n    # are actually reflected in Git\n    if ds.repo:\n        ds.repo.precommit()\n\n    if mode == 'ignore':\n        return\n    elif mode == 'fail':\n        if not ds.repo or ds.repo.dirty:\n            raise RuntimeError('dataset {} has unsaved changes'.format(ds))\n    elif mode == 'save-before':\n        if not ds.is_installed():\n            raise RuntimeError('dataset {} is not yet installed'.format(ds))\n        from datalad.core.local.save import Save\n        Save.__call__(dataset=ds, message=msg, updated=True)\n    else:\n        raise ValueError(\"unknown if-dirty mode '{}'\".format(mode))\n\n\ndef get_tree_roots(paths):\n    \"\"\"Return common root paths for a set of paths\n\n    This function determines the smallest set of common root\n    paths and sorts all given paths under the respective\n    root.\n\n    Returns\n    -------\n    dict\n      paths by root\n    \"\"\"\n    paths_ws = [_with_sep(p) for p in paths]\n    # sort all paths under their potential roots\n    roots = {}\n    # start from the top to get all paths down the line\n    # and collate them into as few roots as possible\n    for s in sorted(paths_ws):\n        if any([s.startswith(r) for r in roots]):\n            # this path is already covered by a known root\n            continue\n        # find all sub paths\n        subs = [p for p in paths if p.startswith(s)]\n        roots[s.rstrip(sep)] = subs\n    return roots\n\n\n# TODO remove\n# only `remove` and `uninstall` use this, the uses path `path_is_subpath`\ndef path_is_under(values, path=None):\n    \"\"\"Whether a given path is a subdirectory of any of the given test values\n\n    Parameters\n    ----------\n    values : sequence or dict\n      Paths to be tested against. This can be a dictionary in which case\n      all values from all keys will be tested against.\n    path : path or None\n      Test path. If None is given, the process' working directory is\n      used.\n\n    Returns\n    -------\n    bool\n    \"\"\"\n    if path is None:\n        from datalad.utils import getpwd\n        path = getpwd()\n    if isinstance(values, dict):\n        values = chain(*values.values())\n    path_drive, _ = op.splitdrive(path)\n    for p in values:\n        p_drive, _ = op.splitdrive(p)\n        # need to protect against unsupported use of relpath() with\n        # abspaths on windows from different drives (gh-3724)\n        if path_drive != p_drive:\n        # different drives, enough evidence for \"not under\"\n            continue\n        rpath = relpath(p, start=path)\n        if rpath == curdir \\\n                or rpath == pardir \\\n                or set(psplit(rpath)) == {pardir}:\n            # first match is enough\n            return True\n    return False\n\n\n# TODO(OPT)? YOH: from a cursory review seems like possibly an expensive function\n# whenever many paths were provided (e.g. via shell glob).\n# Might be worth testing on some usecase and py-spy'ing if notable portion\n# of time is spent.\ndef discover_dataset_trace_to_targets(basepath, targetpaths, current_trace,\n                                      spec, includeds=None):\n    \"\"\"Discover the edges and nodes in a dataset tree to given target paths\n\n    Parameters\n    ----------\n    basepath : path\n      Path to a start or top-level dataset. Really has to be a path to a\n      dataset!\n    targetpaths : list(path)\n      Any non-zero number of paths that are termination points for the\n      search algorithm. Can be paths to datasets, directories, or files\n      (and any combination thereof).\n    current_trace : list\n      For a top-level call this should probably always be `[]`\n    spec : dict\n      `content_by_ds`-style dictionary that will receive information about the\n      discovered datasets. Specifically, for each discovered dataset there\n      will be an item with its path under the key (path) of the respective\n      superdataset.\n    includeds : sequence, optional\n      Any paths given are treated as existing subdatasets, regardless of\n      whether they can be found in the filesystem. Such subdatasets will appear\n      under the key of the closest existing dataset in the `spec`.\n\n    Returns\n    -------\n    None\n      Function calls itself recursively and populates `spec` dict in-place.\n      Keys are dataset paths, values are sets of subdataset paths\n    \"\"\"\n    # convert to set for faster lookup\n    includeds = includeds if isinstance(includeds, set) else \\\n        set() if includeds is None else set(includeds)\n    # this beast walks the directory tree from a given `basepath` until\n    # it discovers any of the given `targetpaths`\n    # if it finds one, it commits any accummulated trace of visited\n    # datasets on this edge to the spec\n    valid_repo = GitRepo.is_valid_repo(basepath)\n    if valid_repo:\n        # we are passing into a new dataset, extend the dataset trace\n        current_trace = current_trace + [basepath]\n    # this edge is not done, we need to try to reach any downstream\n    # dataset\n    undiscovered_ds = set(t for t in targetpaths) # if t != basepath)\n    # whether anything in this directory matched a targetpath\n    filematch = False\n    if isdir(basepath):\n        for p in listdir(basepath):\n            p = ensure_unicode(opj(basepath, p))\n            if not isdir(p):\n                if p in targetpaths:\n                    filematch = True\n                # we cannot have anything below this one\n                continue\n            # OPT listdir might be large and we could have only few items\n            # in `targetpaths` -- so traverse only those in spec which have\n            # leading dir basepath\n            # filter targets matching this downward path\n            downward_targets = set(\n                t for t in targetpaths if path_startswith(t, p))\n            if not downward_targets:\n                continue\n            # remove the matching ones from the \"todo\" list\n            undiscovered_ds.difference_update(downward_targets)\n            # go one deeper\n            discover_dataset_trace_to_targets(\n                p, downward_targets, current_trace, spec,\n                includeds=includeds if not includeds else includeds.intersection(\n                    downward_targets))\n    undiscovered_ds = [t for t in undiscovered_ds\n                       if includeds and\n                          path_is_subpath(t, current_trace[-1]) and\n                          t in includeds]\n    if filematch or basepath in targetpaths or undiscovered_ds:\n        for i, p in enumerate(current_trace[:-1]):\n            # TODO RF prepare proper annotated path dicts\n            subds = spec.get(p, set())\n            subds.add(current_trace[i + 1])\n            spec[p] = subds\n        if undiscovered_ds:\n            spec[current_trace[-1]] = spec.get(current_trace[-1], set()).union(\n                undiscovered_ds)\n\n\ndef get_result_filter(fx):\n    \"\"\"Wrap a filter into a helper to be able to accept additional\n    arguments, if the filter doesn't support it already\"\"\"\n    _fx = fx\n    if fx and not getargspec(fx).keywords:\n        def _fx(res, **kwargs):\n            return fx(res)\n    return _fx\n\n\ndef eval_results(func):\n    \"\"\"Decorator for return value evaluation of datalad commands.\n\n    Note, this decorator is only compatible with commands that return\n    status dict sequences!\n\n    Two basic modes of operation are supported: 1) \"generator mode\" that\n    `yields` individual results, and 2) \"list mode\" that returns a sequence of\n    results. The behavior can be selected via the kwarg `return_type`.\n    Default is \"list mode\".\n\n    This decorator implements common functionality for result rendering/output,\n    error detection/handling, and logging.\n\n    Result rendering/output can be triggered via the\n    `datalad.api.result-renderer` configuration variable, or the\n    `result_renderer` keyword argument of each decorated command. Supported\n    modes are: 'default' (one line per result with action, status, path,\n    and an optional message); 'json' (one object per result, like git-annex),\n    'json_pp' (like 'json', but pretty-printed spanning multiple lines),\n    'tailored' custom output formatting provided by each command\n    class (if any).\n\n    Error detection works by inspecting the `status` item of all result\n    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'\n    will cause an IncompleteResultsError exception to be raised that carries\n    the failed actions' status dictionaries in its `failed` attribute.\n\n    Status messages will be logged automatically, by default the following\n    association of result status and log channel will be used: 'ok' (debug),\n    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger\n    instances included in the results are used to capture the origin of a\n    status report.\n\n    Parameters\n    ----------\n    func: function\n      __call__ method of a subclass of Interface,\n      i.e. a datalad command definition\n    \"\"\"\n\n    @wrapt.decorator\n    def eval_func(wrapped, instance, args, kwargs):\n        lgr.log(2, \"Entered eval_func for %s\", func)\n        # for result filters\n        # we need to produce a dict with argname/argvalue pairs for all args\n        # incl. defaults and args given as positionals\n        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)\n\n        # determine the command class associated with `wrapped`\n        wrapped_class = get_wrapped_class(wrapped)\n\n        # retrieve common options from kwargs, and fall back on the command\n        # class attributes, or general defaults if needed\n        kwargs = kwargs.copy()  # we will pop, which might cause side-effect\n        common_params = {\n            p_name: kwargs.pop(\n                # go with any explicitly given default\n                p_name,\n                # otherwise determine the command class and pull any\n                # default set in that class\n                getattr(\n                    wrapped_class,\n                    p_name,\n                    # or the common default\n                    eval_defaults[p_name]))\n            for p_name in eval_params}\n\n        # short cuts and configured setup for common options\n        return_type = common_params['return_type']\n        result_filter = get_result_filter(common_params['result_filter'])\n        # resolve string labels for transformers too\n        result_xfm = known_result_xfms.get(\n            common_params['result_xfm'],\n            # use verbatim, if not a known label\n            common_params['result_xfm'])\n        result_renderer = common_params['result_renderer']\n        # TODO remove this conditional branch entirely, done outside\n        if not result_renderer:\n            result_renderer = dlcfg.get('datalad.api.result-renderer', None)\n        # look for potential override of logging behavior\n        result_log_level = dlcfg.get('datalad.log.result-level', None)\n\n        # query cfg for defaults\n        # .is_installed and .config can be costly, so ensure we do\n        # it only once. See https://github.com/datalad/datalad/issues/3575\n        dataset_arg = allkwargs.get('dataset', None)\n        from datalad.distribution.dataset import Dataset\n        ds = dataset_arg if isinstance(dataset_arg, Dataset) \\\n            else Dataset(dataset_arg) if dataset_arg else None\n        # look for hooks\n        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)\n\n        # this internal helper function actually drives the command\n        # generator-style, it may generate an exception if desired,\n        # on incomplete results\n        def generator_func(*_args, **_kwargs):\n            # flag whether to raise an exception\n            incomplete_results = []\n            # track what actions were performed how many times\n            action_summary = {}\n\n            # if a custom summary is to be provided, collect the results\n            # of the command execution\n            results = []\n            do_custom_result_summary = result_renderer in ('tailored', 'default') \\\n                and hasattr(wrapped_class, 'custom_result_summary_renderer')\n\n            # process main results\n            for r in _process_results(\n                    # execution\n                    wrapped(*_args, **_kwargs),\n                    wrapped_class,\n                    common_params['on_failure'],\n                    # bookkeeping\n                    action_summary,\n                    incomplete_results,\n                    # communication\n                    result_renderer,\n                    result_log_level,\n                    # let renderers get to see how a command was called\n                    allkwargs):\n                for hook, spec in hooks.items():\n                    # run the hooks before we yield the result\n                    # this ensures that they are executed before\n                    # a potentially wrapper command gets to act\n                    # on them\n                    if match_jsonhook2result(hook, r, spec['match']):\n                        lgr.debug('Result %s matches hook %s', r, hook)\n                        # a hook is also a command that yields results\n                        # so yield them outside too\n                        # users need to pay attention to void infinite\n                        # loops, i.e. when a hook yields a result that\n                        # triggers that same hook again\n                        for hr in run_jsonhook(hook, spec, r, dataset_arg):\n                            # apply same logic as for main results, otherwise\n                            # any filters would only tackle the primary results\n                            # and a mixture of return values could happen\n                            if not keep_result(hr, result_filter, **allkwargs):\n                                continue\n                            hr = xfm_result(hr, result_xfm)\n                            # rationale for conditional is a few lines down\n                            if hr:\n                                yield hr\n                if not keep_result(r, result_filter, **allkwargs):\n                    continue\n                r = xfm_result(r, result_xfm)\n                # in case the result_xfm decided to not give us anything\n                # exclude it from the results. There is no particular reason\n                # to do so other than that it was established behavior when\n                # this comment was written. This will not affect any real\n                # result record\n                if r:\n                    yield r\n\n                # collect if summary is desired\n                if do_custom_result_summary:\n                    results.append(r)\n\n            # result summary before a potential exception\n            # custom first\n            if do_custom_result_summary:\n                wrapped_class.custom_result_summary_renderer(results)\n            elif result_renderer == 'default' and action_summary and \\\n                    sum(sum(s.values()) for s in action_summary.values()) > 1:\n                # give a summary in default mode, when there was more than one\n                # action performed\n                ui.message(\"action summary:\\n  {}\".format(\n                    '\\n  '.join('{} ({})'.format(\n                        act,\n                        ', '.join('{}: {}'.format(status, action_summary[act][status])\n                                  for status in sorted(action_summary[act])))\n                                for act in sorted(action_summary))))\n\n            if incomplete_results:\n                raise IncompleteResultsError(\n                    failed=incomplete_results,\n                    msg=\"Command did not complete successfully\")\n\n        if return_type == 'generator':\n            # hand over the generator\n            lgr.log(2, \"Returning generator_func from eval_func for %s\", wrapped_class)\n            return generator_func(*args, **kwargs)\n        else:\n            @wrapt.decorator\n            def return_func(wrapped_, instance_, args_, kwargs_):\n                results = wrapped_(*args_, **kwargs_)\n                if inspect.isgenerator(results):\n                    # unwind generator if there is one, this actually runs\n                    # any processing\n                    results = list(results)\n                if return_type == 'item-or-list' and \\\n                        len(results) < 2:\n                    return results[0] if results else None\n                else:\n                    return results\n            lgr.log(2, \"Returning return_func from eval_func for %s\", wrapped_class)\n            return return_func(generator_func)(*args, **kwargs)\n\n    ret = eval_func(func)\n    ret._eval_results = True\n    return ret\n\n\ndef default_result_renderer(res):\n    if res.get('status', None) != 'notneeded':\n        path = res.get('path', None)\n        if path and res.get('refds'):\n            try:\n                path = relpath(path, res['refds'])\n            except ValueError:\n                # can happen, e.g., on windows with paths from different\n                # drives. just go with the original path in this case\n                pass\n        ui.message('{action}({status}):{path}{type}{msg}'.format(\n            action=ac.color_word(\n                res.get('action', '<action-unspecified>'),\n                ac.BOLD),\n            status=ac.color_status(res.get('status', '<status-unspecified>')),\n            path=' {}'.format(path) if path else '',\n            type=' ({})'.format(\n                ac.color_word(res['type'], ac.MAGENTA)\n            ) if 'type' in res else '',\n            msg=' [{}]'.format(\n                res['message'][0] % res['message'][1:]\n                if isinstance(res['message'], tuple) else res[\n                    'message'])\n            if res.get('message', None) else ''))\n\n\ndef _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):\n    # +1 because there was the original result + nsimilar displayed.\n    n_suppressed = nsimilar - ndisplayed + 1\n    if n_suppressed > 0:\n        ts = time()\n        # rate-limit update of suppression message, with a large number\n        # of fast-paced results updating for each one can result in more\n        # CPU load than the actual processing\n        # arbitrarily go for a 2Hz update frequency -- it \"feels\" good\n        if last_ts is None or final or (ts - last_ts > 0.5):\n            ui.message('  [{} similar {} been suppressed; disable with datalad.ui.suppress-similar-results=off]'\n                       .format(n_suppressed,\n                               single_or_plural(\"message has\",\n                                                \"messages have\",\n                                                n_suppressed, False)),\n                       cr=\"\\n\" if final else \"\\r\")\n            return ts\n    return last_ts\n\n\ndef _process_results(\n        results,\n        cmd_class,\n        on_failure,\n        action_summary,\n        incomplete_results,\n        result_renderer,\n        result_log_level,\n        allkwargs):\n    # private helper pf @eval_results\n    # loop over results generated from some source and handle each\n    # of them according to the requested behavior (logging, rendering, ...)\n\n    # used to track repeated messages in the default renderer\n    last_result = None\n    last_result_ts = None\n    # which result dict keys to inspect for changes to discover repetitions\n    # of similar messages\n    repetition_keys = set(('action', 'status', 'type', 'refds'))\n    # counter for detected repetitions\n    result_repetitions = 0\n    # how many repetitions to show, before suppression kicks in\n    render_n_repetitions = \\\n        dlcfg.obtain('datalad.ui.suppress-similar-results-threshold') \\\n        if sys.stdout.isatty() \\\n        and dlcfg.obtain('datalad.ui.suppress-similar-results') \\\n        else float(\"inf\")\n\n    for res in results:\n        if not res or 'action' not in res:\n            # XXX Yarik has to no clue on how to track the origin of the\n            # record to figure out WTF, so he just skips it\n            # but MIH thinks leaving a trace of that would be good\n            lgr.debug('Drop result record without \"action\": %s', res)\n            continue\n\n        actsum = action_summary.get(res['action'], {})\n        if res['status']:\n            actsum[res['status']] = actsum.get(res['status'], 0) + 1\n            action_summary[res['action']] = actsum\n        ## log message, if there is one and a logger was given\n        msg = res.get('message', None)\n        # remove logger instance from results, as it is no longer useful\n        # after logging was done, it isn't serializable, and generally\n        # pollutes the output\n        res_lgr = res.pop('logger', None)\n        if msg and res_lgr:\n            if isinstance(res_lgr, logging.Logger):\n                # didn't get a particular log function, go with default\n                res_lgr = getattr(\n                    res_lgr,\n                    default_logchannels[res['status']]\n                    if result_log_level is None\n                    else result_log_level)\n            msg = res['message']\n            msgargs = None\n            if isinstance(msg, tuple):\n                msgargs = msg[1:]\n                msg = msg[0]\n            if 'path' in res:\n                # result path could be a path instance\n                path = str(res['path'])\n                if msgargs:\n                    # we will pass the msg for %-polation, so % should be doubled\n                    path = path.replace('%', '%%')\n                msg = '{} [{}({})]'.format(\n                    msg, res['action'], path)\n            if msgargs:\n                # support string expansion of logging to avoid runtime cost\n                try:\n                    res_lgr(msg, *msgargs)\n                except TypeError as exc:\n                    raise TypeError(\n                        \"Failed to render %r with %r from %r: %s\"\n                        % (msg, msgargs, res, exc_str(exc))\n                    )\n            else:\n                res_lgr(msg)\n\n        ## output rendering\n        # TODO RF this in a simple callable that gets passed into this function\n        if result_renderer is None or result_renderer == 'disabled':\n            pass\n        elif result_renderer == 'default':\n            trimmed_result = {k: v for k, v in res.items() if k in repetition_keys}\n            if res.get('status', None) != 'notneeded' \\\n                    and trimmed_result == last_result:\n                # this is a similar report, suppress if too many, but count it\n                result_repetitions += 1\n                if result_repetitions < render_n_repetitions:\n                    default_result_renderer(res)\n                else:\n                    last_result_ts = _display_suppressed_message(\n                        result_repetitions, render_n_repetitions, last_result_ts)\n            else:\n                # this one is new, first report on any prev. suppressed results\n                # by number, and then render this fresh one\n                last_result_ts = _display_suppressed_message(\n                    result_repetitions, render_n_repetitions, last_result_ts,\n                    final=True)\n                default_result_renderer(res)\n                result_repetitions = 0\n            last_result = trimmed_result\n        elif result_renderer in ('json', 'json_pp'):\n            ui.message(json.dumps(\n                {k: v for k, v in res.items()\n                 if k not in ('logger')},\n                sort_keys=True,\n                indent=2 if result_renderer.endswith('_pp') else None,\n                default=str))\n        elif result_renderer in ('tailored', 'default'):\n            if hasattr(cmd_class, 'custom_result_renderer'):\n                cmd_class.custom_result_renderer(res, **allkwargs)\n        elif hasattr(result_renderer, '__call__'):\n            try:\n                result_renderer(res, **allkwargs)\n            except Exception as e:\n                lgr.warning('Result rendering failed for: %s [%s]',\n                            res, exc_str(e))\n        else:\n            raise ValueError('unknown result renderer \"{}\"'.format(result_renderer))\n\n        ## error handling\n        # looks for error status, and report at the end via\n        # an exception\n        if on_failure in ('continue', 'stop') \\\n                and res['status'] in ('impossible', 'error'):\n            incomplete_results.append(res)\n            if on_failure == 'stop':\n                # first fail -> that's it\n                # raise will happen after the loop\n                break\n        yield res\n    # make sure to report on any issues that we had suppressed\n    _display_suppressed_message(\n        result_repetitions, render_n_repetitions, last_result_ts, final=True)\n\n\ndef keep_result(res, rfilter, **kwargs):\n    if not rfilter:\n        return True\n    try:\n        if not rfilter(res, **kwargs):\n            # give the slightest indication which filter was employed\n            raise ValueError(\n                'excluded by filter {} with arguments {}'.format(rfilter, kwargs))\n    except ValueError as e:\n        # make sure to report the excluded result to massively improve\n        # debugging experience\n        lgr.debug('Not reporting result (%s): %s', exc_str(e), res)\n        return False\n    return True\n\n\ndef xfm_result(res, xfm):\n    if not xfm:\n        return res\n\n    return xfm(res)\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** `custom_result_summary_renderer` Raises IndexError When `action_summary` Is Missing

**Description:**
When executing a push action, the `custom_result_summary_renderer` function expects both `results` and `action_summary` as arguments. However, in some scenarios, it is invoked without the `action_summary`, leading to an `IndexError`.

**Example Code:**
```python
@staticmethod
def custom_result_summary_renderer(results, action_summary):  # Expected two arguments
    hints = set([r.get('hints', None) for r in results])
    # Processing logic...
```

**Issue Behavior:**
Calling `custom_result_summary_renderer` with only the `results` parameter causes the following error:
```
IndexError: tuple index out of range
```

**Expected Behavior:**
The `custom_result_summary_renderer` should handle cases where `action_summary` is not provided, allowing it to operate without raising an error.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/utils.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface utility functions

"""

__docformat__ = 'restructuredtext'

import inspect
import logging
import wrapt
import sys
import re
from time import time
from os import curdir
from os import pardir
from os import listdir
import os.path as op
from os.path import join as opj
from os.path import isdir
from os.path import relpath
from os.path import sep
from os.path import split as psplit
from itertools import chain

import json

# avoid import from API to not get into circular imports
from datalad.utils import with_pathsep as _with_sep  # TODO: RF whenever merge conflict is not upon us
from datalad.utils import (
    path_startswith,
    path_is_subpath,
    ensure_unicode,
    getargspec,
    get_wrapped_class,
)
from datalad.support.gitrepo import GitRepo
from datalad.support.exceptions import IncompleteResultsError
from datalad import cfg as dlcfg
from datalad.dochelpers import (
    exc_str,
    single_or_plural,
)

from datalad.ui import ui
import datalad.support.ansi_colors as ac

from datalad.interface.base import default_logchannels
from datalad.interface.base import get_allargs_as_kwargs
from datalad.interface.common_opts import eval_params
from datalad.interface.common_opts import eval_defaults
from .results import known_result_xfms
from datalad.core.local.resulthooks import (
    get_jsonhooks_from_config,
    match_jsonhook2result,
    run_jsonhook,
)


lgr = logging.getLogger('datalad.interface.utils')


def cls2cmdlinename(cls):
    "Return the cmdline command name from an Interface class"
    r = re.compile(r'([a-z0-9])([A-Z])')
    return r.sub('\\1-\\2', cls.__name__).lower()


# TODO remove
# only `drop` and `uninstall` are still using this
def handle_dirty_dataset(ds, mode, msg=None):
    """Detect and treat unsaved changes as instructed by `mode`

    Parameters
    ----------
    ds : Dataset or None
      Dataset to be inspected. Does nothing if `None`.
    mode : {'fail', 'ignore', 'save-before'}
      How to act upon discovering unsaved changes.
    msg : str or None
      Custom message to use for a potential commit.

    Returns
    -------
    None
    """
    if ds is None:
        # nothing to be handled
        return
    if msg is None:
        msg = '[DATALAD] auto-saved changes'

    # make sure that all pending changes (batched annex operations, etc.)
    # are actually reflected in Git
    if ds.repo:
        ds.repo.precommit()

    if mode == 'ignore':
        return
    elif mode == 'fail':
        if not ds.repo or ds.repo.dirty:
            raise RuntimeError('dataset {} has unsaved changes'.format(ds))
    elif mode == 'save-before':
        if not ds.is_installed():
            raise RuntimeError('dataset {} is not yet installed'.format(ds))
        from datalad.core.local.save import Save
        Save.__call__(dataset=ds, message=msg, updated=True)
    else:
        raise ValueError("unknown if-dirty mode '{}'".format(mode))


def get_tree_roots(paths):
    """Return common root paths for a set of paths

    This function determines the smallest set of common root
    paths and sorts all given paths under the respective
    root.

    Returns
    -------
    dict
      paths by root
    """
    paths_ws = [_with_sep(p) for p in paths]
    # sort all paths under their potential roots
    roots = {}
    # start from the top to get all paths down the line
    # and collate them into as few roots as possible
    for s in sorted(paths_ws):
        if any([s.startswith(r) for r in roots]):
            # this path is already covered by a known root
            continue
        # find all sub paths
        subs = [p for p in paths if p.startswith(s)]
        roots[s.rstrip(sep)] = subs
    return roots


# TODO remove
# only `remove` and `uninstall` use this, the uses path `path_is_subpath`
def path_is_under(values, path=None):
    """Whether a given path is a subdirectory of any of the given test values

    Parameters
    ----------
    values : sequence or dict
      Paths to be tested against. This can be a dictionary in which case
      all values from all keys will be tested against.
    path : path or None
      Test path. If None is given, the process' working directory is
      used.

    Returns
    -------
    bool
    """
    if path is None:
        from datalad.utils import getpwd
        path = getpwd()
    if isinstance(values, dict):
        values = chain(*values.values())
    path_drive, _ = op.splitdrive(path)
    for p in values:
        p_drive, _ = op.splitdrive(p)
        # need to protect against unsupported use of relpath() with
        # abspaths on windows from different drives (gh-3724)
        if path_drive != p_drive:
        # different drives, enough evidence for "not under"
            continue
        rpath = relpath(p, start=path)
        if rpath == curdir \
                or rpath == pardir \
                or set(psplit(rpath)) == {pardir}:
            # first match is enough
            return True
    return False


# TODO(OPT)? YOH: from a cursory review seems like possibly an expensive function
# whenever many paths were provided (e.g. via shell glob).
# Might be worth testing on some usecase and py-spy'ing if notable portion
# of time is spent.
def discover_dataset_trace_to_targets(basepath, targetpaths, current_trace,
                                      spec, includeds=None):
    """Discover the edges and nodes in a dataset tree to given target paths

    Parameters
    ----------
    basepath : path
      Path to a start or top-level dataset. Really has to be a path to a
      dataset!
    targetpaths : list(path)
      Any non-zero number of paths that are termination points for the
      search algorithm. Can be paths to datasets, directories, or files
      (and any combination thereof).
    current_trace : list
      For a top-level call this should probably always be `[]`
    spec : dict
      `content_by_ds`-style dictionary that will receive information about the
      discovered datasets. Specifically, for each discovered dataset there
      will be an item with its path under the key (path) of the respective
      superdataset.
    includeds : sequence, optional
      Any paths given are treated as existing subdatasets, regardless of
      whether they can be found in the filesystem. Such subdatasets will appear
      under the key of the closest existing dataset in the `spec`.

    Returns
    -------
    None
      Function calls itself recursively and populates `spec` dict in-place.
      Keys are dataset paths, values are sets of subdataset paths
    """
    # convert to set for faster lookup
    includeds = includeds if isinstance(includeds, set) else \
        set() if includeds is None else set(includeds)
    # this beast walks the directory tree from a given `basepath` until
    # it discovers any of the given `targetpaths`
    # if it finds one, it commits any accummulated trace of visited
    # datasets on this edge to the spec
    valid_repo = GitRepo.is_valid_repo(basepath)
    if valid_repo:
        # we are passing into a new dataset, extend the dataset trace
        current_trace = current_trace + [basepath]
    # this edge is not done, we need to try to reach any downstream
    # dataset
    undiscovered_ds = set(t for t in targetpaths) # if t != basepath)
    # whether anything in this directory matched a targetpath
    filematch = False
    if isdir(basepath):
        for p in listdir(basepath):
            p = ensure_unicode(opj(basepath, p))
            if not isdir(p):
                if p in targetpaths:
                    filematch = True
                # we cannot have anything below this one
                continue
            # OPT listdir might be large and we could have only few items
            # in `targetpaths` -- so traverse only those in spec which have
            # leading dir basepath
            # filter targets matching this downward path
            downward_targets = set(
                t for t in targetpaths if path_startswith(t, p))
            if not downward_targets:
                continue
            # remove the matching ones from the "todo" list
            undiscovered_ds.difference_update(downward_targets)
            # go one deeper
            discover_dataset_trace_to_targets(
                p, downward_targets, current_trace, spec,
                includeds=includeds if not includeds else includeds.intersection(
                    downward_targets))
    undiscovered_ds = [t for t in undiscovered_ds
                       if includeds and
                          path_is_subpath(t, current_trace[-1]) and
                          t in includeds]
    if filematch or basepath in targetpaths or undiscovered_ds:
        for i, p in enumerate(current_trace[:-1]):
            # TODO RF prepare proper annotated path dicts
            subds = spec.get(p, set())
            subds.add(current_trace[i + 1])
            spec[p] = subds
        if undiscovered_ds:
            spec[current_trace[-1]] = spec.get(current_trace[-1], set()).union(
                undiscovered_ds)


def get_result_filter(fx):
    """Wrap a filter into a helper to be able to accept additional
    arguments, if the filter doesn't support it already"""
    _fx = fx
    if fx and not getargspec(fx).keywords:
        def _fx(res, **kwargs):
            return fx(res)
    return _fx


def eval_results(func):
    """Decorator for return value evaluation of datalad commands.

    Note, this decorator is only compatible with commands that return
    status dict sequences!

    Two basic modes of operation are supported: 1) "generator mode" that
    `yields` individual results, and 2) "list mode" that returns a sequence of
    results. The behavior can be selected via the kwarg `return_type`.
    Default is "list mode".

    This decorator implements common functionality for result rendering/output,
    error detection/handling, and logging.

    Result rendering/output can be triggered via the
    `datalad.api.result-renderer` configuration variable, or the
    `result_renderer` keyword argument of each decorated command. Supported
    modes are: 'default' (one line per result with action, status, path,
    and an optional message); 'json' (one object per result, like git-annex),
    'json_pp' (like 'json', but pretty-printed spanning multiple lines),
    'tailored' custom output formatting provided by each command
    class (if any).

    Error detection works by inspecting the `status` item of all result
    dictionaries. Any occurrence of a status other than 'ok' or 'notneeded'
    will cause an IncompleteResultsError exception to be raised that carries
    the failed actions' status dictionaries in its `failed` attribute.

    Status messages will be logged automatically, by default the following
    association of result status and log channel will be used: 'ok' (debug),
    'notneeded' (debug), 'impossible' (warning), 'error' (error).  Logger
    instances included in the results are used to capture the origin of a
    status report.

    Parameters
    ----------
    func: function
      __call__ method of a subclass of Interface,
      i.e. a datalad command definition
    """

    @wrapt.decorator
    def eval_func(wrapped, instance, args, kwargs):
        lgr.log(2, "Entered eval_func for %s", func)
        # for result filters
        # we need to produce a dict with argname/argvalue pairs for all args
        # incl. defaults and args given as positionals
        allkwargs = get_allargs_as_kwargs(wrapped, args, kwargs)

        # determine the command class associated with `wrapped`
        wrapped_class = get_wrapped_class(wrapped)

        # retrieve common options from kwargs, and fall back on the command
        # class attributes, or general defaults if needed
        kwargs = kwargs.copy()  # we will pop, which might cause side-effect
        common_params = {
            p_name: kwargs.pop(
                # go with any explicitly given default
                p_name,
                # otherwise determine the command class and pull any
                # default set in that class
                getattr(
                    wrapped_class,
                    p_name,
                    # or the common default
                    eval_defaults[p_name]))
            for p_name in eval_params}

        # short cuts and configured setup for common options
        return_type = common_params['return_type']
        result_filter = get_result_filter(common_params['result_filter'])
        # resolve string labels for transformers too
        result_xfm = known_result_xfms.get(
            common_params['result_xfm'],
            # use verbatim, if not a known label
            common_params['result_xfm'])
        result_renderer = common_params['result_renderer']
        # TODO remove this conditional branch entirely, done outside
        if not result_renderer:
            result_renderer = dlcfg.get('datalad.api.result-renderer', None)
        # look for potential override of logging behavior
        result_log_level = dlcfg.get('datalad.log.result-level', None)

        # query cfg for defaults
        # .is_installed and .config can be costly, so ensure we do
        # it only once. See https://github.com/datalad/datalad/issues/3575
        dataset_arg = allkwargs.get('dataset', None)
        from datalad.distribution.dataset import Dataset
        ds = dataset_arg if isinstance(dataset_arg, Dataset) \
            else Dataset(dataset_arg) if dataset_arg else None
        # look for hooks
        hooks = get_jsonhooks_from_config(ds.config if ds else dlcfg)

        # this internal helper function actually drives the command
        # generator-style, it may generate an exception if desired,
        # on incomplete results
        def generator_func(*_args, **_kwargs):
            # flag whether to raise an exception
            incomplete_results = []
            # track what actions were performed how many times
            action_summary = {}

            # if a custom summary is to be provided, collect the results
            # of the command execution
            results = []
            do_custom_result_summary = result_renderer in ('tailored', 'default') \
                and hasattr(wrapped_class, 'custom_result_summary_renderer')

            # process main results
            for r in _process_results(
                    # execution
                    wrapped(*_args, **_kwargs),
                    wrapped_class,
                    common_params['on_failure'],
                    # bookkeeping
                    action_summary,
                    incomplete_results,
                    # communication
                    result_renderer,
                    result_log_level,
                    # let renderers get to see how a command was called
                    allkwargs):
                for hook, spec in hooks.items():
                    # run the hooks before we yield the result
                    # this ensures that they are executed before
                    # a potentially wrapper command gets to act
                    # on them
                    if match_jsonhook2result(hook, r, spec['match']):
                        lgr.debug('Result %s matches hook %s', r, hook)
                        # a hook is also a command that yields results
                        # so yield them outside too
                        # users need to pay attention to void infinite
                        # loops, i.e. when a hook yields a result that
                        # triggers that same hook again
                        for hr in run_jsonhook(hook, spec, r, dataset_arg):
                            # apply same logic as for main results, otherwise
                            # any filters would only tackle the primary results
                            # and a mixture of return values could happen
                            if not keep_result(hr, result_filter, **allkwargs):
                                continue
                            hr = xfm_result(hr, result_xfm)
                            # rationale for conditional is a few lines down
                            if hr:
                                yield hr
                if not keep_result(r, result_filter, **allkwargs):
                    continue
                r = xfm_result(r, result_xfm)
                # in case the result_xfm decided to not give us anything
                # exclude it from the results. There is no particular reason
                # to do so other than that it was established behavior when
                # this comment was written. This will not affect any real
                # result record
                if r:
                    yield r

                # collect if summary is desired
                if do_custom_result_summary:
                    results.append(r)

            # result summary before a potential exception
            # custom first
            if do_custom_result_summary:
                wrapped_class.custom_result_summary_renderer(results)
            elif result_renderer == 'default' and action_summary and \
                    sum(sum(s.values()) for s in action_summary.values()) > 1:
                # give a summary in default mode, when there was more than one
                # action performed
                ui.message("action summary:\n  {}".format(
                    '\n  '.join('{} ({})'.format(
                        act,
                        ', '.join('{}: {}'.format(status, action_summary[act][status])
                                  for status in sorted(action_summary[act])))
                                for act in sorted(action_summary))))

            if incomplete_results:
                raise IncompleteResultsError(
                    failed=incomplete_results,
                    msg="Command did not complete successfully")

        if return_type == 'generator':
            # hand over the generator
            lgr.log(2, "Returning generator_func from eval_func for %s", wrapped_class)
            return generator_func(*args, **kwargs)
        else:
            @wrapt.decorator
            def return_func(wrapped_, instance_, args_, kwargs_):
                results = wrapped_(*args_, **kwargs_)
                if inspect.isgenerator(results):
                    # unwind generator if there is one, this actually runs
                    # any processing
                    results = list(results)
                if return_type == 'item-or-list' and \
                        len(results) < 2:
                    return results[0] if results else None
                else:
                    return results
            lgr.log(2, "Returning return_func from eval_func for %s", wrapped_class)
            return return_func(generator_func)(*args, **kwargs)

    ret = eval_func(func)
    ret._eval_results = True
    return ret


def default_result_renderer(res):
    if res.get('status', None) != 'notneeded':
        path = res.get('path', None)
        if path and res.get('refds'):
            try:
                path = relpath(path, res['refds'])
            except ValueError:
                # can happen, e.g., on windows with paths from different
                # drives. just go with the original path in this case
                pass
        ui.message('{action}({status}):{path}{type}{msg}'.format(
            action=ac.color_word(
                res.get('action', '<action-unspecified>'),
                ac.BOLD),
            status=ac.color_status(res.get('status', '<status-unspecified>')),
            path=' {}'.format(path) if path else '',
            type=' ({})'.format(
                ac.color_word(res['type'], ac.MAGENTA)
            ) if 'type' in res else '',
            msg=' [{}]'.format(
                res['message'][0] % res['message'][1:]
                if isinstance(res['message'], tuple) else res[
                    'message'])
            if res.get('message', None) else ''))


def _display_suppressed_message(nsimilar, ndisplayed, last_ts, final=False):
    # +1 because there was the original result + nsimilar displayed.
    n_suppressed = nsimilar - ndisplayed + 1
    if n_suppressed > 0:
        ts = time()
        # rate-limit update of suppression message, with a large number
        # of fast-paced results updating for each one can result in more
        # CPU load than the actual processing
        # arbitrarily go for a 2Hz update frequency -- it "feels" good
        if last_ts is None or final or (ts - last_ts > 0.5):
            ui.message('  [{} similar {} been suppressed; disable with datalad.ui.suppress-similar-results=off]'
                       .format(n_suppressed,
                               single_or_plural("message has",
                                                "messages have",
                                                n_suppressed, False)),
                       cr="\n" if final else "\r")
            return ts
    return last_ts


def _process_results(
        results,
        cmd_class,
        on_failure,
        action_summary,
        incomplete_results,
        result_renderer,
        result_log_level,
        allkwargs):
    # private helper pf @eval_results
    # loop over results generated from some source and handle each
    # of them according to the requested behavior (logging, rendering, ...)

    # used to track repeated messages in the default renderer
    last_result = None
    last_result_ts = None
    # which result dict keys to inspect for changes to discover repetitions
    # of similar messages
    repetition_keys = set(('action', 'status', 'type', 'refds'))
    # counter for detected repetitions
    result_repetitions = 0
    # how many repetitions to show, before suppression kicks in
    render_n_repetitions = \
        dlcfg.obtain('datalad.ui.suppress-similar-results-threshold') \
        if sys.stdout.isatty() \
        and dlcfg.obtain('datalad.ui.suppress-similar-results') \
        else float("inf")

    for res in results:
        if not res or 'action' not in res:
            # XXX Yarik has to no clue on how to track the origin of the
            # record to figure out WTF, so he just skips it
            # but MIH thinks leaving a trace of that would be good
            lgr.debug('Drop result record without "action": %s', res)
            continue

        actsum = action_summary.get(res['action'], {})
        if res['status']:
            actsum[res['status']] = actsum.get(res['status'], 0) + 1
            action_summary[res['action']] = actsum
        ## log message, if there is one and a logger was given
        msg = res.get('message', None)
        # remove logger instance from results, as it is no longer useful
        # after logging was done, it isn't serializable, and generally
        # pollutes the output
        res_lgr = res.pop('logger', None)
        if msg and res_lgr:
            if isinstance(res_lgr, logging.Logger):
                # didn't get a particular log function, go with default
                res_lgr = getattr(
                    res_lgr,
                    default_logchannels[res['status']]
                    if result_log_level is None
                    else result_log_level)
            msg = res['message']
            msgargs = None
            if isinstance(msg, tuple):
                msgargs = msg[1:]
                msg = msg[0]
            if 'path' in res:
                # result path could be a path instance
                path = str(res['path'])
                if msgargs:
                    # we will pass the msg for %-polation, so % should be doubled
                    path = path.replace('%', '%%')
                msg = '{} [{}({})]'.format(
                    msg, res['action'], path)
            if msgargs:
                # support string expansion of logging to avoid runtime cost
                try:
                    res_lgr(msg, *msgargs)
                except TypeError as exc:
                    raise TypeError(
                        "Failed to render %r with %r from %r: %s"
                        % (msg, msgargs, res, exc_str(exc))
                    )
            else:
                res_lgr(msg)

        ## output rendering
        # TODO RF this in a simple callable that gets passed into this function
        if result_renderer is None or result_renderer == 'disabled':
            pass
        elif result_renderer == 'default':
            trimmed_result = {k: v for k, v in res.items() if k in repetition_keys}
            if res.get('status', None) != 'notneeded' \
                    and trimmed_result == last_result:
                # this is a similar report, suppress if too many, but count it
                result_repetitions += 1
                if result_repetitions < render_n_repetitions:
                    default_result_renderer(res)
                else:
                    last_result_ts = _display_suppressed_message(
                        result_repetitions, render_n_repetitions, last_result_ts)
            else:
                # this one is new, first report on any prev. suppressed results
                # by number, and then render this fresh one
                last_result_ts = _display_suppressed_message(
                    result_repetitions, render_n_repetitions, last_result_ts,
                    final=True)
                default_result_renderer(res)
                result_repetitions = 0
            last_result = trimmed_result
        elif result_renderer in ('json', 'json_pp'):
            ui.message(json.dumps(
                {k: v for k, v in res.items()
                 if k not in ('logger')},
                sort_keys=True,
                indent=2 if result_renderer.endswith('_pp') else None,
                default=str))
        elif result_renderer in ('tailored', 'default'):
            if hasattr(cmd_class, 'custom_result_renderer'):
                cmd_class.custom_result_renderer(res, **allkwargs)
        elif hasattr(result_renderer, '__call__'):
            try:
                result_renderer(res, **allkwargs)
            except Exception as e:
                lgr.warning('Result rendering failed for: %s [%s]',
                            res, exc_str(e))
        else:
            raise ValueError('unknown result renderer "{}"'.format(result_renderer))

        ## error handling
        # looks for error status, and report at the end via
        # an exception
        if on_failure in ('continue', 'stop') \
                and res['status'] in ('impossible', 'error'):
            incomplete_results.append(res)
            if on_failure == 'stop':
                # first fail -> that's it
                # raise will happen after the loop
                break
        yield res
    # make sure to report on any issues that we had suppressed
    _display_suppressed_message(
        result_repetitions, render_n_repetitions, last_result_ts, final=True)


def keep_result(res, rfilter, **kwargs):
    if not rfilter:
        return True
    try:
        if not rfilter(res, **kwargs):
            # give the slightest indication which filter was employed
            raise ValueError(
                'excluded by filter {} with arguments {}'.format(rfilter, kwargs))
    except ValueError as e:
        # make sure to report the excluded result to massively improve
        # debugging experience
        lgr.debug('Not reporting result (%s): %s', exc_str(e), res)
        return False
    return True


def xfm_result(res, xfm):
    if not xfm:
        return res

    return xfm(res)

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user