{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\"\"\"\nfrom datalad.ui.utils import get_console_width\n\n__docformat__ = 'restructuredtext'\n\nimport logging\nlgr = logging.getLogger('datalad.cmdline')\n\nlgr.log(5, \"Importing cmdline.main\")\n\nimport argparse\nfrom collections import defaultdict\nimport sys\nimport textwrap\nimport os\n\nfrom six import text_type\n\nimport datalad\n\nfrom datalad.cmdline import helpers\nfrom datalad.support.exceptions import InsufficientArgumentsError\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.exceptions import CommandError\nfrom .helpers import strip_arg_from_argv\nfrom ..utils import setup_exceptionhook, chpwd\nfrom ..utils import assure_unicode\nfrom ..utils import on_msys_tainted_paths\nfrom ..dochelpers import exc_str\n\n\ndef _license_info():\n    return \"\"\"\\\nCopyright (c) 2013-2018 DataLad developers\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n\"\"\"\n\n\nclass ArgumentParserDisableAbbrev(argparse.ArgumentParser):\n    # Don't accept abbreviations for long options. With py3.5 and above, we\n    # could just use allow_abbrev=False.\n    #\n    # Modified from the solution posted at\n    # https://bugs.python.org/issue14910#msg204678\n    def _get_option_tuples(self, option_string):\n        chars = self.prefix_chars\n        if option_string[0] in chars and option_string[1] in chars:\n            # option_string is a long flag. Disable abbreviation.\n            return []\n        return super(ArgumentParserDisableAbbrev, self)._get_option_tuples(\n            option_string)\n\n\n# TODO:  OPT look into making setup_parser smarter to become faster\n# Now it seems to take up to 200ms to do all the parser setup\n# even though it might not be necessary to know about all the commands etc.\n# I wondered if it could somehow decide on what commands to worry about etc\n# by going through sys.args first\ndef setup_parser(\n        cmdlineargs,\n        formatter_class=argparse.RawDescriptionHelpFormatter,\n        return_subparsers=False):\n    lgr.log(5, \"Starting to setup_parser\")\n    # delay since it can be a heavy import\n    from ..interface.base import dedent_docstring, get_interface_groups, \\\n        get_cmdline_command_name, alter_interface_docs_for_cmdline, \\\n        load_interface, get_cmd_doc\n    # setup cmdline args parser\n    parts = {}\n    # main parser\n    parser = ArgumentParserDisableAbbrev(\n        fromfile_prefix_chars=None,\n        # usage=\"%(prog)s ...\",\n        description=dedent_docstring(\"\"\"\\\n            Comprehensive data management solution\n\n            DataLad provides a unified data distribution system built on the Git\n            and Git-annex. DataLad command line tools allow to manipulate (obtain,\n            create, update, publish, etc.) datasets and provide a comprehensive\n            toolbox for joint management of data and code. Compared to Git/annex\n            it primarly extends their functionality to transparently and\n            simultaneously work with multiple inter-related repositories.\"\"\"),\n        epilog='\"Be happy!\"',\n        formatter_class=formatter_class,\n        add_help=False)\n    # common options\n    helpers.parser_add_common_opt(parser, 'log_level')\n    helpers.parser_add_common_opt(parser, 'pbs_runner')\n    helpers.parser_add_common_opt(parser, 'change_path')\n    helpers.parser_add_common_opt(\n        parser,\n        'version',\n        version='datalad %s\\n' % datalad.__version__)\n    if __debug__:\n        parser.add_argument(\n            '--dbg', action='store_true', dest='common_debug',\n            help=\"enter Python debugger when uncaught exception happens\")\n        parser.add_argument(\n            '--idbg', action='store_true', dest='common_idebug',\n            help=\"enter IPython debugger when uncaught exception happens\")\n    parser.add_argument(\n        '-c', action='append', dest='cfg_overrides', metavar='KEY=VALUE',\n        help=\"\"\"configuration variable setting. Overrides any configuration\n        read from a file, but is potentially overridden itself by configuration\n        variables in the process environment.\"\"\")\n    parser.add_argument(\n        '-f', '--output-format', dest='common_output_format',\n        default='default',\n        type=assure_unicode,\n        metavar=\"{default,json,json_pp,tailored,'<template>'}\",\n        help=\"\"\"select format for returned command results. 'default' give one line\n        per result reporting action, status, path and an optional message;\n        'json' renders a JSON object with all properties for each result (one per\n        line); 'json_pp' pretty-prints JSON spanning multiple lines; 'tailored'\n        enables a command-specific rendering style that is typically\n        tailored to human consumption (no result output otherwise),\n        '<template>' reports any value(s) of any result properties in any format\n        indicated by the template (e.g. '{path}'; compare with JSON\n        output for all key-value choices). The template syntax follows the Python\n        \"format() language\". It is possible to report individual\n        dictionary values, e.g. '{metadata[name]}'. If a 2nd-level key contains\n        a colon, e.g. 'music:Genre', ':' must be substituted by '#' in the template,\n        like so: '{metadata[music#Genre]}'.\"\"\")\n    parser.add_argument(\n        '--report-status', dest='common_report_status',\n        choices=['success', 'failure', 'ok', 'notneeded', 'impossible', 'error'],\n        help=\"\"\"constrain command result report to records matching the given\n        status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands\n        for 'impossible' OR 'error'.\"\"\")\n    parser.add_argument(\n        '--report-type', dest='common_report_type',\n        choices=['dataset', 'file'],\n        action='append',\n        help=\"\"\"constrain command result report to records matching the given\n        type. Can be given more than once to match multiple types.\"\"\")\n    parser.add_argument(\n        '--on-failure', dest='common_on_failure',\n        choices=['ignore', 'continue', 'stop'],\n        # no default: better be configure per-command\n        help=\"\"\"when an operation fails: 'ignore' and continue with remaining\n        operations, the error is logged but does not lead to a non-zero exit code\n        of the command; 'continue' works like 'ignore', but an error causes a\n        non-zero exit code; 'stop' halts on first failure and yields non-zero exit\n        code. A failure is any result with status 'impossible' or 'error'.\"\"\")\n    parser.add_argument(\n        '--proc-pre', dest='common_proc_pre',\n        nargs='+',\n        action='append',\n        metavar=('<PROCEDURE NAME>', 'ARGS'),\n        help=\"\"\"Dataset procedure to run before the main command (see run-procedure\n        command for details). This option can be given more than once to run\n        multiple procedures in the order in which they were given.\n        It is important to specify the target dataset via the --dataset argument\n        of the main command.\"\"\"),\n    parser.add_argument(\n        '--proc-post', dest='common_proc_post',\n        nargs='+',\n        action='append',\n        metavar=('<PROCEDURE NAME>', 'ARGS'),\n        help=\"\"\"Like --proc-pre, but procedures are executed after the main command\n        has finished.\"\"\"),\n    parser.add_argument(\n        '--cmd', dest='_', action='store_true',\n        help=\"\"\"syntactical helper that can be used to end the list of global\n        command line options before the subcommand label. Options like\n        --proc-pre can take an arbitrary number of arguments and may require\n        to be followed by a single --cmd in order to enable identification\n        of the subcommand.\"\"\")\n\n    # yoh: atm we only dump to console.  Might adopt the same separation later on\n    #      and for consistency will call it --verbose-level as well for now\n    # log-level is set via common_opts ATM\n    # parser.add_argument('--log-level',\n    #                     choices=('critical', 'error', 'warning', 'info', 'debug'),\n    #                     dest='common_log_level',\n    #                     help=\"\"\"level of verbosity in log files. By default\n    #                          everything, including debug messages is logged.\"\"\")\n    #parser.add_argument('-l', '--verbose-level',\n    #                    choices=('critical', 'error', 'warning', 'info', 'debug'),\n    #                    dest='common_verbose_level',\n    #                    help=\"\"\"level of verbosity of console output. By default\n    #                         only warnings and errors are printed.\"\"\")\n\n    # Before doing anything additional and possibly expensive see may be that\n    # we have got the command already\n    need_single_subparser = False if return_subparsers else None\n    fail_handler = (lambda *a, **kw: True) \\\n        if return_subparsers else fail_with_short_help\n    try:\n        parsed_args, unparsed_args = parser._parse_known_args(\n            cmdlineargs[1:], argparse.Namespace())\n        if not unparsed_args:\n            fail_handler(parser, msg=\"too few arguments\", exit_code=2)\n        lgr.debug(\"Command line args 1st pass. Parsed: %s Unparsed: %s\",\n                  parsed_args, unparsed_args)\n    except Exception as exc:\n        lgr.debug(\"Early parsing failed with %s\", exc_str(exc))\n        need_single_subparser = False\n        unparsed_args = cmdlineargs[1:]  # referenced before assignment otherwise\n\n    interface_groups = get_interface_groups(include_plugins=True)\n\n    # First unparsed could be either unknown option to top level \"datalad\"\n    # or a command. Among unknown could be --help/--help-np which would\n    # need to be dealt with\n    unparsed_arg = unparsed_args[0] if unparsed_args else None\n    if need_single_subparser is not None \\\n            or unparsed_arg in ('--help', '--help-np', '-h'):\n        need_single_subparser = False\n        add_entrypoints_to_interface_groups(interface_groups)\n    elif unparsed_arg.startswith('-'):  # unknown option\n        fail_with_short_help(parser,\n                             msg=\"unrecognized argument %s\" % unparsed_arg,\n                             exit_code=2)\n        # if we could get a list of options known to parser,\n        # we could suggest them\n        # known=get_all_options(parser), provided=unparsed_arg)\n    else:  # the command to handle\n        known_commands = get_commands_from_groups(interface_groups)\n        if unparsed_arg not in known_commands:\n            # need to load all the extensions and try again\n            add_entrypoints_to_interface_groups(interface_groups)\n            known_commands = get_commands_from_groups(interface_groups)\n\n        if unparsed_arg not in known_commands:\n            # check if might be coming from known extensions\n            from ..interface import _known_extension_commands\n            extension_commands = {\n                c: e\n                for e, commands in _known_extension_commands.items()\n                for c in commands\n            }\n            hint = None\n            if unparsed_arg in extension_commands:\n                hint = \"Command %s is provided by (not installed) extension %s.\" \\\n                      % (unparsed_arg, extension_commands[unparsed_arg])\n            fail_with_short_help(\n                parser,\n                hint=hint,\n                provided=unparsed_arg,\n                known=list(known_commands.keys()) + list(extension_commands.keys())\n            )\n        if need_single_subparser is None:\n            need_single_subparser = unparsed_arg\n\n    # --help specification was delayed since it causes immediate printout of\n    # --help output before we setup --help for each command\n    helpers.parser_add_common_opt(parser, 'help')\n\n    grp_short_descriptions = defaultdict(list)\n    # create subparser, use module suffix as cmd name\n    subparsers = parser.add_subparsers()\n    for group_name, _, _interfaces \\\n            in sorted(interface_groups, key=lambda x: x[1]):\n        for _intfspec in _interfaces:\n            cmd_name = get_cmdline_command_name(_intfspec)\n            if need_single_subparser and cmd_name != need_single_subparser:\n                continue\n            _intf = load_interface(_intfspec)\n            if _intf is None:\n                # TODO(yoh):  add doc why we could skip this one... makes this\n                # loop harder to extract into a dedicated function\n                continue\n            # deal with optional parser args\n            if hasattr(_intf, 'parser_args'):\n                parser_args = _intf.parser_args\n            else:\n                parser_args = dict(formatter_class=formatter_class)\n                # use class description, if no explicit description is available\n                intf_doc = get_cmd_doc(_intf)\n                parser_args['description'] = alter_interface_docs_for_cmdline(\n                    intf_doc)\n            subparser = subparsers.add_parser(cmd_name, add_help=False, **parser_args)\n            # our own custom help for all commands\n            helpers.parser_add_common_opt(subparser, 'help')\n            # let module configure the parser\n            _intf.setup_parser(subparser)\n            # logger for command\n\n            # configure 'run' function for this command\n            plumbing_args = dict(\n                func=_intf.call_from_parser,\n                logger=logging.getLogger(_intf.__module__),\n                subparser=subparser)\n            if hasattr(_intf, 'result_renderer_cmdline'):\n                plumbing_args['result_renderer'] = _intf.result_renderer_cmdline\n            subparser.set_defaults(**plumbing_args)\n            parts[cmd_name] = subparser\n            # store short description for later\n            sdescr = getattr(_intf, 'short_description',\n                             parser_args['description'].split('\\n')[0])\n            grp_short_descriptions[group_name].append((cmd_name, sdescr))\n\n    # create command summary\n    if '--help' in cmdlineargs or '--help-np' in cmdlineargs:\n        parser.description = get_description_with_cmd_summary(\n            grp_short_descriptions,\n            interface_groups,\n            parser.description)\n\n    parts['datalad'] = parser\n    lgr.log(5, \"Finished setup_parser\")\n    if return_subparsers:\n        return parts\n    else:\n        return parser\n\n\ndef fail_with_short_help(parser=None,\n                         msg=None,\n                         known=None, provided=None,\n                         hint=None,\n                         exit_code=1,\n                         what=\"command\",\n                         out=None):\n    \"\"\"Generic helper to fail\n    with short help possibly hinting on what was intended if `known`\n    were provided\n    \"\"\"\n    out = out or sys.stderr\n    if msg:\n        out.write(\"error: %s\\n\" % msg)\n    if not known:\n        if parser:\n            # just to appear in print_usage also consistent with --help output\n            parser.add_argument(\"command [command-opts]\")\n            parser.print_usage(file=out)\n    else:\n        out.write(\n            \"datalad: Unknown %s %r.  See 'datalad --help'.\\n\\n\"\n            % (what, provided,))\n        if provided not in known:\n            import difflib\n            suggestions = difflib.get_close_matches(provided, known)\n            if suggestions:\n                out.write(\n                    \"Did you mean one of these?\\n        %s\\n\"\n                    % \"\\n        \".join(suggestions))\n        # Too noisy\n        # sys.stderr.write(\" All known %ss: %s\\n\"\n        #                   % (what, \", \".join(sorted(known))))\n    if hint:\n        out.write(\"Hint: %s\\n\" % hint)\n    raise SystemExit(exit_code)\n\n\ndef get_description_with_cmd_summary(grp_short_descriptions, interface_groups,\n                                     parser_description):\n    from ..interface.base import dedent_docstring\n    from ..interface.base import get_cmd_summaries\n    lgr.debug(\"Generating detailed description for the parser\")\n\n    console_width = get_console_width()\n    cmd_summary = get_cmd_summaries(grp_short_descriptions, interface_groups,\n                                    width=console_width)\n    # we need one last formal section to not have the trailed be\n    # confused with the last command group\n    cmd_summary.append('\\n*General information*\\n')\n    detailed_description = '%s\\n%s\\n\\n%s' \\\n                           % (parser_description,\n                              '\\n'.join(cmd_summary),\n                              textwrap.fill(dedent_docstring(\"\"\"\\\n    Detailed usage information for individual commands is\n    available via command-specific --help, i.e.:\n    datalad <command> --help\"\"\"),\n                                            console_width - 5,\n                                            initial_indent='',\n                                            subsequent_indent=''))\n    return detailed_description\n\n\ndef get_commands_from_groups(groups):\n    \"\"\"Get a dictionary of command: interface_spec\"\"\"\n    from ..interface.base import get_cmdline_command_name\n    return {\n        get_cmdline_command_name(_intfspec): _intfspec\n        for _, _, _interfaces in groups\n        for _intfspec in _interfaces\n    }\n\n\ndef add_entrypoints_to_interface_groups(interface_groups):\n    lgr.debug(\"Loading entrypoints\")\n    from pkg_resources import iter_entry_points  # delay expensive import\n    for ep in iter_entry_points('datalad.extensions'):\n        lgr.debug(\n            'Loading entrypoint %s from datalad.extensions for docs building',\n            ep.name)\n        try:\n            spec = ep.load()\n            interface_groups.append((ep.name, spec[0], spec[1]))\n            lgr.debug('Loaded entrypoint %s', ep.name)\n        except Exception as e:\n            lgr.warning('Failed to load entrypoint %s: %s', ep.name, exc_str(e))\n            continue\n\ndef _fix_datalad_ri(s):\n    \"\"\"Fixup argument if it was a DataLadRI and had leading / removed\n\n    See gh-2643\n    \"\"\"\n    if s.startswith('//') and (len(s) == 2 or (len(s) > 2 and s[2] != '/')):\n        lgr.info(\n            \"Changing %s back to /%s as it was probably changed by MINGW/MSYS, \"\n            \"see http://www.mingw.org/wiki/Posix_path_conversion\", s, s)\n        return \"/\" + s\n    return s\n\n\ndef main(args=None):\n    lgr.log(5, \"Starting main(%r)\", args)\n    args = args or sys.argv\n    if on_msys_tainted_paths:\n        # Possibly present DataLadRIs were stripped of a leading /\n        args = [_fix_datalad_ri(s) for s in args]\n    # PYTHON_ARGCOMPLETE_OK\n    parser = setup_parser(args)\n    try:\n        import argcomplete\n        argcomplete.autocomplete(parser)\n    except ImportError:\n        pass\n\n    # parse cmd args\n    lgr.debug(\"Parsing known args among %s\", repr(args))\n    cmdlineargs, unparsed_args = parser.parse_known_args(args[1:])\n    has_func = hasattr(cmdlineargs, 'func') and cmdlineargs.func is not None\n    if unparsed_args:\n        if has_func and cmdlineargs.func.__self__.__name__ != 'Export':\n            lgr.error('unknown argument{}: {}'.format(\n                's' if len(unparsed_args) > 1 else '',\n                unparsed_args if len(unparsed_args) > 1 else unparsed_args[0]))\n            cmdlineargs.subparser.print_usage()\n            sys.exit(1)\n        else:\n            # store all unparsed arguments\n            cmdlineargs.datalad_unparsed_args = unparsed_args\n\n    # to possibly be passed into PBS scheduled call\n    args_ = args or sys.argv\n\n    if cmdlineargs.cfg_overrides is not None:\n        overrides = dict([\n            (o.split('=')[0], '='.join(o.split('=')[1:]))\n            for o in cmdlineargs.cfg_overrides])\n        datalad.cfg.overrides.update(overrides)\n\n    # enable overrides\n    datalad.cfg.reload(force=True)\n\n    if cmdlineargs.change_path is not None:\n        from .common_args import change_path as change_path_opt\n        for path in cmdlineargs.change_path:\n            chpwd(path)\n            args_ = strip_arg_from_argv(args_, path, change_path_opt[1])\n\n    ret = None\n    if cmdlineargs.pbs_runner:\n        from .helpers import run_via_pbs\n        from .common_args import pbs_runner as pbs_runner_opt\n        args_ = strip_arg_from_argv(args_, cmdlineargs.pbs_runner, pbs_runner_opt[1])\n        # run the function associated with the selected command\n        run_via_pbs(args_, cmdlineargs.pbs_runner)\n    elif has_func:\n        if cmdlineargs.common_debug or cmdlineargs.common_idebug:\n            # so we could see/stop clearly at the point of failure\n            setup_exceptionhook(ipython=cmdlineargs.common_idebug)\n            from datalad.interface.base import Interface\n            Interface._interrupted_exit_code = None\n            ret = cmdlineargs.func(cmdlineargs)\n        else:\n            # otherwise - guard and only log the summary. Postmortem is not\n            # as convenient if being caught in this ultimate except\n            try:\n                ret = cmdlineargs.func(cmdlineargs)\n            except InsufficientArgumentsError as exc:\n                # if the func reports inappropriate usage, give help output\n                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))\n                cmdlineargs.subparser.print_usage(sys.stderr)\n                sys.exit(2)\n            except IncompleteResultsError as exc:\n                # rendering for almost all commands now happens 'online'\n                # hence we are no longer attempting to render the actual\n                # results in an IncompleteResultsError, ubt rather trust that\n                # this happened before\n\n                # in general we do not want to see the error again, but\n                # present in debug output\n                lgr.debug('could not perform all requested actions: %s',\n                          exc_str(exc))\n                sys.exit(1)\n            except CommandError as exc:\n                # behave as if the command ran directly, importantly pass\n                # exit code as is\n                exc_msg = str(exc)\n                if exc_msg:\n                    msg = exc_msg.encode() if isinstance(exc_msg, text_type) else exc_msg\n                    os.write(2, msg + b\"\\n\")\n                if exc.stdout:\n                    os.write(1, exc.stdout.encode() if isinstance(exc.stdout, text_type) else exc.stdout)\n                if exc.stderr:\n                    os.write(2, exc.stderr.encode() if isinstance(exc.stderr, text_type) else exc.stderr)\n                # We must not exit with 0 code if any exception got here but\n                # had no code defined\n                sys.exit(exc.code if exc.code is not None else 1)\n            except Exception as exc:\n                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))\n                sys.exit(1)\n    else:\n        # just let argparser spit out its error, since there is smth wrong\n        parser.parse_args(args)\n        # if that one didn't puke -- we should\n        parser.print_usage()\n        lgr.error(\"Please specify the command\")\n        sys.exit(2)\n\n    try:\n        if hasattr(cmdlineargs, 'result_renderer'):\n            cmdlineargs.result_renderer(ret, cmdlineargs)\n    except Exception as exc:\n        lgr.error(\"Failed to render results due to %s\", exc_str(exc))\n        sys.exit(1)\n\nlgr.log(5, \"Done importing cmdline.main\")\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\"\"\"\nfrom datalad.ui.utils import get_console_width\n\n__docformat__ = 'restructuredtext'\n\nimport logging\nlgr = logging.getLogger('datalad.cmdline')\n\nlgr.log(5, \"Importing cmdline.main\")\n\nimport argparse\nfrom collections import defaultdict\nimport sys\nimport textwrap\nimport os\n\nfrom six import text_type\n\nimport datalad\n\nfrom datalad.cmdline import helpers\nfrom datalad.support.exceptions import InsufficientArgumentsError\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.exceptions import CommandError\nfrom .helpers import strip_arg_from_argv\nfrom ..utils import (\n    assure_unicode,\n    chpwd,\n    get_suggestions_msg,\n    on_msys_tainted_paths,\n    setup_exceptionhook,\n)\nfrom ..dochelpers import exc_str\n\n\ndef _license_info():\n    return \"\"\"\\\nCopyright (c) 2013-2018 DataLad developers\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n\"\"\"\n\n\nclass ArgumentParserDisableAbbrev(argparse.ArgumentParser):\n    # Don't accept abbreviations for long options. With py3.5 and above, we\n    # could just use allow_abbrev=False.\n    #\n    # Modified from the solution posted at\n    # https://bugs.python.org/issue14910#msg204678\n    def _get_option_tuples(self, option_string):\n        chars = self.prefix_chars\n        if option_string[0] in chars and option_string[1] in chars:\n            # option_string is a long flag. Disable abbreviation.\n            return []\n        return super(ArgumentParserDisableAbbrev, self)._get_option_tuples(\n            option_string)\n\n\n# TODO:  OPT look into making setup_parser smarter to become faster\n# Now it seems to take up to 200ms to do all the parser setup\n# even though it might not be necessary to know about all the commands etc.\n# I wondered if it could somehow decide on what commands to worry about etc\n# by going through sys.args first\ndef setup_parser(\n        cmdlineargs,\n        formatter_class=argparse.RawDescriptionHelpFormatter,\n        return_subparsers=False):\n    lgr.log(5, \"Starting to setup_parser\")\n    # delay since it can be a heavy import\n    from ..interface.base import dedent_docstring, get_interface_groups, \\\n        get_cmdline_command_name, alter_interface_docs_for_cmdline, \\\n        load_interface, get_cmd_doc\n    # setup cmdline args parser\n    parts = {}\n    # main parser\n    parser = ArgumentParserDisableAbbrev(\n        fromfile_prefix_chars=None,\n        # usage=\"%(prog)s ...\",\n        description=dedent_docstring(\"\"\"\\\n            Comprehensive data management solution\n\n            DataLad provides a unified data distribution system built on the Git\n            and Git-annex. DataLad command line tools allow to manipulate (obtain,\n            create, update, publish, etc.) datasets and provide a comprehensive\n            toolbox for joint management of data and code. Compared to Git/annex\n            it primarly extends their functionality to transparently and\n            simultaneously work with multiple inter-related repositories.\"\"\"),\n        epilog='\"Be happy!\"',\n        formatter_class=formatter_class,\n        add_help=False)\n    # common options\n    helpers.parser_add_common_opt(parser, 'log_level')\n    helpers.parser_add_common_opt(parser, 'pbs_runner')\n    helpers.parser_add_common_opt(parser, 'change_path')\n    helpers.parser_add_common_opt(\n        parser,\n        'version',\n        version='datalad %s\\n' % datalad.__version__)\n    if __debug__:\n        parser.add_argument(\n            '--dbg', action='store_true', dest='common_debug',\n            help=\"enter Python debugger when uncaught exception happens\")\n        parser.add_argument(\n            '--idbg', action='store_true', dest='common_idebug',\n            help=\"enter IPython debugger when uncaught exception happens\")\n    parser.add_argument(\n        '-c', action='append', dest='cfg_overrides', metavar='KEY=VALUE',\n        help=\"\"\"configuration variable setting. Overrides any configuration\n        read from a file, but is potentially overridden itself by configuration\n        variables in the process environment.\"\"\")\n    parser.add_argument(\n        '-f', '--output-format', dest='common_output_format',\n        default='default',\n        type=assure_unicode,\n        metavar=\"{default,json,json_pp,tailored,'<template>'}\",\n        help=\"\"\"select format for returned command results. 'default' give one line\n        per result reporting action, status, path and an optional message;\n        'json' renders a JSON object with all properties for each result (one per\n        line); 'json_pp' pretty-prints JSON spanning multiple lines; 'tailored'\n        enables a command-specific rendering style that is typically\n        tailored to human consumption (no result output otherwise),\n        '<template>' reports any value(s) of any result properties in any format\n        indicated by the template (e.g. '{path}'; compare with JSON\n        output for all key-value choices). The template syntax follows the Python\n        \"format() language\". It is possible to report individual\n        dictionary values, e.g. '{metadata[name]}'. If a 2nd-level key contains\n        a colon, e.g. 'music:Genre', ':' must be substituted by '#' in the template,\n        like so: '{metadata[music#Genre]}'.\"\"\")\n    parser.add_argument(\n        '--report-status', dest='common_report_status',\n        choices=['success', 'failure', 'ok', 'notneeded', 'impossible', 'error'],\n        help=\"\"\"constrain command result report to records matching the given\n        status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands\n        for 'impossible' OR 'error'.\"\"\")\n    parser.add_argument(\n        '--report-type', dest='common_report_type',\n        choices=['dataset', 'file'],\n        action='append',\n        help=\"\"\"constrain command result report to records matching the given\n        type. Can be given more than once to match multiple types.\"\"\")\n    parser.add_argument(\n        '--on-failure', dest='common_on_failure',\n        choices=['ignore', 'continue', 'stop'],\n        # no default: better be configure per-command\n        help=\"\"\"when an operation fails: 'ignore' and continue with remaining\n        operations, the error is logged but does not lead to a non-zero exit code\n        of the command; 'continue' works like 'ignore', but an error causes a\n        non-zero exit code; 'stop' halts on first failure and yields non-zero exit\n        code. A failure is any result with status 'impossible' or 'error'.\"\"\")\n    parser.add_argument(\n        '--proc-pre', dest='common_proc_pre',\n        nargs='+',\n        action='append',\n        metavar=('<PROCEDURE NAME>', 'ARGS'),\n        help=\"\"\"Dataset procedure to run before the main command (see run-procedure\n        command for details). This option can be given more than once to run\n        multiple procedures in the order in which they were given.\n        It is important to specify the target dataset via the --dataset argument\n        of the main command.\"\"\"),\n    parser.add_argument(\n        '--proc-post', dest='common_proc_post',\n        nargs='+',\n        action='append',\n        metavar=('<PROCEDURE NAME>', 'ARGS'),\n        help=\"\"\"Like --proc-pre, but procedures are executed after the main command\n        has finished.\"\"\"),\n    parser.add_argument(\n        '--cmd', dest='_', action='store_true',\n        help=\"\"\"syntactical helper that can be used to end the list of global\n        command line options before the subcommand label. Options like\n        --proc-pre can take an arbitrary number of arguments and may require\n        to be followed by a single --cmd in order to enable identification\n        of the subcommand.\"\"\")\n\n    # yoh: atm we only dump to console.  Might adopt the same separation later on\n    #      and for consistency will call it --verbose-level as well for now\n    # log-level is set via common_opts ATM\n    # parser.add_argument('--log-level',\n    #                     choices=('critical', 'error', 'warning', 'info', 'debug'),\n    #                     dest='common_log_level',\n    #                     help=\"\"\"level of verbosity in log files. By default\n    #                          everything, including debug messages is logged.\"\"\")\n    #parser.add_argument('-l', '--verbose-level',\n    #                    choices=('critical', 'error', 'warning', 'info', 'debug'),\n    #                    dest='common_verbose_level',\n    #                    help=\"\"\"level of verbosity of console output. By default\n    #                         only warnings and errors are printed.\"\"\")\n\n    # Before doing anything additional and possibly expensive see may be that\n    # we have got the command already\n    need_single_subparser = False if return_subparsers else None\n    fail_handler = (lambda *a, **kw: True) \\\n        if return_subparsers else fail_with_short_help\n    try:\n        parsed_args, unparsed_args = parser._parse_known_args(\n            cmdlineargs[1:], argparse.Namespace())\n        if not unparsed_args:\n            fail_handler(parser, msg=\"too few arguments\", exit_code=2)\n        lgr.debug(\"Command line args 1st pass. Parsed: %s Unparsed: %s\",\n                  parsed_args, unparsed_args)\n    except Exception as exc:\n        lgr.debug(\"Early parsing failed with %s\", exc_str(exc))\n        need_single_subparser = False\n        unparsed_args = cmdlineargs[1:]  # referenced before assignment otherwise\n\n    interface_groups = get_interface_groups(include_plugins=True)\n\n    # First unparsed could be either unknown option to top level \"datalad\"\n    # or a command. Among unknown could be --help/--help-np which would\n    # need to be dealt with\n    unparsed_arg = unparsed_args[0] if unparsed_args else None\n    if need_single_subparser is not None \\\n            or unparsed_arg in ('--help', '--help-np', '-h'):\n        need_single_subparser = False\n        add_entrypoints_to_interface_groups(interface_groups)\n    elif unparsed_arg.startswith('-'):  # unknown option\n        fail_with_short_help(parser,\n                             msg=\"unrecognized argument %s\" % unparsed_arg,\n                             exit_code=2)\n        # if we could get a list of options known to parser,\n        # we could suggest them\n        # known=get_all_options(parser), provided=unparsed_arg)\n    else:  # the command to handle\n        known_commands = get_commands_from_groups(interface_groups)\n        if unparsed_arg not in known_commands:\n            # need to load all the extensions and try again\n            add_entrypoints_to_interface_groups(interface_groups)\n            known_commands = get_commands_from_groups(interface_groups)\n\n        if unparsed_arg not in known_commands:\n            # check if might be coming from known extensions\n            from ..interface import _known_extension_commands\n            extension_commands = {\n                c: e\n                for e, commands in _known_extension_commands.items()\n                for c in commands\n            }\n            hint = None\n            if unparsed_arg in extension_commands:\n                hint = \"Command %s is provided by (not installed) extension %s.\" \\\n                      % (unparsed_arg, extension_commands[unparsed_arg])\n            fail_with_short_help(\n                parser,\n                hint=hint,\n                provided=unparsed_arg,\n                known=list(known_commands.keys()) + list(extension_commands.keys())\n            )\n        if need_single_subparser is None:\n            need_single_subparser = unparsed_arg\n\n    # --help specification was delayed since it causes immediate printout of\n    # --help output before we setup --help for each command\n    helpers.parser_add_common_opt(parser, 'help')\n\n    grp_short_descriptions = defaultdict(list)\n    # create subparser, use module suffix as cmd name\n    subparsers = parser.add_subparsers()\n    for group_name, _, _interfaces \\\n            in sorted(interface_groups, key=lambda x: x[1]):\n        for _intfspec in _interfaces:\n            cmd_name = get_cmdline_command_name(_intfspec)\n            if need_single_subparser and cmd_name != need_single_subparser:\n                continue\n            _intf = load_interface(_intfspec)\n            if _intf is None:\n                # TODO(yoh):  add doc why we could skip this one... makes this\n                # loop harder to extract into a dedicated function\n                continue\n            # deal with optional parser args\n            if hasattr(_intf, 'parser_args'):\n                parser_args = _intf.parser_args\n            else:\n                parser_args = dict(formatter_class=formatter_class)\n                # use class description, if no explicit description is available\n                intf_doc = get_cmd_doc(_intf)\n                parser_args['description'] = alter_interface_docs_for_cmdline(\n                    intf_doc)\n            subparser = subparsers.add_parser(cmd_name, add_help=False, **parser_args)\n            # our own custom help for all commands\n            helpers.parser_add_common_opt(subparser, 'help')\n            # let module configure the parser\n            _intf.setup_parser(subparser)\n            # logger for command\n\n            # configure 'run' function for this command\n            plumbing_args = dict(\n                func=_intf.call_from_parser,\n                logger=logging.getLogger(_intf.__module__),\n                subparser=subparser)\n            if hasattr(_intf, 'result_renderer_cmdline'):\n                plumbing_args['result_renderer'] = _intf.result_renderer_cmdline\n            subparser.set_defaults(**plumbing_args)\n            parts[cmd_name] = subparser\n            # store short description for later\n            sdescr = getattr(_intf, 'short_description',\n                             parser_args['description'].split('\\n')[0])\n            grp_short_descriptions[group_name].append((cmd_name, sdescr))\n\n    # create command summary\n    if '--help' in cmdlineargs or '--help-np' in cmdlineargs:\n        parser.description = get_description_with_cmd_summary(\n            grp_short_descriptions,\n            interface_groups,\n            parser.description)\n\n    parts['datalad'] = parser\n    lgr.log(5, \"Finished setup_parser\")\n    if return_subparsers:\n        return parts\n    else:\n        return parser\n\n\ndef fail_with_short_help(parser=None,\n                         msg=None,\n                         known=None, provided=None,\n                         hint=None,\n                         exit_code=1,\n                         what=\"command\",\n                         out=None):\n    \"\"\"Generic helper to fail\n    with short help possibly hinting on what was intended if `known`\n    were provided\n    \"\"\"\n    out = out or sys.stderr\n    if msg:\n        out.write(\"error: %s\\n\" % msg)\n    if not known:\n        if parser:\n            # just to appear in print_usage also consistent with --help output\n            parser.add_argument(\"command [command-opts]\")\n            parser.print_usage(file=out)\n    else:\n        out.write(\n            \"datalad: Unknown %s %r.  See 'datalad --help'.\\n\\n\"\n            % (what, provided,))\n        if provided not in known:\n            out.write(get_suggestions_msg(provided, known))\n        # Too noisy\n        # sys.stderr.write(\" All known %ss: %s\\n\"\n        #                   % (what, \", \".join(sorted(known))))\n    if hint:\n        out.write(\"Hint: %s\\n\" % hint)\n    raise SystemExit(exit_code)\n\n\ndef get_description_with_cmd_summary(grp_short_descriptions, interface_groups,\n                                     parser_description):\n    from ..interface.base import dedent_docstring\n    from ..interface.base import get_cmd_summaries\n    lgr.debug(\"Generating detailed description for the parser\")\n\n    console_width = get_console_width()\n    cmd_summary = get_cmd_summaries(grp_short_descriptions, interface_groups,\n                                    width=console_width)\n    # we need one last formal section to not have the trailed be\n    # confused with the last command group\n    cmd_summary.append('\\n*General information*\\n')\n    detailed_description = '%s\\n%s\\n\\n%s' \\\n                           % (parser_description,\n                              '\\n'.join(cmd_summary),\n                              textwrap.fill(dedent_docstring(\"\"\"\\\n    Detailed usage information for individual commands is\n    available via command-specific --help, i.e.:\n    datalad <command> --help\"\"\"),\n                                            console_width - 5,\n                                            initial_indent='',\n                                            subsequent_indent=''))\n    return detailed_description\n\n\ndef get_commands_from_groups(groups):\n    \"\"\"Get a dictionary of command: interface_spec\"\"\"\n    from ..interface.base import get_cmdline_command_name\n    return {\n        get_cmdline_command_name(_intfspec): _intfspec\n        for _, _, _interfaces in groups\n        for _intfspec in _interfaces\n    }\n\n\ndef add_entrypoints_to_interface_groups(interface_groups):\n    lgr.debug(\"Loading entrypoints\")\n    from pkg_resources import iter_entry_points  # delay expensive import\n    for ep in iter_entry_points('datalad.extensions'):\n        lgr.debug(\n            'Loading entrypoint %s from datalad.extensions for docs building',\n            ep.name)\n        try:\n            spec = ep.load()\n            interface_groups.append((ep.name, spec[0], spec[1]))\n            lgr.debug('Loaded entrypoint %s', ep.name)\n        except Exception as e:\n            lgr.warning('Failed to load entrypoint %s: %s', ep.name, exc_str(e))\n            continue\n\ndef _fix_datalad_ri(s):\n    \"\"\"Fixup argument if it was a DataLadRI and had leading / removed\n\n    See gh-2643\n    \"\"\"\n    if s.startswith('//') and (len(s) == 2 or (len(s) > 2 and s[2] != '/')):\n        lgr.info(\n            \"Changing %s back to /%s as it was probably changed by MINGW/MSYS, \"\n            \"see http://www.mingw.org/wiki/Posix_path_conversion\", s, s)\n        return \"/\" + s\n    return s\n\n\ndef main(args=None):\n    lgr.log(5, \"Starting main(%r)\", args)\n    args = args or sys.argv\n    if on_msys_tainted_paths:\n        # Possibly present DataLadRIs were stripped of a leading /\n        args = [_fix_datalad_ri(s) for s in args]\n    # PYTHON_ARGCOMPLETE_OK\n    parser = setup_parser(args)\n    try:\n        import argcomplete\n        argcomplete.autocomplete(parser)\n    except ImportError:\n        pass\n\n    # parse cmd args\n    lgr.debug(\"Parsing known args among %s\", repr(args))\n    cmdlineargs, unparsed_args = parser.parse_known_args(args[1:])\n    has_func = hasattr(cmdlineargs, 'func') and cmdlineargs.func is not None\n    if unparsed_args:\n        if has_func and cmdlineargs.func.__self__.__name__ != 'Export':\n            lgr.error('unknown argument{}: {}'.format(\n                's' if len(unparsed_args) > 1 else '',\n                unparsed_args if len(unparsed_args) > 1 else unparsed_args[0]))\n            cmdlineargs.subparser.print_usage()\n            sys.exit(1)\n        else:\n            # store all unparsed arguments\n            cmdlineargs.datalad_unparsed_args = unparsed_args\n\n    # to possibly be passed into PBS scheduled call\n    args_ = args or sys.argv\n\n    if cmdlineargs.cfg_overrides is not None:\n        overrides = dict([\n            (o.split('=')[0], '='.join(o.split('=')[1:]))\n            for o in cmdlineargs.cfg_overrides])\n        datalad.cfg.overrides.update(overrides)\n\n    # enable overrides\n    datalad.cfg.reload(force=True)\n\n    if cmdlineargs.change_path is not None:\n        from .common_args import change_path as change_path_opt\n        for path in cmdlineargs.change_path:\n            chpwd(path)\n            args_ = strip_arg_from_argv(args_, path, change_path_opt[1])\n\n    ret = None\n    if cmdlineargs.pbs_runner:\n        from .helpers import run_via_pbs\n        from .common_args import pbs_runner as pbs_runner_opt\n        args_ = strip_arg_from_argv(args_, cmdlineargs.pbs_runner, pbs_runner_opt[1])\n        # run the function associated with the selected command\n        run_via_pbs(args_, cmdlineargs.pbs_runner)\n    elif has_func:\n        if cmdlineargs.common_debug or cmdlineargs.common_idebug:\n            # so we could see/stop clearly at the point of failure\n            setup_exceptionhook(ipython=cmdlineargs.common_idebug)\n            from datalad.interface.base import Interface\n            Interface._interrupted_exit_code = None\n            ret = cmdlineargs.func(cmdlineargs)\n        else:\n            # otherwise - guard and only log the summary. Postmortem is not\n            # as convenient if being caught in this ultimate except\n            try:\n                ret = cmdlineargs.func(cmdlineargs)\n            except InsufficientArgumentsError as exc:\n                # if the func reports inappropriate usage, give help output\n                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))\n                cmdlineargs.subparser.print_usage(sys.stderr)\n                sys.exit(2)\n            except IncompleteResultsError as exc:\n                # rendering for almost all commands now happens 'online'\n                # hence we are no longer attempting to render the actual\n                # results in an IncompleteResultsError, ubt rather trust that\n                # this happened before\n\n                # in general we do not want to see the error again, but\n                # present in debug output\n                lgr.debug('could not perform all requested actions: %s',\n                          exc_str(exc))\n                sys.exit(1)\n            except CommandError as exc:\n                # behave as if the command ran directly, importantly pass\n                # exit code as is\n                exc_msg = str(exc)\n                if exc_msg:\n                    msg = exc_msg.encode() if isinstance(exc_msg, text_type) else exc_msg\n                    os.write(2, msg + b\"\\n\")\n                if exc.stdout:\n                    os.write(1, exc.stdout.encode() if isinstance(exc.stdout, text_type) else exc.stdout)\n                if exc.stderr:\n                    os.write(2, exc.stderr.encode() if isinstance(exc.stderr, text_type) else exc.stderr)\n                # We must not exit with 0 code if any exception got here but\n                # had no code defined\n                sys.exit(exc.code if exc.code is not None else 1)\n            except Exception as exc:\n                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))\n                sys.exit(1)\n    else:\n        # just let argparser spit out its error, since there is smth wrong\n        parser.parse_args(args)\n        # if that one didn't puke -- we should\n        parser.print_usage()\n        lgr.error(\"Please specify the command\")\n        sys.exit(2)\n\n    try:\n        if hasattr(cmdlineargs, 'result_renderer'):\n            cmdlineargs.result_renderer(ret, cmdlineargs)\n    except Exception as exc:\n        lgr.error(\"Failed to render results due to %s\", exc_str(exc))\n        sys.exit(1)\n\nlgr.log(5, \"Done importing cmdline.main\")\n",
            "header": {
                "file": {
                    "path": "datalad/cmdline/main.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "c7d1c2a43",
                "new_commit_hash": "b995d4eaf",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/cmdline/main.py"
            },
            "plus_file": {
                "path": "b/datalad/cmdline/main.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 31,
                            "length": 9
                        },
                        "new_range": {
                            "start": 31,
                            "length": 13
                        },
                        "section": "from datalad.support.exceptions import InsufficientArgumentsError"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "from datalad.support.exceptions import IncompleteResultsError",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.support.exceptions import CommandError",
                                "type": "context"
                            },
                            {
                                "content": "from .helpers import strip_arg_from_argv",
                                "type": "context"
                            },
                            {
                                "content": "from ..utils import setup_exceptionhook, chpwd",
                                "type": "deleted"
                            },
                            {
                                "content": "from ..utils import assure_unicode",
                                "type": "deleted"
                            },
                            {
                                "content": "from ..utils import on_msys_tainted_paths",
                                "type": "deleted"
                            },
                            {
                                "content": "from ..utils import (",
                                "type": "added"
                            },
                            {
                                "content": "    assure_unicode,",
                                "type": "added"
                            },
                            {
                                "content": "    chpwd,",
                                "type": "added"
                            },
                            {
                                "content": "    get_suggestions_msg,",
                                "type": "added"
                            },
                            {
                                "content": "    on_msys_tainted_paths,",
                                "type": "added"
                            },
                            {
                                "content": "    setup_exceptionhook,",
                                "type": "added"
                            },
                            {
                                "content": ")",
                                "type": "added"
                            },
                            {
                                "content": "from ..dochelpers import exc_str",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/cmdline/main.py",
                            "type": "statement",
                            "name": "",
                            "content": "from ..utils import (\n    assure_unicode,\n    chpwd,\n    get_suggestions_msg,\n    on_msys_tainted_paths,\n    setup_exceptionhook,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 34,
                            "end_lineno": 40,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/cmdline/main.py",
                            "type": "statement",
                            "name": "",
                            "content": "from ..utils import assure_unicode",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 35,
                            "end_lineno": 35,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/cmdline/main.py",
                            "type": "statement",
                            "name": "",
                            "content": "from ..utils import setup_exceptionhook, chpwd",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 34,
                            "end_lineno": 34,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/cmdline/main.py",
                            "type": "statement",
                            "name": "",
                            "content": "from ..utils import on_msys_tainted_paths",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 36,
                            "end_lineno": 36,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 355,
                            "length": 12
                        },
                        "new_range": {
                            "start": 359,
                            "length": 7
                        },
                        "section": "def fail_with_short_help(parser=None,"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            \"datalad: Unknown %s %r.  See 'datalad --help'.\\n\\n\"",
                                "type": "context"
                            },
                            {
                                "content": "            % (what, provided,))",
                                "type": "context"
                            },
                            {
                                "content": "        if provided not in known:",
                                "type": "context"
                            },
                            {
                                "content": "            import difflib",
                                "type": "deleted"
                            },
                            {
                                "content": "            suggestions = difflib.get_close_matches(provided, known)",
                                "type": "deleted"
                            },
                            {
                                "content": "            if suggestions:",
                                "type": "deleted"
                            },
                            {
                                "content": "                out.write(",
                                "type": "deleted"
                            },
                            {
                                "content": "                    \"Did you mean one of these?\\n        %s\\n\"",
                                "type": "deleted"
                            },
                            {
                                "content": "                    % \"\\n        \".join(suggestions))",
                                "type": "deleted"
                            },
                            {
                                "content": "            out.write(get_suggestions_msg(provided, known))",
                                "type": "added"
                            },
                            {
                                "content": "        # Too noisy",
                                "type": "context"
                            },
                            {
                                "content": "        # sys.stderr.write(\" All known %ss: %s\\n\"",
                                "type": "context"
                            },
                            {
                                "content": "        #                   % (what, \", \".join(sorted(known))))",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cmdline/main.py",
                            "type": "function",
                            "name": "fail_with_short_help",
                            "content": "def fail_with_short_help(parser=None,\n                         msg=None,\n                         known=None, provided=None,\n                         hint=None,\n                         exit_code=1,\n                         what=\"command\",\n                         out=None):\n    \"\"\"Generic helper to fail\n    with short help possibly hinting on what was intended if `known`\n    were provided\n    \"\"\"\n    out = out or sys.stderr\n    if msg:\n        out.write(\"error: %s\\n\" % msg)\n    if not known:\n        if parser:\n            # just to appear in print_usage also consistent with --help output\n            parser.add_argument(\"command [command-opts]\")\n            parser.print_usage(file=out)\n    else:\n        out.write(\n            \"datalad: Unknown %s %r.  See 'datalad --help'.\\n\\n\"\n            % (what, provided,))\n        if provided not in known:\n            import difflib\n            suggestions = difflib.get_close_matches(provided, known)\n            if suggestions:\n                out.write(\n                    \"Did you mean one of these?\\n        %s\\n\"\n                    % \"\\n        \".join(suggestions))\n        # Too noisy\n        # sys.stderr.write(\" All known %ss: %s\\n\"\n        #                   % (what, \", \".join(sorted(known))))\n    if hint:\n        out.write(\"Hint: %s\\n\" % hint)\n    raise SystemExit(exit_code)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 334,
                            "end_lineno": 369,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test functioning of the datalad main cmdline utility \"\"\"\n\nfrom datalad.tests.utils import known_failure_direct_mode\nfrom datalad.tests.utils import on_windows\n\n\nimport re\nimport sys\nfrom six.moves import StringIO\nfrom mock import patch\n\nimport datalad\nfrom ..main import (\n    main,\n    fail_with_short_help,\n    _fix_datalad_ri,\n)\nfrom datalad import __version__\nfrom datalad.cmd import Runner\nfrom datalad.ui.utils import get_console_width\nfrom datalad.api import create\nfrom datalad.utils import chpwd\nfrom datalad.tests.utils import with_tempfile\nfrom datalad.tests.utils import assert_equal, assert_raises, in_, ok_startswith\nfrom datalad.tests.utils import assert_in\nfrom datalad.tests.utils import assert_re_in\nfrom datalad.tests.utils import assert_not_in\nfrom datalad.tests.utils import slow\n\n\ndef run_main(args, exit_code=0, expect_stderr=False):\n    \"\"\"Run main() of the datalad, do basic checks and provide outputs\n\n    Parameters\n    ----------\n    args : list\n        List of string cmdline arguments to pass\n    exit_code : int\n        Expected exit code. Would raise AssertionError if differs\n    expect_stderr : bool or string\n        Whether to expect stderr output. If string -- match\n\n    Returns\n    -------\n    stdout, stderr  strings\n       Output produced\n    \"\"\"\n    with patch('sys.stderr', new_callable=StringIO) as cmerr:\n        with patch('sys.stdout', new_callable=StringIO) as cmout:\n            with assert_raises(SystemExit) as cm:\n                main([\"datalad\"] + list(args))\n            assert_equal(cm.exception.code, exit_code)\n            stdout = cmout.getvalue()\n            stderr = cmerr.getvalue()\n            if expect_stderr is False:\n                assert_equal(stderr, \"\")\n            elif expect_stderr is True:\n                # do nothing -- just return\n                pass\n            else:\n                # must be a string\n                assert_equal(stderr, expect_stderr)\n    return stdout, stderr\n\n\n# TODO: switch to stdout for --version output\ndef test_version():\n    stdout, stderr = run_main(['--version'], expect_stderr=True)\n\n    # and output should contain our version, copyright, license\n\n    # https://hg.python.org/cpython/file/default/Doc/whatsnew/3.4.rst#l1952\n    out = stdout if sys.version_info >= (3, 4) else stderr\n    ok_startswith(out, 'datalad %s\\n' % datalad.__version__)\n    # since https://github.com/datalad/datalad/pull/2733 no license in --version\n    assert_not_in(\"Copyright\", out)\n    assert_not_in(\"Permission is hereby granted\", out)\n\n\ndef test_help_np():\n    stdout, stderr = run_main(['--help-np'])\n\n    # Let's extract section titles:\n    # enough of bin/datalad and .tox/py27/bin/datalad -- guarantee consistency! ;)\n    ok_startswith(stdout, 'Usage: datalad')\n    # Sections start/end with * if ran under DATALAD_HELP2MAN mode\n    sections = [l[1:-1] for l in filter(re.compile('^\\*.*\\*$').match, stdout.split('\\n'))]\n    # but order is still not guaranteed (dict somewhere)! TODO\n    # see https://travis-ci.org/datalad/datalad/jobs/80519004\n    # thus testing sets\n    for s in {'Commands for dataset operations',\n              'Commands for metadata handling',\n              'Miscellaneous commands',\n              'General information',\n              'Global options',\n              'Plumbing commands',\n              'Plugins'}:\n        assert_in(s, sections)\n\n    # none of the lines must be longer than 80 chars\n    # TODO: decide on   create-sibling and possibly\n    # rewrite-urls\n    accepted_width = get_console_width()\n\n    long_lines = [\"%d %s\" % (len(l), l) for l in stdout.split('\\n')\n                  if len(l) > accepted_width and\n                  '{' not in l  # on nd70 summary line is unsplit\n                  ]\n    if long_lines:\n        raise AssertionError(\n            \"Following lines in --help output were longer than %s chars:\\n%s\"\n            % (accepted_width, '\\n'.join(long_lines))\n        )\n\n\ndef test_usage_on_insufficient_args():\n    stdout, stderr = run_main(['install'], exit_code=2, expect_stderr=True)\n    ok_startswith(stderr, 'usage:')\n\n\ndef test_subcmd_usage_on_unknown_args():\n    stdout, stderr = run_main(['get', '--murks'], exit_code=1, expect_stderr=True)\n    in_('get', stdout)\n\n\ndef test_combined_short_option():\n    stdout, stderr = run_main(['-fjson'], exit_code=2, expect_stderr=True)\n    assert_not_in(\"unrecognized argument\", stderr)\n    assert_in(\"too few arguments\", stderr)\n\n\ndef check_incorrect_option(opts, err_str):\n    # The first line used to be:\n    # stdout, stderr = run_main((sys.argv[0],) + opts, expect_stderr=True, exit_code=2)\n    # But: what do we expect to be in sys.argv[0] here?\n    # It depends on how we invoke the test.\n    # - nosetests -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0}=='nosetests'\n    # - python -m nose -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0}=='python -m nose'\n    # - python -c \"import nose; nose.main()\" -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0]=='-c'\n    # This led to failure in case sys.argv[0] contained an option, that was\n    # defined to be a datalad option too, therefore was a 'known_arg' and was\n    # checked to meet its constraints.\n    # But sys.argv[0] actually isn't used by main at all. It simply doesn't\n    # matter what's in there. The only thing important to pass here is `opts`.\n    stdout, stderr = run_main(opts, expect_stderr=True, exit_code=2)\n    out = stdout + stderr\n    assert_in(\"usage: \", out)\n    assert_re_in(err_str, out, match=False)\n\n\ndef test_incorrect_options():\n    # apparently a bit different if following a good one so let's do both\n    err_invalid = \"error: (invalid|too few arguments|unrecognized argument)\"\n    yield check_incorrect_option, ('--buga',), err_invalid\n    yield check_incorrect_option, ('--dbg', '--buga'), err_invalid\n\n    err_insufficient = err_invalid  # \"specify\"\n    yield check_incorrect_option, ('--dbg',), err_insufficient\n    yield check_incorrect_option, tuple(), err_insufficient\n\n\ndef test_script_shims():\n    runner = Runner()\n    for script in [\n        'datalad',\n        'git-annex-remote-datalad-archives',\n        'git-annex-remote-datalad']:\n        if not on_windows:\n            # those must be available for execution, and should not contain\n            which, _ = runner(['which', script])\n            # test if there is no easy install shim in there\n            with open(which.rstrip()) as f:\n                content = f.read()\n        else:\n            from distutils.spawn import find_executable\n            content = find_executable(script)\n        assert_not_in('EASY', content) # NOTHING easy should be there\n        assert_not_in('pkg_resources', content)\n\n        # and let's check that it is our script\n        out, err = runner([script, '--version'])\n        version = (out + err).splitlines()[0].split(' ', 1)[1]\n        # we can get git and non git .dev version... so for now\n        # relax\n        get_numeric_portion = lambda v: [x for x in v.split('.') if x.isdigit()]\n        # extract numeric portion\n        assert get_numeric_portion(version) # that my lambda is correctish\n        assert_equal(get_numeric_portion(__version__),\n                     get_numeric_portion(version))\n\n\n@slow  # 11.2591s\n@with_tempfile(mkdir=True)\ndef test_cfg_override(path):\n    with chpwd(path):\n        # control\n        out, err = Runner()('datalad wtf -s some', shell=True)\n        assert_not_in('datalad.dummy: this', out)\n        # ensure that this is not a dataset's cfg manager\n        assert_not_in('datalad.dataset.id', out)\n        # env var\n        if on_windows:\n            cmd_str = 'set DATALAD_DUMMY=this&& datalad wtf -s some'\n        else:\n            cmd_str = 'DATALAD_DUMMY=this datalad wtf -s some'\n        out, err = Runner()(cmd_str, shell=True)\n        assert_in('datalad.dummy: this', out)\n        # cmdline arg\n        out, err = Runner()('datalad -c datalad.dummy=this wtf -s some', shell=True)\n        assert_in('datalad.dummy: this', out)\n\n        # now create a dataset in the path. the wtf plugin will switch to\n        # using the dataset's config manager, which must inherit the overrides\n        create(dataset=path)\n        # control\n        out, err = Runner()('datalad wtf -s some', shell=True)\n        assert_not_in('datalad.dummy: this', out)\n        # ensure that this is a dataset's cfg manager\n        assert_in('datalad.dataset.id', out)\n        # env var\n        if on_windows:\n            cmd_str = 'set DATALAD_DUMMY=this&& datalad wtf -s some'\n        else:\n            cmd_str = 'DATALAD_DUMMY=this datalad wtf -s some'\n        out, err = Runner()(cmd_str, shell=True)\n        assert_in('datalad.dummy: this', out)\n        # cmdline arg\n        out, err = Runner()('datalad -c datalad.dummy=this wtf -s some', shell=True)\n        assert_in('datalad.dummy: this', out)\n\n\ndef test_fail_with_short_help():\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(exit_code=3, out=out)\n    assert_equal(cme.exception.code, 3)\n    assert_equal(out.getvalue(), \"\")\n\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(msg=\"Failed badly\", out=out)\n    assert_equal(cme.exception.code, 1)\n    assert_equal(out.getvalue(), \"error: Failed badly\\n\")\n\n    # Suggestions, hint, etc\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(\n            msg=\"Failed badly\",\n            known=[\"mother\", \"mutter\", \"father\", \"son\"],\n            provided=\"muther\",\n            hint=\"You can become one\",\n            exit_code=0,  # noone forbids\n            what=\"parent\",\n            out=out)\n    assert_equal(cme.exception.code, 0)\n    assert_equal(out.getvalue(),\n                 \"error: Failed badly\\n\"\n                 \"datalad: Unknown parent 'muther'.  See 'datalad --help'.\\n\\n\"\n                 \"Did you mean one of these?\\n\"\n                 \"        mutter\\n\"\n                 \"        mother\\n\"\n                 \"        father\\n\"\n                 \"Hint: You can become one\\n\")\n\ndef test_fix_datalad_ri():\n    assert_equal(_fix_datalad_ri('/'), '/')\n    assert_equal(_fix_datalad_ri('/a/b'), '/a/b')\n    assert_equal(_fix_datalad_ri('//'), '///')\n    assert_equal(_fix_datalad_ri('///'), '///')\n    assert_equal(_fix_datalad_ri('//a'), '///a')\n    assert_equal(_fix_datalad_ri('///a'), '///a')\n    assert_equal(_fix_datalad_ri('//a/b'), '///a/b')\n    assert_equal(_fix_datalad_ri('///a/b'), '///a/b')",
            "new_file_content": "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test functioning of the datalad main cmdline utility \"\"\"\n\nfrom datalad.tests.utils import known_failure_direct_mode\nfrom datalad.tests.utils import on_windows\n\n\nimport re\nimport sys\nfrom six.moves import StringIO\nfrom mock import patch\n\nimport datalad\nfrom ..main import (\n    main,\n    fail_with_short_help,\n    _fix_datalad_ri,\n)\nfrom datalad import __version__\nfrom datalad.cmd import Runner\nfrom datalad.ui.utils import get_console_width\nfrom datalad.api import create\nfrom datalad.utils import chpwd\nfrom datalad.tests.utils import with_tempfile\nfrom datalad.tests.utils import assert_equal, assert_raises, in_, ok_startswith\nfrom datalad.tests.utils import assert_in\nfrom datalad.tests.utils import assert_re_in\nfrom datalad.tests.utils import assert_not_in\nfrom datalad.tests.utils import slow\n\n\ndef run_main(args, exit_code=0, expect_stderr=False):\n    \"\"\"Run main() of the datalad, do basic checks and provide outputs\n\n    Parameters\n    ----------\n    args : list\n        List of string cmdline arguments to pass\n    exit_code : int\n        Expected exit code. Would raise AssertionError if differs\n    expect_stderr : bool or string\n        Whether to expect stderr output. If string -- match\n\n    Returns\n    -------\n    stdout, stderr  strings\n       Output produced\n    \"\"\"\n    with patch('sys.stderr', new_callable=StringIO) as cmerr:\n        with patch('sys.stdout', new_callable=StringIO) as cmout:\n            with assert_raises(SystemExit) as cm:\n                main([\"datalad\"] + list(args))\n            assert_equal(cm.exception.code, exit_code)\n            stdout = cmout.getvalue()\n            stderr = cmerr.getvalue()\n            if expect_stderr is False:\n                assert_equal(stderr, \"\")\n            elif expect_stderr is True:\n                # do nothing -- just return\n                pass\n            else:\n                # must be a string\n                assert_equal(stderr, expect_stderr)\n    return stdout, stderr\n\n\n# TODO: switch to stdout for --version output\ndef test_version():\n    stdout, stderr = run_main(['--version'], expect_stderr=True)\n\n    # and output should contain our version, copyright, license\n\n    # https://hg.python.org/cpython/file/default/Doc/whatsnew/3.4.rst#l1952\n    out = stdout if sys.version_info >= (3, 4) else stderr\n    ok_startswith(out, 'datalad %s\\n' % datalad.__version__)\n    # since https://github.com/datalad/datalad/pull/2733 no license in --version\n    assert_not_in(\"Copyright\", out)\n    assert_not_in(\"Permission is hereby granted\", out)\n\n\ndef test_help_np():\n    stdout, stderr = run_main(['--help-np'])\n\n    # Let's extract section titles:\n    # enough of bin/datalad and .tox/py27/bin/datalad -- guarantee consistency! ;)\n    ok_startswith(stdout, 'Usage: datalad')\n    # Sections start/end with * if ran under DATALAD_HELP2MAN mode\n    sections = [l[1:-1] for l in filter(re.compile('^\\*.*\\*$').match, stdout.split('\\n'))]\n    # but order is still not guaranteed (dict somewhere)! TODO\n    # see https://travis-ci.org/datalad/datalad/jobs/80519004\n    # thus testing sets\n    for s in {'Commands for dataset operations',\n              'Commands for metadata handling',\n              'Miscellaneous commands',\n              'General information',\n              'Global options',\n              'Plumbing commands',\n              'Plugins'}:\n        assert_in(s, sections)\n\n    # none of the lines must be longer than 80 chars\n    # TODO: decide on   create-sibling and possibly\n    # rewrite-urls\n    accepted_width = get_console_width()\n\n    long_lines = [\"%d %s\" % (len(l), l) for l in stdout.split('\\n')\n                  if len(l) > accepted_width and\n                  '{' not in l  # on nd70 summary line is unsplit\n                  ]\n    if long_lines:\n        raise AssertionError(\n            \"Following lines in --help output were longer than %s chars:\\n%s\"\n            % (accepted_width, '\\n'.join(long_lines))\n        )\n\n\ndef test_usage_on_insufficient_args():\n    stdout, stderr = run_main(['install'], exit_code=2, expect_stderr=True)\n    ok_startswith(stderr, 'usage:')\n\n\ndef test_subcmd_usage_on_unknown_args():\n    stdout, stderr = run_main(['get', '--murks'], exit_code=1, expect_stderr=True)\n    in_('get', stdout)\n\n\ndef test_combined_short_option():\n    stdout, stderr = run_main(['-fjson'], exit_code=2, expect_stderr=True)\n    assert_not_in(\"unrecognized argument\", stderr)\n    assert_in(\"too few arguments\", stderr)\n\n\ndef check_incorrect_option(opts, err_str):\n    # The first line used to be:\n    # stdout, stderr = run_main((sys.argv[0],) + opts, expect_stderr=True, exit_code=2)\n    # But: what do we expect to be in sys.argv[0] here?\n    # It depends on how we invoke the test.\n    # - nosetests -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0}=='nosetests'\n    # - python -m nose -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0}=='python -m nose'\n    # - python -c \"import nose; nose.main()\" -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0]=='-c'\n    # This led to failure in case sys.argv[0] contained an option, that was\n    # defined to be a datalad option too, therefore was a 'known_arg' and was\n    # checked to meet its constraints.\n    # But sys.argv[0] actually isn't used by main at all. It simply doesn't\n    # matter what's in there. The only thing important to pass here is `opts`.\n    stdout, stderr = run_main(opts, expect_stderr=True, exit_code=2)\n    out = stdout + stderr\n    assert_in(\"usage: \", out)\n    assert_re_in(err_str, out, match=False)\n\n\ndef test_incorrect_options():\n    # apparently a bit different if following a good one so let's do both\n    err_invalid = \"error: (invalid|too few arguments|unrecognized argument)\"\n    yield check_incorrect_option, ('--buga',), err_invalid\n    yield check_incorrect_option, ('--dbg', '--buga'), err_invalid\n\n    err_insufficient = err_invalid  # \"specify\"\n    yield check_incorrect_option, ('--dbg',), err_insufficient\n    yield check_incorrect_option, tuple(), err_insufficient\n\n\ndef test_script_shims():\n    runner = Runner()\n    for script in [\n        'datalad',\n        'git-annex-remote-datalad-archives',\n        'git-annex-remote-datalad']:\n        if not on_windows:\n            # those must be available for execution, and should not contain\n            which, _ = runner(['which', script])\n            # test if there is no easy install shim in there\n            with open(which.rstrip()) as f:\n                content = f.read()\n        else:\n            from distutils.spawn import find_executable\n            content = find_executable(script)\n        assert_not_in('EASY', content) # NOTHING easy should be there\n        assert_not_in('pkg_resources', content)\n\n        # and let's check that it is our script\n        out, err = runner([script, '--version'])\n        version = (out + err).splitlines()[0].split(' ', 1)[1]\n        # we can get git and non git .dev version... so for now\n        # relax\n        get_numeric_portion = lambda v: [x for x in v.split('.') if x.isdigit()]\n        # extract numeric portion\n        assert get_numeric_portion(version) # that my lambda is correctish\n        assert_equal(get_numeric_portion(__version__),\n                     get_numeric_portion(version))\n\n\n@slow  # 11.2591s\n@with_tempfile(mkdir=True)\ndef test_cfg_override(path):\n    with chpwd(path):\n        # control\n        out, err = Runner()('datalad wtf -s some', shell=True)\n        assert_not_in('datalad.dummy: this', out)\n        # ensure that this is not a dataset's cfg manager\n        assert_not_in('datalad.dataset.id', out)\n        # env var\n        if on_windows:\n            cmd_str = 'set DATALAD_DUMMY=this&& datalad wtf -s some'\n        else:\n            cmd_str = 'DATALAD_DUMMY=this datalad wtf -s some'\n        out, err = Runner()(cmd_str, shell=True)\n        assert_in('datalad.dummy: this', out)\n        # cmdline arg\n        out, err = Runner()('datalad -c datalad.dummy=this wtf -s some', shell=True)\n        assert_in('datalad.dummy: this', out)\n\n        # now create a dataset in the path. the wtf plugin will switch to\n        # using the dataset's config manager, which must inherit the overrides\n        create(dataset=path)\n        # control\n        out, err = Runner()('datalad wtf -s some', shell=True)\n        assert_not_in('datalad.dummy: this', out)\n        # ensure that this is a dataset's cfg manager\n        assert_in('datalad.dataset.id', out)\n        # env var\n        if on_windows:\n            cmd_str = 'set DATALAD_DUMMY=this&& datalad wtf -s some'\n        else:\n            cmd_str = 'DATALAD_DUMMY=this datalad wtf -s some'\n        out, err = Runner()(cmd_str, shell=True)\n        assert_in('datalad.dummy: this', out)\n        # cmdline arg\n        out, err = Runner()('datalad -c datalad.dummy=this wtf -s some', shell=True)\n        assert_in('datalad.dummy: this', out)\n\n\ndef test_fail_with_short_help():\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(exit_code=3, out=out)\n    assert_equal(cme.exception.code, 3)\n    assert_equal(out.getvalue(), \"\")\n\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(msg=\"Failed badly\", out=out)\n    assert_equal(cme.exception.code, 1)\n    assert_equal(out.getvalue(), \"error: Failed badly\\n\")\n\n    # Suggestions, hint, etc\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(\n            msg=\"Failed badly\",\n            known=[\"mother\", \"mutter\", \"father\", \"son\"],\n            provided=\"muther\",\n            hint=\"You can become one\",\n            exit_code=0,  # noone forbids\n            what=\"parent\",\n            out=out)\n    assert_equal(cme.exception.code, 0)\n    assert_equal(out.getvalue(),\n                 \"error: Failed badly\\n\"\n                 \"datalad: Unknown parent 'muther'.  See 'datalad --help'.\\n\\n\"\n                 \"Did you mean any of these?\\n\"\n                 \"        mutter\\n\"\n                 \"        mother\\n\"\n                 \"        father\\n\"\n                 \"Hint: You can become one\\n\")\n\ndef test_fix_datalad_ri():\n    assert_equal(_fix_datalad_ri('/'), '/')\n    assert_equal(_fix_datalad_ri('/a/b'), '/a/b')\n    assert_equal(_fix_datalad_ri('//'), '///')\n    assert_equal(_fix_datalad_ri('///'), '///')\n    assert_equal(_fix_datalad_ri('//a'), '///a')\n    assert_equal(_fix_datalad_ri('///a'), '///a')\n    assert_equal(_fix_datalad_ri('//a/b'), '///a/b')\n    assert_equal(_fix_datalad_ri('///a/b'), '///a/b')",
            "header": {
                "file": {
                    "path": "datalad/cmdline/tests/test_main.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "bde1abb26",
                "new_commit_hash": "211780bcb",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/cmdline/tests/test_main.py"
            },
            "plus_file": {
                "path": "b/datalad/cmdline/tests/test_main.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 268,
                            "length": 7
                        },
                        "new_range": {
                            "start": 268,
                            "length": 7
                        },
                        "section": "def test_fail_with_short_help():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_equal(out.getvalue(),",
                                "type": "context"
                            },
                            {
                                "content": "                 \"error: Failed badly\\n\"",
                                "type": "context"
                            },
                            {
                                "content": "                 \"datalad: Unknown parent 'muther'.  See 'datalad --help'.\\n\\n\"",
                                "type": "context"
                            },
                            {
                                "content": "                 \"Did you mean one of these?\\n\"",
                                "type": "deleted"
                            },
                            {
                                "content": "                 \"Did you mean any of these?\\n\"",
                                "type": "added"
                            },
                            {
                                "content": "                 \"        mutter\\n\"",
                                "type": "context"
                            },
                            {
                                "content": "                 \"        mother\\n\"",
                                "type": "context"
                            },
                            {
                                "content": "                 \"        father\\n\"",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cmdline/tests/test_main.py",
                            "type": "function",
                            "name": "test_fail_with_short_help",
                            "content": "def test_fail_with_short_help():\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(exit_code=3, out=out)\n    assert_equal(cme.exception.code, 3)\n    assert_equal(out.getvalue(), \"\")\n\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(msg=\"Failed badly\", out=out)\n    assert_equal(cme.exception.code, 1)\n    assert_equal(out.getvalue(), \"error: Failed badly\\n\")\n\n    # Suggestions, hint, etc\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(\n            msg=\"Failed badly\",\n            known=[\"mother\", \"mutter\", \"father\", \"son\"],\n            provided=\"muther\",\n            hint=\"You can become one\",\n            exit_code=0,  # noone forbids\n            what=\"parent\",\n            out=out)\n    assert_equal(cme.exception.code, 0)\n    assert_equal(out.getvalue(),\n                 \"error: Failed badly\\n\"\n                 \"datalad: Unknown parent 'muther'.  See 'datalad --help'.\\n\\n\"\n                 \"Did you mean one of these?\\n\"\n                 \"        mutter\\n\"\n                 \"        mother\\n\"\n                 \"        father\\n\"\n                 \"Hint: You can become one\\n\")",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 243,
                            "end_lineno": 275,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\nimport collections\nimport hashlib\nimport re\nimport six.moves.builtins as __builtin__\nimport time\n\nimport logging\nimport shutil\nimport os\nimport sys\nimport tempfile\nimport platform\nimport gc\nimport glob\nimport gzip\nimport string\nimport wrapt\n\nfrom copy import copy as shallow_copy\nfrom contextlib import contextmanager\nfrom functools import wraps\nfrom time import sleep\nfrom inspect import getargspec\nfrom itertools import tee\n\nimport os.path as op\nfrom os.path import sep as dirsep\nfrom os.path import commonprefix\nfrom os.path import curdir, basename, exists, realpath, islink, join as opj\nfrom os.path import isabs, normpath, expandvars, expanduser, abspath, sep\nfrom os.path import isdir\nfrom os.path import relpath\nfrom os.path import stat\nfrom os.path import dirname\nfrom os.path import split as psplit\nimport posixpath\n\n\nfrom six import PY2, text_type, binary_type, string_types\n\n# from datalad.dochelpers import get_docstring_split\nfrom datalad.consts import TIMESTAMP_FMT\n\n\nif PY2:\n    unicode_srctypes = string_types\nelse:\n    unicode_srctypes = string_types + (bytes,)\n\n\nlgr = logging.getLogger(\"datalad.utils\")\n\nlgr.log(5, \"Importing datalad.utils\")\n#\n# Some useful variables\n#\nplatform_system = platform.system().lower()\non_windows = platform_system == 'windows'\non_osx = platform_system == 'darwin'\non_linux = platform_system == 'linux'\non_msys_tainted_paths = on_windows \\\n                        and 'MSYS_NO_PATHCONV' not in os.environ \\\n                        and os.environ.get('MSYSTEM', '')[:4] in ('MSYS', 'MING')\ntry:\n    linux_distribution_name, linux_distribution_release \\\n        = platform.linux_distribution()[:2]\n    on_debian_wheezy = on_linux \\\n                       and linux_distribution_name == 'debian' \\\n                       and linux_distribution_release.startswith('7.')\nexcept:  # pragma: no cover\n    # MIH: IndexError?\n    on_debian_wheezy = False\n    linux_distribution_name = linux_distribution_release = None\n\n# Maximal length of cmdline string\n# Query the system and use hardcoded \"knowledge\" if None\n# probably   getconf ARG_MAX   might not be available\n# The last one would be the most conservative/Windows\nCMD_MAX_ARG_HARDCODED = 2097152 if on_linux else 262144 if on_osx else 32767\ntry:\n    CMD_MAX_ARG = os.sysconf('SC_ARG_MAX')\n    assert CMD_MAX_ARG > 0\nexcept Exception as exc:\n    # ATM (20181005) SC_ARG_MAX available only on POSIX systems\n    # so exception would be thrown e.g. on Windows, or\n    # somehow during Debian build for nd14.04 it is coming up with -1:\n    # https://github.com/datalad/datalad/issues/3015\n    CMD_MAX_ARG = CMD_MAX_ARG_HARDCODED\n    lgr.debug(\n        \"Failed to query or got useless SC_ARG_MAX sysconf, \"\n        \"will use hardcoded value: %s\", exc)\n# Even with all careful computations we do, due to necessity to account for\n# environment and what not, we still could not figure out \"exact\" way to\n# estimate it, but it was shown that 300k safety margin on linux was sufficient.\n# https://github.com/datalad/datalad/pull/2977#issuecomment-436264710\n# 300k is ~15%, so to be safe, and for paranoid us we will just use up to 50%\n# of the length for \"safety margin\".  We might probably still blow due to\n# env vars, unicode, etc...  so any hard limit imho is not a proper solution\nCMD_MAX_ARG = int(0.5 * CMD_MAX_ARG)\nlgr.debug(\n    \"Maximal length of cmdline string (adjusted for safety margin): %d\",\n    CMD_MAX_ARG)\n\n#\n# Little helpers\n#\n\n\ndef get_func_kwargs_doc(func):\n    \"\"\" Provides args for a function\n\n    Parameters\n    ----------\n    func: str\n      name of the function from which args are being requested\n\n    Returns\n    -------\n    list\n      of the args that a function takes in\n    \"\"\"\n    return getargspec(func)[0]\n\n    # TODO: format error message with descriptions of args\n    # return [repr(dict(get_docstring_split(func)[1]).get(x)) for x in getargspec(func)[0]]\n\n\ndef any_re_search(regexes, value):\n    \"\"\"Return if any of regexes (list or str) searches succesfully for value\"\"\"\n    for regex in assure_tuple_or_list(regexes):\n        if re.search(regex, value):\n            return True\n    return False\n\n\ndef not_supported_on_windows(msg=None):\n    \"\"\"A little helper to be invoked to consistently fail whenever functionality is\n    not supported (yet) on Windows\n    \"\"\"\n    if on_windows:\n        raise NotImplementedError(\"This functionality is not yet implemented for Windows OS\"\n                                  + (\": %s\" % msg if msg else \"\"))\n\n\ndef shortened_repr(value, l=30):\n    try:\n        if hasattr(value, '__repr__') and (value.__repr__ is not object.__repr__):\n            value_repr = repr(value)\n            if not value_repr.startswith('<') and len(value_repr) > l:\n                value_repr = \"<<%s...>>\" % (value_repr[:l - 8])\n            elif value_repr.startswith('<') and value_repr.endswith('>') and ' object at 0x':\n                raise ValueError(\"I hate those useless long reprs\")\n        else:\n            raise ValueError(\"gimme class\")\n    except Exception as e:\n        value_repr = \"<%s>\" % value.__class__.__name__.split('.')[-1]\n    return value_repr\n\n\ndef __auto_repr__(obj):\n    attr_names = tuple()\n    if hasattr(obj, '__dict__'):\n        attr_names += tuple(obj.__dict__.keys())\n    if hasattr(obj, '__slots__'):\n        attr_names += tuple(obj.__slots__)\n\n    items = []\n    for attr in sorted(set(attr_names)):\n        if attr.startswith('_'):\n            continue\n        value = getattr(obj, attr)\n        # TODO:  should we add this feature to minimize some talktative reprs\n        # such as of URL?\n        #if value is None:\n        #    continue\n        items.append(\"%s=%s\" % (attr, shortened_repr(value)))\n\n    return \"%s(%s)\" % (obj.__class__.__name__, ', '.join(items))\n\n\ndef auto_repr(cls):\n    \"\"\"Decorator for a class to assign it an automagic quick and dirty __repr__\n\n    It uses public class attributes to prepare repr of a class\n\n    Original idea: http://stackoverflow.com/a/27799004/1265472\n    \"\"\"\n\n    cls.__repr__ = __auto_repr__\n    return cls\n\n\ndef is_interactive():\n    \"\"\"Return True if all in/outs are tty\"\"\"\n    # TODO: check on windows if hasattr check would work correctly and add value:\n    #\n    return sys.stdin.isatty() and sys.stdout.isatty() and sys.stderr.isatty()\n\n\ndef get_ipython_shell():\n    \"\"\"Detect if running within IPython and returns its `ip` (shell) object\n\n    Returns None if not under ipython (no `get_ipython` function)\n    \"\"\"\n    try:\n        return get_ipython()\n    except NameError:\n        return None\n\n\ndef md5sum(filename):\n    with open(filename, 'rb') as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef sorted_files(dout):\n    \"\"\"Return a (sorted) list of files under dout\n    \"\"\"\n    return sorted(sum([[opj(r, f)[len(dout) + 1:] for f in files]\n                       for r, d, files in os.walk(dout)\n                       if not '.git' in r], []))\n\n_VCS_REGEX = '%s\\.(?:git|gitattributes|svn|bzr|hg)(?:%s|$)' % (dirsep, dirsep)\n_DATALAD_REGEX = '%s\\.(?:datalad)(?:%s|$)' % (dirsep, dirsep)\n\n\ndef find_files(regex, topdir=curdir, exclude=None, exclude_vcs=True, exclude_datalad=False, dirs=False):\n    \"\"\"Generator to find files matching regex\n\n    Parameters\n    ----------\n    regex: basestring\n    exclude: basestring, optional\n      Matches to exclude\n    exclude_vcs:\n      If True, excludes commonly known VCS subdirectories.  If string, used\n      as regex to exclude those files (regex: `%r`)\n    exclude_datalad:\n      If True, excludes files known to be datalad meta-data files (e.g. under\n      .datalad/ subdirectory) (regex: `%r`)\n    topdir: basestring, optional\n      Directory where to search\n    dirs: bool, optional\n      Whether to match directories as well as files\n    \"\"\"\n\n    for dirpath, dirnames, filenames in os.walk(topdir):\n        names = (dirnames + filenames) if dirs else filenames\n        # TODO: might want to uniformize on windows to use '/'\n        paths = (opj(dirpath, name) for name in names)\n        for path in filter(re.compile(regex).search, paths):\n            path = path.rstrip(dirsep)\n            if exclude and re.search(exclude, path):\n                continue\n            if exclude_vcs and re.search(_VCS_REGEX, path):\n                continue\n            if exclude_datalad and re.search(_DATALAD_REGEX, path):\n                continue\n            yield path\nfind_files.__doc__ %= (_VCS_REGEX, _DATALAD_REGEX)\n\n\ndef expandpath(path, force_absolute=True):\n    \"\"\"Expand all variables and user handles in a path.\n\n    By default return an absolute path\n    \"\"\"\n    path = expandvars(expanduser(path))\n    if force_absolute:\n        path = abspath(path)\n    return path\n\n\ndef posix_relpath(path, start=None):\n    \"\"\"Behave like os.path.relpath, but always return POSIX paths...\n\n    on any platform.\"\"\"\n    # join POSIX style\n    return posixpath.join(\n        # split and relpath native style\n        # python2.7 ntpath implementation of relpath cannot handle start=None\n        *psplit(\n            relpath(path, start=start if start is not None else '')))\n\n\ndef is_explicit_path(path):\n    \"\"\"Return whether a path explicitly points to a location\n\n    Any absolute path, or relative path starting with either '../' or\n    './' is assumed to indicate a location on the filesystem. Any other\n    path format is not considered explicit.\"\"\"\n    path = expandpath(path, force_absolute=False)\n    return isabs(path) \\\n        or path.startswith(os.curdir + os.sep) \\\n        or path.startswith(os.pardir + os.sep)\n\n\ndef rotree(path, ro=True, chmod_files=True):\n    \"\"\"To make tree read-only or writable\n\n    Parameters\n    ----------\n    path : string\n      Path to the tree/directory to chmod\n    ro : bool, optional\n      Whether to make it R/O (default) or RW\n    chmod_files : bool, optional\n      Whether to operate also on files (not just directories)\n    \"\"\"\n    if ro:\n        chmod = lambda f: os.chmod(f, os.stat(f).st_mode & ~stat.S_IWRITE)\n    else:\n        chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)\n\n    for root, dirs, files in os.walk(path, followlinks=False):\n        if chmod_files:\n            for f in files:\n                fullf = opj(root, f)\n                # might be the \"broken\" symlink which would fail to stat etc\n                if exists(fullf):\n                    chmod(fullf)\n        chmod(root)\n\n\ndef rmtree(path, chmod_files='auto', children_only=False, *args, **kwargs):\n    \"\"\"To remove git-annex .git it is needed to make all files and directories writable again first\n\n    Parameters\n    ----------\n    chmod_files : string or bool, optional\n       Whether to make files writable also before removal.  Usually it is just\n       a matter of directories to have write permissions.\n       If 'auto' it would chmod files on windows by default\n    children_only : bool, optional\n       If set, all files and subdirectories would be removed while the path\n       itself (must be a directory) would be preserved\n    `*args` :\n    `**kwargs` :\n       Passed into shutil.rmtree call\n    \"\"\"\n    # Give W permissions back only to directories, no need to bother with files\n    if chmod_files == 'auto':\n        chmod_files = on_windows\n    # TODO:  yoh thinks that if we could quickly check our Flyweight for\n    #        repos if any of them is under the path, and could call .precommit\n    #        on those to possibly stop batched processes etc, we did not have\n    #        to do it on case by case\n    # Check for open files\n    assert_no_open_files(path)\n\n    if children_only:\n        if not os.path.isdir(path):\n            raise ValueError(\"Can remove children only of directories\")\n        for p in os.listdir(path):\n            rmtree(op.join(path, p))\n        return\n    if not (os.path.islink(path) or not os.path.isdir(path)):\n        rotree(path, ro=False, chmod_files=chmod_files)\n        _rmtree(path, *args, **kwargs)\n    else:\n        # just remove the symlink\n        unlink(path)\n\n\ndef rmdir(path, *args, **kwargs):\n    \"\"\"os.rmdir with our optional checking for open files\"\"\"\n    assert_no_open_files(path)\n    os.rmdir(path)\n\n\ndef get_open_files(path, log_open=False):\n    \"\"\"Get open files under a path\n\n    Parameters\n    ----------\n    path : str\n      File or directory to check for open files under\n    log_open : bool or int\n      If set - logger level to use\n\n    Returns\n    -------\n    dict\n      path : pid\n\n    \"\"\"\n    # Original idea: https://stackoverflow.com/a/11115521/1265472\n    import psutil\n    files = {}\n    # since the ones returned by psutil would not be aware of symlinks in the\n    # path we should also get realpath for path\n    path = realpath(path)\n    for proc in psutil.process_iter():\n        try:\n            open_paths = [p.path for p in proc.open_files()] + [proc.cwd()]\n            for p in open_paths:\n                # note: could be done more efficiently so we do not\n                # renormalize path over and over again etc\n                if path_startswith(p, path):\n                    files[p] = proc.pid\n        # Catch a race condition where a process ends\n        # before we can examine its files\n        except psutil.NoSuchProcess:\n            pass\n        except psutil.AccessDenied:\n            pass\n\n    if files and log_open:\n        lgr.log(log_open, \"Open files under %s: %s\", path, files)\n    return files\n\n\n_assert_no_open_files_cfg = os.environ.get('DATALAD_ASSERT_NO_OPEN_FILES')\nif _assert_no_open_files_cfg:\n    def assert_no_open_files(path):\n        files = get_open_files(path, log_open=40)\n        if _assert_no_open_files_cfg == 'assert':\n            assert not files\n        elif files:\n            if _assert_no_open_files_cfg == 'pdb':\n                import pdb\n                pdb.set_trace()\n            elif _assert_no_open_files_cfg == 'epdb':\n                import epdb\n                epdb.serve()\n            pass\n        # otherwise we would just issue that error message in the log\nelse:\n    def assert_no_open_files(*args, **kwargs):\n        pass\n\n\ndef rmtemp(f, *args, **kwargs):\n    \"\"\"Wrapper to centralize removing of temp files so we could keep them around\n\n    It will not remove the temporary file/directory if DATALAD_TESTS_TEMP_KEEP\n    environment variable is defined\n    \"\"\"\n    if not os.environ.get('DATALAD_TESTS_TEMP_KEEP'):\n        if not os.path.lexists(f):\n            lgr.debug(\"Path %s does not exist, so can't be removed\" % f)\n            return\n        lgr.log(5, \"Removing temp file: %s\" % f)\n        # Can also be a directory\n        if os.path.isdir(f):\n            rmtree(f, *args, **kwargs)\n        else:\n            unlink(f)\n    else:\n        lgr.info(\"Keeping temp file: %s\" % f)\n\n\ndef file_basename(name, return_ext=False):\n    \"\"\"\n    Strips up to 2 extensions of length up to 4 characters and starting with alpha\n    not a digit, so we could get rid of .tar.gz etc\n    \"\"\"\n    bname = basename(name)\n    fbname = re.sub('(\\.[a-zA-Z_]\\S{1,4}){0,2}$', '', bname)\n    if return_ext:\n        return fbname, bname[len(fbname) + 1:]\n    else:\n        return fbname\n\n\ndef escape_filename(filename):\n    \"\"\"Surround filename in \"\" and escape \" in the filename\n    \"\"\"\n    filename = filename.replace('\"', r'\\\"').replace('`', r'\\`')\n    filename = '\"%s\"' % filename\n    return filename\n\n\ndef encode_filename(filename):\n    \"\"\"Encode unicode filename\n    \"\"\"\n    if isinstance(filename, text_type):\n        return filename.encode(sys.getfilesystemencoding())\n    else:\n        return filename\n\n\ndef decode_input(s):\n    \"\"\"Given input string/bytes, decode according to stdin codepage (or UTF-8)\n    if not defined\n\n    If fails -- issue warning and decode allowing for errors\n    being replaced\n    \"\"\"\n    if isinstance(s, text_type):\n        return s\n    else:\n        encoding = sys.stdin.encoding or 'UTF-8'\n        try:\n            return s.decode(encoding)\n        except UnicodeDecodeError as exc:\n            lgr.warning(\n                \"Failed to decode input string using %s encoding. \"\n                \"Decoding allowing for errors\", encoding)\n            return s.decode(encoding, errors='replace')\n\n\nif on_windows:\n    def lmtime(filepath, mtime):\n        \"\"\"Set mtime for files.  On Windows a merely adapter to os.utime\n        \"\"\"\n        os.utime(filepath, (time.time(), mtime))\nelse:\n    def lmtime(filepath, mtime):\n        \"\"\"Set mtime for files, while not de-referencing symlinks.\n\n        To overcome absence of os.lutime\n\n        Works only on linux and OSX ATM\n        \"\"\"\n        from .cmd import Runner\n        # convert mtime to format touch understands [[CC]YY]MMDDhhmm[.SS]\n        smtime = time.strftime(\"%Y%m%d%H%M.%S\", time.localtime(mtime))\n        lgr.log(3, \"Setting mtime for %s to %s == %s\", filepath, mtime, smtime)\n        Runner().run(['touch', '-h', '-t', '%s' % smtime, filepath])\n        rfilepath = realpath(filepath)\n        if islink(filepath) and exists(rfilepath):\n            # trust noone - adjust also of the target file\n            # since it seemed like downloading under OSX (was it using curl?)\n            # didn't bother with timestamps\n            lgr.log(3, \"File is a symlink to %s Setting mtime for it to %s\",\n                    rfilepath, mtime)\n            os.utime(rfilepath, (time.time(), mtime))\n        # doesn't work on OSX\n        # Runner().run(['touch', '-h', '-d', '@%s' % mtime, filepath])\n\n\ndef assure_tuple_or_list(obj):\n    \"\"\"Given an object, wrap into a tuple if not list or tuple\n    \"\"\"\n    if isinstance(obj, (list, tuple)):\n        return obj\n    return (obj,)\n\n\ndef assure_iter(s, cls, copy=False, iterate=True):\n    \"\"\"Given not a list, would place it into a list. If None - empty list is returned\n\n    Parameters\n    ----------\n    s: list or anything\n    cls: class\n      Which iterable class to assure\n    copy: bool, optional\n      If correct iterable is passed, it would generate its shallow copy\n    iterate: bool, optional\n      If it is not a list, but something iterable (but not a text_type)\n      iterate over it.\n    \"\"\"\n\n    if isinstance(s, cls):\n        return s if not copy else shallow_copy(s)\n    elif isinstance(s, text_type):\n        return cls((s,))\n    elif iterate and hasattr(s, '__iter__'):\n        return cls(s)\n    elif s is None:\n        return cls()\n    else:\n        return cls((s,))\n\n\ndef assure_list(s, copy=False, iterate=True):\n    \"\"\"Given not a list, would place it into a list. If None - empty list is returned\n\n    Parameters\n    ----------\n    s: list or anything\n    copy: bool, optional\n      If list is passed, it would generate a shallow copy of the list\n    iterate: bool, optional\n      If it is not a list, but something iterable (but not a text_type)\n      iterate over it.\n    \"\"\"\n    return assure_iter(s, list, copy=copy, iterate=iterate)\n\n\ndef assure_list_from_str(s, sep='\\n'):\n    \"\"\"Given a multiline string convert it to a list of return None if empty\n\n    Parameters\n    ----------\n    s: str or list\n    \"\"\"\n\n    if not s:\n        return None\n\n    if isinstance(s, list):\n        return s\n    return s.split(sep)\n\n\ndef assure_dict_from_str(s, **kwargs):\n    \"\"\"Given a multiline string with key=value items convert it to a dictionary\n\n    Parameters\n    ----------\n    s: str or dict\n\n    Returns None if input s is empty\n    \"\"\"\n\n    if not s:\n        return None\n\n    if isinstance(s, dict):\n        return s\n\n    out = {}\n    for value_str in assure_list_from_str(s, **kwargs):\n        if '=' not in value_str:\n            raise ValueError(\"{} is not in key=value format\".format(repr(value_str)))\n        k, v = value_str.split('=', 1)\n        if k in out:\n            err = \"key {} was already defined in {}, but new value {} was provided\".format(k, out, v)\n            raise ValueError(err)\n        out[k] = v\n    return out\n\n\ndef assure_bytes(s, encoding='utf-8'):\n    \"\"\"Convert/encode unicode to str (PY2) or bytes (PY3) if of 'text_type'\n\n    Parameters\n    ----------\n    encoding: str, optional\n      Encoding to use.  \"utf-8\" is the default\n    \"\"\"\n    if not isinstance(s, text_type):\n        return s\n    return s.encode(encoding)\n\n\ndef assure_unicode(s, encoding=None, confidence=None):\n    \"\"\"Convert/decode to unicode (PY2) or str (PY3) if of 'binary_type'\n\n    Parameters\n    ----------\n    encoding: str, optional\n      Encoding to use.  If None, \"utf-8\" is tried, and then if not a valid\n      UTF-8, encoding will be guessed\n    confidence: float, optional\n      A value between 0 and 1, so if guessing of encoding is of lower than\n      specified confidence, ValueError is raised\n    \"\"\"\n    if not isinstance(s, binary_type):\n        return s\n    if encoding is None:\n        # Figure out encoding, defaulting to 'utf-8' which is our common\n        # target in contemporary digital society\n        try:\n            return s.decode('utf-8')\n        except UnicodeDecodeError as exc:\n            from .dochelpers import exc_str\n            lgr.debug(\"Failed to decode a string as utf-8: %s\", exc_str(exc))\n        # And now we could try to guess\n        from chardet import detect\n        enc = detect(s)\n        denc = enc.get('encoding', None)\n        if denc:\n            denc_confidence = enc.get('confidence', 0)\n            if confidence is not None and  denc_confidence < confidence:\n                raise ValueError(\n                    \"Failed to auto-detect encoding with high enough \"\n                    \"confidence. Highest confidence was %s for %s\"\n                    % (denc_confidence, denc)\n                )\n            return s.decode(denc)\n        else:\n            raise ValueError(\n                \"Could not decode value as utf-8, or to guess its encoding: %s\"\n                % repr(s)\n            )\n    else:\n        return s.decode(encoding)\n\n\ndef assure_bool(s):\n    \"\"\"Convert value into boolean following convention for strings\n\n    to recognize on,True,yes as True, off,False,no as False\n    \"\"\"\n    if isinstance(s, string_types):\n        if s.isdigit():\n            return bool(int(s))\n        sl = s.lower()\n        if sl in {'y', 'yes', 'true', 'on'}:\n            return True\n        elif sl in {'n', 'no', 'false', 'off'}:\n            return False\n        else:\n            raise ValueError(\"Do not know how to treat %r as a boolean\" % s)\n    return bool(s)\n\n\ndef as_unicode(val, cast_types=object):\n    \"\"\"Given an arbitrary value, would try to obtain unicode value of it\n    \n    For unicode it would return original value, for python2 str or python3\n    bytes it would use assure_unicode, for None - an empty (unicode) string,\n    and for any other type (see `cast_types`) - would apply the unicode \n    constructor.  If value is not an instance of `cast_types`, TypeError\n    is thrown\n    \n    Parameters\n    ----------\n    cast_types: type\n      Which types to cast to unicode by providing to constructor\n    \"\"\"\n    if val is None:\n        return u''\n    elif isinstance(val, text_type):\n        return val\n    elif isinstance(val, unicode_srctypes):\n        return assure_unicode(val)\n    elif isinstance(val, cast_types):\n        return text_type(val)\n    else:\n        raise TypeError(\n            \"Value %r is not of any of known or provided %s types\"\n            % (val, cast_types))\n\n\ndef unique(seq, key=None):\n    \"\"\"Given a sequence return a list only with unique elements while maintaining order\n\n    This is the fastest solution.  See\n    https://www.peterbe.com/plog/uniqifiers-benchmark\n    and\n    http://stackoverflow.com/a/480227/1265472\n    for more information.\n    Enhancement -- added ability to compare for uniqueness using a key function\n\n    Parameters\n    ----------\n    seq:\n      Sequence to analyze\n    key: callable, optional\n      Function to call on each element so we could decide not on a full\n      element, but on its member etc\n    \"\"\"\n    seen = set()\n    seen_add = seen.add\n    if not key:\n        return [x for x in seq if not (x in seen or seen_add(x))]\n    else:\n        # OPT: could be optimized, since key is called twice, but for our cases\n        # should be just as fine\n        return [x for x in seq if not (key(x) in seen or seen_add(key(x)))]\n\n\ndef all_same(items):\n    \"\"\"Quick check if all items are the same.\n\n    Identical to a check like len(set(items)) == 1 but\n    should be more efficient while working on generators, since would\n    return False as soon as any difference detected thus possibly avoiding\n    unnecessary evaluations\n    \"\"\"\n    first = True\n    first_item = None\n    for item in items:\n        if first:\n            first = False\n            first_item = item\n        else:\n            if item != first_item:\n                return False\n    # So we return False if was empty\n    return not first\n\n\ndef map_items(func, v):\n    \"\"\"A helper to apply `func` to all elements (keys and values) within dict\n\n    No type checking of values passed to func is done, so `func`\n    should be resilient to values which it should not handle\n\n    Initial usecase - apply_recursive(url_fragment, assure_unicode)\n    \"\"\"\n    # map all elements within item\n    return v.__class__(\n        item.__class__(map(func, item))\n        for item in v.items()\n    )\n\n\ndef partition(items, predicate=bool):\n    \"\"\"Partition `items` by `predicate`.\n\n    Parameters\n    ----------\n    items : iterable\n    predicate : callable\n        A function that will be mapped over each element in `items`. The\n        elements will partitioned based on whether the return value is false or\n        true.\n\n    Returns\n    -------\n    A tuple with two generators, the first for 'false' items and the second for\n    'true' ones.\n\n    Notes\n    -----\n    Taken from Peter Otten's snippet posted at\n    https://nedbatchelder.com/blog/201306/filter_a_list_into_two_parts.html\n    \"\"\"\n    a, b = tee((predicate(item), item) for item in items)\n    return ((item for pred, item in a if not pred),\n            (item for pred, item in b if pred))\n\n\ndef generate_chunks(container, size):\n    \"\"\"Given a container, generate chunks from it with size up to `size`\n    \"\"\"\n    # There could be a \"smarter\" solution but I think this would suffice\n    assert size > 0,  \"Size should be non-0 positive\"\n    while container:\n        yield container[:size]\n        container = container[size:]\n\n#\n# Generators helpers\n#\n\ndef saved_generator(gen):\n    \"\"\"Given a generator returns two generators, where 2nd one just replays\n\n    So the first one would be going through the generated items and 2nd one\n    would be yielding saved items\n    \"\"\"\n    saved = []\n\n    def gen1():\n        for x in gen:  # iterating over original generator\n            saved.append(x)\n            yield x\n\n    def gen2():\n        for x in saved:  # yielding saved entries\n            yield x\n\n    return gen1(), gen2()\n\n\n#\n# Decorators\n#\ndef better_wraps(to_be_wrapped):\n    \"\"\"Decorator to replace `functools.wraps`\n\n    This is based on `wrapt` instead of `functools` and in opposition to `wraps`\n    preserves the correct signature of the decorated function.\n    It is written with the intention to replace the use of `wraps` without any\n    need to rewrite the actual decorators.\n    \"\"\"\n\n    @wrapt.decorator(adapter=to_be_wrapped)\n    def intermediator(to_be_wrapper, instance, args, kwargs):\n        return to_be_wrapper(*args, **kwargs)\n\n    return intermediator\n\n\n# Borrowed from pandas\n# Copyright: 2011-2014, Lambda Foundry, Inc. and PyData Development Team\n# License: BSD-3\ndef optional_args(decorator):\n    \"\"\"allows a decorator to take optional positional and keyword arguments.\n        Assumes that taking a single, callable, positional argument means that\n        it is decorating a function, i.e. something like this::\n\n            @my_decorator\n            def function(): pass\n\n        Calls decorator with decorator(f, `*args`, `**kwargs`)\"\"\"\n\n    @better_wraps(decorator)\n    def wrapper(*args, **kwargs):\n        def dec(f):\n            return decorator(f, *args, **kwargs)\n\n        is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable)\n        if is_decorating:\n            f = args[0]\n            args = []\n            return dec(f)\n        else:\n            return dec\n\n    return wrapper\n\n\n# TODO: just provide decorators for tempfile.mk* functions. This is ugly!\ndef get_tempfile_kwargs(tkwargs=None, prefix=\"\", wrapped=None):\n    \"\"\"Updates kwargs to be passed to tempfile. calls depending on env vars\n    \"\"\"\n    if tkwargs is None:\n        tkwargs_ = {}\n    else:\n        # operate on a copy of tkwargs to avoid any side-effects\n        tkwargs_ = tkwargs.copy()\n\n    # TODO: don't remember why I had this one originally\n    # if len(targs)<2 and \\\n    if 'prefix' not in tkwargs_:\n        tkwargs_['prefix'] = '_'.join(\n            ['datalad_temp'] +\n            ([prefix] if prefix else []) +\n            ([''] if (on_windows or not wrapped) else [wrapped.__name__]))\n\n    directory = os.environ.get('DATALAD_TESTS_TEMP_DIR')\n    if directory and 'dir' not in tkwargs_:\n        tkwargs_['dir'] = directory\n\n    return tkwargs_\n\n\n@optional_args\ndef line_profile(func):\n    \"\"\"Q&D helper to line profile the function and spit out stats\n    \"\"\"\n    import line_profiler\n    prof = line_profiler.LineProfiler()\n\n    @wraps(func)\n    def newfunc(*args, **kwargs):\n        try:\n            pfunc = prof(func)\n            return pfunc(*args, **kwargs)\n        finally:\n            prof.print_stats()\n    return newfunc\n\n#\n# Context Managers\n#\n\n\n@contextmanager\ndef nothing_cm():\n    \"\"\"Just a dummy cm to programmically switch context managers\"\"\"\n    yield\n\n\n@contextmanager\ndef swallow_outputs():\n    \"\"\"Context manager to help consuming both stdout and stderr, and print()\n\n    stdout is available as cm.out and stderr as cm.err whenever cm is the\n    yielded context manager.\n    Internally uses temporary files to guarantee absent side-effects of swallowing\n    into StringIO which lacks .fileno.\n\n    print mocking is necessary for some uses where sys.stdout was already bound\n    to original sys.stdout, thus mocking it later had no effect. Overriding\n    print function had desired effect\n    \"\"\"\n\n    class StringIOAdapter(object):\n        \"\"\"Little adapter to help getting out/err values\n        \"\"\"\n        def __init__(self):\n            kw = get_tempfile_kwargs({}, prefix=\"outputs\")\n\n            self._out = open(tempfile.mktemp(**kw), 'w')\n            self._err = open(tempfile.mktemp(**kw), 'w')\n\n        def _read(self, h):\n            with open(h.name) as f:\n                return f.read()\n\n        @property\n        def out(self):\n            if not self._out.closed:\n                self._out.flush()\n            return self._read(self._out)\n\n        @property\n        def err(self):\n            if not self._err.closed:\n                self._err.flush()\n            return self._read(self._err)\n\n        @property\n        def handles(self):\n            return self._out, self._err\n\n        def cleanup(self):\n            self._out.close()\n            self._err.close()\n            out_name = self._out.name\n            err_name = self._err.name\n            del self._out\n            del self._err\n            gc.collect()\n            rmtemp(out_name)\n            rmtemp(err_name)\n\n    def fake_print(*args, **kwargs):\n        sep = kwargs.pop('sep', ' ')\n        end = kwargs.pop('end', '\\n')\n        file = kwargs.pop('file', sys.stdout)\n\n        if file in (oldout, olderr, sys.stdout, sys.stderr):\n            # we mock\n            try:\n                sys.stdout.write(sep.join(args) + end)\n            except UnicodeEncodeError as exc:\n                lgr.error(\n                    \"Failed to write to mocked stdout, got %s, continue as it \"\n                    \"didn't happen\",  exc)\n        else:\n            # must be some other file one -- leave it alone\n            oldprint(*args, sep=sep, end=end, file=file)\n\n    from .ui import ui\n    # preserve -- they could have been mocked already\n    oldprint = getattr(__builtin__, 'print')\n    oldout, olderr = sys.stdout, sys.stderr\n    olduiout = ui.out\n    adapter = StringIOAdapter()\n\n    try:\n        sys.stdout, sys.stderr = adapter.handles\n        ui.out = adapter.handles[0]\n        setattr(__builtin__, 'print', fake_print)\n\n        yield adapter\n    finally:\n        sys.stdout, sys.stderr, ui.out = oldout, olderr, olduiout\n        setattr(__builtin__, 'print',  oldprint)\n        adapter.cleanup()\n\n\n@contextmanager\ndef swallow_logs(new_level=None, file_=None, name='datalad'):\n    \"\"\"Context manager to consume all logs.\n\n    \"\"\"\n    lgr = logging.getLogger(name)\n\n    # Keep old settings\n    old_level = lgr.level\n    old_handlers = lgr.handlers\n\n    # Let's log everything into a string\n    # TODO: generalize with the one for swallow_outputs\n    class StringIOAdapter(object):\n        \"\"\"Little adapter to help getting out values\n\n        And to stay consistent with how swallow_outputs behaves\n        \"\"\"\n        def __init__(self):\n            if file_ is None:\n                kw = get_tempfile_kwargs({}, prefix=\"logs\")\n                out_file = tempfile.mktemp(**kw)\n            else:\n                out_file = file_\n            # PY3 requires clearly one or another.  race condition possible\n            self._out = open(out_file, 'a')\n            self._final_out = None\n\n        def _read(self, h):\n            with open(h.name) as f:\n                return f.read()\n\n        @property\n        def out(self):\n            if self._final_out is not None:\n                # we closed and cleaned up already\n                return self._final_out\n            else:\n                self._out.flush()\n                return self._read(self._out)\n\n        @property\n        def lines(self):\n            return self.out.split('\\n')\n\n        @property\n        def handle(self):\n            return self._out\n\n        def cleanup(self):\n            # store for access while object exists\n            self._final_out = self.out\n            self._out.close()\n            out_name = self._out.name\n            del self._out\n            gc.collect()\n            if not file_:\n                rmtemp(out_name)\n\n        def assert_logged(self, msg=None, level=None, regex=True, **kwargs):\n            \"\"\"Provide assertion on whether a msg was logged at a given level\n\n            If neither `msg` nor `level` provided, checks if anything was logged\n            at all.\n\n            Parameters\n            ----------\n            msg: str, optional\n              Message (as a regular expression, if `regex`) to be searched.\n              If no msg provided, checks if anything was logged at a given level.\n            level: str, optional\n              String representing the level to be logged\n            regex: bool, optional\n              If False, regular `assert_in` is used\n            **kwargs: str, optional\n              Passed to `assert_re_in` or `assert_in`\n            \"\"\"\n            from datalad.tests.utils import assert_re_in\n            from datalad.tests.utils import assert_in\n\n            if regex:\n                match = '\\[%s\\] ' % level if level else \"\\[\\S+\\] \"\n            else:\n                match = '[%s] ' % level if level else ''\n\n            if msg:\n                match += msg\n\n            if match:\n                (assert_re_in if regex else assert_in)(match, self.out, **kwargs)\n            else:\n                assert not kwargs, \"no kwargs to be passed anywhere\"\n                assert self.out, \"Nothing was logged!?\"\n\n    adapter = StringIOAdapter()\n    # TODO: it does store messages but without any formatting, i.e. even without\n    # date/time prefix etc.  IMHO it should preserve formatting in case if file_ is\n    # set\n    swallow_handler = logging.StreamHandler(adapter.handle)\n    # we want to log levelname so we could test against it\n    swallow_handler.setFormatter(\n        logging.Formatter('[%(levelname)s] %(message)s'))\n    # Inherit filters\n    from datalad.log import ProgressHandler\n    swallow_handler.filters = sum([h.filters for h in old_handlers\n                                   if not isinstance(h, ProgressHandler)],\n                                  [])\n    lgr.handlers = [swallow_handler]\n    if old_level < logging.DEBUG:  # so if HEAVYDEBUG etc -- show them!\n        lgr.handlers += old_handlers\n\n    if isinstance(new_level, str):\n        new_level = getattr(logging, new_level)\n\n    if new_level is not None:\n        lgr.setLevel(new_level)\n\n    try:\n        yield adapter\n        # TODO: if file_ and there was an exception -- most probably worth logging it?\n        # although ideally it should be the next log outside added to that file_ ... oh well\n    finally:\n        lgr.handlers, lgr.level = old_handlers, old_level\n        adapter.cleanup()\n\n\n# TODO: May be melt in with swallow_logs at some point:\n@contextmanager\ndef disable_logger(logger=None):\n    \"\"\"context manager to temporarily disable logging\n\n    This is to provide one of swallow_logs' purposes without unnecessarily\n    creating temp files (see gh-1865)\n\n    Parameters\n    ----------\n    logger: Logger\n        Logger whose handlers will be ordered to not log anything.\n        Default: datalad's topmost Logger ('datalad')\n    \"\"\"\n\n    class NullFilter(logging.Filter):\n        \"\"\"Filter class to reject all records\n        \"\"\"\n        def filter(self, record):\n            return 0\n\n    if logger is None:\n        # default: all of datalad's logging:\n        logger = logging.getLogger('datalad')\n\n    filter_ = NullFilter(logger.name)\n    [h.addFilter(filter_) for h in logger.handlers]\n\n    try:\n        yield logger\n    finally:\n        [h.removeFilter(filter_) for h in logger.handlers]\n\n\n#\n# Additional handlers\n#\n_sys_excepthook = sys.excepthook  # Just in case we ever need original one\n\n\ndef setup_exceptionhook(ipython=False):\n    \"\"\"Overloads default sys.excepthook with our exceptionhook handler.\n\n       If interactive, our exceptionhook handler will invoke\n       pdb.post_mortem; if not interactive, then invokes default handler.\n    \"\"\"\n\n    def _datalad_pdb_excepthook(type, value, tb):\n        import traceback\n        traceback.print_exception(type, value, tb)\n        print()\n        if is_interactive():\n            import pdb\n            pdb.post_mortem(tb)\n\n    if ipython:\n        from IPython.core import ultratb\n        sys.excepthook = ultratb.FormattedTB(mode='Verbose',\n                                             # color_scheme='Linux',\n                                             call_pdb=is_interactive())\n    else:\n        sys.excepthook = _datalad_pdb_excepthook\n\n\ndef assure_dir(*args):\n    \"\"\"Make sure directory exists.\n\n    Joins the list of arguments to an os-specific path to the desired\n    directory and creates it, if it not exists yet.\n    \"\"\"\n    dirname = opj(*args)\n    if not exists(dirname):\n        os.makedirs(dirname)\n    return dirname\n\n\ndef updated(d, update):\n    \"\"\"Return a copy of the input with the 'update'\n\n    Primarily for updating dictionaries\n    \"\"\"\n    d = d.copy()\n    d.update(update)\n    return d\n\n\n_pwd_mode = None\n\n\ndef _switch_to_getcwd(msg, *args):\n    global _pwd_mode\n    _pwd_mode = 'cwd'\n    lgr.warning(\n        msg + \". From now on will be returning os.getcwd(). Directory\"\n               \" symlinks in the paths will be resolved\",\n        *args\n    )\n    # TODO:  we might want to mitigate by going through all flywheighted\n    # repos and tuning up their .paths to be resolved?\n\n\ndef getpwd():\n    \"\"\"Try to return a CWD without dereferencing possible symlinks\n\n    This function will try to use PWD environment variable to provide a current\n    working directory, possibly with some directories along the path being\n    symlinks to other directories.  Unfortunately, PWD is used/set only by the\n    shell and such functions as `os.chdir` and `os.getcwd` nohow use or modify\n    it, thus `os.getcwd()` returns path with links dereferenced.\n\n    While returning current working directory based on PWD env variable we\n    verify that the directory is the same as `os.getcwd()` after resolving all\n    symlinks.  If that verification fails, we fall back to always use\n    `os.getcwd()`.\n\n    Initial decision to either use PWD env variable or os.getcwd() is done upon\n    the first call of this function.\n    \"\"\"\n    global _pwd_mode\n    if _pwd_mode is None:\n        # we need to decide!\n        try:\n            pwd = os.environ['PWD']\n            if on_windows and pwd and pwd.startswith('/'):\n                # It should be a path from MSYS.\n                # - it might start with a drive letter or not\n                # - it seems to be \"illegal\" to have a single letter directories\n                #   under / path, i.e. if created - they aren't found\n                # - 'ln -s' does not fail to create a \"symlink\" but it just\n                # copies!\n                #   so we are not likely to need original PWD purpose on\n                # those systems\n                # Verdict:\n                _pwd_mode = 'cwd'\n            else:\n                _pwd_mode = 'PWD'\n        except KeyError:\n            _pwd_mode = 'cwd'\n\n    if _pwd_mode == 'cwd':\n        return os.getcwd()\n    elif _pwd_mode == 'PWD':\n        try:\n            cwd = os.getcwd()\n        except OSError as exc:\n            if \"o such file\" in str(exc):\n                # directory was removed but we promised to be robust and\n                # still report the path we might know since we are still in PWD\n                # mode\n                cwd = None\n            else:\n                raise\n        try:\n            pwd = os.environ['PWD']\n            pwd_real = op.realpath(pwd)\n            # This logic would fail to catch the case where chdir did happen\n            # to the directory where current PWD is pointing to, e.g.\n            # $> ls -ld $PWD\n            # lrwxrwxrwx 1 yoh yoh 5 Oct 11 13:27 /home/yoh/.tmp/tmp -> /tmp//\n            # hopa:~/.tmp/tmp\n            # $> python -c 'import os; os.chdir(\"/tmp\"); from datalad.utils import getpwd; print(getpwd(), os.getcwd())'\n            # ('/home/yoh/.tmp/tmp', '/tmp')\n            # but I guess that should not be too harmful\n            if cwd is not None and pwd_real != cwd:\n                _switch_to_getcwd(\n                    \"realpath of PWD=%s is %s whenever os.getcwd()=%s\",\n                    pwd, pwd_real, cwd\n                )\n                return cwd\n            return pwd\n        except KeyError:\n            _switch_to_getcwd(\"PWD env variable is no longer available\")\n            return cwd  # Must not happen, but may be someone\n                        # evil purges PWD from environ?\n    else:\n        raise RuntimeError(\n            \"Must have not got here. \"\n            \"pwd_mode must be either cwd or PWD. And it is now %r\" % (_pwd_mode,)\n        )\n\n\nclass chpwd(object):\n    \"\"\"Wrapper around os.chdir which also adjusts environ['PWD']\n\n    The reason is that otherwise PWD is simply inherited from the shell\n    and we have no ability to assess directory path without dereferencing\n    symlinks.\n\n    If used as a context manager it allows to temporarily change directory\n    to the given path\n    \"\"\"\n    def __init__(self, path, mkdir=False, logsuffix=''):\n\n        if path:\n            pwd = getpwd()\n            self._prev_pwd = pwd\n        else:\n            self._prev_pwd = None\n            return\n\n        if not isabs(path):\n            path = normpath(opj(pwd, path))\n        if not os.path.exists(path) and mkdir:\n            self._mkdir = True\n            os.mkdir(path)\n        else:\n            self._mkdir = False\n        lgr.debug(\"chdir %r -> %r %s\", self._prev_pwd, path, logsuffix)\n        os.chdir(path)  # for grep people -- ok, to chdir here!\n        os.environ['PWD'] = path\n\n    def __enter__(self):\n        # nothing more to do really, chdir was in the constructor\n        pass\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        if self._prev_pwd:\n            # Need to use self.__class__ so this instance, if the entire\n            # thing mocked during the test, still would use correct chpwd\n            self.__class__(self._prev_pwd, logsuffix=\"(coming back)\")\n\n\ndef dlabspath(path, norm=False):\n    \"\"\"Symlinks-in-the-cwd aware abspath\n\n    os.path.abspath relies on os.getcwd() which would not know about symlinks\n    in the path\n\n    TODO: we might want to norm=True by default to match behavior of\n    os .path.abspath?\n    \"\"\"\n    if not isabs(path):\n        # if not absolute -- relative to pwd\n        path = opj(getpwd(), path)\n    return normpath(path) if norm else path\n\n\ndef with_pathsep(path):\n    \"\"\"Little helper to guarantee that path ends with /\"\"\"\n    return path + sep if not path.endswith(sep) else path\n\n\ndef get_path_prefix(path, pwd=None):\n    \"\"\"Get path prefix (for current directory)\n\n    Returns relative path to the topdir, if we are under topdir, and if not\n    absolute path to topdir.  If `pwd` is not specified - current directory\n    assumed\n    \"\"\"\n    pwd = pwd or getpwd()\n    path = dlabspath(path)\n    path_ = with_pathsep(path)\n    pwd_ = with_pathsep(pwd)\n    common = commonprefix((path_, pwd_))\n    if common.endswith(sep) and common in {path_, pwd_}:\n        # we are in subdir or above the path = use relative path\n        location_prefix = relpath(path, pwd)\n        # if benign \"here\" - cut off\n        if location_prefix in (curdir, curdir + sep):\n            location_prefix = ''\n        return location_prefix\n    else:\n        # just return absolute path\n        return path\n\n\ndef _get_normalized_paths(path, prefix):\n    if isabs(path) != isabs(prefix):\n        raise ValueError(\"Both paths must either be absolute or relative. \"\n                         \"Got %r and %r\" % (path, prefix))\n    path = with_pathsep(path)\n    prefix = with_pathsep(prefix)\n    return path, prefix\n\n\ndef path_startswith(path, prefix):\n    \"\"\"Return True if path starts with prefix path\n\n    Parameters\n    ----------\n    path: str\n    prefix: str\n    \"\"\"\n    path, prefix = _get_normalized_paths(path, prefix)\n    return path.startswith(prefix)\n\n\ndef path_is_subpath(path, prefix):\n    \"\"\"Return True if path is a subpath of prefix\n\n    It will return False if path == prefix.\n\n    Parameters\n    ----------\n    path: str\n    prefix: str\n    \"\"\"\n    path, prefix = _get_normalized_paths(path, prefix)\n    return (len(prefix) < len(path)) and path.startswith(prefix)\n\n\ndef knows_annex(path):\n    \"\"\"Returns whether at a given path there is information about an annex\n\n    It is just a thin wrapper around GitRepo.is_with_annex() classmethod\n    which also checks for `path` to exist first.\n\n    This includes actually present annexes, but also uninitialized ones, or\n    even the presence of a remote annex branch.\n    \"\"\"\n    from os.path import exists\n    if not exists(path):\n        lgr.debug(\"No annex: test path {0} doesn't exist\".format(path))\n        return False\n    from datalad.support.gitrepo import GitRepo\n    return GitRepo(path, init=False, create=False).is_with_annex()\n\n\n@contextmanager\ndef make_tempfile(content=None, wrapped=None, **tkwargs):\n    \"\"\"Helper class to provide a temporary file name and remove it at the end (context manager)\n\n    Parameters\n    ----------\n    mkdir : bool, optional (default: False)\n        If True, temporary directory created using tempfile.mkdtemp()\n    content : str or bytes, optional\n        Content to be stored in the file created\n    wrapped : function, optional\n        If set, function name used to prefix temporary file name\n    `**tkwargs`:\n        All other arguments are passed into the call to tempfile.mk{,d}temp(),\n        and resultant temporary filename is passed as the first argument into\n        the function t.  If no 'prefix' argument is provided, it will be\n        constructed using module and function names ('.' replaced with\n        '_').\n\n    To change the used directory without providing keyword argument 'dir' set\n    DATALAD_TESTS_TEMP_DIR.\n\n    Examples\n    --------\n        >>> from os.path import exists\n        >>> from datalad.utils import make_tempfile\n        >>> with make_tempfile() as fname:\n        ...    k = open(fname, 'w').write('silly test')\n        >>> assert not exists(fname)  # was removed\n\n        >>> with make_tempfile(content=\"blah\") as fname:\n        ...    assert open(fname).read() == \"blah\"\n    \"\"\"\n\n    if tkwargs.get('mkdir', None) and content is not None:\n        raise ValueError(\"mkdir=True while providing content makes no sense\")\n\n    tkwargs_ = get_tempfile_kwargs(tkwargs, wrapped=wrapped)\n\n    # if DATALAD_TESTS_TEMP_DIR is set, use that as directory,\n    # let mktemp handle it otherwise. However, an explicitly provided\n    # dir=... will override this.\n    mkdir = tkwargs_.pop('mkdir', False)\n\n    filename = {False: tempfile.mktemp,\n                True: tempfile.mkdtemp}[mkdir](**tkwargs_)\n    filename = realpath(filename)\n\n    if content:\n        with open(filename, 'w' + ('b' if isinstance(content, binary_type) else '')) as f:\n            f.write(content)\n\n    if __debug__:\n        # TODO mkdir\n        lgr.debug('Created temporary thing named %s\"' % filename)\n    try:\n        yield filename\n    finally:\n        # glob here for all files with the same name (-suffix)\n        # would be useful whenever we requested .img filename,\n        # and function creates .hdr as well\n        lsuffix = len(tkwargs_.get('suffix', ''))\n        filename_ = lsuffix and filename[:-lsuffix] or filename\n        filenames = glob.glob(filename_ + '*')\n        if len(filename_) < 3 or len(filenames) > 5:\n            # For paranoid yoh who stepped into this already ones ;-)\n            lgr.warning(\"It is unlikely that it was intended to remove all\"\n                        \" files matching %r. Skipping\" % filename_)\n            return\n        for f in filenames:\n            try:\n                rmtemp(f)\n            except OSError:  # pragma: no cover\n                pass\n\n\ndef _path_(*p):\n    \"\"\"Given a path in POSIX\" notation, regenerate one in native to the env one\"\"\"\n    if on_windows:\n        return opj(*map(lambda x: opj(*x.split('/')), p))\n    else:\n        # Assume that all others as POSIX compliant so nothing to be done\n        return opj(*p)\n\n\ndef get_timestamp_suffix(time_=None, prefix='-'):\n    \"\"\"Return a time stamp (full date and time up to second)\n\n    primarily to be used for generation of log files names\n    \"\"\"\n    args = []\n    if time_ is not None:\n        if isinstance(time_, int):\n            time_ = time.gmtime(time_)\n        args.append(time_)\n    return time.strftime(prefix + TIMESTAMP_FMT, *args)\n\n\ndef get_logfilename(dspath, cmd='datalad'):\n    \"\"\"Return a filename to use for logging under a dataset/repository\n\n    directory would be created if doesn't exist, but dspath must exist\n    and be a directory\n    \"\"\"\n    assert(exists(dspath))\n    assert(isdir(dspath))\n    ds_logdir = assure_dir(dspath, '.git', 'datalad', 'logs')  # TODO: use WEB_META_LOG whenever #789 merged\n    return opj(ds_logdir, 'crawl-%s.log' % get_timestamp_suffix())\n\n\ndef get_trace(edges, start, end, trace=None):\n    \"\"\"Return the trace/path to reach a node in a tree.\n\n    Parameters\n    ----------\n    edges : sequence(2-tuple)\n      The tree given by a sequence of edges (parent, child) tuples. The\n      nodes can be identified by any value and data type that supports\n      the '==' operation.\n    start :\n      Identifier of the start node. Must be present as a value in the parent\n      location of an edge tuple in order to be found.\n    end :\n      Identifier of the target/end node. Must be present as a value in the child\n      location of an edge tuple in order to be found.\n    trace : list\n      Mostly useful for recursive calls, and used internally.\n\n    Returns\n    -------\n    None or list\n      Returns a list with the trace to the target (the starts and the target\n      are not included in the trace, hence if start and end are directly connected\n      an empty list is returned), or None when no trace to the target can be found,\n      or start and end are identical.\n    \"\"\"\n    # the term trace is used to avoid confusion with a path in the sense\n    # of a filesystem path, but the analogy fits and nodes can be paths\n    if trace is None:\n        trace = []\n    if not edges:\n        raise ValueError(\"no edges given\")\n    for cand in edges:\n        cand_super, cand_sub = cand\n        if cand_sub in trace:\n            # only DAGs, skip any cyclic traces\n            continue\n        if trace and cand_super != trace[-1]:\n            # only consider edges that lead off the end of the trace\n            continue\n        if not trace and cand_super != start:\n            # we got nothing yet, and this edges is not matching the start\n            continue\n        if cand_sub == end:\n            return trace\n        # dive into potential subnodes\n        cand_trace = get_trace(\n            edges,\n            start,\n            end,\n            trace + [cand_sub])\n        if cand_trace:\n            return cand_trace\n    return None\n\n\ndef get_dataset_root(path):\n    \"\"\"Return the root of an existent dataset containing a given path\n\n    The root path is returned in the same absolute or relative form\n    as the input argument. If no associated dataset exists, or the\n    input path doesn't exist, None is returned.\n    \"\"\"\n    suffix = '.git'\n    if not isdir(path):\n        path = dirname(path)\n    apath = abspath(path)\n    # while we can still go up\n    while psplit(apath)[1]:\n        if exists(opj(path, suffix)):\n            return path\n        # new test path in the format we got it\n        path = normpath(opj(path, os.pardir))\n        # no luck, next round\n        apath = abspath(path)\n    return None\n\n\ndef get_dataset_pwds(dataset):\n    \"\"\"Return the current directory for the dataset.\n\n    Parameters\n    ----------\n    dataset : Dataset\n\n    Returns\n    -------\n    A tuple, where the first item is the absolute path of the pwd and the\n    second is the pwd relative to the dataset's path.\n    \"\"\"\n    if dataset:\n        pwd = dataset.path\n        rel_pwd = curdir\n    else:\n        # act on the whole dataset if nothing else was specified\n\n        # Follow our generic semantic that if dataset is specified,\n        # paths are relative to it, if not -- relative to pwd\n        pwd = getpwd()\n        # Pass pwd to get_dataset_root instead of os.path.curdir to handle\n        # repos whose leading paths have a symlinked directory (see the\n        # TMPDIR=\"/var/tmp/sym link\" test case).\n        dataset = get_dataset_root(pwd)\n\n        if dataset:\n            rel_pwd = relpath(pwd, dataset)\n        else:\n            rel_pwd = pwd  # and leave handling to caller\n    return pwd, rel_pwd\n\n\n# ATM used in datalad_crawler extension, so do not remove yet\ndef try_multiple(ntrials, exception, base, f, *args, **kwargs):\n    \"\"\"Call f multiple times making exponentially growing delay between the calls\"\"\"\n    from .dochelpers import exc_str\n    for trial in range(1, ntrials+1):\n        try:\n            return f(*args, **kwargs)\n        except exception as exc:\n            if trial == ntrials:\n                raise  # just reraise on the last trial\n            t = base ** trial\n            lgr.warning(\"Caught %s on trial #%d. Sleeping %f and retrying\",\n                        exc_str(exc), trial, t)\n            sleep(t)\n\n\n@optional_args\ndef try_multiple_dec(f, ntrials=None, duration=0.1, exceptions=None, increment_type=None):\n    \"\"\"Decorator to try function multiple times.\n\n    Main purpose is to decorate functions dealing with removal of files/directories\n    and which might need a few seconds to work correctly on Windows which takes\n    its time to release files/directories.\n\n    Parameters\n    ----------\n    ntrials: int, optional\n    duration: float, optional\n      Seconds to sleep before retrying.\n    increment_type: {None, 'exponential'}\n      Note that if it is exponential, duration should typically be > 1.0\n      so it grows with higher power\n\n    \"\"\"\n    from .dochelpers import exc_str\n    if not exceptions:\n        exceptions = (OSError, WindowsError, PermissionError) \\\n            if on_windows else OSError\n    if not ntrials:\n        # Life goes fast on proper systems, no need to delay it much\n        ntrials = 50 if on_windows else 3\n\n    assert increment_type in {None, 'exponential'}\n\n    @wraps(f)\n    def wrapped(*args, **kwargs):\n        t = duration\n        for trial in range(ntrials):\n            try:\n                return f(*args, **kwargs)\n            except exceptions as exc:\n                if increment_type == 'exponential':\n                    t = duration ** (trial + 1)\n                lgr.log(\n                    5,\n                    \"Caught %s on trial #%d. Sleeping %f and retrying\",\n                    exc_str(exc), trial, t)\n                if trial < ntrials - 1:\n                    sleep(t)\n                else:\n                    raise\n\n    return wrapped\n\n\n@try_multiple_dec\ndef unlink(f):\n    \"\"\"'Robust' unlink.  Would try multiple times\n\n    On windows boxes there is evidence for a latency of more than a second\n    until a file is considered no longer \"in-use\".\n    WindowsError is not known on Linux, and if IOError or any other\n    exception\n    is thrown then if except statement has WindowsError in it -- NameError\n    also see gh-2533\n    \"\"\"\n    # Check for open files\n    assert_no_open_files(f)\n    return os.unlink(f)\n\n\n@try_multiple_dec\ndef _rmtree(*args, **kwargs):\n    \"\"\"Just a helper to decorate shutil.rmtree.\n\n    rmtree defined above does more and ideally should not itself be decorated\n    since a recursive definition and does checks for open files inside etc -\n    might be too runtime expensive\n    \"\"\"\n    return shutil.rmtree(*args, **kwargs)\n\n\ndef slash_join(base, extension):\n    \"\"\"Join two strings with a '/', avoiding duplicate slashes\n\n    If any of the strings is None the other is returned as is.\n    \"\"\"\n    if extension is None:\n        return base\n    if base is None:\n        return extension\n    return '/'.join(\n        (base.rstrip('/'),\n         extension.lstrip('/')))\n\n\ndef safe_print(s):\n    \"\"\"Print with protection against UTF-8 encoding errors\"\"\"\n    # A little bit of dance to be able to test this code\n    print_f = getattr(__builtin__, \"print\")\n    try:\n        print_f(s)\n    except UnicodeEncodeError:\n        # failed to encode so let's do encoding while ignoring errors\n        # to print at least something\n        # explicit `or ascii` since somehow on buildbot it seemed to return None\n        s = s.encode(getattr(sys.stdout, 'encoding', 'ascii') or 'ascii', errors='ignore') \\\n            if hasattr(s, 'encode') else s\n        print_f(s.decode())\n\n#\n# IO Helpers\n#\n\ndef open_r_encdetect(fname, readahead=1000):\n    \"\"\"Return a file object in read mode with auto-detected encoding\n\n    This is helpful when dealing with files of unknown encoding.\n\n    Parameters\n    ----------\n    readahead: int, optional\n      How many bytes to read for guessing the encoding type.  If\n      negative - full file will be read\n    \"\"\"\n    from chardet import detect\n    import io\n    # read some bytes from the file\n    with open(fname, 'rb') as f:\n        head = f.read(readahead)\n    enc = detect(head)\n    denc = enc.get('encoding', None)\n    lgr.debug(\"Auto-detected encoding %s for file %s (confidence: %s)\",\n              denc,\n              fname,\n              enc.get('confidence', 'unknown'))\n    return io.open(fname, encoding=denc)\n\n\ndef read_csv_lines(fname, dialect=None, readahead=16384, **kwargs):\n    \"\"\"A generator of dict records from a CSV/TSV\n\n    Automatically guesses the encoding for each record to convert to UTF-8\n\n    Parameters\n    ----------\n    fname: str\n      Filename\n    dialect: str, optional\n      Dialect to specify to csv.reader. If not specified -- guessed from\n      the file, if fails to guess, \"excel-tab\" is assumed\n    readahead: int, optional\n      How many bytes to read from the file to guess the type\n    **kwargs\n      Passed to `csv.reader`\n    \"\"\"\n    import csv\n    if dialect is None:\n        with open(fname) as tsvfile:\n            # add robustness, use a sniffer\n            try:\n                dialect = csv.Sniffer().sniff(tsvfile.read(readahead))\n            except Exception as exc:\n                from .dochelpers import exc_str\n                lgr.warning(\n                    'Could not determine file-format, assuming TSV: %s',\n                    exc_str(exc)\n                )\n                dialect = 'excel-tab'\n\n    kw = {} if PY2 else dict(encoding='utf-8')\n    with open(fname, 'rb' if PY2 else 'r', **kw) as tsvfile:\n        # csv.py doesn't do Unicode; encode temporarily as UTF-8:\n        csv_reader = csv.reader(\n            tsvfile,\n            dialect=dialect,\n            **kwargs\n        )\n        header = None\n        for row in csv_reader:\n            # decode UTF-8 back to Unicode, cell by cell:\n            row_unicode = map(assure_unicode, row)\n            if header is None:\n                header = list(row_unicode)\n            else:\n                yield dict(zip(header, row_unicode))\n\n\ndef import_modules(modnames, pkg, msg=\"Failed to import {module}\", log=lgr.debug):\n    \"\"\"Helper to import a list of modules without failing if N/A\n\n    Parameters\n    ----------\n    modnames: list of str\n      List of module names to import\n    pkg: str\n      Package under which to import\n    msg: str, optional\n      Message template for .format() to log at DEBUG level if import fails.\n      Keys {module} and {package} will be provided and ': {exception}' appended\n    log: callable, optional\n      Logger call to use for logging messages\n    \"\"\"\n    from importlib import import_module\n    _globals = globals()\n    mods_loaded = []\n    if pkg and not pkg in sys.modules:\n        # with python 3.5.1 (ok with 3.5.5) somehow kept running into\n        #  Failed to import dlsub1: Parent module 'dltestm1' not loaded\n        # while running the test. Preloading pkg resolved the issue\n        import_module(pkg)\n    for modname in modnames:\n        try:\n            _globals[modname] = mod = import_module(\n                '.{}'.format(modname),\n                pkg)\n            mods_loaded.append(mod)\n        except Exception as exc:\n            from datalad.dochelpers import exc_str\n            log((msg + ': {exception}').format(\n                module=modname, package=pkg, exception=exc_str(exc)))\n    return mods_loaded\n\n\ndef import_module_from_file(modpath, pkg=None, log=lgr.debug):\n    \"\"\"Import provided module given a path\n\n    TODO:\n    - RF/make use of it in pipeline.py which has similar logic\n    - join with import_modules above?\n\n    Parameters\n    ----------\n    pkg: module, optional\n       If provided, and modpath is under pkg.__path__, relative import will be\n       used\n    \"\"\"\n    assert(modpath.endswith('.py'))  # for now just for .py files\n\n    log(\"Importing %s\" % modpath)\n\n    modname = basename(modpath)[:-3]\n    relmodpath = None\n    if pkg:\n        for pkgpath in pkg.__path__:\n            if path_is_subpath(modpath, pkgpath):\n                # for now relying on having .py extension -- assertion above\n                relmodpath = '.' + relpath(modpath[:-3], pkgpath).replace(sep, '.')\n                break\n\n    try:\n        if relmodpath:\n            from importlib import import_module\n            mod = import_module(relmodpath, pkg.__name__)\n        else:\n            dirname_ = dirname(modpath)\n            try:\n                sys.path.insert(0, dirname_)\n                mod = __import__(modname, level=0)\n            finally:\n                if dirname_ in sys.path:\n                    sys.path.pop(sys.path.index(dirname_))\n                else:\n                    log(\"Expected path %s to be within sys.path, but it was gone!\" % dirname_)\n    except Exception as e:\n        from datalad.dochelpers import exc_str\n        raise RuntimeError(\n            \"Failed to import module from %s: %s\" % (modpath, exc_str(e)))\n\n    return mod\n\n\ndef get_encoding_info():\n    \"\"\"Return a dictionary with various encoding/locale information\"\"\"\n    import sys, locale\n    from collections import OrderedDict\n    return OrderedDict([\n        ('default', sys.getdefaultencoding()),\n        ('filesystem', sys.getfilesystemencoding()),\n        ('locale.prefered', locale.getpreferredencoding()),\n    ])\n\n\ndef get_envvars_info():\n    from collections import OrderedDict\n    envs = []\n    for var, val in os.environ.items():\n        if (\n                var.startswith('PYTHON') or\n                var.startswith('LC_') or\n                var.startswith('GIT_') or\n                var in ('LANG', 'LANGUAGE', 'PATH')\n        ):\n            envs.append((var, val))\n    return OrderedDict(envs)\n\n\n# This class is modified from Snakemake (v5.1.4)\nclass SequenceFormatter(string.Formatter):\n    \"\"\"string.Formatter subclass with special behavior for sequences.\n\n    This class delegates formatting of individual elements to another\n    formatter object. Non-list objects are formatted by calling the\n    delegate formatter's \"format_field\" method. List-like objects\n    (list, tuple, set, frozenset) are formatted by formatting each\n    element of the list according to the specified format spec using\n    the delegate formatter and then joining the resulting strings with\n    a separator (space by default).\n    \"\"\"\n\n    def __init__(self, separator=\" \", element_formatter=string.Formatter(),\n                 *args, **kwargs):\n        self.separator = separator\n        self.element_formatter = element_formatter\n\n    def format_element(self, elem, format_spec):\n        \"\"\"Format a single element\n\n        For sequences, this is called once for each element in a\n        sequence. For anything else, it is called on the entire\n        object. It is intended to be overridden in subclases.\n        \"\"\"\n        return self.element_formatter.format_field(elem, format_spec)\n\n    def format_field(self, value, format_spec):\n        if isinstance(value, (list, tuple, set, frozenset)):\n            return self.separator.join(self.format_element(v, format_spec)\n                                       for v in value)\n        else:\n            return self.format_element(value, format_spec)\n\n\n# TODO: eventually we might want to make use of attr module\nclass File(object):\n    \"\"\"Helper for a file entry in the create_tree/@with_tree\n\n    It allows to define additional settings for entries\n    \"\"\"\n    def __init__(self, name, executable=False):\n        \"\"\"\n\n        Parameters\n        ----------\n        name : str\n          Name of the file\n        executable: bool, optional\n          Make it executable\n        \"\"\"\n        self.name = name\n        self.executable = executable\n\n    def __str__(self):\n        return self.name\n\n\ndef create_tree_archive(path, name, load, overwrite=False, archives_leading_dir=True):\n    \"\"\"Given an archive `name`, create under `path` with specified `load` tree\n    \"\"\"\n    from datalad.support.archives import compress_files\n    dirname = file_basename(name)\n    full_dirname = op.join(path, dirname)\n    os.makedirs(full_dirname)\n    create_tree(full_dirname, load, archives_leading_dir=archives_leading_dir)\n    # create archive\n    if archives_leading_dir:\n        compress_files([dirname], name, path=path, overwrite=overwrite)\n    else:\n        compress_files(list(map(op.basename, glob.glob(opj(full_dirname, '*')))),\n                       opj(op.pardir, name),\n                       path=op.join(path, dirname),\n                       overwrite=overwrite)\n    # remove original tree\n    rmtree(full_dirname)\n\n\ndef create_tree(path, tree, archives_leading_dir=True, remove_existing=False):\n    \"\"\"Given a list of tuples (name, load) create such a tree\n\n    if load is a tuple itself -- that would create either a subtree or an archive\n    with that content and place it into the tree if name ends with .tar.gz\n    \"\"\"\n    lgr.log(5, \"Creating a tree under %s\", path)\n    if not op.exists(path):\n        os.makedirs(path)\n\n    if isinstance(tree, dict):\n        tree = tree.items()\n\n    for file_, load in tree:\n        if isinstance(file_, File):\n            executable = file_.executable\n            name = file_.name\n        else:\n            executable = False\n            name = file_\n        full_name = op.join(path, name)\n        if remove_existing and op.lexists(full_name):\n            rmtree(full_name, chmod_files=True)\n        if isinstance(load, (tuple, list, dict)):\n            if name.endswith('.tar.gz') or name.endswith('.tar') or name.endswith('.zip'):\n                create_tree_archive(\n                    path, name, load,\n                    archives_leading_dir=archives_leading_dir)\n            else:\n                create_tree(\n                    full_name, load,\n                    archives_leading_dir=archives_leading_dir,\n                    remove_existing=remove_existing)\n        else:\n            open_func = open\n            if full_name.endswith('.gz'):\n                open_func = gzip.open\n            with open_func(full_name, \"wb\") as f:\n                f.write(assure_bytes(load, 'utf-8'))\n        if executable:\n            os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)\n\n\nlgr.log(5, \"Done importing datalad.utils\")\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\nimport collections\nimport hashlib\nimport re\nimport six.moves.builtins as __builtin__\nimport time\n\nimport logging\nimport shutil\nimport os\nimport sys\nimport tempfile\nimport platform\nimport gc\nimport glob\nimport gzip\nimport string\nimport wrapt\n\nfrom copy import copy as shallow_copy\nfrom contextlib import contextmanager\nfrom functools import wraps\nfrom time import sleep\nfrom inspect import getargspec\nfrom itertools import tee\n\nimport os.path as op\nfrom os.path import sep as dirsep\nfrom os.path import commonprefix\nfrom os.path import curdir, basename, exists, realpath, islink, join as opj\nfrom os.path import isabs, normpath, expandvars, expanduser, abspath, sep\nfrom os.path import isdir\nfrom os.path import relpath\nfrom os.path import stat\nfrom os.path import dirname\nfrom os.path import split as psplit\nimport posixpath\n\n\nfrom six import PY2, text_type, binary_type, string_types\n\n# from datalad.dochelpers import get_docstring_split\nfrom datalad.consts import TIMESTAMP_FMT\n\n\nif PY2:\n    unicode_srctypes = string_types\nelse:\n    unicode_srctypes = string_types + (bytes,)\n\n\nlgr = logging.getLogger(\"datalad.utils\")\n\nlgr.log(5, \"Importing datalad.utils\")\n#\n# Some useful variables\n#\nplatform_system = platform.system().lower()\non_windows = platform_system == 'windows'\non_osx = platform_system == 'darwin'\non_linux = platform_system == 'linux'\non_msys_tainted_paths = on_windows \\\n                        and 'MSYS_NO_PATHCONV' not in os.environ \\\n                        and os.environ.get('MSYSTEM', '')[:4] in ('MSYS', 'MING')\ntry:\n    linux_distribution_name, linux_distribution_release \\\n        = platform.linux_distribution()[:2]\n    on_debian_wheezy = on_linux \\\n                       and linux_distribution_name == 'debian' \\\n                       and linux_distribution_release.startswith('7.')\nexcept:  # pragma: no cover\n    # MIH: IndexError?\n    on_debian_wheezy = False\n    linux_distribution_name = linux_distribution_release = None\n\n# Maximal length of cmdline string\n# Query the system and use hardcoded \"knowledge\" if None\n# probably   getconf ARG_MAX   might not be available\n# The last one would be the most conservative/Windows\nCMD_MAX_ARG_HARDCODED = 2097152 if on_linux else 262144 if on_osx else 32767\ntry:\n    CMD_MAX_ARG = os.sysconf('SC_ARG_MAX')\n    assert CMD_MAX_ARG > 0\nexcept Exception as exc:\n    # ATM (20181005) SC_ARG_MAX available only on POSIX systems\n    # so exception would be thrown e.g. on Windows, or\n    # somehow during Debian build for nd14.04 it is coming up with -1:\n    # https://github.com/datalad/datalad/issues/3015\n    CMD_MAX_ARG = CMD_MAX_ARG_HARDCODED\n    lgr.debug(\n        \"Failed to query or got useless SC_ARG_MAX sysconf, \"\n        \"will use hardcoded value: %s\", exc)\n# Even with all careful computations we do, due to necessity to account for\n# environment and what not, we still could not figure out \"exact\" way to\n# estimate it, but it was shown that 300k safety margin on linux was sufficient.\n# https://github.com/datalad/datalad/pull/2977#issuecomment-436264710\n# 300k is ~15%, so to be safe, and for paranoid us we will just use up to 50%\n# of the length for \"safety margin\".  We might probably still blow due to\n# env vars, unicode, etc...  so any hard limit imho is not a proper solution\nCMD_MAX_ARG = int(0.5 * CMD_MAX_ARG)\nlgr.debug(\n    \"Maximal length of cmdline string (adjusted for safety margin): %d\",\n    CMD_MAX_ARG)\n\n#\n# Little helpers\n#\n\n\ndef get_func_kwargs_doc(func):\n    \"\"\" Provides args for a function\n\n    Parameters\n    ----------\n    func: str\n      name of the function from which args are being requested\n\n    Returns\n    -------\n    list\n      of the args that a function takes in\n    \"\"\"\n    return getargspec(func)[0]\n\n    # TODO: format error message with descriptions of args\n    # return [repr(dict(get_docstring_split(func)[1]).get(x)) for x in getargspec(func)[0]]\n\n\ndef any_re_search(regexes, value):\n    \"\"\"Return if any of regexes (list or str) searches succesfully for value\"\"\"\n    for regex in assure_tuple_or_list(regexes):\n        if re.search(regex, value):\n            return True\n    return False\n\n\ndef not_supported_on_windows(msg=None):\n    \"\"\"A little helper to be invoked to consistently fail whenever functionality is\n    not supported (yet) on Windows\n    \"\"\"\n    if on_windows:\n        raise NotImplementedError(\"This functionality is not yet implemented for Windows OS\"\n                                  + (\": %s\" % msg if msg else \"\"))\n\n\ndef shortened_repr(value, l=30):\n    try:\n        if hasattr(value, '__repr__') and (value.__repr__ is not object.__repr__):\n            value_repr = repr(value)\n            if not value_repr.startswith('<') and len(value_repr) > l:\n                value_repr = \"<<%s...>>\" % (value_repr[:l - 8])\n            elif value_repr.startswith('<') and value_repr.endswith('>') and ' object at 0x':\n                raise ValueError(\"I hate those useless long reprs\")\n        else:\n            raise ValueError(\"gimme class\")\n    except Exception as e:\n        value_repr = \"<%s>\" % value.__class__.__name__.split('.')[-1]\n    return value_repr\n\n\ndef __auto_repr__(obj):\n    attr_names = tuple()\n    if hasattr(obj, '__dict__'):\n        attr_names += tuple(obj.__dict__.keys())\n    if hasattr(obj, '__slots__'):\n        attr_names += tuple(obj.__slots__)\n\n    items = []\n    for attr in sorted(set(attr_names)):\n        if attr.startswith('_'):\n            continue\n        value = getattr(obj, attr)\n        # TODO:  should we add this feature to minimize some talktative reprs\n        # such as of URL?\n        #if value is None:\n        #    continue\n        items.append(\"%s=%s\" % (attr, shortened_repr(value)))\n\n    return \"%s(%s)\" % (obj.__class__.__name__, ', '.join(items))\n\n\ndef auto_repr(cls):\n    \"\"\"Decorator for a class to assign it an automagic quick and dirty __repr__\n\n    It uses public class attributes to prepare repr of a class\n\n    Original idea: http://stackoverflow.com/a/27799004/1265472\n    \"\"\"\n\n    cls.__repr__ = __auto_repr__\n    return cls\n\n\ndef is_interactive():\n    \"\"\"Return True if all in/outs are tty\"\"\"\n    # TODO: check on windows if hasattr check would work correctly and add value:\n    #\n    return sys.stdin.isatty() and sys.stdout.isatty() and sys.stderr.isatty()\n\n\ndef get_ipython_shell():\n    \"\"\"Detect if running within IPython and returns its `ip` (shell) object\n\n    Returns None if not under ipython (no `get_ipython` function)\n    \"\"\"\n    try:\n        return get_ipython()\n    except NameError:\n        return None\n\n\ndef md5sum(filename):\n    with open(filename, 'rb') as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef sorted_files(dout):\n    \"\"\"Return a (sorted) list of files under dout\n    \"\"\"\n    return sorted(sum([[opj(r, f)[len(dout) + 1:] for f in files]\n                       for r, d, files in os.walk(dout)\n                       if not '.git' in r], []))\n\n_VCS_REGEX = '%s\\.(?:git|gitattributes|svn|bzr|hg)(?:%s|$)' % (dirsep, dirsep)\n_DATALAD_REGEX = '%s\\.(?:datalad)(?:%s|$)' % (dirsep, dirsep)\n\n\ndef find_files(regex, topdir=curdir, exclude=None, exclude_vcs=True, exclude_datalad=False, dirs=False):\n    \"\"\"Generator to find files matching regex\n\n    Parameters\n    ----------\n    regex: basestring\n    exclude: basestring, optional\n      Matches to exclude\n    exclude_vcs:\n      If True, excludes commonly known VCS subdirectories.  If string, used\n      as regex to exclude those files (regex: `%r`)\n    exclude_datalad:\n      If True, excludes files known to be datalad meta-data files (e.g. under\n      .datalad/ subdirectory) (regex: `%r`)\n    topdir: basestring, optional\n      Directory where to search\n    dirs: bool, optional\n      Whether to match directories as well as files\n    \"\"\"\n\n    for dirpath, dirnames, filenames in os.walk(topdir):\n        names = (dirnames + filenames) if dirs else filenames\n        # TODO: might want to uniformize on windows to use '/'\n        paths = (opj(dirpath, name) for name in names)\n        for path in filter(re.compile(regex).search, paths):\n            path = path.rstrip(dirsep)\n            if exclude and re.search(exclude, path):\n                continue\n            if exclude_vcs and re.search(_VCS_REGEX, path):\n                continue\n            if exclude_datalad and re.search(_DATALAD_REGEX, path):\n                continue\n            yield path\nfind_files.__doc__ %= (_VCS_REGEX, _DATALAD_REGEX)\n\n\ndef expandpath(path, force_absolute=True):\n    \"\"\"Expand all variables and user handles in a path.\n\n    By default return an absolute path\n    \"\"\"\n    path = expandvars(expanduser(path))\n    if force_absolute:\n        path = abspath(path)\n    return path\n\n\ndef posix_relpath(path, start=None):\n    \"\"\"Behave like os.path.relpath, but always return POSIX paths...\n\n    on any platform.\"\"\"\n    # join POSIX style\n    return posixpath.join(\n        # split and relpath native style\n        # python2.7 ntpath implementation of relpath cannot handle start=None\n        *psplit(\n            relpath(path, start=start if start is not None else '')))\n\n\ndef is_explicit_path(path):\n    \"\"\"Return whether a path explicitly points to a location\n\n    Any absolute path, or relative path starting with either '../' or\n    './' is assumed to indicate a location on the filesystem. Any other\n    path format is not considered explicit.\"\"\"\n    path = expandpath(path, force_absolute=False)\n    return isabs(path) \\\n        or path.startswith(os.curdir + os.sep) \\\n        or path.startswith(os.pardir + os.sep)\n\n\ndef rotree(path, ro=True, chmod_files=True):\n    \"\"\"To make tree read-only or writable\n\n    Parameters\n    ----------\n    path : string\n      Path to the tree/directory to chmod\n    ro : bool, optional\n      Whether to make it R/O (default) or RW\n    chmod_files : bool, optional\n      Whether to operate also on files (not just directories)\n    \"\"\"\n    if ro:\n        chmod = lambda f: os.chmod(f, os.stat(f).st_mode & ~stat.S_IWRITE)\n    else:\n        chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)\n\n    for root, dirs, files in os.walk(path, followlinks=False):\n        if chmod_files:\n            for f in files:\n                fullf = opj(root, f)\n                # might be the \"broken\" symlink which would fail to stat etc\n                if exists(fullf):\n                    chmod(fullf)\n        chmod(root)\n\n\ndef rmtree(path, chmod_files='auto', children_only=False, *args, **kwargs):\n    \"\"\"To remove git-annex .git it is needed to make all files and directories writable again first\n\n    Parameters\n    ----------\n    chmod_files : string or bool, optional\n       Whether to make files writable also before removal.  Usually it is just\n       a matter of directories to have write permissions.\n       If 'auto' it would chmod files on windows by default\n    children_only : bool, optional\n       If set, all files and subdirectories would be removed while the path\n       itself (must be a directory) would be preserved\n    `*args` :\n    `**kwargs` :\n       Passed into shutil.rmtree call\n    \"\"\"\n    # Give W permissions back only to directories, no need to bother with files\n    if chmod_files == 'auto':\n        chmod_files = on_windows\n    # TODO:  yoh thinks that if we could quickly check our Flyweight for\n    #        repos if any of them is under the path, and could call .precommit\n    #        on those to possibly stop batched processes etc, we did not have\n    #        to do it on case by case\n    # Check for open files\n    assert_no_open_files(path)\n\n    if children_only:\n        if not os.path.isdir(path):\n            raise ValueError(\"Can remove children only of directories\")\n        for p in os.listdir(path):\n            rmtree(op.join(path, p))\n        return\n    if not (os.path.islink(path) or not os.path.isdir(path)):\n        rotree(path, ro=False, chmod_files=chmod_files)\n        _rmtree(path, *args, **kwargs)\n    else:\n        # just remove the symlink\n        unlink(path)\n\n\ndef rmdir(path, *args, **kwargs):\n    \"\"\"os.rmdir with our optional checking for open files\"\"\"\n    assert_no_open_files(path)\n    os.rmdir(path)\n\n\ndef get_open_files(path, log_open=False):\n    \"\"\"Get open files under a path\n\n    Parameters\n    ----------\n    path : str\n      File or directory to check for open files under\n    log_open : bool or int\n      If set - logger level to use\n\n    Returns\n    -------\n    dict\n      path : pid\n\n    \"\"\"\n    # Original idea: https://stackoverflow.com/a/11115521/1265472\n    import psutil\n    files = {}\n    # since the ones returned by psutil would not be aware of symlinks in the\n    # path we should also get realpath for path\n    path = realpath(path)\n    for proc in psutil.process_iter():\n        try:\n            open_paths = [p.path for p in proc.open_files()] + [proc.cwd()]\n            for p in open_paths:\n                # note: could be done more efficiently so we do not\n                # renormalize path over and over again etc\n                if path_startswith(p, path):\n                    files[p] = proc.pid\n        # Catch a race condition where a process ends\n        # before we can examine its files\n        except psutil.NoSuchProcess:\n            pass\n        except psutil.AccessDenied:\n            pass\n\n    if files and log_open:\n        lgr.log(log_open, \"Open files under %s: %s\", path, files)\n    return files\n\n\n_assert_no_open_files_cfg = os.environ.get('DATALAD_ASSERT_NO_OPEN_FILES')\nif _assert_no_open_files_cfg:\n    def assert_no_open_files(path):\n        files = get_open_files(path, log_open=40)\n        if _assert_no_open_files_cfg == 'assert':\n            assert not files\n        elif files:\n            if _assert_no_open_files_cfg == 'pdb':\n                import pdb\n                pdb.set_trace()\n            elif _assert_no_open_files_cfg == 'epdb':\n                import epdb\n                epdb.serve()\n            pass\n        # otherwise we would just issue that error message in the log\nelse:\n    def assert_no_open_files(*args, **kwargs):\n        pass\n\n\ndef rmtemp(f, *args, **kwargs):\n    \"\"\"Wrapper to centralize removing of temp files so we could keep them around\n\n    It will not remove the temporary file/directory if DATALAD_TESTS_TEMP_KEEP\n    environment variable is defined\n    \"\"\"\n    if not os.environ.get('DATALAD_TESTS_TEMP_KEEP'):\n        if not os.path.lexists(f):\n            lgr.debug(\"Path %s does not exist, so can't be removed\" % f)\n            return\n        lgr.log(5, \"Removing temp file: %s\" % f)\n        # Can also be a directory\n        if os.path.isdir(f):\n            rmtree(f, *args, **kwargs)\n        else:\n            unlink(f)\n    else:\n        lgr.info(\"Keeping temp file: %s\" % f)\n\n\ndef file_basename(name, return_ext=False):\n    \"\"\"\n    Strips up to 2 extensions of length up to 4 characters and starting with alpha\n    not a digit, so we could get rid of .tar.gz etc\n    \"\"\"\n    bname = basename(name)\n    fbname = re.sub('(\\.[a-zA-Z_]\\S{1,4}){0,2}$', '', bname)\n    if return_ext:\n        return fbname, bname[len(fbname) + 1:]\n    else:\n        return fbname\n\n\ndef escape_filename(filename):\n    \"\"\"Surround filename in \"\" and escape \" in the filename\n    \"\"\"\n    filename = filename.replace('\"', r'\\\"').replace('`', r'\\`')\n    filename = '\"%s\"' % filename\n    return filename\n\n\ndef encode_filename(filename):\n    \"\"\"Encode unicode filename\n    \"\"\"\n    if isinstance(filename, text_type):\n        return filename.encode(sys.getfilesystemencoding())\n    else:\n        return filename\n\n\ndef decode_input(s):\n    \"\"\"Given input string/bytes, decode according to stdin codepage (or UTF-8)\n    if not defined\n\n    If fails -- issue warning and decode allowing for errors\n    being replaced\n    \"\"\"\n    if isinstance(s, text_type):\n        return s\n    else:\n        encoding = sys.stdin.encoding or 'UTF-8'\n        try:\n            return s.decode(encoding)\n        except UnicodeDecodeError as exc:\n            lgr.warning(\n                \"Failed to decode input string using %s encoding. \"\n                \"Decoding allowing for errors\", encoding)\n            return s.decode(encoding, errors='replace')\n\n\nif on_windows:\n    def lmtime(filepath, mtime):\n        \"\"\"Set mtime for files.  On Windows a merely adapter to os.utime\n        \"\"\"\n        os.utime(filepath, (time.time(), mtime))\nelse:\n    def lmtime(filepath, mtime):\n        \"\"\"Set mtime for files, while not de-referencing symlinks.\n\n        To overcome absence of os.lutime\n\n        Works only on linux and OSX ATM\n        \"\"\"\n        from .cmd import Runner\n        # convert mtime to format touch understands [[CC]YY]MMDDhhmm[.SS]\n        smtime = time.strftime(\"%Y%m%d%H%M.%S\", time.localtime(mtime))\n        lgr.log(3, \"Setting mtime for %s to %s == %s\", filepath, mtime, smtime)\n        Runner().run(['touch', '-h', '-t', '%s' % smtime, filepath])\n        rfilepath = realpath(filepath)\n        if islink(filepath) and exists(rfilepath):\n            # trust noone - adjust also of the target file\n            # since it seemed like downloading under OSX (was it using curl?)\n            # didn't bother with timestamps\n            lgr.log(3, \"File is a symlink to %s Setting mtime for it to %s\",\n                    rfilepath, mtime)\n            os.utime(rfilepath, (time.time(), mtime))\n        # doesn't work on OSX\n        # Runner().run(['touch', '-h', '-d', '@%s' % mtime, filepath])\n\n\ndef assure_tuple_or_list(obj):\n    \"\"\"Given an object, wrap into a tuple if not list or tuple\n    \"\"\"\n    if isinstance(obj, (list, tuple)):\n        return obj\n    return (obj,)\n\n\ndef assure_iter(s, cls, copy=False, iterate=True):\n    \"\"\"Given not a list, would place it into a list. If None - empty list is returned\n\n    Parameters\n    ----------\n    s: list or anything\n    cls: class\n      Which iterable class to assure\n    copy: bool, optional\n      If correct iterable is passed, it would generate its shallow copy\n    iterate: bool, optional\n      If it is not a list, but something iterable (but not a text_type)\n      iterate over it.\n    \"\"\"\n\n    if isinstance(s, cls):\n        return s if not copy else shallow_copy(s)\n    elif isinstance(s, text_type):\n        return cls((s,))\n    elif iterate and hasattr(s, '__iter__'):\n        return cls(s)\n    elif s is None:\n        return cls()\n    else:\n        return cls((s,))\n\n\ndef assure_list(s, copy=False, iterate=True):\n    \"\"\"Given not a list, would place it into a list. If None - empty list is returned\n\n    Parameters\n    ----------\n    s: list or anything\n    copy: bool, optional\n      If list is passed, it would generate a shallow copy of the list\n    iterate: bool, optional\n      If it is not a list, but something iterable (but not a text_type)\n      iterate over it.\n    \"\"\"\n    return assure_iter(s, list, copy=copy, iterate=iterate)\n\n\ndef assure_list_from_str(s, sep='\\n'):\n    \"\"\"Given a multiline string convert it to a list of return None if empty\n\n    Parameters\n    ----------\n    s: str or list\n    \"\"\"\n\n    if not s:\n        return None\n\n    if isinstance(s, list):\n        return s\n    return s.split(sep)\n\n\ndef assure_dict_from_str(s, **kwargs):\n    \"\"\"Given a multiline string with key=value items convert it to a dictionary\n\n    Parameters\n    ----------\n    s: str or dict\n\n    Returns None if input s is empty\n    \"\"\"\n\n    if not s:\n        return None\n\n    if isinstance(s, dict):\n        return s\n\n    out = {}\n    for value_str in assure_list_from_str(s, **kwargs):\n        if '=' not in value_str:\n            raise ValueError(\"{} is not in key=value format\".format(repr(value_str)))\n        k, v = value_str.split('=', 1)\n        if k in out:\n            err = \"key {} was already defined in {}, but new value {} was provided\".format(k, out, v)\n            raise ValueError(err)\n        out[k] = v\n    return out\n\n\ndef assure_bytes(s, encoding='utf-8'):\n    \"\"\"Convert/encode unicode to str (PY2) or bytes (PY3) if of 'text_type'\n\n    Parameters\n    ----------\n    encoding: str, optional\n      Encoding to use.  \"utf-8\" is the default\n    \"\"\"\n    if not isinstance(s, text_type):\n        return s\n    return s.encode(encoding)\n\n\ndef assure_unicode(s, encoding=None, confidence=None):\n    \"\"\"Convert/decode to unicode (PY2) or str (PY3) if of 'binary_type'\n\n    Parameters\n    ----------\n    encoding: str, optional\n      Encoding to use.  If None, \"utf-8\" is tried, and then if not a valid\n      UTF-8, encoding will be guessed\n    confidence: float, optional\n      A value between 0 and 1, so if guessing of encoding is of lower than\n      specified confidence, ValueError is raised\n    \"\"\"\n    if not isinstance(s, binary_type):\n        return s\n    if encoding is None:\n        # Figure out encoding, defaulting to 'utf-8' which is our common\n        # target in contemporary digital society\n        try:\n            return s.decode('utf-8')\n        except UnicodeDecodeError as exc:\n            from .dochelpers import exc_str\n            lgr.debug(\"Failed to decode a string as utf-8: %s\", exc_str(exc))\n        # And now we could try to guess\n        from chardet import detect\n        enc = detect(s)\n        denc = enc.get('encoding', None)\n        if denc:\n            denc_confidence = enc.get('confidence', 0)\n            if confidence is not None and  denc_confidence < confidence:\n                raise ValueError(\n                    \"Failed to auto-detect encoding with high enough \"\n                    \"confidence. Highest confidence was %s for %s\"\n                    % (denc_confidence, denc)\n                )\n            return s.decode(denc)\n        else:\n            raise ValueError(\n                \"Could not decode value as utf-8, or to guess its encoding: %s\"\n                % repr(s)\n            )\n    else:\n        return s.decode(encoding)\n\n\ndef assure_bool(s):\n    \"\"\"Convert value into boolean following convention for strings\n\n    to recognize on,True,yes as True, off,False,no as False\n    \"\"\"\n    if isinstance(s, string_types):\n        if s.isdigit():\n            return bool(int(s))\n        sl = s.lower()\n        if sl in {'y', 'yes', 'true', 'on'}:\n            return True\n        elif sl in {'n', 'no', 'false', 'off'}:\n            return False\n        else:\n            raise ValueError(\"Do not know how to treat %r as a boolean\" % s)\n    return bool(s)\n\n\ndef as_unicode(val, cast_types=object):\n    \"\"\"Given an arbitrary value, would try to obtain unicode value of it\n    \n    For unicode it would return original value, for python2 str or python3\n    bytes it would use assure_unicode, for None - an empty (unicode) string,\n    and for any other type (see `cast_types`) - would apply the unicode \n    constructor.  If value is not an instance of `cast_types`, TypeError\n    is thrown\n    \n    Parameters\n    ----------\n    cast_types: type\n      Which types to cast to unicode by providing to constructor\n    \"\"\"\n    if val is None:\n        return u''\n    elif isinstance(val, text_type):\n        return val\n    elif isinstance(val, unicode_srctypes):\n        return assure_unicode(val)\n    elif isinstance(val, cast_types):\n        return text_type(val)\n    else:\n        raise TypeError(\n            \"Value %r is not of any of known or provided %s types\"\n            % (val, cast_types))\n\n\ndef unique(seq, key=None):\n    \"\"\"Given a sequence return a list only with unique elements while maintaining order\n\n    This is the fastest solution.  See\n    https://www.peterbe.com/plog/uniqifiers-benchmark\n    and\n    http://stackoverflow.com/a/480227/1265472\n    for more information.\n    Enhancement -- added ability to compare for uniqueness using a key function\n\n    Parameters\n    ----------\n    seq:\n      Sequence to analyze\n    key: callable, optional\n      Function to call on each element so we could decide not on a full\n      element, but on its member etc\n    \"\"\"\n    seen = set()\n    seen_add = seen.add\n    if not key:\n        return [x for x in seq if not (x in seen or seen_add(x))]\n    else:\n        # OPT: could be optimized, since key is called twice, but for our cases\n        # should be just as fine\n        return [x for x in seq if not (key(x) in seen or seen_add(key(x)))]\n\n\ndef all_same(items):\n    \"\"\"Quick check if all items are the same.\n\n    Identical to a check like len(set(items)) == 1 but\n    should be more efficient while working on generators, since would\n    return False as soon as any difference detected thus possibly avoiding\n    unnecessary evaluations\n    \"\"\"\n    first = True\n    first_item = None\n    for item in items:\n        if first:\n            first = False\n            first_item = item\n        else:\n            if item != first_item:\n                return False\n    # So we return False if was empty\n    return not first\n\n\ndef map_items(func, v):\n    \"\"\"A helper to apply `func` to all elements (keys and values) within dict\n\n    No type checking of values passed to func is done, so `func`\n    should be resilient to values which it should not handle\n\n    Initial usecase - apply_recursive(url_fragment, assure_unicode)\n    \"\"\"\n    # map all elements within item\n    return v.__class__(\n        item.__class__(map(func, item))\n        for item in v.items()\n    )\n\n\ndef partition(items, predicate=bool):\n    \"\"\"Partition `items` by `predicate`.\n\n    Parameters\n    ----------\n    items : iterable\n    predicate : callable\n        A function that will be mapped over each element in `items`. The\n        elements will partitioned based on whether the return value is false or\n        true.\n\n    Returns\n    -------\n    A tuple with two generators, the first for 'false' items and the second for\n    'true' ones.\n\n    Notes\n    -----\n    Taken from Peter Otten's snippet posted at\n    https://nedbatchelder.com/blog/201306/filter_a_list_into_two_parts.html\n    \"\"\"\n    a, b = tee((predicate(item), item) for item in items)\n    return ((item for pred, item in a if not pred),\n            (item for pred, item in b if pred))\n\n\ndef generate_chunks(container, size):\n    \"\"\"Given a container, generate chunks from it with size up to `size`\n    \"\"\"\n    # There could be a \"smarter\" solution but I think this would suffice\n    assert size > 0,  \"Size should be non-0 positive\"\n    while container:\n        yield container[:size]\n        container = container[size:]\n\n#\n# Generators helpers\n#\n\ndef saved_generator(gen):\n    \"\"\"Given a generator returns two generators, where 2nd one just replays\n\n    So the first one would be going through the generated items and 2nd one\n    would be yielding saved items\n    \"\"\"\n    saved = []\n\n    def gen1():\n        for x in gen:  # iterating over original generator\n            saved.append(x)\n            yield x\n\n    def gen2():\n        for x in saved:  # yielding saved entries\n            yield x\n\n    return gen1(), gen2()\n\n\n#\n# Decorators\n#\ndef better_wraps(to_be_wrapped):\n    \"\"\"Decorator to replace `functools.wraps`\n\n    This is based on `wrapt` instead of `functools` and in opposition to `wraps`\n    preserves the correct signature of the decorated function.\n    It is written with the intention to replace the use of `wraps` without any\n    need to rewrite the actual decorators.\n    \"\"\"\n\n    @wrapt.decorator(adapter=to_be_wrapped)\n    def intermediator(to_be_wrapper, instance, args, kwargs):\n        return to_be_wrapper(*args, **kwargs)\n\n    return intermediator\n\n\n# Borrowed from pandas\n# Copyright: 2011-2014, Lambda Foundry, Inc. and PyData Development Team\n# License: BSD-3\ndef optional_args(decorator):\n    \"\"\"allows a decorator to take optional positional and keyword arguments.\n        Assumes that taking a single, callable, positional argument means that\n        it is decorating a function, i.e. something like this::\n\n            @my_decorator\n            def function(): pass\n\n        Calls decorator with decorator(f, `*args`, `**kwargs`)\"\"\"\n\n    @better_wraps(decorator)\n    def wrapper(*args, **kwargs):\n        def dec(f):\n            return decorator(f, *args, **kwargs)\n\n        is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable)\n        if is_decorating:\n            f = args[0]\n            args = []\n            return dec(f)\n        else:\n            return dec\n\n    return wrapper\n\n\n# TODO: just provide decorators for tempfile.mk* functions. This is ugly!\ndef get_tempfile_kwargs(tkwargs=None, prefix=\"\", wrapped=None):\n    \"\"\"Updates kwargs to be passed to tempfile. calls depending on env vars\n    \"\"\"\n    if tkwargs is None:\n        tkwargs_ = {}\n    else:\n        # operate on a copy of tkwargs to avoid any side-effects\n        tkwargs_ = tkwargs.copy()\n\n    # TODO: don't remember why I had this one originally\n    # if len(targs)<2 and \\\n    if 'prefix' not in tkwargs_:\n        tkwargs_['prefix'] = '_'.join(\n            ['datalad_temp'] +\n            ([prefix] if prefix else []) +\n            ([''] if (on_windows or not wrapped) else [wrapped.__name__]))\n\n    directory = os.environ.get('DATALAD_TESTS_TEMP_DIR')\n    if directory and 'dir' not in tkwargs_:\n        tkwargs_['dir'] = directory\n\n    return tkwargs_\n\n\n@optional_args\ndef line_profile(func):\n    \"\"\"Q&D helper to line profile the function and spit out stats\n    \"\"\"\n    import line_profiler\n    prof = line_profiler.LineProfiler()\n\n    @wraps(func)\n    def newfunc(*args, **kwargs):\n        try:\n            pfunc = prof(func)\n            return pfunc(*args, **kwargs)\n        finally:\n            prof.print_stats()\n    return newfunc\n\n#\n# Context Managers\n#\n\n\n@contextmanager\ndef nothing_cm():\n    \"\"\"Just a dummy cm to programmically switch context managers\"\"\"\n    yield\n\n\n@contextmanager\ndef swallow_outputs():\n    \"\"\"Context manager to help consuming both stdout and stderr, and print()\n\n    stdout is available as cm.out and stderr as cm.err whenever cm is the\n    yielded context manager.\n    Internally uses temporary files to guarantee absent side-effects of swallowing\n    into StringIO which lacks .fileno.\n\n    print mocking is necessary for some uses where sys.stdout was already bound\n    to original sys.stdout, thus mocking it later had no effect. Overriding\n    print function had desired effect\n    \"\"\"\n\n    class StringIOAdapter(object):\n        \"\"\"Little adapter to help getting out/err values\n        \"\"\"\n        def __init__(self):\n            kw = get_tempfile_kwargs({}, prefix=\"outputs\")\n\n            self._out = open(tempfile.mktemp(**kw), 'w')\n            self._err = open(tempfile.mktemp(**kw), 'w')\n\n        def _read(self, h):\n            with open(h.name) as f:\n                return f.read()\n\n        @property\n        def out(self):\n            if not self._out.closed:\n                self._out.flush()\n            return self._read(self._out)\n\n        @property\n        def err(self):\n            if not self._err.closed:\n                self._err.flush()\n            return self._read(self._err)\n\n        @property\n        def handles(self):\n            return self._out, self._err\n\n        def cleanup(self):\n            self._out.close()\n            self._err.close()\n            out_name = self._out.name\n            err_name = self._err.name\n            del self._out\n            del self._err\n            gc.collect()\n            rmtemp(out_name)\n            rmtemp(err_name)\n\n    def fake_print(*args, **kwargs):\n        sep = kwargs.pop('sep', ' ')\n        end = kwargs.pop('end', '\\n')\n        file = kwargs.pop('file', sys.stdout)\n\n        if file in (oldout, olderr, sys.stdout, sys.stderr):\n            # we mock\n            try:\n                sys.stdout.write(sep.join(args) + end)\n            except UnicodeEncodeError as exc:\n                lgr.error(\n                    \"Failed to write to mocked stdout, got %s, continue as it \"\n                    \"didn't happen\",  exc)\n        else:\n            # must be some other file one -- leave it alone\n            oldprint(*args, sep=sep, end=end, file=file)\n\n    from .ui import ui\n    # preserve -- they could have been mocked already\n    oldprint = getattr(__builtin__, 'print')\n    oldout, olderr = sys.stdout, sys.stderr\n    olduiout = ui.out\n    adapter = StringIOAdapter()\n\n    try:\n        sys.stdout, sys.stderr = adapter.handles\n        ui.out = adapter.handles[0]\n        setattr(__builtin__, 'print', fake_print)\n\n        yield adapter\n    finally:\n        sys.stdout, sys.stderr, ui.out = oldout, olderr, olduiout\n        setattr(__builtin__, 'print',  oldprint)\n        adapter.cleanup()\n\n\n@contextmanager\ndef swallow_logs(new_level=None, file_=None, name='datalad'):\n    \"\"\"Context manager to consume all logs.\n\n    \"\"\"\n    lgr = logging.getLogger(name)\n\n    # Keep old settings\n    old_level = lgr.level\n    old_handlers = lgr.handlers\n\n    # Let's log everything into a string\n    # TODO: generalize with the one for swallow_outputs\n    class StringIOAdapter(object):\n        \"\"\"Little adapter to help getting out values\n\n        And to stay consistent with how swallow_outputs behaves\n        \"\"\"\n        def __init__(self):\n            if file_ is None:\n                kw = get_tempfile_kwargs({}, prefix=\"logs\")\n                out_file = tempfile.mktemp(**kw)\n            else:\n                out_file = file_\n            # PY3 requires clearly one or another.  race condition possible\n            self._out = open(out_file, 'a')\n            self._final_out = None\n\n        def _read(self, h):\n            with open(h.name) as f:\n                return f.read()\n\n        @property\n        def out(self):\n            if self._final_out is not None:\n                # we closed and cleaned up already\n                return self._final_out\n            else:\n                self._out.flush()\n                return self._read(self._out)\n\n        @property\n        def lines(self):\n            return self.out.split('\\n')\n\n        @property\n        def handle(self):\n            return self._out\n\n        def cleanup(self):\n            # store for access while object exists\n            self._final_out = self.out\n            self._out.close()\n            out_name = self._out.name\n            del self._out\n            gc.collect()\n            if not file_:\n                rmtemp(out_name)\n\n        def assert_logged(self, msg=None, level=None, regex=True, **kwargs):\n            \"\"\"Provide assertion on whether a msg was logged at a given level\n\n            If neither `msg` nor `level` provided, checks if anything was logged\n            at all.\n\n            Parameters\n            ----------\n            msg: str, optional\n              Message (as a regular expression, if `regex`) to be searched.\n              If no msg provided, checks if anything was logged at a given level.\n            level: str, optional\n              String representing the level to be logged\n            regex: bool, optional\n              If False, regular `assert_in` is used\n            **kwargs: str, optional\n              Passed to `assert_re_in` or `assert_in`\n            \"\"\"\n            from datalad.tests.utils import assert_re_in\n            from datalad.tests.utils import assert_in\n\n            if regex:\n                match = '\\[%s\\] ' % level if level else \"\\[\\S+\\] \"\n            else:\n                match = '[%s] ' % level if level else ''\n\n            if msg:\n                match += msg\n\n            if match:\n                (assert_re_in if regex else assert_in)(match, self.out, **kwargs)\n            else:\n                assert not kwargs, \"no kwargs to be passed anywhere\"\n                assert self.out, \"Nothing was logged!?\"\n\n    adapter = StringIOAdapter()\n    # TODO: it does store messages but without any formatting, i.e. even without\n    # date/time prefix etc.  IMHO it should preserve formatting in case if file_ is\n    # set\n    swallow_handler = logging.StreamHandler(adapter.handle)\n    # we want to log levelname so we could test against it\n    swallow_handler.setFormatter(\n        logging.Formatter('[%(levelname)s] %(message)s'))\n    # Inherit filters\n    from datalad.log import ProgressHandler\n    swallow_handler.filters = sum([h.filters for h in old_handlers\n                                   if not isinstance(h, ProgressHandler)],\n                                  [])\n    lgr.handlers = [swallow_handler]\n    if old_level < logging.DEBUG:  # so if HEAVYDEBUG etc -- show them!\n        lgr.handlers += old_handlers\n\n    if isinstance(new_level, str):\n        new_level = getattr(logging, new_level)\n\n    if new_level is not None:\n        lgr.setLevel(new_level)\n\n    try:\n        yield adapter\n        # TODO: if file_ and there was an exception -- most probably worth logging it?\n        # although ideally it should be the next log outside added to that file_ ... oh well\n    finally:\n        lgr.handlers, lgr.level = old_handlers, old_level\n        adapter.cleanup()\n\n\n# TODO: May be melt in with swallow_logs at some point:\n@contextmanager\ndef disable_logger(logger=None):\n    \"\"\"context manager to temporarily disable logging\n\n    This is to provide one of swallow_logs' purposes without unnecessarily\n    creating temp files (see gh-1865)\n\n    Parameters\n    ----------\n    logger: Logger\n        Logger whose handlers will be ordered to not log anything.\n        Default: datalad's topmost Logger ('datalad')\n    \"\"\"\n\n    class NullFilter(logging.Filter):\n        \"\"\"Filter class to reject all records\n        \"\"\"\n        def filter(self, record):\n            return 0\n\n    if logger is None:\n        # default: all of datalad's logging:\n        logger = logging.getLogger('datalad')\n\n    filter_ = NullFilter(logger.name)\n    [h.addFilter(filter_) for h in logger.handlers]\n\n    try:\n        yield logger\n    finally:\n        [h.removeFilter(filter_) for h in logger.handlers]\n\n\n#\n# Additional handlers\n#\n_sys_excepthook = sys.excepthook  # Just in case we ever need original one\n\n\ndef setup_exceptionhook(ipython=False):\n    \"\"\"Overloads default sys.excepthook with our exceptionhook handler.\n\n       If interactive, our exceptionhook handler will invoke\n       pdb.post_mortem; if not interactive, then invokes default handler.\n    \"\"\"\n\n    def _datalad_pdb_excepthook(type, value, tb):\n        import traceback\n        traceback.print_exception(type, value, tb)\n        print()\n        if is_interactive():\n            import pdb\n            pdb.post_mortem(tb)\n\n    if ipython:\n        from IPython.core import ultratb\n        sys.excepthook = ultratb.FormattedTB(mode='Verbose',\n                                             # color_scheme='Linux',\n                                             call_pdb=is_interactive())\n    else:\n        sys.excepthook = _datalad_pdb_excepthook\n\n\ndef assure_dir(*args):\n    \"\"\"Make sure directory exists.\n\n    Joins the list of arguments to an os-specific path to the desired\n    directory and creates it, if it not exists yet.\n    \"\"\"\n    dirname = opj(*args)\n    if not exists(dirname):\n        os.makedirs(dirname)\n    return dirname\n\n\ndef updated(d, update):\n    \"\"\"Return a copy of the input with the 'update'\n\n    Primarily for updating dictionaries\n    \"\"\"\n    d = d.copy()\n    d.update(update)\n    return d\n\n\n_pwd_mode = None\n\n\ndef _switch_to_getcwd(msg, *args):\n    global _pwd_mode\n    _pwd_mode = 'cwd'\n    lgr.warning(\n        msg + \". From now on will be returning os.getcwd(). Directory\"\n               \" symlinks in the paths will be resolved\",\n        *args\n    )\n    # TODO:  we might want to mitigate by going through all flywheighted\n    # repos and tuning up their .paths to be resolved?\n\n\ndef getpwd():\n    \"\"\"Try to return a CWD without dereferencing possible symlinks\n\n    This function will try to use PWD environment variable to provide a current\n    working directory, possibly with some directories along the path being\n    symlinks to other directories.  Unfortunately, PWD is used/set only by the\n    shell and such functions as `os.chdir` and `os.getcwd` nohow use or modify\n    it, thus `os.getcwd()` returns path with links dereferenced.\n\n    While returning current working directory based on PWD env variable we\n    verify that the directory is the same as `os.getcwd()` after resolving all\n    symlinks.  If that verification fails, we fall back to always use\n    `os.getcwd()`.\n\n    Initial decision to either use PWD env variable or os.getcwd() is done upon\n    the first call of this function.\n    \"\"\"\n    global _pwd_mode\n    if _pwd_mode is None:\n        # we need to decide!\n        try:\n            pwd = os.environ['PWD']\n            if on_windows and pwd and pwd.startswith('/'):\n                # It should be a path from MSYS.\n                # - it might start with a drive letter or not\n                # - it seems to be \"illegal\" to have a single letter directories\n                #   under / path, i.e. if created - they aren't found\n                # - 'ln -s' does not fail to create a \"symlink\" but it just\n                # copies!\n                #   so we are not likely to need original PWD purpose on\n                # those systems\n                # Verdict:\n                _pwd_mode = 'cwd'\n            else:\n                _pwd_mode = 'PWD'\n        except KeyError:\n            _pwd_mode = 'cwd'\n\n    if _pwd_mode == 'cwd':\n        return os.getcwd()\n    elif _pwd_mode == 'PWD':\n        try:\n            cwd = os.getcwd()\n        except OSError as exc:\n            if \"o such file\" in str(exc):\n                # directory was removed but we promised to be robust and\n                # still report the path we might know since we are still in PWD\n                # mode\n                cwd = None\n            else:\n                raise\n        try:\n            pwd = os.environ['PWD']\n            pwd_real = op.realpath(pwd)\n            # This logic would fail to catch the case where chdir did happen\n            # to the directory where current PWD is pointing to, e.g.\n            # $> ls -ld $PWD\n            # lrwxrwxrwx 1 yoh yoh 5 Oct 11 13:27 /home/yoh/.tmp/tmp -> /tmp//\n            # hopa:~/.tmp/tmp\n            # $> python -c 'import os; os.chdir(\"/tmp\"); from datalad.utils import getpwd; print(getpwd(), os.getcwd())'\n            # ('/home/yoh/.tmp/tmp', '/tmp')\n            # but I guess that should not be too harmful\n            if cwd is not None and pwd_real != cwd:\n                _switch_to_getcwd(\n                    \"realpath of PWD=%s is %s whenever os.getcwd()=%s\",\n                    pwd, pwd_real, cwd\n                )\n                return cwd\n            return pwd\n        except KeyError:\n            _switch_to_getcwd(\"PWD env variable is no longer available\")\n            return cwd  # Must not happen, but may be someone\n                        # evil purges PWD from environ?\n    else:\n        raise RuntimeError(\n            \"Must have not got here. \"\n            \"pwd_mode must be either cwd or PWD. And it is now %r\" % (_pwd_mode,)\n        )\n\n\nclass chpwd(object):\n    \"\"\"Wrapper around os.chdir which also adjusts environ['PWD']\n\n    The reason is that otherwise PWD is simply inherited from the shell\n    and we have no ability to assess directory path without dereferencing\n    symlinks.\n\n    If used as a context manager it allows to temporarily change directory\n    to the given path\n    \"\"\"\n    def __init__(self, path, mkdir=False, logsuffix=''):\n\n        if path:\n            pwd = getpwd()\n            self._prev_pwd = pwd\n        else:\n            self._prev_pwd = None\n            return\n\n        if not isabs(path):\n            path = normpath(opj(pwd, path))\n        if not os.path.exists(path) and mkdir:\n            self._mkdir = True\n            os.mkdir(path)\n        else:\n            self._mkdir = False\n        lgr.debug(\"chdir %r -> %r %s\", self._prev_pwd, path, logsuffix)\n        os.chdir(path)  # for grep people -- ok, to chdir here!\n        os.environ['PWD'] = path\n\n    def __enter__(self):\n        # nothing more to do really, chdir was in the constructor\n        pass\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        if self._prev_pwd:\n            # Need to use self.__class__ so this instance, if the entire\n            # thing mocked during the test, still would use correct chpwd\n            self.__class__(self._prev_pwd, logsuffix=\"(coming back)\")\n\n\ndef dlabspath(path, norm=False):\n    \"\"\"Symlinks-in-the-cwd aware abspath\n\n    os.path.abspath relies on os.getcwd() which would not know about symlinks\n    in the path\n\n    TODO: we might want to norm=True by default to match behavior of\n    os .path.abspath?\n    \"\"\"\n    if not isabs(path):\n        # if not absolute -- relative to pwd\n        path = opj(getpwd(), path)\n    return normpath(path) if norm else path\n\n\ndef with_pathsep(path):\n    \"\"\"Little helper to guarantee that path ends with /\"\"\"\n    return path + sep if not path.endswith(sep) else path\n\n\ndef get_path_prefix(path, pwd=None):\n    \"\"\"Get path prefix (for current directory)\n\n    Returns relative path to the topdir, if we are under topdir, and if not\n    absolute path to topdir.  If `pwd` is not specified - current directory\n    assumed\n    \"\"\"\n    pwd = pwd or getpwd()\n    path = dlabspath(path)\n    path_ = with_pathsep(path)\n    pwd_ = with_pathsep(pwd)\n    common = commonprefix((path_, pwd_))\n    if common.endswith(sep) and common in {path_, pwd_}:\n        # we are in subdir or above the path = use relative path\n        location_prefix = relpath(path, pwd)\n        # if benign \"here\" - cut off\n        if location_prefix in (curdir, curdir + sep):\n            location_prefix = ''\n        return location_prefix\n    else:\n        # just return absolute path\n        return path\n\n\ndef _get_normalized_paths(path, prefix):\n    if isabs(path) != isabs(prefix):\n        raise ValueError(\"Both paths must either be absolute or relative. \"\n                         \"Got %r and %r\" % (path, prefix))\n    path = with_pathsep(path)\n    prefix = with_pathsep(prefix)\n    return path, prefix\n\n\ndef path_startswith(path, prefix):\n    \"\"\"Return True if path starts with prefix path\n\n    Parameters\n    ----------\n    path: str\n    prefix: str\n    \"\"\"\n    path, prefix = _get_normalized_paths(path, prefix)\n    return path.startswith(prefix)\n\n\ndef path_is_subpath(path, prefix):\n    \"\"\"Return True if path is a subpath of prefix\n\n    It will return False if path == prefix.\n\n    Parameters\n    ----------\n    path: str\n    prefix: str\n    \"\"\"\n    path, prefix = _get_normalized_paths(path, prefix)\n    return (len(prefix) < len(path)) and path.startswith(prefix)\n\n\ndef knows_annex(path):\n    \"\"\"Returns whether at a given path there is information about an annex\n\n    It is just a thin wrapper around GitRepo.is_with_annex() classmethod\n    which also checks for `path` to exist first.\n\n    This includes actually present annexes, but also uninitialized ones, or\n    even the presence of a remote annex branch.\n    \"\"\"\n    from os.path import exists\n    if not exists(path):\n        lgr.debug(\"No annex: test path {0} doesn't exist\".format(path))\n        return False\n    from datalad.support.gitrepo import GitRepo\n    return GitRepo(path, init=False, create=False).is_with_annex()\n\n\n@contextmanager\ndef make_tempfile(content=None, wrapped=None, **tkwargs):\n    \"\"\"Helper class to provide a temporary file name and remove it at the end (context manager)\n\n    Parameters\n    ----------\n    mkdir : bool, optional (default: False)\n        If True, temporary directory created using tempfile.mkdtemp()\n    content : str or bytes, optional\n        Content to be stored in the file created\n    wrapped : function, optional\n        If set, function name used to prefix temporary file name\n    `**tkwargs`:\n        All other arguments are passed into the call to tempfile.mk{,d}temp(),\n        and resultant temporary filename is passed as the first argument into\n        the function t.  If no 'prefix' argument is provided, it will be\n        constructed using module and function names ('.' replaced with\n        '_').\n\n    To change the used directory without providing keyword argument 'dir' set\n    DATALAD_TESTS_TEMP_DIR.\n\n    Examples\n    --------\n        >>> from os.path import exists\n        >>> from datalad.utils import make_tempfile\n        >>> with make_tempfile() as fname:\n        ...    k = open(fname, 'w').write('silly test')\n        >>> assert not exists(fname)  # was removed\n\n        >>> with make_tempfile(content=\"blah\") as fname:\n        ...    assert open(fname).read() == \"blah\"\n    \"\"\"\n\n    if tkwargs.get('mkdir', None) and content is not None:\n        raise ValueError(\"mkdir=True while providing content makes no sense\")\n\n    tkwargs_ = get_tempfile_kwargs(tkwargs, wrapped=wrapped)\n\n    # if DATALAD_TESTS_TEMP_DIR is set, use that as directory,\n    # let mktemp handle it otherwise. However, an explicitly provided\n    # dir=... will override this.\n    mkdir = tkwargs_.pop('mkdir', False)\n\n    filename = {False: tempfile.mktemp,\n                True: tempfile.mkdtemp}[mkdir](**tkwargs_)\n    filename = realpath(filename)\n\n    if content:\n        with open(filename, 'w' + ('b' if isinstance(content, binary_type) else '')) as f:\n            f.write(content)\n\n    if __debug__:\n        # TODO mkdir\n        lgr.debug('Created temporary thing named %s\"' % filename)\n    try:\n        yield filename\n    finally:\n        # glob here for all files with the same name (-suffix)\n        # would be useful whenever we requested .img filename,\n        # and function creates .hdr as well\n        lsuffix = len(tkwargs_.get('suffix', ''))\n        filename_ = lsuffix and filename[:-lsuffix] or filename\n        filenames = glob.glob(filename_ + '*')\n        if len(filename_) < 3 or len(filenames) > 5:\n            # For paranoid yoh who stepped into this already ones ;-)\n            lgr.warning(\"It is unlikely that it was intended to remove all\"\n                        \" files matching %r. Skipping\" % filename_)\n            return\n        for f in filenames:\n            try:\n                rmtemp(f)\n            except OSError:  # pragma: no cover\n                pass\n\n\ndef _path_(*p):\n    \"\"\"Given a path in POSIX\" notation, regenerate one in native to the env one\"\"\"\n    if on_windows:\n        return opj(*map(lambda x: opj(*x.split('/')), p))\n    else:\n        # Assume that all others as POSIX compliant so nothing to be done\n        return opj(*p)\n\n\ndef get_timestamp_suffix(time_=None, prefix='-'):\n    \"\"\"Return a time stamp (full date and time up to second)\n\n    primarily to be used for generation of log files names\n    \"\"\"\n    args = []\n    if time_ is not None:\n        if isinstance(time_, int):\n            time_ = time.gmtime(time_)\n        args.append(time_)\n    return time.strftime(prefix + TIMESTAMP_FMT, *args)\n\n\ndef get_logfilename(dspath, cmd='datalad'):\n    \"\"\"Return a filename to use for logging under a dataset/repository\n\n    directory would be created if doesn't exist, but dspath must exist\n    and be a directory\n    \"\"\"\n    assert(exists(dspath))\n    assert(isdir(dspath))\n    ds_logdir = assure_dir(dspath, '.git', 'datalad', 'logs')  # TODO: use WEB_META_LOG whenever #789 merged\n    return opj(ds_logdir, 'crawl-%s.log' % get_timestamp_suffix())\n\n\ndef get_trace(edges, start, end, trace=None):\n    \"\"\"Return the trace/path to reach a node in a tree.\n\n    Parameters\n    ----------\n    edges : sequence(2-tuple)\n      The tree given by a sequence of edges (parent, child) tuples. The\n      nodes can be identified by any value and data type that supports\n      the '==' operation.\n    start :\n      Identifier of the start node. Must be present as a value in the parent\n      location of an edge tuple in order to be found.\n    end :\n      Identifier of the target/end node. Must be present as a value in the child\n      location of an edge tuple in order to be found.\n    trace : list\n      Mostly useful for recursive calls, and used internally.\n\n    Returns\n    -------\n    None or list\n      Returns a list with the trace to the target (the starts and the target\n      are not included in the trace, hence if start and end are directly connected\n      an empty list is returned), or None when no trace to the target can be found,\n      or start and end are identical.\n    \"\"\"\n    # the term trace is used to avoid confusion with a path in the sense\n    # of a filesystem path, but the analogy fits and nodes can be paths\n    if trace is None:\n        trace = []\n    if not edges:\n        raise ValueError(\"no edges given\")\n    for cand in edges:\n        cand_super, cand_sub = cand\n        if cand_sub in trace:\n            # only DAGs, skip any cyclic traces\n            continue\n        if trace and cand_super != trace[-1]:\n            # only consider edges that lead off the end of the trace\n            continue\n        if not trace and cand_super != start:\n            # we got nothing yet, and this edges is not matching the start\n            continue\n        if cand_sub == end:\n            return trace\n        # dive into potential subnodes\n        cand_trace = get_trace(\n            edges,\n            start,\n            end,\n            trace + [cand_sub])\n        if cand_trace:\n            return cand_trace\n    return None\n\n\ndef get_dataset_root(path):\n    \"\"\"Return the root of an existent dataset containing a given path\n\n    The root path is returned in the same absolute or relative form\n    as the input argument. If no associated dataset exists, or the\n    input path doesn't exist, None is returned.\n    \"\"\"\n    suffix = '.git'\n    if not isdir(path):\n        path = dirname(path)\n    apath = abspath(path)\n    # while we can still go up\n    while psplit(apath)[1]:\n        if exists(opj(path, suffix)):\n            return path\n        # new test path in the format we got it\n        path = normpath(opj(path, os.pardir))\n        # no luck, next round\n        apath = abspath(path)\n    return None\n\n\ndef get_dataset_pwds(dataset):\n    \"\"\"Return the current directory for the dataset.\n\n    Parameters\n    ----------\n    dataset : Dataset\n\n    Returns\n    -------\n    A tuple, where the first item is the absolute path of the pwd and the\n    second is the pwd relative to the dataset's path.\n    \"\"\"\n    if dataset:\n        pwd = dataset.path\n        rel_pwd = curdir\n    else:\n        # act on the whole dataset if nothing else was specified\n\n        # Follow our generic semantic that if dataset is specified,\n        # paths are relative to it, if not -- relative to pwd\n        pwd = getpwd()\n        # Pass pwd to get_dataset_root instead of os.path.curdir to handle\n        # repos whose leading paths have a symlinked directory (see the\n        # TMPDIR=\"/var/tmp/sym link\" test case).\n        dataset = get_dataset_root(pwd)\n\n        if dataset:\n            rel_pwd = relpath(pwd, dataset)\n        else:\n            rel_pwd = pwd  # and leave handling to caller\n    return pwd, rel_pwd\n\n\n# ATM used in datalad_crawler extension, so do not remove yet\ndef try_multiple(ntrials, exception, base, f, *args, **kwargs):\n    \"\"\"Call f multiple times making exponentially growing delay between the calls\"\"\"\n    from .dochelpers import exc_str\n    for trial in range(1, ntrials+1):\n        try:\n            return f(*args, **kwargs)\n        except exception as exc:\n            if trial == ntrials:\n                raise  # just reraise on the last trial\n            t = base ** trial\n            lgr.warning(\"Caught %s on trial #%d. Sleeping %f and retrying\",\n                        exc_str(exc), trial, t)\n            sleep(t)\n\n\n@optional_args\ndef try_multiple_dec(f, ntrials=None, duration=0.1, exceptions=None, increment_type=None):\n    \"\"\"Decorator to try function multiple times.\n\n    Main purpose is to decorate functions dealing with removal of files/directories\n    and which might need a few seconds to work correctly on Windows which takes\n    its time to release files/directories.\n\n    Parameters\n    ----------\n    ntrials: int, optional\n    duration: float, optional\n      Seconds to sleep before retrying.\n    increment_type: {None, 'exponential'}\n      Note that if it is exponential, duration should typically be > 1.0\n      so it grows with higher power\n\n    \"\"\"\n    from .dochelpers import exc_str\n    if not exceptions:\n        exceptions = (OSError, WindowsError, PermissionError) \\\n            if on_windows else OSError\n    if not ntrials:\n        # Life goes fast on proper systems, no need to delay it much\n        ntrials = 50 if on_windows else 3\n\n    assert increment_type in {None, 'exponential'}\n\n    @wraps(f)\n    def wrapped(*args, **kwargs):\n        t = duration\n        for trial in range(ntrials):\n            try:\n                return f(*args, **kwargs)\n            except exceptions as exc:\n                if increment_type == 'exponential':\n                    t = duration ** (trial + 1)\n                lgr.log(\n                    5,\n                    \"Caught %s on trial #%d. Sleeping %f and retrying\",\n                    exc_str(exc), trial, t)\n                if trial < ntrials - 1:\n                    sleep(t)\n                else:\n                    raise\n\n    return wrapped\n\n\n@try_multiple_dec\ndef unlink(f):\n    \"\"\"'Robust' unlink.  Would try multiple times\n\n    On windows boxes there is evidence for a latency of more than a second\n    until a file is considered no longer \"in-use\".\n    WindowsError is not known on Linux, and if IOError or any other\n    exception\n    is thrown then if except statement has WindowsError in it -- NameError\n    also see gh-2533\n    \"\"\"\n    # Check for open files\n    assert_no_open_files(f)\n    return os.unlink(f)\n\n\n@try_multiple_dec\ndef _rmtree(*args, **kwargs):\n    \"\"\"Just a helper to decorate shutil.rmtree.\n\n    rmtree defined above does more and ideally should not itself be decorated\n    since a recursive definition and does checks for open files inside etc -\n    might be too runtime expensive\n    \"\"\"\n    return shutil.rmtree(*args, **kwargs)\n\n\ndef slash_join(base, extension):\n    \"\"\"Join two strings with a '/', avoiding duplicate slashes\n\n    If any of the strings is None the other is returned as is.\n    \"\"\"\n    if extension is None:\n        return base\n    if base is None:\n        return extension\n    return '/'.join(\n        (base.rstrip('/'),\n         extension.lstrip('/')))\n\n\ndef safe_print(s):\n    \"\"\"Print with protection against UTF-8 encoding errors\"\"\"\n    # A little bit of dance to be able to test this code\n    print_f = getattr(__builtin__, \"print\")\n    try:\n        print_f(s)\n    except UnicodeEncodeError:\n        # failed to encode so let's do encoding while ignoring errors\n        # to print at least something\n        # explicit `or ascii` since somehow on buildbot it seemed to return None\n        s = s.encode(getattr(sys.stdout, 'encoding', 'ascii') or 'ascii', errors='ignore') \\\n            if hasattr(s, 'encode') else s\n        print_f(s.decode())\n\n#\n# IO Helpers\n#\n\ndef open_r_encdetect(fname, readahead=1000):\n    \"\"\"Return a file object in read mode with auto-detected encoding\n\n    This is helpful when dealing with files of unknown encoding.\n\n    Parameters\n    ----------\n    readahead: int, optional\n      How many bytes to read for guessing the encoding type.  If\n      negative - full file will be read\n    \"\"\"\n    from chardet import detect\n    import io\n    # read some bytes from the file\n    with open(fname, 'rb') as f:\n        head = f.read(readahead)\n    enc = detect(head)\n    denc = enc.get('encoding', None)\n    lgr.debug(\"Auto-detected encoding %s for file %s (confidence: %s)\",\n              denc,\n              fname,\n              enc.get('confidence', 'unknown'))\n    return io.open(fname, encoding=denc)\n\n\ndef read_csv_lines(fname, dialect=None, readahead=16384, **kwargs):\n    \"\"\"A generator of dict records from a CSV/TSV\n\n    Automatically guesses the encoding for each record to convert to UTF-8\n\n    Parameters\n    ----------\n    fname: str\n      Filename\n    dialect: str, optional\n      Dialect to specify to csv.reader. If not specified -- guessed from\n      the file, if fails to guess, \"excel-tab\" is assumed\n    readahead: int, optional\n      How many bytes to read from the file to guess the type\n    **kwargs\n      Passed to `csv.reader`\n    \"\"\"\n    import csv\n    if dialect is None:\n        with open(fname) as tsvfile:\n            # add robustness, use a sniffer\n            try:\n                dialect = csv.Sniffer().sniff(tsvfile.read(readahead))\n            except Exception as exc:\n                from .dochelpers import exc_str\n                lgr.warning(\n                    'Could not determine file-format, assuming TSV: %s',\n                    exc_str(exc)\n                )\n                dialect = 'excel-tab'\n\n    kw = {} if PY2 else dict(encoding='utf-8')\n    with open(fname, 'rb' if PY2 else 'r', **kw) as tsvfile:\n        # csv.py doesn't do Unicode; encode temporarily as UTF-8:\n        csv_reader = csv.reader(\n            tsvfile,\n            dialect=dialect,\n            **kwargs\n        )\n        header = None\n        for row in csv_reader:\n            # decode UTF-8 back to Unicode, cell by cell:\n            row_unicode = map(assure_unicode, row)\n            if header is None:\n                header = list(row_unicode)\n            else:\n                yield dict(zip(header, row_unicode))\n\n\ndef import_modules(modnames, pkg, msg=\"Failed to import {module}\", log=lgr.debug):\n    \"\"\"Helper to import a list of modules without failing if N/A\n\n    Parameters\n    ----------\n    modnames: list of str\n      List of module names to import\n    pkg: str\n      Package under which to import\n    msg: str, optional\n      Message template for .format() to log at DEBUG level if import fails.\n      Keys {module} and {package} will be provided and ': {exception}' appended\n    log: callable, optional\n      Logger call to use for logging messages\n    \"\"\"\n    from importlib import import_module\n    _globals = globals()\n    mods_loaded = []\n    if pkg and not pkg in sys.modules:\n        # with python 3.5.1 (ok with 3.5.5) somehow kept running into\n        #  Failed to import dlsub1: Parent module 'dltestm1' not loaded\n        # while running the test. Preloading pkg resolved the issue\n        import_module(pkg)\n    for modname in modnames:\n        try:\n            _globals[modname] = mod = import_module(\n                '.{}'.format(modname),\n                pkg)\n            mods_loaded.append(mod)\n        except Exception as exc:\n            from datalad.dochelpers import exc_str\n            log((msg + ': {exception}').format(\n                module=modname, package=pkg, exception=exc_str(exc)))\n    return mods_loaded\n\n\ndef import_module_from_file(modpath, pkg=None, log=lgr.debug):\n    \"\"\"Import provided module given a path\n\n    TODO:\n    - RF/make use of it in pipeline.py which has similar logic\n    - join with import_modules above?\n\n    Parameters\n    ----------\n    pkg: module, optional\n       If provided, and modpath is under pkg.__path__, relative import will be\n       used\n    \"\"\"\n    assert(modpath.endswith('.py'))  # for now just for .py files\n\n    log(\"Importing %s\" % modpath)\n\n    modname = basename(modpath)[:-3]\n    relmodpath = None\n    if pkg:\n        for pkgpath in pkg.__path__:\n            if path_is_subpath(modpath, pkgpath):\n                # for now relying on having .py extension -- assertion above\n                relmodpath = '.' + relpath(modpath[:-3], pkgpath).replace(sep, '.')\n                break\n\n    try:\n        if relmodpath:\n            from importlib import import_module\n            mod = import_module(relmodpath, pkg.__name__)\n        else:\n            dirname_ = dirname(modpath)\n            try:\n                sys.path.insert(0, dirname_)\n                mod = __import__(modname, level=0)\n            finally:\n                if dirname_ in sys.path:\n                    sys.path.pop(sys.path.index(dirname_))\n                else:\n                    log(\"Expected path %s to be within sys.path, but it was gone!\" % dirname_)\n    except Exception as e:\n        from datalad.dochelpers import exc_str\n        raise RuntimeError(\n            \"Failed to import module from %s: %s\" % (modpath, exc_str(e)))\n\n    return mod\n\n\ndef get_encoding_info():\n    \"\"\"Return a dictionary with various encoding/locale information\"\"\"\n    import sys, locale\n    from collections import OrderedDict\n    return OrderedDict([\n        ('default', sys.getdefaultencoding()),\n        ('filesystem', sys.getfilesystemencoding()),\n        ('locale.prefered', locale.getpreferredencoding()),\n    ])\n\n\ndef get_envvars_info():\n    from collections import OrderedDict\n    envs = []\n    for var, val in os.environ.items():\n        if (\n                var.startswith('PYTHON') or\n                var.startswith('LC_') or\n                var.startswith('GIT_') or\n                var in ('LANG', 'LANGUAGE', 'PATH')\n        ):\n            envs.append((var, val))\n    return OrderedDict(envs)\n\n\n# This class is modified from Snakemake (v5.1.4)\nclass SequenceFormatter(string.Formatter):\n    \"\"\"string.Formatter subclass with special behavior for sequences.\n\n    This class delegates formatting of individual elements to another\n    formatter object. Non-list objects are formatted by calling the\n    delegate formatter's \"format_field\" method. List-like objects\n    (list, tuple, set, frozenset) are formatted by formatting each\n    element of the list according to the specified format spec using\n    the delegate formatter and then joining the resulting strings with\n    a separator (space by default).\n    \"\"\"\n\n    def __init__(self, separator=\" \", element_formatter=string.Formatter(),\n                 *args, **kwargs):\n        self.separator = separator\n        self.element_formatter = element_formatter\n\n    def format_element(self, elem, format_spec):\n        \"\"\"Format a single element\n\n        For sequences, this is called once for each element in a\n        sequence. For anything else, it is called on the entire\n        object. It is intended to be overridden in subclases.\n        \"\"\"\n        return self.element_formatter.format_field(elem, format_spec)\n\n    def format_field(self, value, format_spec):\n        if isinstance(value, (list, tuple, set, frozenset)):\n            return self.separator.join(self.format_element(v, format_spec)\n                                       for v in value)\n        else:\n            return self.format_element(value, format_spec)\n\n\n# TODO: eventually we might want to make use of attr module\nclass File(object):\n    \"\"\"Helper for a file entry in the create_tree/@with_tree\n\n    It allows to define additional settings for entries\n    \"\"\"\n    def __init__(self, name, executable=False):\n        \"\"\"\n\n        Parameters\n        ----------\n        name : str\n          Name of the file\n        executable: bool, optional\n          Make it executable\n        \"\"\"\n        self.name = name\n        self.executable = executable\n\n    def __str__(self):\n        return self.name\n\n\ndef create_tree_archive(path, name, load, overwrite=False, archives_leading_dir=True):\n    \"\"\"Given an archive `name`, create under `path` with specified `load` tree\n    \"\"\"\n    from datalad.support.archives import compress_files\n    dirname = file_basename(name)\n    full_dirname = op.join(path, dirname)\n    os.makedirs(full_dirname)\n    create_tree(full_dirname, load, archives_leading_dir=archives_leading_dir)\n    # create archive\n    if archives_leading_dir:\n        compress_files([dirname], name, path=path, overwrite=overwrite)\n    else:\n        compress_files(list(map(op.basename, glob.glob(opj(full_dirname, '*')))),\n                       opj(op.pardir, name),\n                       path=op.join(path, dirname),\n                       overwrite=overwrite)\n    # remove original tree\n    rmtree(full_dirname)\n\n\ndef create_tree(path, tree, archives_leading_dir=True, remove_existing=False):\n    \"\"\"Given a list of tuples (name, load) create such a tree\n\n    if load is a tuple itself -- that would create either a subtree or an archive\n    with that content and place it into the tree if name ends with .tar.gz\n    \"\"\"\n    lgr.log(5, \"Creating a tree under %s\", path)\n    if not op.exists(path):\n        os.makedirs(path)\n\n    if isinstance(tree, dict):\n        tree = tree.items()\n\n    for file_, load in tree:\n        if isinstance(file_, File):\n            executable = file_.executable\n            name = file_.name\n        else:\n            executable = False\n            name = file_\n        full_name = op.join(path, name)\n        if remove_existing and op.lexists(full_name):\n            rmtree(full_name, chmod_files=True)\n        if isinstance(load, (tuple, list, dict)):\n            if name.endswith('.tar.gz') or name.endswith('.tar') or name.endswith('.zip'):\n                create_tree_archive(\n                    path, name, load,\n                    archives_leading_dir=archives_leading_dir)\n            else:\n                create_tree(\n                    full_name, load,\n                    archives_leading_dir=archives_leading_dir,\n                    remove_existing=remove_existing)\n        else:\n            open_func = open\n            if full_name.endswith('.gz'):\n                open_func = gzip.open\n            with open_func(full_name, \"wb\") as f:\n                f.write(assure_bytes(load, 'utf-8'))\n        if executable:\n            os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)\n\n\ndef get_suggestions_msg(values, known, sep=\"\\n        \"):\n    \"\"\"Return a formatted string with suggestions for values given the known ones\n    \"\"\"\n    import difflib\n    suggestions = []\n    for value in assure_list(values):  # might not want to do it if we change presentation below\n        suggestions += difflib.get_close_matches(value, known)\n    suggestions = unique(suggestions)\n    msg = \"Did you mean any of these?\"\n    if suggestions:\n        if '\\n' in sep:\n            # if separator includes new line - we add entire separator right away\n            msg += sep\n        else:\n            msg += ' '\n        return msg + \"%s\\n\" % sep.join(suggestions)\n    return ''\n\n\n\nlgr.log(5, \"Done importing datalad.utils\")\n",
            "header": {
                "file": {
                    "path": "datalad/utils.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "2aa396928",
                "new_commit_hash": "5938111b6",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/utils.py"
            },
            "plus_file": {
                "path": "b/datalad/utils.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 2142,
                            "length": 4
                        },
                        "new_range": {
                            "start": 2142,
                            "length": 24
                        },
                        "section": "def create_tree(path, tree, archives_leading_dir=True, remove_existing=False):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def get_suggestions_msg(values, known, sep=\"\\n        \"):",
                                "type": "added"
                            },
                            {
                                "content": "    \"\"\"Return a formatted string with suggestions for values given the known ones",
                                "type": "added"
                            },
                            {
                                "content": "    \"\"\"",
                                "type": "added"
                            },
                            {
                                "content": "    import difflib",
                                "type": "added"
                            },
                            {
                                "content": "    suggestions = []",
                                "type": "added"
                            },
                            {
                                "content": "    for value in assure_list(values):  # might not want to do it if we change presentation below",
                                "type": "added"
                            },
                            {
                                "content": "        suggestions += difflib.get_close_matches(value, known)",
                                "type": "added"
                            },
                            {
                                "content": "    suggestions = unique(suggestions)",
                                "type": "added"
                            },
                            {
                                "content": "    msg = \"Did you mean any of these?\"",
                                "type": "added"
                            },
                            {
                                "content": "    if suggestions:",
                                "type": "added"
                            },
                            {
                                "content": "        if '\\n' in sep:",
                                "type": "added"
                            },
                            {
                                "content": "            # if separator includes new line - we add entire separator right away",
                                "type": "added"
                            },
                            {
                                "content": "            msg += sep",
                                "type": "added"
                            },
                            {
                                "content": "        else:",
                                "type": "added"
                            },
                            {
                                "content": "            msg += ' '",
                                "type": "added"
                            },
                            {
                                "content": "        return msg + \"%s\\n\" % sep.join(suggestions)",
                                "type": "added"
                            },
                            {
                                "content": "    return ''",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "lgr.log(5, \"Done importing datalad.utils\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/utils.py",
                            "type": "function",
                            "name": "get_suggestions_msg",
                            "content": "def get_suggestions_msg(values, known, sep=\"\\n        \"):\n    \"\"\"Return a formatted string with suggestions for values given the known ones\n    \"\"\"\n    import difflib\n    suggestions = []\n    for value in assure_list(values):  # might not want to do it if we change presentation below\n        suggestions += difflib.get_close_matches(value, known)\n    suggestions = unique(suggestions)\n    msg = \"Did you mean any of these?\"\n    if suggestions:\n        if '\\n' in sep:\n            # if separator includes new line - we add entire separator right away\n            msg += sep\n        else:\n            msg += ' '\n        return msg + \"%s\\n\" % sep.join(suggestions)\n    return ''",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 2145,
                            "end_lineno": 2161,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "ade51ea0dab18977814b9f09beedc77b52f2e9e0^",
    "new_commit_hash": "ade51ea0dab18977814b9f09beedc77b52f2e9e0",
    "commit_message": "RF: get_suggestions_msg() to provide uniform msg composition for presenting suggestions\n\n'any' is used now instead of 'one' since we can query for multiple terms at once\nso 'one' really doesn't fit there, while 'any' fits both scenarios",
    "commit_date": "2018-12-13T10:02:21-05:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "ade51ea0dab18977814b9f09beedc77b52f2e9e0",
    "test_file_codes": [
        "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test functioning of the datalad main cmdline utility \"\"\"\n\nfrom datalad.tests.utils import known_failure_direct_mode\nfrom datalad.tests.utils import on_windows\n\n\nimport re\nimport sys\nfrom six.moves import StringIO\nfrom mock import patch\n\nimport datalad\nfrom datalad.cmdline.main import (\n    main,\n    fail_with_short_help,\n    _fix_datalad_ri,\n)\nfrom datalad import __version__\nfrom datalad.cmd import Runner\nfrom datalad.ui.utils import get_console_width\nfrom datalad.api import create\nfrom datalad.utils import chpwd\nfrom datalad.tests.utils import with_tempfile\nfrom datalad.tests.utils import assert_equal, assert_raises, in_, ok_startswith\nfrom datalad.tests.utils import assert_in\nfrom datalad.tests.utils import assert_re_in\nfrom datalad.tests.utils import assert_not_in\nfrom datalad.tests.utils import slow\n\n\ndef run_main(args, exit_code=0, expect_stderr=False):\n    \"\"\"Run main() of the datalad, do basic checks and provide outputs\n\n    Parameters\n    ----------\n    args : list\n        List of string cmdline arguments to pass\n    exit_code : int\n        Expected exit code. Would raise AssertionError if differs\n    expect_stderr : bool or string\n        Whether to expect stderr output. If string -- match\n\n    Returns\n    -------\n    stdout, stderr  strings\n       Output produced\n    \"\"\"\n    with patch('sys.stderr', new_callable=StringIO) as cmerr:\n        with patch('sys.stdout', new_callable=StringIO) as cmout:\n            with assert_raises(SystemExit) as cm:\n                main([\"datalad\"] + list(args))\n            assert_equal(cm.exception.code, exit_code)\n            stdout = cmout.getvalue()\n            stderr = cmerr.getvalue()\n            if expect_stderr is False:\n                assert_equal(stderr, \"\")\n            elif expect_stderr is True:\n                # do nothing -- just return\n                pass\n            else:\n                # must be a string\n                assert_equal(stderr, expect_stderr)\n    return stdout, stderr\n\n\n# TODO: switch to stdout for --version output\ndef test_version():\n    stdout, stderr = run_main(['--version'], expect_stderr=True)\n\n    # and output should contain our version, copyright, license\n\n    # https://hg.python.org/cpython/file/default/Doc/whatsnew/3.4.rst#l1952\n    out = stdout if sys.version_info >= (3, 4) else stderr\n    ok_startswith(out, 'datalad %s\\n' % datalad.__version__)\n    # since https://github.com/datalad/datalad/pull/2733 no license in --version\n    assert_not_in(\"Copyright\", out)\n    assert_not_in(\"Permission is hereby granted\", out)\n\n\ndef test_help_np():\n    stdout, stderr = run_main(['--help-np'])\n\n    # Let's extract section titles:\n    # enough of bin/datalad and .tox/py27/bin/datalad -- guarantee consistency! ;)\n    ok_startswith(stdout, 'Usage: datalad')\n    # Sections start/end with * if ran under DATALAD_HELP2MAN mode\n    sections = [l[1:-1] for l in filter(re.compile('^\\*.*\\*$').match, stdout.split('\\n'))]\n    # but order is still not guaranteed (dict somewhere)! TODO\n    # see https://travis-ci.org/datalad/datalad/jobs/80519004\n    # thus testing sets\n    for s in {'Commands for dataset operations',\n              'Commands for metadata handling',\n              'Miscellaneous commands',\n              'General information',\n              'Global options',\n              'Plumbing commands',\n              'Plugins'}:\n        assert_in(s, sections)\n\n    # none of the lines must be longer than 80 chars\n    # TODO: decide on   create-sibling and possibly\n    # rewrite-urls\n    accepted_width = get_console_width()\n\n    long_lines = [\"%d %s\" % (len(l), l) for l in stdout.split('\\n')\n                  if len(l) > accepted_width and\n                  '{' not in l  # on nd70 summary line is unsplit\n                  ]\n    if long_lines:\n        raise AssertionError(\n            \"Following lines in --help output were longer than %s chars:\\n%s\"\n            % (accepted_width, '\\n'.join(long_lines))\n        )\n\n\ndef test_usage_on_insufficient_args():\n    stdout, stderr = run_main(['install'], exit_code=2, expect_stderr=True)\n    ok_startswith(stderr, 'usage:')\n\n\ndef test_subcmd_usage_on_unknown_args():\n    stdout, stderr = run_main(['get', '--murks'], exit_code=1, expect_stderr=True)\n    in_('get', stdout)\n\n\ndef test_combined_short_option():\n    stdout, stderr = run_main(['-fjson'], exit_code=2, expect_stderr=True)\n    assert_not_in(\"unrecognized argument\", stderr)\n    assert_in(\"too few arguments\", stderr)\n\n\ndef check_incorrect_option(opts, err_str):\n    # The first line used to be:\n    # stdout, stderr = run_main((sys.argv[0],) + opts, expect_stderr=True, exit_code=2)\n    # But: what do we expect to be in sys.argv[0] here?\n    # It depends on how we invoke the test.\n    # - nosetests -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0}=='nosetests'\n    # - python -m nose -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0}=='python -m nose'\n    # - python -c \"import nose; nose.main()\" -s -v datalad/cmdline/tests/test_main.py would result in:\n    #   sys.argv[0]=='-c'\n    # This led to failure in case sys.argv[0] contained an option, that was\n    # defined to be a datalad option too, therefore was a 'known_arg' and was\n    # checked to meet its constraints.\n    # But sys.argv[0] actually isn't used by main at all. It simply doesn't\n    # matter what's in there. The only thing important to pass here is `opts`.\n    stdout, stderr = run_main(opts, expect_stderr=True, exit_code=2)\n    out = stdout + stderr\n    assert_in(\"usage: \", out)\n    assert_re_in(err_str, out, match=False)\n\n\ndef test_incorrect_options():\n    # apparently a bit different if following a good one so let's do both\n    err_invalid = \"error: (invalid|too few arguments|unrecognized argument)\"\n    yield check_incorrect_option, ('--buga',), err_invalid\n    yield check_incorrect_option, ('--dbg', '--buga'), err_invalid\n\n    err_insufficient = err_invalid  # \"specify\"\n    yield check_incorrect_option, ('--dbg',), err_insufficient\n    yield check_incorrect_option, tuple(), err_insufficient\n\n\ndef test_script_shims():\n    runner = Runner()\n    for script in [\n        'datalad',\n        'git-annex-remote-datalad-archives',\n        'git-annex-remote-datalad']:\n        if not on_windows:\n            # those must be available for execution, and should not contain\n            which, _ = runner(['which', script])\n            # test if there is no easy install shim in there\n            with open(which.rstrip()) as f:\n                content = f.read()\n        else:\n            from distutils.spawn import find_executable\n            content = find_executable(script)\n        assert_not_in('EASY', content) # NOTHING easy should be there\n        assert_not_in('pkg_resources', content)\n\n        # and let's check that it is our script\n        out, err = runner([script, '--version'])\n        version = (out + err).splitlines()[0].split(' ', 1)[1]\n        # we can get git and non git .dev version... so for now\n        # relax\n        get_numeric_portion = lambda v: [x for x in v.split('.') if x.isdigit()]\n        # extract numeric portion\n        assert get_numeric_portion(version) # that my lambda is correctish\n        assert_equal(get_numeric_portion(__version__),\n                     get_numeric_portion(version))\n\n\n@slow  # 11.2591s\n@with_tempfile(mkdir=True)\ndef test_cfg_override(path):\n    with chpwd(path):\n        # control\n        out, err = Runner()('datalad wtf -s some', shell=True)\n        assert_not_in('datalad.dummy: this', out)\n        # ensure that this is not a dataset's cfg manager\n        assert_not_in('datalad.dataset.id', out)\n        # env var\n        if on_windows:\n            cmd_str = 'set DATALAD_DUMMY=this&& datalad wtf -s some'\n        else:\n            cmd_str = 'DATALAD_DUMMY=this datalad wtf -s some'\n        out, err = Runner()(cmd_str, shell=True)\n        assert_in('datalad.dummy: this', out)\n        # cmdline arg\n        out, err = Runner()('datalad -c datalad.dummy=this wtf -s some', shell=True)\n        assert_in('datalad.dummy: this', out)\n\n        # now create a dataset in the path. the wtf plugin will switch to\n        # using the dataset's config manager, which must inherit the overrides\n        create(dataset=path)\n        # control\n        out, err = Runner()('datalad wtf -s some', shell=True)\n        assert_not_in('datalad.dummy: this', out)\n        # ensure that this is a dataset's cfg manager\n        assert_in('datalad.dataset.id', out)\n        # env var\n        if on_windows:\n            cmd_str = 'set DATALAD_DUMMY=this&& datalad wtf -s some'\n        else:\n            cmd_str = 'DATALAD_DUMMY=this datalad wtf -s some'\n        out, err = Runner()(cmd_str, shell=True)\n        assert_in('datalad.dummy: this', out)\n        # cmdline arg\n        out, err = Runner()('datalad -c datalad.dummy=this wtf -s some', shell=True)\n        assert_in('datalad.dummy: this', out)\n\n\ndef test_fail_with_short_help():\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(exit_code=3, out=out)\n    assert_equal(cme.exception.code, 3)\n    assert_equal(out.getvalue(), \"\")\n\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(msg=\"Failed badly\", out=out)\n    assert_equal(cme.exception.code, 1)\n    assert_equal(out.getvalue(), \"error: Failed badly\\n\")\n\n    # Suggestions, hint, etc\n    out = StringIO()\n    with assert_raises(SystemExit) as cme:\n        fail_with_short_help(\n            msg=\"Failed badly\",\n            known=[\"mother\", \"mutter\", \"father\", \"son\"],\n            provided=\"muther\",\n            hint=\"You can become one\",\n            exit_code=0,  # noone forbids\n            what=\"parent\",\n            out=out)\n    assert_equal(cme.exception.code, 0)\n    assert_equal(out.getvalue(),\n                 \"error: Failed badly\\n\"\n                 \"datalad: Unknown parent 'muther'.  See 'datalad --help'.\\n\\n\"\n                 \"Did you mean any of these?\\n\"\n                 \"        mutter\\n\"\n                 \"        mother\\n\"\n                 \"        father\\n\"\n                 \"Hint: You can become one\\n\")\n\ndef test_fix_datalad_ri():\n    assert_equal(_fix_datalad_ri('/'), '/')\n    assert_equal(_fix_datalad_ri('/a/b'), '/a/b')\n    assert_equal(_fix_datalad_ri('//'), '///')\n    assert_equal(_fix_datalad_ri('///'), '///')\n    assert_equal(_fix_datalad_ri('//a'), '///a')\n    assert_equal(_fix_datalad_ri('///a'), '///a')\n    assert_equal(_fix_datalad_ri('//a/b'), '///a/b')\n    assert_equal(_fix_datalad_ri('///a/b'), '///a/b')",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\n❌ Verification failed\nPython 3.9 installation failed, trying Python 3.7...\nVerifying installation...\nDatalad version: 0.11.1.dev87\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.7\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 97ms\nInstalled 25 packages in 3.72s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpVcVK1m/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpVcVK1m/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpVcVK1m/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpVcVK1m/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpVcVK1m/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 51, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpBuO8EZ/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpBuO8EZ/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpBuO8EZ/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpBuO8EZ/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpBuO8EZ/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 51, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0/datalad/__init__.py\", line 43, in <module>\n    from datalad.cmd import GitRunner\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0/datalad/cmd.py\", line 28, in <module>\n    from .support import path as op\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0/datalad/support/path.py\", line 21, in <module>\n    from ..utils import (\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0/datalad/utils.py\", line 26, in <module>\n    import wrapt\nModuleNotFoundError: No module named 'wrapt'\nUsing CPython 3.7.9\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 26 packages in 350ms\nInstalled 26 packages in 1.94s\n + bids==0.0\n + coverage==7.2.7\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + fasteners==0.19\n + grabbit==0.2.6\n + importlib-metadata==6.7.0\n + iniconfig==2.0.0\n + nibabel==4.0.2\n + num2words==0.5.14\n + numpy==1.21.6\n + packaging==24.0\n + pandas==1.3.5\n + patsy==1.0.1\n + pluggy==1.2.0\n + pybids==0.6.5\n + pytest==7.4.4\n + pytest-cov==4.1.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.7.3\n + setuptools==68.0.0\n + six==1.17.0\n + tomli==2.0.1\n + typing-extensions==4.7.1\n + zipp==3.15.0\nResolved 57 packages in 1.27s\nPrepared 1 package in 1.35s\nInstalled 50 packages in 46ms\n + appdirs==1.4.4\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.15.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + cryptography==44.0.0\n + datalad==0.11.1 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0)\n + deprecated==1.2.15\n + exifread==3.0.0\n + gitdb==4.0.12\n + gitpython==3.1.44\n + httpretty==1.1.4\n + humanize==4.6.0\n + idna==3.10\n + importlib-resources==5.12.0\n + iso8601==2.1.0\n + jaraco-classes==3.2.3\n + jeepney==0.8.0\n + jsmin==3.0.1\n + keyring==24.1.1\n + keyrings-alt==4.2.0\n + mock==5.1.0\n + more-itertools==9.1.0\n + msgpack==1.0.5\n + multidict==6.0.5\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.12\n + pillow==9.5.0\n + pycparser==2.21\n + pygithub==2.3.0\n + pyjwt==2.8.0\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.1\n + requests==2.31.0\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + smmap==5.0.2\n + soupsieve==2.4.1\n + tqdm==4.67.1\n + urllib3==2.0.7\n + vcrpy==4.3.0\n + whoosh==2.7.4\n + wrapt==1.16.0\n + yarl==1.9.4\nResolved 91 packages in 219ms\nInstalled 31 packages in 23ms\n + alabaster==0.7.13\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.14.0\n + build==1.1.1\n + distlib==0.3.9\n + docutils==0.19\n + filelock==3.12.2\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.9.16\n + markupsafe==2.1.5\n + nose-timer==1.0.1\n + platformdirs==4.0.0\n + psutil==6.1.1\n + pygments==2.17.2\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==5.3.0\n + sphinx-rtd-theme==2.0.0\n + sphinxcontrib-applehelp==1.0.2\n + sphinxcontrib-devhelp==1.0.2\n + sphinxcontrib-htmlhelp==2.0.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==1.0.3\n + sphinxcontrib-serializinghtml==1.1.5\n + tabulate==0.9.0\n + virtualenv==20.26.6\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0\nplugins: cov-4.1.0\ncollected 10 items\n\nr2e_tests/test_1.py .....xFF..                                           [100%]\n\n=================================== FAILURES ===================================\n______________________________ test_script_shims _______________________________\n\n    def test_script_shims():\n        runner = Runner()\n        for script in [\n            'datalad',\n            'git-annex-remote-datalad-archives',\n            'git-annex-remote-datalad']:\n            if not on_windows:\n                # those must be available for execution, and should not contain\n                which, _ = runner(['which', script])\n                # test if there is no easy install shim in there\n                with open(which.rstrip()) as f:\n                    content = f.read()\n            else:\n                from distutils.spawn import find_executable\n                content = find_executable(script)\n            assert_not_in('EASY', content) # NOTHING easy should be there\n            assert_not_in('pkg_resources', content)\n    \n            # and let's check that it is our script\n>           out, err = runner([script, '--version'])\n\nr2e_tests/test_1.py:192: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ndatalad/cmd.py:187: in __call__\n    return self.run(cmd, *args, **kwargs)\ndatalad/cmd.py:487: in run\n    stdin=stdin)\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/subprocess.py:800: in __init__\n    restore_signals, start_new_session)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = <subprocess.Popen object at 0x73561367af10>\nargs = ['datalad', '--version'], executable = b'datalad', preexec_fn = None\nclose_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None\ncreationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = 13\nc2pwrite = 15, errread = 16, errwrite = 17, restore_signals = True\nstart_new_session = False\n\n    def _execute_child(self, args, executable, preexec_fn, close_fds,\n                       pass_fds, cwd, env,\n                       startupinfo, creationflags, shell,\n                       p2cread, p2cwrite,\n                       c2pread, c2pwrite,\n                       errread, errwrite,\n                       restore_signals, start_new_session):\n        \"\"\"Execute program (POSIX version)\"\"\"\n    \n        if isinstance(args, (str, bytes)):\n            args = [args]\n        else:\n            args = list(args)\n    \n        if shell:\n            # On Android the default shell is at '/system/bin/sh'.\n            unix_shell = ('/system/bin/sh' if\n                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')\n            args = [unix_shell, \"-c\"] + args\n            if executable:\n                args[0] = executable\n    \n        if executable is None:\n            executable = args[0]\n        orig_executable = executable\n    \n        # For transferring possible exec failure from child to parent.\n        # Data format: \"exception name:hex errno:description\"\n        # Pickle is not used; it is complex and involves memory allocation.\n        errpipe_read, errpipe_write = os.pipe()\n        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.\n        low_fds_to_close = []\n        while errpipe_write < 3:\n            low_fds_to_close.append(errpipe_write)\n            errpipe_write = os.dup(errpipe_write)\n        for low_fd in low_fds_to_close:\n            os.close(low_fd)\n        try:\n            try:\n                # We must avoid complex work that could involve\n                # malloc or free in the child process to avoid\n                # potential deadlocks, thus we do all this here.\n                # and pass it to fork_exec()\n    \n                if env is not None:\n                    env_list = []\n                    for k, v in env.items():\n                        k = os.fsencode(k)\n                        if b'=' in k:\n                            raise ValueError(\"illegal environment variable name\")\n                        env_list.append(k + b'=' + os.fsencode(v))\n                else:\n                    env_list = None  # Use execv instead of execve.\n                executable = os.fsencode(executable)\n                if os.path.dirname(executable):\n                    executable_list = (executable,)\n                else:\n                    # This matches the behavior of os._execvpe().\n                    executable_list = tuple(\n                        os.path.join(os.fsencode(dir), executable)\n                        for dir in os.get_exec_path(env))\n                fds_to_keep = set(pass_fds)\n                fds_to_keep.add(errpipe_write)\n                self.pid = _posixsubprocess.fork_exec(\n                        args, executable_list,\n                        close_fds, tuple(sorted(map(int, fds_to_keep))),\n                        cwd, env_list,\n                        p2cread, p2cwrite, c2pread, c2pwrite,\n                        errread, errwrite,\n                        errpipe_read, errpipe_write,\n                        restore_signals, start_new_session, preexec_fn)\n                self._child_created = True\n            finally:\n                # be sure the FD is closed no matter what\n                os.close(errpipe_write)\n    \n            # self._devnull is not always defined.\n            devnull_fd = getattr(self, '_devnull', None)\n            if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:\n                os.close(p2cread)\n            if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:\n                os.close(c2pwrite)\n            if errwrite != -1 and errread != -1 and errwrite != devnull_fd:\n                os.close(errwrite)\n            if devnull_fd is not None:\n                os.close(devnull_fd)\n            # Prevent a double close of these fds from __init__ on error.\n            self._closed_child_pipe_fds = True\n    \n            # Wait for exec to fail or succeed; possibly raising an\n            # exception (limited in size)\n            errpipe_data = bytearray()\n            while True:\n                part = os.read(errpipe_read, 50000)\n                errpipe_data += part\n                if not part or len(errpipe_data) > 50000:\n                    break\n        finally:\n            # be sure the FD is closed no matter what\n            os.close(errpipe_read)\n    \n        if errpipe_data:\n            try:\n                pid, sts = os.waitpid(self.pid, 0)\n                if pid == self.pid:\n                    self._handle_exitstatus(sts)\n                else:\n                    self.returncode = sys.maxsize\n            except ChildProcessError:\n                pass\n    \n            try:\n                exception_name, hex_errno, err_msg = (\n                        errpipe_data.split(b':', 2))\n                # The encoding here should match the encoding\n                # written in by the subprocess implementations\n                # like _posixsubprocess\n                err_msg = err_msg.decode()\n            except ValueError:\n                exception_name = b'SubprocessError'\n                hex_errno = b'0'\n                err_msg = 'Bad exception data from child: {!r}'.format(\n                              bytes(errpipe_data))\n            child_exception_type = getattr(\n                    builtins, exception_name.decode('ascii'),\n                    SubprocessError)\n            if issubclass(child_exception_type, OSError) and hex_errno:\n                errno_num = int(hex_errno, 16)\n                child_exec_never_called = (err_msg == \"noexec\")\n                if child_exec_never_called:\n                    err_msg = \"\"\n                    # The error must be from chdir(cwd).\n                    err_filename = cwd\n                else:\n                    err_filename = orig_executable\n                if errno_num != 0:\n                    err_msg = os.strerror(errno_num)\n                    if errno_num == errno.ENOENT:\n                        err_msg += ': ' + repr(err_filename)\n>               raise child_exception_type(errno_num, err_msg, err_filename)\nE               FileNotFoundError: [Errno 2] No such file or directory: 'datalad': 'datalad'\n\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/subprocess.py:1551: FileNotFoundError\n______________________________ test_cfg_override _______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexvbvv2tup'}\nfilename = '/tmp/datalad_temp_test_cfg_overridey3m5okkq'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_cfg_override() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:615: TypeError\n==================================== PASSES ====================================\n_______________________ test_usage_on_insufficient_args ________________________\n----------------------------- Captured stderr call -----------------------------\n[ERROR] Please provide at least a source or a path [install.py:__call__:167] (InsufficientArgumentsError) \n------------------------------ Captured log call -------------------------------\nERROR    datalad.cmdline:main.py:502 Please provide at least a source or a path [install.py:__call__:167] (InsufficientArgumentsError)\n______________________ test_subcmd_usage_on_unknown_args _______________________\n----------------------------- Captured stderr call -----------------------------\n[ERROR] unknown argument: --murks \n------------------------------ Captured log call -------------------------------\nERROR    datalad.cmdline:main.py:456 unknown argument: --murks\n_____________________________ test_fix_datalad_ri ______________________________\n----------------------------- Captured stderr call -----------------------------\n[INFO] Changing // back to /// as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion \n[INFO] Changing //a back to ///a as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion \n[INFO] Changing //a/b back to ///a/b as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion \n------------------------------ Captured log call -------------------------------\nINFO     datalad.cmdline:main.py:429 Changing // back to /// as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion\nINFO     datalad.cmdline:main.py:429 Changing //a back to ///a as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion\nINFO     datalad.cmdline:main.py:429 Changing //a/b back to ///a/b as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_version\nPASSED r2e_tests/test_1.py::test_help_np\nPASSED r2e_tests/test_1.py::test_usage_on_insufficient_args\nPASSED r2e_tests/test_1.py::test_subcmd_usage_on_unknown_args\nPASSED r2e_tests/test_1.py::test_combined_short_option\nPASSED r2e_tests/test_1.py::test_fail_with_short_help\nPASSED r2e_tests/test_1.py::test_fix_datalad_ri\nXFAIL r2e_tests/test_1.py::test_incorrect_options - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_incorrect_options will be ignored\nFAILED r2e_tests/test_1.py::test_script_shims - FileNotFoundError: [Errno 2] ...\nFAILED r2e_tests/test_1.py::test_cfg_override - TypeError: test_cfg_override(...\n==================== 2 failed, 7 passed, 1 xfailed in 1.30s ====================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ade51ea0dab18977814b9f09beedc77b52f2e9e0\nplugins: cov-4.1.0\ncollected 10 items\n\nr2e_tests/test_1.py .....xFFF.                                           [100%]\n\n=================================== FAILURES ===================================\n______________________________ test_script_shims _______________________________\n\n    def test_script_shims():\n        runner = Runner()\n        for script in [\n            'datalad',\n            'git-annex-remote-datalad-archives',\n            'git-annex-remote-datalad']:\n            if not on_windows:\n                # those must be available for execution, and should not contain\n                which, _ = runner(['which', script])\n                # test if there is no easy install shim in there\n                with open(which.rstrip()) as f:\n                    content = f.read()\n            else:\n                from distutils.spawn import find_executable\n                content = find_executable(script)\n            assert_not_in('EASY', content) # NOTHING easy should be there\n            assert_not_in('pkg_resources', content)\n    \n            # and let's check that it is our script\n>           out, err = runner([script, '--version'])\n\nr2e_tests/test_1.py:192: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ndatalad/cmd.py:187: in __call__\n    return self.run(cmd, *args, **kwargs)\ndatalad/cmd.py:487: in run\n    stdin=stdin)\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/subprocess.py:800: in __init__\n    restore_signals, start_new_session)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = <subprocess.Popen object at 0x7428fbbadc50>\nargs = ['datalad', '--version'], executable = b'datalad', preexec_fn = None\nclose_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None\ncreationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = 13\nc2pwrite = 15, errread = 16, errwrite = 17, restore_signals = True\nstart_new_session = False\n\n    def _execute_child(self, args, executable, preexec_fn, close_fds,\n                       pass_fds, cwd, env,\n                       startupinfo, creationflags, shell,\n                       p2cread, p2cwrite,\n                       c2pread, c2pwrite,\n                       errread, errwrite,\n                       restore_signals, start_new_session):\n        \"\"\"Execute program (POSIX version)\"\"\"\n    \n        if isinstance(args, (str, bytes)):\n            args = [args]\n        else:\n            args = list(args)\n    \n        if shell:\n            # On Android the default shell is at '/system/bin/sh'.\n            unix_shell = ('/system/bin/sh' if\n                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')\n            args = [unix_shell, \"-c\"] + args\n            if executable:\n                args[0] = executable\n    \n        if executable is None:\n            executable = args[0]\n        orig_executable = executable\n    \n        # For transferring possible exec failure from child to parent.\n        # Data format: \"exception name:hex errno:description\"\n        # Pickle is not used; it is complex and involves memory allocation.\n        errpipe_read, errpipe_write = os.pipe()\n        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.\n        low_fds_to_close = []\n        while errpipe_write < 3:\n            low_fds_to_close.append(errpipe_write)\n            errpipe_write = os.dup(errpipe_write)\n        for low_fd in low_fds_to_close:\n            os.close(low_fd)\n        try:\n            try:\n                # We must avoid complex work that could involve\n                # malloc or free in the child process to avoid\n                # potential deadlocks, thus we do all this here.\n                # and pass it to fork_exec()\n    \n                if env is not None:\n                    env_list = []\n                    for k, v in env.items():\n                        k = os.fsencode(k)\n                        if b'=' in k:\n                            raise ValueError(\"illegal environment variable name\")\n                        env_list.append(k + b'=' + os.fsencode(v))\n                else:\n                    env_list = None  # Use execv instead of execve.\n                executable = os.fsencode(executable)\n                if os.path.dirname(executable):\n                    executable_list = (executable,)\n                else:\n                    # This matches the behavior of os._execvpe().\n                    executable_list = tuple(\n                        os.path.join(os.fsencode(dir), executable)\n                        for dir in os.get_exec_path(env))\n                fds_to_keep = set(pass_fds)\n                fds_to_keep.add(errpipe_write)\n                self.pid = _posixsubprocess.fork_exec(\n                        args, executable_list,\n                        close_fds, tuple(sorted(map(int, fds_to_keep))),\n                        cwd, env_list,\n                        p2cread, p2cwrite, c2pread, c2pwrite,\n                        errread, errwrite,\n                        errpipe_read, errpipe_write,\n                        restore_signals, start_new_session, preexec_fn)\n                self._child_created = True\n            finally:\n                # be sure the FD is closed no matter what\n                os.close(errpipe_write)\n    \n            # self._devnull is not always defined.\n            devnull_fd = getattr(self, '_devnull', None)\n            if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:\n                os.close(p2cread)\n            if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:\n                os.close(c2pwrite)\n            if errwrite != -1 and errread != -1 and errwrite != devnull_fd:\n                os.close(errwrite)\n            if devnull_fd is not None:\n                os.close(devnull_fd)\n            # Prevent a double close of these fds from __init__ on error.\n            self._closed_child_pipe_fds = True\n    \n            # Wait for exec to fail or succeed; possibly raising an\n            # exception (limited in size)\n            errpipe_data = bytearray()\n            while True:\n                part = os.read(errpipe_read, 50000)\n                errpipe_data += part\n                if not part or len(errpipe_data) > 50000:\n                    break\n        finally:\n            # be sure the FD is closed no matter what\n            os.close(errpipe_read)\n    \n        if errpipe_data:\n            try:\n                pid, sts = os.waitpid(self.pid, 0)\n                if pid == self.pid:\n                    self._handle_exitstatus(sts)\n                else:\n                    self.returncode = sys.maxsize\n            except ChildProcessError:\n                pass\n    \n            try:\n                exception_name, hex_errno, err_msg = (\n                        errpipe_data.split(b':', 2))\n                # The encoding here should match the encoding\n                # written in by the subprocess implementations\n                # like _posixsubprocess\n                err_msg = err_msg.decode()\n            except ValueError:\n                exception_name = b'SubprocessError'\n                hex_errno = b'0'\n                err_msg = 'Bad exception data from child: {!r}'.format(\n                              bytes(errpipe_data))\n            child_exception_type = getattr(\n                    builtins, exception_name.decode('ascii'),\n                    SubprocessError)\n            if issubclass(child_exception_type, OSError) and hex_errno:\n                errno_num = int(hex_errno, 16)\n                child_exec_never_called = (err_msg == \"noexec\")\n                if child_exec_never_called:\n                    err_msg = \"\"\n                    # The error must be from chdir(cwd).\n                    err_filename = cwd\n                else:\n                    err_filename = orig_executable\n                if errno_num != 0:\n                    err_msg = os.strerror(errno_num)\n                    if errno_num == errno.ENOENT:\n                        err_msg += ': ' + repr(err_filename)\n>               raise child_exception_type(errno_num, err_msg, err_filename)\nE               FileNotFoundError: [Errno 2] No such file or directory: 'datalad': 'datalad'\n\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/subprocess.py:1551: FileNotFoundError\n______________________________ test_cfg_override _______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexs9o3iqrr'}\nfilename = '/tmp/datalad_temp_test_cfg_overrides643t43s'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_cfg_override() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:615: TypeError\n__________________________ test_fail_with_short_help ___________________________\n\n    def test_fail_with_short_help():\n        out = StringIO()\n        with assert_raises(SystemExit) as cme:\n            fail_with_short_help(exit_code=3, out=out)\n        assert_equal(cme.exception.code, 3)\n        assert_equal(out.getvalue(), \"\")\n    \n        out = StringIO()\n        with assert_raises(SystemExit) as cme:\n            fail_with_short_help(msg=\"Failed badly\", out=out)\n        assert_equal(cme.exception.code, 1)\n        assert_equal(out.getvalue(), \"error: Failed badly\\n\")\n    \n        # Suggestions, hint, etc\n        out = StringIO()\n        with assert_raises(SystemExit) as cme:\n            fail_with_short_help(\n                msg=\"Failed badly\",\n                known=[\"mother\", \"mutter\", \"father\", \"son\"],\n                provided=\"muther\",\n                hint=\"You can become one\",\n                exit_code=0,  # noone forbids\n                what=\"parent\",\n                out=out)\n        assert_equal(cme.exception.code, 0)\n        assert_equal(out.getvalue(),\n>                    \"error: Failed badly\\n\"\n                     \"datalad: Unknown parent 'muther'.  See 'datalad --help'.\\n\\n\"\n                     \"Did you mean any of these?\\n\"\n                     \"        mutter\\n\"\n                     \"        mother\\n\"\n                     \"        father\\n\"\n                     \"Hint: You can become one\\n\")\n\nr2e_tests/test_1.py:269: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/unittest/case.py:852: in assertEqual\n    assertion_func(first, second, msg=msg)\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/unittest/case.py:1233: in assertMultiLineEqual\n    self.fail(self._formatMessage(msg, standardMsg))\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = <nose.tools.trivial.Dummy testMethod=nop>\nmsg = '\"erro[85 chars]mean one of these?\\\\n        mutter\\\\n        mo[44 chars]ne\\\\n\" != \"erro[85 chars]mean any of these?\\... any of these?\\n?              ^ ^\\n          mutter\\n          mother\\n          father\\n  Hint: You can become one\\n'\n\n    def fail(self, msg=None):\n        \"\"\"Fail immediately, with the given message.\"\"\"\n>       raise self.failureException(msg)\nE       AssertionError: \"erro[85 chars]mean one of these?\\n        mutter\\n        mo[44 chars]ne\\n\" != \"erro[85 chars]mean any of these?\\n        mutter\\n        mo[44 chars]ne\\n\"\nE         error: Failed badly\nE         datalad: Unknown parent 'muther'.  See 'datalad --help'.\nE         \nE       - Did you mean one of these?\nE       ?              ^ ^\nE       + Did you mean any of these?\nE       ?              ^ ^\nE                 mutter\nE                 mother\nE                 father\nE         Hint: You can become one\n\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/unittest/case.py:693: AssertionError\n==================================== PASSES ====================================\n_______________________ test_usage_on_insufficient_args ________________________\n----------------------------- Captured stderr call -----------------------------\n[ERROR] Please provide at least a source or a path [install.py:__call__:167] (InsufficientArgumentsError) \n------------------------------ Captured log call -------------------------------\nERROR    datalad.cmdline:main.py:503 Please provide at least a source or a path [install.py:__call__:167] (InsufficientArgumentsError)\n______________________ test_subcmd_usage_on_unknown_args _______________________\n----------------------------- Captured stderr call -----------------------------\n[ERROR] unknown argument: --murks \n------------------------------ Captured log call -------------------------------\nERROR    datalad.cmdline:main.py:457 unknown argument: --murks\n_____________________________ test_fix_datalad_ri ______________________________\n----------------------------- Captured stderr call -----------------------------\n[INFO] Changing // back to /// as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion \n[INFO] Changing //a back to ///a as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion \n[INFO] Changing //a/b back to ///a/b as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion \n------------------------------ Captured log call -------------------------------\nINFO     datalad.cmdline:main.py:430 Changing // back to /// as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion\nINFO     datalad.cmdline:main.py:430 Changing //a back to ///a as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion\nINFO     datalad.cmdline:main.py:430 Changing //a/b back to ///a/b as it was probably changed by MINGW/MSYS, see http://www.mingw.org/wiki/Posix_path_conversion\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_version\nPASSED r2e_tests/test_1.py::test_help_np\nPASSED r2e_tests/test_1.py::test_usage_on_insufficient_args\nPASSED r2e_tests/test_1.py::test_subcmd_usage_on_unknown_args\nPASSED r2e_tests/test_1.py::test_combined_short_option\nPASSED r2e_tests/test_1.py::test_fix_datalad_ri\nXFAIL r2e_tests/test_1.py::test_incorrect_options - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_incorrect_options will be ignored\nFAILED r2e_tests/test_1.py::test_script_shims - FileNotFoundError: [Errno 2] ...\nFAILED r2e_tests/test_1.py::test_cfg_override - TypeError: test_cfg_override(...\nFAILED r2e_tests/test_1.py::test_fail_with_short_help - AssertionError: \"erro...\n==================== 3 failed, 6 passed, 1 xfailed in 0.84s ====================\n",
    "old_commit_res_stderr": ""
}
datalad/cmdline/main.py datalad/cmdline/tests/test_main.py datalad/utils.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Suggestion Message Uses "Any" Instead of "One", Causing Output Inconsistencies

**Description:**
After updating the suggestion message to use "any" instead of "one" when providing suggestions for unknown arguments, the output no longer matches the expected format. This change leads to inconsistencies, especially in scenarios where multiple suggestions are provided.

**Example:**
When an unknown parent `'muther'` is provided, the output is:

```
error: Failed badly
datalad: Unknown parent 'muther'.  See 'datalad --help'.

Did you mean any of these?
        mutter
        mother
        father
Hint: You can become one
```

Previously, the message used "one of these," which aligned with the expected output format.

**Expected Behavior:**
The suggestion message should consistently use "any of these" to accurately reflect the possibility of multiple suggestions. This ensures that the output remains clear and aligns with scenarios where multiple terms are queried simultaneously.

**Actual Behavior:**
The message now uses "any of these," which differs from the original phrasing. This discrepancy can cause confusion and may lead to failures in processes or tests that expect the original "one of these" wording.

**Code Example:**
```python
out = StringIO()
fail_with_short_help(
    msg='Failed badly',
    known=['mother', 'mutter', 'father', 'son'],
    provided='muther',
    hint='You can become one',
    exit_code=0,
    what='parent',
    out=out
)
print(out.getvalue())
```

**Output:**
```
error: Failed badly
datalad: Unknown parent 'muther'.  See 'datalad --help'.

Did you mean any of these?
        mutter
        mother
        father
Hint: You can become one
```
[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/cmdline/main.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
""""""
from datalad.ui.utils import get_console_width

__docformat__ = 'restructuredtext'

import logging
lgr = logging.getLogger('datalad.cmdline')

lgr.log(5, "Importing cmdline.main")

import argparse
from collections import defaultdict
import sys
import textwrap
import os

from six import text_type

import datalad

from datalad.cmdline import helpers
from datalad.support.exceptions import InsufficientArgumentsError
from datalad.support.exceptions import IncompleteResultsError
from datalad.support.exceptions import CommandError
from .helpers import strip_arg_from_argv
from ..utils import setup_exceptionhook, chpwd
from ..utils import assure_unicode
from ..utils import on_msys_tainted_paths
from ..dochelpers import exc_str


def _license_info():
    return """\
Copyright (c) 2013-2018 DataLad developers

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""


class ArgumentParserDisableAbbrev(argparse.ArgumentParser):
    # Don't accept abbreviations for long options. With py3.5 and above, we
    # could just use allow_abbrev=False.
    #
    # Modified from the solution posted at
    # https://bugs.python.org/issue14910#msg204678
    def _get_option_tuples(self, option_string):
        chars = self.prefix_chars
        if option_string[0] in chars and option_string[1] in chars:
            # option_string is a long flag. Disable abbreviation.
            return []
        return super(ArgumentParserDisableAbbrev, self)._get_option_tuples(
            option_string)


# TODO:  OPT look into making setup_parser smarter to become faster
# Now it seems to take up to 200ms to do all the parser setup
# even though it might not be necessary to know about all the commands etc.
# I wondered if it could somehow decide on what commands to worry about etc
# by going through sys.args first
def setup_parser(
        cmdlineargs,
        formatter_class=argparse.RawDescriptionHelpFormatter,
        return_subparsers=False):
    lgr.log(5, "Starting to setup_parser")
    # delay since it can be a heavy import
    from ..interface.base import dedent_docstring, get_interface_groups, \
        get_cmdline_command_name, alter_interface_docs_for_cmdline, \
        load_interface, get_cmd_doc
    # setup cmdline args parser
    parts = {}
    # main parser
    parser = ArgumentParserDisableAbbrev(
        fromfile_prefix_chars=None,
        # usage="%(prog)s ...",
        description=dedent_docstring("""\
            Comprehensive data management solution

            DataLad provides a unified data distribution system built on the Git
            and Git-annex. DataLad command line tools allow to manipulate (obtain,
            create, update, publish, etc.) datasets and provide a comprehensive
            toolbox for joint management of data and code. Compared to Git/annex
            it primarly extends their functionality to transparently and
            simultaneously work with multiple inter-related repositories."""),
        epilog='"Be happy!"',
        formatter_class=formatter_class,
        add_help=False)
    # common options
    helpers.parser_add_common_opt(parser, 'log_level')
    helpers.parser_add_common_opt(parser, 'pbs_runner')
    helpers.parser_add_common_opt(parser, 'change_path')
    helpers.parser_add_common_opt(
        parser,
        'version',
        version='datalad %s\n' % datalad.__version__)
    if __debug__:
        parser.add_argument(
            '--dbg', action='store_true', dest='common_debug',
            help="enter Python debugger when uncaught exception happens")
        parser.add_argument(
            '--idbg', action='store_true', dest='common_idebug',
            help="enter IPython debugger when uncaught exception happens")
    parser.add_argument(
        '-c', action='append', dest='cfg_overrides', metavar='KEY=VALUE',
        help="""configuration variable setting. Overrides any configuration
        read from a file, but is potentially overridden itself by configuration
        variables in the process environment.""")
    parser.add_argument(
        '-f', '--output-format', dest='common_output_format',
        default='default',
        type=assure_unicode,
        metavar="{default,json,json_pp,tailored,'<template>'}",
        help="""select format for returned command results. 'default' give one line
        per result reporting action, status, path and an optional message;
        'json' renders a JSON object with all properties for each result (one per
        line); 'json_pp' pretty-prints JSON spanning multiple lines; 'tailored'
        enables a command-specific rendering style that is typically
        tailored to human consumption (no result output otherwise),
        '<template>' reports any value(s) of any result properties in any format
        indicated by the template (e.g. '{path}'; compare with JSON
        output for all key-value choices). The template syntax follows the Python
        "format() language". It is possible to report individual
        dictionary values, e.g. '{metadata[name]}'. If a 2nd-level key contains
        a colon, e.g. 'music:Genre', ':' must be substituted by '#' in the template,
        like so: '{metadata[music#Genre]}'.""")
    parser.add_argument(
        '--report-status', dest='common_report_status',
        choices=['success', 'failure', 'ok', 'notneeded', 'impossible', 'error'],
        help="""constrain command result report to records matching the given
        status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands
        for 'impossible' OR 'error'.""")
    parser.add_argument(
        '--report-type', dest='common_report_type',
        choices=['dataset', 'file'],
        action='append',
        help="""constrain command result report to records matching the given
        type. Can be given more than once to match multiple types.""")
    parser.add_argument(
        '--on-failure', dest='common_on_failure',
        choices=['ignore', 'continue', 'stop'],
        # no default: better be configure per-command
        help="""when an operation fails: 'ignore' and continue with remaining
        operations, the error is logged but does not lead to a non-zero exit code
        of the command; 'continue' works like 'ignore', but an error causes a
        non-zero exit code; 'stop' halts on first failure and yields non-zero exit
        code. A failure is any result with status 'impossible' or 'error'.""")
    parser.add_argument(
        '--proc-pre', dest='common_proc_pre',
        nargs='+',
        action='append',
        metavar=('<PROCEDURE NAME>', 'ARGS'),
        help="""Dataset procedure to run before the main command (see run-procedure
        command for details). This option can be given more than once to run
        multiple procedures in the order in which they were given.
        It is important to specify the target dataset via the --dataset argument
        of the main command."""),
    parser.add_argument(
        '--proc-post', dest='common_proc_post',
        nargs='+',
        action='append',
        metavar=('<PROCEDURE NAME>', 'ARGS'),
        help="""Like --proc-pre, but procedures are executed after the main command
        has finished."""),
    parser.add_argument(
        '--cmd', dest='_', action='store_true',
        help="""syntactical helper that can be used to end the list of global
        command line options before the subcommand label. Options like
        --proc-pre can take an arbitrary number of arguments and may require
        to be followed by a single --cmd in order to enable identification
        of the subcommand.""")

    # yoh: atm we only dump to console.  Might adopt the same separation later on
    #      and for consistency will call it --verbose-level as well for now
    # log-level is set via common_opts ATM
    # parser.add_argument('--log-level',
    #                     choices=('critical', 'error', 'warning', 'info', 'debug'),
    #                     dest='common_log_level',
    #                     help="""level of verbosity in log files. By default
    #                          everything, including debug messages is logged.""")
    #parser.add_argument('-l', '--verbose-level',
    #                    choices=('critical', 'error', 'warning', 'info', 'debug'),
    #                    dest='common_verbose_level',
    #                    help="""level of verbosity of console output. By default
    #                         only warnings and errors are printed.""")

    # Before doing anything additional and possibly expensive see may be that
    # we have got the command already
    need_single_subparser = False if return_subparsers else None
    fail_handler = (lambda *a, **kw: True) \
        if return_subparsers else fail_with_short_help
    try:
        parsed_args, unparsed_args = parser._parse_known_args(
            cmdlineargs[1:], argparse.Namespace())
        if not unparsed_args:
            fail_handler(parser, msg="too few arguments", exit_code=2)
        lgr.debug("Command line args 1st pass. Parsed: %s Unparsed: %s",
                  parsed_args, unparsed_args)
    except Exception as exc:
        lgr.debug("Early parsing failed with %s", exc_str(exc))
        need_single_subparser = False
        unparsed_args = cmdlineargs[1:]  # referenced before assignment otherwise

    interface_groups = get_interface_groups(include_plugins=True)

    # First unparsed could be either unknown option to top level "datalad"
    # or a command. Among unknown could be --help/--help-np which would
    # need to be dealt with
    unparsed_arg = unparsed_args[0] if unparsed_args else None
    if need_single_subparser is not None \
            or unparsed_arg in ('--help', '--help-np', '-h'):
        need_single_subparser = False
        add_entrypoints_to_interface_groups(interface_groups)
    elif unparsed_arg.startswith('-'):  # unknown option
        fail_with_short_help(parser,
                             msg="unrecognized argument %s" % unparsed_arg,
                             exit_code=2)
        # if we could get a list of options known to parser,
        # we could suggest them
        # known=get_all_options(parser), provided=unparsed_arg)
    else:  # the command to handle
        known_commands = get_commands_from_groups(interface_groups)
        if unparsed_arg not in known_commands:
            # need to load all the extensions and try again
            add_entrypoints_to_interface_groups(interface_groups)
            known_commands = get_commands_from_groups(interface_groups)

        if unparsed_arg not in known_commands:
            # check if might be coming from known extensions
            from ..interface import _known_extension_commands
            extension_commands = {
                c: e
                for e, commands in _known_extension_commands.items()
                for c in commands
            }
            hint = None
            if unparsed_arg in extension_commands:
                hint = "Command %s is provided by (not installed) extension %s." \
                      % (unparsed_arg, extension_commands[unparsed_arg])
            fail_with_short_help(
                parser,
                hint=hint,
                provided=unparsed_arg,
                known=list(known_commands.keys()) + list(extension_commands.keys())
            )
        if need_single_subparser is None:
            need_single_subparser = unparsed_arg

    # --help specification was delayed since it causes immediate printout of
    # --help output before we setup --help for each command
    helpers.parser_add_common_opt(parser, 'help')

    grp_short_descriptions = defaultdict(list)
    # create subparser, use module suffix as cmd name
    subparsers = parser.add_subparsers()
    for group_name, _, _interfaces \
            in sorted(interface_groups, key=lambda x: x[1]):
        for _intfspec in _interfaces:
            cmd_name = get_cmdline_command_name(_intfspec)
            if need_single_subparser and cmd_name != need_single_subparser:
                continue
            _intf = load_interface(_intfspec)
            if _intf is None:
                # TODO(yoh):  add doc why we could skip this one... makes this
                # loop harder to extract into a dedicated function
                continue
            # deal with optional parser args
            if hasattr(_intf, 'parser_args'):
                parser_args = _intf.parser_args
            else:
                parser_args = dict(formatter_class=formatter_class)
                # use class description, if no explicit description is available
                intf_doc = get_cmd_doc(_intf)
                parser_args['description'] = alter_interface_docs_for_cmdline(
                    intf_doc)
            subparser = subparsers.add_parser(cmd_name, add_help=False, **parser_args)
            # our own custom help for all commands
            helpers.parser_add_common_opt(subparser, 'help')
            # let module configure the parser
            _intf.setup_parser(subparser)
            # logger for command

            # configure 'run' function for this command
            plumbing_args = dict(
                func=_intf.call_from_parser,
                logger=logging.getLogger(_intf.__module__),
                subparser=subparser)
            if hasattr(_intf, 'result_renderer_cmdline'):
                plumbing_args['result_renderer'] = _intf.result_renderer_cmdline
            subparser.set_defaults(**plumbing_args)
            parts[cmd_name] = subparser
            # store short description for later
            sdescr = getattr(_intf, 'short_description',
                             parser_args['description'].split('\n')[0])
            grp_short_descriptions[group_name].append((cmd_name, sdescr))

    # create command summary
    if '--help' in cmdlineargs or '--help-np' in cmdlineargs:
        parser.description = get_description_with_cmd_summary(
            grp_short_descriptions,
            interface_groups,
            parser.description)

    parts['datalad'] = parser
    lgr.log(5, "Finished setup_parser")
    if return_subparsers:
        return parts
    else:
        return parser


def fail_with_short_help(parser=None,
                         msg=None,
                         known=None, provided=None,
                         hint=None,
                         exit_code=1,
                         what="command",
                         out=None):
    """Generic helper to fail
    with short help possibly hinting on what was intended if `known`
    were provided
    """
    out = out or sys.stderr
    if msg:
        out.write("error: %s\n" % msg)
    if not known:
        if parser:
            # just to appear in print_usage also consistent with --help output
            parser.add_argument("command [command-opts]")
            parser.print_usage(file=out)
    else:
        out.write(
            "datalad: Unknown %s %r.  See 'datalad --help'.\n\n"
            % (what, provided,))
        if provided not in known:
            import difflib
            suggestions = difflib.get_close_matches(provided, known)
            if suggestions:
                out.write(
                    "Did you mean one of these?\n        %s\n"
                    % "\n        ".join(suggestions))
        # Too noisy
        # sys.stderr.write(" All known %ss: %s\n"
        #                   % (what, ", ".join(sorted(known))))
    if hint:
        out.write("Hint: %s\n" % hint)
    raise SystemExit(exit_code)


def get_description_with_cmd_summary(grp_short_descriptions, interface_groups,
                                     parser_description):
    from ..interface.base import dedent_docstring
    from ..interface.base import get_cmd_summaries
    lgr.debug("Generating detailed description for the parser")

    console_width = get_console_width()
    cmd_summary = get_cmd_summaries(grp_short_descriptions, interface_groups,
                                    width=console_width)
    # we need one last formal section to not have the trailed be
    # confused with the last command group
    cmd_summary.append('\n*General information*\n')
    detailed_description = '%s\n%s\n\n%s' \
                           % (parser_description,
                              '\n'.join(cmd_summary),
                              textwrap.fill(dedent_docstring("""\
    Detailed usage information for individual commands is
    available via command-specific --help, i.e.:
    datalad <command> --help"""),
                                            console_width - 5,
                                            initial_indent='',
                                            subsequent_indent=''))
    return detailed_description


def get_commands_from_groups(groups):
    """Get a dictionary of command: interface_spec"""
    from ..interface.base import get_cmdline_command_name
    return {
        get_cmdline_command_name(_intfspec): _intfspec
        for _, _, _interfaces in groups
        for _intfspec in _interfaces
    }


def add_entrypoints_to_interface_groups(interface_groups):
    lgr.debug("Loading entrypoints")
    from pkg_resources import iter_entry_points  # delay expensive import
    for ep in iter_entry_points('datalad.extensions'):
        lgr.debug(
            'Loading entrypoint %s from datalad.extensions for docs building',
            ep.name)
        try:
            spec = ep.load()
            interface_groups.append((ep.name, spec[0], spec[1]))
            lgr.debug('Loaded entrypoint %s', ep.name)
        except Exception as e:
            lgr.warning('Failed to load entrypoint %s: %s', ep.name, exc_str(e))
            continue

def _fix_datalad_ri(s):
    """Fixup argument if it was a DataLadRI and had leading / removed

    See gh-2643
    """
    if s.startswith('//') and (len(s) == 2 or (len(s) > 2 and s[2] != '/')):
        lgr.info(
            "Changing %s back to /%s as it was probably changed by MINGW/MSYS, "
            "see http://www.mingw.org/wiki/Posix_path_conversion", s, s)
        return "/" + s
    return s


def main(args=None):
    lgr.log(5, "Starting main(%r)", args)
    args = args or sys.argv
    if on_msys_tainted_paths:
        # Possibly present DataLadRIs were stripped of a leading /
        args = [_fix_datalad_ri(s) for s in args]
    # PYTHON_ARGCOMPLETE_OK
    parser = setup_parser(args)
    try:
        import argcomplete
        argcomplete.autocomplete(parser)
    except ImportError:
        pass

    # parse cmd args
    lgr.debug("Parsing known args among %s", repr(args))
    cmdlineargs, unparsed_args = parser.parse_known_args(args[1:])
    has_func = hasattr(cmdlineargs, 'func') and cmdlineargs.func is not None
    if unparsed_args:
        if has_func and cmdlineargs.func.__self__.__name__ != 'Export':
            lgr.error('unknown argument{}: {}'.format(
                's' if len(unparsed_args) > 1 else '',
                unparsed_args if len(unparsed_args) > 1 else unparsed_args[0]))
            cmdlineargs.subparser.print_usage()
            sys.exit(1)
        else:
            # store all unparsed arguments
            cmdlineargs.datalad_unparsed_args = unparsed_args

    # to possibly be passed into PBS scheduled call
    args_ = args or sys.argv

    if cmdlineargs.cfg_overrides is not None:
        overrides = dict([
            (o.split('=')[0], '='.join(o.split('=')[1:]))
            for o in cmdlineargs.cfg_overrides])
        datalad.cfg.overrides.update(overrides)

    # enable overrides
    datalad.cfg.reload(force=True)

    if cmdlineargs.change_path is not None:
        from .common_args import change_path as change_path_opt
        for path in cmdlineargs.change_path:
            chpwd(path)
            args_ = strip_arg_from_argv(args_, path, change_path_opt[1])

    ret = None
    if cmdlineargs.pbs_runner:
        from .helpers import run_via_pbs
        from .common_args import pbs_runner as pbs_runner_opt
        args_ = strip_arg_from_argv(args_, cmdlineargs.pbs_runner, pbs_runner_opt[1])
        # run the function associated with the selected command
        run_via_pbs(args_, cmdlineargs.pbs_runner)
    elif has_func:
        if cmdlineargs.common_debug or cmdlineargs.common_idebug:
            # so we could see/stop clearly at the point of failure
            setup_exceptionhook(ipython=cmdlineargs.common_idebug)
            from datalad.interface.base import Interface
            Interface._interrupted_exit_code = None
            ret = cmdlineargs.func(cmdlineargs)
        else:
            # otherwise - guard and only log the summary. Postmortem is not
            # as convenient if being caught in this ultimate except
            try:
                ret = cmdlineargs.func(cmdlineargs)
            except InsufficientArgumentsError as exc:
                # if the func reports inappropriate usage, give help output
                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))
                cmdlineargs.subparser.print_usage(sys.stderr)
                sys.exit(2)
            except IncompleteResultsError as exc:
                # rendering for almost all commands now happens 'online'
                # hence we are no longer attempting to render the actual
                # results in an IncompleteResultsError, ubt rather trust that
                # this happened before

                # in general we do not want to see the error again, but
                # present in debug output
                lgr.debug('could not perform all requested actions: %s',
                          exc_str(exc))
                sys.exit(1)
            except CommandError as exc:
                # behave as if the command ran directly, importantly pass
                # exit code as is
                exc_msg = str(exc)
                if exc_msg:
                    msg = exc_msg.encode() if isinstance(exc_msg, text_type) else exc_msg
                    os.write(2, msg + b"\n")
                if exc.stdout:
                    os.write(1, exc.stdout.encode() if isinstance(exc.stdout, text_type) else exc.stdout)
                if exc.stderr:
                    os.write(2, exc.stderr.encode() if isinstance(exc.stderr, text_type) else exc.stderr)
                # We must not exit with 0 code if any exception got here but
                # had no code defined
                sys.exit(exc.code if exc.code is not None else 1)
            except Exception as exc:
                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))
                sys.exit(1)
    else:
        # just let argparser spit out its error, since there is smth wrong
        parser.parse_args(args)
        # if that one didn't puke -- we should
        parser.print_usage()
        lgr.error("Please specify the command")
        sys.exit(2)

    try:
        if hasattr(cmdlineargs, 'result_renderer'):
            cmdlineargs.result_renderer(ret, cmdlineargs)
    except Exception as exc:
        lgr.error("Failed to render results due to %s", exc_str(exc))
        sys.exit(1)

lgr.log(5, "Done importing cmdline.main")


### datalad/utils.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##

import collections
import hashlib
import re
import six.moves.builtins as __builtin__
import time

import logging
import shutil
import os
import sys
import tempfile
import platform
import gc
import glob
import gzip
import string
import wrapt

from copy import copy as shallow_copy
from contextlib import contextmanager
from functools import wraps
from time import sleep
from inspect import getargspec
from itertools import tee

import os.path as op
from os.path import sep as dirsep
from os.path import commonprefix
from os.path import curdir, basename, exists, realpath, islink, join as opj
from os.path import isabs, normpath, expandvars, expanduser, abspath, sep
from os.path import isdir
from os.path import relpath
from os.path import stat
from os.path import dirname
from os.path import split as psplit
import posixpath


from six import PY2, text_type, binary_type, string_types

# from datalad.dochelpers import get_docstring_split
from datalad.consts import TIMESTAMP_FMT


if PY2:
    unicode_srctypes = string_types
else:
    unicode_srctypes = string_types + (bytes,)


lgr = logging.getLogger("datalad.utils")

lgr.log(5, "Importing datalad.utils")
#
# Some useful variables
#
platform_system = platform.system().lower()
on_windows = platform_system == 'windows'
on_osx = platform_system == 'darwin'
on_linux = platform_system == 'linux'
on_msys_tainted_paths = on_windows \
                        and 'MSYS_NO_PATHCONV' not in os.environ \
                        and os.environ.get('MSYSTEM', '')[:4] in ('MSYS', 'MING')
try:
    linux_distribution_name, linux_distribution_release \
        = platform.linux_distribution()[:2]
    on_debian_wheezy = on_linux \
                       and linux_distribution_name == 'debian' \
                       and linux_distribution_release.startswith('7.')
except:  # pragma: no cover
    # MIH: IndexError?
    on_debian_wheezy = False
    linux_distribution_name = linux_distribution_release = None

# Maximal length of cmdline string
# Query the system and use hardcoded "knowledge" if None
# probably   getconf ARG_MAX   might not be available
# The last one would be the most conservative/Windows
CMD_MAX_ARG_HARDCODED = 2097152 if on_linux else 262144 if on_osx else 32767
try:
    CMD_MAX_ARG = os.sysconf('SC_ARG_MAX')
    assert CMD_MAX_ARG > 0
except Exception as exc:
    # ATM (20181005) SC_ARG_MAX available only on POSIX systems
    # so exception would be thrown e.g. on Windows, or
    # somehow during Debian build for nd14.04 it is coming up with -1:
    # https://github.com/datalad/datalad/issues/3015
    CMD_MAX_ARG = CMD_MAX_ARG_HARDCODED
    lgr.debug(
        "Failed to query or got useless SC_ARG_MAX sysconf, "
        "will use hardcoded value: %s", exc)
# Even with all careful computations we do, due to necessity to account for
# environment and what not, we still could not figure out "exact" way to
# estimate it, but it was shown that 300k safety margin on linux was sufficient.
# https://github.com/datalad/datalad/pull/2977#issuecomment-436264710
# 300k is ~15%, so to be safe, and for paranoid us we will just use up to 50%
# of the length for "safety margin".  We might probably still blow due to
# env vars, unicode, etc...  so any hard limit imho is not a proper solution
CMD_MAX_ARG = int(0.5 * CMD_MAX_ARG)
lgr.debug(
    "Maximal length of cmdline string (adjusted for safety margin): %d",
    CMD_MAX_ARG)

#
# Little helpers
#


def get_func_kwargs_doc(func):
    """ Provides args for a function

    Parameters
    ----------
    func: str
      name of the function from which args are being requested

    Returns
    -------
    list
      of the args that a function takes in
    """
    return getargspec(func)[0]

    # TODO: format error message with descriptions of args
    # return [repr(dict(get_docstring_split(func)[1]).get(x)) for x in getargspec(func)[0]]


def any_re_search(regexes, value):
    """Return if any of regexes (list or str) searches succesfully for value"""
    for regex in assure_tuple_or_list(regexes):
        if re.search(regex, value):
            return True
    return False


def not_supported_on_windows(msg=None):
    """A little helper to be invoked to consistently fail whenever functionality is
    not supported (yet) on Windows
    """
    if on_windows:
        raise NotImplementedError("This functionality is not yet implemented for Windows OS"
                                  + (": %s" % msg if msg else ""))


def shortened_repr(value, l=30):
    try:
        if hasattr(value, '__repr__') and (value.__repr__ is not object.__repr__):
            value_repr = repr(value)
            if not value_repr.startswith('<') and len(value_repr) > l:
                value_repr = "<<%s...>>" % (value_repr[:l - 8])
            elif value_repr.startswith('<') and value_repr.endswith('>') and ' object at 0x':
                raise ValueError("I hate those useless long reprs")
        else:
            raise ValueError("gimme class")
    except Exception as e:
        value_repr = "<%s>" % value.__class__.__name__.split('.')[-1]
    return value_repr


def __auto_repr__(obj):
    attr_names = tuple()
    if hasattr(obj, '__dict__'):
        attr_names += tuple(obj.__dict__.keys())
    if hasattr(obj, '__slots__'):
        attr_names += tuple(obj.__slots__)

    items = []
    for attr in sorted(set(attr_names)):
        if attr.startswith('_'):
            continue
        value = getattr(obj, attr)
        # TODO:  should we add this feature to minimize some talktative reprs
        # such as of URL?
        #if value is None:
        #    continue
        items.append("%s=%s" % (attr, shortened_repr(value)))

    return "%s(%s)" % (obj.__class__.__name__, ', '.join(items))


def auto_repr(cls):
    """Decorator for a class to assign it an automagic quick and dirty __repr__

    It uses public class attributes to prepare repr of a class

    Original idea: http://stackoverflow.com/a/27799004/1265472
    """

    cls.__repr__ = __auto_repr__
    return cls


def is_interactive():
    """Return True if all in/outs are tty"""
    # TODO: check on windows if hasattr check would work correctly and add value:
    #
    return sys.stdin.isatty() and sys.stdout.isatty() and sys.stderr.isatty()


def get_ipython_shell():
    """Detect if running within IPython and returns its `ip` (shell) object

    Returns None if not under ipython (no `get_ipython` function)
    """
    try:
        return get_ipython()
    except NameError:
        return None


def md5sum(filename):
    with open(filename, 'rb') as f:
        return hashlib.md5(f.read()).hexdigest()


def sorted_files(dout):
    """Return a (sorted) list of files under dout
    """
    return sorted(sum([[opj(r, f)[len(dout) + 1:] for f in files]
                       for r, d, files in os.walk(dout)
                       if not '.git' in r], []))

_VCS_REGEX = '%s\.(?:git|gitattributes|svn|bzr|hg)(?:%s|$)' % (dirsep, dirsep)
_DATALAD_REGEX = '%s\.(?:datalad)(?:%s|$)' % (dirsep, dirsep)


def find_files(regex, topdir=curdir, exclude=None, exclude_vcs=True, exclude_datalad=False, dirs=False):
    """Generator to find files matching regex

    Parameters
    ----------
    regex: basestring
    exclude: basestring, optional
      Matches to exclude
    exclude_vcs:
      If True, excludes commonly known VCS subdirectories.  If string, used
      as regex to exclude those files (regex: `%r`)
    exclude_datalad:
      If True, excludes files known to be datalad meta-data files (e.g. under
      .datalad/ subdirectory) (regex: `%r`)
    topdir: basestring, optional
      Directory where to search
    dirs: bool, optional
      Whether to match directories as well as files
    """

    for dirpath, dirnames, filenames in os.walk(topdir):
        names = (dirnames + filenames) if dirs else filenames
        # TODO: might want to uniformize on windows to use '/'
        paths = (opj(dirpath, name) for name in names)
        for path in filter(re.compile(regex).search, paths):
            path = path.rstrip(dirsep)
            if exclude and re.search(exclude, path):
                continue
            if exclude_vcs and re.search(_VCS_REGEX, path):
                continue
            if exclude_datalad and re.search(_DATALAD_REGEX, path):
                continue
            yield path
find_files.__doc__ %= (_VCS_REGEX, _DATALAD_REGEX)


def expandpath(path, force_absolute=True):
    """Expand all variables and user handles in a path.

    By default return an absolute path
    """
    path = expandvars(expanduser(path))
    if force_absolute:
        path = abspath(path)
    return path


def posix_relpath(path, start=None):
    """Behave like os.path.relpath, but always return POSIX paths...

    on any platform."""
    # join POSIX style
    return posixpath.join(
        # split and relpath native style
        # python2.7 ntpath implementation of relpath cannot handle start=None
        *psplit(
            relpath(path, start=start if start is not None else '')))


def is_explicit_path(path):
    """Return whether a path explicitly points to a location

    Any absolute path, or relative path starting with either '../' or
    './' is assumed to indicate a location on the filesystem. Any other
    path format is not considered explicit."""
    path = expandpath(path, force_absolute=False)
    return isabs(path) \
        or path.startswith(os.curdir + os.sep) \
        or path.startswith(os.pardir + os.sep)


def rotree(path, ro=True, chmod_files=True):
    """To make tree read-only or writable

    Parameters
    ----------
    path : string
      Path to the tree/directory to chmod
    ro : bool, optional
      Whether to make it R/O (default) or RW
    chmod_files : bool, optional
      Whether to operate also on files (not just directories)
    """
    if ro:
        chmod = lambda f: os.chmod(f, os.stat(f).st_mode & ~stat.S_IWRITE)
    else:
        chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)

    for root, dirs, files in os.walk(path, followlinks=False):
        if chmod_files:
            for f in files:
                fullf = opj(root, f)
                # might be the "broken" symlink which would fail to stat etc
                if exists(fullf):
                    chmod(fullf)
        chmod(root)


def rmtree(path, chmod_files='auto', children_only=False, *args, **kwargs):
    """To remove git-annex .git it is needed to make all files and directories writable again first

    Parameters
    ----------
    chmod_files : string or bool, optional
       Whether to make files writable also before removal.  Usually it is just
       a matter of directories to have write permissions.
       If 'auto' it would chmod files on windows by default
    children_only : bool, optional
       If set, all files and subdirectories would be removed while the path
       itself (must be a directory) would be preserved
    `*args` :
    `**kwargs` :
       Passed into shutil.rmtree call
    """
    # Give W permissions back only to directories, no need to bother with files
    if chmod_files == 'auto':
        chmod_files = on_windows
    # TODO:  yoh thinks that if we could quickly check our Flyweight for
    #        repos if any of them is under the path, and could call .precommit
    #        on those to possibly stop batched processes etc, we did not have
    #        to do it on case by case
    # Check for open files
    assert_no_open_files(path)

    if children_only:
        if not os.path.isdir(path):
            raise ValueError("Can remove children only of directories")
        for p in os.listdir(path):
            rmtree(op.join(path, p))
        return
    if not (os.path.islink(path) or not os.path.isdir(path)):
        rotree(path, ro=False, chmod_files=chmod_files)
        _rmtree(path, *args, **kwargs)
    else:
        # just remove the symlink
        unlink(path)


def rmdir(path, *args, **kwargs):
    """os.rmdir with our optional checking for open files"""
    assert_no_open_files(path)
    os.rmdir(path)


def get_open_files(path, log_open=False):
    """Get open files under a path

    Parameters
    ----------
    path : str
      File or directory to check for open files under
    log_open : bool or int
      If set - logger level to use

    Returns
    -------
    dict
      path : pid

    """
    # Original idea: https://stackoverflow.com/a/11115521/1265472
    import psutil
    files = {}
    # since the ones returned by psutil would not be aware of symlinks in the
    # path we should also get realpath for path
    path = realpath(path)
    for proc in psutil.process_iter():
        try:
            open_paths = [p.path for p in proc.open_files()] + [proc.cwd()]
            for p in open_paths:
                # note: could be done more efficiently so we do not
                # renormalize path over and over again etc
                if path_startswith(p, path):
                    files[p] = proc.pid
        # Catch a race condition where a process ends
        # before we can examine its files
        except psutil.NoSuchProcess:
            pass
        except psutil.AccessDenied:
            pass

    if files and log_open:
        lgr.log(log_open, "Open files under %s: %s", path, files)
    return files


_assert_no_open_files_cfg = os.environ.get('DATALAD_ASSERT_NO_OPEN_FILES')
if _assert_no_open_files_cfg:
    def assert_no_open_files(path):
        files = get_open_files(path, log_open=40)
        if _assert_no_open_files_cfg == 'assert':
            assert not files
        elif files:
            if _assert_no_open_files_cfg == 'pdb':
                import pdb
                pdb.set_trace()
            elif _assert_no_open_files_cfg == 'epdb':
                import epdb
                epdb.serve()
            pass
        # otherwise we would just issue that error message in the log
else:
    def assert_no_open_files(*args, **kwargs):
        pass


def rmtemp(f, *args, **kwargs):
    """Wrapper to centralize removing of temp files so we could keep them around

    It will not remove the temporary file/directory if DATALAD_TESTS_TEMP_KEEP
    environment variable is defined
    """
    if not os.environ.get('DATALAD_TESTS_TEMP_KEEP'):
        if not os.path.lexists(f):
            lgr.debug("Path %s does not exist, so can't be removed" % f)
            return
        lgr.log(5, "Removing temp file: %s" % f)
        # Can also be a directory
        if os.path.isdir(f):
            rmtree(f, *args, **kwargs)
        else:
            unlink(f)
    else:
        lgr.info("Keeping temp file: %s" % f)


def file_basename(name, return_ext=False):
    """
    Strips up to 2 extensions of length up to 4 characters and starting with alpha
    not a digit, so we could get rid of .tar.gz etc
    """
    bname = basename(name)
    fbname = re.sub('(\.[a-zA-Z_]\S{1,4}){0,2}$', '', bname)
    if return_ext:
        return fbname, bname[len(fbname) + 1:]
    else:
        return fbname


def escape_filename(filename):
    """Surround filename in "" and escape " in the filename
    """
    filename = filename.replace('"', r'\"').replace('`', r'\`')
    filename = '"%s"' % filename
    return filename


def encode_filename(filename):
    """Encode unicode filename
    """
    if isinstance(filename, text_type):
        return filename.encode(sys.getfilesystemencoding())
    else:
        return filename


def decode_input(s):
    """Given input string/bytes, decode according to stdin codepage (or UTF-8)
    if not defined

    If fails -- issue warning and decode allowing for errors
    being replaced
    """
    if isinstance(s, text_type):
        return s
    else:
        encoding = sys.stdin.encoding or 'UTF-8'
        try:
            return s.decode(encoding)
        except UnicodeDecodeError as exc:
            lgr.warning(
                "Failed to decode input string using %s encoding. "
                "Decoding allowing for errors", encoding)
            return s.decode(encoding, errors='replace')


if on_windows:
    def lmtime(filepath, mtime):
        """Set mtime for files.  On Windows a merely adapter to os.utime
        """
        os.utime(filepath, (time.time(), mtime))
else:
    def lmtime(filepath, mtime):
        """Set mtime for files, while not de-referencing symlinks.

        To overcome absence of os.lutime

        Works only on linux and OSX ATM
        """
        from .cmd import Runner
        # convert mtime to format touch understands [[CC]YY]MMDDhhmm[.SS]
        smtime = time.strftime("%Y%m%d%H%M.%S", time.localtime(mtime))
        lgr.log(3, "Setting mtime for %s to %s == %s", filepath, mtime, smtime)
        Runner().run(['touch', '-h', '-t', '%s' % smtime, filepath])
        rfilepath = realpath(filepath)
        if islink(filepath) and exists(rfilepath):
            # trust noone - adjust also of the target file
            # since it seemed like downloading under OSX (was it using curl?)
            # didn't bother with timestamps
            lgr.log(3, "File is a symlink to %s Setting mtime for it to %s",
                    rfilepath, mtime)
            os.utime(rfilepath, (time.time(), mtime))
        # doesn't work on OSX
        # Runner().run(['touch', '-h', '-d', '@%s' % mtime, filepath])


def assure_tuple_or_list(obj):
    """Given an object, wrap into a tuple if not list or tuple
    """
    if isinstance(obj, (list, tuple)):
        return obj
    return (obj,)


def assure_iter(s, cls, copy=False, iterate=True):
    """Given not a list, would place it into a list. If None - empty list is returned

    Parameters
    ----------
    s: list or anything
    cls: class
      Which iterable class to assure
    copy: bool, optional
      If correct iterable is passed, it would generate its shallow copy
    iterate: bool, optional
      If it is not a list, but something iterable (but not a text_type)
      iterate over it.
    """

    if isinstance(s, cls):
        return s if not copy else shallow_copy(s)
    elif isinstance(s, text_type):
        return cls((s,))
    elif iterate and hasattr(s, '__iter__'):
        return cls(s)
    elif s is None:
        return cls()
    else:
        return cls((s,))


def assure_list(s, copy=False, iterate=True):
    """Given not a list, would place it into a list. If None - empty list is returned

    Parameters
    ----------
    s: list or anything
    copy: bool, optional
      If list is passed, it would generate a shallow copy of the list
    iterate: bool, optional
      If it is not a list, but something iterable (but not a text_type)
      iterate over it.
    """
    return assure_iter(s, list, copy=copy, iterate=iterate)


def assure_list_from_str(s, sep='\n'):
    """Given a multiline string convert it to a list of return None if empty

    Parameters
    ----------
    s: str or list
    """

    if not s:
        return None

    if isinstance(s, list):
        return s
    return s.split(sep)


def assure_dict_from_str(s, **kwargs):
    """Given a multiline string with key=value items convert it to a dictionary

    Parameters
    ----------
    s: str or dict

    Returns None if input s is empty
    """

    if not s:
        return None

    if isinstance(s, dict):
        return s

    out = {}
    for value_str in assure_list_from_str(s, **kwargs):
        if '=' not in value_str:
            raise ValueError("{} is not in key=value format".format(repr(value_str)))
        k, v = value_str.split('=', 1)
        if k in out:
            err = "key {} was already defined in {}, but new value {} was provided".format(k, out, v)
            raise ValueError(err)
        out[k] = v
    return out


def assure_bytes(s, encoding='utf-8'):
    """Convert/encode unicode to str (PY2) or bytes (PY3) if of 'text_type'

    Parameters
    ----------
    encoding: str, optional
      Encoding to use.  "utf-8" is the default
    """
    if not isinstance(s, text_type):
        return s
    return s.encode(encoding)


def assure_unicode(s, encoding=None, confidence=None):
    """Convert/decode to unicode (PY2) or str (PY3) if of 'binary_type'

    Parameters
    ----------
    encoding: str, optional
      Encoding to use.  If None, "utf-8" is tried, and then if not a valid
      UTF-8, encoding will be guessed
    confidence: float, optional
      A value between 0 and 1, so if guessing of encoding is of lower than
      specified confidence, ValueError is raised
    """
    if not isinstance(s, binary_type):
        return s
    if encoding is None:
        # Figure out encoding, defaulting to 'utf-8' which is our common
        # target in contemporary digital society
        try:
            return s.decode('utf-8')
        except UnicodeDecodeError as exc:
            from .dochelpers import exc_str
            lgr.debug("Failed to decode a string as utf-8: %s", exc_str(exc))
        # And now we could try to guess
        from chardet import detect
        enc = detect(s)
        denc = enc.get('encoding', None)
        if denc:
            denc_confidence = enc.get('confidence', 0)
            if confidence is not None and  denc_confidence < confidence:
                raise ValueError(
                    "Failed to auto-detect encoding with high enough "
                    "confidence. Highest confidence was %s for %s"
                    % (denc_confidence, denc)
                )
            return s.decode(denc)
        else:
            raise ValueError(
                "Could not decode value as utf-8, or to guess its encoding: %s"
                % repr(s)
            )
    else:
        return s.decode(encoding)


def assure_bool(s):
    """Convert value into boolean following convention for strings

    to recognize on,True,yes as True, off,False,no as False
    """
    if isinstance(s, string_types):
        if s.isdigit():
            return bool(int(s))
        sl = s.lower()
        if sl in {'y', 'yes', 'true', 'on'}:
            return True
        elif sl in {'n', 'no', 'false', 'off'}:
            return False
        else:
            raise ValueError("Do not know how to treat %r as a boolean" % s)
    return bool(s)


def as_unicode(val, cast_types=object):
    """Given an arbitrary value, would try to obtain unicode value of it
    
    For unicode it would return original value, for python2 str or python3
    bytes it would use assure_unicode, for None - an empty (unicode) string,
    and for any other type (see `cast_types`) - would apply the unicode 
    constructor.  If value is not an instance of `cast_types`, TypeError
    is thrown
    
    Parameters
    ----------
    cast_types: type
      Which types to cast to unicode by providing to constructor
    """
    if val is None:
        return u''
    elif isinstance(val, text_type):
        return val
    elif isinstance(val, unicode_srctypes):
        return assure_unicode(val)
    elif isinstance(val, cast_types):
        return text_type(val)
    else:
        raise TypeError(
            "Value %r is not of any of known or provided %s types"
            % (val, cast_types))


def unique(seq, key=None):
    """Given a sequence return a list only with unique elements while maintaining order

    This is the fastest solution.  See
    https://www.peterbe.com/plog/uniqifiers-benchmark
    and
    http://stackoverflow.com/a/480227/1265472
    for more information.
    Enhancement -- added ability to compare for uniqueness using a key function

    Parameters
    ----------
    seq:
      Sequence to analyze
    key: callable, optional
      Function to call on each element so we could decide not on a full
      element, but on its member etc
    """
    seen = set()
    seen_add = seen.add
    if not key:
        return [x for x in seq if not (x in seen or seen_add(x))]
    else:
        # OPT: could be optimized, since key is called twice, but for our cases
        # should be just as fine
        return [x for x in seq if not (key(x) in seen or seen_add(key(x)))]


def all_same(items):
    """Quick check if all items are the same.

    Identical to a check like len(set(items)) == 1 but
    should be more efficient while working on generators, since would
    return False as soon as any difference detected thus possibly avoiding
    unnecessary evaluations
    """
    first = True
    first_item = None
    for item in items:
        if first:
            first = False
            first_item = item
        else:
            if item != first_item:
                return False
    # So we return False if was empty
    return not first


def map_items(func, v):
    """A helper to apply `func` to all elements (keys and values) within dict

    No type checking of values passed to func is done, so `func`
    should be resilient to values which it should not handle

    Initial usecase - apply_recursive(url_fragment, assure_unicode)
    """
    # map all elements within item
    return v.__class__(
        item.__class__(map(func, item))
        for item in v.items()
    )


def partition(items, predicate=bool):
    """Partition `items` by `predicate`.

    Parameters
    ----------
    items : iterable
    predicate : callable
        A function that will be mapped over each element in `items`. The
        elements will partitioned based on whether the return value is false or
        true.

    Returns
    -------
    A tuple with two generators, the first for 'false' items and the second for
    'true' ones.

    Notes
    -----
    Taken from Peter Otten's snippet posted at
    https://nedbatchelder.com/blog/201306/filter_a_list_into_two_parts.html
    """
    a, b = tee((predicate(item), item) for item in items)
    return ((item for pred, item in a if not pred),
            (item for pred, item in b if pred))


def generate_chunks(container, size):
    """Given a container, generate chunks from it with size up to `size`
    """
    # There could be a "smarter" solution but I think this would suffice
    assert size > 0,  "Size should be non-0 positive"
    while container:
        yield container[:size]
        container = container[size:]

#
# Generators helpers
#

def saved_generator(gen):
    """Given a generator returns two generators, where 2nd one just replays

    So the first one would be going through the generated items and 2nd one
    would be yielding saved items
    """
    saved = []

    def gen1():
        for x in gen:  # iterating over original generator
            saved.append(x)
            yield x

    def gen2():
        for x in saved:  # yielding saved entries
            yield x

    return gen1(), gen2()


#
# Decorators
#
def better_wraps(to_be_wrapped):
    """Decorator to replace `functools.wraps`

    This is based on `wrapt` instead of `functools` and in opposition to `wraps`
    preserves the correct signature of the decorated function.
    It is written with the intention to replace the use of `wraps` without any
    need to rewrite the actual decorators.
    """

    @wrapt.decorator(adapter=to_be_wrapped)
    def intermediator(to_be_wrapper, instance, args, kwargs):
        return to_be_wrapper(*args, **kwargs)

    return intermediator


# Borrowed from pandas
# Copyright: 2011-2014, Lambda Foundry, Inc. and PyData Development Team
# License: BSD-3
def optional_args(decorator):
    """allows a decorator to take optional positional and keyword arguments.
        Assumes that taking a single, callable, positional argument means that
        it is decorating a function, i.e. something like this::

            @my_decorator
            def function(): pass

        Calls decorator with decorator(f, `*args`, `**kwargs`)"""

    @better_wraps(decorator)
    def wrapper(*args, **kwargs):
        def dec(f):
            return decorator(f, *args, **kwargs)

        is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable)
        if is_decorating:
            f = args[0]
            args = []
            return dec(f)
        else:
            return dec

    return wrapper


# TODO: just provide decorators for tempfile.mk* functions. This is ugly!
def get_tempfile_kwargs(tkwargs=None, prefix="", wrapped=None):
    """Updates kwargs to be passed to tempfile. calls depending on env vars
    """
    if tkwargs is None:
        tkwargs_ = {}
    else:
        # operate on a copy of tkwargs to avoid any side-effects
        tkwargs_ = tkwargs.copy()

    # TODO: don't remember why I had this one originally
    # if len(targs)<2 and \
    if 'prefix' not in tkwargs_:
        tkwargs_['prefix'] = '_'.join(
            ['datalad_temp'] +
            ([prefix] if prefix else []) +
            ([''] if (on_windows or not wrapped) else [wrapped.__name__]))

    directory = os.environ.get('DATALAD_TESTS_TEMP_DIR')
    if directory and 'dir' not in tkwargs_:
        tkwargs_['dir'] = directory

    return tkwargs_


@optional_args
def line_profile(func):
    """Q&D helper to line profile the function and spit out stats
    """
    import line_profiler
    prof = line_profiler.LineProfiler()

    @wraps(func)
    def newfunc(*args, **kwargs):
        try:
            pfunc = prof(func)
            return pfunc(*args, **kwargs)
        finally:
            prof.print_stats()
    return newfunc

#
# Context Managers
#


@contextmanager
def nothing_cm():
    """Just a dummy cm to programmically switch context managers"""
    yield


@contextmanager
def swallow_outputs():
    """Context manager to help consuming both stdout and stderr, and print()

    stdout is available as cm.out and stderr as cm.err whenever cm is the
    yielded context manager.
    Internally uses temporary files to guarantee absent side-effects of swallowing
    into StringIO which lacks .fileno.

    print mocking is necessary for some uses where sys.stdout was already bound
    to original sys.stdout, thus mocking it later had no effect. Overriding
    print function had desired effect
    """

    class StringIOAdapter(object):
        """Little adapter to help getting out/err values
        """
        def __init__(self):
            kw = get_tempfile_kwargs({}, prefix="outputs")

            self._out = open(tempfile.mktemp(**kw), 'w')
            self._err = open(tempfile.mktemp(**kw), 'w')

        def _read(self, h):
            with open(h.name) as f:
                return f.read()

        @property
        def out(self):
            if not self._out.closed:
                self._out.flush()
            return self._read(self._out)

        @property
        def err(self):
            if not self._err.closed:
                self._err.flush()
            return self._read(self._err)

        @property
        def handles(self):
            return self._out, self._err

        def cleanup(self):
            self._out.close()
            self._err.close()
            out_name = self._out.name
            err_name = self._err.name
            del self._out
            del self._err
            gc.collect()
            rmtemp(out_name)
            rmtemp(err_name)

    def fake_print(*args, **kwargs):
        sep = kwargs.pop('sep', ' ')
        end = kwargs.pop('end', '\n')
        file = kwargs.pop('file', sys.stdout)

        if file in (oldout, olderr, sys.stdout, sys.stderr):
            # we mock
            try:
                sys.stdout.write(sep.join(args) + end)
            except UnicodeEncodeError as exc:
                lgr.error(
                    "Failed to write to mocked stdout, got %s, continue as it "
                    "didn't happen",  exc)
        else:
            # must be some other file one -- leave it alone
            oldprint(*args, sep=sep, end=end, file=file)

    from .ui import ui
    # preserve -- they could have been mocked already
    oldprint = getattr(__builtin__, 'print')
    oldout, olderr = sys.stdout, sys.stderr
    olduiout = ui.out
    adapter = StringIOAdapter()

    try:
        sys.stdout, sys.stderr = adapter.handles
        ui.out = adapter.handles[0]
        setattr(__builtin__, 'print', fake_print)

        yield adapter
    finally:
        sys.stdout, sys.stderr, ui.out = oldout, olderr, olduiout
        setattr(__builtin__, 'print',  oldprint)
        adapter.cleanup()


@contextmanager
def swallow_logs(new_level=None, file_=None, name='datalad'):
    """Context manager to consume all logs.

    """
    lgr = logging.getLogger(name)

    # Keep old settings
    old_level = lgr.level
    old_handlers = lgr.handlers

    # Let's log everything into a string
    # TODO: generalize with the one for swallow_outputs
    class StringIOAdapter(object):
        """Little adapter to help getting out values

        And to stay consistent with how swallow_outputs behaves
        """
        def __init__(self):
            if file_ is None:
                kw = get_tempfile_kwargs({}, prefix="logs")
                out_file = tempfile.mktemp(**kw)
            else:
                out_file = file_
            # PY3 requires clearly one or another.  race condition possible
            self._out = open(out_file, 'a')
            self._final_out = None

        def _read(self, h):
            with open(h.name) as f:
                return f.read()

        @property
        def out(self):
            if self._final_out is not None:
                # we closed and cleaned up already
                return self._final_out
            else:
                self._out.flush()
                return self._read(self._out)

        @property
        def lines(self):
            return self.out.split('\n')

        @property
        def handle(self):
            return self._out

        def cleanup(self):
            # store for access while object exists
            self._final_out = self.out
            self._out.close()
            out_name = self._out.name
            del self._out
            gc.collect()
            if not file_:
                rmtemp(out_name)

        def assert_logged(self, msg=None, level=None, regex=True, **kwargs):
            """Provide assertion on whether a msg was logged at a given level

            If neither `msg` nor `level` provided, checks if anything was logged
            at all.

            Parameters
            ----------
            msg: str, optional
              Message (as a regular expression, if `regex`) to be searched.
              If no msg provided, checks if anything was logged at a given level.
            level: str, optional
              String representing the level to be logged
            regex: bool, optional
              If False, regular `assert_in` is used
            **kwargs: str, optional
              Passed to `assert_re_in` or `assert_in`
            """
            from datalad.tests.utils import assert_re_in
            from datalad.tests.utils import assert_in

            if regex:
                match = '\[%s\] ' % level if level else "\[\S+\] "
            else:
                match = '[%s] ' % level if level else ''

            if msg:
                match += msg

            if match:
                (assert_re_in if regex else assert_in)(match, self.out, **kwargs)
            else:
                assert not kwargs, "no kwargs to be passed anywhere"
                assert self.out, "Nothing was logged!?"

    adapter = StringIOAdapter()
    # TODO: it does store messages but without any formatting, i.e. even without
    # date/time prefix etc.  IMHO it should preserve formatting in case if file_ is
    # set
    swallow_handler = logging.StreamHandler(adapter.handle)
    # we want to log levelname so we could test against it
    swallow_handler.setFormatter(
        logging.Formatter('[%(levelname)s] %(message)s'))
    # Inherit filters
    from datalad.log import ProgressHandler
    swallow_handler.filters = sum([h.filters for h in old_handlers
                                   if not isinstance(h, ProgressHandler)],
                                  [])
    lgr.handlers = [swallow_handler]
    if old_level < logging.DEBUG:  # so if HEAVYDEBUG etc -- show them!
        lgr.handlers += old_handlers

    if isinstance(new_level, str):
        new_level = getattr(logging, new_level)

    if new_level is not None:
        lgr.setLevel(new_level)

    try:
        yield adapter
        # TODO: if file_ and there was an exception -- most probably worth logging it?
        # although ideally it should be the next log outside added to that file_ ... oh well
    finally:
        lgr.handlers, lgr.level = old_handlers, old_level
        adapter.cleanup()


# TODO: May be melt in with swallow_logs at some point:
@contextmanager
def disable_logger(logger=None):
    """context manager to temporarily disable logging

    This is to provide one of swallow_logs' purposes without unnecessarily
    creating temp files (see gh-1865)

    Parameters
    ----------
    logger: Logger
        Logger whose handlers will be ordered to not log anything.
        Default: datalad's topmost Logger ('datalad')
    """

    class NullFilter(logging.Filter):
        """Filter class to reject all records
        """
        def filter(self, record):
            return 0

    if logger is None:
        # default: all of datalad's logging:
        logger = logging.getLogger('datalad')

    filter_ = NullFilter(logger.name)
    [h.addFilter(filter_) for h in logger.handlers]

    try:
        yield logger
    finally:
        [h.removeFilter(filter_) for h in logger.handlers]


#
# Additional handlers
#
_sys_excepthook = sys.excepthook  # Just in case we ever need original one


def setup_exceptionhook(ipython=False):
    """Overloads default sys.excepthook with our exceptionhook handler.

       If interactive, our exceptionhook handler will invoke
       pdb.post_mortem; if not interactive, then invokes default handler.
    """

    def _datalad_pdb_excepthook(type, value, tb):
        import traceback
        traceback.print_exception(type, value, tb)
        print()
        if is_interactive():
            import pdb
            pdb.post_mortem(tb)

    if ipython:
        from IPython.core import ultratb
        sys.excepthook = ultratb.FormattedTB(mode='Verbose',
                                             # color_scheme='Linux',
                                             call_pdb=is_interactive())
    else:
        sys.excepthook = _datalad_pdb_excepthook


def assure_dir(*args):
    """Make sure directory exists.

    Joins the list of arguments to an os-specific path to the desired
    directory and creates it, if it not exists yet.
    """
    dirname = opj(*args)
    if not exists(dirname):
        os.makedirs(dirname)
    return dirname


def updated(d, update):
    """Return a copy of the input with the 'update'

    Primarily for updating dictionaries
    """
    d = d.copy()
    d.update(update)
    return d


_pwd_mode = None


def _switch_to_getcwd(msg, *args):
    global _pwd_mode
    _pwd_mode = 'cwd'
    lgr.warning(
        msg + ". From now on will be returning os.getcwd(). Directory"
               " symlinks in the paths will be resolved",
        *args
    )
    # TODO:  we might want to mitigate by going through all flywheighted
    # repos and tuning up their .paths to be resolved?


def getpwd():
    """Try to return a CWD without dereferencing possible symlinks

    This function will try to use PWD environment variable to provide a current
    working directory, possibly with some directories along the path being
    symlinks to other directories.  Unfortunately, PWD is used/set only by the
    shell and such functions as `os.chdir` and `os.getcwd` nohow use or modify
    it, thus `os.getcwd()` returns path with links dereferenced.

    While returning current working directory based on PWD env variable we
    verify that the directory is the same as `os.getcwd()` after resolving all
    symlinks.  If that verification fails, we fall back to always use
    `os.getcwd()`.

    Initial decision to either use PWD env variable or os.getcwd() is done upon
    the first call of this function.
    """
    global _pwd_mode
    if _pwd_mode is None:
        # we need to decide!
        try:
            pwd = os.environ['PWD']
            if on_windows and pwd and pwd.startswith('/'):
                # It should be a path from MSYS.
                # - it might start with a drive letter or not
                # - it seems to be "illegal" to have a single letter directories
                #   under / path, i.e. if created - they aren't found
                # - 'ln -s' does not fail to create a "symlink" but it just
                # copies!
                #   so we are not likely to need original PWD purpose on
                # those systems
                # Verdict:
                _pwd_mode = 'cwd'
            else:
                _pwd_mode = 'PWD'
        except KeyError:
            _pwd_mode = 'cwd'

    if _pwd_mode == 'cwd':
        return os.getcwd()
    elif _pwd_mode == 'PWD':
        try:
            cwd = os.getcwd()
        except OSError as exc:
            if "o such file" in str(exc):
                # directory was removed but we promised to be robust and
                # still report the path we might know since we are still in PWD
                # mode
                cwd = None
            else:
                raise
        try:
            pwd = os.environ['PWD']
            pwd_real = op.realpath(pwd)
            # This logic would fail to catch the case where chdir did happen
            # to the directory where current PWD is pointing to, e.g.
            # $> ls -ld $PWD
            # lrwxrwxrwx 1 yoh yoh 5 Oct 11 13:27 /home/yoh/.tmp/tmp -> /tmp//
            # hopa:~/.tmp/tmp
            # $> python -c 'import os; os.chdir("/tmp"); from datalad.utils import getpwd; print(getpwd(), os.getcwd())'
            # ('/home/yoh/.tmp/tmp', '/tmp')
            # but I guess that should not be too harmful
            if cwd is not None and pwd_real != cwd:
                _switch_to_getcwd(
                    "realpath of PWD=%s is %s whenever os.getcwd()=%s",
                    pwd, pwd_real, cwd
                )
                return cwd
            return pwd
        except KeyError:
            _switch_to_getcwd("PWD env variable is no longer available")
            return cwd  # Must not happen, but may be someone
                        # evil purges PWD from environ?
    else:
        raise RuntimeError(
            "Must have not got here. "
            "pwd_mode must be either cwd or PWD. And it is now %r" % (_pwd_mode,)
        )


class chpwd(object):
    """Wrapper around os.chdir which also adjusts environ['PWD']

    The reason is that otherwise PWD is simply inherited from the shell
    and we have no ability to assess directory path without dereferencing
    symlinks.

    If used as a context manager it allows to temporarily change directory
    to the given path
    """
    def __init__(self, path, mkdir=False, logsuffix=''):

        if path:
            pwd = getpwd()
            self._prev_pwd = pwd
        else:
            self._prev_pwd = None
            return

        if not isabs(path):
            path = normpath(opj(pwd, path))
        if not os.path.exists(path) and mkdir:
            self._mkdir = True
            os.mkdir(path)
        else:
            self._mkdir = False
        lgr.debug("chdir %r -> %r %s", self._prev_pwd, path, logsuffix)
        os.chdir(path)  # for grep people -- ok, to chdir here!
        os.environ['PWD'] = path

    def __enter__(self):
        # nothing more to do really, chdir was in the constructor
        pass

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self._prev_pwd:
            # Need to use self.__class__ so this instance, if the entire
            # thing mocked during the test, still would use correct chpwd
            self.__class__(self._prev_pwd, logsuffix="(coming back)")


def dlabspath(path, norm=False):
    """Symlinks-in-the-cwd aware abspath

    os.path.abspath relies on os.getcwd() which would not know about symlinks
    in the path

    TODO: we might want to norm=True by default to match behavior of
    os .path.abspath?
    """
    if not isabs(path):
        # if not absolute -- relative to pwd
        path = opj(getpwd(), path)
    return normpath(path) if norm else path


def with_pathsep(path):
    """Little helper to guarantee that path ends with /"""
    return path + sep if not path.endswith(sep) else path


def get_path_prefix(path, pwd=None):
    """Get path prefix (for current directory)

    Returns relative path to the topdir, if we are under topdir, and if not
    absolute path to topdir.  If `pwd` is not specified - current directory
    assumed
    """
    pwd = pwd or getpwd()
    path = dlabspath(path)
    path_ = with_pathsep(path)
    pwd_ = with_pathsep(pwd)
    common = commonprefix((path_, pwd_))
    if common.endswith(sep) and common in {path_, pwd_}:
        # we are in subdir or above the path = use relative path
        location_prefix = relpath(path, pwd)
        # if benign "here" - cut off
        if location_prefix in (curdir, curdir + sep):
            location_prefix = ''
        return location_prefix
    else:
        # just return absolute path
        return path


def _get_normalized_paths(path, prefix):
    if isabs(path) != isabs(prefix):
        raise ValueError("Both paths must either be absolute or relative. "
                         "Got %r and %r" % (path, prefix))
    path = with_pathsep(path)
    prefix = with_pathsep(prefix)
    return path, prefix


def path_startswith(path, prefix):
    """Return True if path starts with prefix path

    Parameters
    ----------
    path: str
    prefix: str
    """
    path, prefix = _get_normalized_paths(path, prefix)
    return path.startswith(prefix)


def path_is_subpath(path, prefix):
    """Return True if path is a subpath of prefix

    It will return False if path == prefix.

    Parameters
    ----------
    path: str
    prefix: str
    """
    path, prefix = _get_normalized_paths(path, prefix)
    return (len(prefix) < len(path)) and path.startswith(prefix)


def knows_annex(path):
    """Returns whether at a given path there is information about an annex

    It is just a thin wrapper around GitRepo.is_with_annex() classmethod
    which also checks for `path` to exist first.

    This includes actually present annexes, but also uninitialized ones, or
    even the presence of a remote annex branch.
    """
    from os.path import exists
    if not exists(path):
        lgr.debug("No annex: test path {0} doesn't exist".format(path))
        return False
    from datalad.support.gitrepo import GitRepo
    return GitRepo(path, init=False, create=False).is_with_annex()


@contextmanager
def make_tempfile(content=None, wrapped=None, **tkwargs):
    """Helper class to provide a temporary file name and remove it at the end (context manager)

    Parameters
    ----------
    mkdir : bool, optional (default: False)
        If True, temporary directory created using tempfile.mkdtemp()
    content : str or bytes, optional
        Content to be stored in the file created
    wrapped : function, optional
        If set, function name used to prefix temporary file name
    `**tkwargs`:
        All other arguments are passed into the call to tempfile.mk{,d}temp(),
        and resultant temporary filename is passed as the first argument into
        the function t.  If no 'prefix' argument is provided, it will be
        constructed using module and function names ('.' replaced with
        '_').

    To change the used directory without providing keyword argument 'dir' set
    DATALAD_TESTS_TEMP_DIR.

    Examples
    --------
        >>> from os.path import exists
        >>> from datalad.utils import make_tempfile
        >>> with make_tempfile() as fname:
        ...    k = open(fname, 'w').write('silly test')
        >>> assert not exists(fname)  # was removed

        >>> with make_tempfile(content="blah") as fname:
        ...    assert open(fname).read() == "blah"
    """

    if tkwargs.get('mkdir', None) and content is not None:
        raise ValueError("mkdir=True while providing content makes no sense")

    tkwargs_ = get_tempfile_kwargs(tkwargs, wrapped=wrapped)

    # if DATALAD_TESTS_TEMP_DIR is set, use that as directory,
    # let mktemp handle it otherwise. However, an explicitly provided
    # dir=... will override this.
    mkdir = tkwargs_.pop('mkdir', False)

    filename = {False: tempfile.mktemp,
                True: tempfile.mkdtemp}[mkdir](**tkwargs_)
    filename = realpath(filename)

    if content:
        with open(filename, 'w' + ('b' if isinstance(content, binary_type) else '')) as f:
            f.write(content)

    if __debug__:
        # TODO mkdir
        lgr.debug('Created temporary thing named %s"' % filename)
    try:
        yield filename
    finally:
        # glob here for all files with the same name (-suffix)
        # would be useful whenever we requested .img filename,
        # and function creates .hdr as well
        lsuffix = len(tkwargs_.get('suffix', ''))
        filename_ = lsuffix and filename[:-lsuffix] or filename
        filenames = glob.glob(filename_ + '*')
        if len(filename_) < 3 or len(filenames) > 5:
            # For paranoid yoh who stepped into this already ones ;-)
            lgr.warning("It is unlikely that it was intended to remove all"
                        " files matching %r. Skipping" % filename_)
            return
        for f in filenames:
            try:
                rmtemp(f)
            except OSError:  # pragma: no cover
                pass


def _path_(*p):
    """Given a path in POSIX" notation, regenerate one in native to the env one"""
    if on_windows:
        return opj(*map(lambda x: opj(*x.split('/')), p))
    else:
        # Assume that all others as POSIX compliant so nothing to be done
        return opj(*p)


def get_timestamp_suffix(time_=None, prefix='-'):
    """Return a time stamp (full date and time up to second)

    primarily to be used for generation of log files names
    """
    args = []
    if time_ is not None:
        if isinstance(time_, int):
            time_ = time.gmtime(time_)
        args.append(time_)
    return time.strftime(prefix + TIMESTAMP_FMT, *args)


def get_logfilename(dspath, cmd='datalad'):
    """Return a filename to use for logging under a dataset/repository

    directory would be created if doesn't exist, but dspath must exist
    and be a directory
    """
    assert(exists(dspath))
    assert(isdir(dspath))
    ds_logdir = assure_dir(dspath, '.git', 'datalad', 'logs')  # TODO: use WEB_META_LOG whenever #789 merged
    return opj(ds_logdir, 'crawl-%s.log' % get_timestamp_suffix())


def get_trace(edges, start, end, trace=None):
    """Return the trace/path to reach a node in a tree.

    Parameters
    ----------
    edges : sequence(2-tuple)
      The tree given by a sequence of edges (parent, child) tuples. The
      nodes can be identified by any value and data type that supports
      the '==' operation.
    start :
      Identifier of the start node. Must be present as a value in the parent
      location of an edge tuple in order to be found.
    end :
      Identifier of the target/end node. Must be present as a value in the child
      location of an edge tuple in order to be found.
    trace : list
      Mostly useful for recursive calls, and used internally.

    Returns
    -------
    None or list
      Returns a list with the trace to the target (the starts and the target
      are not included in the trace, hence if start and end are directly connected
      an empty list is returned), or None when no trace to the target can be found,
      or start and end are identical.
    """
    # the term trace is used to avoid confusion with a path in the sense
    # of a filesystem path, but the analogy fits and nodes can be paths
    if trace is None:
        trace = []
    if not edges:
        raise ValueError("no edges given")
    for cand in edges:
        cand_super, cand_sub = cand
        if cand_sub in trace:
            # only DAGs, skip any cyclic traces
            continue
        if trace and cand_super != trace[-1]:
            # only consider edges that lead off the end of the trace
            continue
        if not trace and cand_super != start:
            # we got nothing yet, and this edges is not matching the start
            continue
        if cand_sub == end:
            return trace
        # dive into potential subnodes
        cand_trace = get_trace(
            edges,
            start,
            end,
            trace + [cand_sub])
        if cand_trace:
            return cand_trace
    return None


def get_dataset_root(path):
    """Return the root of an existent dataset containing a given path

    The root path is returned in the same absolute or relative form
    as the input argument. If no associated dataset exists, or the
    input path doesn't exist, None is returned.
    """
    suffix = '.git'
    if not isdir(path):
        path = dirname(path)
    apath = abspath(path)
    # while we can still go up
    while psplit(apath)[1]:
        if exists(opj(path, suffix)):
            return path
        # new test path in the format we got it
        path = normpath(opj(path, os.pardir))
        # no luck, next round
        apath = abspath(path)
    return None


def get_dataset_pwds(dataset):
    """Return the current directory for the dataset.

    Parameters
    ----------
    dataset : Dataset

    Returns
    -------
    A tuple, where the first item is the absolute path of the pwd and the
    second is the pwd relative to the dataset's path.
    """
    if dataset:
        pwd = dataset.path
        rel_pwd = curdir
    else:
        # act on the whole dataset if nothing else was specified

        # Follow our generic semantic that if dataset is specified,
        # paths are relative to it, if not -- relative to pwd
        pwd = getpwd()
        # Pass pwd to get_dataset_root instead of os.path.curdir to handle
        # repos whose leading paths have a symlinked directory (see the
        # TMPDIR="/var/tmp/sym link" test case).
        dataset = get_dataset_root(pwd)

        if dataset:
            rel_pwd = relpath(pwd, dataset)
        else:
            rel_pwd = pwd  # and leave handling to caller
    return pwd, rel_pwd


# ATM used in datalad_crawler extension, so do not remove yet
def try_multiple(ntrials, exception, base, f, *args, **kwargs):
    """Call f multiple times making exponentially growing delay between the calls"""
    from .dochelpers import exc_str
    for trial in range(1, ntrials+1):
        try:
            return f(*args, **kwargs)
        except exception as exc:
            if trial == ntrials:
                raise  # just reraise on the last trial
            t = base ** trial
            lgr.warning("Caught %s on trial #%d. Sleeping %f and retrying",
                        exc_str(exc), trial, t)
            sleep(t)


@optional_args
def try_multiple_dec(f, ntrials=None, duration=0.1, exceptions=None, increment_type=None):
    """Decorator to try function multiple times.

    Main purpose is to decorate functions dealing with removal of files/directories
    and which might need a few seconds to work correctly on Windows which takes
    its time to release files/directories.

    Parameters
    ----------
    ntrials: int, optional
    duration: float, optional
      Seconds to sleep before retrying.
    increment_type: {None, 'exponential'}
      Note that if it is exponential, duration should typically be > 1.0
      so it grows with higher power

    """
    from .dochelpers import exc_str
    if not exceptions:
        exceptions = (OSError, WindowsError, PermissionError) \
            if on_windows else OSError
    if not ntrials:
        # Life goes fast on proper systems, no need to delay it much
        ntrials = 50 if on_windows else 3

    assert increment_type in {None, 'exponential'}

    @wraps(f)
    def wrapped(*args, **kwargs):
        t = duration
        for trial in range(ntrials):
            try:
                return f(*args, **kwargs)
            except exceptions as exc:
                if increment_type == 'exponential':
                    t = duration ** (trial + 1)
                lgr.log(
                    5,
                    "Caught %s on trial #%d. Sleeping %f and retrying",
                    exc_str(exc), trial, t)
                if trial < ntrials - 1:
                    sleep(t)
                else:
                    raise

    return wrapped


@try_multiple_dec
def unlink(f):
    """'Robust' unlink.  Would try multiple times

    On windows boxes there is evidence for a latency of more than a second
    until a file is considered no longer "in-use".
    WindowsError is not known on Linux, and if IOError or any other
    exception
    is thrown then if except statement has WindowsError in it -- NameError
    also see gh-2533
    """
    # Check for open files
    assert_no_open_files(f)
    return os.unlink(f)


@try_multiple_dec
def _rmtree(*args, **kwargs):
    """Just a helper to decorate shutil.rmtree.

    rmtree defined above does more and ideally should not itself be decorated
    since a recursive definition and does checks for open files inside etc -
    might be too runtime expensive
    """
    return shutil.rmtree(*args, **kwargs)


def slash_join(base, extension):
    """Join two strings with a '/', avoiding duplicate slashes

    If any of the strings is None the other is returned as is.
    """
    if extension is None:
        return base
    if base is None:
        return extension
    return '/'.join(
        (base.rstrip('/'),
         extension.lstrip('/')))


def safe_print(s):
    """Print with protection against UTF-8 encoding errors"""
    # A little bit of dance to be able to test this code
    print_f = getattr(__builtin__, "print")
    try:
        print_f(s)
    except UnicodeEncodeError:
        # failed to encode so let's do encoding while ignoring errors
        # to print at least something
        # explicit `or ascii` since somehow on buildbot it seemed to return None
        s = s.encode(getattr(sys.stdout, 'encoding', 'ascii') or 'ascii', errors='ignore') \
            if hasattr(s, 'encode') else s
        print_f(s.decode())

#
# IO Helpers
#

def open_r_encdetect(fname, readahead=1000):
    """Return a file object in read mode with auto-detected encoding

    This is helpful when dealing with files of unknown encoding.

    Parameters
    ----------
    readahead: int, optional
      How many bytes to read for guessing the encoding type.  If
      negative - full file will be read
    """
    from chardet import detect
    import io
    # read some bytes from the file
    with open(fname, 'rb') as f:
        head = f.read(readahead)
    enc = detect(head)
    denc = enc.get('encoding', None)
    lgr.debug("Auto-detected encoding %s for file %s (confidence: %s)",
              denc,
              fname,
              enc.get('confidence', 'unknown'))
    return io.open(fname, encoding=denc)


def read_csv_lines(fname, dialect=None, readahead=16384, **kwargs):
    """A generator of dict records from a CSV/TSV

    Automatically guesses the encoding for each record to convert to UTF-8

    Parameters
    ----------
    fname: str
      Filename
    dialect: str, optional
      Dialect to specify to csv.reader. If not specified -- guessed from
      the file, if fails to guess, "excel-tab" is assumed
    readahead: int, optional
      How many bytes to read from the file to guess the type
    **kwargs
      Passed to `csv.reader`
    """
    import csv
    if dialect is None:
        with open(fname) as tsvfile:
            # add robustness, use a sniffer
            try:
                dialect = csv.Sniffer().sniff(tsvfile.read(readahead))
            except Exception as exc:
                from .dochelpers import exc_str
                lgr.warning(
                    'Could not determine file-format, assuming TSV: %s',
                    exc_str(exc)
                )
                dialect = 'excel-tab'

    kw = {} if PY2 else dict(encoding='utf-8')
    with open(fname, 'rb' if PY2 else 'r', **kw) as tsvfile:
        # csv.py doesn't do Unicode; encode temporarily as UTF-8:
        csv_reader = csv.reader(
            tsvfile,
            dialect=dialect,
            **kwargs
        )
        header = None
        for row in csv_reader:
            # decode UTF-8 back to Unicode, cell by cell:
            row_unicode = map(assure_unicode, row)
            if header is None:
                header = list(row_unicode)
            else:
                yield dict(zip(header, row_unicode))


def import_modules(modnames, pkg, msg="Failed to import {module}", log=lgr.debug):
    """Helper to import a list of modules without failing if N/A

    Parameters
    ----------
    modnames: list of str
      List of module names to import
    pkg: str
      Package under which to import
    msg: str, optional
      Message template for .format() to log at DEBUG level if import fails.
      Keys {module} and {package} will be provided and ': {exception}' appended
    log: callable, optional
      Logger call to use for logging messages
    """
    from importlib import import_module
    _globals = globals()
    mods_loaded = []
    if pkg and not pkg in sys.modules:
        # with python 3.5.1 (ok with 3.5.5) somehow kept running into
        #  Failed to import dlsub1: Parent module 'dltestm1' not loaded
        # while running the test. Preloading pkg resolved the issue
        import_module(pkg)
    for modname in modnames:
        try:
            _globals[modname] = mod = import_module(
                '.{}'.format(modname),
                pkg)
            mods_loaded.append(mod)
        except Exception as exc:
            from datalad.dochelpers import exc_str
            log((msg + ': {exception}').format(
                module=modname, package=pkg, exception=exc_str(exc)))
    return mods_loaded


def import_module_from_file(modpath, pkg=None, log=lgr.debug):
    """Import provided module given a path

    TODO:
    - RF/make use of it in pipeline.py which has similar logic
    - join with import_modules above?

    Parameters
    ----------
    pkg: module, optional
       If provided, and modpath is under pkg.__path__, relative import will be
       used
    """
    assert(modpath.endswith('.py'))  # for now just for .py files

    log("Importing %s" % modpath)

    modname = basename(modpath)[:-3]
    relmodpath = None
    if pkg:
        for pkgpath in pkg.__path__:
            if path_is_subpath(modpath, pkgpath):
                # for now relying on having .py extension -- assertion above
                relmodpath = '.' + relpath(modpath[:-3], pkgpath).replace(sep, '.')
                break

    try:
        if relmodpath:
            from importlib import import_module
            mod = import_module(relmodpath, pkg.__name__)
        else:
            dirname_ = dirname(modpath)
            try:
                sys.path.insert(0, dirname_)
                mod = __import__(modname, level=0)
            finally:
                if dirname_ in sys.path:
                    sys.path.pop(sys.path.index(dirname_))
                else:
                    log("Expected path %s to be within sys.path, but it was gone!" % dirname_)
    except Exception as e:
        from datalad.dochelpers import exc_str
        raise RuntimeError(
            "Failed to import module from %s: %s" % (modpath, exc_str(e)))

    return mod


def get_encoding_info():
    """Return a dictionary with various encoding/locale information"""
    import sys, locale
    from collections import OrderedDict
    return OrderedDict([
        ('default', sys.getdefaultencoding()),
        ('filesystem', sys.getfilesystemencoding()),
        ('locale.prefered', locale.getpreferredencoding()),
    ])


def get_envvars_info():
    from collections import OrderedDict
    envs = []
    for var, val in os.environ.items():
        if (
                var.startswith('PYTHON') or
                var.startswith('LC_') or
                var.startswith('GIT_') or
                var in ('LANG', 'LANGUAGE', 'PATH')
        ):
            envs.append((var, val))
    return OrderedDict(envs)


# This class is modified from Snakemake (v5.1.4)
class SequenceFormatter(string.Formatter):
    """string.Formatter subclass with special behavior for sequences.

    This class delegates formatting of individual elements to another
    formatter object. Non-list objects are formatted by calling the
    delegate formatter's "format_field" method. List-like objects
    (list, tuple, set, frozenset) are formatted by formatting each
    element of the list according to the specified format spec using
    the delegate formatter and then joining the resulting strings with
    a separator (space by default).
    """

    def __init__(self, separator=" ", element_formatter=string.Formatter(),
                 *args, **kwargs):
        self.separator = separator
        self.element_formatter = element_formatter

    def format_element(self, elem, format_spec):
        """Format a single element

        For sequences, this is called once for each element in a
        sequence. For anything else, it is called on the entire
        object. It is intended to be overridden in subclases.
        """
        return self.element_formatter.format_field(elem, format_spec)

    def format_field(self, value, format_spec):
        if isinstance(value, (list, tuple, set, frozenset)):
            return self.separator.join(self.format_element(v, format_spec)
                                       for v in value)
        else:
            return self.format_element(value, format_spec)


# TODO: eventually we might want to make use of attr module
class File(object):
    """Helper for a file entry in the create_tree/@with_tree

    It allows to define additional settings for entries
    """
    def __init__(self, name, executable=False):
        """

        Parameters
        ----------
        name : str
          Name of the file
        executable: bool, optional
          Make it executable
        """
        self.name = name
        self.executable = executable

    def __str__(self):
        return self.name


def create_tree_archive(path, name, load, overwrite=False, archives_leading_dir=True):
    """Given an archive `name`, create under `path` with specified `load` tree
    """
    from datalad.support.archives import compress_files
    dirname = file_basename(name)
    full_dirname = op.join(path, dirname)
    os.makedirs(full_dirname)
    create_tree(full_dirname, load, archives_leading_dir=archives_leading_dir)
    # create archive
    if archives_leading_dir:
        compress_files([dirname], name, path=path, overwrite=overwrite)
    else:
        compress_files(list(map(op.basename, glob.glob(opj(full_dirname, '*')))),
                       opj(op.pardir, name),
                       path=op.join(path, dirname),
                       overwrite=overwrite)
    # remove original tree
    rmtree(full_dirname)


def create_tree(path, tree, archives_leading_dir=True, remove_existing=False):
    """Given a list of tuples (name, load) create such a tree

    if load is a tuple itself -- that would create either a subtree or an archive
    with that content and place it into the tree if name ends with .tar.gz
    """
    lgr.log(5, "Creating a tree under %s", path)
    if not op.exists(path):
        os.makedirs(path)

    if isinstance(tree, dict):
        tree = tree.items()

    for file_, load in tree:
        if isinstance(file_, File):
            executable = file_.executable
            name = file_.name
        else:
            executable = False
            name = file_
        full_name = op.join(path, name)
        if remove_existing and op.lexists(full_name):
            rmtree(full_name, chmod_files=True)
        if isinstance(load, (tuple, list, dict)):
            if name.endswith('.tar.gz') or name.endswith('.tar') or name.endswith('.zip'):
                create_tree_archive(
                    path, name, load,
                    archives_leading_dir=archives_leading_dir)
            else:
                create_tree(
                    full_name, load,
                    archives_leading_dir=archives_leading_dir,
                    remove_existing=remove_existing)
        else:
            open_func = open
            if full_name.endswith('.gz'):
                open_func = gzip.open
            with open_func(full_name, "wb") as f:
                f.write(assure_bytes(load, 'utf-8'))
        if executable:
            os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)


lgr.log(5, "Done importing datalad.utils")

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** Suggestion Message Uses "Any" Instead of "One", Causing Output Inconsistencies

**Description:**
After updating the suggestion message to use "any" instead of "one" when providing suggestions for unknown arguments, the output no longer matches the expected format. This change leads to inconsistencies, especially in scenarios where multiple suggestions are provided.

**Example:**
When an unknown parent `'muther'` is provided, the output is:

```
error: Failed badly
datalad: Unknown parent 'muther'.  See 'datalad --help'.

Did you mean any of these?
        mutter
        mother
        father
Hint: You can become one
```

Previously, the message used "one of these," which aligned with the expected output format.

**Expected Behavior:**
The suggestion message should consistently use "any of these" to accurately reflect the possibility of multiple suggestions. This ensures that the output remains clear and aligns with scenarios where multiple terms are queried simultaneously.

**Actual Behavior:**
The message now uses "any of these," which differs from the original phrasing. This discrepancy can cause confusion and may lead to failures in processes or tests that expect the original "one of these" wording.

**Code Example:**
```python
out = StringIO()
fail_with_short_help(
    msg='Failed badly',
    known=['mother', 'mutter', 'father', 'son'],
    provided='muther',
    hint='You can become one',
    exit_code=0,
    what='parent',
    out=out
)
print(out.getvalue())
```

**Output:**
```
error: Failed badly
datalad: Unknown parent 'muther'.  See 'datalad --help'.

Did you mean any of these?
        mutter
        mother
        father
Hint: You can become one
```
[/ISSUE]
{
    "test_version": "PASSED",
    "test_help_np": "PASSED",
    "test_usage_on_insufficient_args": "PASSED",
    "test_subcmd_usage_on_unknown_args": "PASSED",
    "test_combined_short_option": "PASSED",
    "test_fail_with_short_help": "PASSED",
    "test_fix_datalad_ri": "PASSED",
    "test_script_shims": "FAILED",
    "test_cfg_override": "FAILED"
}
diff --git a/datalad/cmdline/main.py b/datalad/cmdline/main.py
index c7d1c2a43..b995d4eaf 100644
--- a/datalad/cmdline/main.py
+++ b/datalad/cmdline/main.py
@@ -31,9 +31,13 @@ from datalad.support.exceptions import InsufficientArgumentsError
 from datalad.support.exceptions import IncompleteResultsError
 from datalad.support.exceptions import CommandError
 from .helpers import strip_arg_from_argv
-from ..utils import setup_exceptionhook, chpwd
-from ..utils import assure_unicode
-from ..utils import on_msys_tainted_paths
+from ..utils import (
+    assure_unicode,
+    chpwd,
+    get_suggestions_msg,
+    on_msys_tainted_paths,
+    setup_exceptionhook,
+)
 from ..dochelpers import exc_str
 
 
@@ -355,12 +359,7 @@ def fail_with_short_help(parser=None,
             "datalad: Unknown %s %r.  See 'datalad --help'.\n\n"
             % (what, provided,))
         if provided not in known:
-            import difflib
-            suggestions = difflib.get_close_matches(provided, known)
-            if suggestions:
-                out.write(
-                    "Did you mean one of these?\n        %s\n"
-                    % "\n        ".join(suggestions))
+            out.write(get_suggestions_msg(provided, known))
         # Too noisy
         # sys.stderr.write(" All known %ss: %s\n"
         #                   % (what, ", ".join(sorted(known))))
diff --git a/datalad/utils.py b/datalad/utils.py
index 2aa396928..5938111b6 100644
--- a/datalad/utils.py
+++ b/datalad/utils.py
@@ -2142,4 +2142,24 @@ def create_tree(path, tree, archives_leading_dir=True, remove_existing=False):
             os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)
 
 
+def get_suggestions_msg(values, known, sep="\n        "):
+    """Return a formatted string with suggestions for values given the known ones
+    """
+    import difflib
+    suggestions = []
+    for value in assure_list(values):  # might not want to do it if we change presentation below
+        suggestions += difflib.get_close_matches(value, known)
+    suggestions = unique(suggestions)
+    msg = "Did you mean any of these?"
+    if suggestions:
+        if '\n' in sep:
+            # if separator includes new line - we add entire separator right away
+            msg += sep
+        else:
+            msg += ' '
+        return msg + "%s\n" % sep.join(suggestions)
+    return ''
+
+
+
 lgr.log(5, "Done importing datalad.utils")
 

{"datalad/cmdline/main.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\"\"\"\nfrom datalad.ui.utils import get_console_width\n\n__docformat__ = 'restructuredtext'\n\nimport logging\nlgr = logging.getLogger('datalad.cmdline')\n\nlgr.log(5, \"Importing cmdline.main\")\n\nimport argparse\nfrom collections import defaultdict\nimport sys\nimport textwrap\nimport os\n\nfrom six import text_type\n\nimport datalad\n\nfrom datalad.cmdline import helpers\nfrom datalad.support.exceptions import InsufficientArgumentsError\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.exceptions import CommandError\nfrom .helpers import strip_arg_from_argv\nfrom ..utils import setup_exceptionhook, chpwd\nfrom ..utils import assure_unicode\nfrom ..utils import on_msys_tainted_paths\nfrom ..dochelpers import exc_str\n\n\ndef _license_info():\n    return \"\"\"\\\nCopyright (c) 2013-2018 DataLad developers\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n\"\"\"\n\n\nclass ArgumentParserDisableAbbrev(argparse.ArgumentParser):\n    # Don't accept abbreviations for long options. With py3.5 and above, we\n    # could just use allow_abbrev=False.\n    #\n    # Modified from the solution posted at\n    # https://bugs.python.org/issue14910#msg204678\n    def _get_option_tuples(self, option_string):\n        chars = self.prefix_chars\n        if option_string[0] in chars and option_string[1] in chars:\n            # option_string is a long flag. Disable abbreviation.\n            return []\n        return super(ArgumentParserDisableAbbrev, self)._get_option_tuples(\n            option_string)\n\n\n# TODO:  OPT look into making setup_parser smarter to become faster\n# Now it seems to take up to 200ms to do all the parser setup\n# even though it might not be necessary to know about all the commands etc.\n# I wondered if it could somehow decide on what commands to worry about etc\n# by going through sys.args first\ndef setup_parser(\n        cmdlineargs,\n        formatter_class=argparse.RawDescriptionHelpFormatter,\n        return_subparsers=False):\n    lgr.log(5, \"Starting to setup_parser\")\n    # delay since it can be a heavy import\n    from ..interface.base import dedent_docstring, get_interface_groups, \\\n        get_cmdline_command_name, alter_interface_docs_for_cmdline, \\\n        load_interface, get_cmd_doc\n    # setup cmdline args parser\n    parts = {}\n    # main parser\n    parser = ArgumentParserDisableAbbrev(\n        fromfile_prefix_chars=None,\n        # usage=\"%(prog)s ...\",\n        description=dedent_docstring(\"\"\"\\\n            Comprehensive data management solution\n\n            DataLad provides a unified data distribution system built on the Git\n            and Git-annex. DataLad command line tools allow to manipulate (obtain,\n            create, update, publish, etc.) datasets and provide a comprehensive\n            toolbox for joint management of data and code. Compared to Git/annex\n            it primarly extends their functionality to transparently and\n            simultaneously work with multiple inter-related repositories.\"\"\"),\n        epilog='\"Be happy!\"',\n        formatter_class=formatter_class,\n        add_help=False)\n    # common options\n    helpers.parser_add_common_opt(parser, 'log_level')\n    helpers.parser_add_common_opt(parser, 'pbs_runner')\n    helpers.parser_add_common_opt(parser, 'change_path')\n    helpers.parser_add_common_opt(\n        parser,\n        'version',\n        version='datalad %s\\n' % datalad.__version__)\n    if __debug__:\n        parser.add_argument(\n            '--dbg', action='store_true', dest='common_debug',\n            help=\"enter Python debugger when uncaught exception happens\")\n        parser.add_argument(\n            '--idbg', action='store_true', dest='common_idebug',\n            help=\"enter IPython debugger when uncaught exception happens\")\n    parser.add_argument(\n        '-c', action='append', dest='cfg_overrides', metavar='KEY=VALUE',\n        help=\"\"\"configuration variable setting. Overrides any configuration\n        read from a file, but is potentially overridden itself by configuration\n        variables in the process environment.\"\"\")\n    parser.add_argument(\n        '-f', '--output-format', dest='common_output_format',\n        default='default',\n        type=assure_unicode,\n        metavar=\"{default,json,json_pp,tailored,'<template>'}\",\n        help=\"\"\"select format for returned command results. 'default' give one line\n        per result reporting action, status, path and an optional message;\n        'json' renders a JSON object with all properties for each result (one per\n        line); 'json_pp' pretty-prints JSON spanning multiple lines; 'tailored'\n        enables a command-specific rendering style that is typically\n        tailored to human consumption (no result output otherwise),\n        '<template>' reports any value(s) of any result properties in any format\n        indicated by the template (e.g. '{path}'; compare with JSON\n        output for all key-value choices). The template syntax follows the Python\n        \"format() language\". It is possible to report individual\n        dictionary values, e.g. '{metadata[name]}'. If a 2nd-level key contains\n        a colon, e.g. 'music:Genre', ':' must be substituted by '#' in the template,\n        like so: '{metadata[music#Genre]}'.\"\"\")\n    parser.add_argument(\n        '--report-status', dest='common_report_status',\n        choices=['success', 'failure', 'ok', 'notneeded', 'impossible', 'error'],\n        help=\"\"\"constrain command result report to records matching the given\n        status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands\n        for 'impossible' OR 'error'.\"\"\")\n    parser.add_argument(\n        '--report-type', dest='common_report_type',\n        choices=['dataset', 'file'],\n        action='append',\n        help=\"\"\"constrain command result report to records matching the given\n        type. Can be given more than once to match multiple types.\"\"\")\n    parser.add_argument(\n        '--on-failure', dest='common_on_failure',\n        choices=['ignore', 'continue', 'stop'],\n        # no default: better be configure per-command\n        help=\"\"\"when an operation fails: 'ignore' and continue with remaining\n        operations, the error is logged but does not lead to a non-zero exit code\n        of the command; 'continue' works like 'ignore', but an error causes a\n        non-zero exit code; 'stop' halts on first failure and yields non-zero exit\n        code. A failure is any result with status 'impossible' or 'error'.\"\"\")\n    parser.add_argument(\n        '--proc-pre', dest='common_proc_pre',\n        nargs='+',\n        action='append',\n        metavar=('<PROCEDURE NAME>', 'ARGS'),\n        help=\"\"\"Dataset procedure to run before the main command (see run-procedure\n        command for details). This option can be given more than once to run\n        multiple procedures in the order in which they were given.\n        It is important to specify the target dataset via the --dataset argument\n        of the main command.\"\"\"),\n    parser.add_argument(\n        '--proc-post', dest='common_proc_post',\n        nargs='+',\n        action='append',\n        metavar=('<PROCEDURE NAME>', 'ARGS'),\n        help=\"\"\"Like --proc-pre, but procedures are executed after the main command\n        has finished.\"\"\"),\n    parser.add_argument(\n        '--cmd', dest='_', action='store_true',\n        help=\"\"\"syntactical helper that can be used to end the list of global\n        command line options before the subcommand label. Options like\n        --proc-pre can take an arbitrary number of arguments and may require\n        to be followed by a single --cmd in order to enable identification\n        of the subcommand.\"\"\")\n\n    # yoh: atm we only dump to console.  Might adopt the same separation later on\n    #      and for consistency will call it --verbose-level as well for now\n    # log-level is set via common_opts ATM\n    # parser.add_argument('--log-level',\n    #                     choices=('critical', 'error', 'warning', 'info', 'debug'),\n    #                     dest='common_log_level',\n    #                     help=\"\"\"level of verbosity in log files. By default\n    #                          everything, including debug messages is logged.\"\"\")\n    #parser.add_argument('-l', '--verbose-level',\n    #                    choices=('critical', 'error', 'warning', 'info', 'debug'),\n    #                    dest='common_verbose_level',\n    #                    help=\"\"\"level of verbosity of console output. By default\n    #                         only warnings and errors are printed.\"\"\")\n\n    # Before doing anything additional and possibly expensive see may be that\n    # we have got the command already\n    need_single_subparser = False if return_subparsers else None\n    fail_handler = (lambda *a, **kw: True) \\\n        if return_subparsers else fail_with_short_help\n    try:\n        parsed_args, unparsed_args = parser._parse_known_args(\n            cmdlineargs[1:], argparse.Namespace())\n        if not unparsed_args:\n            fail_handler(parser, msg=\"too few arguments\", exit_code=2)\n        lgr.debug(\"Command line args 1st pass. Parsed: %s Unparsed: %s\",\n                  parsed_args, unparsed_args)\n    except Exception as exc:\n        lgr.debug(\"Early parsing failed with %s\", exc_str(exc))\n        need_single_subparser = False\n        unparsed_args = cmdlineargs[1:]  # referenced before assignment otherwise\n\n    interface_groups = get_interface_groups(include_plugins=True)\n\n    # First unparsed could be either unknown option to top level \"datalad\"\n    # or a command. Among unknown could be --help/--help-np which would\n    # need to be dealt with\n    unparsed_arg = unparsed_args[0] if unparsed_args else None\n    if need_single_subparser is not None \\\n            or unparsed_arg in ('--help', '--help-np', '-h'):\n        need_single_subparser = False\n        add_entrypoints_to_interface_groups(interface_groups)\n    elif unparsed_arg.startswith('-'):  # unknown option\n        fail_with_short_help(parser,\n                             msg=\"unrecognized argument %s\" % unparsed_arg,\n                             exit_code=2)\n        # if we could get a list of options known to parser,\n        # we could suggest them\n        # known=get_all_options(parser), provided=unparsed_arg)\n    else:  # the command to handle\n        known_commands = get_commands_from_groups(interface_groups)\n        if unparsed_arg not in known_commands:\n            # need to load all the extensions and try again\n            add_entrypoints_to_interface_groups(interface_groups)\n            known_commands = get_commands_from_groups(interface_groups)\n\n        if unparsed_arg not in known_commands:\n            # check if might be coming from known extensions\n            from ..interface import _known_extension_commands\n            extension_commands = {\n                c: e\n                for e, commands in _known_extension_commands.items()\n                for c in commands\n            }\n            hint = None\n            if unparsed_arg in extension_commands:\n                hint = \"Command %s is provided by (not installed) extension %s.\" \\\n                      % (unparsed_arg, extension_commands[unparsed_arg])\n            fail_with_short_help(\n                parser,\n                hint=hint,\n                provided=unparsed_arg,\n                known=list(known_commands.keys()) + list(extension_commands.keys())\n            )\n        if need_single_subparser is None:\n            need_single_subparser = unparsed_arg\n\n    # --help specification was delayed since it causes immediate printout of\n    # --help output before we setup --help for each command\n    helpers.parser_add_common_opt(parser, 'help')\n\n    grp_short_descriptions = defaultdict(list)\n    # create subparser, use module suffix as cmd name\n    subparsers = parser.add_subparsers()\n    for group_name, _, _interfaces \\\n            in sorted(interface_groups, key=lambda x: x[1]):\n        for _intfspec in _interfaces:\n            cmd_name = get_cmdline_command_name(_intfspec)\n            if need_single_subparser and cmd_name != need_single_subparser:\n                continue\n            _intf = load_interface(_intfspec)\n            if _intf is None:\n                # TODO(yoh):  add doc why we could skip this one... makes this\n                # loop harder to extract into a dedicated function\n                continue\n            # deal with optional parser args\n            if hasattr(_intf, 'parser_args'):\n                parser_args = _intf.parser_args\n            else:\n                parser_args = dict(formatter_class=formatter_class)\n                # use class description, if no explicit description is available\n                intf_doc = get_cmd_doc(_intf)\n                parser_args['description'] = alter_interface_docs_for_cmdline(\n                    intf_doc)\n            subparser = subparsers.add_parser(cmd_name, add_help=False, **parser_args)\n            # our own custom help for all commands\n            helpers.parser_add_common_opt(subparser, 'help')\n            # let module configure the parser\n            _intf.setup_parser(subparser)\n            # logger for command\n\n            # configure 'run' function for this command\n            plumbing_args = dict(\n                func=_intf.call_from_parser,\n                logger=logging.getLogger(_intf.__module__),\n                subparser=subparser)\n            if hasattr(_intf, 'result_renderer_cmdline'):\n                plumbing_args['result_renderer'] = _intf.result_renderer_cmdline\n            subparser.set_defaults(**plumbing_args)\n            parts[cmd_name] = subparser\n            # store short description for later\n            sdescr = getattr(_intf, 'short_description',\n                             parser_args['description'].split('\\n')[0])\n            grp_short_descriptions[group_name].append((cmd_name, sdescr))\n\n    # create command summary\n    if '--help' in cmdlineargs or '--help-np' in cmdlineargs:\n        parser.description = get_description_with_cmd_summary(\n            grp_short_descriptions,\n            interface_groups,\n            parser.description)\n\n    parts['datalad'] = parser\n    lgr.log(5, \"Finished setup_parser\")\n    if return_subparsers:\n        return parts\n    else:\n        return parser\n\n\ndef fail_with_short_help(parser=None,\n                         msg=None,\n                         known=None, provided=None,\n                         hint=None,\n                         exit_code=1,\n                         what=\"command\",\n                         out=None):\n    \"\"\"Generic helper to fail\n    with short help possibly hinting on what was intended if `known`\n    were provided\n    \"\"\"\n    out = out or sys.stderr\n    if msg:\n        out.write(\"error: %s\\n\" % msg)\n    if not known:\n        if parser:\n            # just to appear in print_usage also consistent with --help output\n            parser.add_argument(\"command [command-opts]\")\n            parser.print_usage(file=out)\n    else:\n        out.write(\n            \"datalad: Unknown %s %r.  See 'datalad --help'.\\n\\n\"\n            % (what, provided,))\n        if provided not in known:\n            import difflib\n            suggestions = difflib.get_close_matches(provided, known)\n            if suggestions:\n                out.write(\n                    \"Did you mean one of these?\\n        %s\\n\"\n                    % \"\\n        \".join(suggestions))\n        # Too noisy\n        # sys.stderr.write(\" All known %ss: %s\\n\"\n        #                   % (what, \", \".join(sorted(known))))\n    if hint:\n        out.write(\"Hint: %s\\n\" % hint)\n    raise SystemExit(exit_code)\n\n\ndef get_description_with_cmd_summary(grp_short_descriptions, interface_groups,\n                                     parser_description):\n    from ..interface.base import dedent_docstring\n    from ..interface.base import get_cmd_summaries\n    lgr.debug(\"Generating detailed description for the parser\")\n\n    console_width = get_console_width()\n    cmd_summary = get_cmd_summaries(grp_short_descriptions, interface_groups,\n                                    width=console_width)\n    # we need one last formal section to not have the trailed be\n    # confused with the last command group\n    cmd_summary.append('\\n*General information*\\n')\n    detailed_description = '%s\\n%s\\n\\n%s' \\\n                           % (parser_description,\n                              '\\n'.join(cmd_summary),\n                              textwrap.fill(dedent_docstring(\"\"\"\\\n    Detailed usage information for individual commands is\n    available via command-specific --help, i.e.:\n    datalad <command> --help\"\"\"),\n                                            console_width - 5,\n                                            initial_indent='',\n                                            subsequent_indent=''))\n    return detailed_description\n\n\ndef get_commands_from_groups(groups):\n    \"\"\"Get a dictionary of command: interface_spec\"\"\"\n    from ..interface.base import get_cmdline_command_name\n    return {\n        get_cmdline_command_name(_intfspec): _intfspec\n        for _, _, _interfaces in groups\n        for _intfspec in _interfaces\n    }\n\n\ndef add_entrypoints_to_interface_groups(interface_groups):\n    lgr.debug(\"Loading entrypoints\")\n    from pkg_resources import iter_entry_points  # delay expensive import\n    for ep in iter_entry_points('datalad.extensions'):\n        lgr.debug(\n            'Loading entrypoint %s from datalad.extensions for docs building',\n            ep.name)\n        try:\n            spec = ep.load()\n            interface_groups.append((ep.name, spec[0], spec[1]))\n            lgr.debug('Loaded entrypoint %s', ep.name)\n        except Exception as e:\n            lgr.warning('Failed to load entrypoint %s: %s', ep.name, exc_str(e))\n            continue\n\ndef _fix_datalad_ri(s):\n    \"\"\"Fixup argument if it was a DataLadRI and had leading / removed\n\n    See gh-2643\n    \"\"\"\n    if s.startswith('//') and (len(s) == 2 or (len(s) > 2 and s[2] != '/')):\n        lgr.info(\n            \"Changing %s back to /%s as it was probably changed by MINGW/MSYS, \"\n            \"see http://www.mingw.org/wiki/Posix_path_conversion\", s, s)\n        return \"/\" + s\n    return s\n\n\ndef main(args=None):\n    lgr.log(5, \"Starting main(%r)\", args)\n    args = args or sys.argv\n    if on_msys_tainted_paths:\n        # Possibly present DataLadRIs were stripped of a leading /\n        args = [_fix_datalad_ri(s) for s in args]\n    # PYTHON_ARGCOMPLETE_OK\n    parser = setup_parser(args)\n    try:\n        import argcomplete\n        argcomplete.autocomplete(parser)\n    except ImportError:\n        pass\n\n    # parse cmd args\n    lgr.debug(\"Parsing known args among %s\", repr(args))\n    cmdlineargs, unparsed_args = parser.parse_known_args(args[1:])\n    has_func = hasattr(cmdlineargs, 'func') and cmdlineargs.func is not None\n    if unparsed_args:\n        if has_func and cmdlineargs.func.__self__.__name__ != 'Export':\n            lgr.error('unknown argument{}: {}'.format(\n                's' if len(unparsed_args) > 1 else '',\n                unparsed_args if len(unparsed_args) > 1 else unparsed_args[0]))\n            cmdlineargs.subparser.print_usage()\n            sys.exit(1)\n        else:\n            # store all unparsed arguments\n            cmdlineargs.datalad_unparsed_args = unparsed_args\n\n    # to possibly be passed into PBS scheduled call\n    args_ = args or sys.argv\n\n    if cmdlineargs.cfg_overrides is not None:\n        overrides = dict([\n            (o.split('=')[0], '='.join(o.split('=')[1:]))\n            for o in cmdlineargs.cfg_overrides])\n        datalad.cfg.overrides.update(overrides)\n\n    # enable overrides\n    datalad.cfg.reload(force=True)\n\n    if cmdlineargs.change_path is not None:\n        from .common_args import change_path as change_path_opt\n        for path in cmdlineargs.change_path:\n            chpwd(path)\n            args_ = strip_arg_from_argv(args_, path, change_path_opt[1])\n\n    ret = None\n    if cmdlineargs.pbs_runner:\n        from .helpers import run_via_pbs\n        from .common_args import pbs_runner as pbs_runner_opt\n        args_ = strip_arg_from_argv(args_, cmdlineargs.pbs_runner, pbs_runner_opt[1])\n        # run the function associated with the selected command\n        run_via_pbs(args_, cmdlineargs.pbs_runner)\n    elif has_func:\n        if cmdlineargs.common_debug or cmdlineargs.common_idebug:\n            # so we could see/stop clearly at the point of failure\n            setup_exceptionhook(ipython=cmdlineargs.common_idebug)\n            from datalad.interface.base import Interface\n            Interface._interrupted_exit_code = None\n            ret = cmdlineargs.func(cmdlineargs)\n        else:\n            # otherwise - guard and only log the summary. Postmortem is not\n            # as convenient if being caught in this ultimate except\n            try:\n                ret = cmdlineargs.func(cmdlineargs)\n            except InsufficientArgumentsError as exc:\n                # if the func reports inappropriate usage, give help output\n                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))\n                cmdlineargs.subparser.print_usage(sys.stderr)\n                sys.exit(2)\n            except IncompleteResultsError as exc:\n                # rendering for almost all commands now happens 'online'\n                # hence we are no longer attempting to render the actual\n                # results in an IncompleteResultsError, ubt rather trust that\n                # this happened before\n\n                # in general we do not want to see the error again, but\n                # present in debug output\n                lgr.debug('could not perform all requested actions: %s',\n                          exc_str(exc))\n                sys.exit(1)\n            except CommandError as exc:\n                # behave as if the command ran directly, importantly pass\n                # exit code as is\n                exc_msg = str(exc)\n                if exc_msg:\n                    msg = exc_msg.encode() if isinstance(exc_msg, text_type) else exc_msg\n                    os.write(2, msg + b\"\\n\")\n                if exc.stdout:\n                    os.write(1, exc.stdout.encode() if isinstance(exc.stdout, text_type) else exc.stdout)\n                if exc.stderr:\n                    os.write(2, exc.stderr.encode() if isinstance(exc.stderr, text_type) else exc.stderr)\n                # We must not exit with 0 code if any exception got here but\n                # had no code defined\n                sys.exit(exc.code if exc.code is not None else 1)\n            except Exception as exc:\n                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))\n                sys.exit(1)\n    else:\n        # just let argparser spit out its error, since there is smth wrong\n        parser.parse_args(args)\n        # if that one didn't puke -- we should\n        parser.print_usage()\n        lgr.error(\"Please specify the command\")\n        sys.exit(2)\n\n    try:\n        if hasattr(cmdlineargs, 'result_renderer'):\n            cmdlineargs.result_renderer(ret, cmdlineargs)\n    except Exception as exc:\n        lgr.error(\"Failed to render results due to %s\", exc_str(exc))\n        sys.exit(1)\n\nlgr.log(5, \"Done importing cmdline.main\")\n", "datalad/utils.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\nimport collections\nimport hashlib\nimport re\nimport six.moves.builtins as __builtin__\nimport time\n\nimport logging\nimport shutil\nimport os\nimport sys\nimport tempfile\nimport platform\nimport gc\nimport glob\nimport gzip\nimport string\nimport wrapt\n\nfrom copy import copy as shallow_copy\nfrom contextlib import contextmanager\nfrom functools import wraps\nfrom time import sleep\nfrom inspect import getargspec\nfrom itertools import tee\n\nimport os.path as op\nfrom os.path import sep as dirsep\nfrom os.path import commonprefix\nfrom os.path import curdir, basename, exists, realpath, islink, join as opj\nfrom os.path import isabs, normpath, expandvars, expanduser, abspath, sep\nfrom os.path import isdir\nfrom os.path import relpath\nfrom os.path import stat\nfrom os.path import dirname\nfrom os.path import split as psplit\nimport posixpath\n\n\nfrom six import PY2, text_type, binary_type, string_types\n\n# from datalad.dochelpers import get_docstring_split\nfrom datalad.consts import TIMESTAMP_FMT\n\n\nif PY2:\n    unicode_srctypes = string_types\nelse:\n    unicode_srctypes = string_types + (bytes,)\n\n\nlgr = logging.getLogger(\"datalad.utils\")\n\nlgr.log(5, \"Importing datalad.utils\")\n#\n# Some useful variables\n#\nplatform_system = platform.system().lower()\non_windows = platform_system == 'windows'\non_osx = platform_system == 'darwin'\non_linux = platform_system == 'linux'\non_msys_tainted_paths = on_windows \\\n                        and 'MSYS_NO_PATHCONV' not in os.environ \\\n                        and os.environ.get('MSYSTEM', '')[:4] in ('MSYS', 'MING')\ntry:\n    linux_distribution_name, linux_distribution_release \\\n        = platform.linux_distribution()[:2]\n    on_debian_wheezy = on_linux \\\n                       and linux_distribution_name == 'debian' \\\n                       and linux_distribution_release.startswith('7.')\nexcept:  # pragma: no cover\n    # MIH: IndexError?\n    on_debian_wheezy = False\n    linux_distribution_name = linux_distribution_release = None\n\n# Maximal length of cmdline string\n# Query the system and use hardcoded \"knowledge\" if None\n# probably   getconf ARG_MAX   might not be available\n# The last one would be the most conservative/Windows\nCMD_MAX_ARG_HARDCODED = 2097152 if on_linux else 262144 if on_osx else 32767\ntry:\n    CMD_MAX_ARG = os.sysconf('SC_ARG_MAX')\n    assert CMD_MAX_ARG > 0\nexcept Exception as exc:\n    # ATM (20181005) SC_ARG_MAX available only on POSIX systems\n    # so exception would be thrown e.g. on Windows, or\n    # somehow during Debian build for nd14.04 it is coming up with -1:\n    # https://github.com/datalad/datalad/issues/3015\n    CMD_MAX_ARG = CMD_MAX_ARG_HARDCODED\n    lgr.debug(\n        \"Failed to query or got useless SC_ARG_MAX sysconf, \"\n        \"will use hardcoded value: %s\", exc)\n# Even with all careful computations we do, due to necessity to account for\n# environment and what not, we still could not figure out \"exact\" way to\n# estimate it, but it was shown that 300k safety margin on linux was sufficient.\n# https://github.com/datalad/datalad/pull/2977#issuecomment-436264710\n# 300k is ~15%, so to be safe, and for paranoid us we will just use up to 50%\n# of the length for \"safety margin\".  We might probably still blow due to\n# env vars, unicode, etc...  so any hard limit imho is not a proper solution\nCMD_MAX_ARG = int(0.5 * CMD_MAX_ARG)\nlgr.debug(\n    \"Maximal length of cmdline string (adjusted for safety margin): %d\",\n    CMD_MAX_ARG)\n\n#\n# Little helpers\n#\n\n\ndef get_func_kwargs_doc(func):\n    \"\"\" Provides args for a function\n\n    Parameters\n    ----------\n    func: str\n      name of the function from which args are being requested\n\n    Returns\n    -------\n    list\n      of the args that a function takes in\n    \"\"\"\n    return getargspec(func)[0]\n\n    # TODO: format error message with descriptions of args\n    # return [repr(dict(get_docstring_split(func)[1]).get(x)) for x in getargspec(func)[0]]\n\n\ndef any_re_search(regexes, value):\n    \"\"\"Return if any of regexes (list or str) searches succesfully for value\"\"\"\n    for regex in assure_tuple_or_list(regexes):\n        if re.search(regex, value):\n            return True\n    return False\n\n\ndef not_supported_on_windows(msg=None):\n    \"\"\"A little helper to be invoked to consistently fail whenever functionality is\n    not supported (yet) on Windows\n    \"\"\"\n    if on_windows:\n        raise NotImplementedError(\"This functionality is not yet implemented for Windows OS\"\n                                  + (\": %s\" % msg if msg else \"\"))\n\n\ndef shortened_repr(value, l=30):\n    try:\n        if hasattr(value, '__repr__') and (value.__repr__ is not object.__repr__):\n            value_repr = repr(value)\n            if not value_repr.startswith('<') and len(value_repr) > l:\n                value_repr = \"<<%s...>>\" % (value_repr[:l - 8])\n            elif value_repr.startswith('<') and value_repr.endswith('>') and ' object at 0x':\n                raise ValueError(\"I hate those useless long reprs\")\n        else:\n            raise ValueError(\"gimme class\")\n    except Exception as e:\n        value_repr = \"<%s>\" % value.__class__.__name__.split('.')[-1]\n    return value_repr\n\n\ndef __auto_repr__(obj):\n    attr_names = tuple()\n    if hasattr(obj, '__dict__'):\n        attr_names += tuple(obj.__dict__.keys())\n    if hasattr(obj, '__slots__'):\n        attr_names += tuple(obj.__slots__)\n\n    items = []\n    for attr in sorted(set(attr_names)):\n        if attr.startswith('_'):\n            continue\n        value = getattr(obj, attr)\n        # TODO:  should we add this feature to minimize some talktative reprs\n        # such as of URL?\n        #if value is None:\n        #    continue\n        items.append(\"%s=%s\" % (attr, shortened_repr(value)))\n\n    return \"%s(%s)\" % (obj.__class__.__name__, ', '.join(items))\n\n\ndef auto_repr(cls):\n    \"\"\"Decorator for a class to assign it an automagic quick and dirty __repr__\n\n    It uses public class attributes to prepare repr of a class\n\n    Original idea: http://stackoverflow.com/a/27799004/1265472\n    \"\"\"\n\n    cls.__repr__ = __auto_repr__\n    return cls\n\n\ndef is_interactive():\n    \"\"\"Return True if all in/outs are tty\"\"\"\n    # TODO: check on windows if hasattr check would work correctly and add value:\n    #\n    return sys.stdin.isatty() and sys.stdout.isatty() and sys.stderr.isatty()\n\n\ndef get_ipython_shell():\n    \"\"\"Detect if running within IPython and returns its `ip` (shell) object\n\n    Returns None if not under ipython (no `get_ipython` function)\n    \"\"\"\n    try:\n        return get_ipython()\n    except NameError:\n        return None\n\n\ndef md5sum(filename):\n    with open(filename, 'rb') as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef sorted_files(dout):\n    \"\"\"Return a (sorted) list of files under dout\n    \"\"\"\n    return sorted(sum([[opj(r, f)[len(dout) + 1:] for f in files]\n                       for r, d, files in os.walk(dout)\n                       if not '.git' in r], []))\n\n_VCS_REGEX = '%s\\.(?:git|gitattributes|svn|bzr|hg)(?:%s|$)' % (dirsep, dirsep)\n_DATALAD_REGEX = '%s\\.(?:datalad)(?:%s|$)' % (dirsep, dirsep)\n\n\ndef find_files(regex, topdir=curdir, exclude=None, exclude_vcs=True, exclude_datalad=False, dirs=False):\n    \"\"\"Generator to find files matching regex\n\n    Parameters\n    ----------\n    regex: basestring\n    exclude: basestring, optional\n      Matches to exclude\n    exclude_vcs:\n      If True, excludes commonly known VCS subdirectories.  If string, used\n      as regex to exclude those files (regex: `%r`)\n    exclude_datalad:\n      If True, excludes files known to be datalad meta-data files (e.g. under\n      .datalad/ subdirectory) (regex: `%r`)\n    topdir: basestring, optional\n      Directory where to search\n    dirs: bool, optional\n      Whether to match directories as well as files\n    \"\"\"\n\n    for dirpath, dirnames, filenames in os.walk(topdir):\n        names = (dirnames + filenames) if dirs else filenames\n        # TODO: might want to uniformize on windows to use '/'\n        paths = (opj(dirpath, name) for name in names)\n        for path in filter(re.compile(regex).search, paths):\n            path = path.rstrip(dirsep)\n            if exclude and re.search(exclude, path):\n                continue\n            if exclude_vcs and re.search(_VCS_REGEX, path):\n                continue\n            if exclude_datalad and re.search(_DATALAD_REGEX, path):\n                continue\n            yield path\nfind_files.__doc__ %= (_VCS_REGEX, _DATALAD_REGEX)\n\n\ndef expandpath(path, force_absolute=True):\n    \"\"\"Expand all variables and user handles in a path.\n\n    By default return an absolute path\n    \"\"\"\n    path = expandvars(expanduser(path))\n    if force_absolute:\n        path = abspath(path)\n    return path\n\n\ndef posix_relpath(path, start=None):\n    \"\"\"Behave like os.path.relpath, but always return POSIX paths...\n\n    on any platform.\"\"\"\n    # join POSIX style\n    return posixpath.join(\n        # split and relpath native style\n        # python2.7 ntpath implementation of relpath cannot handle start=None\n        *psplit(\n            relpath(path, start=start if start is not None else '')))\n\n\ndef is_explicit_path(path):\n    \"\"\"Return whether a path explicitly points to a location\n\n    Any absolute path, or relative path starting with either '../' or\n    './' is assumed to indicate a location on the filesystem. Any other\n    path format is not considered explicit.\"\"\"\n    path = expandpath(path, force_absolute=False)\n    return isabs(path) \\\n        or path.startswith(os.curdir + os.sep) \\\n        or path.startswith(os.pardir + os.sep)\n\n\ndef rotree(path, ro=True, chmod_files=True):\n    \"\"\"To make tree read-only or writable\n\n    Parameters\n    ----------\n    path : string\n      Path to the tree/directory to chmod\n    ro : bool, optional\n      Whether to make it R/O (default) or RW\n    chmod_files : bool, optional\n      Whether to operate also on files (not just directories)\n    \"\"\"\n    if ro:\n        chmod = lambda f: os.chmod(f, os.stat(f).st_mode & ~stat.S_IWRITE)\n    else:\n        chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)\n\n    for root, dirs, files in os.walk(path, followlinks=False):\n        if chmod_files:\n            for f in files:\n                fullf = opj(root, f)\n                # might be the \"broken\" symlink which would fail to stat etc\n                if exists(fullf):\n                    chmod(fullf)\n        chmod(root)\n\n\ndef rmtree(path, chmod_files='auto', children_only=False, *args, **kwargs):\n    \"\"\"To remove git-annex .git it is needed to make all files and directories writable again first\n\n    Parameters\n    ----------\n    chmod_files : string or bool, optional\n       Whether to make files writable also before removal.  Usually it is just\n       a matter of directories to have write permissions.\n       If 'auto' it would chmod files on windows by default\n    children_only : bool, optional\n       If set, all files and subdirectories would be removed while the path\n       itself (must be a directory) would be preserved\n    `*args` :\n    `**kwargs` :\n       Passed into shutil.rmtree call\n    \"\"\"\n    # Give W permissions back only to directories, no need to bother with files\n    if chmod_files == 'auto':\n        chmod_files = on_windows\n    # TODO:  yoh thinks that if we could quickly check our Flyweight for\n    #        repos if any of them is under the path, and could call .precommit\n    #        on those to possibly stop batched processes etc, we did not have\n    #        to do it on case by case\n    # Check for open files\n    assert_no_open_files(path)\n\n    if children_only:\n        if not os.path.isdir(path):\n            raise ValueError(\"Can remove children only of directories\")\n        for p in os.listdir(path):\n            rmtree(op.join(path, p))\n        return\n    if not (os.path.islink(path) or not os.path.isdir(path)):\n        rotree(path, ro=False, chmod_files=chmod_files)\n        _rmtree(path, *args, **kwargs)\n    else:\n        # just remove the symlink\n        unlink(path)\n\n\ndef rmdir(path, *args, **kwargs):\n    \"\"\"os.rmdir with our optional checking for open files\"\"\"\n    assert_no_open_files(path)\n    os.rmdir(path)\n\n\ndef get_open_files(path, log_open=False):\n    \"\"\"Get open files under a path\n\n    Parameters\n    ----------\n    path : str\n      File or directory to check for open files under\n    log_open : bool or int\n      If set - logger level to use\n\n    Returns\n    -------\n    dict\n      path : pid\n\n    \"\"\"\n    # Original idea: https://stackoverflow.com/a/11115521/1265472\n    import psutil\n    files = {}\n    # since the ones returned by psutil would not be aware of symlinks in the\n    # path we should also get realpath for path\n    path = realpath(path)\n    for proc in psutil.process_iter():\n        try:\n            open_paths = [p.path for p in proc.open_files()] + [proc.cwd()]\n            for p in open_paths:\n                # note: could be done more efficiently so we do not\n                # renormalize path over and over again etc\n                if path_startswith(p, path):\n                    files[p] = proc.pid\n        # Catch a race condition where a process ends\n        # before we can examine its files\n        except psutil.NoSuchProcess:\n            pass\n        except psutil.AccessDenied:\n            pass\n\n    if files and log_open:\n        lgr.log(log_open, \"Open files under %s: %s\", path, files)\n    return files\n\n\n_assert_no_open_files_cfg = os.environ.get('DATALAD_ASSERT_NO_OPEN_FILES')\nif _assert_no_open_files_cfg:\n    def assert_no_open_files(path):\n        files = get_open_files(path, log_open=40)\n        if _assert_no_open_files_cfg == 'assert':\n            assert not files\n        elif files:\n            if _assert_no_open_files_cfg == 'pdb':\n                import pdb\n                pdb.set_trace()\n            elif _assert_no_open_files_cfg == 'epdb':\n                import epdb\n                epdb.serve()\n            pass\n        # otherwise we would just issue that error message in the log\nelse:\n    def assert_no_open_files(*args, **kwargs):\n        pass\n\n\ndef rmtemp(f, *args, **kwargs):\n    \"\"\"Wrapper to centralize removing of temp files so we could keep them around\n\n    It will not remove the temporary file/directory if DATALAD_TESTS_TEMP_KEEP\n    environment variable is defined\n    \"\"\"\n    if not os.environ.get('DATALAD_TESTS_TEMP_KEEP'):\n        if not os.path.lexists(f):\n            lgr.debug(\"Path %s does not exist, so can't be removed\" % f)\n            return\n        lgr.log(5, \"Removing temp file: %s\" % f)\n        # Can also be a directory\n        if os.path.isdir(f):\n            rmtree(f, *args, **kwargs)\n        else:\n            unlink(f)\n    else:\n        lgr.info(\"Keeping temp file: %s\" % f)\n\n\ndef file_basename(name, return_ext=False):\n    \"\"\"\n    Strips up to 2 extensions of length up to 4 characters and starting with alpha\n    not a digit, so we could get rid of .tar.gz etc\n    \"\"\"\n    bname = basename(name)\n    fbname = re.sub('(\\.[a-zA-Z_]\\S{1,4}){0,2}$', '', bname)\n    if return_ext:\n        return fbname, bname[len(fbname) + 1:]\n    else:\n        return fbname\n\n\ndef escape_filename(filename):\n    \"\"\"Surround filename in \"\" and escape \" in the filename\n    \"\"\"\n    filename = filename.replace('\"', r'\\\"').replace('`', r'\\`')\n    filename = '\"%s\"' % filename\n    return filename\n\n\ndef encode_filename(filename):\n    \"\"\"Encode unicode filename\n    \"\"\"\n    if isinstance(filename, text_type):\n        return filename.encode(sys.getfilesystemencoding())\n    else:\n        return filename\n\n\ndef decode_input(s):\n    \"\"\"Given input string/bytes, decode according to stdin codepage (or UTF-8)\n    if not defined\n\n    If fails -- issue warning and decode allowing for errors\n    being replaced\n    \"\"\"\n    if isinstance(s, text_type):\n        return s\n    else:\n        encoding = sys.stdin.encoding or 'UTF-8'\n        try:\n            return s.decode(encoding)\n        except UnicodeDecodeError as exc:\n            lgr.warning(\n                \"Failed to decode input string using %s encoding. \"\n                \"Decoding allowing for errors\", encoding)\n            return s.decode(encoding, errors='replace')\n\n\nif on_windows:\n    def lmtime(filepath, mtime):\n        \"\"\"Set mtime for files.  On Windows a merely adapter to os.utime\n        \"\"\"\n        os.utime(filepath, (time.time(), mtime))\nelse:\n    def lmtime(filepath, mtime):\n        \"\"\"Set mtime for files, while not de-referencing symlinks.\n\n        To overcome absence of os.lutime\n\n        Works only on linux and OSX ATM\n        \"\"\"\n        from .cmd import Runner\n        # convert mtime to format touch understands [[CC]YY]MMDDhhmm[.SS]\n        smtime = time.strftime(\"%Y%m%d%H%M.%S\", time.localtime(mtime))\n        lgr.log(3, \"Setting mtime for %s to %s == %s\", filepath, mtime, smtime)\n        Runner().run(['touch', '-h', '-t', '%s' % smtime, filepath])\n        rfilepath = realpath(filepath)\n        if islink(filepath) and exists(rfilepath):\n            # trust noone - adjust also of the target file\n            # since it seemed like downloading under OSX (was it using curl?)\n            # didn't bother with timestamps\n            lgr.log(3, \"File is a symlink to %s Setting mtime for it to %s\",\n                    rfilepath, mtime)\n            os.utime(rfilepath, (time.time(), mtime))\n        # doesn't work on OSX\n        # Runner().run(['touch', '-h', '-d', '@%s' % mtime, filepath])\n\n\ndef assure_tuple_or_list(obj):\n    \"\"\"Given an object, wrap into a tuple if not list or tuple\n    \"\"\"\n    if isinstance(obj, (list, tuple)):\n        return obj\n    return (obj,)\n\n\ndef assure_iter(s, cls, copy=False, iterate=True):\n    \"\"\"Given not a list, would place it into a list. If None - empty list is returned\n\n    Parameters\n    ----------\n    s: list or anything\n    cls: class\n      Which iterable class to assure\n    copy: bool, optional\n      If correct iterable is passed, it would generate its shallow copy\n    iterate: bool, optional\n      If it is not a list, but something iterable (but not a text_type)\n      iterate over it.\n    \"\"\"\n\n    if isinstance(s, cls):\n        return s if not copy else shallow_copy(s)\n    elif isinstance(s, text_type):\n        return cls((s,))\n    elif iterate and hasattr(s, '__iter__'):\n        return cls(s)\n    elif s is None:\n        return cls()\n    else:\n        return cls((s,))\n\n\ndef assure_list(s, copy=False, iterate=True):\n    \"\"\"Given not a list, would place it into a list. If None - empty list is returned\n\n    Parameters\n    ----------\n    s: list or anything\n    copy: bool, optional\n      If list is passed, it would generate a shallow copy of the list\n    iterate: bool, optional\n      If it is not a list, but something iterable (but not a text_type)\n      iterate over it.\n    \"\"\"\n    return assure_iter(s, list, copy=copy, iterate=iterate)\n\n\ndef assure_list_from_str(s, sep='\\n'):\n    \"\"\"Given a multiline string convert it to a list of return None if empty\n\n    Parameters\n    ----------\n    s: str or list\n    \"\"\"\n\n    if not s:\n        return None\n\n    if isinstance(s, list):\n        return s\n    return s.split(sep)\n\n\ndef assure_dict_from_str(s, **kwargs):\n    \"\"\"Given a multiline string with key=value items convert it to a dictionary\n\n    Parameters\n    ----------\n    s: str or dict\n\n    Returns None if input s is empty\n    \"\"\"\n\n    if not s:\n        return None\n\n    if isinstance(s, dict):\n        return s\n\n    out = {}\n    for value_str in assure_list_from_str(s, **kwargs):\n        if '=' not in value_str:\n            raise ValueError(\"{} is not in key=value format\".format(repr(value_str)))\n        k, v = value_str.split('=', 1)\n        if k in out:\n            err = \"key {} was already defined in {}, but new value {} was provided\".format(k, out, v)\n            raise ValueError(err)\n        out[k] = v\n    return out\n\n\ndef assure_bytes(s, encoding='utf-8'):\n    \"\"\"Convert/encode unicode to str (PY2) or bytes (PY3) if of 'text_type'\n\n    Parameters\n    ----------\n    encoding: str, optional\n      Encoding to use.  \"utf-8\" is the default\n    \"\"\"\n    if not isinstance(s, text_type):\n        return s\n    return s.encode(encoding)\n\n\ndef assure_unicode(s, encoding=None, confidence=None):\n    \"\"\"Convert/decode to unicode (PY2) or str (PY3) if of 'binary_type'\n\n    Parameters\n    ----------\n    encoding: str, optional\n      Encoding to use.  If None, \"utf-8\" is tried, and then if not a valid\n      UTF-8, encoding will be guessed\n    confidence: float, optional\n      A value between 0 and 1, so if guessing of encoding is of lower than\n      specified confidence, ValueError is raised\n    \"\"\"\n    if not isinstance(s, binary_type):\n        return s\n    if encoding is None:\n        # Figure out encoding, defaulting to 'utf-8' which is our common\n        # target in contemporary digital society\n        try:\n            return s.decode('utf-8')\n        except UnicodeDecodeError as exc:\n            from .dochelpers import exc_str\n            lgr.debug(\"Failed to decode a string as utf-8: %s\", exc_str(exc))\n        # And now we could try to guess\n        from chardet import detect\n        enc = detect(s)\n        denc = enc.get('encoding', None)\n        if denc:\n            denc_confidence = enc.get('confidence', 0)\n            if confidence is not None and  denc_confidence < confidence:\n                raise ValueError(\n                    \"Failed to auto-detect encoding with high enough \"\n                    \"confidence. Highest confidence was %s for %s\"\n                    % (denc_confidence, denc)\n                )\n            return s.decode(denc)\n        else:\n            raise ValueError(\n                \"Could not decode value as utf-8, or to guess its encoding: %s\"\n                % repr(s)\n            )\n    else:\n        return s.decode(encoding)\n\n\ndef assure_bool(s):\n    \"\"\"Convert value into boolean following convention for strings\n\n    to recognize on,True,yes as True, off,False,no as False\n    \"\"\"\n    if isinstance(s, string_types):\n        if s.isdigit():\n            return bool(int(s))\n        sl = s.lower()\n        if sl in {'y', 'yes', 'true', 'on'}:\n            return True\n        elif sl in {'n', 'no', 'false', 'off'}:\n            return False\n        else:\n            raise ValueError(\"Do not know how to treat %r as a boolean\" % s)\n    return bool(s)\n\n\ndef as_unicode(val, cast_types=object):\n    \"\"\"Given an arbitrary value, would try to obtain unicode value of it\n    \n    For unicode it would return original value, for python2 str or python3\n    bytes it would use assure_unicode, for None - an empty (unicode) string,\n    and for any other type (see `cast_types`) - would apply the unicode \n    constructor.  If value is not an instance of `cast_types`, TypeError\n    is thrown\n    \n    Parameters\n    ----------\n    cast_types: type\n      Which types to cast to unicode by providing to constructor\n    \"\"\"\n    if val is None:\n        return u''\n    elif isinstance(val, text_type):\n        return val\n    elif isinstance(val, unicode_srctypes):\n        return assure_unicode(val)\n    elif isinstance(val, cast_types):\n        return text_type(val)\n    else:\n        raise TypeError(\n            \"Value %r is not of any of known or provided %s types\"\n            % (val, cast_types))\n\n\ndef unique(seq, key=None):\n    \"\"\"Given a sequence return a list only with unique elements while maintaining order\n\n    This is the fastest solution.  See\n    https://www.peterbe.com/plog/uniqifiers-benchmark\n    and\n    http://stackoverflow.com/a/480227/1265472\n    for more information.\n    Enhancement -- added ability to compare for uniqueness using a key function\n\n    Parameters\n    ----------\n    seq:\n      Sequence to analyze\n    key: callable, optional\n      Function to call on each element so we could decide not on a full\n      element, but on its member etc\n    \"\"\"\n    seen = set()\n    seen_add = seen.add\n    if not key:\n        return [x for x in seq if not (x in seen or seen_add(x))]\n    else:\n        # OPT: could be optimized, since key is called twice, but for our cases\n        # should be just as fine\n        return [x for x in seq if not (key(x) in seen or seen_add(key(x)))]\n\n\ndef all_same(items):\n    \"\"\"Quick check if all items are the same.\n\n    Identical to a check like len(set(items)) == 1 but\n    should be more efficient while working on generators, since would\n    return False as soon as any difference detected thus possibly avoiding\n    unnecessary evaluations\n    \"\"\"\n    first = True\n    first_item = None\n    for item in items:\n        if first:\n            first = False\n            first_item = item\n        else:\n            if item != first_item:\n                return False\n    # So we return False if was empty\n    return not first\n\n\ndef map_items(func, v):\n    \"\"\"A helper to apply `func` to all elements (keys and values) within dict\n\n    No type checking of values passed to func is done, so `func`\n    should be resilient to values which it should not handle\n\n    Initial usecase - apply_recursive(url_fragment, assure_unicode)\n    \"\"\"\n    # map all elements within item\n    return v.__class__(\n        item.__class__(map(func, item))\n        for item in v.items()\n    )\n\n\ndef partition(items, predicate=bool):\n    \"\"\"Partition `items` by `predicate`.\n\n    Parameters\n    ----------\n    items : iterable\n    predicate : callable\n        A function that will be mapped over each element in `items`. The\n        elements will partitioned based on whether the return value is false or\n        true.\n\n    Returns\n    -------\n    A tuple with two generators, the first for 'false' items and the second for\n    'true' ones.\n\n    Notes\n    -----\n    Taken from Peter Otten's snippet posted at\n    https://nedbatchelder.com/blog/201306/filter_a_list_into_two_parts.html\n    \"\"\"\n    a, b = tee((predicate(item), item) for item in items)\n    return ((item for pred, item in a if not pred),\n            (item for pred, item in b if pred))\n\n\ndef generate_chunks(container, size):\n    \"\"\"Given a container, generate chunks from it with size up to `size`\n    \"\"\"\n    # There could be a \"smarter\" solution but I think this would suffice\n    assert size > 0,  \"Size should be non-0 positive\"\n    while container:\n        yield container[:size]\n        container = container[size:]\n\n#\n# Generators helpers\n#\n\ndef saved_generator(gen):\n    \"\"\"Given a generator returns two generators, where 2nd one just replays\n\n    So the first one would be going through the generated items and 2nd one\n    would be yielding saved items\n    \"\"\"\n    saved = []\n\n    def gen1():\n        for x in gen:  # iterating over original generator\n            saved.append(x)\n            yield x\n\n    def gen2():\n        for x in saved:  # yielding saved entries\n            yield x\n\n    return gen1(), gen2()\n\n\n#\n# Decorators\n#\ndef better_wraps(to_be_wrapped):\n    \"\"\"Decorator to replace `functools.wraps`\n\n    This is based on `wrapt` instead of `functools` and in opposition to `wraps`\n    preserves the correct signature of the decorated function.\n    It is written with the intention to replace the use of `wraps` without any\n    need to rewrite the actual decorators.\n    \"\"\"\n\n    @wrapt.decorator(adapter=to_be_wrapped)\n    def intermediator(to_be_wrapper, instance, args, kwargs):\n        return to_be_wrapper(*args, **kwargs)\n\n    return intermediator\n\n\n# Borrowed from pandas\n# Copyright: 2011-2014, Lambda Foundry, Inc. and PyData Development Team\n# License: BSD-3\ndef optional_args(decorator):\n    \"\"\"allows a decorator to take optional positional and keyword arguments.\n        Assumes that taking a single, callable, positional argument means that\n        it is decorating a function, i.e. something like this::\n\n            @my_decorator\n            def function(): pass\n\n        Calls decorator with decorator(f, `*args`, `**kwargs`)\"\"\"\n\n    @better_wraps(decorator)\n    def wrapper(*args, **kwargs):\n        def dec(f):\n            return decorator(f, *args, **kwargs)\n\n        is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable)\n        if is_decorating:\n            f = args[0]\n            args = []\n            return dec(f)\n        else:\n            return dec\n\n    return wrapper\n\n\n# TODO: just provide decorators for tempfile.mk* functions. This is ugly!\ndef get_tempfile_kwargs(tkwargs=None, prefix=\"\", wrapped=None):\n    \"\"\"Updates kwargs to be passed to tempfile. calls depending on env vars\n    \"\"\"\n    if tkwargs is None:\n        tkwargs_ = {}\n    else:\n        # operate on a copy of tkwargs to avoid any side-effects\n        tkwargs_ = tkwargs.copy()\n\n    # TODO: don't remember why I had this one originally\n    # if len(targs)<2 and \\\n    if 'prefix' not in tkwargs_:\n        tkwargs_['prefix'] = '_'.join(\n            ['datalad_temp'] +\n            ([prefix] if prefix else []) +\n            ([''] if (on_windows or not wrapped) else [wrapped.__name__]))\n\n    directory = os.environ.get('DATALAD_TESTS_TEMP_DIR')\n    if directory and 'dir' not in tkwargs_:\n        tkwargs_['dir'] = directory\n\n    return tkwargs_\n\n\n@optional_args\ndef line_profile(func):\n    \"\"\"Q&D helper to line profile the function and spit out stats\n    \"\"\"\n    import line_profiler\n    prof = line_profiler.LineProfiler()\n\n    @wraps(func)\n    def newfunc(*args, **kwargs):\n        try:\n            pfunc = prof(func)\n            return pfunc(*args, **kwargs)\n        finally:\n            prof.print_stats()\n    return newfunc\n\n#\n# Context Managers\n#\n\n\n@contextmanager\ndef nothing_cm():\n    \"\"\"Just a dummy cm to programmically switch context managers\"\"\"\n    yield\n\n\n@contextmanager\ndef swallow_outputs():\n    \"\"\"Context manager to help consuming both stdout and stderr, and print()\n\n    stdout is available as cm.out and stderr as cm.err whenever cm is the\n    yielded context manager.\n    Internally uses temporary files to guarantee absent side-effects of swallowing\n    into StringIO which lacks .fileno.\n\n    print mocking is necessary for some uses where sys.stdout was already bound\n    to original sys.stdout, thus mocking it later had no effect. Overriding\n    print function had desired effect\n    \"\"\"\n\n    class StringIOAdapter(object):\n        \"\"\"Little adapter to help getting out/err values\n        \"\"\"\n        def __init__(self):\n            kw = get_tempfile_kwargs({}, prefix=\"outputs\")\n\n            self._out = open(tempfile.mktemp(**kw), 'w')\n            self._err = open(tempfile.mktemp(**kw), 'w')\n\n        def _read(self, h):\n            with open(h.name) as f:\n                return f.read()\n\n        @property\n        def out(self):\n            if not self._out.closed:\n                self._out.flush()\n            return self._read(self._out)\n\n        @property\n        def err(self):\n            if not self._err.closed:\n                self._err.flush()\n            return self._read(self._err)\n\n        @property\n        def handles(self):\n            return self._out, self._err\n\n        def cleanup(self):\n            self._out.close()\n            self._err.close()\n            out_name = self._out.name\n            err_name = self._err.name\n            del self._out\n            del self._err\n            gc.collect()\n            rmtemp(out_name)\n            rmtemp(err_name)\n\n    def fake_print(*args, **kwargs):\n        sep = kwargs.pop('sep', ' ')\n        end = kwargs.pop('end', '\\n')\n        file = kwargs.pop('file', sys.stdout)\n\n        if file in (oldout, olderr, sys.stdout, sys.stderr):\n            # we mock\n            try:\n                sys.stdout.write(sep.join(args) + end)\n            except UnicodeEncodeError as exc:\n                lgr.error(\n                    \"Failed to write to mocked stdout, got %s, continue as it \"\n                    \"didn't happen\",  exc)\n        else:\n            # must be some other file one -- leave it alone\n            oldprint(*args, sep=sep, end=end, file=file)\n\n    from .ui import ui\n    # preserve -- they could have been mocked already\n    oldprint = getattr(__builtin__, 'print')\n    oldout, olderr = sys.stdout, sys.stderr\n    olduiout = ui.out\n    adapter = StringIOAdapter()\n\n    try:\n        sys.stdout, sys.stderr = adapter.handles\n        ui.out = adapter.handles[0]\n        setattr(__builtin__, 'print', fake_print)\n\n        yield adapter\n    finally:\n        sys.stdout, sys.stderr, ui.out = oldout, olderr, olduiout\n        setattr(__builtin__, 'print',  oldprint)\n        adapter.cleanup()\n\n\n@contextmanager\ndef swallow_logs(new_level=None, file_=None, name='datalad'):\n    \"\"\"Context manager to consume all logs.\n\n    \"\"\"\n    lgr = logging.getLogger(name)\n\n    # Keep old settings\n    old_level = lgr.level\n    old_handlers = lgr.handlers\n\n    # Let's log everything into a string\n    # TODO: generalize with the one for swallow_outputs\n    class StringIOAdapter(object):\n        \"\"\"Little adapter to help getting out values\n\n        And to stay consistent with how swallow_outputs behaves\n        \"\"\"\n        def __init__(self):\n            if file_ is None:\n                kw = get_tempfile_kwargs({}, prefix=\"logs\")\n                out_file = tempfile.mktemp(**kw)\n            else:\n                out_file = file_\n            # PY3 requires clearly one or another.  race condition possible\n            self._out = open(out_file, 'a')\n            self._final_out = None\n\n        def _read(self, h):\n            with open(h.name) as f:\n                return f.read()\n\n        @property\n        def out(self):\n            if self._final_out is not None:\n                # we closed and cleaned up already\n                return self._final_out\n            else:\n                self._out.flush()\n                return self._read(self._out)\n\n        @property\n        def lines(self):\n            return self.out.split('\\n')\n\n        @property\n        def handle(self):\n            return self._out\n\n        def cleanup(self):\n            # store for access while object exists\n            self._final_out = self.out\n            self._out.close()\n            out_name = self._out.name\n            del self._out\n            gc.collect()\n            if not file_:\n                rmtemp(out_name)\n\n        def assert_logged(self, msg=None, level=None, regex=True, **kwargs):\n            \"\"\"Provide assertion on whether a msg was logged at a given level\n\n            If neither `msg` nor `level` provided, checks if anything was logged\n            at all.\n\n            Parameters\n            ----------\n            msg: str, optional\n              Message (as a regular expression, if `regex`) to be searched.\n              If no msg provided, checks if anything was logged at a given level.\n            level: str, optional\n              String representing the level to be logged\n            regex: bool, optional\n              If False, regular `assert_in` is used\n            **kwargs: str, optional\n              Passed to `assert_re_in` or `assert_in`\n            \"\"\"\n            from datalad.tests.utils import assert_re_in\n            from datalad.tests.utils import assert_in\n\n            if regex:\n                match = '\\[%s\\] ' % level if level else \"\\[\\S+\\] \"\n            else:\n                match = '[%s] ' % level if level else ''\n\n            if msg:\n                match += msg\n\n            if match:\n                (assert_re_in if regex else assert_in)(match, self.out, **kwargs)\n            else:\n                assert not kwargs, \"no kwargs to be passed anywhere\"\n                assert self.out, \"Nothing was logged!?\"\n\n    adapter = StringIOAdapter()\n    # TODO: it does store messages but without any formatting, i.e. even without\n    # date/time prefix etc.  IMHO it should preserve formatting in case if file_ is\n    # set\n    swallow_handler = logging.StreamHandler(adapter.handle)\n    # we want to log levelname so we could test against it\n    swallow_handler.setFormatter(\n        logging.Formatter('[%(levelname)s] %(message)s'))\n    # Inherit filters\n    from datalad.log import ProgressHandler\n    swallow_handler.filters = sum([h.filters for h in old_handlers\n                                   if not isinstance(h, ProgressHandler)],\n                                  [])\n    lgr.handlers = [swallow_handler]\n    if old_level < logging.DEBUG:  # so if HEAVYDEBUG etc -- show them!\n        lgr.handlers += old_handlers\n\n    if isinstance(new_level, str):\n        new_level = getattr(logging, new_level)\n\n    if new_level is not None:\n        lgr.setLevel(new_level)\n\n    try:\n        yield adapter\n        # TODO: if file_ and there was an exception -- most probably worth logging it?\n        # although ideally it should be the next log outside added to that file_ ... oh well\n    finally:\n        lgr.handlers, lgr.level = old_handlers, old_level\n        adapter.cleanup()\n\n\n# TODO: May be melt in with swallow_logs at some point:\n@contextmanager\ndef disable_logger(logger=None):\n    \"\"\"context manager to temporarily disable logging\n\n    This is to provide one of swallow_logs' purposes without unnecessarily\n    creating temp files (see gh-1865)\n\n    Parameters\n    ----------\n    logger: Logger\n        Logger whose handlers will be ordered to not log anything.\n        Default: datalad's topmost Logger ('datalad')\n    \"\"\"\n\n    class NullFilter(logging.Filter):\n        \"\"\"Filter class to reject all records\n        \"\"\"\n        def filter(self, record):\n            return 0\n\n    if logger is None:\n        # default: all of datalad's logging:\n        logger = logging.getLogger('datalad')\n\n    filter_ = NullFilter(logger.name)\n    [h.addFilter(filter_) for h in logger.handlers]\n\n    try:\n        yield logger\n    finally:\n        [h.removeFilter(filter_) for h in logger.handlers]\n\n\n#\n# Additional handlers\n#\n_sys_excepthook = sys.excepthook  # Just in case we ever need original one\n\n\ndef setup_exceptionhook(ipython=False):\n    \"\"\"Overloads default sys.excepthook with our exceptionhook handler.\n\n       If interactive, our exceptionhook handler will invoke\n       pdb.post_mortem; if not interactive, then invokes default handler.\n    \"\"\"\n\n    def _datalad_pdb_excepthook(type, value, tb):\n        import traceback\n        traceback.print_exception(type, value, tb)\n        print()\n        if is_interactive():\n            import pdb\n            pdb.post_mortem(tb)\n\n    if ipython:\n        from IPython.core import ultratb\n        sys.excepthook = ultratb.FormattedTB(mode='Verbose',\n                                             # color_scheme='Linux',\n                                             call_pdb=is_interactive())\n    else:\n        sys.excepthook = _datalad_pdb_excepthook\n\n\ndef assure_dir(*args):\n    \"\"\"Make sure directory exists.\n\n    Joins the list of arguments to an os-specific path to the desired\n    directory and creates it, if it not exists yet.\n    \"\"\"\n    dirname = opj(*args)\n    if not exists(dirname):\n        os.makedirs(dirname)\n    return dirname\n\n\ndef updated(d, update):\n    \"\"\"Return a copy of the input with the 'update'\n\n    Primarily for updating dictionaries\n    \"\"\"\n    d = d.copy()\n    d.update(update)\n    return d\n\n\n_pwd_mode = None\n\n\ndef _switch_to_getcwd(msg, *args):\n    global _pwd_mode\n    _pwd_mode = 'cwd'\n    lgr.warning(\n        msg + \". From now on will be returning os.getcwd(). Directory\"\n               \" symlinks in the paths will be resolved\",\n        *args\n    )\n    # TODO:  we might want to mitigate by going through all flywheighted\n    # repos and tuning up their .paths to be resolved?\n\n\ndef getpwd():\n    \"\"\"Try to return a CWD without dereferencing possible symlinks\n\n    This function will try to use PWD environment variable to provide a current\n    working directory, possibly with some directories along the path being\n    symlinks to other directories.  Unfortunately, PWD is used/set only by the\n    shell and such functions as `os.chdir` and `os.getcwd` nohow use or modify\n    it, thus `os.getcwd()` returns path with links dereferenced.\n\n    While returning current working directory based on PWD env variable we\n    verify that the directory is the same as `os.getcwd()` after resolving all\n    symlinks.  If that verification fails, we fall back to always use\n    `os.getcwd()`.\n\n    Initial decision to either use PWD env variable or os.getcwd() is done upon\n    the first call of this function.\n    \"\"\"\n    global _pwd_mode\n    if _pwd_mode is None:\n        # we need to decide!\n        try:\n            pwd = os.environ['PWD']\n            if on_windows and pwd and pwd.startswith('/'):\n                # It should be a path from MSYS.\n                # - it might start with a drive letter or not\n                # - it seems to be \"illegal\" to have a single letter directories\n                #   under / path, i.e. if created - they aren't found\n                # - 'ln -s' does not fail to create a \"symlink\" but it just\n                # copies!\n                #   so we are not likely to need original PWD purpose on\n                # those systems\n                # Verdict:\n                _pwd_mode = 'cwd'\n            else:\n                _pwd_mode = 'PWD'\n        except KeyError:\n            _pwd_mode = 'cwd'\n\n    if _pwd_mode == 'cwd':\n        return os.getcwd()\n    elif _pwd_mode == 'PWD':\n        try:\n            cwd = os.getcwd()\n        except OSError as exc:\n            if \"o such file\" in str(exc):\n                # directory was removed but we promised to be robust and\n                # still report the path we might know since we are still in PWD\n                # mode\n                cwd = None\n            else:\n                raise\n        try:\n            pwd = os.environ['PWD']\n            pwd_real = op.realpath(pwd)\n            # This logic would fail to catch the case where chdir did happen\n            # to the directory where current PWD is pointing to, e.g.\n            # $> ls -ld $PWD\n            # lrwxrwxrwx 1 yoh yoh 5 Oct 11 13:27 /home/yoh/.tmp/tmp -> /tmp//\n            # hopa:~/.tmp/tmp\n            # $> python -c 'import os; os.chdir(\"/tmp\"); from datalad.utils import getpwd; print(getpwd(), os.getcwd())'\n            # ('/home/yoh/.tmp/tmp', '/tmp')\n            # but I guess that should not be too harmful\n            if cwd is not None and pwd_real != cwd:\n                _switch_to_getcwd(\n                    \"realpath of PWD=%s is %s whenever os.getcwd()=%s\",\n                    pwd, pwd_real, cwd\n                )\n                return cwd\n            return pwd\n        except KeyError:\n            _switch_to_getcwd(\"PWD env variable is no longer available\")\n            return cwd  # Must not happen, but may be someone\n                        # evil purges PWD from environ?\n    else:\n        raise RuntimeError(\n            \"Must have not got here. \"\n            \"pwd_mode must be either cwd or PWD. And it is now %r\" % (_pwd_mode,)\n        )\n\n\nclass chpwd(object):\n    \"\"\"Wrapper around os.chdir which also adjusts environ['PWD']\n\n    The reason is that otherwise PWD is simply inherited from the shell\n    and we have no ability to assess directory path without dereferencing\n    symlinks.\n\n    If used as a context manager it allows to temporarily change directory\n    to the given path\n    \"\"\"\n    def __init__(self, path, mkdir=False, logsuffix=''):\n\n        if path:\n            pwd = getpwd()\n            self._prev_pwd = pwd\n        else:\n            self._prev_pwd = None\n            return\n\n        if not isabs(path):\n            path = normpath(opj(pwd, path))\n        if not os.path.exists(path) and mkdir:\n            self._mkdir = True\n            os.mkdir(path)\n        else:\n            self._mkdir = False\n        lgr.debug(\"chdir %r -> %r %s\", self._prev_pwd, path, logsuffix)\n        os.chdir(path)  # for grep people -- ok, to chdir here!\n        os.environ['PWD'] = path\n\n    def __enter__(self):\n        # nothing more to do really, chdir was in the constructor\n        pass\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        if self._prev_pwd:\n            # Need to use self.__class__ so this instance, if the entire\n            # thing mocked during the test, still would use correct chpwd\n            self.__class__(self._prev_pwd, logsuffix=\"(coming back)\")\n\n\ndef dlabspath(path, norm=False):\n    \"\"\"Symlinks-in-the-cwd aware abspath\n\n    os.path.abspath relies on os.getcwd() which would not know about symlinks\n    in the path\n\n    TODO: we might want to norm=True by default to match behavior of\n    os .path.abspath?\n    \"\"\"\n    if not isabs(path):\n        # if not absolute -- relative to pwd\n        path = opj(getpwd(), path)\n    return normpath(path) if norm else path\n\n\ndef with_pathsep(path):\n    \"\"\"Little helper to guarantee that path ends with /\"\"\"\n    return path + sep if not path.endswith(sep) else path\n\n\ndef get_path_prefix(path, pwd=None):\n    \"\"\"Get path prefix (for current directory)\n\n    Returns relative path to the topdir, if we are under topdir, and if not\n    absolute path to topdir.  If `pwd` is not specified - current directory\n    assumed\n    \"\"\"\n    pwd = pwd or getpwd()\n    path = dlabspath(path)\n    path_ = with_pathsep(path)\n    pwd_ = with_pathsep(pwd)\n    common = commonprefix((path_, pwd_))\n    if common.endswith(sep) and common in {path_, pwd_}:\n        # we are in subdir or above the path = use relative path\n        location_prefix = relpath(path, pwd)\n        # if benign \"here\" - cut off\n        if location_prefix in (curdir, curdir + sep):\n            location_prefix = ''\n        return location_prefix\n    else:\n        # just return absolute path\n        return path\n\n\ndef _get_normalized_paths(path, prefix):\n    if isabs(path) != isabs(prefix):\n        raise ValueError(\"Both paths must either be absolute or relative. \"\n                         \"Got %r and %r\" % (path, prefix))\n    path = with_pathsep(path)\n    prefix = with_pathsep(prefix)\n    return path, prefix\n\n\ndef path_startswith(path, prefix):\n    \"\"\"Return True if path starts with prefix path\n\n    Parameters\n    ----------\n    path: str\n    prefix: str\n    \"\"\"\n    path, prefix = _get_normalized_paths(path, prefix)\n    return path.startswith(prefix)\n\n\ndef path_is_subpath(path, prefix):\n    \"\"\"Return True if path is a subpath of prefix\n\n    It will return False if path == prefix.\n\n    Parameters\n    ----------\n    path: str\n    prefix: str\n    \"\"\"\n    path, prefix = _get_normalized_paths(path, prefix)\n    return (len(prefix) < len(path)) and path.startswith(prefix)\n\n\ndef knows_annex(path):\n    \"\"\"Returns whether at a given path there is information about an annex\n\n    It is just a thin wrapper around GitRepo.is_with_annex() classmethod\n    which also checks for `path` to exist first.\n\n    This includes actually present annexes, but also uninitialized ones, or\n    even the presence of a remote annex branch.\n    \"\"\"\n    from os.path import exists\n    if not exists(path):\n        lgr.debug(\"No annex: test path {0} doesn't exist\".format(path))\n        return False\n    from datalad.support.gitrepo import GitRepo\n    return GitRepo(path, init=False, create=False).is_with_annex()\n\n\n@contextmanager\ndef make_tempfile(content=None, wrapped=None, **tkwargs):\n    \"\"\"Helper class to provide a temporary file name and remove it at the end (context manager)\n\n    Parameters\n    ----------\n    mkdir : bool, optional (default: False)\n        If True, temporary directory created using tempfile.mkdtemp()\n    content : str or bytes, optional\n        Content to be stored in the file created\n    wrapped : function, optional\n        If set, function name used to prefix temporary file name\n    `**tkwargs`:\n        All other arguments are passed into the call to tempfile.mk{,d}temp(),\n        and resultant temporary filename is passed as the first argument into\n        the function t.  If no 'prefix' argument is provided, it will be\n        constructed using module and function names ('.' replaced with\n        '_').\n\n    To change the used directory without providing keyword argument 'dir' set\n    DATALAD_TESTS_TEMP_DIR.\n\n    Examples\n    --------\n        >>> from os.path import exists\n        >>> from datalad.utils import make_tempfile\n        >>> with make_tempfile() as fname:\n        ...    k = open(fname, 'w').write('silly test')\n        >>> assert not exists(fname)  # was removed\n\n        >>> with make_tempfile(content=\"blah\") as fname:\n        ...    assert open(fname).read() == \"blah\"\n    \"\"\"\n\n    if tkwargs.get('mkdir', None) and content is not None:\n        raise ValueError(\"mkdir=True while providing content makes no sense\")\n\n    tkwargs_ = get_tempfile_kwargs(tkwargs, wrapped=wrapped)\n\n    # if DATALAD_TESTS_TEMP_DIR is set, use that as directory,\n    # let mktemp handle it otherwise. However, an explicitly provided\n    # dir=... will override this.\n    mkdir = tkwargs_.pop('mkdir', False)\n\n    filename = {False: tempfile.mktemp,\n                True: tempfile.mkdtemp}[mkdir](**tkwargs_)\n    filename = realpath(filename)\n\n    if content:\n        with open(filename, 'w' + ('b' if isinstance(content, binary_type) else '')) as f:\n            f.write(content)\n\n    if __debug__:\n        # TODO mkdir\n        lgr.debug('Created temporary thing named %s\"' % filename)\n    try:\n        yield filename\n    finally:\n        # glob here for all files with the same name (-suffix)\n        # would be useful whenever we requested .img filename,\n        # and function creates .hdr as well\n        lsuffix = len(tkwargs_.get('suffix', ''))\n        filename_ = lsuffix and filename[:-lsuffix] or filename\n        filenames = glob.glob(filename_ + '*')\n        if len(filename_) < 3 or len(filenames) > 5:\n            # For paranoid yoh who stepped into this already ones ;-)\n            lgr.warning(\"It is unlikely that it was intended to remove all\"\n                        \" files matching %r. Skipping\" % filename_)\n            return\n        for f in filenames:\n            try:\n                rmtemp(f)\n            except OSError:  # pragma: no cover\n                pass\n\n\ndef _path_(*p):\n    \"\"\"Given a path in POSIX\" notation, regenerate one in native to the env one\"\"\"\n    if on_windows:\n        return opj(*map(lambda x: opj(*x.split('/')), p))\n    else:\n        # Assume that all others as POSIX compliant so nothing to be done\n        return opj(*p)\n\n\ndef get_timestamp_suffix(time_=None, prefix='-'):\n    \"\"\"Return a time stamp (full date and time up to second)\n\n    primarily to be used for generation of log files names\n    \"\"\"\n    args = []\n    if time_ is not None:\n        if isinstance(time_, int):\n            time_ = time.gmtime(time_)\n        args.append(time_)\n    return time.strftime(prefix + TIMESTAMP_FMT, *args)\n\n\ndef get_logfilename(dspath, cmd='datalad'):\n    \"\"\"Return a filename to use for logging under a dataset/repository\n\n    directory would be created if doesn't exist, but dspath must exist\n    and be a directory\n    \"\"\"\n    assert(exists(dspath))\n    assert(isdir(dspath))\n    ds_logdir = assure_dir(dspath, '.git', 'datalad', 'logs')  # TODO: use WEB_META_LOG whenever #789 merged\n    return opj(ds_logdir, 'crawl-%s.log' % get_timestamp_suffix())\n\n\ndef get_trace(edges, start, end, trace=None):\n    \"\"\"Return the trace/path to reach a node in a tree.\n\n    Parameters\n    ----------\n    edges : sequence(2-tuple)\n      The tree given by a sequence of edges (parent, child) tuples. The\n      nodes can be identified by any value and data type that supports\n      the '==' operation.\n    start :\n      Identifier of the start node. Must be present as a value in the parent\n      location of an edge tuple in order to be found.\n    end :\n      Identifier of the target/end node. Must be present as a value in the child\n      location of an edge tuple in order to be found.\n    trace : list\n      Mostly useful for recursive calls, and used internally.\n\n    Returns\n    -------\n    None or list\n      Returns a list with the trace to the target (the starts and the target\n      are not included in the trace, hence if start and end are directly connected\n      an empty list is returned), or None when no trace to the target can be found,\n      or start and end are identical.\n    \"\"\"\n    # the term trace is used to avoid confusion with a path in the sense\n    # of a filesystem path, but the analogy fits and nodes can be paths\n    if trace is None:\n        trace = []\n    if not edges:\n        raise ValueError(\"no edges given\")\n    for cand in edges:\n        cand_super, cand_sub = cand\n        if cand_sub in trace:\n            # only DAGs, skip any cyclic traces\n            continue\n        if trace and cand_super != trace[-1]:\n            # only consider edges that lead off the end of the trace\n            continue\n        if not trace and cand_super != start:\n            # we got nothing yet, and this edges is not matching the start\n            continue\n        if cand_sub == end:\n            return trace\n        # dive into potential subnodes\n        cand_trace = get_trace(\n            edges,\n            start,\n            end,\n            trace + [cand_sub])\n        if cand_trace:\n            return cand_trace\n    return None\n\n\ndef get_dataset_root(path):\n    \"\"\"Return the root of an existent dataset containing a given path\n\n    The root path is returned in the same absolute or relative form\n    as the input argument. If no associated dataset exists, or the\n    input path doesn't exist, None is returned.\n    \"\"\"\n    suffix = '.git'\n    if not isdir(path):\n        path = dirname(path)\n    apath = abspath(path)\n    # while we can still go up\n    while psplit(apath)[1]:\n        if exists(opj(path, suffix)):\n            return path\n        # new test path in the format we got it\n        path = normpath(opj(path, os.pardir))\n        # no luck, next round\n        apath = abspath(path)\n    return None\n\n\ndef get_dataset_pwds(dataset):\n    \"\"\"Return the current directory for the dataset.\n\n    Parameters\n    ----------\n    dataset : Dataset\n\n    Returns\n    -------\n    A tuple, where the first item is the absolute path of the pwd and the\n    second is the pwd relative to the dataset's path.\n    \"\"\"\n    if dataset:\n        pwd = dataset.path\n        rel_pwd = curdir\n    else:\n        # act on the whole dataset if nothing else was specified\n\n        # Follow our generic semantic that if dataset is specified,\n        # paths are relative to it, if not -- relative to pwd\n        pwd = getpwd()\n        # Pass pwd to get_dataset_root instead of os.path.curdir to handle\n        # repos whose leading paths have a symlinked directory (see the\n        # TMPDIR=\"/var/tmp/sym link\" test case).\n        dataset = get_dataset_root(pwd)\n\n        if dataset:\n            rel_pwd = relpath(pwd, dataset)\n        else:\n            rel_pwd = pwd  # and leave handling to caller\n    return pwd, rel_pwd\n\n\n# ATM used in datalad_crawler extension, so do not remove yet\ndef try_multiple(ntrials, exception, base, f, *args, **kwargs):\n    \"\"\"Call f multiple times making exponentially growing delay between the calls\"\"\"\n    from .dochelpers import exc_str\n    for trial in range(1, ntrials+1):\n        try:\n            return f(*args, **kwargs)\n        except exception as exc:\n            if trial == ntrials:\n                raise  # just reraise on the last trial\n            t = base ** trial\n            lgr.warning(\"Caught %s on trial #%d. Sleeping %f and retrying\",\n                        exc_str(exc), trial, t)\n            sleep(t)\n\n\n@optional_args\ndef try_multiple_dec(f, ntrials=None, duration=0.1, exceptions=None, increment_type=None):\n    \"\"\"Decorator to try function multiple times.\n\n    Main purpose is to decorate functions dealing with removal of files/directories\n    and which might need a few seconds to work correctly on Windows which takes\n    its time to release files/directories.\n\n    Parameters\n    ----------\n    ntrials: int, optional\n    duration: float, optional\n      Seconds to sleep before retrying.\n    increment_type: {None, 'exponential'}\n      Note that if it is exponential, duration should typically be > 1.0\n      so it grows with higher power\n\n    \"\"\"\n    from .dochelpers import exc_str\n    if not exceptions:\n        exceptions = (OSError, WindowsError, PermissionError) \\\n            if on_windows else OSError\n    if not ntrials:\n        # Life goes fast on proper systems, no need to delay it much\n        ntrials = 50 if on_windows else 3\n\n    assert increment_type in {None, 'exponential'}\n\n    @wraps(f)\n    def wrapped(*args, **kwargs):\n        t = duration\n        for trial in range(ntrials):\n            try:\n                return f(*args, **kwargs)\n            except exceptions as exc:\n                if increment_type == 'exponential':\n                    t = duration ** (trial + 1)\n                lgr.log(\n                    5,\n                    \"Caught %s on trial #%d. Sleeping %f and retrying\",\n                    exc_str(exc), trial, t)\n                if trial < ntrials - 1:\n                    sleep(t)\n                else:\n                    raise\n\n    return wrapped\n\n\n@try_multiple_dec\ndef unlink(f):\n    \"\"\"'Robust' unlink.  Would try multiple times\n\n    On windows boxes there is evidence for a latency of more than a second\n    until a file is considered no longer \"in-use\".\n    WindowsError is not known on Linux, and if IOError or any other\n    exception\n    is thrown then if except statement has WindowsError in it -- NameError\n    also see gh-2533\n    \"\"\"\n    # Check for open files\n    assert_no_open_files(f)\n    return os.unlink(f)\n\n\n@try_multiple_dec\ndef _rmtree(*args, **kwargs):\n    \"\"\"Just a helper to decorate shutil.rmtree.\n\n    rmtree defined above does more and ideally should not itself be decorated\n    since a recursive definition and does checks for open files inside etc -\n    might be too runtime expensive\n    \"\"\"\n    return shutil.rmtree(*args, **kwargs)\n\n\ndef slash_join(base, extension):\n    \"\"\"Join two strings with a '/', avoiding duplicate slashes\n\n    If any of the strings is None the other is returned as is.\n    \"\"\"\n    if extension is None:\n        return base\n    if base is None:\n        return extension\n    return '/'.join(\n        (base.rstrip('/'),\n         extension.lstrip('/')))\n\n\ndef safe_print(s):\n    \"\"\"Print with protection against UTF-8 encoding errors\"\"\"\n    # A little bit of dance to be able to test this code\n    print_f = getattr(__builtin__, \"print\")\n    try:\n        print_f(s)\n    except UnicodeEncodeError:\n        # failed to encode so let's do encoding while ignoring errors\n        # to print at least something\n        # explicit `or ascii` since somehow on buildbot it seemed to return None\n        s = s.encode(getattr(sys.stdout, 'encoding', 'ascii') or 'ascii', errors='ignore') \\\n            if hasattr(s, 'encode') else s\n        print_f(s.decode())\n\n#\n# IO Helpers\n#\n\ndef open_r_encdetect(fname, readahead=1000):\n    \"\"\"Return a file object in read mode with auto-detected encoding\n\n    This is helpful when dealing with files of unknown encoding.\n\n    Parameters\n    ----------\n    readahead: int, optional\n      How many bytes to read for guessing the encoding type.  If\n      negative - full file will be read\n    \"\"\"\n    from chardet import detect\n    import io\n    # read some bytes from the file\n    with open(fname, 'rb') as f:\n        head = f.read(readahead)\n    enc = detect(head)\n    denc = enc.get('encoding', None)\n    lgr.debug(\"Auto-detected encoding %s for file %s (confidence: %s)\",\n              denc,\n              fname,\n              enc.get('confidence', 'unknown'))\n    return io.open(fname, encoding=denc)\n\n\ndef read_csv_lines(fname, dialect=None, readahead=16384, **kwargs):\n    \"\"\"A generator of dict records from a CSV/TSV\n\n    Automatically guesses the encoding for each record to convert to UTF-8\n\n    Parameters\n    ----------\n    fname: str\n      Filename\n    dialect: str, optional\n      Dialect to specify to csv.reader. If not specified -- guessed from\n      the file, if fails to guess, \"excel-tab\" is assumed\n    readahead: int, optional\n      How many bytes to read from the file to guess the type\n    **kwargs\n      Passed to `csv.reader`\n    \"\"\"\n    import csv\n    if dialect is None:\n        with open(fname) as tsvfile:\n            # add robustness, use a sniffer\n            try:\n                dialect = csv.Sniffer().sniff(tsvfile.read(readahead))\n            except Exception as exc:\n                from .dochelpers import exc_str\n                lgr.warning(\n                    'Could not determine file-format, assuming TSV: %s',\n                    exc_str(exc)\n                )\n                dialect = 'excel-tab'\n\n    kw = {} if PY2 else dict(encoding='utf-8')\n    with open(fname, 'rb' if PY2 else 'r', **kw) as tsvfile:\n        # csv.py doesn't do Unicode; encode temporarily as UTF-8:\n        csv_reader = csv.reader(\n            tsvfile,\n            dialect=dialect,\n            **kwargs\n        )\n        header = None\n        for row in csv_reader:\n            # decode UTF-8 back to Unicode, cell by cell:\n            row_unicode = map(assure_unicode, row)\n            if header is None:\n                header = list(row_unicode)\n            else:\n                yield dict(zip(header, row_unicode))\n\n\ndef import_modules(modnames, pkg, msg=\"Failed to import {module}\", log=lgr.debug):\n    \"\"\"Helper to import a list of modules without failing if N/A\n\n    Parameters\n    ----------\n    modnames: list of str\n      List of module names to import\n    pkg: str\n      Package under which to import\n    msg: str, optional\n      Message template for .format() to log at DEBUG level if import fails.\n      Keys {module} and {package} will be provided and ': {exception}' appended\n    log: callable, optional\n      Logger call to use for logging messages\n    \"\"\"\n    from importlib import import_module\n    _globals = globals()\n    mods_loaded = []\n    if pkg and not pkg in sys.modules:\n        # with python 3.5.1 (ok with 3.5.5) somehow kept running into\n        #  Failed to import dlsub1: Parent module 'dltestm1' not loaded\n        # while running the test. Preloading pkg resolved the issue\n        import_module(pkg)\n    for modname in modnames:\n        try:\n            _globals[modname] = mod = import_module(\n                '.{}'.format(modname),\n                pkg)\n            mods_loaded.append(mod)\n        except Exception as exc:\n            from datalad.dochelpers import exc_str\n            log((msg + ': {exception}').format(\n                module=modname, package=pkg, exception=exc_str(exc)))\n    return mods_loaded\n\n\ndef import_module_from_file(modpath, pkg=None, log=lgr.debug):\n    \"\"\"Import provided module given a path\n\n    TODO:\n    - RF/make use of it in pipeline.py which has similar logic\n    - join with import_modules above?\n\n    Parameters\n    ----------\n    pkg: module, optional\n       If provided, and modpath is under pkg.__path__, relative import will be\n       used\n    \"\"\"\n    assert(modpath.endswith('.py'))  # for now just for .py files\n\n    log(\"Importing %s\" % modpath)\n\n    modname = basename(modpath)[:-3]\n    relmodpath = None\n    if pkg:\n        for pkgpath in pkg.__path__:\n            if path_is_subpath(modpath, pkgpath):\n                # for now relying on having .py extension -- assertion above\n                relmodpath = '.' + relpath(modpath[:-3], pkgpath).replace(sep, '.')\n                break\n\n    try:\n        if relmodpath:\n            from importlib import import_module\n            mod = import_module(relmodpath, pkg.__name__)\n        else:\n            dirname_ = dirname(modpath)\n            try:\n                sys.path.insert(0, dirname_)\n                mod = __import__(modname, level=0)\n            finally:\n                if dirname_ in sys.path:\n                    sys.path.pop(sys.path.index(dirname_))\n                else:\n                    log(\"Expected path %s to be within sys.path, but it was gone!\" % dirname_)\n    except Exception as e:\n        from datalad.dochelpers import exc_str\n        raise RuntimeError(\n            \"Failed to import module from %s: %s\" % (modpath, exc_str(e)))\n\n    return mod\n\n\ndef get_encoding_info():\n    \"\"\"Return a dictionary with various encoding/locale information\"\"\"\n    import sys, locale\n    from collections import OrderedDict\n    return OrderedDict([\n        ('default', sys.getdefaultencoding()),\n        ('filesystem', sys.getfilesystemencoding()),\n        ('locale.prefered', locale.getpreferredencoding()),\n    ])\n\n\ndef get_envvars_info():\n    from collections import OrderedDict\n    envs = []\n    for var, val in os.environ.items():\n        if (\n                var.startswith('PYTHON') or\n                var.startswith('LC_') or\n                var.startswith('GIT_') or\n                var in ('LANG', 'LANGUAGE', 'PATH')\n        ):\n            envs.append((var, val))\n    return OrderedDict(envs)\n\n\n# This class is modified from Snakemake (v5.1.4)\nclass SequenceFormatter(string.Formatter):\n    \"\"\"string.Formatter subclass with special behavior for sequences.\n\n    This class delegates formatting of individual elements to another\n    formatter object. Non-list objects are formatted by calling the\n    delegate formatter's \"format_field\" method. List-like objects\n    (list, tuple, set, frozenset) are formatted by formatting each\n    element of the list according to the specified format spec using\n    the delegate formatter and then joining the resulting strings with\n    a separator (space by default).\n    \"\"\"\n\n    def __init__(self, separator=\" \", element_formatter=string.Formatter(),\n                 *args, **kwargs):\n        self.separator = separator\n        self.element_formatter = element_formatter\n\n    def format_element(self, elem, format_spec):\n        \"\"\"Format a single element\n\n        For sequences, this is called once for each element in a\n        sequence. For anything else, it is called on the entire\n        object. It is intended to be overridden in subclases.\n        \"\"\"\n        return self.element_formatter.format_field(elem, format_spec)\n\n    def format_field(self, value, format_spec):\n        if isinstance(value, (list, tuple, set, frozenset)):\n            return self.separator.join(self.format_element(v, format_spec)\n                                       for v in value)\n        else:\n            return self.format_element(value, format_spec)\n\n\n# TODO: eventually we might want to make use of attr module\nclass File(object):\n    \"\"\"Helper for a file entry in the create_tree/@with_tree\n\n    It allows to define additional settings for entries\n    \"\"\"\n    def __init__(self, name, executable=False):\n        \"\"\"\n\n        Parameters\n        ----------\n        name : str\n          Name of the file\n        executable: bool, optional\n          Make it executable\n        \"\"\"\n        self.name = name\n        self.executable = executable\n\n    def __str__(self):\n        return self.name\n\n\ndef create_tree_archive(path, name, load, overwrite=False, archives_leading_dir=True):\n    \"\"\"Given an archive `name`, create under `path` with specified `load` tree\n    \"\"\"\n    from datalad.support.archives import compress_files\n    dirname = file_basename(name)\n    full_dirname = op.join(path, dirname)\n    os.makedirs(full_dirname)\n    create_tree(full_dirname, load, archives_leading_dir=archives_leading_dir)\n    # create archive\n    if archives_leading_dir:\n        compress_files([dirname], name, path=path, overwrite=overwrite)\n    else:\n        compress_files(list(map(op.basename, glob.glob(opj(full_dirname, '*')))),\n                       opj(op.pardir, name),\n                       path=op.join(path, dirname),\n                       overwrite=overwrite)\n    # remove original tree\n    rmtree(full_dirname)\n\n\ndef create_tree(path, tree, archives_leading_dir=True, remove_existing=False):\n    \"\"\"Given a list of tuples (name, load) create such a tree\n\n    if load is a tuple itself -- that would create either a subtree or an archive\n    with that content and place it into the tree if name ends with .tar.gz\n    \"\"\"\n    lgr.log(5, \"Creating a tree under %s\", path)\n    if not op.exists(path):\n        os.makedirs(path)\n\n    if isinstance(tree, dict):\n        tree = tree.items()\n\n    for file_, load in tree:\n        if isinstance(file_, File):\n            executable = file_.executable\n            name = file_.name\n        else:\n            executable = False\n            name = file_\n        full_name = op.join(path, name)\n        if remove_existing and op.lexists(full_name):\n            rmtree(full_name, chmod_files=True)\n        if isinstance(load, (tuple, list, dict)):\n            if name.endswith('.tar.gz') or name.endswith('.tar') or name.endswith('.zip'):\n                create_tree_archive(\n                    path, name, load,\n                    archives_leading_dir=archives_leading_dir)\n            else:\n                create_tree(\n                    full_name, load,\n                    archives_leading_dir=archives_leading_dir,\n                    remove_existing=remove_existing)\n        else:\n            open_func = open\n            if full_name.endswith('.gz'):\n                open_func = gzip.open\n            with open_func(full_name, \"wb\") as f:\n                f.write(assure_bytes(load, 'utf-8'))\n        if executable:\n            os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)\n\n\nlgr.log(5, \"Done importing datalad.utils\")\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Suggestion Message Uses "Any" Instead of "One", Causing Output Inconsistencies

**Description:**
After updating the suggestion message to use "any" instead of "one" when providing suggestions for unknown arguments, the output no longer matches the expected format. This change leads to inconsistencies, especially in scenarios where multiple suggestions are provided.

**Example:**
When an unknown parent `'muther'` is provided, the output is:

```
error: Failed badly
datalad: Unknown parent 'muther'.  See 'datalad --help'.

Did you mean any of these?
        mutter
        mother
        father
Hint: You can become one
```

Previously, the message used "one of these," which aligned with the expected output format.

**Expected Behavior:**
The suggestion message should consistently use "any of these" to accurately reflect the possibility of multiple suggestions. This ensures that the output remains clear and aligns with scenarios where multiple terms are queried simultaneously.

**Actual Behavior:**
The message now uses "any of these," which differs from the original phrasing. This discrepancy can cause confusion and may lead to failures in processes or tests that expect the original "one of these" wording.

**Code Example:**
```python
out = StringIO()
fail_with_short_help(
    msg='Failed badly',
    known=['mother', 'mutter', 'father', 'son'],
    provided='muther',
    hint='You can become one',
    exit_code=0,
    what='parent',
    out=out
)
print(out.getvalue())
```

**Output:**
```
error: Failed badly
datalad: Unknown parent 'muther'.  See 'datalad --help'.

Did you mean any of these?
        mutter
        mother
        father
Hint: You can become one
```
[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/cmdline/main.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
""""""
from datalad.ui.utils import get_console_width

__docformat__ = 'restructuredtext'

import logging
lgr = logging.getLogger('datalad.cmdline')

lgr.log(5, "Importing cmdline.main")

import argparse
from collections import defaultdict
import sys
import textwrap
import os

from six import text_type

import datalad

from datalad.cmdline import helpers
from datalad.support.exceptions import InsufficientArgumentsError
from datalad.support.exceptions import IncompleteResultsError
from datalad.support.exceptions import CommandError
from .helpers import strip_arg_from_argv
from ..utils import setup_exceptionhook, chpwd
from ..utils import assure_unicode
from ..utils import on_msys_tainted_paths
from ..dochelpers import exc_str


def _license_info():
    return """\
Copyright (c) 2013-2018 DataLad developers

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""


class ArgumentParserDisableAbbrev(argparse.ArgumentParser):
    # Don't accept abbreviations for long options. With py3.5 and above, we
    # could just use allow_abbrev=False.
    #
    # Modified from the solution posted at
    # https://bugs.python.org/issue14910#msg204678
    def _get_option_tuples(self, option_string):
        chars = self.prefix_chars
        if option_string[0] in chars and option_string[1] in chars:
            # option_string is a long flag. Disable abbreviation.
            return []
        return super(ArgumentParserDisableAbbrev, self)._get_option_tuples(
            option_string)


# TODO:  OPT look into making setup_parser smarter to become faster
# Now it seems to take up to 200ms to do all the parser setup
# even though it might not be necessary to know about all the commands etc.
# I wondered if it could somehow decide on what commands to worry about etc
# by going through sys.args first
def setup_parser(
        cmdlineargs,
        formatter_class=argparse.RawDescriptionHelpFormatter,
        return_subparsers=False):
    lgr.log(5, "Starting to setup_parser")
    # delay since it can be a heavy import
    from ..interface.base import dedent_docstring, get_interface_groups, \
        get_cmdline_command_name, alter_interface_docs_for_cmdline, \
        load_interface, get_cmd_doc
    # setup cmdline args parser
    parts = {}
    # main parser
    parser = ArgumentParserDisableAbbrev(
        fromfile_prefix_chars=None,
        # usage="%(prog)s ...",
        description=dedent_docstring("""\
            Comprehensive data management solution

            DataLad provides a unified data distribution system built on the Git
            and Git-annex. DataLad command line tools allow to manipulate (obtain,
            create, update, publish, etc.) datasets and provide a comprehensive
            toolbox for joint management of data and code. Compared to Git/annex
            it primarly extends their functionality to transparently and
            simultaneously work with multiple inter-related repositories."""),
        epilog='"Be happy!"',
        formatter_class=formatter_class,
        add_help=False)
    # common options
    helpers.parser_add_common_opt(parser, 'log_level')
    helpers.parser_add_common_opt(parser, 'pbs_runner')
    helpers.parser_add_common_opt(parser, 'change_path')
    helpers.parser_add_common_opt(
        parser,
        'version',
        version='datalad %s\n' % datalad.__version__)
    if __debug__:
        parser.add_argument(
            '--dbg', action='store_true', dest='common_debug',
            help="enter Python debugger when uncaught exception happens")
        parser.add_argument(
            '--idbg', action='store_true', dest='common_idebug',
            help="enter IPython debugger when uncaught exception happens")
    parser.add_argument(
        '-c', action='append', dest='cfg_overrides', metavar='KEY=VALUE',
        help="""configuration variable setting. Overrides any configuration
        read from a file, but is potentially overridden itself by configuration
        variables in the process environment.""")
    parser.add_argument(
        '-f', '--output-format', dest='common_output_format',
        default='default',
        type=assure_unicode,
        metavar="{default,json,json_pp,tailored,'<template>'}",
        help="""select format for returned command results. 'default' give one line
        per result reporting action, status, path and an optional message;
        'json' renders a JSON object with all properties for each result (one per
        line); 'json_pp' pretty-prints JSON spanning multiple lines; 'tailored'
        enables a command-specific rendering style that is typically
        tailored to human consumption (no result output otherwise),
        '<template>' reports any value(s) of any result properties in any format
        indicated by the template (e.g. '{path}'; compare with JSON
        output for all key-value choices). The template syntax follows the Python
        "format() language". It is possible to report individual
        dictionary values, e.g. '{metadata[name]}'. If a 2nd-level key contains
        a colon, e.g. 'music:Genre', ':' must be substituted by '#' in the template,
        like so: '{metadata[music#Genre]}'.""")
    parser.add_argument(
        '--report-status', dest='common_report_status',
        choices=['success', 'failure', 'ok', 'notneeded', 'impossible', 'error'],
        help="""constrain command result report to records matching the given
        status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands
        for 'impossible' OR 'error'.""")
    parser.add_argument(
        '--report-type', dest='common_report_type',
        choices=['dataset', 'file'],
        action='append',
        help="""constrain command result report to records matching the given
        type. Can be given more than once to match multiple types.""")
    parser.add_argument(
        '--on-failure', dest='common_on_failure',
        choices=['ignore', 'continue', 'stop'],
        # no default: better be configure per-command
        help="""when an operation fails: 'ignore' and continue with remaining
        operations, the error is logged but does not lead to a non-zero exit code
        of the command; 'continue' works like 'ignore', but an error causes a
        non-zero exit code; 'stop' halts on first failure and yields non-zero exit
        code. A failure is any result with status 'impossible' or 'error'.""")
    parser.add_argument(
        '--proc-pre', dest='common_proc_pre',
        nargs='+',
        action='append',
        metavar=('<PROCEDURE NAME>', 'ARGS'),
        help="""Dataset procedure to run before the main command (see run-procedure
        command for details). This option can be given more than once to run
        multiple procedures in the order in which they were given.
        It is important to specify the target dataset via the --dataset argument
        of the main command."""),
    parser.add_argument(
        '--proc-post', dest='common_proc_post',
        nargs='+',
        action='append',
        metavar=('<PROCEDURE NAME>', 'ARGS'),
        help="""Like --proc-pre, but procedures are executed after the main command
        has finished."""),
    parser.add_argument(
        '--cmd', dest='_', action='store_true',
        help="""syntactical helper that can be used to end the list of global
        command line options before the subcommand label. Options like
        --proc-pre can take an arbitrary number of arguments and may require
        to be followed by a single --cmd in order to enable identification
        of the subcommand.""")

    # yoh: atm we only dump to console.  Might adopt the same separation later on
    #      and for consistency will call it --verbose-level as well for now
    # log-level is set via common_opts ATM
    # parser.add_argument('--log-level',
    #                     choices=('critical', 'error', 'warning', 'info', 'debug'),
    #                     dest='common_log_level',
    #                     help="""level of verbosity in log files. By default
    #                          everything, including debug messages is logged.""")
    #parser.add_argument('-l', '--verbose-level',
    #                    choices=('critical', 'error', 'warning', 'info', 'debug'),
    #                    dest='common_verbose_level',
    #                    help="""level of verbosity of console output. By default
    #                         only warnings and errors are printed.""")

    # Before doing anything additional and possibly expensive see may be that
    # we have got the command already
    need_single_subparser = False if return_subparsers else None
    fail_handler = (lambda *a, **kw: True) \
        if return_subparsers else fail_with_short_help
    try:
        parsed_args, unparsed_args = parser._parse_known_args(
            cmdlineargs[1:], argparse.Namespace())
        if not unparsed_args:
            fail_handler(parser, msg="too few arguments", exit_code=2)
        lgr.debug("Command line args 1st pass. Parsed: %s Unparsed: %s",
                  parsed_args, unparsed_args)
    except Exception as exc:
        lgr.debug("Early parsing failed with %s", exc_str(exc))
        need_single_subparser = False
        unparsed_args = cmdlineargs[1:]  # referenced before assignment otherwise

    interface_groups = get_interface_groups(include_plugins=True)

    # First unparsed could be either unknown option to top level "datalad"
    # or a command. Among unknown could be --help/--help-np which would
    # need to be dealt with
    unparsed_arg = unparsed_args[0] if unparsed_args else None
    if need_single_subparser is not None \
            or unparsed_arg in ('--help', '--help-np', '-h'):
        need_single_subparser = False
        add_entrypoints_to_interface_groups(interface_groups)
    elif unparsed_arg.startswith('-'):  # unknown option
        fail_with_short_help(parser,
                             msg="unrecognized argument %s" % unparsed_arg,
                             exit_code=2)
        # if we could get a list of options known to parser,
        # we could suggest them
        # known=get_all_options(parser), provided=unparsed_arg)
    else:  # the command to handle
        known_commands = get_commands_from_groups(interface_groups)
        if unparsed_arg not in known_commands:
            # need to load all the extensions and try again
            add_entrypoints_to_interface_groups(interface_groups)
            known_commands = get_commands_from_groups(interface_groups)

        if unparsed_arg not in known_commands:
            # check if might be coming from known extensions
            from ..interface import _known_extension_commands
            extension_commands = {
                c: e
                for e, commands in _known_extension_commands.items()
                for c in commands
            }
            hint = None
            if unparsed_arg in extension_commands:
                hint = "Command %s is provided by (not installed) extension %s." \
                      % (unparsed_arg, extension_commands[unparsed_arg])
            fail_with_short_help(
                parser,
                hint=hint,
                provided=unparsed_arg,
                known=list(known_commands.keys()) + list(extension_commands.keys())
            )
        if need_single_subparser is None:
            need_single_subparser = unparsed_arg

    # --help specification was delayed since it causes immediate printout of
    # --help output before we setup --help for each command
    helpers.parser_add_common_opt(parser, 'help')

    grp_short_descriptions = defaultdict(list)
    # create subparser, use module suffix as cmd name
    subparsers = parser.add_subparsers()
    for group_name, _, _interfaces \
            in sorted(interface_groups, key=lambda x: x[1]):
        for _intfspec in _interfaces:
            cmd_name = get_cmdline_command_name(_intfspec)
            if need_single_subparser and cmd_name != need_single_subparser:
                continue
            _intf = load_interface(_intfspec)
            if _intf is None:
                # TODO(yoh):  add doc why we could skip this one... makes this
                # loop harder to extract into a dedicated function
                continue
            # deal with optional parser args
            if hasattr(_intf, 'parser_args'):
                parser_args = _intf.parser_args
            else:
                parser_args = dict(formatter_class=formatter_class)
                # use class description, if no explicit description is available
                intf_doc = get_cmd_doc(_intf)
                parser_args['description'] = alter_interface_docs_for_cmdline(
                    intf_doc)
            subparser = subparsers.add_parser(cmd_name, add_help=False, **parser_args)
            # our own custom help for all commands
            helpers.parser_add_common_opt(subparser, 'help')
            # let module configure the parser
            _intf.setup_parser(subparser)
            # logger for command

            # configure 'run' function for this command
            plumbing_args = dict(
                func=_intf.call_from_parser,
                logger=logging.getLogger(_intf.__module__),
                subparser=subparser)
            if hasattr(_intf, 'result_renderer_cmdline'):
                plumbing_args['result_renderer'] = _intf.result_renderer_cmdline
            subparser.set_defaults(**plumbing_args)
            parts[cmd_name] = subparser
            # store short description for later
            sdescr = getattr(_intf, 'short_description',
                             parser_args['description'].split('\n')[0])
            grp_short_descriptions[group_name].append((cmd_name, sdescr))

    # create command summary
    if '--help' in cmdlineargs or '--help-np' in cmdlineargs:
        parser.description = get_description_with_cmd_summary(
            grp_short_descriptions,
            interface_groups,
            parser.description)

    parts['datalad'] = parser
    lgr.log(5, "Finished setup_parser")
    if return_subparsers:
        return parts
    else:
        return parser


def fail_with_short_help(parser=None,
                         msg=None,
                         known=None, provided=None,
                         hint=None,
                         exit_code=1,
                         what="command",
                         out=None):
    """Generic helper to fail
    with short help possibly hinting on what was intended if `known`
    were provided
    """
    out = out or sys.stderr
    if msg:
        out.write("error: %s\n" % msg)
    if not known:
        if parser:
            # just to appear in print_usage also consistent with --help output
            parser.add_argument("command [command-opts]")
            parser.print_usage(file=out)
    else:
        out.write(
            "datalad: Unknown %s %r.  See 'datalad --help'.\n\n"
            % (what, provided,))
        if provided not in known:
            import difflib
            suggestions = difflib.get_close_matches(provided, known)
            if suggestions:
                out.write(
                    "Did you mean one of these?\n        %s\n"
                    % "\n        ".join(suggestions))
        # Too noisy
        # sys.stderr.write(" All known %ss: %s\n"
        #                   % (what, ", ".join(sorted(known))))
    if hint:
        out.write("Hint: %s\n" % hint)
    raise SystemExit(exit_code)


def get_description_with_cmd_summary(grp_short_descriptions, interface_groups,
                                     parser_description):
    from ..interface.base import dedent_docstring
    from ..interface.base import get_cmd_summaries
    lgr.debug("Generating detailed description for the parser")

    console_width = get_console_width()
    cmd_summary = get_cmd_summaries(grp_short_descriptions, interface_groups,
                                    width=console_width)
    # we need one last formal section to not have the trailed be
    # confused with the last command group
    cmd_summary.append('\n*General information*\n')
    detailed_description = '%s\n%s\n\n%s' \
                           % (parser_description,
                              '\n'.join(cmd_summary),
                              textwrap.fill(dedent_docstring("""\
    Detailed usage information for individual commands is
    available via command-specific --help, i.e.:
    datalad <command> --help"""),
                                            console_width - 5,
                                            initial_indent='',
                                            subsequent_indent=''))
    return detailed_description


def get_commands_from_groups(groups):
    """Get a dictionary of command: interface_spec"""
    from ..interface.base import get_cmdline_command_name
    return {
        get_cmdline_command_name(_intfspec): _intfspec
        for _, _, _interfaces in groups
        for _intfspec in _interfaces
    }


def add_entrypoints_to_interface_groups(interface_groups):
    lgr.debug("Loading entrypoints")
    from pkg_resources import iter_entry_points  # delay expensive import
    for ep in iter_entry_points('datalad.extensions'):
        lgr.debug(
            'Loading entrypoint %s from datalad.extensions for docs building',
            ep.name)
        try:
            spec = ep.load()
            interface_groups.append((ep.name, spec[0], spec[1]))
            lgr.debug('Loaded entrypoint %s', ep.name)
        except Exception as e:
            lgr.warning('Failed to load entrypoint %s: %s', ep.name, exc_str(e))
            continue

def _fix_datalad_ri(s):
    """Fixup argument if it was a DataLadRI and had leading / removed

    See gh-2643
    """
    if s.startswith('//') and (len(s) == 2 or (len(s) > 2 and s[2] != '/')):
        lgr.info(
            "Changing %s back to /%s as it was probably changed by MINGW/MSYS, "
            "see http://www.mingw.org/wiki/Posix_path_conversion", s, s)
        return "/" + s
    return s


def main(args=None):
    lgr.log(5, "Starting main(%r)", args)
    args = args or sys.argv
    if on_msys_tainted_paths:
        # Possibly present DataLadRIs were stripped of a leading /
        args = [_fix_datalad_ri(s) for s in args]
    # PYTHON_ARGCOMPLETE_OK
    parser = setup_parser(args)
    try:
        import argcomplete
        argcomplete.autocomplete(parser)
    except ImportError:
        pass

    # parse cmd args
    lgr.debug("Parsing known args among %s", repr(args))
    cmdlineargs, unparsed_args = parser.parse_known_args(args[1:])
    has_func = hasattr(cmdlineargs, 'func') and cmdlineargs.func is not None
    if unparsed_args:
        if has_func and cmdlineargs.func.__self__.__name__ != 'Export':
            lgr.error('unknown argument{}: {}'.format(
                's' if len(unparsed_args) > 1 else '',
                unparsed_args if len(unparsed_args) > 1 else unparsed_args[0]))
            cmdlineargs.subparser.print_usage()
            sys.exit(1)
        else:
            # store all unparsed arguments
            cmdlineargs.datalad_unparsed_args = unparsed_args

    # to possibly be passed into PBS scheduled call
    args_ = args or sys.argv

    if cmdlineargs.cfg_overrides is not None:
        overrides = dict([
            (o.split('=')[0], '='.join(o.split('=')[1:]))
            for o in cmdlineargs.cfg_overrides])
        datalad.cfg.overrides.update(overrides)

    # enable overrides
    datalad.cfg.reload(force=True)

    if cmdlineargs.change_path is not None:
        from .common_args import change_path as change_path_opt
        for path in cmdlineargs.change_path:
            chpwd(path)
            args_ = strip_arg_from_argv(args_, path, change_path_opt[1])

    ret = None
    if cmdlineargs.pbs_runner:
        from .helpers import run_via_pbs
        from .common_args import pbs_runner as pbs_runner_opt
        args_ = strip_arg_from_argv(args_, cmdlineargs.pbs_runner, pbs_runner_opt[1])
        # run the function associated with the selected command
        run_via_pbs(args_, cmdlineargs.pbs_runner)
    elif has_func:
        if cmdlineargs.common_debug or cmdlineargs.common_idebug:
            # so we could see/stop clearly at the point of failure
            setup_exceptionhook(ipython=cmdlineargs.common_idebug)
            from datalad.interface.base import Interface
            Interface._interrupted_exit_code = None
            ret = cmdlineargs.func(cmdlineargs)
        else:
            # otherwise - guard and only log the summary. Postmortem is not
            # as convenient if being caught in this ultimate except
            try:
                ret = cmdlineargs.func(cmdlineargs)
            except InsufficientArgumentsError as exc:
                # if the func reports inappropriate usage, give help output
                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))
                cmdlineargs.subparser.print_usage(sys.stderr)
                sys.exit(2)
            except IncompleteResultsError as exc:
                # rendering for almost all commands now happens 'online'
                # hence we are no longer attempting to render the actual
                # results in an IncompleteResultsError, ubt rather trust that
                # this happened before

                # in general we do not want to see the error again, but
                # present in debug output
                lgr.debug('could not perform all requested actions: %s',
                          exc_str(exc))
                sys.exit(1)
            except CommandError as exc:
                # behave as if the command ran directly, importantly pass
                # exit code as is
                exc_msg = str(exc)
                if exc_msg:
                    msg = exc_msg.encode() if isinstance(exc_msg, text_type) else exc_msg
                    os.write(2, msg + b"\n")
                if exc.stdout:
                    os.write(1, exc.stdout.encode() if isinstance(exc.stdout, text_type) else exc.stdout)
                if exc.stderr:
                    os.write(2, exc.stderr.encode() if isinstance(exc.stderr, text_type) else exc.stderr)
                # We must not exit with 0 code if any exception got here but
                # had no code defined
                sys.exit(exc.code if exc.code is not None else 1)
            except Exception as exc:
                lgr.error('%s (%s)' % (exc_str(exc), exc.__class__.__name__))
                sys.exit(1)
    else:
        # just let argparser spit out its error, since there is smth wrong
        parser.parse_args(args)
        # if that one didn't puke -- we should
        parser.print_usage()
        lgr.error("Please specify the command")
        sys.exit(2)

    try:
        if hasattr(cmdlineargs, 'result_renderer'):
            cmdlineargs.result_renderer(ret, cmdlineargs)
    except Exception as exc:
        lgr.error("Failed to render results due to %s", exc_str(exc))
        sys.exit(1)

lgr.log(5, "Done importing cmdline.main")


### datalad/utils.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##

import collections
import hashlib
import re
import six.moves.builtins as __builtin__
import time

import logging
import shutil
import os
import sys
import tempfile
import platform
import gc
import glob
import gzip
import string
import wrapt

from copy import copy as shallow_copy
from contextlib import contextmanager
from functools import wraps
from time import sleep
from inspect import getargspec
from itertools import tee

import os.path as op
from os.path import sep as dirsep
from os.path import commonprefix
from os.path import curdir, basename, exists, realpath, islink, join as opj
from os.path import isabs, normpath, expandvars, expanduser, abspath, sep
from os.path import isdir
from os.path import relpath
from os.path import stat
from os.path import dirname
from os.path import split as psplit
import posixpath


from six import PY2, text_type, binary_type, string_types

# from datalad.dochelpers import get_docstring_split
from datalad.consts import TIMESTAMP_FMT


if PY2:
    unicode_srctypes = string_types
else:
    unicode_srctypes = string_types + (bytes,)


lgr = logging.getLogger("datalad.utils")

lgr.log(5, "Importing datalad.utils")
#
# Some useful variables
#
platform_system = platform.system().lower()
on_windows = platform_system == 'windows'
on_osx = platform_system == 'darwin'
on_linux = platform_system == 'linux'
on_msys_tainted_paths = on_windows \
                        and 'MSYS_NO_PATHCONV' not in os.environ \
                        and os.environ.get('MSYSTEM', '')[:4] in ('MSYS', 'MING')
try:
    linux_distribution_name, linux_distribution_release \
        = platform.linux_distribution()[:2]
    on_debian_wheezy = on_linux \
                       and linux_distribution_name == 'debian' \
                       and linux_distribution_release.startswith('7.')
except:  # pragma: no cover
    # MIH: IndexError?
    on_debian_wheezy = False
    linux_distribution_name = linux_distribution_release = None

# Maximal length of cmdline string
# Query the system and use hardcoded "knowledge" if None
# probably   getconf ARG_MAX   might not be available
# The last one would be the most conservative/Windows
CMD_MAX_ARG_HARDCODED = 2097152 if on_linux else 262144 if on_osx else 32767
try:
    CMD_MAX_ARG = os.sysconf('SC_ARG_MAX')
    assert CMD_MAX_ARG > 0
except Exception as exc:
    # ATM (20181005) SC_ARG_MAX available only on POSIX systems
    # so exception would be thrown e.g. on Windows, or
    # somehow during Debian build for nd14.04 it is coming up with -1:
    # https://github.com/datalad/datalad/issues/3015
    CMD_MAX_ARG = CMD_MAX_ARG_HARDCODED
    lgr.debug(
        "Failed to query or got useless SC_ARG_MAX sysconf, "
        "will use hardcoded value: %s", exc)
# Even with all careful computations we do, due to necessity to account for
# environment and what not, we still could not figure out "exact" way to
# estimate it, but it was shown that 300k safety margin on linux was sufficient.
# https://github.com/datalad/datalad/pull/2977#issuecomment-436264710
# 300k is ~15%, so to be safe, and for paranoid us we will just use up to 50%
# of the length for "safety margin".  We might probably still blow due to
# env vars, unicode, etc...  so any hard limit imho is not a proper solution
CMD_MAX_ARG = int(0.5 * CMD_MAX_ARG)
lgr.debug(
    "Maximal length of cmdline string (adjusted for safety margin): %d",
    CMD_MAX_ARG)

#
# Little helpers
#


def get_func_kwargs_doc(func):
    """ Provides args for a function

    Parameters
    ----------
    func: str
      name of the function from which args are being requested

    Returns
    -------
    list
      of the args that a function takes in
    """
    return getargspec(func)[0]

    # TODO: format error message with descriptions of args
    # return [repr(dict(get_docstring_split(func)[1]).get(x)) for x in getargspec(func)[0]]


def any_re_search(regexes, value):
    """Return if any of regexes (list or str) searches succesfully for value"""
    for regex in assure_tuple_or_list(regexes):
        if re.search(regex, value):
            return True
    return False


def not_supported_on_windows(msg=None):
    """A little helper to be invoked to consistently fail whenever functionality is
    not supported (yet) on Windows
    """
    if on_windows:
        raise NotImplementedError("This functionality is not yet implemented for Windows OS"
                                  + (": %s" % msg if msg else ""))


def shortened_repr(value, l=30):
    try:
        if hasattr(value, '__repr__') and (value.__repr__ is not object.__repr__):
            value_repr = repr(value)
            if not value_repr.startswith('<') and len(value_repr) > l:
                value_repr = "<<%s...>>" % (value_repr[:l - 8])
            elif value_repr.startswith('<') and value_repr.endswith('>') and ' object at 0x':
                raise ValueError("I hate those useless long reprs")
        else:
            raise ValueError("gimme class")
    except Exception as e:
        value_repr = "<%s>" % value.__class__.__name__.split('.')[-1]
    return value_repr


def __auto_repr__(obj):
    attr_names = tuple()
    if hasattr(obj, '__dict__'):
        attr_names += tuple(obj.__dict__.keys())
    if hasattr(obj, '__slots__'):
        attr_names += tuple(obj.__slots__)

    items = []
    for attr in sorted(set(attr_names)):
        if attr.startswith('_'):
            continue
        value = getattr(obj, attr)
        # TODO:  should we add this feature to minimize some talktative reprs
        # such as of URL?
        #if value is None:
        #    continue
        items.append("%s=%s" % (attr, shortened_repr(value)))

    return "%s(%s)" % (obj.__class__.__name__, ', '.join(items))


def auto_repr(cls):
    """Decorator for a class to assign it an automagic quick and dirty __repr__

    It uses public class attributes to prepare repr of a class

    Original idea: http://stackoverflow.com/a/27799004/1265472
    """

    cls.__repr__ = __auto_repr__
    return cls


def is_interactive():
    """Return True if all in/outs are tty"""
    # TODO: check on windows if hasattr check would work correctly and add value:
    #
    return sys.stdin.isatty() and sys.stdout.isatty() and sys.stderr.isatty()


def get_ipython_shell():
    """Detect if running within IPython and returns its `ip` (shell) object

    Returns None if not under ipython (no `get_ipython` function)
    """
    try:
        return get_ipython()
    except NameError:
        return None


def md5sum(filename):
    with open(filename, 'rb') as f:
        return hashlib.md5(f.read()).hexdigest()


def sorted_files(dout):
    """Return a (sorted) list of files under dout
    """
    return sorted(sum([[opj(r, f)[len(dout) + 1:] for f in files]
                       for r, d, files in os.walk(dout)
                       if not '.git' in r], []))

_VCS_REGEX = '%s\.(?:git|gitattributes|svn|bzr|hg)(?:%s|$)' % (dirsep, dirsep)
_DATALAD_REGEX = '%s\.(?:datalad)(?:%s|$)' % (dirsep, dirsep)


def find_files(regex, topdir=curdir, exclude=None, exclude_vcs=True, exclude_datalad=False, dirs=False):
    """Generator to find files matching regex

    Parameters
    ----------
    regex: basestring
    exclude: basestring, optional
      Matches to exclude
    exclude_vcs:
      If True, excludes commonly known VCS subdirectories.  If string, used
      as regex to exclude those files (regex: `%r`)
    exclude_datalad:
      If True, excludes files known to be datalad meta-data files (e.g. under
      .datalad/ subdirectory) (regex: `%r`)
    topdir: basestring, optional
      Directory where to search
    dirs: bool, optional
      Whether to match directories as well as files
    """

    for dirpath, dirnames, filenames in os.walk(topdir):
        names = (dirnames + filenames) if dirs else filenames
        # TODO: might want to uniformize on windows to use '/'
        paths = (opj(dirpath, name) for name in names)
        for path in filter(re.compile(regex).search, paths):
            path = path.rstrip(dirsep)
            if exclude and re.search(exclude, path):
                continue
            if exclude_vcs and re.search(_VCS_REGEX, path):
                continue
            if exclude_datalad and re.search(_DATALAD_REGEX, path):
                continue
            yield path
find_files.__doc__ %= (_VCS_REGEX, _DATALAD_REGEX)


def expandpath(path, force_absolute=True):
    """Expand all variables and user handles in a path.

    By default return an absolute path
    """
    path = expandvars(expanduser(path))
    if force_absolute:
        path = abspath(path)
    return path


def posix_relpath(path, start=None):
    """Behave like os.path.relpath, but always return POSIX paths...

    on any platform."""
    # join POSIX style
    return posixpath.join(
        # split and relpath native style
        # python2.7 ntpath implementation of relpath cannot handle start=None
        *psplit(
            relpath(path, start=start if start is not None else '')))


def is_explicit_path(path):
    """Return whether a path explicitly points to a location

    Any absolute path, or relative path starting with either '../' or
    './' is assumed to indicate a location on the filesystem. Any other
    path format is not considered explicit."""
    path = expandpath(path, force_absolute=False)
    return isabs(path) \
        or path.startswith(os.curdir + os.sep) \
        or path.startswith(os.pardir + os.sep)


def rotree(path, ro=True, chmod_files=True):
    """To make tree read-only or writable

    Parameters
    ----------
    path : string
      Path to the tree/directory to chmod
    ro : bool, optional
      Whether to make it R/O (default) or RW
    chmod_files : bool, optional
      Whether to operate also on files (not just directories)
    """
    if ro:
        chmod = lambda f: os.chmod(f, os.stat(f).st_mode & ~stat.S_IWRITE)
    else:
        chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)

    for root, dirs, files in os.walk(path, followlinks=False):
        if chmod_files:
            for f in files:
                fullf = opj(root, f)
                # might be the "broken" symlink which would fail to stat etc
                if exists(fullf):
                    chmod(fullf)
        chmod(root)


def rmtree(path, chmod_files='auto', children_only=False, *args, **kwargs):
    """To remove git-annex .git it is needed to make all files and directories writable again first

    Parameters
    ----------
    chmod_files : string or bool, optional
       Whether to make files writable also before removal.  Usually it is just
       a matter of directories to have write permissions.
       If 'auto' it would chmod files on windows by default
    children_only : bool, optional
       If set, all files and subdirectories would be removed while the path
       itself (must be a directory) would be preserved
    `*args` :
    `**kwargs` :
       Passed into shutil.rmtree call
    """
    # Give W permissions back only to directories, no need to bother with files
    if chmod_files == 'auto':
        chmod_files = on_windows
    # TODO:  yoh thinks that if we could quickly check our Flyweight for
    #        repos if any of them is under the path, and could call .precommit
    #        on those to possibly stop batched processes etc, we did not have
    #        to do it on case by case
    # Check for open files
    assert_no_open_files(path)

    if children_only:
        if not os.path.isdir(path):
            raise ValueError("Can remove children only of directories")
        for p in os.listdir(path):
            rmtree(op.join(path, p))
        return
    if not (os.path.islink(path) or not os.path.isdir(path)):
        rotree(path, ro=False, chmod_files=chmod_files)
        _rmtree(path, *args, **kwargs)
    else:
        # just remove the symlink
        unlink(path)


def rmdir(path, *args, **kwargs):
    """os.rmdir with our optional checking for open files"""
    assert_no_open_files(path)
    os.rmdir(path)


def get_open_files(path, log_open=False):
    """Get open files under a path

    Parameters
    ----------
    path : str
      File or directory to check for open files under
    log_open : bool or int
      If set - logger level to use

    Returns
    -------
    dict
      path : pid

    """
    # Original idea: https://stackoverflow.com/a/11115521/1265472
    import psutil
    files = {}
    # since the ones returned by psutil would not be aware of symlinks in the
    # path we should also get realpath for path
    path = realpath(path)
    for proc in psutil.process_iter():
        try:
            open_paths = [p.path for p in proc.open_files()] + [proc.cwd()]
            for p in open_paths:
                # note: could be done more efficiently so we do not
                # renormalize path over and over again etc
                if path_startswith(p, path):
                    files[p] = proc.pid
        # Catch a race condition where a process ends
        # before we can examine its files
        except psutil.NoSuchProcess:
            pass
        except psutil.AccessDenied:
            pass

    if files and log_open:
        lgr.log(log_open, "Open files under %s: %s", path, files)
    return files


_assert_no_open_files_cfg = os.environ.get('DATALAD_ASSERT_NO_OPEN_FILES')
if _assert_no_open_files_cfg:
    def assert_no_open_files(path):
        files = get_open_files(path, log_open=40)
        if _assert_no_open_files_cfg == 'assert':
            assert not files
        elif files:
            if _assert_no_open_files_cfg == 'pdb':
                import pdb
                pdb.set_trace()
            elif _assert_no_open_files_cfg == 'epdb':
                import epdb
                epdb.serve()
            pass
        # otherwise we would just issue that error message in the log
else:
    def assert_no_open_files(*args, **kwargs):
        pass


def rmtemp(f, *args, **kwargs):
    """Wrapper to centralize removing of temp files so we could keep them around

    It will not remove the temporary file/directory if DATALAD_TESTS_TEMP_KEEP
    environment variable is defined
    """
    if not os.environ.get('DATALAD_TESTS_TEMP_KEEP'):
        if not os.path.lexists(f):
            lgr.debug("Path %s does not exist, so can't be removed" % f)
            return
        lgr.log(5, "Removing temp file: %s" % f)
        # Can also be a directory
        if os.path.isdir(f):
            rmtree(f, *args, **kwargs)
        else:
            unlink(f)
    else:
        lgr.info("Keeping temp file: %s" % f)


def file_basename(name, return_ext=False):
    """
    Strips up to 2 extensions of length up to 4 characters and starting with alpha
    not a digit, so we could get rid of .tar.gz etc
    """
    bname = basename(name)
    fbname = re.sub('(\.[a-zA-Z_]\S{1,4}){0,2}$', '', bname)
    if return_ext:
        return fbname, bname[len(fbname) + 1:]
    else:
        return fbname


def escape_filename(filename):
    """Surround filename in "" and escape " in the filename
    """
    filename = filename.replace('"', r'\"').replace('`', r'\`')
    filename = '"%s"' % filename
    return filename


def encode_filename(filename):
    """Encode unicode filename
    """
    if isinstance(filename, text_type):
        return filename.encode(sys.getfilesystemencoding())
    else:
        return filename


def decode_input(s):
    """Given input string/bytes, decode according to stdin codepage (or UTF-8)
    if not defined

    If fails -- issue warning and decode allowing for errors
    being replaced
    """
    if isinstance(s, text_type):
        return s
    else:
        encoding = sys.stdin.encoding or 'UTF-8'
        try:
            return s.decode(encoding)
        except UnicodeDecodeError as exc:
            lgr.warning(
                "Failed to decode input string using %s encoding. "
                "Decoding allowing for errors", encoding)
            return s.decode(encoding, errors='replace')


if on_windows:
    def lmtime(filepath, mtime):
        """Set mtime for files.  On Windows a merely adapter to os.utime
        """
        os.utime(filepath, (time.time(), mtime))
else:
    def lmtime(filepath, mtime):
        """Set mtime for files, while not de-referencing symlinks.

        To overcome absence of os.lutime

        Works only on linux and OSX ATM
        """
        from .cmd import Runner
        # convert mtime to format touch understands [[CC]YY]MMDDhhmm[.SS]
        smtime = time.strftime("%Y%m%d%H%M.%S", time.localtime(mtime))
        lgr.log(3, "Setting mtime for %s to %s == %s", filepath, mtime, smtime)
        Runner().run(['touch', '-h', '-t', '%s' % smtime, filepath])
        rfilepath = realpath(filepath)
        if islink(filepath) and exists(rfilepath):
            # trust noone - adjust also of the target file
            # since it seemed like downloading under OSX (was it using curl?)
            # didn't bother with timestamps
            lgr.log(3, "File is a symlink to %s Setting mtime for it to %s",
                    rfilepath, mtime)
            os.utime(rfilepath, (time.time(), mtime))
        # doesn't work on OSX
        # Runner().run(['touch', '-h', '-d', '@%s' % mtime, filepath])


def assure_tuple_or_list(obj):
    """Given an object, wrap into a tuple if not list or tuple
    """
    if isinstance(obj, (list, tuple)):
        return obj
    return (obj,)


def assure_iter(s, cls, copy=False, iterate=True):
    """Given not a list, would place it into a list. If None - empty list is returned

    Parameters
    ----------
    s: list or anything
    cls: class
      Which iterable class to assure
    copy: bool, optional
      If correct iterable is passed, it would generate its shallow copy
    iterate: bool, optional
      If it is not a list, but something iterable (but not a text_type)
      iterate over it.
    """

    if isinstance(s, cls):
        return s if not copy else shallow_copy(s)
    elif isinstance(s, text_type):
        return cls((s,))
    elif iterate and hasattr(s, '__iter__'):
        return cls(s)
    elif s is None:
        return cls()
    else:
        return cls((s,))


def assure_list(s, copy=False, iterate=True):
    """Given not a list, would place it into a list. If None - empty list is returned

    Parameters
    ----------
    s: list or anything
    copy: bool, optional
      If list is passed, it would generate a shallow copy of the list
    iterate: bool, optional
      If it is not a list, but something iterable (but not a text_type)
      iterate over it.
    """
    return assure_iter(s, list, copy=copy, iterate=iterate)


def assure_list_from_str(s, sep='\n'):
    """Given a multiline string convert it to a list of return None if empty

    Parameters
    ----------
    s: str or list
    """

    if not s:
        return None

    if isinstance(s, list):
        return s
    return s.split(sep)


def assure_dict_from_str(s, **kwargs):
    """Given a multiline string with key=value items convert it to a dictionary

    Parameters
    ----------
    s: str or dict

    Returns None if input s is empty
    """

    if not s:
        return None

    if isinstance(s, dict):
        return s

    out = {}
    for value_str in assure_list_from_str(s, **kwargs):
        if '=' not in value_str:
            raise ValueError("{} is not in key=value format".format(repr(value_str)))
        k, v = value_str.split('=', 1)
        if k in out:
            err = "key {} was already defined in {}, but new value {} was provided".format(k, out, v)
            raise ValueError(err)
        out[k] = v
    return out


def assure_bytes(s, encoding='utf-8'):
    """Convert/encode unicode to str (PY2) or bytes (PY3) if of 'text_type'

    Parameters
    ----------
    encoding: str, optional
      Encoding to use.  "utf-8" is the default
    """
    if not isinstance(s, text_type):
        return s
    return s.encode(encoding)


def assure_unicode(s, encoding=None, confidence=None):
    """Convert/decode to unicode (PY2) or str (PY3) if of 'binary_type'

    Parameters
    ----------
    encoding: str, optional
      Encoding to use.  If None, "utf-8" is tried, and then if not a valid
      UTF-8, encoding will be guessed
    confidence: float, optional
      A value between 0 and 1, so if guessing of encoding is of lower than
      specified confidence, ValueError is raised
    """
    if not isinstance(s, binary_type):
        return s
    if encoding is None:
        # Figure out encoding, defaulting to 'utf-8' which is our common
        # target in contemporary digital society
        try:
            return s.decode('utf-8')
        except UnicodeDecodeError as exc:
            from .dochelpers import exc_str
            lgr.debug("Failed to decode a string as utf-8: %s", exc_str(exc))
        # And now we could try to guess
        from chardet import detect
        enc = detect(s)
        denc = enc.get('encoding', None)
        if denc:
            denc_confidence = enc.get('confidence', 0)
            if confidence is not None and  denc_confidence < confidence:
                raise ValueError(
                    "Failed to auto-detect encoding with high enough "
                    "confidence. Highest confidence was %s for %s"
                    % (denc_confidence, denc)
                )
            return s.decode(denc)
        else:
            raise ValueError(
                "Could not decode value as utf-8, or to guess its encoding: %s"
                % repr(s)
            )
    else:
        return s.decode(encoding)


def assure_bool(s):
    """Convert value into boolean following convention for strings

    to recognize on,True,yes as True, off,False,no as False
    """
    if isinstance(s, string_types):
        if s.isdigit():
            return bool(int(s))
        sl = s.lower()
        if sl in {'y', 'yes', 'true', 'on'}:
            return True
        elif sl in {'n', 'no', 'false', 'off'}:
            return False
        else:
            raise ValueError("Do not know how to treat %r as a boolean" % s)
    return bool(s)


def as_unicode(val, cast_types=object):
    """Given an arbitrary value, would try to obtain unicode value of it
    
    For unicode it would return original value, for python2 str or python3
    bytes it would use assure_unicode, for None - an empty (unicode) string,
    and for any other type (see `cast_types`) - would apply the unicode 
    constructor.  If value is not an instance of `cast_types`, TypeError
    is thrown
    
    Parameters
    ----------
    cast_types: type
      Which types to cast to unicode by providing to constructor
    """
    if val is None:
        return u''
    elif isinstance(val, text_type):
        return val
    elif isinstance(val, unicode_srctypes):
        return assure_unicode(val)
    elif isinstance(val, cast_types):
        return text_type(val)
    else:
        raise TypeError(
            "Value %r is not of any of known or provided %s types"
            % (val, cast_types))


def unique(seq, key=None):
    """Given a sequence return a list only with unique elements while maintaining order

    This is the fastest solution.  See
    https://www.peterbe.com/plog/uniqifiers-benchmark
    and
    http://stackoverflow.com/a/480227/1265472
    for more information.
    Enhancement -- added ability to compare for uniqueness using a key function

    Parameters
    ----------
    seq:
      Sequence to analyze
    key: callable, optional
      Function to call on each element so we could decide not on a full
      element, but on its member etc
    """
    seen = set()
    seen_add = seen.add
    if not key:
        return [x for x in seq if not (x in seen or seen_add(x))]
    else:
        # OPT: could be optimized, since key is called twice, but for our cases
        # should be just as fine
        return [x for x in seq if not (key(x) in seen or seen_add(key(x)))]


def all_same(items):
    """Quick check if all items are the same.

    Identical to a check like len(set(items)) == 1 but
    should be more efficient while working on generators, since would
    return False as soon as any difference detected thus possibly avoiding
    unnecessary evaluations
    """
    first = True
    first_item = None
    for item in items:
        if first:
            first = False
            first_item = item
        else:
            if item != first_item:
                return False
    # So we return False if was empty
    return not first


def map_items(func, v):
    """A helper to apply `func` to all elements (keys and values) within dict

    No type checking of values passed to func is done, so `func`
    should be resilient to values which it should not handle

    Initial usecase - apply_recursive(url_fragment, assure_unicode)
    """
    # map all elements within item
    return v.__class__(
        item.__class__(map(func, item))
        for item in v.items()
    )


def partition(items, predicate=bool):
    """Partition `items` by `predicate`.

    Parameters
    ----------
    items : iterable
    predicate : callable
        A function that will be mapped over each element in `items`. The
        elements will partitioned based on whether the return value is false or
        true.

    Returns
    -------
    A tuple with two generators, the first for 'false' items and the second for
    'true' ones.

    Notes
    -----
    Taken from Peter Otten's snippet posted at
    https://nedbatchelder.com/blog/201306/filter_a_list_into_two_parts.html
    """
    a, b = tee((predicate(item), item) for item in items)
    return ((item for pred, item in a if not pred),
            (item for pred, item in b if pred))


def generate_chunks(container, size):
    """Given a container, generate chunks from it with size up to `size`
    """
    # There could be a "smarter" solution but I think this would suffice
    assert size > 0,  "Size should be non-0 positive"
    while container:
        yield container[:size]
        container = container[size:]

#
# Generators helpers
#

def saved_generator(gen):
    """Given a generator returns two generators, where 2nd one just replays

    So the first one would be going through the generated items and 2nd one
    would be yielding saved items
    """
    saved = []

    def gen1():
        for x in gen:  # iterating over original generator
            saved.append(x)
            yield x

    def gen2():
        for x in saved:  # yielding saved entries
            yield x

    return gen1(), gen2()


#
# Decorators
#
def better_wraps(to_be_wrapped):
    """Decorator to replace `functools.wraps`

    This is based on `wrapt` instead of `functools` and in opposition to `wraps`
    preserves the correct signature of the decorated function.
    It is written with the intention to replace the use of `wraps` without any
    need to rewrite the actual decorators.
    """

    @wrapt.decorator(adapter=to_be_wrapped)
    def intermediator(to_be_wrapper, instance, args, kwargs):
        return to_be_wrapper(*args, **kwargs)

    return intermediator


# Borrowed from pandas
# Copyright: 2011-2014, Lambda Foundry, Inc. and PyData Development Team
# License: BSD-3
def optional_args(decorator):
    """allows a decorator to take optional positional and keyword arguments.
        Assumes that taking a single, callable, positional argument means that
        it is decorating a function, i.e. something like this::

            @my_decorator
            def function(): pass

        Calls decorator with decorator(f, `*args`, `**kwargs`)"""

    @better_wraps(decorator)
    def wrapper(*args, **kwargs):
        def dec(f):
            return decorator(f, *args, **kwargs)

        is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable)
        if is_decorating:
            f = args[0]
            args = []
            return dec(f)
        else:
            return dec

    return wrapper


# TODO: just provide decorators for tempfile.mk* functions. This is ugly!
def get_tempfile_kwargs(tkwargs=None, prefix="", wrapped=None):
    """Updates kwargs to be passed to tempfile. calls depending on env vars
    """
    if tkwargs is None:
        tkwargs_ = {}
    else:
        # operate on a copy of tkwargs to avoid any side-effects
        tkwargs_ = tkwargs.copy()

    # TODO: don't remember why I had this one originally
    # if len(targs)<2 and \
    if 'prefix' not in tkwargs_:
        tkwargs_['prefix'] = '_'.join(
            ['datalad_temp'] +
            ([prefix] if prefix else []) +
            ([''] if (on_windows or not wrapped) else [wrapped.__name__]))

    directory = os.environ.get('DATALAD_TESTS_TEMP_DIR')
    if directory and 'dir' not in tkwargs_:
        tkwargs_['dir'] = directory

    return tkwargs_


@optional_args
def line_profile(func):
    """Q&D helper to line profile the function and spit out stats
    """
    import line_profiler
    prof = line_profiler.LineProfiler()

    @wraps(func)
    def newfunc(*args, **kwargs):
        try:
            pfunc = prof(func)
            return pfunc(*args, **kwargs)
        finally:
            prof.print_stats()
    return newfunc

#
# Context Managers
#


@contextmanager
def nothing_cm():
    """Just a dummy cm to programmically switch context managers"""
    yield


@contextmanager
def swallow_outputs():
    """Context manager to help consuming both stdout and stderr, and print()

    stdout is available as cm.out and stderr as cm.err whenever cm is the
    yielded context manager.
    Internally uses temporary files to guarantee absent side-effects of swallowing
    into StringIO which lacks .fileno.

    print mocking is necessary for some uses where sys.stdout was already bound
    to original sys.stdout, thus mocking it later had no effect. Overriding
    print function had desired effect
    """

    class StringIOAdapter(object):
        """Little adapter to help getting out/err values
        """
        def __init__(self):
            kw = get_tempfile_kwargs({}, prefix="outputs")

            self._out = open(tempfile.mktemp(**kw), 'w')
            self._err = open(tempfile.mktemp(**kw), 'w')

        def _read(self, h):
            with open(h.name) as f:
                return f.read()

        @property
        def out(self):
            if not self._out.closed:
                self._out.flush()
            return self._read(self._out)

        @property
        def err(self):
            if not self._err.closed:
                self._err.flush()
            return self._read(self._err)

        @property
        def handles(self):
            return self._out, self._err

        def cleanup(self):
            self._out.close()
            self._err.close()
            out_name = self._out.name
            err_name = self._err.name
            del self._out
            del self._err
            gc.collect()
            rmtemp(out_name)
            rmtemp(err_name)

    def fake_print(*args, **kwargs):
        sep = kwargs.pop('sep', ' ')
        end = kwargs.pop('end', '\n')
        file = kwargs.pop('file', sys.stdout)

        if file in (oldout, olderr, sys.stdout, sys.stderr):
            # we mock
            try:
                sys.stdout.write(sep.join(args) + end)
            except UnicodeEncodeError as exc:
                lgr.error(
                    "Failed to write to mocked stdout, got %s, continue as it "
                    "didn't happen",  exc)
        else:
            # must be some other file one -- leave it alone
            oldprint(*args, sep=sep, end=end, file=file)

    from .ui import ui
    # preserve -- they could have been mocked already
    oldprint = getattr(__builtin__, 'print')
    oldout, olderr = sys.stdout, sys.stderr
    olduiout = ui.out
    adapter = StringIOAdapter()

    try:
        sys.stdout, sys.stderr = adapter.handles
        ui.out = adapter.handles[0]
        setattr(__builtin__, 'print', fake_print)

        yield adapter
    finally:
        sys.stdout, sys.stderr, ui.out = oldout, olderr, olduiout
        setattr(__builtin__, 'print',  oldprint)
        adapter.cleanup()


@contextmanager
def swallow_logs(new_level=None, file_=None, name='datalad'):
    """Context manager to consume all logs.

    """
    lgr = logging.getLogger(name)

    # Keep old settings
    old_level = lgr.level
    old_handlers = lgr.handlers

    # Let's log everything into a string
    # TODO: generalize with the one for swallow_outputs
    class StringIOAdapter(object):
        """Little adapter to help getting out values

        And to stay consistent with how swallow_outputs behaves
        """
        def __init__(self):
            if file_ is None:
                kw = get_tempfile_kwargs({}, prefix="logs")
                out_file = tempfile.mktemp(**kw)
            else:
                out_file = file_
            # PY3 requires clearly one or another.  race condition possible
            self._out = open(out_file, 'a')
            self._final_out = None

        def _read(self, h):
            with open(h.name) as f:
                return f.read()

        @property
        def out(self):
            if self._final_out is not None:
                # we closed and cleaned up already
                return self._final_out
            else:
                self._out.flush()
                return self._read(self._out)

        @property
        def lines(self):
            return self.out.split('\n')

        @property
        def handle(self):
            return self._out

        def cleanup(self):
            # store for access while object exists
            self._final_out = self.out
            self._out.close()
            out_name = self._out.name
            del self._out
            gc.collect()
            if not file_:
                rmtemp(out_name)

        def assert_logged(self, msg=None, level=None, regex=True, **kwargs):
            """Provide assertion on whether a msg was logged at a given level

            If neither `msg` nor `level` provided, checks if anything was logged
            at all.

            Parameters
            ----------
            msg: str, optional
              Message (as a regular expression, if `regex`) to be searched.
              If no msg provided, checks if anything was logged at a given level.
            level: str, optional
              String representing the level to be logged
            regex: bool, optional
              If False, regular `assert_in` is used
            **kwargs: str, optional
              Passed to `assert_re_in` or `assert_in`
            """
            from datalad.tests.utils import assert_re_in
            from datalad.tests.utils import assert_in

            if regex:
                match = '\[%s\] ' % level if level else "\[\S+\] "
            else:
                match = '[%s] ' % level if level else ''

            if msg:
                match += msg

            if match:
                (assert_re_in if regex else assert_in)(match, self.out, **kwargs)
            else:
                assert not kwargs, "no kwargs to be passed anywhere"
                assert self.out, "Nothing was logged!?"

    adapter = StringIOAdapter()
    # TODO: it does store messages but without any formatting, i.e. even without
    # date/time prefix etc.  IMHO it should preserve formatting in case if file_ is
    # set
    swallow_handler = logging.StreamHandler(adapter.handle)
    # we want to log levelname so we could test against it
    swallow_handler.setFormatter(
        logging.Formatter('[%(levelname)s] %(message)s'))
    # Inherit filters
    from datalad.log import ProgressHandler
    swallow_handler.filters = sum([h.filters for h in old_handlers
                                   if not isinstance(h, ProgressHandler)],
                                  [])
    lgr.handlers = [swallow_handler]
    if old_level < logging.DEBUG:  # so if HEAVYDEBUG etc -- show them!
        lgr.handlers += old_handlers

    if isinstance(new_level, str):
        new_level = getattr(logging, new_level)

    if new_level is not None:
        lgr.setLevel(new_level)

    try:
        yield adapter
        # TODO: if file_ and there was an exception -- most probably worth logging it?
        # although ideally it should be the next log outside added to that file_ ... oh well
    finally:
        lgr.handlers, lgr.level = old_handlers, old_level
        adapter.cleanup()


# TODO: May be melt in with swallow_logs at some point:
@contextmanager
def disable_logger(logger=None):
    """context manager to temporarily disable logging

    This is to provide one of swallow_logs' purposes without unnecessarily
    creating temp files (see gh-1865)

    Parameters
    ----------
    logger: Logger
        Logger whose handlers will be ordered to not log anything.
        Default: datalad's topmost Logger ('datalad')
    """

    class NullFilter(logging.Filter):
        """Filter class to reject all records
        """
        def filter(self, record):
            return 0

    if logger is None:
        # default: all of datalad's logging:
        logger = logging.getLogger('datalad')

    filter_ = NullFilter(logger.name)
    [h.addFilter(filter_) for h in logger.handlers]

    try:
        yield logger
    finally:
        [h.removeFilter(filter_) for h in logger.handlers]


#
# Additional handlers
#
_sys_excepthook = sys.excepthook  # Just in case we ever need original one


def setup_exceptionhook(ipython=False):
    """Overloads default sys.excepthook with our exceptionhook handler.

       If interactive, our exceptionhook handler will invoke
       pdb.post_mortem; if not interactive, then invokes default handler.
    """

    def _datalad_pdb_excepthook(type, value, tb):
        import traceback
        traceback.print_exception(type, value, tb)
        print()
        if is_interactive():
            import pdb
            pdb.post_mortem(tb)

    if ipython:
        from IPython.core import ultratb
        sys.excepthook = ultratb.FormattedTB(mode='Verbose',
                                             # color_scheme='Linux',
                                             call_pdb=is_interactive())
    else:
        sys.excepthook = _datalad_pdb_excepthook


def assure_dir(*args):
    """Make sure directory exists.

    Joins the list of arguments to an os-specific path to the desired
    directory and creates it, if it not exists yet.
    """
    dirname = opj(*args)
    if not exists(dirname):
        os.makedirs(dirname)
    return dirname


def updated(d, update):
    """Return a copy of the input with the 'update'

    Primarily for updating dictionaries
    """
    d = d.copy()
    d.update(update)
    return d


_pwd_mode = None


def _switch_to_getcwd(msg, *args):
    global _pwd_mode
    _pwd_mode = 'cwd'
    lgr.warning(
        msg + ". From now on will be returning os.getcwd(). Directory"
               " symlinks in the paths will be resolved",
        *args
    )
    # TODO:  we might want to mitigate by going through all flywheighted
    # repos and tuning up their .paths to be resolved?


def getpwd():
    """Try to return a CWD without dereferencing possible symlinks

    This function will try to use PWD environment variable to provide a current
    working directory, possibly with some directories along the path being
    symlinks to other directories.  Unfortunately, PWD is used/set only by the
    shell and such functions as `os.chdir` and `os.getcwd` nohow use or modify
    it, thus `os.getcwd()` returns path with links dereferenced.

    While returning current working directory based on PWD env variable we
    verify that the directory is the same as `os.getcwd()` after resolving all
    symlinks.  If that verification fails, we fall back to always use
    `os.getcwd()`.

    Initial decision to either use PWD env variable or os.getcwd() is done upon
    the first call of this function.
    """
    global _pwd_mode
    if _pwd_mode is None:
        # we need to decide!
        try:
            pwd = os.environ['PWD']
            if on_windows and pwd and pwd.startswith('/'):
                # It should be a path from MSYS.
                # - it might start with a drive letter or not
                # - it seems to be "illegal" to have a single letter directories
                #   under / path, i.e. if created - they aren't found
                # - 'ln -s' does not fail to create a "symlink" but it just
                # copies!
                #   so we are not likely to need original PWD purpose on
                # those systems
                # Verdict:
                _pwd_mode = 'cwd'
            else:
                _pwd_mode = 'PWD'
        except KeyError:
            _pwd_mode = 'cwd'

    if _pwd_mode == 'cwd':
        return os.getcwd()
    elif _pwd_mode == 'PWD':
        try:
            cwd = os.getcwd()
        except OSError as exc:
            if "o such file" in str(exc):
                # directory was removed but we promised to be robust and
                # still report the path we might know since we are still in PWD
                # mode
                cwd = None
            else:
                raise
        try:
            pwd = os.environ['PWD']
            pwd_real = op.realpath(pwd)
            # This logic would fail to catch the case where chdir did happen
            # to the directory where current PWD is pointing to, e.g.
            # $> ls -ld $PWD
            # lrwxrwxrwx 1 yoh yoh 5 Oct 11 13:27 /home/yoh/.tmp/tmp -> /tmp//
            # hopa:~/.tmp/tmp
            # $> python -c 'import os; os.chdir("/tmp"); from datalad.utils import getpwd; print(getpwd(), os.getcwd())'
            # ('/home/yoh/.tmp/tmp', '/tmp')
            # but I guess that should not be too harmful
            if cwd is not None and pwd_real != cwd:
                _switch_to_getcwd(
                    "realpath of PWD=%s is %s whenever os.getcwd()=%s",
                    pwd, pwd_real, cwd
                )
                return cwd
            return pwd
        except KeyError:
            _switch_to_getcwd("PWD env variable is no longer available")
            return cwd  # Must not happen, but may be someone
                        # evil purges PWD from environ?
    else:
        raise RuntimeError(
            "Must have not got here. "
            "pwd_mode must be either cwd or PWD. And it is now %r" % (_pwd_mode,)
        )


class chpwd(object):
    """Wrapper around os.chdir which also adjusts environ['PWD']

    The reason is that otherwise PWD is simply inherited from the shell
    and we have no ability to assess directory path without dereferencing
    symlinks.

    If used as a context manager it allows to temporarily change directory
    to the given path
    """
    def __init__(self, path, mkdir=False, logsuffix=''):

        if path:
            pwd = getpwd()
            self._prev_pwd = pwd
        else:
            self._prev_pwd = None
            return

        if not isabs(path):
            path = normpath(opj(pwd, path))
        if not os.path.exists(path) and mkdir:
            self._mkdir = True
            os.mkdir(path)
        else:
            self._mkdir = False
        lgr.debug("chdir %r -> %r %s", self._prev_pwd, path, logsuffix)
        os.chdir(path)  # for grep people -- ok, to chdir here!
        os.environ['PWD'] = path

    def __enter__(self):
        # nothing more to do really, chdir was in the constructor
        pass

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self._prev_pwd:
            # Need to use self.__class__ so this instance, if the entire
            # thing mocked during the test, still would use correct chpwd
            self.__class__(self._prev_pwd, logsuffix="(coming back)")


def dlabspath(path, norm=False):
    """Symlinks-in-the-cwd aware abspath

    os.path.abspath relies on os.getcwd() which would not know about symlinks
    in the path

    TODO: we might want to norm=True by default to match behavior of
    os .path.abspath?
    """
    if not isabs(path):
        # if not absolute -- relative to pwd
        path = opj(getpwd(), path)
    return normpath(path) if norm else path


def with_pathsep(path):
    """Little helper to guarantee that path ends with /"""
    return path + sep if not path.endswith(sep) else path


def get_path_prefix(path, pwd=None):
    """Get path prefix (for current directory)

    Returns relative path to the topdir, if we are under topdir, and if not
    absolute path to topdir.  If `pwd` is not specified - current directory
    assumed
    """
    pwd = pwd or getpwd()
    path = dlabspath(path)
    path_ = with_pathsep(path)
    pwd_ = with_pathsep(pwd)
    common = commonprefix((path_, pwd_))
    if common.endswith(sep) and common in {path_, pwd_}:
        # we are in subdir or above the path = use relative path
        location_prefix = relpath(path, pwd)
        # if benign "here" - cut off
        if location_prefix in (curdir, curdir + sep):
            location_prefix = ''
        return location_prefix
    else:
        # just return absolute path
        return path


def _get_normalized_paths(path, prefix):
    if isabs(path) != isabs(prefix):
        raise ValueError("Both paths must either be absolute or relative. "
                         "Got %r and %r" % (path, prefix))
    path = with_pathsep(path)
    prefix = with_pathsep(prefix)
    return path, prefix


def path_startswith(path, prefix):
    """Return True if path starts with prefix path

    Parameters
    ----------
    path: str
    prefix: str
    """
    path, prefix = _get_normalized_paths(path, prefix)
    return path.startswith(prefix)


def path_is_subpath(path, prefix):
    """Return True if path is a subpath of prefix

    It will return False if path == prefix.

    Parameters
    ----------
    path: str
    prefix: str
    """
    path, prefix = _get_normalized_paths(path, prefix)
    return (len(prefix) < len(path)) and path.startswith(prefix)


def knows_annex(path):
    """Returns whether at a given path there is information about an annex

    It is just a thin wrapper around GitRepo.is_with_annex() classmethod
    which also checks for `path` to exist first.

    This includes actually present annexes, but also uninitialized ones, or
    even the presence of a remote annex branch.
    """
    from os.path import exists
    if not exists(path):
        lgr.debug("No annex: test path {0} doesn't exist".format(path))
        return False
    from datalad.support.gitrepo import GitRepo
    return GitRepo(path, init=False, create=False).is_with_annex()


@contextmanager
def make_tempfile(content=None, wrapped=None, **tkwargs):
    """Helper class to provide a temporary file name and remove it at the end (context manager)

    Parameters
    ----------
    mkdir : bool, optional (default: False)
        If True, temporary directory created using tempfile.mkdtemp()
    content : str or bytes, optional
        Content to be stored in the file created
    wrapped : function, optional
        If set, function name used to prefix temporary file name
    `**tkwargs`:
        All other arguments are passed into the call to tempfile.mk{,d}temp(),
        and resultant temporary filename is passed as the first argument into
        the function t.  If no 'prefix' argument is provided, it will be
        constructed using module and function names ('.' replaced with
        '_').

    To change the used directory without providing keyword argument 'dir' set
    DATALAD_TESTS_TEMP_DIR.

    Examples
    --------
        >>> from os.path import exists
        >>> from datalad.utils import make_tempfile
        >>> with make_tempfile() as fname:
        ...    k = open(fname, 'w').write('silly test')
        >>> assert not exists(fname)  # was removed

        >>> with make_tempfile(content="blah") as fname:
        ...    assert open(fname).read() == "blah"
    """

    if tkwargs.get('mkdir', None) and content is not None:
        raise ValueError("mkdir=True while providing content makes no sense")

    tkwargs_ = get_tempfile_kwargs(tkwargs, wrapped=wrapped)

    # if DATALAD_TESTS_TEMP_DIR is set, use that as directory,
    # let mktemp handle it otherwise. However, an explicitly provided
    # dir=... will override this.
    mkdir = tkwargs_.pop('mkdir', False)

    filename = {False: tempfile.mktemp,
                True: tempfile.mkdtemp}[mkdir](**tkwargs_)
    filename = realpath(filename)

    if content:
        with open(filename, 'w' + ('b' if isinstance(content, binary_type) else '')) as f:
            f.write(content)

    if __debug__:
        # TODO mkdir
        lgr.debug('Created temporary thing named %s"' % filename)
    try:
        yield filename
    finally:
        # glob here for all files with the same name (-suffix)
        # would be useful whenever we requested .img filename,
        # and function creates .hdr as well
        lsuffix = len(tkwargs_.get('suffix', ''))
        filename_ = lsuffix and filename[:-lsuffix] or filename
        filenames = glob.glob(filename_ + '*')
        if len(filename_) < 3 or len(filenames) > 5:
            # For paranoid yoh who stepped into this already ones ;-)
            lgr.warning("It is unlikely that it was intended to remove all"
                        " files matching %r. Skipping" % filename_)
            return
        for f in filenames:
            try:
                rmtemp(f)
            except OSError:  # pragma: no cover
                pass


def _path_(*p):
    """Given a path in POSIX" notation, regenerate one in native to the env one"""
    if on_windows:
        return opj(*map(lambda x: opj(*x.split('/')), p))
    else:
        # Assume that all others as POSIX compliant so nothing to be done
        return opj(*p)


def get_timestamp_suffix(time_=None, prefix='-'):
    """Return a time stamp (full date and time up to second)

    primarily to be used for generation of log files names
    """
    args = []
    if time_ is not None:
        if isinstance(time_, int):
            time_ = time.gmtime(time_)
        args.append(time_)
    return time.strftime(prefix + TIMESTAMP_FMT, *args)


def get_logfilename(dspath, cmd='datalad'):
    """Return a filename to use for logging under a dataset/repository

    directory would be created if doesn't exist, but dspath must exist
    and be a directory
    """
    assert(exists(dspath))
    assert(isdir(dspath))
    ds_logdir = assure_dir(dspath, '.git', 'datalad', 'logs')  # TODO: use WEB_META_LOG whenever #789 merged
    return opj(ds_logdir, 'crawl-%s.log' % get_timestamp_suffix())


def get_trace(edges, start, end, trace=None):
    """Return the trace/path to reach a node in a tree.

    Parameters
    ----------
    edges : sequence(2-tuple)
      The tree given by a sequence of edges (parent, child) tuples. The
      nodes can be identified by any value and data type that supports
      the '==' operation.
    start :
      Identifier of the start node. Must be present as a value in the parent
      location of an edge tuple in order to be found.
    end :
      Identifier of the target/end node. Must be present as a value in the child
      location of an edge tuple in order to be found.
    trace : list
      Mostly useful for recursive calls, and used internally.

    Returns
    -------
    None or list
      Returns a list with the trace to the target (the starts and the target
      are not included in the trace, hence if start and end are directly connected
      an empty list is returned), or None when no trace to the target can be found,
      or start and end are identical.
    """
    # the term trace is used to avoid confusion with a path in the sense
    # of a filesystem path, but the analogy fits and nodes can be paths
    if trace is None:
        trace = []
    if not edges:
        raise ValueError("no edges given")
    for cand in edges:
        cand_super, cand_sub = cand
        if cand_sub in trace:
            # only DAGs, skip any cyclic traces
            continue
        if trace and cand_super != trace[-1]:
            # only consider edges that lead off the end of the trace
            continue
        if not trace and cand_super != start:
            # we got nothing yet, and this edges is not matching the start
            continue
        if cand_sub == end:
            return trace
        # dive into potential subnodes
        cand_trace = get_trace(
            edges,
            start,
            end,
            trace + [cand_sub])
        if cand_trace:
            return cand_trace
    return None


def get_dataset_root(path):
    """Return the root of an existent dataset containing a given path

    The root path is returned in the same absolute or relative form
    as the input argument. If no associated dataset exists, or the
    input path doesn't exist, None is returned.
    """
    suffix = '.git'
    if not isdir(path):
        path = dirname(path)
    apath = abspath(path)
    # while we can still go up
    while psplit(apath)[1]:
        if exists(opj(path, suffix)):
            return path
        # new test path in the format we got it
        path = normpath(opj(path, os.pardir))
        # no luck, next round
        apath = abspath(path)
    return None


def get_dataset_pwds(dataset):
    """Return the current directory for the dataset.

    Parameters
    ----------
    dataset : Dataset

    Returns
    -------
    A tuple, where the first item is the absolute path of the pwd and the
    second is the pwd relative to the dataset's path.
    """
    if dataset:
        pwd = dataset.path
        rel_pwd = curdir
    else:
        # act on the whole dataset if nothing else was specified

        # Follow our generic semantic that if dataset is specified,
        # paths are relative to it, if not -- relative to pwd
        pwd = getpwd()
        # Pass pwd to get_dataset_root instead of os.path.curdir to handle
        # repos whose leading paths have a symlinked directory (see the
        # TMPDIR="/var/tmp/sym link" test case).
        dataset = get_dataset_root(pwd)

        if dataset:
            rel_pwd = relpath(pwd, dataset)
        else:
            rel_pwd = pwd  # and leave handling to caller
    return pwd, rel_pwd


# ATM used in datalad_crawler extension, so do not remove yet
def try_multiple(ntrials, exception, base, f, *args, **kwargs):
    """Call f multiple times making exponentially growing delay between the calls"""
    from .dochelpers import exc_str
    for trial in range(1, ntrials+1):
        try:
            return f(*args, **kwargs)
        except exception as exc:
            if trial == ntrials:
                raise  # just reraise on the last trial
            t = base ** trial
            lgr.warning("Caught %s on trial #%d. Sleeping %f and retrying",
                        exc_str(exc), trial, t)
            sleep(t)


@optional_args
def try_multiple_dec(f, ntrials=None, duration=0.1, exceptions=None, increment_type=None):
    """Decorator to try function multiple times.

    Main purpose is to decorate functions dealing with removal of files/directories
    and which might need a few seconds to work correctly on Windows which takes
    its time to release files/directories.

    Parameters
    ----------
    ntrials: int, optional
    duration: float, optional
      Seconds to sleep before retrying.
    increment_type: {None, 'exponential'}
      Note that if it is exponential, duration should typically be > 1.0
      so it grows with higher power

    """
    from .dochelpers import exc_str
    if not exceptions:
        exceptions = (OSError, WindowsError, PermissionError) \
            if on_windows else OSError
    if not ntrials:
        # Life goes fast on proper systems, no need to delay it much
        ntrials = 50 if on_windows else 3

    assert increment_type in {None, 'exponential'}

    @wraps(f)
    def wrapped(*args, **kwargs):
        t = duration
        for trial in range(ntrials):
            try:
                return f(*args, **kwargs)
            except exceptions as exc:
                if increment_type == 'exponential':
                    t = duration ** (trial + 1)
                lgr.log(
                    5,
                    "Caught %s on trial #%d. Sleeping %f and retrying",
                    exc_str(exc), trial, t)
                if trial < ntrials - 1:
                    sleep(t)
                else:
                    raise

    return wrapped


@try_multiple_dec
def unlink(f):
    """'Robust' unlink.  Would try multiple times

    On windows boxes there is evidence for a latency of more than a second
    until a file is considered no longer "in-use".
    WindowsError is not known on Linux, and if IOError or any other
    exception
    is thrown then if except statement has WindowsError in it -- NameError
    also see gh-2533
    """
    # Check for open files
    assert_no_open_files(f)
    return os.unlink(f)


@try_multiple_dec
def _rmtree(*args, **kwargs):
    """Just a helper to decorate shutil.rmtree.

    rmtree defined above does more and ideally should not itself be decorated
    since a recursive definition and does checks for open files inside etc -
    might be too runtime expensive
    """
    return shutil.rmtree(*args, **kwargs)


def slash_join(base, extension):
    """Join two strings with a '/', avoiding duplicate slashes

    If any of the strings is None the other is returned as is.
    """
    if extension is None:
        return base
    if base is None:
        return extension
    return '/'.join(
        (base.rstrip('/'),
         extension.lstrip('/')))


def safe_print(s):
    """Print with protection against UTF-8 encoding errors"""
    # A little bit of dance to be able to test this code
    print_f = getattr(__builtin__, "print")
    try:
        print_f(s)
    except UnicodeEncodeError:
        # failed to encode so let's do encoding while ignoring errors
        # to print at least something
        # explicit `or ascii` since somehow on buildbot it seemed to return None
        s = s.encode(getattr(sys.stdout, 'encoding', 'ascii') or 'ascii', errors='ignore') \
            if hasattr(s, 'encode') else s
        print_f(s.decode())

#
# IO Helpers
#

def open_r_encdetect(fname, readahead=1000):
    """Return a file object in read mode with auto-detected encoding

    This is helpful when dealing with files of unknown encoding.

    Parameters
    ----------
    readahead: int, optional
      How many bytes to read for guessing the encoding type.  If
      negative - full file will be read
    """
    from chardet import detect
    import io
    # read some bytes from the file
    with open(fname, 'rb') as f:
        head = f.read(readahead)
    enc = detect(head)
    denc = enc.get('encoding', None)
    lgr.debug("Auto-detected encoding %s for file %s (confidence: %s)",
              denc,
              fname,
              enc.get('confidence', 'unknown'))
    return io.open(fname, encoding=denc)


def read_csv_lines(fname, dialect=None, readahead=16384, **kwargs):
    """A generator of dict records from a CSV/TSV

    Automatically guesses the encoding for each record to convert to UTF-8

    Parameters
    ----------
    fname: str
      Filename
    dialect: str, optional
      Dialect to specify to csv.reader. If not specified -- guessed from
      the file, if fails to guess, "excel-tab" is assumed
    readahead: int, optional
      How many bytes to read from the file to guess the type
    **kwargs
      Passed to `csv.reader`
    """
    import csv
    if dialect is None:
        with open(fname) as tsvfile:
            # add robustness, use a sniffer
            try:
                dialect = csv.Sniffer().sniff(tsvfile.read(readahead))
            except Exception as exc:
                from .dochelpers import exc_str
                lgr.warning(
                    'Could not determine file-format, assuming TSV: %s',
                    exc_str(exc)
                )
                dialect = 'excel-tab'

    kw = {} if PY2 else dict(encoding='utf-8')
    with open(fname, 'rb' if PY2 else 'r', **kw) as tsvfile:
        # csv.py doesn't do Unicode; encode temporarily as UTF-8:
        csv_reader = csv.reader(
            tsvfile,
            dialect=dialect,
            **kwargs
        )
        header = None
        for row in csv_reader:
            # decode UTF-8 back to Unicode, cell by cell:
            row_unicode = map(assure_unicode, row)
            if header is None:
                header = list(row_unicode)
            else:
                yield dict(zip(header, row_unicode))


def import_modules(modnames, pkg, msg="Failed to import {module}", log=lgr.debug):
    """Helper to import a list of modules without failing if N/A

    Parameters
    ----------
    modnames: list of str
      List of module names to import
    pkg: str
      Package under which to import
    msg: str, optional
      Message template for .format() to log at DEBUG level if import fails.
      Keys {module} and {package} will be provided and ': {exception}' appended
    log: callable, optional
      Logger call to use for logging messages
    """
    from importlib import import_module
    _globals = globals()
    mods_loaded = []
    if pkg and not pkg in sys.modules:
        # with python 3.5.1 (ok with 3.5.5) somehow kept running into
        #  Failed to import dlsub1: Parent module 'dltestm1' not loaded
        # while running the test. Preloading pkg resolved the issue
        import_module(pkg)
    for modname in modnames:
        try:
            _globals[modname] = mod = import_module(
                '.{}'.format(modname),
                pkg)
            mods_loaded.append(mod)
        except Exception as exc:
            from datalad.dochelpers import exc_str
            log((msg + ': {exception}').format(
                module=modname, package=pkg, exception=exc_str(exc)))
    return mods_loaded


def import_module_from_file(modpath, pkg=None, log=lgr.debug):
    """Import provided module given a path

    TODO:
    - RF/make use of it in pipeline.py which has similar logic
    - join with import_modules above?

    Parameters
    ----------
    pkg: module, optional
       If provided, and modpath is under pkg.__path__, relative import will be
       used
    """
    assert(modpath.endswith('.py'))  # for now just for .py files

    log("Importing %s" % modpath)

    modname = basename(modpath)[:-3]
    relmodpath = None
    if pkg:
        for pkgpath in pkg.__path__:
            if path_is_subpath(modpath, pkgpath):
                # for now relying on having .py extension -- assertion above
                relmodpath = '.' + relpath(modpath[:-3], pkgpath).replace(sep, '.')
                break

    try:
        if relmodpath:
            from importlib import import_module
            mod = import_module(relmodpath, pkg.__name__)
        else:
            dirname_ = dirname(modpath)
            try:
                sys.path.insert(0, dirname_)
                mod = __import__(modname, level=0)
            finally:
                if dirname_ in sys.path:
                    sys.path.pop(sys.path.index(dirname_))
                else:
                    log("Expected path %s to be within sys.path, but it was gone!" % dirname_)
    except Exception as e:
        from datalad.dochelpers import exc_str
        raise RuntimeError(
            "Failed to import module from %s: %s" % (modpath, exc_str(e)))

    return mod


def get_encoding_info():
    """Return a dictionary with various encoding/locale information"""
    import sys, locale
    from collections import OrderedDict
    return OrderedDict([
        ('default', sys.getdefaultencoding()),
        ('filesystem', sys.getfilesystemencoding()),
        ('locale.prefered', locale.getpreferredencoding()),
    ])


def get_envvars_info():
    from collections import OrderedDict
    envs = []
    for var, val in os.environ.items():
        if (
                var.startswith('PYTHON') or
                var.startswith('LC_') or
                var.startswith('GIT_') or
                var in ('LANG', 'LANGUAGE', 'PATH')
        ):
            envs.append((var, val))
    return OrderedDict(envs)


# This class is modified from Snakemake (v5.1.4)
class SequenceFormatter(string.Formatter):
    """string.Formatter subclass with special behavior for sequences.

    This class delegates formatting of individual elements to another
    formatter object. Non-list objects are formatted by calling the
    delegate formatter's "format_field" method. List-like objects
    (list, tuple, set, frozenset) are formatted by formatting each
    element of the list according to the specified format spec using
    the delegate formatter and then joining the resulting strings with
    a separator (space by default).
    """

    def __init__(self, separator=" ", element_formatter=string.Formatter(),
                 *args, **kwargs):
        self.separator = separator
        self.element_formatter = element_formatter

    def format_element(self, elem, format_spec):
        """Format a single element

        For sequences, this is called once for each element in a
        sequence. For anything else, it is called on the entire
        object. It is intended to be overridden in subclases.
        """
        return self.element_formatter.format_field(elem, format_spec)

    def format_field(self, value, format_spec):
        if isinstance(value, (list, tuple, set, frozenset)):
            return self.separator.join(self.format_element(v, format_spec)
                                       for v in value)
        else:
            return self.format_element(value, format_spec)


# TODO: eventually we might want to make use of attr module
class File(object):
    """Helper for a file entry in the create_tree/@with_tree

    It allows to define additional settings for entries
    """
    def __init__(self, name, executable=False):
        """

        Parameters
        ----------
        name : str
          Name of the file
        executable: bool, optional
          Make it executable
        """
        self.name = name
        self.executable = executable

    def __str__(self):
        return self.name


def create_tree_archive(path, name, load, overwrite=False, archives_leading_dir=True):
    """Given an archive `name`, create under `path` with specified `load` tree
    """
    from datalad.support.archives import compress_files
    dirname = file_basename(name)
    full_dirname = op.join(path, dirname)
    os.makedirs(full_dirname)
    create_tree(full_dirname, load, archives_leading_dir=archives_leading_dir)
    # create archive
    if archives_leading_dir:
        compress_files([dirname], name, path=path, overwrite=overwrite)
    else:
        compress_files(list(map(op.basename, glob.glob(opj(full_dirname, '*')))),
                       opj(op.pardir, name),
                       path=op.join(path, dirname),
                       overwrite=overwrite)
    # remove original tree
    rmtree(full_dirname)


def create_tree(path, tree, archives_leading_dir=True, remove_existing=False):
    """Given a list of tuples (name, load) create such a tree

    if load is a tuple itself -- that would create either a subtree or an archive
    with that content and place it into the tree if name ends with .tar.gz
    """
    lgr.log(5, "Creating a tree under %s", path)
    if not op.exists(path):
        os.makedirs(path)

    if isinstance(tree, dict):
        tree = tree.items()

    for file_, load in tree:
        if isinstance(file_, File):
            executable = file_.executable
            name = file_.name
        else:
            executable = False
            name = file_
        full_name = op.join(path, name)
        if remove_existing and op.lexists(full_name):
            rmtree(full_name, chmod_files=True)
        if isinstance(load, (tuple, list, dict)):
            if name.endswith('.tar.gz') or name.endswith('.tar') or name.endswith('.zip'):
                create_tree_archive(
                    path, name, load,
                    archives_leading_dir=archives_leading_dir)
            else:
                create_tree(
                    full_name, load,
                    archives_leading_dir=archives_leading_dir,
                    remove_existing=remove_existing)
        else:
            open_func = open
            if full_name.endswith('.gz'):
                open_func = gzip.open
            with open_func(full_name, "wb") as f:
                f.write(assure_bytes(load, 'utf-8'))
        if executable:
            os.chmod(full_name, os.stat(full_name).st_mode | stat.S_IEXEC)


lgr.log(5, "Done importing datalad.utils")

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user