{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\nWrapper for command and function calls, allowing for dry runs and output handling\n\n\"\"\"\n\nimport time\nimport subprocess\nimport sys\nimport logging\nimport os\nimport atexit\nimport functools\nimport tempfile\n\nfrom collections import OrderedDict\nfrom .support import path as op\nfrom .consts import GIT_SSH_COMMAND\nfrom .dochelpers import exc_str\nfrom .support.exceptions import CommandError\nfrom .support.protocol import (\n    NullProtocol,\n    ExecutionTimeProtocol,\n    ExecutionTimeExternalsProtocol,\n)\nfrom .utils import (\n    on_windows,\n    get_tempfile_kwargs,\n    assure_unicode,\n    assure_bytes,\n    unlink,\n    auto_repr,\n    split_cmdline,\n)\nfrom .dochelpers import borrowdoc\n\nlgr = logging.getLogger('datalad.cmd')\n\n# In python3 to split byte stream on newline, it must be bytes\nlinesep_bytes = os.linesep.encode()\n\n_TEMP_std = sys.stdout, sys.stderr\n# To be used in the temp file name to distinguish the ones we create\n# in Runner so we take care about their removal, in contrast to those\n# which might be created outside and passed into Runner\n_MAGICAL_OUTPUT_MARKER = \"_runneroutput_\"\n\nfrom io import IOBase as file_class\n\n\ndef _decide_to_log(v):\n    \"\"\"Hacky workaround for now so we could specify per each which to\n    log online and which to the log\"\"\"\n    if isinstance(v, bool) or callable(v):\n        return v\n    elif v in {'online'}:\n        return True\n    elif v in {'offline'}:\n        return False\n    else:\n        raise ValueError(\"can be bool, callable, 'online' or 'offline'\")\n\n\ndef _get_output_stream(log_std, false_value):\n    \"\"\"Helper to prepare output stream for Popen and use file for 'offline'\n\n    Necessary to avoid lockdowns when both stdout and stderr are pipes\n    \"\"\"\n    if log_std:\n        if log_std == 'offline':\n            # we will open a temporary file\n\n            tf = tempfile.mktemp(\n                **get_tempfile_kwargs({}, prefix=_MAGICAL_OUTPUT_MARKER)\n            )\n            return open(tf, 'w')  # XXX PY3 should be 'b' may be?\n        else:\n            return subprocess.PIPE\n    else:\n        return false_value\n\n\ndef _cleanup_output(stream, std):\n    if isinstance(stream, file_class) and \\\n        _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):\n        if not stream.closed:\n            stream.close()\n        if op.exists(stream.name):\n            unlink(stream.name)\n    elif stream == subprocess.PIPE:\n        std.close()\n\n\ndef kill_output(output):\n    \"\"\"Helper for WitlessRunner to swallow all output and neither\n    relay it to the parent process's stdout, nor provide it as the\n    return value of WitlessRunner.run().\n    \"\"\"\n    return b'', 0\n\n\ndef capture_output(output):\n    \"\"\"Helper for WitlessRunner to capture all output and\n    provide it as the return value of WitlessRunner.run().\n    \"\"\"\n    return output, 0\n\n\nclass WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out\n\n\nclass Runner(WitlessRunner):\n    \"\"\"Provides a wrapper for calling functions and commands.\n\n    An object of this class provides a methods that calls shell commands or\n    python functions, allowing for protocolling the calls and output handling.\n\n    Outputs (stdout and stderr) can be either logged or streamed to system's\n    stdout/stderr during execution.\n    This can be enabled or disabled for both of them independently.\n    Additionally, a protocol object can be a used with the Runner. Such a\n    protocol has to implement datalad.support.protocol.ProtocolInterface, is\n    able to record calls and allows for dry runs.\n    \"\"\"\n\n    __slots__ = ['commands', 'dry', 'cwd', 'env', 'protocol',\n                 '_log_opts']\n\n    def __init__(self, cwd=None, env=None, protocol=None, log_outputs=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd: string, optional\n             Base current working directory for commands.  Could be overridden\n             per run call via cwd option\n        env: dict, optional\n             Custom environment to use for calls. Could be overridden per run\n             call via env option\n        protocol: ProtocolInterface\n             Protocol object to write to.\n        log_outputs : bool, optional\n             Switch to instruct whether outputs should be logged or not.  If not\n             set (default), config 'datalad.log.outputs' would be consulted\n        \"\"\"\n        super(Runner, self).__init__(cwd=cwd, env=env)\n\n        if protocol is None:\n            # TODO: config cmd.protocol = null\n            protocol_str = os.environ.get('DATALAD_CMD_PROTOCOL', 'null')\n            protocol = {\n                'externals-time': ExecutionTimeExternalsProtocol,\n                'time': ExecutionTimeProtocol,\n                'null': NullProtocol\n            }[protocol_str]()\n            if protocol_str != 'null':\n                # we need to dump it into a file at the end\n                # TODO: config cmd.protocol_prefix = protocol\n                filename = '%s-%s.log' % (\n                    os.environ.get('DATALAD_CMD_PROTOCOL_PREFIX', 'protocol'),\n                    id(self)\n                )\n                atexit.register(functools.partial(protocol.write_to_file, filename))\n\n        self.protocol = protocol\n        # Various options for logging\n        self._log_opts = {}\n        # we don't know yet whether we need to log every output or not\n        if log_outputs is not None:\n            self._log_opts['outputs'] = log_outputs\n\n    def __call__(self, cmd, *args, **kwargs):\n        \"\"\"Convenience method\n\n        This will call run() or call() depending on the kind of `cmd`.\n        If `cmd` is a string it is interpreted as the to be executed command.\n        Otherwise it is expected to be a callable.\n        Any other argument is passed to the respective method.\n\n        Parameters\n        ----------\n        cmd: str or callable\n           command string to be executed via shell or callable to be called.\n\n        `*args`:\n        `**kwargs`:\n           see Runner.run() and Runner.call() for available arguments.\n\n        Raises\n        ------\n        TypeError\n          if cmd is neither a string nor a callable.\n        \"\"\"\n\n        if isinstance(cmd, str) or isinstance(cmd, list):\n            return self.run(cmd, *args, **kwargs)\n        elif callable(cmd):\n            return self.call(cmd, *args, **kwargs)\n        else:\n            raise TypeError(\"Argument 'command' is neither a string, \"\n                            \"nor a list nor a callable.\")\n\n    def _opt_env_adapter(v):\n        \"\"\"If value is a string, split by ,\"\"\"\n        if v:\n            if v.isdigit():\n                log_env = bool(int(v))\n            else:\n                log_env = v.split(',')\n            return log_env\n        else:\n            return False\n\n    _LOG_OPTS_ADAPTERS = OrderedDict([\n        ('outputs', None),\n        ('cwd', None),\n        ('env', _opt_env_adapter),\n        ('stdin', None),\n    ])\n\n    def _get_log_setting(self, opt, default=False):\n        try:\n            return self._log_opts[opt]\n        except KeyError:\n            try:\n                from . import cfg\n            except ImportError:\n                return default\n            adapter = self._LOG_OPTS_ADAPTERS.get(opt, None)\n            self._log_opts[opt] = \\\n                (cfg.getbool if not adapter else cfg.get_value)(\n                    'datalad.log.cmd', opt, default=default)\n            if adapter:\n                self._log_opts[opt] = adapter(self._log_opts[opt])\n            return self._log_opts[opt]\n\n    @property\n    def log_outputs(self):\n        return self._get_log_setting('outputs')\n\n    @property\n    def log_cwd(self):\n        return self._get_log_setting('cwd')\n\n    @property\n    def log_stdin(self):\n        return self._get_log_setting('stdin')\n\n    @property\n    def log_env(self):\n        return self._get_log_setting('env')\n\n    # Two helpers to encapsulate formatting/output\n    def _log_out(self, line):\n        if line and self.log_outputs:\n            self.log(\"stdout| \" + line.rstrip('\\n'))\n\n    def _log_err(self, line, expected=False):\n        if line and self.log_outputs:\n            self.log(\"stderr| \" + line.rstrip('\\n'),\n                     level={True: 9,\n                            False: 11}[expected])\n\n    def _get_output_online(self, proc,\n                           log_stdout, log_stderr,\n                           outputstream, errstream,\n                           expect_stderr=False, expect_fail=False):\n        \"\"\"\n\n        If log_stdout or log_stderr are callables, they will be given a read\n        line to be processed, and return processed result.  So if they need to\n        'swallow' the line from being logged, should just return None\n\n        Parameters\n        ----------\n        proc\n        log_stdout: bool or callable or 'online' or 'offline'\n        log_stderr: : bool or callable or 'online' or 'offline'\n          If any of those 'offline', we would call proc.communicate at the\n          end to grab possibly outstanding output from it\n        expect_stderr\n        expect_fail\n\n        Returns\n        -------\n\n        \"\"\"\n        stdout, stderr = bytes(), bytes()\n\n        log_stdout_ = _decide_to_log(log_stdout)\n        log_stderr_ = _decide_to_log(log_stderr)\n        log_stdout_is_callable = callable(log_stdout_)\n        log_stderr_is_callable = callable(log_stderr_)\n\n        # arguments to be passed into _process_one_line\n        stdout_args = (\n                'stdout',\n                proc, log_stdout_, log_stdout_is_callable\n        )\n        stderr_args = (\n                'stderr',\n                proc, log_stderr_, log_stderr_is_callable,\n                expect_stderr or expect_fail\n        )\n\n        while proc.poll() is None:\n            # see for a possibly useful approach to processing output\n            # in another thread http://codereview.stackexchange.com/a/17959\n            # current problem is that if there is no output on stderr\n            # it stalls\n            # Monitor if anything was output and if nothing, sleep a bit\n            stdout_, stderr_ = None, None\n            if log_stdout_:\n                stdout_ = self._process_one_line(*stdout_args)\n                stdout += stdout_\n            if log_stderr_:\n                stderr_ = self._process_one_line(*stderr_args)\n                stderr += stderr_\n            if stdout_ is None and stderr_ is None:\n                # no output was really produced, so sleep a tiny bit\n                time.sleep(0.001)\n\n        # Handle possible remaining output\n        if log_stdout_ and log_stderr_:\n            # If Popen was called with more than two pipes, calling\n            # communicate() after we partially read the stream will return\n            # empty output.\n            stdout += self._process_remaining_output(\n                outputstream, proc.stdout.read(), *stdout_args)\n            stderr += self._process_remaining_output(\n                errstream, proc.stderr.read(), *stderr_args)\n        stdout_, stderr_ = proc.communicate()\n        # ??? should we condition it on log_stdout in {'offline'} ???\n        stdout += self._process_remaining_output(outputstream, stdout_, *stdout_args)\n        stderr += self._process_remaining_output(errstream, stderr_, *stderr_args)\n\n        return stdout, stderr\n\n    def _process_remaining_output(self, stream, out_, *pargs):\n        \"\"\"Helper to process output which might have been obtained from popen or\n        should be loaded from file\"\"\"\n        out = bytes()\n        if isinstance(stream, file_class) and \\\n                _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):\n            assert out_ is None, \"should have gone into a file\"\n            if not stream.closed:\n                stream.close()\n            with open(stream.name, 'rb') as f:\n                for line in f:\n                    out += self._process_one_line(*pargs, line=line)\n        else:\n            if out_:\n                # resolving a once in a while failing test #2185\n                if isinstance(out_, str):\n                    out_ = out_.encode('utf-8')\n                for line in out_.split(linesep_bytes):\n                    out += self._process_one_line(\n                        *pargs, line=line, suf=linesep_bytes)\n        return out\n\n    def _process_one_line(self, out_type, proc, log_, log_is_callable,\n                          expected=False, line=None, suf=None):\n        if line is None:\n            lgr.log(3, \"Reading line from %s\", out_type)\n            line = {'stdout': proc.stdout, 'stderr': proc.stderr}[out_type].readline()\n        else:\n            lgr.log(3, \"Processing provided line\")\n        if line and log_is_callable:\n            # Let it be processed\n            line = log_(assure_unicode(line))\n            if line is not None:\n                # we are working with binary type here\n                line = assure_bytes(line)\n        if line:\n            if out_type == 'stdout':\n                self._log_out(assure_unicode(line))\n            elif out_type == 'stderr':\n                self._log_err(line.decode('utf-8'),\n                              expected)\n            else:  # pragma: no cover\n                raise RuntimeError(\"must not get here\")\n            return (line + suf) if suf else line\n        # it was output already directly but for code to work, return \"\"\n        return bytes()\n\n    def run(self, cmd, log_stdout=True, log_stderr=True, log_online=False,\n            expect_stderr=False, expect_fail=False,\n            cwd=None, env=None, shell=None, stdin=None):\n        \"\"\"Runs the command `cmd` using shell.\n\n        In case of dry-mode `cmd` is just added to `commands` and it is\n        actually executed otherwise.\n        Allows for separately logging stdout and stderr  or streaming it to\n        system's stdout or stderr respectively.\n\n        Note: Using a string as `cmd` and shell=True allows for piping,\n              multiple commands, etc., but that implies split_cmdline() is not\n              used. This is considered to be a security hazard.\n              So be careful with input.\n\n        Parameters\n        ----------\n        cmd : str, list\n          String (or list) defining the command call.  No shell is used if cmd\n          is specified as a list\n\n        log_stdout: bool, optional\n            If True, stdout is logged. Goes to sys.stdout otherwise.\n\n        log_stderr: bool, optional\n            If True, stderr is logged. Goes to sys.stderr otherwise.\n\n        log_online: bool, optional\n            Whether to log as output comes in.  Setting to True is preferable\n            for running user-invoked actions to provide timely output\n\n        expect_stderr: bool, optional\n            Normally, having stderr output is a signal of a problem and thus it\n            gets logged at level 11.  But some utilities, e.g. wget, use\n            stderr for their progress output.  Whenever such output is expected,\n            set it to True and output will be logged at level 9 unless\n            exit status is non-0 (in non-online mode only, in online -- would\n            log at 9)\n\n        expect_fail: bool, optional\n            Normally, if command exits with non-0 status, it is considered an\n            error and logged at level 11 (above DEBUG). But if the call intended\n            for checking routine, such messages are usually not needed, thus\n            it will be logged at level 9.\n\n        cwd : string, optional\n            Directory under which run the command (passed to Popen)\n\n        env : string, optional\n            Custom environment to pass\n\n        shell: bool, optional\n            Run command in a shell.  If not specified, then it runs in a shell\n            only if command is specified as a string (not a list)\n\n        stdin: file descriptor\n            input stream to connect to stdin of the process.\n\n        Returns\n        -------\n        (stdout, stderr) - bytes!\n\n        Raises\n        ------\n        CommandError\n           if command's exitcode wasn't 0 or None. exitcode is passed to\n           CommandError's `code`-field. Command's stdout and stderr are stored\n           in CommandError's `stdout` and `stderr` fields respectively.\n        \"\"\"\n        outputstream = _get_output_stream(log_stdout, sys.stdout)\n        errstream = _get_output_stream(log_stderr, sys.stderr)\n\n        popen_env = env or self.env\n        popen_cwd = cwd or self.cwd\n\n        if popen_cwd and popen_env and 'PWD' in popen_env:\n            # we must have inherited PWD, but cwd was provided, so we must\n            # adjust it\n            popen_env = popen_env.copy()  # to avoid side-effects\n            popen_env['PWD'] = popen_cwd\n\n        # TODO: if outputstream is sys.stdout and that one is set to StringIO\n        #       we have to \"shim\" it with something providing fileno().\n        # This happens when we do not swallow outputs, while allowing nosetest's\n        # StringIO to be provided as stdout, crashing the Popen requiring\n        # fileno().  In out swallow_outputs, we just use temporary files\n        # to overcome this problem.\n        # For now necessary test code should be wrapped into swallow_outputs cm\n        # to avoid the problem\n        log_msgs = [\"Running: %s\"]\n        log_args = [cmd]\n        if self.log_cwd:\n            log_msgs += ['cwd=%r']\n            log_args += [popen_cwd]\n        if self.log_stdin:\n            log_msgs += ['stdin=%r']\n            log_args += [stdin]\n        log_env = self.log_env\n        if log_env and popen_env:\n            log_msgs += [\"env=%r\"]\n            log_args.append(\n                popen_env if log_env is True\n                else {k: popen_env[k] for k in log_env if k in popen_env}\n            )\n        log_msg = '\\n'.join(log_msgs)\n        self.log(log_msg, *log_args)\n\n        if self.protocol.do_execute_ext_commands:\n\n            if shell is None:\n                shell = isinstance(cmd, str)\n\n            if self.protocol.records_ext_commands:\n                prot_exc = None\n                prot_id = self.protocol.start_section(\n                    split_cmdline(cmd)\n                    if isinstance(cmd, str)\n                    else cmd)\n            try:\n                proc = subprocess.Popen(cmd,\n                                        stdout=outputstream,\n                                        stderr=errstream,\n                                        shell=shell,\n                                        cwd=popen_cwd,\n                                        env=popen_env,\n                                        stdin=stdin)\n\n            except Exception as e:\n                prot_exc = e\n                lgr.log(11, \"Failed to start %r%r: %s\" %\n                        (cmd, \" under %r\" % cwd if cwd else '', exc_str(e)))\n                raise\n\n            finally:\n                if self.protocol.records_ext_commands:\n                    self.protocol.end_section(prot_id, prot_exc)\n\n            try:\n                if log_online:\n                    out = self._get_output_online(proc,\n                                                  log_stdout, log_stderr,\n                                                  outputstream, errstream,\n                                                  expect_stderr=expect_stderr,\n                                                  expect_fail=expect_fail)\n                else:\n                    out = proc.communicate()\n\n                # Decoding was delayed to this point\n                def decode_if_not_None(x):\n                    return \"\" if x is None else bytes.decode(x)\n                out = tuple(map(decode_if_not_None, out))\n\n                status = proc.poll()\n\n                # needs to be done after we know status\n                if not log_online:\n                    self._log_out(out[0])\n                    if status not in [0, None]:\n                        self._log_err(out[1], expected=expect_fail)\n                    else:\n                        # as directed\n                        self._log_err(out[1], expected=expect_stderr)\n\n                if status not in [0, None]:\n                    msg = \"Failed to run %r%s. Exit code=%d.%s%s\" \\\n                        % (cmd, \" under %r\" % (popen_cwd), status,\n                           \"\" if log_online else \" out=%s\" % out[0],\n                           \"\" if log_online else \" err=%s\" % out[1])\n                    lgr.log(9 if expect_fail else 11, msg)\n                    raise CommandError(str(cmd), msg, status, out[0], out[1])\n                else:\n                    self.log(\"Finished running %r with status %s\" % (cmd, status),\n                             level=8)\n\n            except CommandError:\n                # do not bother with reacting to \"regular\" CommandError\n                # exceptions.  Somehow if we also terminate here for them\n                # some processes elsewhere might stall:\n                # see https://github.com/datalad/datalad/pull/3794\n                raise\n\n            except BaseException as exc:\n                exc_info = sys.exc_info()\n                # KeyboardInterrupt is subclass of BaseException\n                lgr.debug(\"Terminating process for %s upon exception: %s\",\n                          cmd, exc_str(exc))\n                try:\n                    # there are still possible (although unlikely) cases when\n                    # we fail to interrupt but we\n                    # should not crash if we fail to terminate the process\n                    proc.terminate()\n                except BaseException as exc2:\n                    lgr.warning(\"Failed to terminate process for %s: %s\",\n                                cmd, exc_str(exc2))\n                raise exc_info[1]\n\n            finally:\n                # Those streams are for us to close if we asked for a PIPE\n                # TODO -- assure closing the files\n                _cleanup_output(outputstream, proc.stdout)\n                _cleanup_output(errstream, proc.stderr)\n\n        else:\n            if self.protocol.records_ext_commands:\n                self.protocol.add_section(split_cmdline(cmd)\n                                          if isinstance(cmd, str)\n                                          else cmd, None)\n            out = (\"DRY\", \"DRY\")\n\n        return out\n\n    def call(self, f, *args, **kwargs):\n        \"\"\"Helper to unify collection of logging all \"dry\" actions.\n\n        Calls `f` if `Runner`-object is not in dry-mode. Adds `f` along with\n        its arguments to `commands` otherwise.\n\n        Parameters\n        ----------\n        f: callable\n        \"\"\"\n        if self.protocol.do_execute_callables:\n            if self.protocol.records_callables:\n                prot_exc = None\n                prot_id = self.protocol.start_section(\n                    [str(f), \"args=%s\" % str(args), \"kwargs=%s\" % str(kwargs)])\n\n            try:\n                return f(*args, **kwargs)\n            except Exception as e:\n                prot_exc = e\n                raise\n            finally:\n                if self.protocol.records_callables:\n                    self.protocol.end_section(prot_id, prot_exc)\n        else:\n            if self.protocol.records_callables:\n                self.protocol.add_section(\n                    [str(f), \"args=%s\" % str(args), \"kwargs=%s\" % str(kwargs)],\n                    None)\n\n    def log(self, msg, *args, **kwargs):\n        \"\"\"log helper\n\n        Logs at level 9 by default and adds \"Protocol:\"-prefix in order to\n        log the used protocol.\n        \"\"\"\n        level = kwargs.pop('level', 9)\n        if isinstance(self.protocol, NullProtocol):\n            lgr.log(level, msg, *args, **kwargs)\n        else:\n            if args:\n                msg = msg % args\n            lgr.log(level, \"{%s} %s\" % (\n                self.protocol.__class__.__name__, msg)\n            )\n\n\nclass GitRunner(Runner):\n    \"\"\"\n    Runner to be used to run git and git annex commands\n\n    Overloads the runner class to check & update GIT_DIR and\n    GIT_WORK_TREE environment variables set to the absolute path\n    if is defined and is relative path\n    \"\"\"\n    _GIT_PATH = None\n\n    @borrowdoc(Runner)\n    def __init__(self, *args, **kwargs):\n        super(GitRunner, self).__init__(*args, **kwargs)\n        self._check_git_path()\n\n    @staticmethod\n    def _check_git_path():\n        \"\"\"If using bundled git-annex, we would like to use bundled with it git\n\n        Thus we will store _GIT_PATH a path to git in the same directory as annex\n        if found.  If it is empty (but not None), we do nothing\n        \"\"\"\n        if GitRunner._GIT_PATH is None:\n            from distutils.spawn import find_executable\n            # with all the nesting of config and this runner, cannot use our\n            # cfg here, so will resort to dark magic of environment options\n            if (os.environ.get('DATALAD_USE_DEFAULT_GIT', '0').lower()\n                    in ('1', 'on', 'true', 'yes')):\n                git_fpath = find_executable(\"git\")\n                if git_fpath:\n                    GitRunner._GIT_PATH = ''\n                    lgr.log(9, \"Will use default git %s\", git_fpath)\n                    return  # we are done - there is a default git avail.\n                # if not -- we will look for a bundled one\n            GitRunner._GIT_PATH = GitRunner._get_bundled_path()\n            lgr.log(9, \"Will use git under %r (no adjustments to PATH if empty \"\n                       \"string)\", GitRunner._GIT_PATH)\n            assert(GitRunner._GIT_PATH is not None)  # we made the decision!\n\n    @staticmethod\n    def _get_bundled_path():\n        from distutils.spawn import find_executable\n        annex_fpath = find_executable(\"git-annex\")\n        if not annex_fpath:\n            # not sure how to live further anyways! ;)\n            alongside = False\n        else:\n            annex_path = op.dirname(op.realpath(annex_fpath))\n            alongside = op.lexists(op.join(annex_path, 'git'))\n        return annex_path if alongside else ''\n\n    @staticmethod\n    def get_git_environ_adjusted(env=None):\n        \"\"\"\n        Replaces GIT_DIR and GIT_WORK_TREE with absolute paths if relative path and defined\n        \"\"\"\n        # if env set copy else get os environment\n        git_env = env.copy() if env else os.environ.copy()\n        if GitRunner._GIT_PATH:\n            git_env['PATH'] = op.pathsep.join([GitRunner._GIT_PATH, git_env['PATH']]) \\\n                if 'PATH' in git_env \\\n                else GitRunner._GIT_PATH\n\n        for varstring in ['GIT_DIR', 'GIT_WORK_TREE']:\n            var = git_env.get(varstring)\n            if var:                                    # if env variable set\n                if not op.isabs(var):                   # and it's a relative path\n                    git_env[varstring] = op.abspath(var)  # to absolute path\n                    lgr.log(9, \"Updated %s to %s\", varstring, git_env[varstring])\n\n        if 'GIT_SSH_COMMAND' not in git_env:\n            git_env['GIT_SSH_COMMAND'] = GIT_SSH_COMMAND\n            git_env['GIT_SSH_VARIANT'] = 'ssh'\n\n        return git_env\n\n    def run(self, cmd, env=None, *args, **kwargs):\n        out, err = super(GitRunner, self).run(\n            cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)\n        # All communication here will be returned as unicode\n        # TODO: do that instead within the super's run!\n        return assure_unicode(out), assure_unicode(err)\n\n\ndef readline_rstripped(stdout):\n    #return iter(stdout.readline, b'').next().rstrip()\n    return stdout.readline().rstrip()\n\n\nclass SafeDelCloseMixin(object):\n    \"\"\"A helper class to use where __del__ would call .close() which might\n    fail if \"too late in GC game\"\n    \"\"\"\n    def __del__(self):\n        try:\n            self.close()\n        except TypeError:\n            if os.fdopen is None or lgr.debug is None:\n                # if we are late in the game and things already gc'ed in py3,\n                # it is Ok\n                return\n            raise\n\n\n@auto_repr\nclass BatchedCommand(SafeDelCloseMixin):\n    \"\"\"Container for a process which would allow for persistent communication\n    \"\"\"\n\n    def __init__(self, cmd, path=None, output_proc=None):\n        if not isinstance(cmd, list):\n            cmd = [cmd]\n        self.cmd = cmd\n        self.path = path\n        self.output_proc = output_proc if output_proc else readline_rstripped\n        self._process = None\n        self._stderr_out = None\n        self._stderr_out_fname = None\n\n    def _initialize(self):\n        lgr.debug(\"Initiating a new process for %s\" % repr(self))\n        lgr.log(5, \"Command: %s\" % self.cmd)\n        # according to the internet wisdom there is no easy way with subprocess\n        # while avoid deadlocks etc.  We would need to start a thread/subprocess\n        # to timeout etc\n        # kwargs = dict(bufsize=1, universal_newlines=True) if PY3 else {}\n        self._stderr_out, self._stderr_out_fname = tempfile.mkstemp()\n        self._process = subprocess.Popen(\n            self.cmd,\n            stdin=subprocess.PIPE,\n            stdout=subprocess.PIPE,\n            stderr=self._stderr_out,\n            env=GitRunner.get_git_environ_adjusted(),\n            cwd=self.path,\n            bufsize=1,\n            universal_newlines=True  # **kwargs\n        )\n\n    def _check_process(self, restart=False):\n        \"\"\"Check if the process was terminated and restart if restart\n\n        Returns\n        -------\n        bool\n          True if process was alive.\n        str\n          stderr if any recorded if was terminated\n        \"\"\"\n        process = self._process\n        ret = True\n        ret_stderr = None\n        if process and process.poll():\n            lgr.warning(\"Process %s was terminated with returncode %s\" % (process, process.returncode))\n            ret_stderr = self.close(return_stderr=True)\n            ret = False\n        if self._process is None and restart:\n            lgr.warning(\"Restarting the process due to previous failure\")\n            self._initialize()\n        return ret, ret_stderr\n\n    def __call__(self, cmds):\n        \"\"\"\n\n        Parameters\n        ----------\n        cmds : str or tuple or list of (str or tuple)\n\n        Returns\n        -------\n        str or list\n          Output received from process.  list in case if cmds was a list\n        \"\"\"\n        input_multiple = isinstance(cmds, list)\n        if not input_multiple:\n            cmds = [cmds]\n\n        output = [o for o in self.yield_(cmds)]\n        return output if input_multiple else output[0]\n\n    def yield_(self, cmds):\n        \"\"\"Same as __call__, but requires `cmds` to be an iterable\n\n        and yields results for each item.\"\"\"\n        for entry in cmds:\n            if not isinstance(entry, str):\n                entry = ' '.join(entry)\n            yield self.proc1(entry)\n\n    def proc1(self, arg):\n        \"\"\"Same as __call__, but only takes a single command argument\n\n        and returns a single result.\n        \"\"\"\n        # TODO: add checks -- may be process died off and needs to be reinitiated\n        if not self._process:\n            self._initialize()\n\n        entry = arg + '\\n'\n        lgr.log(5, \"Sending %r to batched command %s\" % (entry, self))\n        # apparently communicate is just a one time show\n        # stdout, stderr = self._process.communicate(entry)\n        # according to the internet wisdom there is no easy way with subprocess\n        self._check_process(restart=True)\n        process = self._process  # _check_process might have restarted it\n        process.stdin.write(entry)\n        process.stdin.flush()\n        lgr.log(5, \"Done sending.\")\n        still_alive, stderr = self._check_process(restart=False)\n        # TODO: we might want to handle still_alive, e.g. to allow for\n        #       a number of restarts/resends, but it should be per command\n        #       since for some we cannot just resend the same query. But if\n        #       it is just a \"get\"er - we could resend it few times\n        # The default output_proc expects a single line output.\n        # TODO: timeouts etc\n        stdout = assure_unicode(self.output_proc(process.stdout)) \\\n            if not process.stdout.closed else None\n        if stderr:\n            lgr.warning(\"Received output in stderr: %r\", stderr)\n        lgr.log(5, \"Received output: %r\" % stdout)\n        return stdout\n\n    def close(self, return_stderr=False):\n        \"\"\"Close communication and wait for process to terminate\n\n        Returns\n        -------\n        str\n          stderr output if return_stderr and stderr file was there.\n          None otherwise\n        \"\"\"\n        ret = None\n        if self._stderr_out:\n            # close possibly still open fd\n            os.fdopen(self._stderr_out).close()\n            self._stderr_out = None\n        if self._process:\n            process = self._process\n            lgr.debug(\n                \"Closing stdin of %s and waiting process to finish\", process)\n            process.stdin.close()\n            process.stdout.close()\n            process.wait()\n            self._process = None\n            lgr.debug(\"Process %s has finished\", process)\n        if self._stderr_out_fname and os.path.exists(self._stderr_out_fname):\n            if return_stderr:\n                with open(self._stderr_out_fname, 'r') as f:\n                    ret = f.read()\n            # remove the file where we kept dumping stderr\n            unlink(self._stderr_out_fname)\n            self._stderr_out_fname = None\n        return ret\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\nWrapper for command and function calls, allowing for dry runs and output handling\n\n\"\"\"\n\nimport time\nimport subprocess\nimport sys\nimport logging\nimport os\nimport atexit\nimport functools\nimport tempfile\n\nfrom collections import OrderedDict\nfrom .support import path as op\nfrom .consts import GIT_SSH_COMMAND\nfrom .dochelpers import exc_str\nfrom .support.exceptions import CommandError\nfrom .support.protocol import (\n    NullProtocol,\n    ExecutionTimeProtocol,\n    ExecutionTimeExternalsProtocol,\n)\nfrom .utils import (\n    on_windows,\n    get_tempfile_kwargs,\n    assure_unicode,\n    assure_bytes,\n    unlink,\n    auto_repr,\n    split_cmdline,\n)\nfrom .dochelpers import borrowdoc\n\nlgr = logging.getLogger('datalad.cmd')\n\n# In python3 to split byte stream on newline, it must be bytes\nlinesep_bytes = os.linesep.encode()\n\n_TEMP_std = sys.stdout, sys.stderr\n# To be used in the temp file name to distinguish the ones we create\n# in Runner so we take care about their removal, in contrast to those\n# which might be created outside and passed into Runner\n_MAGICAL_OUTPUT_MARKER = \"_runneroutput_\"\n\nfrom io import IOBase as file_class\n\n\ndef _decide_to_log(v):\n    \"\"\"Hacky workaround for now so we could specify per each which to\n    log online and which to the log\"\"\"\n    if isinstance(v, bool) or callable(v):\n        return v\n    elif v in {'online'}:\n        return True\n    elif v in {'offline'}:\n        return False\n    else:\n        raise ValueError(\"can be bool, callable, 'online' or 'offline'\")\n\n\ndef _get_output_stream(log_std, false_value):\n    \"\"\"Helper to prepare output stream for Popen and use file for 'offline'\n\n    Necessary to avoid lockdowns when both stdout and stderr are pipes\n    \"\"\"\n    if log_std:\n        if log_std == 'offline':\n            # we will open a temporary file\n\n            tf = tempfile.mktemp(\n                **get_tempfile_kwargs({}, prefix=_MAGICAL_OUTPUT_MARKER)\n            )\n            return open(tf, 'w')  # XXX PY3 should be 'b' may be?\n        else:\n            return subprocess.PIPE\n    else:\n        return false_value\n\n\ndef _cleanup_output(stream, std):\n    if isinstance(stream, file_class) and \\\n        _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):\n        if not stream.closed:\n            stream.close()\n        if op.exists(stream.name):\n            unlink(stream.name)\n    elif stream == subprocess.PIPE:\n        std.close()\n\n\ndef kill_output(output):\n    \"\"\"Helper for WitlessRunner to swallow all output and neither\n    relay it to the parent process's stdout, nor provide it as the\n    return value of WitlessRunner.run().\n    \"\"\"\n    return b'', 0\n\n\ndef capture_output(output):\n    \"\"\"Helper for WitlessRunner to capture all output and\n    provide it as the return value of WitlessRunner.run().\n    \"\"\"\n    return output, 0\n\n\nclass WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_latency=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval. When the effective polling frequency is\n          too low to keep the output buffers below their maximum size,\n          a process will deadlock.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_latency)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out\n\n\nclass Runner(WitlessRunner):\n    \"\"\"Provides a wrapper for calling functions and commands.\n\n    An object of this class provides a methods that calls shell commands or\n    python functions, allowing for protocolling the calls and output handling.\n\n    Outputs (stdout and stderr) can be either logged or streamed to system's\n    stdout/stderr during execution.\n    This can be enabled or disabled for both of them independently.\n    Additionally, a protocol object can be a used with the Runner. Such a\n    protocol has to implement datalad.support.protocol.ProtocolInterface, is\n    able to record calls and allows for dry runs.\n    \"\"\"\n\n    __slots__ = ['commands', 'dry', 'cwd', 'env', 'protocol',\n                 '_log_opts']\n\n    def __init__(self, cwd=None, env=None, protocol=None, log_outputs=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd: string, optional\n             Base current working directory for commands.  Could be overridden\n             per run call via cwd option\n        env: dict, optional\n             Custom environment to use for calls. Could be overridden per run\n             call via env option\n        protocol: ProtocolInterface\n             Protocol object to write to.\n        log_outputs : bool, optional\n             Switch to instruct whether outputs should be logged or not.  If not\n             set (default), config 'datalad.log.outputs' would be consulted\n        \"\"\"\n        super(Runner, self).__init__(cwd=cwd, env=env)\n\n        if protocol is None:\n            # TODO: config cmd.protocol = null\n            protocol_str = os.environ.get('DATALAD_CMD_PROTOCOL', 'null')\n            protocol = {\n                'externals-time': ExecutionTimeExternalsProtocol,\n                'time': ExecutionTimeProtocol,\n                'null': NullProtocol\n            }[protocol_str]()\n            if protocol_str != 'null':\n                # we need to dump it into a file at the end\n                # TODO: config cmd.protocol_prefix = protocol\n                filename = '%s-%s.log' % (\n                    os.environ.get('DATALAD_CMD_PROTOCOL_PREFIX', 'protocol'),\n                    id(self)\n                )\n                atexit.register(functools.partial(protocol.write_to_file, filename))\n\n        self.protocol = protocol\n        # Various options for logging\n        self._log_opts = {}\n        # we don't know yet whether we need to log every output or not\n        if log_outputs is not None:\n            self._log_opts['outputs'] = log_outputs\n\n    def __call__(self, cmd, *args, **kwargs):\n        \"\"\"Convenience method\n\n        This will call run() or call() depending on the kind of `cmd`.\n        If `cmd` is a string it is interpreted as the to be executed command.\n        Otherwise it is expected to be a callable.\n        Any other argument is passed to the respective method.\n\n        Parameters\n        ----------\n        cmd: str or callable\n           command string to be executed via shell or callable to be called.\n\n        `*args`:\n        `**kwargs`:\n           see Runner.run() and Runner.call() for available arguments.\n\n        Raises\n        ------\n        TypeError\n          if cmd is neither a string nor a callable.\n        \"\"\"\n\n        if isinstance(cmd, str) or isinstance(cmd, list):\n            return self.run(cmd, *args, **kwargs)\n        elif callable(cmd):\n            return self.call(cmd, *args, **kwargs)\n        else:\n            raise TypeError(\"Argument 'command' is neither a string, \"\n                            \"nor a list nor a callable.\")\n\n    def _opt_env_adapter(v):\n        \"\"\"If value is a string, split by ,\"\"\"\n        if v:\n            if v.isdigit():\n                log_env = bool(int(v))\n            else:\n                log_env = v.split(',')\n            return log_env\n        else:\n            return False\n\n    _LOG_OPTS_ADAPTERS = OrderedDict([\n        ('outputs', None),\n        ('cwd', None),\n        ('env', _opt_env_adapter),\n        ('stdin', None),\n    ])\n\n    def _get_log_setting(self, opt, default=False):\n        try:\n            return self._log_opts[opt]\n        except KeyError:\n            try:\n                from . import cfg\n            except ImportError:\n                return default\n            adapter = self._LOG_OPTS_ADAPTERS.get(opt, None)\n            self._log_opts[opt] = \\\n                (cfg.getbool if not adapter else cfg.get_value)(\n                    'datalad.log.cmd', opt, default=default)\n            if adapter:\n                self._log_opts[opt] = adapter(self._log_opts[opt])\n            return self._log_opts[opt]\n\n    @property\n    def log_outputs(self):\n        return self._get_log_setting('outputs')\n\n    @property\n    def log_cwd(self):\n        return self._get_log_setting('cwd')\n\n    @property\n    def log_stdin(self):\n        return self._get_log_setting('stdin')\n\n    @property\n    def log_env(self):\n        return self._get_log_setting('env')\n\n    # Two helpers to encapsulate formatting/output\n    def _log_out(self, line):\n        if line and self.log_outputs:\n            self.log(\"stdout| \" + line.rstrip('\\n'))\n\n    def _log_err(self, line, expected=False):\n        if line and self.log_outputs:\n            self.log(\"stderr| \" + line.rstrip('\\n'),\n                     level={True: 9,\n                            False: 11}[expected])\n\n    def _get_output_online(self, proc,\n                           log_stdout, log_stderr,\n                           outputstream, errstream,\n                           expect_stderr=False, expect_fail=False):\n        \"\"\"\n\n        If log_stdout or log_stderr are callables, they will be given a read\n        line to be processed, and return processed result.  So if they need to\n        'swallow' the line from being logged, should just return None\n\n        Parameters\n        ----------\n        proc\n        log_stdout: bool or callable or 'online' or 'offline'\n        log_stderr: : bool or callable or 'online' or 'offline'\n          If any of those 'offline', we would call proc.communicate at the\n          end to grab possibly outstanding output from it\n        expect_stderr\n        expect_fail\n\n        Returns\n        -------\n\n        \"\"\"\n        stdout, stderr = bytes(), bytes()\n\n        log_stdout_ = _decide_to_log(log_stdout)\n        log_stderr_ = _decide_to_log(log_stderr)\n        log_stdout_is_callable = callable(log_stdout_)\n        log_stderr_is_callable = callable(log_stderr_)\n\n        # arguments to be passed into _process_one_line\n        stdout_args = (\n                'stdout',\n                proc, log_stdout_, log_stdout_is_callable\n        )\n        stderr_args = (\n                'stderr',\n                proc, log_stderr_, log_stderr_is_callable,\n                expect_stderr or expect_fail\n        )\n\n        while proc.poll() is None:\n            # see for a possibly useful approach to processing output\n            # in another thread http://codereview.stackexchange.com/a/17959\n            # current problem is that if there is no output on stderr\n            # it stalls\n            # Monitor if anything was output and if nothing, sleep a bit\n            stdout_, stderr_ = None, None\n            if log_stdout_:\n                stdout_ = self._process_one_line(*stdout_args)\n                stdout += stdout_\n            if log_stderr_:\n                stderr_ = self._process_one_line(*stderr_args)\n                stderr += stderr_\n            if stdout_ is None and stderr_ is None:\n                # no output was really produced, so sleep a tiny bit\n                time.sleep(0.001)\n\n        # Handle possible remaining output\n        if log_stdout_ and log_stderr_:\n            # If Popen was called with more than two pipes, calling\n            # communicate() after we partially read the stream will return\n            # empty output.\n            stdout += self._process_remaining_output(\n                outputstream, proc.stdout.read(), *stdout_args)\n            stderr += self._process_remaining_output(\n                errstream, proc.stderr.read(), *stderr_args)\n        stdout_, stderr_ = proc.communicate()\n        # ??? should we condition it on log_stdout in {'offline'} ???\n        stdout += self._process_remaining_output(outputstream, stdout_, *stdout_args)\n        stderr += self._process_remaining_output(errstream, stderr_, *stderr_args)\n\n        return stdout, stderr\n\n    def _process_remaining_output(self, stream, out_, *pargs):\n        \"\"\"Helper to process output which might have been obtained from popen or\n        should be loaded from file\"\"\"\n        out = bytes()\n        if isinstance(stream, file_class) and \\\n                _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):\n            assert out_ is None, \"should have gone into a file\"\n            if not stream.closed:\n                stream.close()\n            with open(stream.name, 'rb') as f:\n                for line in f:\n                    out += self._process_one_line(*pargs, line=line)\n        else:\n            if out_:\n                # resolving a once in a while failing test #2185\n                if isinstance(out_, str):\n                    out_ = out_.encode('utf-8')\n                for line in out_.split(linesep_bytes):\n                    out += self._process_one_line(\n                        *pargs, line=line, suf=linesep_bytes)\n        return out\n\n    def _process_one_line(self, out_type, proc, log_, log_is_callable,\n                          expected=False, line=None, suf=None):\n        if line is None:\n            lgr.log(3, \"Reading line from %s\", out_type)\n            line = {'stdout': proc.stdout, 'stderr': proc.stderr}[out_type].readline()\n        else:\n            lgr.log(3, \"Processing provided line\")\n        if line and log_is_callable:\n            # Let it be processed\n            line = log_(assure_unicode(line))\n            if line is not None:\n                # we are working with binary type here\n                line = assure_bytes(line)\n        if line:\n            if out_type == 'stdout':\n                self._log_out(assure_unicode(line))\n            elif out_type == 'stderr':\n                self._log_err(line.decode('utf-8'),\n                              expected)\n            else:  # pragma: no cover\n                raise RuntimeError(\"must not get here\")\n            return (line + suf) if suf else line\n        # it was output already directly but for code to work, return \"\"\n        return bytes()\n\n    def run(self, cmd, log_stdout=True, log_stderr=True, log_online=False,\n            expect_stderr=False, expect_fail=False,\n            cwd=None, env=None, shell=None, stdin=None):\n        \"\"\"Runs the command `cmd` using shell.\n\n        In case of dry-mode `cmd` is just added to `commands` and it is\n        actually executed otherwise.\n        Allows for separately logging stdout and stderr  or streaming it to\n        system's stdout or stderr respectively.\n\n        Note: Using a string as `cmd` and shell=True allows for piping,\n              multiple commands, etc., but that implies split_cmdline() is not\n              used. This is considered to be a security hazard.\n              So be careful with input.\n\n        Parameters\n        ----------\n        cmd : str, list\n          String (or list) defining the command call.  No shell is used if cmd\n          is specified as a list\n\n        log_stdout: bool, optional\n            If True, stdout is logged. Goes to sys.stdout otherwise.\n\n        log_stderr: bool, optional\n            If True, stderr is logged. Goes to sys.stderr otherwise.\n\n        log_online: bool, optional\n            Whether to log as output comes in.  Setting to True is preferable\n            for running user-invoked actions to provide timely output\n\n        expect_stderr: bool, optional\n            Normally, having stderr output is a signal of a problem and thus it\n            gets logged at level 11.  But some utilities, e.g. wget, use\n            stderr for their progress output.  Whenever such output is expected,\n            set it to True and output will be logged at level 9 unless\n            exit status is non-0 (in non-online mode only, in online -- would\n            log at 9)\n\n        expect_fail: bool, optional\n            Normally, if command exits with non-0 status, it is considered an\n            error and logged at level 11 (above DEBUG). But if the call intended\n            for checking routine, such messages are usually not needed, thus\n            it will be logged at level 9.\n\n        cwd : string, optional\n            Directory under which run the command (passed to Popen)\n\n        env : string, optional\n            Custom environment to pass\n\n        shell: bool, optional\n            Run command in a shell.  If not specified, then it runs in a shell\n            only if command is specified as a string (not a list)\n\n        stdin: file descriptor\n            input stream to connect to stdin of the process.\n\n        Returns\n        -------\n        (stdout, stderr) - bytes!\n\n        Raises\n        ------\n        CommandError\n           if command's exitcode wasn't 0 or None. exitcode is passed to\n           CommandError's `code`-field. Command's stdout and stderr are stored\n           in CommandError's `stdout` and `stderr` fields respectively.\n        \"\"\"\n        outputstream = _get_output_stream(log_stdout, sys.stdout)\n        errstream = _get_output_stream(log_stderr, sys.stderr)\n\n        popen_env = env or self.env\n        popen_cwd = cwd or self.cwd\n\n        if popen_cwd and popen_env and 'PWD' in popen_env:\n            # we must have inherited PWD, but cwd was provided, so we must\n            # adjust it\n            popen_env = popen_env.copy()  # to avoid side-effects\n            popen_env['PWD'] = popen_cwd\n\n        # TODO: if outputstream is sys.stdout and that one is set to StringIO\n        #       we have to \"shim\" it with something providing fileno().\n        # This happens when we do not swallow outputs, while allowing nosetest's\n        # StringIO to be provided as stdout, crashing the Popen requiring\n        # fileno().  In out swallow_outputs, we just use temporary files\n        # to overcome this problem.\n        # For now necessary test code should be wrapped into swallow_outputs cm\n        # to avoid the problem\n        log_msgs = [\"Running: %s\"]\n        log_args = [cmd]\n        if self.log_cwd:\n            log_msgs += ['cwd=%r']\n            log_args += [popen_cwd]\n        if self.log_stdin:\n            log_msgs += ['stdin=%r']\n            log_args += [stdin]\n        log_env = self.log_env\n        if log_env and popen_env:\n            log_msgs += [\"env=%r\"]\n            log_args.append(\n                popen_env if log_env is True\n                else {k: popen_env[k] for k in log_env if k in popen_env}\n            )\n        log_msg = '\\n'.join(log_msgs)\n        self.log(log_msg, *log_args)\n\n        if self.protocol.do_execute_ext_commands:\n\n            if shell is None:\n                shell = isinstance(cmd, str)\n\n            if self.protocol.records_ext_commands:\n                prot_exc = None\n                prot_id = self.protocol.start_section(\n                    split_cmdline(cmd)\n                    if isinstance(cmd, str)\n                    else cmd)\n            try:\n                proc = subprocess.Popen(cmd,\n                                        stdout=outputstream,\n                                        stderr=errstream,\n                                        shell=shell,\n                                        cwd=popen_cwd,\n                                        env=popen_env,\n                                        stdin=stdin)\n\n            except Exception as e:\n                prot_exc = e\n                lgr.log(11, \"Failed to start %r%r: %s\" %\n                        (cmd, \" under %r\" % cwd if cwd else '', exc_str(e)))\n                raise\n\n            finally:\n                if self.protocol.records_ext_commands:\n                    self.protocol.end_section(prot_id, prot_exc)\n\n            try:\n                if log_online:\n                    out = self._get_output_online(proc,\n                                                  log_stdout, log_stderr,\n                                                  outputstream, errstream,\n                                                  expect_stderr=expect_stderr,\n                                                  expect_fail=expect_fail)\n                else:\n                    out = proc.communicate()\n\n                # Decoding was delayed to this point\n                def decode_if_not_None(x):\n                    return \"\" if x is None else bytes.decode(x)\n                out = tuple(map(decode_if_not_None, out))\n\n                status = proc.poll()\n\n                # needs to be done after we know status\n                if not log_online:\n                    self._log_out(out[0])\n                    if status not in [0, None]:\n                        self._log_err(out[1], expected=expect_fail)\n                    else:\n                        # as directed\n                        self._log_err(out[1], expected=expect_stderr)\n\n                if status not in [0, None]:\n                    msg = \"Failed to run %r%s. Exit code=%d.%s%s\" \\\n                        % (cmd, \" under %r\" % (popen_cwd), status,\n                           \"\" if log_online else \" out=%s\" % out[0],\n                           \"\" if log_online else \" err=%s\" % out[1])\n                    lgr.log(9 if expect_fail else 11, msg)\n                    raise CommandError(str(cmd), msg, status, out[0], out[1])\n                else:\n                    self.log(\"Finished running %r with status %s\" % (cmd, status),\n                             level=8)\n\n            except CommandError:\n                # do not bother with reacting to \"regular\" CommandError\n                # exceptions.  Somehow if we also terminate here for them\n                # some processes elsewhere might stall:\n                # see https://github.com/datalad/datalad/pull/3794\n                raise\n\n            except BaseException as exc:\n                exc_info = sys.exc_info()\n                # KeyboardInterrupt is subclass of BaseException\n                lgr.debug(\"Terminating process for %s upon exception: %s\",\n                          cmd, exc_str(exc))\n                try:\n                    # there are still possible (although unlikely) cases when\n                    # we fail to interrupt but we\n                    # should not crash if we fail to terminate the process\n                    proc.terminate()\n                except BaseException as exc2:\n                    lgr.warning(\"Failed to terminate process for %s: %s\",\n                                cmd, exc_str(exc2))\n                raise exc_info[1]\n\n            finally:\n                # Those streams are for us to close if we asked for a PIPE\n                # TODO -- assure closing the files\n                _cleanup_output(outputstream, proc.stdout)\n                _cleanup_output(errstream, proc.stderr)\n\n        else:\n            if self.protocol.records_ext_commands:\n                self.protocol.add_section(split_cmdline(cmd)\n                                          if isinstance(cmd, str)\n                                          else cmd, None)\n            out = (\"DRY\", \"DRY\")\n\n        return out\n\n    def call(self, f, *args, **kwargs):\n        \"\"\"Helper to unify collection of logging all \"dry\" actions.\n\n        Calls `f` if `Runner`-object is not in dry-mode. Adds `f` along with\n        its arguments to `commands` otherwise.\n\n        Parameters\n        ----------\n        f: callable\n        \"\"\"\n        if self.protocol.do_execute_callables:\n            if self.protocol.records_callables:\n                prot_exc = None\n                prot_id = self.protocol.start_section(\n                    [str(f), \"args=%s\" % str(args), \"kwargs=%s\" % str(kwargs)])\n\n            try:\n                return f(*args, **kwargs)\n            except Exception as e:\n                prot_exc = e\n                raise\n            finally:\n                if self.protocol.records_callables:\n                    self.protocol.end_section(prot_id, prot_exc)\n        else:\n            if self.protocol.records_callables:\n                self.protocol.add_section(\n                    [str(f), \"args=%s\" % str(args), \"kwargs=%s\" % str(kwargs)],\n                    None)\n\n    def log(self, msg, *args, **kwargs):\n        \"\"\"log helper\n\n        Logs at level 9 by default and adds \"Protocol:\"-prefix in order to\n        log the used protocol.\n        \"\"\"\n        level = kwargs.pop('level', 9)\n        if isinstance(self.protocol, NullProtocol):\n            lgr.log(level, msg, *args, **kwargs)\n        else:\n            if args:\n                msg = msg % args\n            lgr.log(level, \"{%s} %s\" % (\n                self.protocol.__class__.__name__, msg)\n            )\n\n\nclass GitRunner(Runner):\n    \"\"\"\n    Runner to be used to run git and git annex commands\n\n    Overloads the runner class to check & update GIT_DIR and\n    GIT_WORK_TREE environment variables set to the absolute path\n    if is defined and is relative path\n    \"\"\"\n    _GIT_PATH = None\n\n    @borrowdoc(Runner)\n    def __init__(self, *args, **kwargs):\n        super(GitRunner, self).__init__(*args, **kwargs)\n        self._check_git_path()\n\n    @staticmethod\n    def _check_git_path():\n        \"\"\"If using bundled git-annex, we would like to use bundled with it git\n\n        Thus we will store _GIT_PATH a path to git in the same directory as annex\n        if found.  If it is empty (but not None), we do nothing\n        \"\"\"\n        if GitRunner._GIT_PATH is None:\n            from distutils.spawn import find_executable\n            # with all the nesting of config and this runner, cannot use our\n            # cfg here, so will resort to dark magic of environment options\n            if (os.environ.get('DATALAD_USE_DEFAULT_GIT', '0').lower()\n                    in ('1', 'on', 'true', 'yes')):\n                git_fpath = find_executable(\"git\")\n                if git_fpath:\n                    GitRunner._GIT_PATH = ''\n                    lgr.log(9, \"Will use default git %s\", git_fpath)\n                    return  # we are done - there is a default git avail.\n                # if not -- we will look for a bundled one\n            GitRunner._GIT_PATH = GitRunner._get_bundled_path()\n            lgr.log(9, \"Will use git under %r (no adjustments to PATH if empty \"\n                       \"string)\", GitRunner._GIT_PATH)\n            assert(GitRunner._GIT_PATH is not None)  # we made the decision!\n\n    @staticmethod\n    def _get_bundled_path():\n        from distutils.spawn import find_executable\n        annex_fpath = find_executable(\"git-annex\")\n        if not annex_fpath:\n            # not sure how to live further anyways! ;)\n            alongside = False\n        else:\n            annex_path = op.dirname(op.realpath(annex_fpath))\n            alongside = op.lexists(op.join(annex_path, 'git'))\n        return annex_path if alongside else ''\n\n    @staticmethod\n    def get_git_environ_adjusted(env=None):\n        \"\"\"\n        Replaces GIT_DIR and GIT_WORK_TREE with absolute paths if relative path and defined\n        \"\"\"\n        # if env set copy else get os environment\n        git_env = env.copy() if env else os.environ.copy()\n        if GitRunner._GIT_PATH:\n            git_env['PATH'] = op.pathsep.join([GitRunner._GIT_PATH, git_env['PATH']]) \\\n                if 'PATH' in git_env \\\n                else GitRunner._GIT_PATH\n\n        for varstring in ['GIT_DIR', 'GIT_WORK_TREE']:\n            var = git_env.get(varstring)\n            if var:                                    # if env variable set\n                if not op.isabs(var):                   # and it's a relative path\n                    git_env[varstring] = op.abspath(var)  # to absolute path\n                    lgr.log(9, \"Updated %s to %s\", varstring, git_env[varstring])\n\n        if 'GIT_SSH_COMMAND' not in git_env:\n            git_env['GIT_SSH_COMMAND'] = GIT_SSH_COMMAND\n            git_env['GIT_SSH_VARIANT'] = 'ssh'\n\n        return git_env\n\n    def run(self, cmd, env=None, *args, **kwargs):\n        out, err = super(GitRunner, self).run(\n            cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)\n        # All communication here will be returned as unicode\n        # TODO: do that instead within the super's run!\n        return assure_unicode(out), assure_unicode(err)\n\n\ndef readline_rstripped(stdout):\n    #return iter(stdout.readline, b'').next().rstrip()\n    return stdout.readline().rstrip()\n\n\nclass SafeDelCloseMixin(object):\n    \"\"\"A helper class to use where __del__ would call .close() which might\n    fail if \"too late in GC game\"\n    \"\"\"\n    def __del__(self):\n        try:\n            self.close()\n        except TypeError:\n            if os.fdopen is None or lgr.debug is None:\n                # if we are late in the game and things already gc'ed in py3,\n                # it is Ok\n                return\n            raise\n\n\n@auto_repr\nclass BatchedCommand(SafeDelCloseMixin):\n    \"\"\"Container for a process which would allow for persistent communication\n    \"\"\"\n\n    def __init__(self, cmd, path=None, output_proc=None):\n        if not isinstance(cmd, list):\n            cmd = [cmd]\n        self.cmd = cmd\n        self.path = path\n        self.output_proc = output_proc if output_proc else readline_rstripped\n        self._process = None\n        self._stderr_out = None\n        self._stderr_out_fname = None\n\n    def _initialize(self):\n        lgr.debug(\"Initiating a new process for %s\" % repr(self))\n        lgr.log(5, \"Command: %s\" % self.cmd)\n        # according to the internet wisdom there is no easy way with subprocess\n        # while avoid deadlocks etc.  We would need to start a thread/subprocess\n        # to timeout etc\n        # kwargs = dict(bufsize=1, universal_newlines=True) if PY3 else {}\n        self._stderr_out, self._stderr_out_fname = tempfile.mkstemp()\n        self._process = subprocess.Popen(\n            self.cmd,\n            stdin=subprocess.PIPE,\n            stdout=subprocess.PIPE,\n            stderr=self._stderr_out,\n            env=GitRunner.get_git_environ_adjusted(),\n            cwd=self.path,\n            bufsize=1,\n            universal_newlines=True  # **kwargs\n        )\n\n    def _check_process(self, restart=False):\n        \"\"\"Check if the process was terminated and restart if restart\n\n        Returns\n        -------\n        bool\n          True if process was alive.\n        str\n          stderr if any recorded if was terminated\n        \"\"\"\n        process = self._process\n        ret = True\n        ret_stderr = None\n        if process and process.poll():\n            lgr.warning(\"Process %s was terminated with returncode %s\" % (process, process.returncode))\n            ret_stderr = self.close(return_stderr=True)\n            ret = False\n        if self._process is None and restart:\n            lgr.warning(\"Restarting the process due to previous failure\")\n            self._initialize()\n        return ret, ret_stderr\n\n    def __call__(self, cmds):\n        \"\"\"\n\n        Parameters\n        ----------\n        cmds : str or tuple or list of (str or tuple)\n\n        Returns\n        -------\n        str or list\n          Output received from process.  list in case if cmds was a list\n        \"\"\"\n        input_multiple = isinstance(cmds, list)\n        if not input_multiple:\n            cmds = [cmds]\n\n        output = [o for o in self.yield_(cmds)]\n        return output if input_multiple else output[0]\n\n    def yield_(self, cmds):\n        \"\"\"Same as __call__, but requires `cmds` to be an iterable\n\n        and yields results for each item.\"\"\"\n        for entry in cmds:\n            if not isinstance(entry, str):\n                entry = ' '.join(entry)\n            yield self.proc1(entry)\n\n    def proc1(self, arg):\n        \"\"\"Same as __call__, but only takes a single command argument\n\n        and returns a single result.\n        \"\"\"\n        # TODO: add checks -- may be process died off and needs to be reinitiated\n        if not self._process:\n            self._initialize()\n\n        entry = arg + '\\n'\n        lgr.log(5, \"Sending %r to batched command %s\" % (entry, self))\n        # apparently communicate is just a one time show\n        # stdout, stderr = self._process.communicate(entry)\n        # according to the internet wisdom there is no easy way with subprocess\n        self._check_process(restart=True)\n        process = self._process  # _check_process might have restarted it\n        process.stdin.write(entry)\n        process.stdin.flush()\n        lgr.log(5, \"Done sending.\")\n        still_alive, stderr = self._check_process(restart=False)\n        # TODO: we might want to handle still_alive, e.g. to allow for\n        #       a number of restarts/resends, but it should be per command\n        #       since for some we cannot just resend the same query. But if\n        #       it is just a \"get\"er - we could resend it few times\n        # The default output_proc expects a single line output.\n        # TODO: timeouts etc\n        stdout = assure_unicode(self.output_proc(process.stdout)) \\\n            if not process.stdout.closed else None\n        if stderr:\n            lgr.warning(\"Received output in stderr: %r\", stderr)\n        lgr.log(5, \"Received output: %r\" % stdout)\n        return stdout\n\n    def close(self, return_stderr=False):\n        \"\"\"Close communication and wait for process to terminate\n\n        Returns\n        -------\n        str\n          stderr output if return_stderr and stderr file was there.\n          None otherwise\n        \"\"\"\n        ret = None\n        if self._stderr_out:\n            # close possibly still open fd\n            os.fdopen(self._stderr_out).close()\n            self._stderr_out = None\n        if self._process:\n            process = self._process\n            lgr.debug(\n                \"Closing stdin of %s and waiting process to finish\", process)\n            process.stdin.close()\n            process.stdout.close()\n            process.wait()\n            self._process = None\n            lgr.debug(\"Process %s has finished\", process)\n        if self._stderr_out_fname and os.path.exists(self._stderr_out_fname):\n            if return_stderr:\n                with open(self._stderr_out_fname, 'r') as f:\n                    ret = f.read()\n            # remove the file where we kept dumping stderr\n            unlink(self._stderr_out_fname)\n            self._stderr_out_fname = None\n        return ret\n",
            "header": {
                "file": {
                    "path": "datalad/cmd.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "e14de0983",
                "new_commit_hash": "3fbe70df6",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/cmd.py"
            },
            "plus_file": {
                "path": "b/datalad/cmd.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 144,
                            "length": 7
                        },
                        "new_range": {
                            "start": 144,
                            "length": 7
                        },
                        "section": "class WitlessRunner(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            self.env['PWD'] = self.cwd",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,",
                                "type": "context"
                            },
                            {
                                "content": "            poll_period=0.1):",
                                "type": "deleted"
                            },
                            {
                                "content": "            poll_latency=0.1):",
                                "type": "added"
                            },
                            {
                                "content": "        \"\"\"Execute a command and communicate with it.",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        Parameters",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "class",
                            "name": "WitlessRunner",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 116,
                            "end_lineno": 297,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "method",
                            "name": "WitlessRunner.run",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 146,
                            "end_lineno": 297,
                            "parent": {
                                "file_name": "datalad/cmd.py",
                                "type": "class",
                                "name": "WitlessRunner",
                                "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 116,
                                "end_lineno": 297,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 171,
                            "length": 7
                        },
                        "new_range": {
                            "start": 171,
                            "length": 9
                        },
                        "section": "class WitlessRunner(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        poll_latency : float, optional",
                                "type": "context"
                            },
                            {
                                "content": "          Shortest interval at which the running process is queried for",
                                "type": "context"
                            },
                            {
                                "content": "          output (in seconds). Any potential output processing will increase",
                                "type": "context"
                            },
                            {
                                "content": "          the effective interval.",
                                "type": "deleted"
                            },
                            {
                                "content": "          the effective interval. When the effective polling frequency is",
                                "type": "added"
                            },
                            {
                                "content": "          too low to keep the output buffers below their maximum size,",
                                "type": "added"
                            },
                            {
                                "content": "          a process will deadlock.",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        Returns",
                                "type": "context"
                            },
                            {
                                "content": "        -------",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "class",
                            "name": "WitlessRunner",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 116,
                            "end_lineno": 297,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "method",
                            "name": "WitlessRunner.run",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 146,
                            "end_lineno": 297,
                            "parent": {
                                "file_name": "datalad/cmd.py",
                                "type": "class",
                                "name": "WitlessRunner",
                                "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 116,
                                "end_lineno": 297,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 226,
                            "length": 8
                        },
                        "new_range": {
                            "start": 228,
                            "length": 7
                        },
                        "section": "class WitlessRunner(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            while process.poll() is None or keep_going:",
                                "type": "context"
                            },
                            {
                                "content": "                # one last read?",
                                "type": "context"
                            },
                            {
                                "content": "                keep_going = process.returncode is None",
                                "type": "context"
                            },
                            {
                                "content": "                # get a chunk of output for the specific period",
                                "type": "deleted"
                            },
                            {
                                "content": "                # of time",
                                "type": "deleted"
                            },
                            {
                                "content": "                # get a chunk of output",
                                "type": "added"
                            },
                            {
                                "content": "                pout = [",
                                "type": "context"
                            },
                            {
                                "content": "                    # read whatever is available, should not block",
                                "type": "context"
                            },
                            {
                                "content": "                    o.read1(-1) if p else None",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "class",
                            "name": "WitlessRunner",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 116,
                            "end_lineno": 297,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "method",
                            "name": "WitlessRunner.run",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 146,
                            "end_lineno": 297,
                            "parent": {
                                "file_name": "datalad/cmd.py",
                                "type": "class",
                                "name": "WitlessRunner",
                                "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 116,
                                "end_lineno": 297,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 249,
                            "length": 7
                        },
                        "new_range": {
                            "start": 250,
                            "length": 7
                        },
                        "section": "class WitlessRunner(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                    buffer = unprocessed[i] + o if unprocessed[i] else o",
                                "type": "context"
                            },
                            {
                                "content": "                    # engage output processor",
                                "type": "context"
                            },
                            {
                                "content": "                    unprocessed[i] = _handle_output(buffer, out[i], proc)",
                                "type": "context"
                            },
                            {
                                "content": "                time.sleep(poll_period)",
                                "type": "deleted"
                            },
                            {
                                "content": "                time.sleep(poll_latency)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "            # obtain exit code",
                                "type": "context"
                            },
                            {
                                "content": "            status = process.poll()",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "class",
                            "name": "WitlessRunner",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 116,
                            "end_lineno": 297,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/cmd.py",
                            "type": "method",
                            "name": "WitlessRunner.run",
                            "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 146,
                            "end_lineno": 297,
                            "parent": {
                                "file_name": "datalad/cmd.py",
                                "type": "class",
                                "name": "WitlessRunner",
                                "content": "class WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 116,
                                "end_lineno": 297,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test WitlessRunner\n\"\"\"\n\nimport os\nimport sys\n\nfrom .utils import (\n    ok_,\n    eq_,\n    assert_raises,\n    assert_in,\n    with_tempfile,\n    assert_cwd_unchanged,\n    ok_file_has_content,\n    OBSCURE_FILENAME,\n)\nfrom datalad.cmd import (\n    WitlessRunner as Runner,\n    capture_output,\n)\nfrom datalad.utils import Path\nfrom datalad.support.exceptions import CommandError\n\n\nclass TweakOutput(object):\n    \"\"\"Test helper to twist and turn output from WitlessRunner\"\"\"\n    def __init__(self, rtruncate_nbytes=None, report_truncation=True):\n        self._rtruncate_nbytes = rtruncate_nbytes\n        self._report_truncation = report_truncation\n        self.__enter__()\n\n    def __enter__(self):\n        self.received = []\n        return self\n\n    def __exit__(self, exc_type, exc_value, traceback):\n        pass\n\n    def __call__(self, byts):\n        self.received.append(byts)\n        rtrunc = (-1) * self._rtruncate_nbytes \\\n            if self._rtruncate_nbytes else None\n        byts = byts[:rtrunc]\n        return byts, self._rtruncate_nbytes if self._report_truncation else 0\n\n\ndef py2cmd(code):\n    \"\"\"Helper to invoke some Python code through a cmdline invokation of\n    the Python interpreter.\n\n    This should be more portable in some cases.\n    \"\"\"\n    return [sys.executable, '-c', code]\n\n\n@assert_cwd_unchanged\n@with_tempfile\ndef test_runner(tempfile):\n    runner = Runner()\n    content = 'Testing äöü東 real run'\n    cmd = ['sh', '-c', 'echo %s > %r' % (content, tempfile)]\n    out, err = runner.run(cmd)\n    # no capture of any kind, by default\n    ok_(not out)\n    ok_(not err)\n    ok_file_has_content(tempfile, content, strip=True)\n    os.unlink(tempfile)\n\n\ndef test_runner_stderr_capture():\n    runner = Runner()\n    test_msg = \"stderr-Message\"\n    out, err = runner.run(py2cmd(\n        'import sys; print(%r, file=sys.stderr)' % test_msg),\n        proc_stdout=capture_output,\n        proc_stderr=capture_output,\n    )\n    eq_(err.rstrip(), test_msg)\n    ok_(not out)\n\n\ndef test_runner_stdout_capture():\n    runner = Runner()\n    test_msg = \"stdout-Message\"\n    out, err = runner.run(py2cmd(\n        'import sys; print(%r, file=sys.stdout)' % test_msg),\n        proc_stdout=capture_output,\n        proc_stderr=capture_output,\n    )\n    eq_(out.rstrip(), test_msg)\n    ok_(not err)\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_failure(dir_):\n    runner = Runner()\n    with assert_raises(CommandError) as cme:\n        runner.run(\n            py2cmd('import sys; sys.exit(53)')\n        )\n    eq_(53, cme.exception.code)\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_fix_PWD(path):\n    env = os.environ.copy()\n    env['PWD'] = orig_cwd = os.getcwd()\n    runner = Runner(cwd=path, env=env)\n    out, err = runner.run(\n        py2cmd('import os; print(os.environ[\"PWD\"])'),\n        proc_stdout=capture_output,\n    )\n    eq_(out.strip(), path)  # was fixed up to point to point to cwd's path\n    eq_(env['PWD'], orig_cwd)  # no side-effect\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_cwd_encoding(path):\n    env = os.environ.copy()\n    # Add PWD to env so that runner will temporarily adjust it to point to cwd.\n    env['PWD'] = os.getcwd()\n    cwd = Path(path) / OBSCURE_FILENAME\n    cwd.mkdir()\n    # Running doesn't fail if cwd or env has unicode value.\n    Runner(cwd=cwd, env=env).run(\n        py2cmd(\n            'from pathlib import Path; (Path.cwd() / \"foo\").write_text(\"t\")'))\n    (cwd / 'foo').exists()\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_stdin(path):\n    runner = Runner()\n    fakestdin = Path(path) / 'io'\n    # go for diffcult content\n    fakestdin.write_text(OBSCURE_FILENAME)\n\n    out, err = runner.run(\n        py2cmd('import fileinput; print(fileinput.input().readline())'),\n        stdin=fakestdin.open(),\n        proc_stdout=capture_output,\n    )\n    assert_in(OBSCURE_FILENAME, out)\n\n\npy_9bytes_plus_6bytes = \"\"\"\\\nimport sys\nprint(\"123456789\", end=\"\", file=sys.stdout, flush=True)\nimport time\ntime.sleep(1.5)\nprint(\"abcdefg\", end=\"\", file=sys.stdout, flush=True)\n\"\"\"\n\n\ndef test_runner_incomplete_capture():\n    runner = Runner()\n    with TweakOutput(rtruncate_nbytes=3) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            # make sure the runner polls faster than the output is coming\n            poll_period=0.1)\n    # we must not loose any output, except for the very last three bytes\n    # even though we poll at a higher frequency\n    eq_(out, '123456789abcd')\n    # conditional protect against slow execution\n    # we see the first batch received\n    if len(outproc.received) > 1:\n        eq_(outproc.received[0], b'123456789')\n        # we see the last batch that carries the pending 3 bytes upfront\n        eq_(outproc.received[-1], b'789abcdefg')\n    if len(outproc.received) > 2:\n        # we see the truncated 3 bytes of the first batch, repeatedly\n        # sent (but the processor rejects them)\n        eq_(outproc.received[1:-1], (len(outproc.received) - 2) * [b'789'])\n\n    # no the same, but the processor doesn't tell that it ignored\n    # 3 bytes\n    with TweakOutput(rtruncate_nbytes=3, report_truncation=False) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            poll_period=0.1)\n    # we miss three bytes at the end of each batch\n    if len(outproc.received) > 1:\n        eq_(out, '123456abcd')\n",
            "new_file_content": "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test WitlessRunner\n\"\"\"\n\nimport os\nimport sys\n\nfrom .utils import (\n    ok_,\n    eq_,\n    assert_raises,\n    assert_in,\n    with_tempfile,\n    assert_cwd_unchanged,\n    ok_file_has_content,\n    OBSCURE_FILENAME,\n)\nfrom datalad.cmd import (\n    WitlessRunner as Runner,\n    capture_output,\n)\nfrom datalad.utils import Path\nfrom datalad.support.exceptions import CommandError\n\n\nclass TweakOutput(object):\n    \"\"\"Test helper to twist and turn output from WitlessRunner\"\"\"\n    def __init__(self, rtruncate_nbytes=None, report_truncation=True):\n        self._rtruncate_nbytes = rtruncate_nbytes\n        self._report_truncation = report_truncation\n        self.__enter__()\n\n    def __enter__(self):\n        self.received = []\n        return self\n\n    def __exit__(self, exc_type, exc_value, traceback):\n        pass\n\n    def __call__(self, byts):\n        self.received.append(byts)\n        rtrunc = (-1) * self._rtruncate_nbytes \\\n            if self._rtruncate_nbytes else None\n        byts = byts[:rtrunc]\n        return byts, self._rtruncate_nbytes if self._report_truncation else 0\n\n\ndef py2cmd(code):\n    \"\"\"Helper to invoke some Python code through a cmdline invokation of\n    the Python interpreter.\n\n    This should be more portable in some cases.\n    \"\"\"\n    return [sys.executable, '-c', code]\n\n\n@assert_cwd_unchanged\n@with_tempfile\ndef test_runner(tempfile):\n    runner = Runner()\n    content = 'Testing äöü東 real run'\n    cmd = ['sh', '-c', 'echo %s > %r' % (content, tempfile)]\n    out, err = runner.run(cmd)\n    # no capture of any kind, by default\n    ok_(not out)\n    ok_(not err)\n    ok_file_has_content(tempfile, content, strip=True)\n    os.unlink(tempfile)\n\n\ndef test_runner_stderr_capture():\n    runner = Runner()\n    test_msg = \"stderr-Message\"\n    out, err = runner.run(py2cmd(\n        'import sys; print(%r, file=sys.stderr)' % test_msg),\n        proc_stdout=capture_output,\n        proc_stderr=capture_output,\n    )\n    eq_(err.rstrip(), test_msg)\n    ok_(not out)\n\n\ndef test_runner_stdout_capture():\n    runner = Runner()\n    test_msg = \"stdout-Message\"\n    out, err = runner.run(py2cmd(\n        'import sys; print(%r, file=sys.stdout)' % test_msg),\n        proc_stdout=capture_output,\n        proc_stderr=capture_output,\n    )\n    eq_(out.rstrip(), test_msg)\n    ok_(not err)\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_failure(dir_):\n    runner = Runner()\n    with assert_raises(CommandError) as cme:\n        runner.run(\n            py2cmd('import sys; sys.exit(53)')\n        )\n    eq_(53, cme.exception.code)\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_fix_PWD(path):\n    env = os.environ.copy()\n    env['PWD'] = orig_cwd = os.getcwd()\n    runner = Runner(cwd=path, env=env)\n    out, err = runner.run(\n        py2cmd('import os; print(os.environ[\"PWD\"])'),\n        proc_stdout=capture_output,\n    )\n    eq_(out.strip(), path)  # was fixed up to point to point to cwd's path\n    eq_(env['PWD'], orig_cwd)  # no side-effect\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_cwd_encoding(path):\n    env = os.environ.copy()\n    # Add PWD to env so that runner will temporarily adjust it to point to cwd.\n    env['PWD'] = os.getcwd()\n    cwd = Path(path) / OBSCURE_FILENAME\n    cwd.mkdir()\n    # Running doesn't fail if cwd or env has unicode value.\n    Runner(cwd=cwd, env=env).run(\n        py2cmd(\n            'from pathlib import Path; (Path.cwd() / \"foo\").write_text(\"t\")'))\n    (cwd / 'foo').exists()\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_stdin(path):\n    runner = Runner()\n    fakestdin = Path(path) / 'io'\n    # go for diffcult content\n    fakestdin.write_text(OBSCURE_FILENAME)\n\n    out, err = runner.run(\n        py2cmd('import fileinput; print(fileinput.input().readline())'),\n        stdin=fakestdin.open(),\n        proc_stdout=capture_output,\n    )\n    assert_in(OBSCURE_FILENAME, out)\n\n\npy_9bytes_plus_6bytes = \"\"\"\\\nimport sys\nprint(\"123456789\", end=\"\", file=sys.stdout, flush=True)\nimport time\ntime.sleep(1.5)\nprint(\"abcdefg\", end=\"\", file=sys.stdout, flush=True)\n\"\"\"\n\n\ndef test_runner_incomplete_capture():\n    runner = Runner()\n    with TweakOutput(rtruncate_nbytes=3) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            # make sure the runner polls faster than the output is coming\n            poll_latency=0.1)\n    # we must not loose any output, except for the very last three bytes\n    # even though we poll at a higher frequency\n    eq_(out, '123456789abcd')\n    # conditional protect against slow execution\n    # we see the first batch received\n    if len(outproc.received) > 1:\n        eq_(outproc.received[0], b'123456789')\n        # we see the last batch that carries the pending 3 bytes upfront\n        eq_(outproc.received[-1], b'789abcdefg')\n    if len(outproc.received) > 2:\n        # we see the truncated 3 bytes of the first batch, repeatedly\n        # sent (but the processor rejects them)\n        eq_(outproc.received[1:-1], (len(outproc.received) - 2) * [b'789'])\n\n    # no the same, but the processor doesn't tell that it ignored\n    # 3 bytes\n    with TweakOutput(rtruncate_nbytes=3, report_truncation=False) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            poll_latency=0.1)\n    # we miss three bytes at the end of each batch\n    if len(outproc.received) > 1:\n        eq_(out, '123456abcd')\n",
            "header": {
                "file": {
                    "path": "datalad/tests/test_witless_runner.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "d7533eb67",
                "new_commit_hash": "1f90c69d0",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/tests/test_witless_runner.py"
            },
            "plus_file": {
                "path": "b/datalad/tests/test_witless_runner.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 169,
                            "length": 7
                        },
                        "new_range": {
                            "start": 169,
                            "length": 7
                        },
                        "section": "def test_runner_incomplete_capture():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            # report that to the runner",
                                "type": "context"
                            },
                            {
                                "content": "            proc_stdout=outproc,",
                                "type": "context"
                            },
                            {
                                "content": "            # make sure the runner polls faster than the output is coming",
                                "type": "context"
                            },
                            {
                                "content": "            poll_period=0.1)",
                                "type": "deleted"
                            },
                            {
                                "content": "            poll_latency=0.1)",
                                "type": "added"
                            },
                            {
                                "content": "    # we must not loose any output, except for the very last three bytes",
                                "type": "context"
                            },
                            {
                                "content": "    # even though we poll at a higher frequency",
                                "type": "context"
                            },
                            {
                                "content": "    eq_(out, '123456789abcd')",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/tests/test_witless_runner.py",
                            "type": "function",
                            "name": "test_runner_incomplete_capture",
                            "content": "def test_runner_incomplete_capture():\n    runner = Runner()\n    with TweakOutput(rtruncate_nbytes=3) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            # make sure the runner polls faster than the output is coming\n            poll_period=0.1)\n    # we must not loose any output, except for the very last three bytes\n    # even though we poll at a higher frequency\n    eq_(out, '123456789abcd')\n    # conditional protect against slow execution\n    # we see the first batch received\n    if len(outproc.received) > 1:\n        eq_(outproc.received[0], b'123456789')\n        # we see the last batch that carries the pending 3 bytes upfront\n        eq_(outproc.received[-1], b'789abcdefg')\n    if len(outproc.received) > 2:\n        # we see the truncated 3 bytes of the first batch, repeatedly\n        # sent (but the processor rejects them)\n        eq_(outproc.received[1:-1], (len(outproc.received) - 2) * [b'789'])\n\n    # no the same, but the processor doesn't tell that it ignored\n    # 3 bytes\n    with TweakOutput(rtruncate_nbytes=3, report_truncation=False) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            poll_period=0.1)\n    # we miss three bytes at the end of each batch\n    if len(outproc.received) > 1:\n        eq_(out, '123456abcd')",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 163,
                            "end_lineno": 198,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 192,
                            "length": 7
                        },
                        "new_range": {
                            "start": 192,
                            "length": 7
                        },
                        "section": "def test_runner_incomplete_capture():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            # we don't process the last three in the output, but we",
                                "type": "context"
                            },
                            {
                                "content": "            # report that to the runner",
                                "type": "context"
                            },
                            {
                                "content": "            proc_stdout=outproc,",
                                "type": "context"
                            },
                            {
                                "content": "            poll_period=0.1)",
                                "type": "deleted"
                            },
                            {
                                "content": "            poll_latency=0.1)",
                                "type": "added"
                            },
                            {
                                "content": "    # we miss three bytes at the end of each batch",
                                "type": "context"
                            },
                            {
                                "content": "    if len(outproc.received) > 1:",
                                "type": "context"
                            },
                            {
                                "content": "        eq_(out, '123456abcd')",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/tests/test_witless_runner.py",
                            "type": "function",
                            "name": "test_runner_incomplete_capture",
                            "content": "def test_runner_incomplete_capture():\n    runner = Runner()\n    with TweakOutput(rtruncate_nbytes=3) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            # make sure the runner polls faster than the output is coming\n            poll_period=0.1)\n    # we must not loose any output, except for the very last three bytes\n    # even though we poll at a higher frequency\n    eq_(out, '123456789abcd')\n    # conditional protect against slow execution\n    # we see the first batch received\n    if len(outproc.received) > 1:\n        eq_(outproc.received[0], b'123456789')\n        # we see the last batch that carries the pending 3 bytes upfront\n        eq_(outproc.received[-1], b'789abcdefg')\n    if len(outproc.received) > 2:\n        # we see the truncated 3 bytes of the first batch, repeatedly\n        # sent (but the processor rejects them)\n        eq_(outproc.received[1:-1], (len(outproc.received) - 2) * [b'789'])\n\n    # no the same, but the processor doesn't tell that it ignored\n    # 3 bytes\n    with TweakOutput(rtruncate_nbytes=3, report_truncation=False) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            poll_period=0.1)\n    # we miss three bytes at the end of each batch\n    if len(outproc.received) > 1:\n        eq_(out, '123456abcd')",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 163,
                            "end_lineno": 198,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "32d46b6b7d1fea1653836adfa33a61c653ff6bc7^",
    "new_commit_hash": "32d46b6b7d1fea1653836adfa33a61c653ff6bc7",
    "commit_message": "RF: poll_period -> poll_latency\n\nTo make clearer that no fixed frequency is maintained.",
    "commit_date": "2020-02-05T15:14:12+01:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "32d46b6b7d1fea1653836adfa33a61c653ff6bc7",
    "test_file_codes": [
        "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test WitlessRunner\n\"\"\"\n\nimport os\nimport sys\n\nfrom datalad.tests.utils import (\n    ok_,\n    eq_,\n    assert_raises,\n    assert_in,\n    with_tempfile,\n    assert_cwd_unchanged,\n    ok_file_has_content,\n    OBSCURE_FILENAME,\n)\nfrom datalad.cmd import (\n    WitlessRunner as Runner,\n    capture_output,\n)\nfrom datalad.utils import Path\nfrom datalad.support.exceptions import CommandError\n\n\nclass TweakOutput(object):\n    \"\"\"Test helper to twist and turn output from WitlessRunner\"\"\"\n    def __init__(self, rtruncate_nbytes=None, report_truncation=True):\n        self._rtruncate_nbytes = rtruncate_nbytes\n        self._report_truncation = report_truncation\n        self.__enter__()\n\n    def __enter__(self):\n        self.received = []\n        return self\n\n    def __exit__(self, exc_type, exc_value, traceback):\n        pass\n\n    def __call__(self, byts):\n        self.received.append(byts)\n        rtrunc = (-1) * self._rtruncate_nbytes \\\n            if self._rtruncate_nbytes else None\n        byts = byts[:rtrunc]\n        return byts, self._rtruncate_nbytes if self._report_truncation else 0\n\n\ndef py2cmd(code):\n    \"\"\"Helper to invoke some Python code through a cmdline invokation of\n    the Python interpreter.\n\n    This should be more portable in some cases.\n    \"\"\"\n    return [sys.executable, '-c', code]\n\n\n@assert_cwd_unchanged\n@with_tempfile\ndef test_runner(tempfile):\n    runner = Runner()\n    content = 'Testing äöü東 real run'\n    cmd = ['sh', '-c', 'echo %s > %r' % (content, tempfile)]\n    out, err = runner.run(cmd)\n    # no capture of any kind, by default\n    ok_(not out)\n    ok_(not err)\n    ok_file_has_content(tempfile, content, strip=True)\n    os.unlink(tempfile)\n\n\ndef test_runner_stderr_capture():\n    runner = Runner()\n    test_msg = \"stderr-Message\"\n    out, err = runner.run(py2cmd(\n        'import sys; print(%r, file=sys.stderr)' % test_msg),\n        proc_stdout=capture_output,\n        proc_stderr=capture_output,\n    )\n    eq_(err.rstrip(), test_msg)\n    ok_(not out)\n\n\ndef test_runner_stdout_capture():\n    runner = Runner()\n    test_msg = \"stdout-Message\"\n    out, err = runner.run(py2cmd(\n        'import sys; print(%r, file=sys.stdout)' % test_msg),\n        proc_stdout=capture_output,\n        proc_stderr=capture_output,\n    )\n    eq_(out.rstrip(), test_msg)\n    ok_(not err)\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_failure(dir_):\n    runner = Runner()\n    with assert_raises(CommandError) as cme:\n        runner.run(\n            py2cmd('import sys; sys.exit(53)')\n        )\n    eq_(53, cme.exception.code)\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_fix_PWD(path):\n    env = os.environ.copy()\n    env['PWD'] = orig_cwd = os.getcwd()\n    runner = Runner(cwd=path, env=env)\n    out, err = runner.run(\n        py2cmd('import os; print(os.environ[\"PWD\"])'),\n        proc_stdout=capture_output,\n    )\n    eq_(out.strip(), path)  # was fixed up to point to point to cwd's path\n    eq_(env['PWD'], orig_cwd)  # no side-effect\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_cwd_encoding(path):\n    env = os.environ.copy()\n    # Add PWD to env so that runner will temporarily adjust it to point to cwd.\n    env['PWD'] = os.getcwd()\n    cwd = Path(path) / OBSCURE_FILENAME\n    cwd.mkdir()\n    # Running doesn't fail if cwd or env has unicode value.\n    Runner(cwd=cwd, env=env).run(\n        py2cmd(\n            'from pathlib import Path; (Path.cwd() / \"foo\").write_text(\"t\")'))\n    (cwd / 'foo').exists()\n\n\n@with_tempfile(mkdir=True)\ndef test_runner_stdin(path):\n    runner = Runner()\n    fakestdin = Path(path) / 'io'\n    # go for diffcult content\n    fakestdin.write_text(OBSCURE_FILENAME)\n\n    out, err = runner.run(\n        py2cmd('import fileinput; print(fileinput.input().readline())'),\n        stdin=fakestdin.open(),\n        proc_stdout=capture_output,\n    )\n    assert_in(OBSCURE_FILENAME, out)\n\n\npy_9bytes_plus_6bytes = \"\"\"\\\nimport sys\nprint(\"123456789\", end=\"\", file=sys.stdout, flush=True)\nimport time\ntime.sleep(1.5)\nprint(\"abcdefg\", end=\"\", file=sys.stdout, flush=True)\n\"\"\"\n\n\ndef test_runner_incomplete_capture():\n    runner = Runner()\n    with TweakOutput(rtruncate_nbytes=3) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            # make sure the runner polls faster than the output is coming\n            poll_latency=0.1)\n    # we must not loose any output, except for the very last three bytes\n    # even though we poll at a higher frequency\n    eq_(out, '123456789abcd')\n    # conditional protect against slow execution\n    # we see the first batch received\n    if len(outproc.received) > 1:\n        eq_(outproc.received[0], b'123456789')\n        # we see the last batch that carries the pending 3 bytes upfront\n        eq_(outproc.received[-1], b'789abcdefg')\n    if len(outproc.received) > 2:\n        # we see the truncated 3 bytes of the first batch, repeatedly\n        # sent (but the processor rejects them)\n        eq_(outproc.received[1:-1], (len(outproc.received) - 2) * [b'789'])\n\n    # no the same, but the processor doesn't tell that it ignored\n    # 3 bytes\n    with TweakOutput(rtruncate_nbytes=3, report_truncation=False) as outproc:\n        out, err = runner.run(\n            py2cmd(py_9bytes_plus_6bytes),\n            # we don't process the last three in the output, but we\n            # report that to the runner\n            proc_stdout=outproc,\n            poll_latency=0.1)\n    # we miss three bytes at the end of each batch\n    if len(outproc.received) > 1:\n        eq_(out, '123456abcd')\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.12.2.dev30\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 96ms\nInstalled 25 packages in 6.53s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 65 packages in 4.60s\nPrepared 1 package in 4.60s\nInstalled 59 packages in 453ms\n + appdirs==1.4.4\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.12.2 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + gitdb==4.0.11\n + gitpython==3.1.43\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + jsmin==3.0.1\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.0.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + smmap==5.0.1\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 99 packages in 311ms\nInstalled 32 packages in 427ms\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7\nplugins: cov-6.0.0\ncollected 8 items\n\nr2e_tests/test_1.py E..EFFF.                                             [100%]\n\n==================================== ERRORS ====================================\n________________________ ERROR at setup of test_runner _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py, line 64\n  @assert_cwd_unchanged\n  @with_tempfile\n  def test_runner(tempfile):\nE       fixture 'tempfile' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py:64\n____________________ ERROR at setup of test_runner_failure _____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py, line 102\n  @with_tempfile(mkdir=True)\n  def test_runner_failure(dir_):\nE       fixture 'dir_' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py:102\n=================================== FAILURES ===================================\n_____________________________ test_runner_fix_PWD ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex788x90a2'}\nfilename = '/tmp/datalad_temp_test_runner_fix_PWD6ex1avd0'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_runner_fix_PWD() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n___________________________ test_runner_cwd_encoding ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexf6uuhrj3'}\nfilename = '/tmp/datalad_temp_test_runner_cwd_encodingzzwr9kcs'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_runner_cwd_encoding() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n______________________________ test_runner_stdin _______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex5n9jo2m1'}\nfilename = '/tmp/datalad_temp_test_runner_stdinckq0ekkn'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_runner_stdin() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_runner_stderr_capture\nPASSED r2e_tests/test_1.py::test_runner_stdout_capture\nPASSED r2e_tests/test_1.py::test_runner_incomplete_capture\nERROR r2e_tests/test_1.py::test_runner\nERROR r2e_tests/test_1.py::test_runner_failure\nFAILED r2e_tests/test_1.py::test_runner_fix_PWD - TypeError: test_runner_fix_...\nFAILED r2e_tests/test_1.py::test_runner_cwd_encoding - TypeError: test_runner...\nFAILED r2e_tests/test_1.py::test_runner_stdin - TypeError: test_runner_stdin(...\n==================== 3 failed, 3 passed, 2 errors in 5.40s =====================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7\nplugins: cov-6.0.0\ncollected 8 items\n\nr2e_tests/test_1.py E..EFFFF                                             [100%]\n\n==================================== ERRORS ====================================\n________________________ ERROR at setup of test_runner _________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py, line 64\n  @assert_cwd_unchanged\n  @with_tempfile\n  def test_runner(tempfile):\nE       fixture 'tempfile' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py:64\n____________________ ERROR at setup of test_runner_failure _____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py, line 102\n  @with_tempfile(mkdir=True)\n  def test_runner_failure(dir_):\nE       fixture 'dir_' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_32d46b6b7d1fea1653836adfa33a61c653ff6bc7/r2e_tests/test_1.py:102\n=================================== FAILURES ===================================\n_____________________________ test_runner_fix_PWD ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex__9_t2it'}\nfilename = '/tmp/datalad_temp_test_runner_fix_PWDnlh8u9fm'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_runner_fix_PWD() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n___________________________ test_runner_cwd_encoding ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex96a1r6tn'}\nfilename = '/tmp/datalad_temp_test_runner_cwd_encodingos3x6vgs'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_runner_cwd_encoding() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n______________________________ test_runner_stdin _______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexy_e8je1u'}\nfilename = '/tmp/datalad_temp_test_runner_stdinhcvslnwu'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_runner_stdin() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n________________________ test_runner_incomplete_capture ________________________\n\n    def test_runner_incomplete_capture():\n        runner = Runner()\n        with TweakOutput(rtruncate_nbytes=3) as outproc:\n>           out, err = runner.run(\n                py2cmd(py_9bytes_plus_6bytes),\n                # we don't process the last three in the output, but we\n                # report that to the runner\n                proc_stdout=outproc,\n                # make sure the runner polls faster than the output is coming\n                poll_latency=0.1)\nE           TypeError: run() got an unexpected keyword argument 'poll_latency'\n\nr2e_tests/test_1.py:166: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_runner_stderr_capture\nPASSED r2e_tests/test_1.py::test_runner_stdout_capture\nERROR r2e_tests/test_1.py::test_runner\nERROR r2e_tests/test_1.py::test_runner_failure\nFAILED r2e_tests/test_1.py::test_runner_fix_PWD - TypeError: test_runner_fix_...\nFAILED r2e_tests/test_1.py::test_runner_cwd_encoding - TypeError: test_runner...\nFAILED r2e_tests/test_1.py::test_runner_stdin - TypeError: test_runner_stdin(...\nFAILED r2e_tests/test_1.py::test_runner_incomplete_capture - TypeError: run()...\n==================== 4 failed, 2 passed, 2 errors in 1.00s =====================\n",
    "old_commit_res_stderr": ""
}
datalad/cmd.py datalad/tests/test_witless_runner.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Renaming `poll_period` to `poll_latency` Causes Multiple Values for `path` Argument

**Description:**
After renaming the `poll_period` parameter to `poll_latency` in the `run` method of `WitlessRunner`, certain function calls are resulting in a `TypeError`. Specifically, when attempting to pass `poll_latency` as a keyword argument, the function receives multiple values for the `path` parameter, leading to unexpected errors.

**Example Buggy Code:**
```python
runner.run(cmd, proc_stdout=outproc, poll_latency=0.1)
```

**Error Message:**
```
TypeError: run() got multiple values for argument 'path'
```

**Expected Behavior:**
The `run` method should accept `poll_latency` as a valid keyword argument without causing any conflicts or multiple assignments of other parameters like `path`. Functions calling `run` with `poll_latency` should execute successfully without raising errors related to argument values.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/cmd.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""
Wrapper for command and function calls, allowing for dry runs and output handling

"""

import time
import subprocess
import sys
import logging
import os
import atexit
import functools
import tempfile

from collections import OrderedDict
from .support import path as op
from .consts import GIT_SSH_COMMAND
from .dochelpers import exc_str
from .support.exceptions import CommandError
from .support.protocol import (
    NullProtocol,
    ExecutionTimeProtocol,
    ExecutionTimeExternalsProtocol,
)
from .utils import (
    on_windows,
    get_tempfile_kwargs,
    assure_unicode,
    assure_bytes,
    unlink,
    auto_repr,
    split_cmdline,
)
from .dochelpers import borrowdoc

lgr = logging.getLogger('datalad.cmd')

# In python3 to split byte stream on newline, it must be bytes
linesep_bytes = os.linesep.encode()

_TEMP_std = sys.stdout, sys.stderr
# To be used in the temp file name to distinguish the ones we create
# in Runner so we take care about their removal, in contrast to those
# which might be created outside and passed into Runner
_MAGICAL_OUTPUT_MARKER = "_runneroutput_"

from io import IOBase as file_class


def _decide_to_log(v):
    """Hacky workaround for now so we could specify per each which to
    log online and which to the log"""
    if isinstance(v, bool) or callable(v):
        return v
    elif v in {'online'}:
        return True
    elif v in {'offline'}:
        return False
    else:
        raise ValueError("can be bool, callable, 'online' or 'offline'")


def _get_output_stream(log_std, false_value):
    """Helper to prepare output stream for Popen and use file for 'offline'

    Necessary to avoid lockdowns when both stdout and stderr are pipes
    """
    if log_std:
        if log_std == 'offline':
            # we will open a temporary file

            tf = tempfile.mktemp(
                **get_tempfile_kwargs({}, prefix=_MAGICAL_OUTPUT_MARKER)
            )
            return open(tf, 'w')  # XXX PY3 should be 'b' may be?
        else:
            return subprocess.PIPE
    else:
        return false_value


def _cleanup_output(stream, std):
    if isinstance(stream, file_class) and \
        _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):
        if not stream.closed:
            stream.close()
        if op.exists(stream.name):
            unlink(stream.name)
    elif stream == subprocess.PIPE:
        std.close()


def kill_output(output):
    """Helper for WitlessRunner to swallow all output and neither
    relay it to the parent process's stdout, nor provide it as the
    return value of WitlessRunner.run().
    """
    return b'', 0


def capture_output(output):
    """Helper for WitlessRunner to capture all output and
    provide it as the return value of WitlessRunner.run().
    """
    return output, 0


class WitlessRunner(object):
    """Minimal Runner with support for online command output processing

    It aims to be as simple as possible, providing only essential
    functionality. Derived classes should be used for additional
    specializations and convenience features.
    """
    __slots__ = ['cwd', 'env']

    def __init__(self, cwd=None, env=None):
        """
        Parameters
        ----------
        cwd : path-like, optional
          If given, commands are executed with this path as PWD,
          the PWD of the parent process is used otherwise.
        env : dict, optional
          Environment to be passed to subprocess.Popen(). If `cwd`
          was given, 'PWD' in the environment is set to its value.
          This must be a complete environment definition, no values
          from the current environment will be inherited.
        """
        self.env = env.copy() if env else None
        # stringify to support Path instances on PY35
        self.cwd = str(cwd) if cwd is not None else None
        if self.cwd and env is not None:
            # if CWD was provided, we must not make it conflict with
            # a potential PWD setting
            self.env['PWD'] = self.cwd

    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,
            poll_period=0.1):
        """Execute a command and communicate with it.

        Parameters
        ----------
        cmd : list
          Sequence of program arguments. Passing a single string means
          that it is simply the name of the program, no complex shell
          commands are supported.
        proc_stdout : callable, optional
          By default no stdout is captured, but relayed to the parent
          process's stdout. If given, all stdout is passed as a byte-string
          to this callable, in the chunks it was received by polling the
          processing. The callable may transform it in any way, its output
          (byte-string) is concatenated and provided as stdout return value.
          The helper functions 'kill_output' and 'capture_output' are
          provided to either swallow all output (and not relay it to the
          parent) or to capture all output and provide it as the return
          value.
        proc_stderr : callable, optional
          Like proc_stdout, but for stderr.
        stdin : byte stream, optional
          File descriptor like, used as stdin for the process. Passed
          verbatim to subprocess.Popen().
        poll_latency : float, optional
          Shortest interval at which the running process is queried for
          output (in seconds). Any potential output processing will increase
          the effective interval.

        Returns
        -------
        stdout, stderr
          Unicode string with the cumulative standard output and error
          of the process.

        Raises
        ------
        CommandError
          On execution failure (non-zero exit code) this exception is
          raised which provides the command (cmd), stdout, stderr,
          exit code (status), and a message identifying the failed
          command, as properties.
        """
        proc_out = (proc_stdout, proc_stderr)
        try:
            lgr.log(8, "Start running %r", cmd)
            process = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE if proc_stdout else None,
                stderr=subprocess.PIPE if proc_stderr else None,
                shell=False,
                cwd=self.cwd,
                env=self.env,
                stdin=stdin,
                # intermediate reports are never decoded anyways
                # from PY37 onwards
                #text=False,
                universal_newlines=False,
            )
        except Exception as e:
            lgr.log(11, "Failed to start %r%r: %s" %
                    (cmd,
                     (" under %r" % self.cwd) if self.cwd else '',
                     exc_str(e)))
            raise

        try:
            out = [[], []]
            unprocessed = [None, None]
            # make sure to run this loop at least once, even if the
            # process is already dead
            keep_going = True

            def _handle_output(u, o, p):
                processed, unprocessed_len = proc(u)
                if processed:
                    o.append(processed)
                return u[-(unprocessed_len):] if unprocessed_len else None

            while process.poll() is None or keep_going:
                # one last read?
                keep_going = process.returncode is None
                # get a chunk of output for the specific period
                # of time
                pout = [
                    # read whatever is available, should not block
                    o.read1(-1) if p else None
                    for o, p in zip(
                        (process.stdout, process.stderr),
                        proc_out)
                ]

                for i, (o, proc) in enumerate(zip(pout, proc_out)):
                    if not o:
                        # nothing read
                        continue
                    if proc is None:
                        # no index update needed, can never change, hence no unprocessed
                        # output either
                        out[i].append(o)
                        continue
                    # make sure to feed back any unprocessed stuff
                    buffer = unprocessed[i] + o if unprocessed[i] else o
                    # engage output processor
                    unprocessed[i] = _handle_output(buffer, out[i], proc)
                time.sleep(poll_period)

            # obtain exit code
            status = process.poll()

            # decode bytes to string
            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)

            if status not in [0, None]:
                msg = "Failed to run %r%s." % (
                    cmd,
                    (" under %r" % self.cwd) if self.cwd else '',
                )
                raise CommandError(
                    cmd=str(cmd),
                    msg=msg,
                    code=status,
                    stdout=out[0],
                    stderr=out[1],
                )
            else:
                lgr.log(8, "Finished running %r with status %s", cmd, status)

        except CommandError:
            # do not bother with reacting to "regular" CommandError
            # exceptions.  Somehow if we also terminate here for them
            # some processes elsewhere might stall:
            # see https://github.com/datalad/datalad/pull/3794
            raise

        except BaseException as exc:
            exc_info = sys.exc_info()
            # KeyboardInterrupt is subclass of BaseException
            lgr.debug("Terminating process for %s upon exception: %s",
                      cmd, exc_str(exc))
            try:
                # there are still possible (although unlikely) cases when
                # we fail to interrupt but we
                # should not crash if we fail to terminate the process
                process.terminate()
            except BaseException as exc2:
                lgr.warning("Failed to terminate process for %s: %s",
                            cmd, exc_str(exc2))
            raise exc_info[1]

        return out


class Runner(WitlessRunner):
    """Provides a wrapper for calling functions and commands.

    An object of this class provides a methods that calls shell commands or
    python functions, allowing for protocolling the calls and output handling.

    Outputs (stdout and stderr) can be either logged or streamed to system's
    stdout/stderr during execution.
    This can be enabled or disabled for both of them independently.
    Additionally, a protocol object can be a used with the Runner. Such a
    protocol has to implement datalad.support.protocol.ProtocolInterface, is
    able to record calls and allows for dry runs.
    """

    __slots__ = ['commands', 'dry', 'cwd', 'env', 'protocol',
                 '_log_opts']

    def __init__(self, cwd=None, env=None, protocol=None, log_outputs=None):
        """
        Parameters
        ----------
        cwd: string, optional
             Base current working directory for commands.  Could be overridden
             per run call via cwd option
        env: dict, optional
             Custom environment to use for calls. Could be overridden per run
             call via env option
        protocol: ProtocolInterface
             Protocol object to write to.
        log_outputs : bool, optional
             Switch to instruct whether outputs should be logged or not.  If not
             set (default), config 'datalad.log.outputs' would be consulted
        """
        super(Runner, self).__init__(cwd=cwd, env=env)

        if protocol is None:
            # TODO: config cmd.protocol = null
            protocol_str = os.environ.get('DATALAD_CMD_PROTOCOL', 'null')
            protocol = {
                'externals-time': ExecutionTimeExternalsProtocol,
                'time': ExecutionTimeProtocol,
                'null': NullProtocol
            }[protocol_str]()
            if protocol_str != 'null':
                # we need to dump it into a file at the end
                # TODO: config cmd.protocol_prefix = protocol
                filename = '%s-%s.log' % (
                    os.environ.get('DATALAD_CMD_PROTOCOL_PREFIX', 'protocol'),
                    id(self)
                )
                atexit.register(functools.partial(protocol.write_to_file, filename))

        self.protocol = protocol
        # Various options for logging
        self._log_opts = {}
        # we don't know yet whether we need to log every output or not
        if log_outputs is not None:
            self._log_opts['outputs'] = log_outputs

    def __call__(self, cmd, *args, **kwargs):
        """Convenience method

        This will call run() or call() depending on the kind of `cmd`.
        If `cmd` is a string it is interpreted as the to be executed command.
        Otherwise it is expected to be a callable.
        Any other argument is passed to the respective method.

        Parameters
        ----------
        cmd: str or callable
           command string to be executed via shell or callable to be called.

        `*args`:
        `**kwargs`:
           see Runner.run() and Runner.call() for available arguments.

        Raises
        ------
        TypeError
          if cmd is neither a string nor a callable.
        """

        if isinstance(cmd, str) or isinstance(cmd, list):
            return self.run(cmd, *args, **kwargs)
        elif callable(cmd):
            return self.call(cmd, *args, **kwargs)
        else:
            raise TypeError("Argument 'command' is neither a string, "
                            "nor a list nor a callable.")

    def _opt_env_adapter(v):
        """If value is a string, split by ,"""
        if v:
            if v.isdigit():
                log_env = bool(int(v))
            else:
                log_env = v.split(',')
            return log_env
        else:
            return False

    _LOG_OPTS_ADAPTERS = OrderedDict([
        ('outputs', None),
        ('cwd', None),
        ('env', _opt_env_adapter),
        ('stdin', None),
    ])

    def _get_log_setting(self, opt, default=False):
        try:
            return self._log_opts[opt]
        except KeyError:
            try:
                from . import cfg
            except ImportError:
                return default
            adapter = self._LOG_OPTS_ADAPTERS.get(opt, None)
            self._log_opts[opt] = \
                (cfg.getbool if not adapter else cfg.get_value)(
                    'datalad.log.cmd', opt, default=default)
            if adapter:
                self._log_opts[opt] = adapter(self._log_opts[opt])
            return self._log_opts[opt]

    @property
    def log_outputs(self):
        return self._get_log_setting('outputs')

    @property
    def log_cwd(self):
        return self._get_log_setting('cwd')

    @property
    def log_stdin(self):
        return self._get_log_setting('stdin')

    @property
    def log_env(self):
        return self._get_log_setting('env')

    # Two helpers to encapsulate formatting/output
    def _log_out(self, line):
        if line and self.log_outputs:
            self.log("stdout| " + line.rstrip('\n'))

    def _log_err(self, line, expected=False):
        if line and self.log_outputs:
            self.log("stderr| " + line.rstrip('\n'),
                     level={True: 9,
                            False: 11}[expected])

    def _get_output_online(self, proc,
                           log_stdout, log_stderr,
                           outputstream, errstream,
                           expect_stderr=False, expect_fail=False):
        """

        If log_stdout or log_stderr are callables, they will be given a read
        line to be processed, and return processed result.  So if they need to
        'swallow' the line from being logged, should just return None

        Parameters
        ----------
        proc
        log_stdout: bool or callable or 'online' or 'offline'
        log_stderr: : bool or callable or 'online' or 'offline'
          If any of those 'offline', we would call proc.communicate at the
          end to grab possibly outstanding output from it
        expect_stderr
        expect_fail

        Returns
        -------

        """
        stdout, stderr = bytes(), bytes()

        log_stdout_ = _decide_to_log(log_stdout)
        log_stderr_ = _decide_to_log(log_stderr)
        log_stdout_is_callable = callable(log_stdout_)
        log_stderr_is_callable = callable(log_stderr_)

        # arguments to be passed into _process_one_line
        stdout_args = (
                'stdout',
                proc, log_stdout_, log_stdout_is_callable
        )
        stderr_args = (
                'stderr',
                proc, log_stderr_, log_stderr_is_callable,
                expect_stderr or expect_fail
        )

        while proc.poll() is None:
            # see for a possibly useful approach to processing output
            # in another thread http://codereview.stackexchange.com/a/17959
            # current problem is that if there is no output on stderr
            # it stalls
            # Monitor if anything was output and if nothing, sleep a bit
            stdout_, stderr_ = None, None
            if log_stdout_:
                stdout_ = self._process_one_line(*stdout_args)
                stdout += stdout_
            if log_stderr_:
                stderr_ = self._process_one_line(*stderr_args)
                stderr += stderr_
            if stdout_ is None and stderr_ is None:
                # no output was really produced, so sleep a tiny bit
                time.sleep(0.001)

        # Handle possible remaining output
        if log_stdout_ and log_stderr_:
            # If Popen was called with more than two pipes, calling
            # communicate() after we partially read the stream will return
            # empty output.
            stdout += self._process_remaining_output(
                outputstream, proc.stdout.read(), *stdout_args)
            stderr += self._process_remaining_output(
                errstream, proc.stderr.read(), *stderr_args)
        stdout_, stderr_ = proc.communicate()
        # ??? should we condition it on log_stdout in {'offline'} ???
        stdout += self._process_remaining_output(outputstream, stdout_, *stdout_args)
        stderr += self._process_remaining_output(errstream, stderr_, *stderr_args)

        return stdout, stderr

    def _process_remaining_output(self, stream, out_, *pargs):
        """Helper to process output which might have been obtained from popen or
        should be loaded from file"""
        out = bytes()
        if isinstance(stream, file_class) and \
                _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):
            assert out_ is None, "should have gone into a file"
            if not stream.closed:
                stream.close()
            with open(stream.name, 'rb') as f:
                for line in f:
                    out += self._process_one_line(*pargs, line=line)
        else:
            if out_:
                # resolving a once in a while failing test #2185
                if isinstance(out_, str):
                    out_ = out_.encode('utf-8')
                for line in out_.split(linesep_bytes):
                    out += self._process_one_line(
                        *pargs, line=line, suf=linesep_bytes)
        return out

    def _process_one_line(self, out_type, proc, log_, log_is_callable,
                          expected=False, line=None, suf=None):
        if line is None:
            lgr.log(3, "Reading line from %s", out_type)
            line = {'stdout': proc.stdout, 'stderr': proc.stderr}[out_type].readline()
        else:
            lgr.log(3, "Processing provided line")
        if line and log_is_callable:
            # Let it be processed
            line = log_(assure_unicode(line))
            if line is not None:
                # we are working with binary type here
                line = assure_bytes(line)
        if line:
            if out_type == 'stdout':
                self._log_out(assure_unicode(line))
            elif out_type == 'stderr':
                self._log_err(line.decode('utf-8'),
                              expected)
            else:  # pragma: no cover
                raise RuntimeError("must not get here")
            return (line + suf) if suf else line
        # it was output already directly but for code to work, return ""
        return bytes()

    def run(self, cmd, log_stdout=True, log_stderr=True, log_online=False,
            expect_stderr=False, expect_fail=False,
            cwd=None, env=None, shell=None, stdin=None):
        """Runs the command `cmd` using shell.

        In case of dry-mode `cmd` is just added to `commands` and it is
        actually executed otherwise.
        Allows for separately logging stdout and stderr  or streaming it to
        system's stdout or stderr respectively.

        Note: Using a string as `cmd` and shell=True allows for piping,
              multiple commands, etc., but that implies split_cmdline() is not
              used. This is considered to be a security hazard.
              So be careful with input.

        Parameters
        ----------
        cmd : str, list
          String (or list) defining the command call.  No shell is used if cmd
          is specified as a list

        log_stdout: bool, optional
            If True, stdout is logged. Goes to sys.stdout otherwise.

        log_stderr: bool, optional
            If True, stderr is logged. Goes to sys.stderr otherwise.

        log_online: bool, optional
            Whether to log as output comes in.  Setting to True is preferable
            for running user-invoked actions to provide timely output

        expect_stderr: bool, optional
            Normally, having stderr output is a signal of a problem and thus it
            gets logged at level 11.  But some utilities, e.g. wget, use
            stderr for their progress output.  Whenever such output is expected,
            set it to True and output will be logged at level 9 unless
            exit status is non-0 (in non-online mode only, in online -- would
            log at 9)

        expect_fail: bool, optional
            Normally, if command exits with non-0 status, it is considered an
            error and logged at level 11 (above DEBUG). But if the call intended
            for checking routine, such messages are usually not needed, thus
            it will be logged at level 9.

        cwd : string, optional
            Directory under which run the command (passed to Popen)

        env : string, optional
            Custom environment to pass

        shell: bool, optional
            Run command in a shell.  If not specified, then it runs in a shell
            only if command is specified as a string (not a list)

        stdin: file descriptor
            input stream to connect to stdin of the process.

        Returns
        -------
        (stdout, stderr) - bytes!

        Raises
        ------
        CommandError
           if command's exitcode wasn't 0 or None. exitcode is passed to
           CommandError's `code`-field. Command's stdout and stderr are stored
           in CommandError's `stdout` and `stderr` fields respectively.
        """
        outputstream = _get_output_stream(log_stdout, sys.stdout)
        errstream = _get_output_stream(log_stderr, sys.stderr)

        popen_env = env or self.env
        popen_cwd = cwd or self.cwd

        if popen_cwd and popen_env and 'PWD' in popen_env:
            # we must have inherited PWD, but cwd was provided, so we must
            # adjust it
            popen_env = popen_env.copy()  # to avoid side-effects
            popen_env['PWD'] = popen_cwd

        # TODO: if outputstream is sys.stdout and that one is set to StringIO
        #       we have to "shim" it with something providing fileno().
        # This happens when we do not swallow outputs, while allowing nosetest's
        # StringIO to be provided as stdout, crashing the Popen requiring
        # fileno().  In out swallow_outputs, we just use temporary files
        # to overcome this problem.
        # For now necessary test code should be wrapped into swallow_outputs cm
        # to avoid the problem
        log_msgs = ["Running: %s"]
        log_args = [cmd]
        if self.log_cwd:
            log_msgs += ['cwd=%r']
            log_args += [popen_cwd]
        if self.log_stdin:
            log_msgs += ['stdin=%r']
            log_args += [stdin]
        log_env = self.log_env
        if log_env and popen_env:
            log_msgs += ["env=%r"]
            log_args.append(
                popen_env if log_env is True
                else {k: popen_env[k] for k in log_env if k in popen_env}
            )
        log_msg = '\n'.join(log_msgs)
        self.log(log_msg, *log_args)

        if self.protocol.do_execute_ext_commands:

            if shell is None:
                shell = isinstance(cmd, str)

            if self.protocol.records_ext_commands:
                prot_exc = None
                prot_id = self.protocol.start_section(
                    split_cmdline(cmd)
                    if isinstance(cmd, str)
                    else cmd)
            try:
                proc = subprocess.Popen(cmd,
                                        stdout=outputstream,
                                        stderr=errstream,
                                        shell=shell,
                                        cwd=popen_cwd,
                                        env=popen_env,
                                        stdin=stdin)

            except Exception as e:
                prot_exc = e
                lgr.log(11, "Failed to start %r%r: %s" %
                        (cmd, " under %r" % cwd if cwd else '', exc_str(e)))
                raise

            finally:
                if self.protocol.records_ext_commands:
                    self.protocol.end_section(prot_id, prot_exc)

            try:
                if log_online:
                    out = self._get_output_online(proc,
                                                  log_stdout, log_stderr,
                                                  outputstream, errstream,
                                                  expect_stderr=expect_stderr,
                                                  expect_fail=expect_fail)
                else:
                    out = proc.communicate()

                # Decoding was delayed to this point
                def decode_if_not_None(x):
                    return "" if x is None else bytes.decode(x)
                out = tuple(map(decode_if_not_None, out))

                status = proc.poll()

                # needs to be done after we know status
                if not log_online:
                    self._log_out(out[0])
                    if status not in [0, None]:
                        self._log_err(out[1], expected=expect_fail)
                    else:
                        # as directed
                        self._log_err(out[1], expected=expect_stderr)

                if status not in [0, None]:
                    msg = "Failed to run %r%s. Exit code=%d.%s%s" \
                        % (cmd, " under %r" % (popen_cwd), status,
                           "" if log_online else " out=%s" % out[0],
                           "" if log_online else " err=%s" % out[1])
                    lgr.log(9 if expect_fail else 11, msg)
                    raise CommandError(str(cmd), msg, status, out[0], out[1])
                else:
                    self.log("Finished running %r with status %s" % (cmd, status),
                             level=8)

            except CommandError:
                # do not bother with reacting to "regular" CommandError
                # exceptions.  Somehow if we also terminate here for them
                # some processes elsewhere might stall:
                # see https://github.com/datalad/datalad/pull/3794
                raise

            except BaseException as exc:
                exc_info = sys.exc_info()
                # KeyboardInterrupt is subclass of BaseException
                lgr.debug("Terminating process for %s upon exception: %s",
                          cmd, exc_str(exc))
                try:
                    # there are still possible (although unlikely) cases when
                    # we fail to interrupt but we
                    # should not crash if we fail to terminate the process
                    proc.terminate()
                except BaseException as exc2:
                    lgr.warning("Failed to terminate process for %s: %s",
                                cmd, exc_str(exc2))
                raise exc_info[1]

            finally:
                # Those streams are for us to close if we asked for a PIPE
                # TODO -- assure closing the files
                _cleanup_output(outputstream, proc.stdout)
                _cleanup_output(errstream, proc.stderr)

        else:
            if self.protocol.records_ext_commands:
                self.protocol.add_section(split_cmdline(cmd)
                                          if isinstance(cmd, str)
                                          else cmd, None)
            out = ("DRY", "DRY")

        return out

    def call(self, f, *args, **kwargs):
        """Helper to unify collection of logging all "dry" actions.

        Calls `f` if `Runner`-object is not in dry-mode. Adds `f` along with
        its arguments to `commands` otherwise.

        Parameters
        ----------
        f: callable
        """
        if self.protocol.do_execute_callables:
            if self.protocol.records_callables:
                prot_exc = None
                prot_id = self.protocol.start_section(
                    [str(f), "args=%s" % str(args), "kwargs=%s" % str(kwargs)])

            try:
                return f(*args, **kwargs)
            except Exception as e:
                prot_exc = e
                raise
            finally:
                if self.protocol.records_callables:
                    self.protocol.end_section(prot_id, prot_exc)
        else:
            if self.protocol.records_callables:
                self.protocol.add_section(
                    [str(f), "args=%s" % str(args), "kwargs=%s" % str(kwargs)],
                    None)

    def log(self, msg, *args, **kwargs):
        """log helper

        Logs at level 9 by default and adds "Protocol:"-prefix in order to
        log the used protocol.
        """
        level = kwargs.pop('level', 9)
        if isinstance(self.protocol, NullProtocol):
            lgr.log(level, msg, *args, **kwargs)
        else:
            if args:
                msg = msg % args
            lgr.log(level, "{%s} %s" % (
                self.protocol.__class__.__name__, msg)
            )


class GitRunner(Runner):
    """
    Runner to be used to run git and git annex commands

    Overloads the runner class to check & update GIT_DIR and
    GIT_WORK_TREE environment variables set to the absolute path
    if is defined and is relative path
    """
    _GIT_PATH = None

    @borrowdoc(Runner)
    def __init__(self, *args, **kwargs):
        super(GitRunner, self).__init__(*args, **kwargs)
        self._check_git_path()

    @staticmethod
    def _check_git_path():
        """If using bundled git-annex, we would like to use bundled with it git

        Thus we will store _GIT_PATH a path to git in the same directory as annex
        if found.  If it is empty (but not None), we do nothing
        """
        if GitRunner._GIT_PATH is None:
            from distutils.spawn import find_executable
            # with all the nesting of config and this runner, cannot use our
            # cfg here, so will resort to dark magic of environment options
            if (os.environ.get('DATALAD_USE_DEFAULT_GIT', '0').lower()
                    in ('1', 'on', 'true', 'yes')):
                git_fpath = find_executable("git")
                if git_fpath:
                    GitRunner._GIT_PATH = ''
                    lgr.log(9, "Will use default git %s", git_fpath)
                    return  # we are done - there is a default git avail.
                # if not -- we will look for a bundled one
            GitRunner._GIT_PATH = GitRunner._get_bundled_path()
            lgr.log(9, "Will use git under %r (no adjustments to PATH if empty "
                       "string)", GitRunner._GIT_PATH)
            assert(GitRunner._GIT_PATH is not None)  # we made the decision!

    @staticmethod
    def _get_bundled_path():
        from distutils.spawn import find_executable
        annex_fpath = find_executable("git-annex")
        if not annex_fpath:
            # not sure how to live further anyways! ;)
            alongside = False
        else:
            annex_path = op.dirname(op.realpath(annex_fpath))
            alongside = op.lexists(op.join(annex_path, 'git'))
        return annex_path if alongside else ''

    @staticmethod
    def get_git_environ_adjusted(env=None):
        """
        Replaces GIT_DIR and GIT_WORK_TREE with absolute paths if relative path and defined
        """
        # if env set copy else get os environment
        git_env = env.copy() if env else os.environ.copy()
        if GitRunner._GIT_PATH:
            git_env['PATH'] = op.pathsep.join([GitRunner._GIT_PATH, git_env['PATH']]) \
                if 'PATH' in git_env \
                else GitRunner._GIT_PATH

        for varstring in ['GIT_DIR', 'GIT_WORK_TREE']:
            var = git_env.get(varstring)
            if var:                                    # if env variable set
                if not op.isabs(var):                   # and it's a relative path
                    git_env[varstring] = op.abspath(var)  # to absolute path
                    lgr.log(9, "Updated %s to %s", varstring, git_env[varstring])

        if 'GIT_SSH_COMMAND' not in git_env:
            git_env['GIT_SSH_COMMAND'] = GIT_SSH_COMMAND
            git_env['GIT_SSH_VARIANT'] = 'ssh'

        return git_env

    def run(self, cmd, env=None, *args, **kwargs):
        out, err = super(GitRunner, self).run(
            cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
        # All communication here will be returned as unicode
        # TODO: do that instead within the super's run!
        return assure_unicode(out), assure_unicode(err)


def readline_rstripped(stdout):
    #return iter(stdout.readline, b'').next().rstrip()
    return stdout.readline().rstrip()


class SafeDelCloseMixin(object):
    """A helper class to use where __del__ would call .close() which might
    fail if "too late in GC game"
    """
    def __del__(self):
        try:
            self.close()
        except TypeError:
            if os.fdopen is None or lgr.debug is None:
                # if we are late in the game and things already gc'ed in py3,
                # it is Ok
                return
            raise


@auto_repr
class BatchedCommand(SafeDelCloseMixin):
    """Container for a process which would allow for persistent communication
    """

    def __init__(self, cmd, path=None, output_proc=None):
        if not isinstance(cmd, list):
            cmd = [cmd]
        self.cmd = cmd
        self.path = path
        self.output_proc = output_proc if output_proc else readline_rstripped
        self._process = None
        self._stderr_out = None
        self._stderr_out_fname = None

    def _initialize(self):
        lgr.debug("Initiating a new process for %s" % repr(self))
        lgr.log(5, "Command: %s" % self.cmd)
        # according to the internet wisdom there is no easy way with subprocess
        # while avoid deadlocks etc.  We would need to start a thread/subprocess
        # to timeout etc
        # kwargs = dict(bufsize=1, universal_newlines=True) if PY3 else {}
        self._stderr_out, self._stderr_out_fname = tempfile.mkstemp()
        self._process = subprocess.Popen(
            self.cmd,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=self._stderr_out,
            env=GitRunner.get_git_environ_adjusted(),
            cwd=self.path,
            bufsize=1,
            universal_newlines=True  # **kwargs
        )

    def _check_process(self, restart=False):
        """Check if the process was terminated and restart if restart

        Returns
        -------
        bool
          True if process was alive.
        str
          stderr if any recorded if was terminated
        """
        process = self._process
        ret = True
        ret_stderr = None
        if process and process.poll():
            lgr.warning("Process %s was terminated with returncode %s" % (process, process.returncode))
            ret_stderr = self.close(return_stderr=True)
            ret = False
        if self._process is None and restart:
            lgr.warning("Restarting the process due to previous failure")
            self._initialize()
        return ret, ret_stderr

    def __call__(self, cmds):
        """

        Parameters
        ----------
        cmds : str or tuple or list of (str or tuple)

        Returns
        -------
        str or list
          Output received from process.  list in case if cmds was a list
        """
        input_multiple = isinstance(cmds, list)
        if not input_multiple:
            cmds = [cmds]

        output = [o for o in self.yield_(cmds)]
        return output if input_multiple else output[0]

    def yield_(self, cmds):
        """Same as __call__, but requires `cmds` to be an iterable

        and yields results for each item."""
        for entry in cmds:
            if not isinstance(entry, str):
                entry = ' '.join(entry)
            yield self.proc1(entry)

    def proc1(self, arg):
        """Same as __call__, but only takes a single command argument

        and returns a single result.
        """
        # TODO: add checks -- may be process died off and needs to be reinitiated
        if not self._process:
            self._initialize()

        entry = arg + '\n'
        lgr.log(5, "Sending %r to batched command %s" % (entry, self))
        # apparently communicate is just a one time show
        # stdout, stderr = self._process.communicate(entry)
        # according to the internet wisdom there is no easy way with subprocess
        self._check_process(restart=True)
        process = self._process  # _check_process might have restarted it
        process.stdin.write(entry)
        process.stdin.flush()
        lgr.log(5, "Done sending.")
        still_alive, stderr = self._check_process(restart=False)
        # TODO: we might want to handle still_alive, e.g. to allow for
        #       a number of restarts/resends, but it should be per command
        #       since for some we cannot just resend the same query. But if
        #       it is just a "get"er - we could resend it few times
        # The default output_proc expects a single line output.
        # TODO: timeouts etc
        stdout = assure_unicode(self.output_proc(process.stdout)) \
            if not process.stdout.closed else None
        if stderr:
            lgr.warning("Received output in stderr: %r", stderr)
        lgr.log(5, "Received output: %r" % stdout)
        return stdout

    def close(self, return_stderr=False):
        """Close communication and wait for process to terminate

        Returns
        -------
        str
          stderr output if return_stderr and stderr file was there.
          None otherwise
        """
        ret = None
        if self._stderr_out:
            # close possibly still open fd
            os.fdopen(self._stderr_out).close()
            self._stderr_out = None
        if self._process:
            process = self._process
            lgr.debug(
                "Closing stdin of %s and waiting process to finish", process)
            process.stdin.close()
            process.stdout.close()
            process.wait()
            self._process = None
            lgr.debug("Process %s has finished", process)
        if self._stderr_out_fname and os.path.exists(self._stderr_out_fname):
            if return_stderr:
                with open(self._stderr_out_fname, 'r') as f:
                    ret = f.read()
            # remove the file where we kept dumping stderr
            unlink(self._stderr_out_fname)
            self._stderr_out_fname = None
        return ret

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** Renaming `poll_period` to `poll_latency` Causes Multiple Values for `path` Argument

**Description:**
After renaming the `poll_period` parameter to `poll_latency` in the `run` method of `WitlessRunner`, certain function calls are resulting in a `TypeError`. Specifically, when attempting to pass `poll_latency` as a keyword argument, the function receives multiple values for the `path` parameter, leading to unexpected errors.

**Example Buggy Code:**
```python
runner.run(cmd, proc_stdout=outproc, poll_latency=0.1)
```

**Error Message:**
```
TypeError: run() got multiple values for argument 'path'
```

**Expected Behavior:**
The `run` method should accept `poll_latency` as a valid keyword argument without causing any conflicts or multiple assignments of other parameters like `path`. Functions calling `run` with `poll_latency` should execute successfully without raising errors related to argument values.

[/ISSUE]
{
    "test_runner_stderr_capture": "PASSED",
    "test_runner_stdout_capture": "PASSED",
    "test_runner_incomplete_capture": "PASSED",
    "test_runner": "ERROR",
    "test_runner_failure": "ERROR",
    "test_runner_fix_PWD": "FAILED",
    "test_runner_cwd_encoding": "FAILED",
    "test_runner_stdin": "FAILED"
}
diff --git a/datalad/cmd.py b/datalad/cmd.py
index e14de0983..3fbe70df6 100644
--- a/datalad/cmd.py
+++ b/datalad/cmd.py
@@ -144,7 +144,7 @@ class WitlessRunner(object):
             self.env['PWD'] = self.cwd
 
     def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,
-            poll_period=0.1):
+            poll_latency=0.1):
         """Execute a command and communicate with it.
 
         Parameters
@@ -171,7 +171,9 @@ class WitlessRunner(object):
         poll_latency : float, optional
           Shortest interval at which the running process is queried for
           output (in seconds). Any potential output processing will increase
-          the effective interval.
+          the effective interval. When the effective polling frequency is
+          too low to keep the output buffers below their maximum size,
+          a process will deadlock.
 
         Returns
         -------
@@ -226,8 +228,7 @@ class WitlessRunner(object):
             while process.poll() is None or keep_going:
                 # one last read?
                 keep_going = process.returncode is None
-                # get a chunk of output for the specific period
-                # of time
+                # get a chunk of output
                 pout = [
                     # read whatever is available, should not block
                     o.read1(-1) if p else None
@@ -249,7 +250,7 @@ class WitlessRunner(object):
                     buffer = unprocessed[i] + o if unprocessed[i] else o
                     # engage output processor
                     unprocessed[i] = _handle_output(buffer, out[i], proc)
-                time.sleep(poll_period)
+                time.sleep(poll_latency)
 
             # obtain exit code
             status = process.poll()

{"datalad/cmd.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\nWrapper for command and function calls, allowing for dry runs and output handling\n\n\"\"\"\n\nimport time\nimport subprocess\nimport sys\nimport logging\nimport os\nimport atexit\nimport functools\nimport tempfile\n\nfrom collections import OrderedDict\nfrom .support import path as op\nfrom .consts import GIT_SSH_COMMAND\nfrom .dochelpers import exc_str\nfrom .support.exceptions import CommandError\nfrom .support.protocol import (\n    NullProtocol,\n    ExecutionTimeProtocol,\n    ExecutionTimeExternalsProtocol,\n)\nfrom .utils import (\n    on_windows,\n    get_tempfile_kwargs,\n    assure_unicode,\n    assure_bytes,\n    unlink,\n    auto_repr,\n    split_cmdline,\n)\nfrom .dochelpers import borrowdoc\n\nlgr = logging.getLogger('datalad.cmd')\n\n# In python3 to split byte stream on newline, it must be bytes\nlinesep_bytes = os.linesep.encode()\n\n_TEMP_std = sys.stdout, sys.stderr\n# To be used in the temp file name to distinguish the ones we create\n# in Runner so we take care about their removal, in contrast to those\n# which might be created outside and passed into Runner\n_MAGICAL_OUTPUT_MARKER = \"_runneroutput_\"\n\nfrom io import IOBase as file_class\n\n\ndef _decide_to_log(v):\n    \"\"\"Hacky workaround for now so we could specify per each which to\n    log online and which to the log\"\"\"\n    if isinstance(v, bool) or callable(v):\n        return v\n    elif v in {'online'}:\n        return True\n    elif v in {'offline'}:\n        return False\n    else:\n        raise ValueError(\"can be bool, callable, 'online' or 'offline'\")\n\n\ndef _get_output_stream(log_std, false_value):\n    \"\"\"Helper to prepare output stream for Popen and use file for 'offline'\n\n    Necessary to avoid lockdowns when both stdout and stderr are pipes\n    \"\"\"\n    if log_std:\n        if log_std == 'offline':\n            # we will open a temporary file\n\n            tf = tempfile.mktemp(\n                **get_tempfile_kwargs({}, prefix=_MAGICAL_OUTPUT_MARKER)\n            )\n            return open(tf, 'w')  # XXX PY3 should be 'b' may be?\n        else:\n            return subprocess.PIPE\n    else:\n        return false_value\n\n\ndef _cleanup_output(stream, std):\n    if isinstance(stream, file_class) and \\\n        _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):\n        if not stream.closed:\n            stream.close()\n        if op.exists(stream.name):\n            unlink(stream.name)\n    elif stream == subprocess.PIPE:\n        std.close()\n\n\ndef kill_output(output):\n    \"\"\"Helper for WitlessRunner to swallow all output and neither\n    relay it to the parent process's stdout, nor provide it as the\n    return value of WitlessRunner.run().\n    \"\"\"\n    return b'', 0\n\n\ndef capture_output(output):\n    \"\"\"Helper for WitlessRunner to capture all output and\n    provide it as the return value of WitlessRunner.run().\n    \"\"\"\n    return output, 0\n\n\nclass WitlessRunner(object):\n    \"\"\"Minimal Runner with support for online command output processing\n\n    It aims to be as simple as possible, providing only essential\n    functionality. Derived classes should be used for additional\n    specializations and convenience features.\n    \"\"\"\n    __slots__ = ['cwd', 'env']\n\n    def __init__(self, cwd=None, env=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd : path-like, optional\n          If given, commands are executed with this path as PWD,\n          the PWD of the parent process is used otherwise.\n        env : dict, optional\n          Environment to be passed to subprocess.Popen(). If `cwd`\n          was given, 'PWD' in the environment is set to its value.\n          This must be a complete environment definition, no values\n          from the current environment will be inherited.\n        \"\"\"\n        self.env = env.copy() if env else None\n        # stringify to support Path instances on PY35\n        self.cwd = str(cwd) if cwd is not None else None\n        if self.cwd and env is not None:\n            # if CWD was provided, we must not make it conflict with\n            # a potential PWD setting\n            self.env['PWD'] = self.cwd\n\n    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,\n            poll_period=0.1):\n        \"\"\"Execute a command and communicate with it.\n\n        Parameters\n        ----------\n        cmd : list\n          Sequence of program arguments. Passing a single string means\n          that it is simply the name of the program, no complex shell\n          commands are supported.\n        proc_stdout : callable, optional\n          By default no stdout is captured, but relayed to the parent\n          process's stdout. If given, all stdout is passed as a byte-string\n          to this callable, in the chunks it was received by polling the\n          processing. The callable may transform it in any way, its output\n          (byte-string) is concatenated and provided as stdout return value.\n          The helper functions 'kill_output' and 'capture_output' are\n          provided to either swallow all output (and not relay it to the\n          parent) or to capture all output and provide it as the return\n          value.\n        proc_stderr : callable, optional\n          Like proc_stdout, but for stderr.\n        stdin : byte stream, optional\n          File descriptor like, used as stdin for the process. Passed\n          verbatim to subprocess.Popen().\n        poll_latency : float, optional\n          Shortest interval at which the running process is queried for\n          output (in seconds). Any potential output processing will increase\n          the effective interval.\n\n        Returns\n        -------\n        stdout, stderr\n          Unicode string with the cumulative standard output and error\n          of the process.\n\n        Raises\n        ------\n        CommandError\n          On execution failure (non-zero exit code) this exception is\n          raised which provides the command (cmd), stdout, stderr,\n          exit code (status), and a message identifying the failed\n          command, as properties.\n        \"\"\"\n        proc_out = (proc_stdout, proc_stderr)\n        try:\n            lgr.log(8, \"Start running %r\", cmd)\n            process = subprocess.Popen(\n                cmd,\n                stdout=subprocess.PIPE if proc_stdout else None,\n                stderr=subprocess.PIPE if proc_stderr else None,\n                shell=False,\n                cwd=self.cwd,\n                env=self.env,\n                stdin=stdin,\n                # intermediate reports are never decoded anyways\n                # from PY37 onwards\n                #text=False,\n                universal_newlines=False,\n            )\n        except Exception as e:\n            lgr.log(11, \"Failed to start %r%r: %s\" %\n                    (cmd,\n                     (\" under %r\" % self.cwd) if self.cwd else '',\n                     exc_str(e)))\n            raise\n\n        try:\n            out = [[], []]\n            unprocessed = [None, None]\n            # make sure to run this loop at least once, even if the\n            # process is already dead\n            keep_going = True\n\n            def _handle_output(u, o, p):\n                processed, unprocessed_len = proc(u)\n                if processed:\n                    o.append(processed)\n                return u[-(unprocessed_len):] if unprocessed_len else None\n\n            while process.poll() is None or keep_going:\n                # one last read?\n                keep_going = process.returncode is None\n                # get a chunk of output for the specific period\n                # of time\n                pout = [\n                    # read whatever is available, should not block\n                    o.read1(-1) if p else None\n                    for o, p in zip(\n                        (process.stdout, process.stderr),\n                        proc_out)\n                ]\n\n                for i, (o, proc) in enumerate(zip(pout, proc_out)):\n                    if not o:\n                        # nothing read\n                        continue\n                    if proc is None:\n                        # no index update needed, can never change, hence no unprocessed\n                        # output either\n                        out[i].append(o)\n                        continue\n                    # make sure to feed back any unprocessed stuff\n                    buffer = unprocessed[i] + o if unprocessed[i] else o\n                    # engage output processor\n                    unprocessed[i] = _handle_output(buffer, out[i], proc)\n                time.sleep(poll_period)\n\n            # obtain exit code\n            status = process.poll()\n\n            # decode bytes to string\n            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)\n\n            if status not in [0, None]:\n                msg = \"Failed to run %r%s.\" % (\n                    cmd,\n                    (\" under %r\" % self.cwd) if self.cwd else '',\n                )\n                raise CommandError(\n                    cmd=str(cmd),\n                    msg=msg,\n                    code=status,\n                    stdout=out[0],\n                    stderr=out[1],\n                )\n            else:\n                lgr.log(8, \"Finished running %r with status %s\", cmd, status)\n\n        except CommandError:\n            # do not bother with reacting to \"regular\" CommandError\n            # exceptions.  Somehow if we also terminate here for them\n            # some processes elsewhere might stall:\n            # see https://github.com/datalad/datalad/pull/3794\n            raise\n\n        except BaseException as exc:\n            exc_info = sys.exc_info()\n            # KeyboardInterrupt is subclass of BaseException\n            lgr.debug(\"Terminating process for %s upon exception: %s\",\n                      cmd, exc_str(exc))\n            try:\n                # there are still possible (although unlikely) cases when\n                # we fail to interrupt but we\n                # should not crash if we fail to terminate the process\n                process.terminate()\n            except BaseException as exc2:\n                lgr.warning(\"Failed to terminate process for %s: %s\",\n                            cmd, exc_str(exc2))\n            raise exc_info[1]\n\n        return out\n\n\nclass Runner(WitlessRunner):\n    \"\"\"Provides a wrapper for calling functions and commands.\n\n    An object of this class provides a methods that calls shell commands or\n    python functions, allowing for protocolling the calls and output handling.\n\n    Outputs (stdout and stderr) can be either logged or streamed to system's\n    stdout/stderr during execution.\n    This can be enabled or disabled for both of them independently.\n    Additionally, a protocol object can be a used with the Runner. Such a\n    protocol has to implement datalad.support.protocol.ProtocolInterface, is\n    able to record calls and allows for dry runs.\n    \"\"\"\n\n    __slots__ = ['commands', 'dry', 'cwd', 'env', 'protocol',\n                 '_log_opts']\n\n    def __init__(self, cwd=None, env=None, protocol=None, log_outputs=None):\n        \"\"\"\n        Parameters\n        ----------\n        cwd: string, optional\n             Base current working directory for commands.  Could be overridden\n             per run call via cwd option\n        env: dict, optional\n             Custom environment to use for calls. Could be overridden per run\n             call via env option\n        protocol: ProtocolInterface\n             Protocol object to write to.\n        log_outputs : bool, optional\n             Switch to instruct whether outputs should be logged or not.  If not\n             set (default), config 'datalad.log.outputs' would be consulted\n        \"\"\"\n        super(Runner, self).__init__(cwd=cwd, env=env)\n\n        if protocol is None:\n            # TODO: config cmd.protocol = null\n            protocol_str = os.environ.get('DATALAD_CMD_PROTOCOL', 'null')\n            protocol = {\n                'externals-time': ExecutionTimeExternalsProtocol,\n                'time': ExecutionTimeProtocol,\n                'null': NullProtocol\n            }[protocol_str]()\n            if protocol_str != 'null':\n                # we need to dump it into a file at the end\n                # TODO: config cmd.protocol_prefix = protocol\n                filename = '%s-%s.log' % (\n                    os.environ.get('DATALAD_CMD_PROTOCOL_PREFIX', 'protocol'),\n                    id(self)\n                )\n                atexit.register(functools.partial(protocol.write_to_file, filename))\n\n        self.protocol = protocol\n        # Various options for logging\n        self._log_opts = {}\n        # we don't know yet whether we need to log every output or not\n        if log_outputs is not None:\n            self._log_opts['outputs'] = log_outputs\n\n    def __call__(self, cmd, *args, **kwargs):\n        \"\"\"Convenience method\n\n        This will call run() or call() depending on the kind of `cmd`.\n        If `cmd` is a string it is interpreted as the to be executed command.\n        Otherwise it is expected to be a callable.\n        Any other argument is passed to the respective method.\n\n        Parameters\n        ----------\n        cmd: str or callable\n           command string to be executed via shell or callable to be called.\n\n        `*args`:\n        `**kwargs`:\n           see Runner.run() and Runner.call() for available arguments.\n\n        Raises\n        ------\n        TypeError\n          if cmd is neither a string nor a callable.\n        \"\"\"\n\n        if isinstance(cmd, str) or isinstance(cmd, list):\n            return self.run(cmd, *args, **kwargs)\n        elif callable(cmd):\n            return self.call(cmd, *args, **kwargs)\n        else:\n            raise TypeError(\"Argument 'command' is neither a string, \"\n                            \"nor a list nor a callable.\")\n\n    def _opt_env_adapter(v):\n        \"\"\"If value is a string, split by ,\"\"\"\n        if v:\n            if v.isdigit():\n                log_env = bool(int(v))\n            else:\n                log_env = v.split(',')\n            return log_env\n        else:\n            return False\n\n    _LOG_OPTS_ADAPTERS = OrderedDict([\n        ('outputs', None),\n        ('cwd', None),\n        ('env', _opt_env_adapter),\n        ('stdin', None),\n    ])\n\n    def _get_log_setting(self, opt, default=False):\n        try:\n            return self._log_opts[opt]\n        except KeyError:\n            try:\n                from . import cfg\n            except ImportError:\n                return default\n            adapter = self._LOG_OPTS_ADAPTERS.get(opt, None)\n            self._log_opts[opt] = \\\n                (cfg.getbool if not adapter else cfg.get_value)(\n                    'datalad.log.cmd', opt, default=default)\n            if adapter:\n                self._log_opts[opt] = adapter(self._log_opts[opt])\n            return self._log_opts[opt]\n\n    @property\n    def log_outputs(self):\n        return self._get_log_setting('outputs')\n\n    @property\n    def log_cwd(self):\n        return self._get_log_setting('cwd')\n\n    @property\n    def log_stdin(self):\n        return self._get_log_setting('stdin')\n\n    @property\n    def log_env(self):\n        return self._get_log_setting('env')\n\n    # Two helpers to encapsulate formatting/output\n    def _log_out(self, line):\n        if line and self.log_outputs:\n            self.log(\"stdout| \" + line.rstrip('\\n'))\n\n    def _log_err(self, line, expected=False):\n        if line and self.log_outputs:\n            self.log(\"stderr| \" + line.rstrip('\\n'),\n                     level={True: 9,\n                            False: 11}[expected])\n\n    def _get_output_online(self, proc,\n                           log_stdout, log_stderr,\n                           outputstream, errstream,\n                           expect_stderr=False, expect_fail=False):\n        \"\"\"\n\n        If log_stdout or log_stderr are callables, they will be given a read\n        line to be processed, and return processed result.  So if they need to\n        'swallow' the line from being logged, should just return None\n\n        Parameters\n        ----------\n        proc\n        log_stdout: bool or callable or 'online' or 'offline'\n        log_stderr: : bool or callable or 'online' or 'offline'\n          If any of those 'offline', we would call proc.communicate at the\n          end to grab possibly outstanding output from it\n        expect_stderr\n        expect_fail\n\n        Returns\n        -------\n\n        \"\"\"\n        stdout, stderr = bytes(), bytes()\n\n        log_stdout_ = _decide_to_log(log_stdout)\n        log_stderr_ = _decide_to_log(log_stderr)\n        log_stdout_is_callable = callable(log_stdout_)\n        log_stderr_is_callable = callable(log_stderr_)\n\n        # arguments to be passed into _process_one_line\n        stdout_args = (\n                'stdout',\n                proc, log_stdout_, log_stdout_is_callable\n        )\n        stderr_args = (\n                'stderr',\n                proc, log_stderr_, log_stderr_is_callable,\n                expect_stderr or expect_fail\n        )\n\n        while proc.poll() is None:\n            # see for a possibly useful approach to processing output\n            # in another thread http://codereview.stackexchange.com/a/17959\n            # current problem is that if there is no output on stderr\n            # it stalls\n            # Monitor if anything was output and if nothing, sleep a bit\n            stdout_, stderr_ = None, None\n            if log_stdout_:\n                stdout_ = self._process_one_line(*stdout_args)\n                stdout += stdout_\n            if log_stderr_:\n                stderr_ = self._process_one_line(*stderr_args)\n                stderr += stderr_\n            if stdout_ is None and stderr_ is None:\n                # no output was really produced, so sleep a tiny bit\n                time.sleep(0.001)\n\n        # Handle possible remaining output\n        if log_stdout_ and log_stderr_:\n            # If Popen was called with more than two pipes, calling\n            # communicate() after we partially read the stream will return\n            # empty output.\n            stdout += self._process_remaining_output(\n                outputstream, proc.stdout.read(), *stdout_args)\n            stderr += self._process_remaining_output(\n                errstream, proc.stderr.read(), *stderr_args)\n        stdout_, stderr_ = proc.communicate()\n        # ??? should we condition it on log_stdout in {'offline'} ???\n        stdout += self._process_remaining_output(outputstream, stdout_, *stdout_args)\n        stderr += self._process_remaining_output(errstream, stderr_, *stderr_args)\n\n        return stdout, stderr\n\n    def _process_remaining_output(self, stream, out_, *pargs):\n        \"\"\"Helper to process output which might have been obtained from popen or\n        should be loaded from file\"\"\"\n        out = bytes()\n        if isinstance(stream, file_class) and \\\n                _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):\n            assert out_ is None, \"should have gone into a file\"\n            if not stream.closed:\n                stream.close()\n            with open(stream.name, 'rb') as f:\n                for line in f:\n                    out += self._process_one_line(*pargs, line=line)\n        else:\n            if out_:\n                # resolving a once in a while failing test #2185\n                if isinstance(out_, str):\n                    out_ = out_.encode('utf-8')\n                for line in out_.split(linesep_bytes):\n                    out += self._process_one_line(\n                        *pargs, line=line, suf=linesep_bytes)\n        return out\n\n    def _process_one_line(self, out_type, proc, log_, log_is_callable,\n                          expected=False, line=None, suf=None):\n        if line is None:\n            lgr.log(3, \"Reading line from %s\", out_type)\n            line = {'stdout': proc.stdout, 'stderr': proc.stderr}[out_type].readline()\n        else:\n            lgr.log(3, \"Processing provided line\")\n        if line and log_is_callable:\n            # Let it be processed\n            line = log_(assure_unicode(line))\n            if line is not None:\n                # we are working with binary type here\n                line = assure_bytes(line)\n        if line:\n            if out_type == 'stdout':\n                self._log_out(assure_unicode(line))\n            elif out_type == 'stderr':\n                self._log_err(line.decode('utf-8'),\n                              expected)\n            else:  # pragma: no cover\n                raise RuntimeError(\"must not get here\")\n            return (line + suf) if suf else line\n        # it was output already directly but for code to work, return \"\"\n        return bytes()\n\n    def run(self, cmd, log_stdout=True, log_stderr=True, log_online=False,\n            expect_stderr=False, expect_fail=False,\n            cwd=None, env=None, shell=None, stdin=None):\n        \"\"\"Runs the command `cmd` using shell.\n\n        In case of dry-mode `cmd` is just added to `commands` and it is\n        actually executed otherwise.\n        Allows for separately logging stdout and stderr  or streaming it to\n        system's stdout or stderr respectively.\n\n        Note: Using a string as `cmd` and shell=True allows for piping,\n              multiple commands, etc., but that implies split_cmdline() is not\n              used. This is considered to be a security hazard.\n              So be careful with input.\n\n        Parameters\n        ----------\n        cmd : str, list\n          String (or list) defining the command call.  No shell is used if cmd\n          is specified as a list\n\n        log_stdout: bool, optional\n            If True, stdout is logged. Goes to sys.stdout otherwise.\n\n        log_stderr: bool, optional\n            If True, stderr is logged. Goes to sys.stderr otherwise.\n\n        log_online: bool, optional\n            Whether to log as output comes in.  Setting to True is preferable\n            for running user-invoked actions to provide timely output\n\n        expect_stderr: bool, optional\n            Normally, having stderr output is a signal of a problem and thus it\n            gets logged at level 11.  But some utilities, e.g. wget, use\n            stderr for their progress output.  Whenever such output is expected,\n            set it to True and output will be logged at level 9 unless\n            exit status is non-0 (in non-online mode only, in online -- would\n            log at 9)\n\n        expect_fail: bool, optional\n            Normally, if command exits with non-0 status, it is considered an\n            error and logged at level 11 (above DEBUG). But if the call intended\n            for checking routine, such messages are usually not needed, thus\n            it will be logged at level 9.\n\n        cwd : string, optional\n            Directory under which run the command (passed to Popen)\n\n        env : string, optional\n            Custom environment to pass\n\n        shell: bool, optional\n            Run command in a shell.  If not specified, then it runs in a shell\n            only if command is specified as a string (not a list)\n\n        stdin: file descriptor\n            input stream to connect to stdin of the process.\n\n        Returns\n        -------\n        (stdout, stderr) - bytes!\n\n        Raises\n        ------\n        CommandError\n           if command's exitcode wasn't 0 or None. exitcode is passed to\n           CommandError's `code`-field. Command's stdout and stderr are stored\n           in CommandError's `stdout` and `stderr` fields respectively.\n        \"\"\"\n        outputstream = _get_output_stream(log_stdout, sys.stdout)\n        errstream = _get_output_stream(log_stderr, sys.stderr)\n\n        popen_env = env or self.env\n        popen_cwd = cwd or self.cwd\n\n        if popen_cwd and popen_env and 'PWD' in popen_env:\n            # we must have inherited PWD, but cwd was provided, so we must\n            # adjust it\n            popen_env = popen_env.copy()  # to avoid side-effects\n            popen_env['PWD'] = popen_cwd\n\n        # TODO: if outputstream is sys.stdout and that one is set to StringIO\n        #       we have to \"shim\" it with something providing fileno().\n        # This happens when we do not swallow outputs, while allowing nosetest's\n        # StringIO to be provided as stdout, crashing the Popen requiring\n        # fileno().  In out swallow_outputs, we just use temporary files\n        # to overcome this problem.\n        # For now necessary test code should be wrapped into swallow_outputs cm\n        # to avoid the problem\n        log_msgs = [\"Running: %s\"]\n        log_args = [cmd]\n        if self.log_cwd:\n            log_msgs += ['cwd=%r']\n            log_args += [popen_cwd]\n        if self.log_stdin:\n            log_msgs += ['stdin=%r']\n            log_args += [stdin]\n        log_env = self.log_env\n        if log_env and popen_env:\n            log_msgs += [\"env=%r\"]\n            log_args.append(\n                popen_env if log_env is True\n                else {k: popen_env[k] for k in log_env if k in popen_env}\n            )\n        log_msg = '\\n'.join(log_msgs)\n        self.log(log_msg, *log_args)\n\n        if self.protocol.do_execute_ext_commands:\n\n            if shell is None:\n                shell = isinstance(cmd, str)\n\n            if self.protocol.records_ext_commands:\n                prot_exc = None\n                prot_id = self.protocol.start_section(\n                    split_cmdline(cmd)\n                    if isinstance(cmd, str)\n                    else cmd)\n            try:\n                proc = subprocess.Popen(cmd,\n                                        stdout=outputstream,\n                                        stderr=errstream,\n                                        shell=shell,\n                                        cwd=popen_cwd,\n                                        env=popen_env,\n                                        stdin=stdin)\n\n            except Exception as e:\n                prot_exc = e\n                lgr.log(11, \"Failed to start %r%r: %s\" %\n                        (cmd, \" under %r\" % cwd if cwd else '', exc_str(e)))\n                raise\n\n            finally:\n                if self.protocol.records_ext_commands:\n                    self.protocol.end_section(prot_id, prot_exc)\n\n            try:\n                if log_online:\n                    out = self._get_output_online(proc,\n                                                  log_stdout, log_stderr,\n                                                  outputstream, errstream,\n                                                  expect_stderr=expect_stderr,\n                                                  expect_fail=expect_fail)\n                else:\n                    out = proc.communicate()\n\n                # Decoding was delayed to this point\n                def decode_if_not_None(x):\n                    return \"\" if x is None else bytes.decode(x)\n                out = tuple(map(decode_if_not_None, out))\n\n                status = proc.poll()\n\n                # needs to be done after we know status\n                if not log_online:\n                    self._log_out(out[0])\n                    if status not in [0, None]:\n                        self._log_err(out[1], expected=expect_fail)\n                    else:\n                        # as directed\n                        self._log_err(out[1], expected=expect_stderr)\n\n                if status not in [0, None]:\n                    msg = \"Failed to run %r%s. Exit code=%d.%s%s\" \\\n                        % (cmd, \" under %r\" % (popen_cwd), status,\n                           \"\" if log_online else \" out=%s\" % out[0],\n                           \"\" if log_online else \" err=%s\" % out[1])\n                    lgr.log(9 if expect_fail else 11, msg)\n                    raise CommandError(str(cmd), msg, status, out[0], out[1])\n                else:\n                    self.log(\"Finished running %r with status %s\" % (cmd, status),\n                             level=8)\n\n            except CommandError:\n                # do not bother with reacting to \"regular\" CommandError\n                # exceptions.  Somehow if we also terminate here for them\n                # some processes elsewhere might stall:\n                # see https://github.com/datalad/datalad/pull/3794\n                raise\n\n            except BaseException as exc:\n                exc_info = sys.exc_info()\n                # KeyboardInterrupt is subclass of BaseException\n                lgr.debug(\"Terminating process for %s upon exception: %s\",\n                          cmd, exc_str(exc))\n                try:\n                    # there are still possible (although unlikely) cases when\n                    # we fail to interrupt but we\n                    # should not crash if we fail to terminate the process\n                    proc.terminate()\n                except BaseException as exc2:\n                    lgr.warning(\"Failed to terminate process for %s: %s\",\n                                cmd, exc_str(exc2))\n                raise exc_info[1]\n\n            finally:\n                # Those streams are for us to close if we asked for a PIPE\n                # TODO -- assure closing the files\n                _cleanup_output(outputstream, proc.stdout)\n                _cleanup_output(errstream, proc.stderr)\n\n        else:\n            if self.protocol.records_ext_commands:\n                self.protocol.add_section(split_cmdline(cmd)\n                                          if isinstance(cmd, str)\n                                          else cmd, None)\n            out = (\"DRY\", \"DRY\")\n\n        return out\n\n    def call(self, f, *args, **kwargs):\n        \"\"\"Helper to unify collection of logging all \"dry\" actions.\n\n        Calls `f` if `Runner`-object is not in dry-mode. Adds `f` along with\n        its arguments to `commands` otherwise.\n\n        Parameters\n        ----------\n        f: callable\n        \"\"\"\n        if self.protocol.do_execute_callables:\n            if self.protocol.records_callables:\n                prot_exc = None\n                prot_id = self.protocol.start_section(\n                    [str(f), \"args=%s\" % str(args), \"kwargs=%s\" % str(kwargs)])\n\n            try:\n                return f(*args, **kwargs)\n            except Exception as e:\n                prot_exc = e\n                raise\n            finally:\n                if self.protocol.records_callables:\n                    self.protocol.end_section(prot_id, prot_exc)\n        else:\n            if self.protocol.records_callables:\n                self.protocol.add_section(\n                    [str(f), \"args=%s\" % str(args), \"kwargs=%s\" % str(kwargs)],\n                    None)\n\n    def log(self, msg, *args, **kwargs):\n        \"\"\"log helper\n\n        Logs at level 9 by default and adds \"Protocol:\"-prefix in order to\n        log the used protocol.\n        \"\"\"\n        level = kwargs.pop('level', 9)\n        if isinstance(self.protocol, NullProtocol):\n            lgr.log(level, msg, *args, **kwargs)\n        else:\n            if args:\n                msg = msg % args\n            lgr.log(level, \"{%s} %s\" % (\n                self.protocol.__class__.__name__, msg)\n            )\n\n\nclass GitRunner(Runner):\n    \"\"\"\n    Runner to be used to run git and git annex commands\n\n    Overloads the runner class to check & update GIT_DIR and\n    GIT_WORK_TREE environment variables set to the absolute path\n    if is defined and is relative path\n    \"\"\"\n    _GIT_PATH = None\n\n    @borrowdoc(Runner)\n    def __init__(self, *args, **kwargs):\n        super(GitRunner, self).__init__(*args, **kwargs)\n        self._check_git_path()\n\n    @staticmethod\n    def _check_git_path():\n        \"\"\"If using bundled git-annex, we would like to use bundled with it git\n\n        Thus we will store _GIT_PATH a path to git in the same directory as annex\n        if found.  If it is empty (but not None), we do nothing\n        \"\"\"\n        if GitRunner._GIT_PATH is None:\n            from distutils.spawn import find_executable\n            # with all the nesting of config and this runner, cannot use our\n            # cfg here, so will resort to dark magic of environment options\n            if (os.environ.get('DATALAD_USE_DEFAULT_GIT', '0').lower()\n                    in ('1', 'on', 'true', 'yes')):\n                git_fpath = find_executable(\"git\")\n                if git_fpath:\n                    GitRunner._GIT_PATH = ''\n                    lgr.log(9, \"Will use default git %s\", git_fpath)\n                    return  # we are done - there is a default git avail.\n                # if not -- we will look for a bundled one\n            GitRunner._GIT_PATH = GitRunner._get_bundled_path()\n            lgr.log(9, \"Will use git under %r (no adjustments to PATH if empty \"\n                       \"string)\", GitRunner._GIT_PATH)\n            assert(GitRunner._GIT_PATH is not None)  # we made the decision!\n\n    @staticmethod\n    def _get_bundled_path():\n        from distutils.spawn import find_executable\n        annex_fpath = find_executable(\"git-annex\")\n        if not annex_fpath:\n            # not sure how to live further anyways! ;)\n            alongside = False\n        else:\n            annex_path = op.dirname(op.realpath(annex_fpath))\n            alongside = op.lexists(op.join(annex_path, 'git'))\n        return annex_path if alongside else ''\n\n    @staticmethod\n    def get_git_environ_adjusted(env=None):\n        \"\"\"\n        Replaces GIT_DIR and GIT_WORK_TREE with absolute paths if relative path and defined\n        \"\"\"\n        # if env set copy else get os environment\n        git_env = env.copy() if env else os.environ.copy()\n        if GitRunner._GIT_PATH:\n            git_env['PATH'] = op.pathsep.join([GitRunner._GIT_PATH, git_env['PATH']]) \\\n                if 'PATH' in git_env \\\n                else GitRunner._GIT_PATH\n\n        for varstring in ['GIT_DIR', 'GIT_WORK_TREE']:\n            var = git_env.get(varstring)\n            if var:                                    # if env variable set\n                if not op.isabs(var):                   # and it's a relative path\n                    git_env[varstring] = op.abspath(var)  # to absolute path\n                    lgr.log(9, \"Updated %s to %s\", varstring, git_env[varstring])\n\n        if 'GIT_SSH_COMMAND' not in git_env:\n            git_env['GIT_SSH_COMMAND'] = GIT_SSH_COMMAND\n            git_env['GIT_SSH_VARIANT'] = 'ssh'\n\n        return git_env\n\n    def run(self, cmd, env=None, *args, **kwargs):\n        out, err = super(GitRunner, self).run(\n            cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)\n        # All communication here will be returned as unicode\n        # TODO: do that instead within the super's run!\n        return assure_unicode(out), assure_unicode(err)\n\n\ndef readline_rstripped(stdout):\n    #return iter(stdout.readline, b'').next().rstrip()\n    return stdout.readline().rstrip()\n\n\nclass SafeDelCloseMixin(object):\n    \"\"\"A helper class to use where __del__ would call .close() which might\n    fail if \"too late in GC game\"\n    \"\"\"\n    def __del__(self):\n        try:\n            self.close()\n        except TypeError:\n            if os.fdopen is None or lgr.debug is None:\n                # if we are late in the game and things already gc'ed in py3,\n                # it is Ok\n                return\n            raise\n\n\n@auto_repr\nclass BatchedCommand(SafeDelCloseMixin):\n    \"\"\"Container for a process which would allow for persistent communication\n    \"\"\"\n\n    def __init__(self, cmd, path=None, output_proc=None):\n        if not isinstance(cmd, list):\n            cmd = [cmd]\n        self.cmd = cmd\n        self.path = path\n        self.output_proc = output_proc if output_proc else readline_rstripped\n        self._process = None\n        self._stderr_out = None\n        self._stderr_out_fname = None\n\n    def _initialize(self):\n        lgr.debug(\"Initiating a new process for %s\" % repr(self))\n        lgr.log(5, \"Command: %s\" % self.cmd)\n        # according to the internet wisdom there is no easy way with subprocess\n        # while avoid deadlocks etc.  We would need to start a thread/subprocess\n        # to timeout etc\n        # kwargs = dict(bufsize=1, universal_newlines=True) if PY3 else {}\n        self._stderr_out, self._stderr_out_fname = tempfile.mkstemp()\n        self._process = subprocess.Popen(\n            self.cmd,\n            stdin=subprocess.PIPE,\n            stdout=subprocess.PIPE,\n            stderr=self._stderr_out,\n            env=GitRunner.get_git_environ_adjusted(),\n            cwd=self.path,\n            bufsize=1,\n            universal_newlines=True  # **kwargs\n        )\n\n    def _check_process(self, restart=False):\n        \"\"\"Check if the process was terminated and restart if restart\n\n        Returns\n        -------\n        bool\n          True if process was alive.\n        str\n          stderr if any recorded if was terminated\n        \"\"\"\n        process = self._process\n        ret = True\n        ret_stderr = None\n        if process and process.poll():\n            lgr.warning(\"Process %s was terminated with returncode %s\" % (process, process.returncode))\n            ret_stderr = self.close(return_stderr=True)\n            ret = False\n        if self._process is None and restart:\n            lgr.warning(\"Restarting the process due to previous failure\")\n            self._initialize()\n        return ret, ret_stderr\n\n    def __call__(self, cmds):\n        \"\"\"\n\n        Parameters\n        ----------\n        cmds : str or tuple or list of (str or tuple)\n\n        Returns\n        -------\n        str or list\n          Output received from process.  list in case if cmds was a list\n        \"\"\"\n        input_multiple = isinstance(cmds, list)\n        if not input_multiple:\n            cmds = [cmds]\n\n        output = [o for o in self.yield_(cmds)]\n        return output if input_multiple else output[0]\n\n    def yield_(self, cmds):\n        \"\"\"Same as __call__, but requires `cmds` to be an iterable\n\n        and yields results for each item.\"\"\"\n        for entry in cmds:\n            if not isinstance(entry, str):\n                entry = ' '.join(entry)\n            yield self.proc1(entry)\n\n    def proc1(self, arg):\n        \"\"\"Same as __call__, but only takes a single command argument\n\n        and returns a single result.\n        \"\"\"\n        # TODO: add checks -- may be process died off and needs to be reinitiated\n        if not self._process:\n            self._initialize()\n\n        entry = arg + '\\n'\n        lgr.log(5, \"Sending %r to batched command %s\" % (entry, self))\n        # apparently communicate is just a one time show\n        # stdout, stderr = self._process.communicate(entry)\n        # according to the internet wisdom there is no easy way with subprocess\n        self._check_process(restart=True)\n        process = self._process  # _check_process might have restarted it\n        process.stdin.write(entry)\n        process.stdin.flush()\n        lgr.log(5, \"Done sending.\")\n        still_alive, stderr = self._check_process(restart=False)\n        # TODO: we might want to handle still_alive, e.g. to allow for\n        #       a number of restarts/resends, but it should be per command\n        #       since for some we cannot just resend the same query. But if\n        #       it is just a \"get\"er - we could resend it few times\n        # The default output_proc expects a single line output.\n        # TODO: timeouts etc\n        stdout = assure_unicode(self.output_proc(process.stdout)) \\\n            if not process.stdout.closed else None\n        if stderr:\n            lgr.warning(\"Received output in stderr: %r\", stderr)\n        lgr.log(5, \"Received output: %r\" % stdout)\n        return stdout\n\n    def close(self, return_stderr=False):\n        \"\"\"Close communication and wait for process to terminate\n\n        Returns\n        -------\n        str\n          stderr output if return_stderr and stderr file was there.\n          None otherwise\n        \"\"\"\n        ret = None\n        if self._stderr_out:\n            # close possibly still open fd\n            os.fdopen(self._stderr_out).close()\n            self._stderr_out = None\n        if self._process:\n            process = self._process\n            lgr.debug(\n                \"Closing stdin of %s and waiting process to finish\", process)\n            process.stdin.close()\n            process.stdout.close()\n            process.wait()\n            self._process = None\n            lgr.debug(\"Process %s has finished\", process)\n        if self._stderr_out_fname and os.path.exists(self._stderr_out_fname):\n            if return_stderr:\n                with open(self._stderr_out_fname, 'r') as f:\n                    ret = f.read()\n            # remove the file where we kept dumping stderr\n            unlink(self._stderr_out_fname)\n            self._stderr_out_fname = None\n        return ret\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Renaming `poll_period` to `poll_latency` Causes Multiple Values for `path` Argument

**Description:**
After renaming the `poll_period` parameter to `poll_latency` in the `run` method of `WitlessRunner`, certain function calls are resulting in a `TypeError`. Specifically, when attempting to pass `poll_latency` as a keyword argument, the function receives multiple values for the `path` parameter, leading to unexpected errors.

**Example Buggy Code:**
```python
runner.run(cmd, proc_stdout=outproc, poll_latency=0.1)
```

**Error Message:**
```
TypeError: run() got multiple values for argument 'path'
```

**Expected Behavior:**
The `run` method should accept `poll_latency` as a valid keyword argument without causing any conflicts or multiple assignments of other parameters like `path`. Functions calling `run` with `poll_latency` should execute successfully without raising errors related to argument values.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/cmd.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""
Wrapper for command and function calls, allowing for dry runs and output handling

"""

import time
import subprocess
import sys
import logging
import os
import atexit
import functools
import tempfile

from collections import OrderedDict
from .support import path as op
from .consts import GIT_SSH_COMMAND
from .dochelpers import exc_str
from .support.exceptions import CommandError
from .support.protocol import (
    NullProtocol,
    ExecutionTimeProtocol,
    ExecutionTimeExternalsProtocol,
)
from .utils import (
    on_windows,
    get_tempfile_kwargs,
    assure_unicode,
    assure_bytes,
    unlink,
    auto_repr,
    split_cmdline,
)
from .dochelpers import borrowdoc

lgr = logging.getLogger('datalad.cmd')

# In python3 to split byte stream on newline, it must be bytes
linesep_bytes = os.linesep.encode()

_TEMP_std = sys.stdout, sys.stderr
# To be used in the temp file name to distinguish the ones we create
# in Runner so we take care about their removal, in contrast to those
# which might be created outside and passed into Runner
_MAGICAL_OUTPUT_MARKER = "_runneroutput_"

from io import IOBase as file_class


def _decide_to_log(v):
    """Hacky workaround for now so we could specify per each which to
    log online and which to the log"""
    if isinstance(v, bool) or callable(v):
        return v
    elif v in {'online'}:
        return True
    elif v in {'offline'}:
        return False
    else:
        raise ValueError("can be bool, callable, 'online' or 'offline'")


def _get_output_stream(log_std, false_value):
    """Helper to prepare output stream for Popen and use file for 'offline'

    Necessary to avoid lockdowns when both stdout and stderr are pipes
    """
    if log_std:
        if log_std == 'offline':
            # we will open a temporary file

            tf = tempfile.mktemp(
                **get_tempfile_kwargs({}, prefix=_MAGICAL_OUTPUT_MARKER)
            )
            return open(tf, 'w')  # XXX PY3 should be 'b' may be?
        else:
            return subprocess.PIPE
    else:
        return false_value


def _cleanup_output(stream, std):
    if isinstance(stream, file_class) and \
        _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):
        if not stream.closed:
            stream.close()
        if op.exists(stream.name):
            unlink(stream.name)
    elif stream == subprocess.PIPE:
        std.close()


def kill_output(output):
    """Helper for WitlessRunner to swallow all output and neither
    relay it to the parent process's stdout, nor provide it as the
    return value of WitlessRunner.run().
    """
    return b'', 0


def capture_output(output):
    """Helper for WitlessRunner to capture all output and
    provide it as the return value of WitlessRunner.run().
    """
    return output, 0


class WitlessRunner(object):
    """Minimal Runner with support for online command output processing

    It aims to be as simple as possible, providing only essential
    functionality. Derived classes should be used for additional
    specializations and convenience features.
    """
    __slots__ = ['cwd', 'env']

    def __init__(self, cwd=None, env=None):
        """
        Parameters
        ----------
        cwd : path-like, optional
          If given, commands are executed with this path as PWD,
          the PWD of the parent process is used otherwise.
        env : dict, optional
          Environment to be passed to subprocess.Popen(). If `cwd`
          was given, 'PWD' in the environment is set to its value.
          This must be a complete environment definition, no values
          from the current environment will be inherited.
        """
        self.env = env.copy() if env else None
        # stringify to support Path instances on PY35
        self.cwd = str(cwd) if cwd is not None else None
        if self.cwd and env is not None:
            # if CWD was provided, we must not make it conflict with
            # a potential PWD setting
            self.env['PWD'] = self.cwd

    def run(self, cmd, proc_stdout=None, proc_stderr=None, stdin=None,
            poll_period=0.1):
        """Execute a command and communicate with it.

        Parameters
        ----------
        cmd : list
          Sequence of program arguments. Passing a single string means
          that it is simply the name of the program, no complex shell
          commands are supported.
        proc_stdout : callable, optional
          By default no stdout is captured, but relayed to the parent
          process's stdout. If given, all stdout is passed as a byte-string
          to this callable, in the chunks it was received by polling the
          processing. The callable may transform it in any way, its output
          (byte-string) is concatenated and provided as stdout return value.
          The helper functions 'kill_output' and 'capture_output' are
          provided to either swallow all output (and not relay it to the
          parent) or to capture all output and provide it as the return
          value.
        proc_stderr : callable, optional
          Like proc_stdout, but for stderr.
        stdin : byte stream, optional
          File descriptor like, used as stdin for the process. Passed
          verbatim to subprocess.Popen().
        poll_latency : float, optional
          Shortest interval at which the running process is queried for
          output (in seconds). Any potential output processing will increase
          the effective interval.

        Returns
        -------
        stdout, stderr
          Unicode string with the cumulative standard output and error
          of the process.

        Raises
        ------
        CommandError
          On execution failure (non-zero exit code) this exception is
          raised which provides the command (cmd), stdout, stderr,
          exit code (status), and a message identifying the failed
          command, as properties.
        """
        proc_out = (proc_stdout, proc_stderr)
        try:
            lgr.log(8, "Start running %r", cmd)
            process = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE if proc_stdout else None,
                stderr=subprocess.PIPE if proc_stderr else None,
                shell=False,
                cwd=self.cwd,
                env=self.env,
                stdin=stdin,
                # intermediate reports are never decoded anyways
                # from PY37 onwards
                #text=False,
                universal_newlines=False,
            )
        except Exception as e:
            lgr.log(11, "Failed to start %r%r: %s" %
                    (cmd,
                     (" under %r" % self.cwd) if self.cwd else '',
                     exc_str(e)))
            raise

        try:
            out = [[], []]
            unprocessed = [None, None]
            # make sure to run this loop at least once, even if the
            # process is already dead
            keep_going = True

            def _handle_output(u, o, p):
                processed, unprocessed_len = proc(u)
                if processed:
                    o.append(processed)
                return u[-(unprocessed_len):] if unprocessed_len else None

            while process.poll() is None or keep_going:
                # one last read?
                keep_going = process.returncode is None
                # get a chunk of output for the specific period
                # of time
                pout = [
                    # read whatever is available, should not block
                    o.read1(-1) if p else None
                    for o, p in zip(
                        (process.stdout, process.stderr),
                        proc_out)
                ]

                for i, (o, proc) in enumerate(zip(pout, proc_out)):
                    if not o:
                        # nothing read
                        continue
                    if proc is None:
                        # no index update needed, can never change, hence no unprocessed
                        # output either
                        out[i].append(o)
                        continue
                    # make sure to feed back any unprocessed stuff
                    buffer = unprocessed[i] + o if unprocessed[i] else o
                    # engage output processor
                    unprocessed[i] = _handle_output(buffer, out[i], proc)
                time.sleep(poll_period)

            # obtain exit code
            status = process.poll()

            # decode bytes to string
            out = tuple(b''.join(o).decode('utf-8') if o else '' for o in out)

            if status not in [0, None]:
                msg = "Failed to run %r%s." % (
                    cmd,
                    (" under %r" % self.cwd) if self.cwd else '',
                )
                raise CommandError(
                    cmd=str(cmd),
                    msg=msg,
                    code=status,
                    stdout=out[0],
                    stderr=out[1],
                )
            else:
                lgr.log(8, "Finished running %r with status %s", cmd, status)

        except CommandError:
            # do not bother with reacting to "regular" CommandError
            # exceptions.  Somehow if we also terminate here for them
            # some processes elsewhere might stall:
            # see https://github.com/datalad/datalad/pull/3794
            raise

        except BaseException as exc:
            exc_info = sys.exc_info()
            # KeyboardInterrupt is subclass of BaseException
            lgr.debug("Terminating process for %s upon exception: %s",
                      cmd, exc_str(exc))
            try:
                # there are still possible (although unlikely) cases when
                # we fail to interrupt but we
                # should not crash if we fail to terminate the process
                process.terminate()
            except BaseException as exc2:
                lgr.warning("Failed to terminate process for %s: %s",
                            cmd, exc_str(exc2))
            raise exc_info[1]

        return out


class Runner(WitlessRunner):
    """Provides a wrapper for calling functions and commands.

    An object of this class provides a methods that calls shell commands or
    python functions, allowing for protocolling the calls and output handling.

    Outputs (stdout and stderr) can be either logged or streamed to system's
    stdout/stderr during execution.
    This can be enabled or disabled for both of them independently.
    Additionally, a protocol object can be a used with the Runner. Such a
    protocol has to implement datalad.support.protocol.ProtocolInterface, is
    able to record calls and allows for dry runs.
    """

    __slots__ = ['commands', 'dry', 'cwd', 'env', 'protocol',
                 '_log_opts']

    def __init__(self, cwd=None, env=None, protocol=None, log_outputs=None):
        """
        Parameters
        ----------
        cwd: string, optional
             Base current working directory for commands.  Could be overridden
             per run call via cwd option
        env: dict, optional
             Custom environment to use for calls. Could be overridden per run
             call via env option
        protocol: ProtocolInterface
             Protocol object to write to.
        log_outputs : bool, optional
             Switch to instruct whether outputs should be logged or not.  If not
             set (default), config 'datalad.log.outputs' would be consulted
        """
        super(Runner, self).__init__(cwd=cwd, env=env)

        if protocol is None:
            # TODO: config cmd.protocol = null
            protocol_str = os.environ.get('DATALAD_CMD_PROTOCOL', 'null')
            protocol = {
                'externals-time': ExecutionTimeExternalsProtocol,
                'time': ExecutionTimeProtocol,
                'null': NullProtocol
            }[protocol_str]()
            if protocol_str != 'null':
                # we need to dump it into a file at the end
                # TODO: config cmd.protocol_prefix = protocol
                filename = '%s-%s.log' % (
                    os.environ.get('DATALAD_CMD_PROTOCOL_PREFIX', 'protocol'),
                    id(self)
                )
                atexit.register(functools.partial(protocol.write_to_file, filename))

        self.protocol = protocol
        # Various options for logging
        self._log_opts = {}
        # we don't know yet whether we need to log every output or not
        if log_outputs is not None:
            self._log_opts['outputs'] = log_outputs

    def __call__(self, cmd, *args, **kwargs):
        """Convenience method

        This will call run() or call() depending on the kind of `cmd`.
        If `cmd` is a string it is interpreted as the to be executed command.
        Otherwise it is expected to be a callable.
        Any other argument is passed to the respective method.

        Parameters
        ----------
        cmd: str or callable
           command string to be executed via shell or callable to be called.

        `*args`:
        `**kwargs`:
           see Runner.run() and Runner.call() for available arguments.

        Raises
        ------
        TypeError
          if cmd is neither a string nor a callable.
        """

        if isinstance(cmd, str) or isinstance(cmd, list):
            return self.run(cmd, *args, **kwargs)
        elif callable(cmd):
            return self.call(cmd, *args, **kwargs)
        else:
            raise TypeError("Argument 'command' is neither a string, "
                            "nor a list nor a callable.")

    def _opt_env_adapter(v):
        """If value is a string, split by ,"""
        if v:
            if v.isdigit():
                log_env = bool(int(v))
            else:
                log_env = v.split(',')
            return log_env
        else:
            return False

    _LOG_OPTS_ADAPTERS = OrderedDict([
        ('outputs', None),
        ('cwd', None),
        ('env', _opt_env_adapter),
        ('stdin', None),
    ])

    def _get_log_setting(self, opt, default=False):
        try:
            return self._log_opts[opt]
        except KeyError:
            try:
                from . import cfg
            except ImportError:
                return default
            adapter = self._LOG_OPTS_ADAPTERS.get(opt, None)
            self._log_opts[opt] = \
                (cfg.getbool if not adapter else cfg.get_value)(
                    'datalad.log.cmd', opt, default=default)
            if adapter:
                self._log_opts[opt] = adapter(self._log_opts[opt])
            return self._log_opts[opt]

    @property
    def log_outputs(self):
        return self._get_log_setting('outputs')

    @property
    def log_cwd(self):
        return self._get_log_setting('cwd')

    @property
    def log_stdin(self):
        return self._get_log_setting('stdin')

    @property
    def log_env(self):
        return self._get_log_setting('env')

    # Two helpers to encapsulate formatting/output
    def _log_out(self, line):
        if line and self.log_outputs:
            self.log("stdout| " + line.rstrip('\n'))

    def _log_err(self, line, expected=False):
        if line and self.log_outputs:
            self.log("stderr| " + line.rstrip('\n'),
                     level={True: 9,
                            False: 11}[expected])

    def _get_output_online(self, proc,
                           log_stdout, log_stderr,
                           outputstream, errstream,
                           expect_stderr=False, expect_fail=False):
        """

        If log_stdout or log_stderr are callables, they will be given a read
        line to be processed, and return processed result.  So if they need to
        'swallow' the line from being logged, should just return None

        Parameters
        ----------
        proc
        log_stdout: bool or callable or 'online' or 'offline'
        log_stderr: : bool or callable or 'online' or 'offline'
          If any of those 'offline', we would call proc.communicate at the
          end to grab possibly outstanding output from it
        expect_stderr
        expect_fail

        Returns
        -------

        """
        stdout, stderr = bytes(), bytes()

        log_stdout_ = _decide_to_log(log_stdout)
        log_stderr_ = _decide_to_log(log_stderr)
        log_stdout_is_callable = callable(log_stdout_)
        log_stderr_is_callable = callable(log_stderr_)

        # arguments to be passed into _process_one_line
        stdout_args = (
                'stdout',
                proc, log_stdout_, log_stdout_is_callable
        )
        stderr_args = (
                'stderr',
                proc, log_stderr_, log_stderr_is_callable,
                expect_stderr or expect_fail
        )

        while proc.poll() is None:
            # see for a possibly useful approach to processing output
            # in another thread http://codereview.stackexchange.com/a/17959
            # current problem is that if there is no output on stderr
            # it stalls
            # Monitor if anything was output and if nothing, sleep a bit
            stdout_, stderr_ = None, None
            if log_stdout_:
                stdout_ = self._process_one_line(*stdout_args)
                stdout += stdout_
            if log_stderr_:
                stderr_ = self._process_one_line(*stderr_args)
                stderr += stderr_
            if stdout_ is None and stderr_ is None:
                # no output was really produced, so sleep a tiny bit
                time.sleep(0.001)

        # Handle possible remaining output
        if log_stdout_ and log_stderr_:
            # If Popen was called with more than two pipes, calling
            # communicate() after we partially read the stream will return
            # empty output.
            stdout += self._process_remaining_output(
                outputstream, proc.stdout.read(), *stdout_args)
            stderr += self._process_remaining_output(
                errstream, proc.stderr.read(), *stderr_args)
        stdout_, stderr_ = proc.communicate()
        # ??? should we condition it on log_stdout in {'offline'} ???
        stdout += self._process_remaining_output(outputstream, stdout_, *stdout_args)
        stderr += self._process_remaining_output(errstream, stderr_, *stderr_args)

        return stdout, stderr

    def _process_remaining_output(self, stream, out_, *pargs):
        """Helper to process output which might have been obtained from popen or
        should be loaded from file"""
        out = bytes()
        if isinstance(stream, file_class) and \
                _MAGICAL_OUTPUT_MARKER in getattr(stream, 'name', ''):
            assert out_ is None, "should have gone into a file"
            if not stream.closed:
                stream.close()
            with open(stream.name, 'rb') as f:
                for line in f:
                    out += self._process_one_line(*pargs, line=line)
        else:
            if out_:
                # resolving a once in a while failing test #2185
                if isinstance(out_, str):
                    out_ = out_.encode('utf-8')
                for line in out_.split(linesep_bytes):
                    out += self._process_one_line(
                        *pargs, line=line, suf=linesep_bytes)
        return out

    def _process_one_line(self, out_type, proc, log_, log_is_callable,
                          expected=False, line=None, suf=None):
        if line is None:
            lgr.log(3, "Reading line from %s", out_type)
            line = {'stdout': proc.stdout, 'stderr': proc.stderr}[out_type].readline()
        else:
            lgr.log(3, "Processing provided line")
        if line and log_is_callable:
            # Let it be processed
            line = log_(assure_unicode(line))
            if line is not None:
                # we are working with binary type here
                line = assure_bytes(line)
        if line:
            if out_type == 'stdout':
                self._log_out(assure_unicode(line))
            elif out_type == 'stderr':
                self._log_err(line.decode('utf-8'),
                              expected)
            else:  # pragma: no cover
                raise RuntimeError("must not get here")
            return (line + suf) if suf else line
        # it was output already directly but for code to work, return ""
        return bytes()

    def run(self, cmd, log_stdout=True, log_stderr=True, log_online=False,
            expect_stderr=False, expect_fail=False,
            cwd=None, env=None, shell=None, stdin=None):
        """Runs the command `cmd` using shell.

        In case of dry-mode `cmd` is just added to `commands` and it is
        actually executed otherwise.
        Allows for separately logging stdout and stderr  or streaming it to
        system's stdout or stderr respectively.

        Note: Using a string as `cmd` and shell=True allows for piping,
              multiple commands, etc., but that implies split_cmdline() is not
              used. This is considered to be a security hazard.
              So be careful with input.

        Parameters
        ----------
        cmd : str, list
          String (or list) defining the command call.  No shell is used if cmd
          is specified as a list

        log_stdout: bool, optional
            If True, stdout is logged. Goes to sys.stdout otherwise.

        log_stderr: bool, optional
            If True, stderr is logged. Goes to sys.stderr otherwise.

        log_online: bool, optional
            Whether to log as output comes in.  Setting to True is preferable
            for running user-invoked actions to provide timely output

        expect_stderr: bool, optional
            Normally, having stderr output is a signal of a problem and thus it
            gets logged at level 11.  But some utilities, e.g. wget, use
            stderr for their progress output.  Whenever such output is expected,
            set it to True and output will be logged at level 9 unless
            exit status is non-0 (in non-online mode only, in online -- would
            log at 9)

        expect_fail: bool, optional
            Normally, if command exits with non-0 status, it is considered an
            error and logged at level 11 (above DEBUG). But if the call intended
            for checking routine, such messages are usually not needed, thus
            it will be logged at level 9.

        cwd : string, optional
            Directory under which run the command (passed to Popen)

        env : string, optional
            Custom environment to pass

        shell: bool, optional
            Run command in a shell.  If not specified, then it runs in a shell
            only if command is specified as a string (not a list)

        stdin: file descriptor
            input stream to connect to stdin of the process.

        Returns
        -------
        (stdout, stderr) - bytes!

        Raises
        ------
        CommandError
           if command's exitcode wasn't 0 or None. exitcode is passed to
           CommandError's `code`-field. Command's stdout and stderr are stored
           in CommandError's `stdout` and `stderr` fields respectively.
        """
        outputstream = _get_output_stream(log_stdout, sys.stdout)
        errstream = _get_output_stream(log_stderr, sys.stderr)

        popen_env = env or self.env
        popen_cwd = cwd or self.cwd

        if popen_cwd and popen_env and 'PWD' in popen_env:
            # we must have inherited PWD, but cwd was provided, so we must
            # adjust it
            popen_env = popen_env.copy()  # to avoid side-effects
            popen_env['PWD'] = popen_cwd

        # TODO: if outputstream is sys.stdout and that one is set to StringIO
        #       we have to "shim" it with something providing fileno().
        # This happens when we do not swallow outputs, while allowing nosetest's
        # StringIO to be provided as stdout, crashing the Popen requiring
        # fileno().  In out swallow_outputs, we just use temporary files
        # to overcome this problem.
        # For now necessary test code should be wrapped into swallow_outputs cm
        # to avoid the problem
        log_msgs = ["Running: %s"]
        log_args = [cmd]
        if self.log_cwd:
            log_msgs += ['cwd=%r']
            log_args += [popen_cwd]
        if self.log_stdin:
            log_msgs += ['stdin=%r']
            log_args += [stdin]
        log_env = self.log_env
        if log_env and popen_env:
            log_msgs += ["env=%r"]
            log_args.append(
                popen_env if log_env is True
                else {k: popen_env[k] for k in log_env if k in popen_env}
            )
        log_msg = '\n'.join(log_msgs)
        self.log(log_msg, *log_args)

        if self.protocol.do_execute_ext_commands:

            if shell is None:
                shell = isinstance(cmd, str)

            if self.protocol.records_ext_commands:
                prot_exc = None
                prot_id = self.protocol.start_section(
                    split_cmdline(cmd)
                    if isinstance(cmd, str)
                    else cmd)
            try:
                proc = subprocess.Popen(cmd,
                                        stdout=outputstream,
                                        stderr=errstream,
                                        shell=shell,
                                        cwd=popen_cwd,
                                        env=popen_env,
                                        stdin=stdin)

            except Exception as e:
                prot_exc = e
                lgr.log(11, "Failed to start %r%r: %s" %
                        (cmd, " under %r" % cwd if cwd else '', exc_str(e)))
                raise

            finally:
                if self.protocol.records_ext_commands:
                    self.protocol.end_section(prot_id, prot_exc)

            try:
                if log_online:
                    out = self._get_output_online(proc,
                                                  log_stdout, log_stderr,
                                                  outputstream, errstream,
                                                  expect_stderr=expect_stderr,
                                                  expect_fail=expect_fail)
                else:
                    out = proc.communicate()

                # Decoding was delayed to this point
                def decode_if_not_None(x):
                    return "" if x is None else bytes.decode(x)
                out = tuple(map(decode_if_not_None, out))

                status = proc.poll()

                # needs to be done after we know status
                if not log_online:
                    self._log_out(out[0])
                    if status not in [0, None]:
                        self._log_err(out[1], expected=expect_fail)
                    else:
                        # as directed
                        self._log_err(out[1], expected=expect_stderr)

                if status not in [0, None]:
                    msg = "Failed to run %r%s. Exit code=%d.%s%s" \
                        % (cmd, " under %r" % (popen_cwd), status,
                           "" if log_online else " out=%s" % out[0],
                           "" if log_online else " err=%s" % out[1])
                    lgr.log(9 if expect_fail else 11, msg)
                    raise CommandError(str(cmd), msg, status, out[0], out[1])
                else:
                    self.log("Finished running %r with status %s" % (cmd, status),
                             level=8)

            except CommandError:
                # do not bother with reacting to "regular" CommandError
                # exceptions.  Somehow if we also terminate here for them
                # some processes elsewhere might stall:
                # see https://github.com/datalad/datalad/pull/3794
                raise

            except BaseException as exc:
                exc_info = sys.exc_info()
                # KeyboardInterrupt is subclass of BaseException
                lgr.debug("Terminating process for %s upon exception: %s",
                          cmd, exc_str(exc))
                try:
                    # there are still possible (although unlikely) cases when
                    # we fail to interrupt but we
                    # should not crash if we fail to terminate the process
                    proc.terminate()
                except BaseException as exc2:
                    lgr.warning("Failed to terminate process for %s: %s",
                                cmd, exc_str(exc2))
                raise exc_info[1]

            finally:
                # Those streams are for us to close if we asked for a PIPE
                # TODO -- assure closing the files
                _cleanup_output(outputstream, proc.stdout)
                _cleanup_output(errstream, proc.stderr)

        else:
            if self.protocol.records_ext_commands:
                self.protocol.add_section(split_cmdline(cmd)
                                          if isinstance(cmd, str)
                                          else cmd, None)
            out = ("DRY", "DRY")

        return out

    def call(self, f, *args, **kwargs):
        """Helper to unify collection of logging all "dry" actions.

        Calls `f` if `Runner`-object is not in dry-mode. Adds `f` along with
        its arguments to `commands` otherwise.

        Parameters
        ----------
        f: callable
        """
        if self.protocol.do_execute_callables:
            if self.protocol.records_callables:
                prot_exc = None
                prot_id = self.protocol.start_section(
                    [str(f), "args=%s" % str(args), "kwargs=%s" % str(kwargs)])

            try:
                return f(*args, **kwargs)
            except Exception as e:
                prot_exc = e
                raise
            finally:
                if self.protocol.records_callables:
                    self.protocol.end_section(prot_id, prot_exc)
        else:
            if self.protocol.records_callables:
                self.protocol.add_section(
                    [str(f), "args=%s" % str(args), "kwargs=%s" % str(kwargs)],
                    None)

    def log(self, msg, *args, **kwargs):
        """log helper

        Logs at level 9 by default and adds "Protocol:"-prefix in order to
        log the used protocol.
        """
        level = kwargs.pop('level', 9)
        if isinstance(self.protocol, NullProtocol):
            lgr.log(level, msg, *args, **kwargs)
        else:
            if args:
                msg = msg % args
            lgr.log(level, "{%s} %s" % (
                self.protocol.__class__.__name__, msg)
            )


class GitRunner(Runner):
    """
    Runner to be used to run git and git annex commands

    Overloads the runner class to check & update GIT_DIR and
    GIT_WORK_TREE environment variables set to the absolute path
    if is defined and is relative path
    """
    _GIT_PATH = None

    @borrowdoc(Runner)
    def __init__(self, *args, **kwargs):
        super(GitRunner, self).__init__(*args, **kwargs)
        self._check_git_path()

    @staticmethod
    def _check_git_path():
        """If using bundled git-annex, we would like to use bundled with it git

        Thus we will store _GIT_PATH a path to git in the same directory as annex
        if found.  If it is empty (but not None), we do nothing
        """
        if GitRunner._GIT_PATH is None:
            from distutils.spawn import find_executable
            # with all the nesting of config and this runner, cannot use our
            # cfg here, so will resort to dark magic of environment options
            if (os.environ.get('DATALAD_USE_DEFAULT_GIT', '0').lower()
                    in ('1', 'on', 'true', 'yes')):
                git_fpath = find_executable("git")
                if git_fpath:
                    GitRunner._GIT_PATH = ''
                    lgr.log(9, "Will use default git %s", git_fpath)
                    return  # we are done - there is a default git avail.
                # if not -- we will look for a bundled one
            GitRunner._GIT_PATH = GitRunner._get_bundled_path()
            lgr.log(9, "Will use git under %r (no adjustments to PATH if empty "
                       "string)", GitRunner._GIT_PATH)
            assert(GitRunner._GIT_PATH is not None)  # we made the decision!

    @staticmethod
    def _get_bundled_path():
        from distutils.spawn import find_executable
        annex_fpath = find_executable("git-annex")
        if not annex_fpath:
            # not sure how to live further anyways! ;)
            alongside = False
        else:
            annex_path = op.dirname(op.realpath(annex_fpath))
            alongside = op.lexists(op.join(annex_path, 'git'))
        return annex_path if alongside else ''

    @staticmethod
    def get_git_environ_adjusted(env=None):
        """
        Replaces GIT_DIR and GIT_WORK_TREE with absolute paths if relative path and defined
        """
        # if env set copy else get os environment
        git_env = env.copy() if env else os.environ.copy()
        if GitRunner._GIT_PATH:
            git_env['PATH'] = op.pathsep.join([GitRunner._GIT_PATH, git_env['PATH']]) \
                if 'PATH' in git_env \
                else GitRunner._GIT_PATH

        for varstring in ['GIT_DIR', 'GIT_WORK_TREE']:
            var = git_env.get(varstring)
            if var:                                    # if env variable set
                if not op.isabs(var):                   # and it's a relative path
                    git_env[varstring] = op.abspath(var)  # to absolute path
                    lgr.log(9, "Updated %s to %s", varstring, git_env[varstring])

        if 'GIT_SSH_COMMAND' not in git_env:
            git_env['GIT_SSH_COMMAND'] = GIT_SSH_COMMAND
            git_env['GIT_SSH_VARIANT'] = 'ssh'

        return git_env

    def run(self, cmd, env=None, *args, **kwargs):
        out, err = super(GitRunner, self).run(
            cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
        # All communication here will be returned as unicode
        # TODO: do that instead within the super's run!
        return assure_unicode(out), assure_unicode(err)


def readline_rstripped(stdout):
    #return iter(stdout.readline, b'').next().rstrip()
    return stdout.readline().rstrip()


class SafeDelCloseMixin(object):
    """A helper class to use where __del__ would call .close() which might
    fail if "too late in GC game"
    """
    def __del__(self):
        try:
            self.close()
        except TypeError:
            if os.fdopen is None or lgr.debug is None:
                # if we are late in the game and things already gc'ed in py3,
                # it is Ok
                return
            raise


@auto_repr
class BatchedCommand(SafeDelCloseMixin):
    """Container for a process which would allow for persistent communication
    """

    def __init__(self, cmd, path=None, output_proc=None):
        if not isinstance(cmd, list):
            cmd = [cmd]
        self.cmd = cmd
        self.path = path
        self.output_proc = output_proc if output_proc else readline_rstripped
        self._process = None
        self._stderr_out = None
        self._stderr_out_fname = None

    def _initialize(self):
        lgr.debug("Initiating a new process for %s" % repr(self))
        lgr.log(5, "Command: %s" % self.cmd)
        # according to the internet wisdom there is no easy way with subprocess
        # while avoid deadlocks etc.  We would need to start a thread/subprocess
        # to timeout etc
        # kwargs = dict(bufsize=1, universal_newlines=True) if PY3 else {}
        self._stderr_out, self._stderr_out_fname = tempfile.mkstemp()
        self._process = subprocess.Popen(
            self.cmd,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=self._stderr_out,
            env=GitRunner.get_git_environ_adjusted(),
            cwd=self.path,
            bufsize=1,
            universal_newlines=True  # **kwargs
        )

    def _check_process(self, restart=False):
        """Check if the process was terminated and restart if restart

        Returns
        -------
        bool
          True if process was alive.
        str
          stderr if any recorded if was terminated
        """
        process = self._process
        ret = True
        ret_stderr = None
        if process and process.poll():
            lgr.warning("Process %s was terminated with returncode %s" % (process, process.returncode))
            ret_stderr = self.close(return_stderr=True)
            ret = False
        if self._process is None and restart:
            lgr.warning("Restarting the process due to previous failure")
            self._initialize()
        return ret, ret_stderr

    def __call__(self, cmds):
        """

        Parameters
        ----------
        cmds : str or tuple or list of (str or tuple)

        Returns
        -------
        str or list
          Output received from process.  list in case if cmds was a list
        """
        input_multiple = isinstance(cmds, list)
        if not input_multiple:
            cmds = [cmds]

        output = [o for o in self.yield_(cmds)]
        return output if input_multiple else output[0]

    def yield_(self, cmds):
        """Same as __call__, but requires `cmds` to be an iterable

        and yields results for each item."""
        for entry in cmds:
            if not isinstance(entry, str):
                entry = ' '.join(entry)
            yield self.proc1(entry)

    def proc1(self, arg):
        """Same as __call__, but only takes a single command argument

        and returns a single result.
        """
        # TODO: add checks -- may be process died off and needs to be reinitiated
        if not self._process:
            self._initialize()

        entry = arg + '\n'
        lgr.log(5, "Sending %r to batched command %s" % (entry, self))
        # apparently communicate is just a one time show
        # stdout, stderr = self._process.communicate(entry)
        # according to the internet wisdom there is no easy way with subprocess
        self._check_process(restart=True)
        process = self._process  # _check_process might have restarted it
        process.stdin.write(entry)
        process.stdin.flush()
        lgr.log(5, "Done sending.")
        still_alive, stderr = self._check_process(restart=False)
        # TODO: we might want to handle still_alive, e.g. to allow for
        #       a number of restarts/resends, but it should be per command
        #       since for some we cannot just resend the same query. But if
        #       it is just a "get"er - we could resend it few times
        # The default output_proc expects a single line output.
        # TODO: timeouts etc
        stdout = assure_unicode(self.output_proc(process.stdout)) \
            if not process.stdout.closed else None
        if stderr:
            lgr.warning("Received output in stderr: %r", stderr)
        lgr.log(5, "Received output: %r" % stdout)
        return stdout

    def close(self, return_stderr=False):
        """Close communication and wait for process to terminate

        Returns
        -------
        str
          stderr output if return_stderr and stderr file was there.
          None otherwise
        """
        ret = None
        if self._stderr_out:
            # close possibly still open fd
            os.fdopen(self._stderr_out).close()
            self._stderr_out = None
        if self._process:
            process = self._process
            lgr.debug(
                "Closing stdin of %s and waiting process to finish", process)
            process.stdin.close()
            process.stdout.close()
            process.wait()
            self._process = None
            lgr.debug("Process %s has finished", process)
        if self._stderr_out_fname and os.path.exists(self._stderr_out_fname):
            if return_stderr:
                with open(self._stderr_out_fname, 'r') as f:
                    ret = f.read()
            # remove the file where we kept dumping stderr
            unlink(self._stderr_out_fname)
            self._stderr_out_fname = None
        return ret

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user