{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"DataLad aims to expose (scientific) data available online as a unified data\ndistribution with the convenience of git-annex repositories as a backend.\"\"\"\n\nimport atexit\n\nfrom .version import __version__\nfrom .log import lgr\n\n# Other imports are interspersed with lgr.debug to ease troubleshooting startup\n# delays etc.\nlgr.log(5, \"Instantiating config\")\nfrom .config import ConfigManager\ncfg = ConfigManager()\n\nlgr.log(5, \"Instantiating ssh manager\")\nfrom .support.sshconnector import SSHManager\nssh_manager = SSHManager()\natexit.register(ssh_manager.close)\n\n\ndef test(package='datalad', **kwargs):\n    \"\"\"A helper to run datalad's tests.  Requires numpy and nose\n\n    See numpy.testing.Tester -- **kwargs are passed into the\n    Tester().test call\n    \"\"\"\n    try:\n        from numpy.testing import Tester\n        Tester(package=package).test(**kwargs)\n        # we don't have any benchmarks atm\n        # bench = Tester().bench\n    except ImportError:\n        raise RuntimeError('Need numpy >= 1.2 for datalad.tests().  Nothing is done')\ntest.__test__ = False\n\n# Following fixtures are necessary at the top level __init__ for fixtures which\n# would cover all **/tests and not just datalad/tests/\n\n# To store settings which setup_package changes and teardown_package should return\n_test_states = {\n    'loglevel': None,\n    'DATALAD_LOGLEVEL': None,\n}\n\ndef setup_package():\n    import os\n\n    # To overcome pybuild overriding HOME but us possibly wanting our\n    # own HOME where we pre-setup git for testing (name, email)\n    if 'GIT_HOME' in os.environ:\n        os.environ['HOME'] = os.environ['GIT_HOME']\n\n    # To overcome pybuild by default defining http{,s}_proxy we would need\n    # to define them to e.g. empty value so it wouldn't bother touching them.\n    # But then haskell libraries do not digest empty value nicely, so we just\n    # pop them out from the environment\n    for ev in ('http_proxy', 'https_proxy'):\n        if ev in os.environ and not (os.environ[ev]):\n            lgr.debug(\"Removing %s from the environment since it is empty\", ev)\n            os.environ.pop(ev)\n\n    DATALAD_LOGLEVEL = os.environ.get('DATALAD_LOGLEVEL', None)\n    if DATALAD_LOGLEVEL is None:\n        # very very silent.  Tests introspecting logs should use\n        # swallow_logs(new_level=...)\n        _test_states['loglevel'] = lgr.getEffectiveLevel()\n        lgr.setLevel(100)\n\n        # And we should also set it within environ so underlying commands also stay silent\n        _test_states['DATALAD_LOGLEVEL'] = DATALAD_LOGLEVEL\n        os.environ['DATALAD_LOGLEVEL'] = '100'\n    else:\n        # We are not overriding them, since explicitly were asked to have some log level\n        _test_states['loglevel'] = None\n\n\ndef teardown_package():\n    import os\n    if os.environ.get('DATALAD_TESTS_NOTEARDOWN'):\n        return\n\n    if _test_states['loglevel'] is not None:\n        lgr.setLevel(_test_states['loglevel'])\n        if _test_states['DATALAD_LOGLEVEL'] is None:\n            os.environ.pop('DATALAD_LOGLEVEL')\n        else:\n            os.environ['DATALAD_LOGLEVEL'] = _test_states['DATALAD_LOGLEVEL']\n\n    from datalad.tests import _TEMP_PATHS_GENERATED\n    from datalad.tests.utils import rmtemp\n    if len(_TEMP_PATHS_GENERATED):\n        msg = \"Removing %d dirs/files: %s\" % (len(_TEMP_PATHS_GENERATED), ', '.join(_TEMP_PATHS_GENERATED))\n    else:\n        msg = \"Nothing to remove\"\n    lgr.debug(\"Teardown tests. \" + msg)\n    for path in _TEMP_PATHS_GENERATED:\n        rmtemp(path, ignore_errors=True)\n\nlgr.log(5, \"Done importing main __init__\")\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"DataLad aims to expose (scientific) data available online as a unified data\ndistribution with the convenience of git-annex repositories as a backend.\"\"\"\n\nimport atexit\n\nfrom .version import __version__\nfrom .log import lgr\n\n# Other imports are interspersed with lgr.debug to ease troubleshooting startup\n# delays etc.\nlgr.log(5, \"Instantiating config\")\nfrom .config import ConfigManager\ncfg = ConfigManager()\n\nlgr.log(5, \"Instantiating ssh manager\")\nfrom .support.sshconnector import SSHManager\nssh_manager = SSHManager()\natexit.register(ssh_manager.close, allow_fail=False)\n\n\ndef test(package='datalad', **kwargs):\n    \"\"\"A helper to run datalad's tests.  Requires numpy and nose\n\n    See numpy.testing.Tester -- **kwargs are passed into the\n    Tester().test call\n    \"\"\"\n    try:\n        from numpy.testing import Tester\n        Tester(package=package).test(**kwargs)\n        # we don't have any benchmarks atm\n        # bench = Tester().bench\n    except ImportError:\n        raise RuntimeError('Need numpy >= 1.2 for datalad.tests().  Nothing is done')\ntest.__test__ = False\n\n# Following fixtures are necessary at the top level __init__ for fixtures which\n# would cover all **/tests and not just datalad/tests/\n\n# To store settings which setup_package changes and teardown_package should return\n_test_states = {\n    'loglevel': None,\n    'DATALAD_LOGLEVEL': None,\n}\n\ndef setup_package():\n    import os\n\n    # To overcome pybuild overriding HOME but us possibly wanting our\n    # own HOME where we pre-setup git for testing (name, email)\n    if 'GIT_HOME' in os.environ:\n        os.environ['HOME'] = os.environ['GIT_HOME']\n\n    # To overcome pybuild by default defining http{,s}_proxy we would need\n    # to define them to e.g. empty value so it wouldn't bother touching them.\n    # But then haskell libraries do not digest empty value nicely, so we just\n    # pop them out from the environment\n    for ev in ('http_proxy', 'https_proxy'):\n        if ev in os.environ and not (os.environ[ev]):\n            lgr.debug(\"Removing %s from the environment since it is empty\", ev)\n            os.environ.pop(ev)\n\n    DATALAD_LOGLEVEL = os.environ.get('DATALAD_LOGLEVEL', None)\n    if DATALAD_LOGLEVEL is None:\n        # very very silent.  Tests introspecting logs should use\n        # swallow_logs(new_level=...)\n        _test_states['loglevel'] = lgr.getEffectiveLevel()\n        lgr.setLevel(100)\n\n        # And we should also set it within environ so underlying commands also stay silent\n        _test_states['DATALAD_LOGLEVEL'] = DATALAD_LOGLEVEL\n        os.environ['DATALAD_LOGLEVEL'] = '100'\n    else:\n        # We are not overriding them, since explicitly were asked to have some log level\n        _test_states['loglevel'] = None\n\n\ndef teardown_package():\n    import os\n    if os.environ.get('DATALAD_TESTS_NOTEARDOWN'):\n        return\n\n    if _test_states['loglevel'] is not None:\n        lgr.setLevel(_test_states['loglevel'])\n        if _test_states['DATALAD_LOGLEVEL'] is None:\n            os.environ.pop('DATALAD_LOGLEVEL')\n        else:\n            os.environ['DATALAD_LOGLEVEL'] = _test_states['DATALAD_LOGLEVEL']\n\n    from datalad.tests import _TEMP_PATHS_GENERATED\n    from datalad.tests.utils import rmtemp\n    if len(_TEMP_PATHS_GENERATED):\n        msg = \"Removing %d dirs/files: %s\" % (len(_TEMP_PATHS_GENERATED), ', '.join(_TEMP_PATHS_GENERATED))\n    else:\n        msg = \"Nothing to remove\"\n    lgr.debug(\"Teardown tests. \" + msg)\n    for path in _TEMP_PATHS_GENERATED:\n        rmtemp(path, ignore_errors=True)\n\nlgr.log(5, \"Done importing main __init__\")\n",
            "header": {
                "file": {
                    "path": "datalad/__init__.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "66d61e716",
                "new_commit_hash": "588b85d19",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/__init__.py"
            },
            "plus_file": {
                "path": "b/datalad/__init__.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 23,
                            "length": 7
                        },
                        "new_range": {
                            "start": 23,
                            "length": 7
                        },
                        "section": "cfg = ConfigManager()"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "lgr.log(5, \"Instantiating ssh manager\")",
                                "type": "context"
                            },
                            {
                                "content": "from .support.sshconnector import SSHManager",
                                "type": "context"
                            },
                            {
                                "content": "ssh_manager = SSHManager()",
                                "type": "context"
                            },
                            {
                                "content": "atexit.register(ssh_manager.close)",
                                "type": "deleted"
                            },
                            {
                                "content": "atexit.register(ssh_manager.close, allow_fail=False)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test(package='datalad', **kwargs):",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/__init__.py",
                            "type": "statement",
                            "name": "",
                            "content": "atexit.register(ssh_manager.close, allow_fail=False)",
                            "ast_type_str": "Expr",
                            "start_lineno": 26,
                            "end_lineno": 26,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/__init__.py",
                            "type": "statement",
                            "name": "",
                            "content": "atexit.register(ssh_manager.close)",
                            "ast_type_str": "Expr",
                            "start_lineno": 26,
                            "end_lineno": 26,
                            "parent": null
                        }
                    ]
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface to an ssh connection.\n\nAllows for connecting via ssh and keeping the connection open\n(by using a controlmaster), in order to perform several ssh commands or\ngit calls to a ssh remote without the need to reauthenticate.\n\"\"\"\n\nimport logging\nfrom subprocess import Popen\nfrom shlex import split as sh_split\nfrom os.path import isdir\n\nfrom datalad.support.network import RI, is_ssh\n\n# TODO: centralize AppDirs (=> datalad.config?)\nfrom appdirs import AppDirs\nfrom datalad.support.exceptions import CommandError\nfrom datalad.utils import not_supported_on_windows\nfrom datalad.utils import on_windows\nfrom datalad.utils import assure_dir\nfrom datalad.utils import auto_repr\nfrom datalad.cmd import Runner\n\nlgr = logging.getLogger('datalad.ssh')\n\n\n@auto_repr\nclass SSHConnection(object):\n    \"\"\"Representation of a (shared) ssh connection.\n    \"\"\"\n\n    def __init__(self, ctrl_path, host, port=None):\n        \"\"\"Create the connection.\n\n        This does not actually open the connection.\n        It's just its representation.\n\n        Parameters\n        ----------\n        ctrl_path: str\n          path to SSH controlmaster\n        host: str\n          host to connect to. This may include the user ( [user@]host )\n        port: str\n          port to connect over\n        \"\"\"\n        self._runner = None\n\n        # TODO: This may actually also contain \"user@host\".\n        #       So, better name instead of 'host'?\n        self.host = host\n        self.ctrl_path = ctrl_path + \":\" + port if port else ctrl_path\n        self.port = port\n        self.ctrl_options = [\"-o\", \"ControlPath=\" + self.ctrl_path]\n\n    def __call__(self, cmd):\n        \"\"\"Executes a command on the remote.\n\n        Parameters\n        ----------\n        cmd: list or str\n          command to run on the remote\n\n        Returns\n        -------\n        tuple of str\n          stdout, stderr of the command run.\n        \"\"\"\n\n        # TODO: Do we need to check for the connection to be open or just rely\n        # on possible ssh failing?\n\n        ssh_cmd = [\"ssh\"] + self.ctrl_options + [self.host] + cmd if isinstance(cmd, list) \\\n            else sh_split(cmd, posix=not on_windows)\n            # windows check currently not needed, but keep it as a reminder\n\n        # TODO: pass expect parameters from above?\n        # Hard to explain to toplevel users ... So for now, just set True\n        return self.runner.run(ssh_cmd, expect_fail=True, expect_stderr=True)\n\n    @property\n    def runner(self):\n        if self._runner is None:\n            self._runner = Runner()\n        return self._runner\n\n    def open(self):\n        \"\"\"Opens the connection.\n\n        In other words: Creates the SSH controlmaster to be used by this\n        connection, if it is not there already.\n        \"\"\"\n\n        # set control options\n        ctrl_options = [\"-o\", \"ControlMaster=auto\", \"-o\", \"ControlPersist=yes\"] + self.ctrl_options\n        # create ssh control master command\n        cmd = [\"ssh\"] + ctrl_options + [self.host, \"exit\"]\n\n        # start control master:\n        lgr.debug(\"Try starting control master by calling:\\n%s\" % cmd)\n        proc = Popen(cmd)\n        proc.communicate(input=\"\\n\")  # why the f.. this is necessary?\n\n    def close(self):\n        \"\"\"Closes the connection.\n        \"\"\"\n\n        # stop controlmaster:\n        cmd = [\"ssh\", \"-O\", \"stop\", \"-S\", self.ctrl_path, self.host]\n        try:\n            self.runner.run(cmd, expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if \"No such file or directory\" in e.stderr:\n                # nothing to clean up\n                pass\n            else:\n                raise\n\n    def copy(self, source, destination, recursive=False, preserve_attrs=False):\n        \"\"\"Copies source file/folder to destination on the remote.\n\n        Parameters\n        ----------\n        source: str or list\n          file/folder path(s) to copy from on local\n        destination: str\n          file/folder path to copy to on remote\n\n        Returns\n        -------\n        str\n          stdout, stderr of the copy operation.\n        \"\"\"\n\n        # add recursive, preserve_attributes flag if recursive, preserve_attrs set and create scp command\n        scp_options = self.ctrl_options + [\"-r\"] if recursive else self.ctrl_options\n        scp_options += [\"-p\"] if preserve_attrs else []\n        scp_cmd = [\"scp\"] + scp_options\n\n        # add source filepath(s) to scp command\n        scp_cmd += source if isinstance(source, list) \\\n            else [source]\n\n        # add destination path\n        scp_cmd += [self.host + \":\" + destination]\n        return self.runner.run(scp_cmd)\n\n\n@auto_repr\nclass SSHManager(object):\n    \"\"\"Keeps ssh connections to share. Serves singleton representation\n    per connection.\n    \"\"\"\n\n    def __init__(self):\n        not_supported_on_windows(\"TODO: Make this an abstraction to \"\n                                 \"interface platform dependent SSH\")\n\n        self._connections = dict()\n\n        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir\n        assure_dir(self.socket_dir)\n\n    def get_connection(self, url):\n        \"\"\"Get a singleton, representing a shared ssh connection to `url`\n\n        Parameters\n        ----------\n        url: str\n          ssh url\n\n        Returns\n        -------\n        SSHConnection\n        \"\"\"\n        # parse url:\n        sshri = RI(url)\n\n        if not is_ssh(sshri):\n            raise ValueError(\"Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax\".format(url))\n\n        # determine control master:\n        ctrl_path = \"%s/%s\" % (self.socket_dir, sshri.hostname)\n        if sshri.port:\n            ctrl_path += \":%s\" % sshri.port\n\n        # do we know it already?\n        if ctrl_path in self._connections:\n            return self._connections[ctrl_path]\n        else:\n            c = SSHConnection(ctrl_path, sshri.hostname)\n            self._connections[ctrl_path] = c\n            return c\n\n    def close(self):\n        \"\"\"Closes all connections, known to this instance.\n        \"\"\"\n        if self._connections:\n            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))\n            for cnct in self._connections:\n                self._connections[cnct].close()\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface to an ssh connection.\n\nAllows for connecting via ssh and keeping the connection open\n(by using a controlmaster), in order to perform several ssh commands or\ngit calls to a ssh remote without the need to reauthenticate.\n\"\"\"\n\nimport logging\nfrom subprocess import Popen\nfrom shlex import split as sh_split\nfrom os.path import isdir\n\nfrom datalad.support.network import RI, is_ssh\n\n# TODO: centralize AppDirs (=> datalad.config?)\nfrom appdirs import AppDirs\nfrom datalad.support.exceptions import CommandError\nfrom datalad.dochelpers import exc_str\nfrom datalad.utils import not_supported_on_windows\nfrom datalad.utils import on_windows\nfrom datalad.utils import assure_dir\nfrom datalad.utils import auto_repr\nfrom datalad.cmd import Runner\n\nlgr = logging.getLogger('datalad.ssh')\n\n\n@auto_repr\nclass SSHConnection(object):\n    \"\"\"Representation of a (shared) ssh connection.\n    \"\"\"\n\n    def __init__(self, ctrl_path, host, port=None):\n        \"\"\"Create the connection.\n\n        This does not actually open the connection.\n        It's just its representation.\n\n        Parameters\n        ----------\n        ctrl_path: str\n          path to SSH controlmaster\n        host: str\n          host to connect to. This may include the user ( [user@]host )\n        port: str\n          port to connect over\n        \"\"\"\n        self._runner = None\n\n        # TODO: This may actually also contain \"user@host\".\n        #       So, better name instead of 'host'?\n        self.host = host\n        self.ctrl_path = ctrl_path + \":\" + port if port else ctrl_path\n        self.port = port\n        self.ctrl_options = [\"-o\", \"ControlPath=\" + self.ctrl_path]\n\n    def __call__(self, cmd):\n        \"\"\"Executes a command on the remote.\n\n        Parameters\n        ----------\n        cmd: list or str\n          command to run on the remote\n\n        Returns\n        -------\n        tuple of str\n          stdout, stderr of the command run.\n        \"\"\"\n\n        # TODO: Do we need to check for the connection to be open or just rely\n        # on possible ssh failing?\n\n        ssh_cmd = [\"ssh\"] + self.ctrl_options + [self.host] + cmd if isinstance(cmd, list) \\\n            else sh_split(cmd, posix=not on_windows)\n            # windows check currently not needed, but keep it as a reminder\n\n        # TODO: pass expect parameters from above?\n        # Hard to explain to toplevel users ... So for now, just set True\n        return self.runner.run(ssh_cmd, expect_fail=True, expect_stderr=True)\n\n    @property\n    def runner(self):\n        if self._runner is None:\n            self._runner = Runner()\n        return self._runner\n\n    def open(self):\n        \"\"\"Opens the connection.\n\n        In other words: Creates the SSH controlmaster to be used by this\n        connection, if it is not there already.\n        \"\"\"\n\n        # set control options\n        ctrl_options = [\"-o\", \"ControlMaster=auto\", \"-o\", \"ControlPersist=yes\"] + self.ctrl_options\n        # create ssh control master command\n        cmd = [\"ssh\"] + ctrl_options + [self.host, \"exit\"]\n\n        # start control master:\n        lgr.debug(\"Try starting control master by calling:\\n%s\" % cmd)\n        proc = Popen(cmd)\n        proc.communicate(input=\"\\n\")  # why the f.. this is necessary?\n\n    def close(self):\n        \"\"\"Closes the connection.\n        \"\"\"\n\n        # stop controlmaster:\n        cmd = [\"ssh\", \"-O\", \"stop\", \"-S\", self.ctrl_path, self.host]\n        try:\n            self.runner.run(cmd, expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if \"No such file or directory\" in e.stderr:\n                # nothing to clean up\n                pass\n            else:\n                raise\n\n    def copy(self, source, destination, recursive=False, preserve_attrs=False):\n        \"\"\"Copies source file/folder to destination on the remote.\n\n        Parameters\n        ----------\n        source: str or list\n          file/folder path(s) to copy from on local\n        destination: str\n          file/folder path to copy to on remote\n\n        Returns\n        -------\n        str\n          stdout, stderr of the copy operation.\n        \"\"\"\n\n        # add recursive, preserve_attributes flag if recursive, preserve_attrs set and create scp command\n        scp_options = self.ctrl_options + [\"-r\"] if recursive else self.ctrl_options\n        scp_options += [\"-p\"] if preserve_attrs else []\n        scp_cmd = [\"scp\"] + scp_options\n\n        # add source filepath(s) to scp command\n        scp_cmd += source if isinstance(source, list) \\\n            else [source]\n\n        # add destination path\n        scp_cmd += [self.host + \":\" + destination]\n        return self.runner.run(scp_cmd)\n\n\n@auto_repr\nclass SSHManager(object):\n    \"\"\"Keeps ssh connections to share. Serves singleton representation\n    per connection.\n    \"\"\"\n\n    def __init__(self):\n        not_supported_on_windows(\"TODO: Make this an abstraction to \"\n                                 \"interface platform dependent SSH\")\n\n        self._connections = dict()\n\n        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir\n        assure_dir(self.socket_dir)\n\n    def get_connection(self, url):\n        \"\"\"Get a singleton, representing a shared ssh connection to `url`\n\n        Parameters\n        ----------\n        url: str\n          ssh url\n\n        Returns\n        -------\n        SSHConnection\n        \"\"\"\n        # parse url:\n        sshri = RI(url)\n\n        if not is_ssh(sshri):\n            raise ValueError(\"Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax\".format(url))\n\n        # determine control master:\n        ctrl_path = \"%s/%s\" % (self.socket_dir, sshri.hostname)\n        if sshri.port:\n            ctrl_path += \":%s\" % sshri.port\n\n        # do we know it already?\n        if ctrl_path in self._connections:\n            return self._connections[ctrl_path]\n        else:\n            c = SSHConnection(ctrl_path, sshri.hostname)\n            self._connections[ctrl_path] = c\n            return c\n\n    def close(self, allow_fail=True):\n        \"\"\"Closes all connections, known to this instance.\n\n        Parameters\n        ----------\n        allow_fail: bool, optional\n          If True, swallow exceptions which might be thrown during\n          connection.close, and just log them at DEBUG level\n        \"\"\"\n        if self._connections:\n            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))\n            for cnct in self._connections:\n                f = self._connections[cnct].close\n                if allow_fail:\n                    f()\n                else:\n                    try:\n                        f()\n                    except Exception as exc:\n                        lgr.debug(\"Failed to close a connection: %s\", exc_str(exc))",
            "header": {
                "file": {
                    "path": "datalad/support/sshconnector.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "69fe54e0a",
                "new_commit_hash": "13b15e80d",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/support/sshconnector.py"
            },
            "plus_file": {
                "path": "b/datalad/support/sshconnector.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 23,
                            "length": 6
                        },
                        "new_range": {
                            "start": 23,
                            "length": 7
                        },
                        "section": "from datalad.support.network import RI, is_ssh"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "# TODO: centralize AppDirs (=> datalad.config?)",
                                "type": "context"
                            },
                            {
                                "content": "from appdirs import AppDirs",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.support.exceptions import CommandError",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.dochelpers import exc_str",
                                "type": "added"
                            },
                            {
                                "content": "from datalad.utils import not_supported_on_windows",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.utils import on_windows",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.utils import assure_dir",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/support/sshconnector.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.dochelpers import exc_str",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 26,
                            "end_lineno": 26,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 200,
                            "length": 10
                        },
                        "new_range": {
                            "start": 201,
                            "length": 23
                        },
                        "section": "class SSHManager(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            self._connections[ctrl_path] = c",
                                "type": "context"
                            },
                            {
                                "content": "            return c",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def close(self):",
                                "type": "deleted"
                            },
                            {
                                "content": "    def close(self, allow_fail=True):",
                                "type": "added"
                            },
                            {
                                "content": "        \"\"\"Closes all connections, known to this instance.",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "        Parameters",
                                "type": "added"
                            },
                            {
                                "content": "        ----------",
                                "type": "added"
                            },
                            {
                                "content": "        allow_fail: bool, optional",
                                "type": "added"
                            },
                            {
                                "content": "          If True, swallow exceptions which might be thrown during",
                                "type": "added"
                            },
                            {
                                "content": "          connection.close, and just log them at DEBUG level",
                                "type": "added"
                            },
                            {
                                "content": "        \"\"\"",
                                "type": "context"
                            },
                            {
                                "content": "        if self._connections:",
                                "type": "context"
                            },
                            {
                                "content": "            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))",
                                "type": "context"
                            },
                            {
                                "content": "            for cnct in self._connections:",
                                "type": "context"
                            },
                            {
                                "content": "                self._connections[cnct].close()",
                                "type": "deleted"
                            },
                            {
                                "content": "                f = self._connections[cnct].close",
                                "type": "added"
                            },
                            {
                                "content": "                if allow_fail:",
                                "type": "added"
                            },
                            {
                                "content": "                    f()",
                                "type": "added"
                            },
                            {
                                "content": "                else:",
                                "type": "added"
                            },
                            {
                                "content": "                    try:",
                                "type": "added"
                            },
                            {
                                "content": "                        f()",
                                "type": "added"
                            },
                            {
                                "content": "                    except Exception as exc:",
                                "type": "added"
                            },
                            {
                                "content": "                        lgr.debug(\"Failed to close a connection: %s\", exc_str(exc))",
                                "type": "added"
                            },
                            {
                                "content": "No newline at end of file",
                                "type": "note"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/support/sshconnector.py",
                            "type": "method",
                            "name": "SSHManager.close",
                            "content": "class SSHManager(object):\n    \"\"\"Keeps ssh connections to share. Serves singleton representation\n    per connection.\n    \"\"\"\n\n    def __init__(self):\n        not_supported_on_windows(\"TODO: Make this an abstraction to \"\n                                 \"interface platform dependent SSH\")\n\n        self._connections = dict()\n\n        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir\n        assure_dir(self.socket_dir)\n\n    def get_connection(self, url):\n        \"\"\"Get a singleton, representing a shared ssh connection to `url`\n\n        Parameters\n        ----------\n        url: str\n          ssh url\n\n        Returns\n        -------\n        SSHConnection\n        \"\"\"\n        # parse url:\n        sshri = RI(url)\n\n        if not is_ssh(sshri):\n            raise ValueError(\"Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax\".format(url))\n\n        # determine control master:\n        ctrl_path = \"%s/%s\" % (self.socket_dir, sshri.hostname)\n        if sshri.port:\n            ctrl_path += \":%s\" % sshri.port\n\n        # do we know it already?\n        if ctrl_path in self._connections:\n            return self._connections[ctrl_path]\n        else:\n            c = SSHConnection(ctrl_path, sshri.hostname)\n            self._connections[ctrl_path] = c\n            return c\n\n    def close(self):\n        \"\"\"Closes all connections, known to this instance.\n        \"\"\"\n        if self._connections:\n            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))\n            for cnct in self._connections:\n                self._connections[cnct].close()",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 203,
                            "end_lineno": 209,
                            "parent": {
                                "file_name": "datalad/support/sshconnector.py",
                                "type": "class",
                                "name": "SSHManager",
                                "content": "class SSHManager(object):\n    \"\"\"Keeps ssh connections to share. Serves singleton representation\n    per connection.\n    \"\"\"\n\n    def __init__(self):\n        not_supported_on_windows(\"TODO: Make this an abstraction to \"\n                                 \"interface platform dependent SSH\")\n\n        self._connections = dict()\n\n        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir\n        assure_dir(self.socket_dir)\n\n    def get_connection(self, url):\n        \"\"\"Get a singleton, representing a shared ssh connection to `url`\n\n        Parameters\n        ----------\n        url: str\n          ssh url\n\n        Returns\n        -------\n        SSHConnection\n        \"\"\"\n        # parse url:\n        sshri = RI(url)\n\n        if not is_ssh(sshri):\n            raise ValueError(\"Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax\".format(url))\n\n        # determine control master:\n        ctrl_path = \"%s/%s\" % (self.socket_dir, sshri.hostname)\n        if sshri.port:\n            ctrl_path += \":%s\" % sshri.port\n\n        # do we know it already?\n        if ctrl_path in self._connections:\n            return self._connections[ctrl_path]\n        else:\n            c = SSHConnection(ctrl_path, sshri.hostname)\n            self._connections[ctrl_path] = c\n            return c\n\n    def close(self):\n        \"\"\"Closes all connections, known to this instance.\n        \"\"\"\n        if self._connections:\n            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))\n            for cnct in self._connections:\n                self._connections[cnct].close()",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 158,
                                "end_lineno": 209,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/support/sshconnector.py",
                            "type": "class",
                            "name": "SSHManager",
                            "content": "class SSHManager(object):\n    \"\"\"Keeps ssh connections to share. Serves singleton representation\n    per connection.\n    \"\"\"\n\n    def __init__(self):\n        not_supported_on_windows(\"TODO: Make this an abstraction to \"\n                                 \"interface platform dependent SSH\")\n\n        self._connections = dict()\n\n        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir\n        assure_dir(self.socket_dir)\n\n    def get_connection(self, url):\n        \"\"\"Get a singleton, representing a shared ssh connection to `url`\n\n        Parameters\n        ----------\n        url: str\n          ssh url\n\n        Returns\n        -------\n        SSHConnection\n        \"\"\"\n        # parse url:\n        sshri = RI(url)\n\n        if not is_ssh(sshri):\n            raise ValueError(\"Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax\".format(url))\n\n        # determine control master:\n        ctrl_path = \"%s/%s\" % (self.socket_dir, sshri.hostname)\n        if sshri.port:\n            ctrl_path += \":%s\" % sshri.port\n\n        # do we know it already?\n        if ctrl_path in self._connections:\n            return self._connections[ctrl_path]\n        else:\n            c = SSHConnection(ctrl_path, sshri.hostname)\n            self._connections[ctrl_path] = c\n            return c\n\n    def close(self):\n        \"\"\"Closes all connections, known to this instance.\n        \"\"\"\n        if self._connections:\n            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))\n            for cnct in self._connections:\n                self._connections[cnct].close()",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 158,
                            "end_lineno": 209,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test classes SSHConnection and SSHManager\n\n\"\"\"\n\nimport os\nfrom os.path import exists, isdir, getmtime, join as opj\n\nfrom nose.tools import ok_, assert_is_instance\n\nfrom datalad.support.sshconnector import SSHConnection, SSHManager\nfrom datalad.tests.utils import assert_raises, eq_\nfrom datalad.tests.utils import skip_ssh, with_tempfile, get_most_obscure_supported_name\n\n\n@skip_ssh\ndef test_ssh_get_connection():\n\n    manager = SSHManager()\n    c1 = manager.get_connection('ssh://localhost')\n    assert_is_instance(c1, SSHConnection)\n\n    # subsequent call returns the very same instance:\n    ok_(manager.get_connection('ssh://localhost') is c1)\n\n    # fail on malformed URls (meaning: out fancy URL parser can't correctly\n    # deal with them):\n    assert_raises(ValueError, manager.get_connection, 'localhost')\n    # we can do what urlparse cannot\n    #assert_raises(ValueError, manager.get_connection, 'someone@localhost')\n    # next one is considered a proper url by urlparse (netloc:'',\n    # path='/localhost), but eventually gets turned into SSHRI(hostname='ssh',\n    # path='/localhost') -- which is fair IMHO -> invalid test\n    #assert_raises(ValueError, manager.get_connection, 'ssh:/localhost')\n\n\n@skip_ssh\ndef test_ssh_open_close():\n\n    manager = SSHManager()\n    c1 = manager.get_connection('ssh://localhost')\n    path = opj(manager.socket_dir, 'localhost')\n    c1.open()\n    # control master exists:\n    ok_(exists(path))\n\n    # use connection to execute remote command:\n    out, err = c1(['ls', '-a'])\n    remote_ls = [entry for entry in out.splitlines() if entry != '.' and entry != '..']\n    local_ls = os.listdir(os.path.expanduser('~'))\n    eq_(set(remote_ls), set(local_ls))\n\n    c1.close()\n    # control master doesn't exist anymore:\n    ok_(not exists(path))\n\n\n@skip_ssh\ndef test_ssh_manager_close():\n\n    manager = SSHManager()\n    manager.get_connection('ssh://localhost').open()\n    manager.get_connection('ssh://datalad-test').open()\n    ok_(exists(opj(manager.socket_dir, 'localhost')))\n    ok_(exists(opj(manager.socket_dir, 'datalad-test')))\n\n    manager.close()\n\n    ok_(not exists(opj(manager.socket_dir, 'localhost')))\n    ok_(not exists(opj(manager.socket_dir, 'datalad-test')))\n\n\n@skip_ssh\n@with_tempfile(mkdir=True)\n@with_tempfile(content=\"one\")\n@with_tempfile(content=\"two\")\ndef test_ssh_copy(sourcedir, sourcefile1, sourcefile2):\n\n    remote_url = 'ssh://localhost'\n    manager = SSHManager()\n    ssh = manager.get_connection(remote_url)\n    ssh.open()\n\n    # write to obscurely named file in sourcedir\n    obscure_file = opj(sourcedir, get_most_obscure_supported_name())\n    with open(obscure_file, 'w') as f:\n        f.write(\"three\")\n\n    # copy tempfile list to remote_url:sourcedir\n    sourcefiles = [sourcefile1, sourcefile2, obscure_file]\n    ssh.copy(sourcefiles, opj(remote_url, sourcedir))\n\n    # recursive copy tempdir to remote_url:targetdir\n    targetdir = sourcedir + '.copy'\n    ssh.copy(sourcedir, opj(remote_url, targetdir), recursive=True, preserve_attrs=True)\n\n    # check if sourcedir copied to remote_url:targetdir\n    ok_(isdir(targetdir))\n    # check if scp preserved source directory attributes\n    # if source_mtime=1.12s, scp -p sets target_mtime = 1.0s, test that\n    eq_(getmtime(targetdir), int(getmtime(sourcedir)) + 0.0)\n\n    # check if targetfiles(and its content) exist in remote_url:targetdir,\n    # this implies file(s) and recursive directory copying pass\n    for targetfile, content in zip(sourcefiles, [\"one\", \"two\", \"three\"]):\n        targetpath = opj(targetdir, targetfile)\n        ok_(exists(targetpath))\n        with open(targetpath, 'r') as fp:\n            eq_(content, fp.read())\n\n    ssh.close()\n",
            "new_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test classes SSHConnection and SSHManager\n\n\"\"\"\n\nimport os\nfrom os.path import exists, isdir, getmtime, join as opj\n\nfrom nose.tools import ok_, assert_is_instance\n\nfrom datalad.support.sshconnector import SSHConnection, SSHManager\nfrom datalad.tests.utils import assert_raises, eq_\nfrom datalad.tests.utils import skip_ssh, with_tempfile, get_most_obscure_supported_name\nfrom datalad.tests.utils import swallow_logs\nfrom datalad.tests.utils import assert_in\n\nimport logging\n\n\n@skip_ssh\ndef test_ssh_get_connection():\n\n    manager = SSHManager()\n    c1 = manager.get_connection('ssh://localhost')\n    assert_is_instance(c1, SSHConnection)\n\n    # subsequent call returns the very same instance:\n    ok_(manager.get_connection('ssh://localhost') is c1)\n\n    # fail on malformed URls (meaning: out fancy URL parser can't correctly\n    # deal with them):\n    assert_raises(ValueError, manager.get_connection, 'localhost')\n    # we can do what urlparse cannot\n    #assert_raises(ValueError, manager.get_connection, 'someone@localhost')\n    # next one is considered a proper url by urlparse (netloc:'',\n    # path='/localhost), but eventually gets turned into SSHRI(hostname='ssh',\n    # path='/localhost') -- which is fair IMHO -> invalid test\n    #assert_raises(ValueError, manager.get_connection, 'ssh:/localhost')\n\n\n@skip_ssh\ndef test_ssh_open_close():\n\n    manager = SSHManager()\n    c1 = manager.get_connection('ssh://localhost')\n    path = opj(manager.socket_dir, 'localhost')\n    c1.open()\n    # control master exists:\n    ok_(exists(path))\n\n    # use connection to execute remote command:\n    out, err = c1(['ls', '-a'])\n    remote_ls = [entry for entry in out.splitlines() if entry != '.' and entry != '..']\n    local_ls = os.listdir(os.path.expanduser('~'))\n    eq_(set(remote_ls), set(local_ls))\n\n    c1.close()\n    # control master doesn't exist anymore:\n    ok_(not exists(path))\n\n\n@skip_ssh\ndef test_ssh_manager_close():\n\n    manager = SSHManager()\n    manager.get_connection('ssh://localhost').open()\n    manager.get_connection('ssh://datalad-test').open()\n    ok_(exists(opj(manager.socket_dir, 'localhost')))\n    ok_(exists(opj(manager.socket_dir, 'datalad-test')))\n\n    manager.close()\n\n    ok_(not exists(opj(manager.socket_dir, 'localhost')))\n    ok_(not exists(opj(manager.socket_dir, 'datalad-test')))\n\n\ndef test_ssh_manager_close_no_throw():\n    manager = SSHManager()\n    class bogus:\n        def close(self):\n            raise Exception(\"oh I am so bad\")\n\n    manager._connections['bogus'] = bogus()\n    assert_raises(Exception, manager.close)\n    assert_raises(Exception, manager.close)\n\n    # but should proceed just fine if allow_fail=False\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        manager.close(allow_fail=False)\n        assert_in('Failed to close a connection: oh I am so bad', cml.out)\n\n\n@skip_ssh\n@with_tempfile(mkdir=True)\n@with_tempfile(content=\"one\")\n@with_tempfile(content=\"two\")\ndef test_ssh_copy(sourcedir, sourcefile1, sourcefile2):\n\n    remote_url = 'ssh://localhost'\n    manager = SSHManager()\n    ssh = manager.get_connection(remote_url)\n    ssh.open()\n\n    # write to obscurely named file in sourcedir\n    obscure_file = opj(sourcedir, get_most_obscure_supported_name())\n    with open(obscure_file, 'w') as f:\n        f.write(\"three\")\n\n    # copy tempfile list to remote_url:sourcedir\n    sourcefiles = [sourcefile1, sourcefile2, obscure_file]\n    ssh.copy(sourcefiles, opj(remote_url, sourcedir))\n\n    # recursive copy tempdir to remote_url:targetdir\n    targetdir = sourcedir + '.copy'\n    ssh.copy(sourcedir, opj(remote_url, targetdir), recursive=True, preserve_attrs=True)\n\n    # check if sourcedir copied to remote_url:targetdir\n    ok_(isdir(targetdir))\n    # check if scp preserved source directory attributes\n    # if source_mtime=1.12s, scp -p sets target_mtime = 1.0s, test that\n    eq_(getmtime(targetdir), int(getmtime(sourcedir)) + 0.0)\n\n    # check if targetfiles(and its content) exist in remote_url:targetdir,\n    # this implies file(s) and recursive directory copying pass\n    for targetfile, content in zip(sourcefiles, [\"one\", \"two\", \"three\"]):\n        targetpath = opj(targetdir, targetfile)\n        ok_(exists(targetpath))\n        with open(targetpath, 'r') as fp:\n            eq_(content, fp.read())\n\n    ssh.close()\n",
            "header": {
                "file": {
                    "path": "datalad/support/tests/test_sshconnector.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "6d286b138",
                "new_commit_hash": "c7252c913",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/support/tests/test_sshconnector.py"
            },
            "plus_file": {
                "path": "b/datalad/support/tests/test_sshconnector.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 17,
                            "length": 6
                        },
                        "new_range": {
                            "start": 17,
                            "length": 10
                        },
                        "section": "from nose.tools import ok_, assert_is_instance"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "from datalad.support.sshconnector import SSHConnection, SSHManager",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.tests.utils import assert_raises, eq_",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.tests.utils import skip_ssh, with_tempfile, get_most_obscure_supported_name",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.tests.utils import swallow_logs",
                                "type": "added"
                            },
                            {
                                "content": "from datalad.tests.utils import assert_in",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "import logging",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "@skip_ssh",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/support/tests/test_sshconnector.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.tests.utils import swallow_logs",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 20,
                            "end_lineno": 20,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/support/tests/test_sshconnector.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.tests.utils import assert_in",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 21,
                            "end_lineno": 21,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/support/tests/test_sshconnector.py",
                            "type": "import",
                            "name": "logging",
                            "content": "import logging",
                            "ast_type_str": "Import",
                            "start_lineno": 23,
                            "end_lineno": 23,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 76,
                            "length": 6
                        },
                        "new_range": {
                            "start": 80,
                            "length": 22
                        },
                        "section": "def test_ssh_manager_close():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    ok_(not exists(opj(manager.socket_dir, 'datalad-test')))",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_ssh_manager_close_no_throw():",
                                "type": "added"
                            },
                            {
                                "content": "    manager = SSHManager()",
                                "type": "added"
                            },
                            {
                                "content": "    class bogus:",
                                "type": "added"
                            },
                            {
                                "content": "        def close(self):",
                                "type": "added"
                            },
                            {
                                "content": "            raise Exception(\"oh I am so bad\")",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    manager._connections['bogus'] = bogus()",
                                "type": "added"
                            },
                            {
                                "content": "    assert_raises(Exception, manager.close)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_raises(Exception, manager.close)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    # but should proceed just fine if allow_fail=False",
                                "type": "added"
                            },
                            {
                                "content": "    with swallow_logs(new_level=logging.DEBUG) as cml:",
                                "type": "added"
                            },
                            {
                                "content": "        manager.close(allow_fail=False)",
                                "type": "added"
                            },
                            {
                                "content": "        assert_in('Failed to close a connection: oh I am so bad', cml.out)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "@skip_ssh",
                                "type": "context"
                            },
                            {
                                "content": "@with_tempfile(mkdir=True)",
                                "type": "context"
                            },
                            {
                                "content": "@with_tempfile(content=\"one\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/support/tests/test_sshconnector.py",
                            "type": "function",
                            "name": "test_ssh_manager_close_no_throw",
                            "content": "def test_ssh_manager_close_no_throw():\n    manager = SSHManager()\n    class bogus:\n        def close(self):\n            raise Exception(\"oh I am so bad\")\n\n    manager._connections['bogus'] = bogus()\n    assert_raises(Exception, manager.close)\n    assert_raises(Exception, manager.close)\n\n    # but should proceed just fine if allow_fail=False\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        manager.close(allow_fail=False)\n        assert_in('Failed to close a connection: oh I am so bad', cml.out)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 83,
                            "end_lineno": 96,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "5b010c507946dc74381091fb7bd97d0a6d0b1722^",
    "new_commit_hash": "5b010c507946dc74381091fb7bd97d0a6d0b1722",
    "commit_message": "BF: swallow exceptions from SSH manager while atexit (Closes #691)",
    "commit_date": "2016-08-10T17:13:44-04:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "5b010c507946dc74381091fb7bd97d0a6d0b1722",
    "test_file_codes": [
        "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test classes SSHConnection and SSHManager\n\n\"\"\"\n\nimport os\nfrom os.path import exists, isdir, getmtime, join as opj\n\nfrom nose.tools import ok_, assert_is_instance\n\nfrom datalad.support.sshconnector import SSHConnection, SSHManager\nfrom datalad.tests.utils import assert_raises, eq_\nfrom datalad.tests.utils import skip_ssh, with_tempfile, get_most_obscure_supported_name\nfrom datalad.tests.utils import swallow_logs\nfrom datalad.tests.utils import assert_in\n\nimport logging\n\n\n@skip_ssh\ndef test_ssh_get_connection():\n\n    manager = SSHManager()\n    c1 = manager.get_connection('ssh://localhost')\n    assert_is_instance(c1, SSHConnection)\n\n    # subsequent call returns the very same instance:\n    ok_(manager.get_connection('ssh://localhost') is c1)\n\n    # fail on malformed URls (meaning: out fancy URL parser can't correctly\n    # deal with them):\n    assert_raises(ValueError, manager.get_connection, 'localhost')\n    # we can do what urlparse cannot\n    #assert_raises(ValueError, manager.get_connection, 'someone@localhost')\n    # next one is considered a proper url by urlparse (netloc:'',\n    # path='/localhost), but eventually gets turned into SSHRI(hostname='ssh',\n    # path='/localhost') -- which is fair IMHO -> invalid test\n    #assert_raises(ValueError, manager.get_connection, 'ssh:/localhost')\n\n\n@skip_ssh\ndef test_ssh_open_close():\n\n    manager = SSHManager()\n    c1 = manager.get_connection('ssh://localhost')\n    path = opj(manager.socket_dir, 'localhost')\n    c1.open()\n    # control master exists:\n    ok_(exists(path))\n\n    # use connection to execute remote command:\n    out, err = c1(['ls', '-a'])\n    remote_ls = [entry for entry in out.splitlines() if entry != '.' and entry != '..']\n    local_ls = os.listdir(os.path.expanduser('~'))\n    eq_(set(remote_ls), set(local_ls))\n\n    c1.close()\n    # control master doesn't exist anymore:\n    ok_(not exists(path))\n\n\n@skip_ssh\ndef test_ssh_manager_close():\n\n    manager = SSHManager()\n    manager.get_connection('ssh://localhost').open()\n    manager.get_connection('ssh://datalad-test').open()\n    ok_(exists(opj(manager.socket_dir, 'localhost')))\n    ok_(exists(opj(manager.socket_dir, 'datalad-test')))\n\n    manager.close()\n\n    ok_(not exists(opj(manager.socket_dir, 'localhost')))\n    ok_(not exists(opj(manager.socket_dir, 'datalad-test')))\n\n\ndef test_ssh_manager_close_no_throw():\n    manager = SSHManager()\n    class bogus:\n        def close(self):\n            raise Exception(\"oh I am so bad\")\n\n    manager._connections['bogus'] = bogus()\n    assert_raises(Exception, manager.close)\n    assert_raises(Exception, manager.close)\n\n    # but should proceed just fine if allow_fail=False\n    with swallow_logs(new_level=logging.DEBUG) as cml:\n        manager.close(allow_fail=False)\n        assert_in('Failed to close a connection: oh I am so bad', cml.out)\n\n\n@skip_ssh\n@with_tempfile(mkdir=True)\n@with_tempfile(content=\"one\")\n@with_tempfile(content=\"two\")\ndef test_ssh_copy(sourcedir, sourcefile1, sourcefile2):\n\n    remote_url = 'ssh://localhost'\n    manager = SSHManager()\n    ssh = manager.get_connection(remote_url)\n    ssh.open()\n\n    # write to obscurely named file in sourcedir\n    obscure_file = opj(sourcedir, get_most_obscure_supported_name())\n    with open(obscure_file, 'w') as f:\n        f.write(\"three\")\n\n    # copy tempfile list to remote_url:sourcedir\n    sourcefiles = [sourcefile1, sourcefile2, obscure_file]\n    ssh.copy(sourcefiles, opj(remote_url, sourcedir))\n\n    # recursive copy tempdir to remote_url:targetdir\n    targetdir = sourcedir + '.copy'\n    ssh.copy(sourcedir, opj(remote_url, targetdir), recursive=True, preserve_attrs=True)\n\n    # check if sourcedir copied to remote_url:targetdir\n    ok_(isdir(targetdir))\n    # check if scp preserved source directory attributes\n    # if source_mtime=1.12s, scp -p sets target_mtime = 1.0s, test that\n    eq_(getmtime(targetdir), int(getmtime(sourcedir)) + 0.0)\n\n    # check if targetfiles(and its content) exist in remote_url:targetdir,\n    # this implies file(s) and recursive directory copying pass\n    for targetfile, content in zip(sourcefiles, [\"one\", \"two\", \"three\"]):\n        targetpath = opj(targetdir, targetfile)\n        ok_(exists(targetpath))\n        with open(targetpath, 'r') as fp:\n            eq_(content, fp.read())\n\n    ssh.close()\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\n❌ Verification failed\nPython 3.9 installation failed, trying Python 3.7...\nVerifying installation...\nDatalad version: 0.2.3\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.7\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 205ms\nInstalled 25 packages in 3.19s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpcBxFcv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpcBxFcv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpcBxFcv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpcBxFcv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpcBxFcv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 35, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpRfv9kv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpRfv9kv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpRfv9kv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpRfv9kv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpRfv9kv/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 35, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/datalad/__init__.py\", line 20, in <module>\n    from .config import ConfigManager\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/datalad/config.py\", line 17, in <module>\n    from appdirs import AppDirs\nModuleNotFoundError: No module named 'appdirs'\nUsing CPython 3.7.9\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 26 packages in 346ms\nInstalled 26 packages in 5.06s\n + bids==0.0\n + coverage==7.2.7\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + fasteners==0.19\n + grabbit==0.2.6\n + importlib-metadata==6.7.0\n + iniconfig==2.0.0\n + nibabel==4.0.2\n + num2words==0.5.14\n + numpy==1.21.6\n + packaging==24.0\n + pandas==1.3.5\n + patsy==1.0.1\n + pluggy==1.2.0\n + pybids==0.6.5\n + pytest==7.4.4\n + pytest-cov==4.1.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.7.3\n + setuptools==68.0.0\n + six==1.17.0\n + tomli==2.0.1\n + typing-extensions==4.7.1\n + zipp==3.15.0\nResolved 69 packages in 1.80s\nPrepared 1 package in 3.47s\nInstalled 63 packages in 917ms\n + appdirs==1.4.4\n + attrs==24.2.0\n + automat==22.10.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.15.1\n + charset-normalizer==3.4.1\n + constantly==15.1.0\n + cryptography==44.0.0\n + cssselect==1.2.0\n + datalad==0.2.3 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722)\n + filelock==3.12.2\n + gitdb==4.0.11\n + gitpython==3.1.43\n + httpretty==1.1.4\n + humanize==4.6.0\n + hyperlink==21.0.0\n + idna==3.10\n + importlib-resources==5.12.0\n + incremental==22.10.0\n + iso8601==2.1.0\n + itemadapter==0.8.0\n + itemloaders==1.1.0\n + jaraco-classes==3.2.3\n + jeepney==0.8.0\n + jmespath==1.0.1\n + keyring==24.1.1\n + keyrings-alt==4.2.0\n + lxml==5.3.0\n + mock==5.1.0\n + more-itertools==9.1.0\n + msgpack-python==0.5.6\n + multidict==6.0.5\n + nose==1.3.7\n + parsel==1.8.1\n + patool==1.12\n + protego==0.3.0\n + pyasn1==0.5.1\n + pyasn1-modules==0.3.0\n + pycparser==2.21\n + pydispatcher==2.0.7\n + pyopenssl==24.3.0\n + pyyaml==6.0.1\n + queuelib==1.6.2\n + requests==2.31.0\n + requests-file==2.1.0\n + requests-ftp==0.3.1\n + scrapy==2.9.0\n + secretstorage==3.3.3\n + service-identity==21.1.0\n + smmap==5.0.1\n + soupsieve==2.4.1\n + testtools==2.7.1\n + tldextract==4.0.0\n + tqdm==4.67.1\n + twisted==23.8.0\n + urllib3==2.0.7\n + vcrpy==4.3.0\n + w3lib==2.1.2\n + wrapt==1.16.0\n + yarl==1.9.4\n + zope-interface==6.4.post2\nAudited 1 package in 72ms\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/.venv/lib/python3.7/site-packages/urllib3/__init__.py:38: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.1.4'. See: https://github.com/urllib3/urllib3/issues/3020\n  exceptions.NotOpenSSLWarning,\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722\nplugins: cov-4.1.0\ncollected 5 items\n\nr2e_tests/test_1.py sss.E                                                [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_ssh_copy ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/r2e_tests/test_1.py, line 99\n  @skip_ssh\n  @with_tempfile(mkdir=True)\n  @with_tempfile(content=\"one\")\n  @with_tempfile(content=\"two\")\n  def test_ssh_copy(sourcedir, sourcefile1, sourcefile2):\nE       fixture 'sourcedir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/r2e_tests/test_1.py:99\n==================================== PASSES ====================================\n_______________________ test_ssh_manager_close_no_throw ________________________\n------------------------------ Captured log call -------------------------------\nDEBUG    datalad.ssh:sshconnector.py:214 Closing 1 SSH connections...\nDEBUG    datalad.ssh:sshconnector.py:223 Failed to close a connection: oh I am so bad [test_1.py:close:87]\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_ssh_manager_close_no_throw\nSKIPPED [3] .venv/lib/python3.7/site-packages/_pytest/unittest.py:371: Run this test by setting DATALAD_TESTS_SSH\nERROR r2e_tests/test_1.py::test_ssh_copy\n==================== 1 passed, 3 skipped, 1 error in 0.81s =====================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722\nplugins: cov-4.1.0\ncollected 5 items\n\nr2e_tests/test_1.py sssFE                                                [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_ssh_copy ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/r2e_tests/test_1.py, line 99\n  @skip_ssh\n  @with_tempfile(mkdir=True)\n  @with_tempfile(content=\"one\")\n  @with_tempfile(content=\"two\")\n  def test_ssh_copy(sourcedir, sourcefile1, sourcefile2):\nE       fixture 'sourcedir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_5b010c507946dc74381091fb7bd97d0a6d0b1722/r2e_tests/test_1.py:99\n=================================== FAILURES ===================================\n_______________________ test_ssh_manager_close_no_throw ________________________\n\n    def test_ssh_manager_close_no_throw():\n        manager = SSHManager()\n        class bogus:\n            def close(self):\n                raise Exception(\"oh I am so bad\")\n    \n        manager._connections['bogus'] = bogus()\n        assert_raises(Exception, manager.close)\n        assert_raises(Exception, manager.close)\n    \n        # but should proceed just fine if allow_fail=False\n        with swallow_logs(new_level=logging.DEBUG) as cml:\n>           manager.close(allow_fail=False)\nE           TypeError: close() got an unexpected keyword argument 'allow_fail'\n\nr2e_tests/test_1.py:95: TypeError\n=========================== short test summary info ============================\nSKIPPED [3] .venv/lib/python3.7/site-packages/_pytest/unittest.py:371: Run this test by setting DATALAD_TESTS_SSH\nERROR r2e_tests/test_1.py::test_ssh_copy\nFAILED r2e_tests/test_1.py::test_ssh_manager_close_no_throw - TypeError: clos...\n==================== 1 failed, 3 skipped, 1 error in 0.46s =====================\n",
    "old_commit_res_stderr": ""
}
datalad/__init__.py datalad/support/sshconnector.py datalad/support/tests/test_sshconnector.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** SSHManager.close Method Lacks 'allow_fail' Parameter Causing Unhandled Exceptions on Exit

**Description:**
When the application attempts to terminate, the `SSHManager`'s `close` method is responsible for closing all active SSH connections. However, if any connection fails to close properly, it raises an exception that is not handled, leading to abrupt termination of the program.

**Example Buggy Code:**
```python
manager = SSHManager()

class BogusConnection:
    def close(self):
        raise Exception("Failed to close connection")

manager._connections['bogus'] = BogusConnection()
manager.close(allow_fail=False)  # This raises TypeError
```

**Expected Behavior:**
The `close` method should accept an `allow_fail` parameter. When `allow_fail` is set to `False`, it should gracefully handle any exceptions raised by individual connections, log the errors appropriately, and allow the program to exit cleanly without interruption.

**Actual Behavior:**
Passing the `allow_fail` argument to the `close` method results in a `TypeError` because the method does not accept this parameter. Consequently, unhandled exceptions prevent the program from exiting gracefully.
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/support/sshconnector.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface to an ssh connection.

Allows for connecting via ssh and keeping the connection open
(by using a controlmaster), in order to perform several ssh commands or
git calls to a ssh remote without the need to reauthenticate.
"""

import logging
from subprocess import Popen
from shlex import split as sh_split
from os.path import isdir

from datalad.support.network import RI, is_ssh

# TODO: centralize AppDirs (=> datalad.config?)
from appdirs import AppDirs
from datalad.support.exceptions import CommandError
from datalad.utils import not_supported_on_windows
from datalad.utils import on_windows
from datalad.utils import assure_dir
from datalad.utils import auto_repr
from datalad.cmd import Runner

lgr = logging.getLogger('datalad.ssh')


@auto_repr
class SSHConnection(object):
    """Representation of a (shared) ssh connection.
    """

    def __init__(self, ctrl_path, host, port=None):
        """Create the connection.

        This does not actually open the connection.
        It's just its representation.

        Parameters
        ----------
        ctrl_path: str
          path to SSH controlmaster
        host: str
          host to connect to. This may include the user ( [user@]host )
        port: str
          port to connect over
        """
        self._runner = None

        # TODO: This may actually also contain "user@host".
        #       So, better name instead of 'host'?
        self.host = host
        self.ctrl_path = ctrl_path + ":" + port if port else ctrl_path
        self.port = port
        self.ctrl_options = ["-o", "ControlPath=" + self.ctrl_path]

    def __call__(self, cmd):
        """Executes a command on the remote.

        Parameters
        ----------
        cmd: list or str
          command to run on the remote

        Returns
        -------
        tuple of str
          stdout, stderr of the command run.
        """

        # TODO: Do we need to check for the connection to be open or just rely
        # on possible ssh failing?

        ssh_cmd = ["ssh"] + self.ctrl_options + [self.host] + cmd if isinstance(cmd, list) \
            else sh_split(cmd, posix=not on_windows)
            # windows check currently not needed, but keep it as a reminder

        # TODO: pass expect parameters from above?
        # Hard to explain to toplevel users ... So for now, just set True
        return self.runner.run(ssh_cmd, expect_fail=True, expect_stderr=True)

    @property
    def runner(self):
        if self._runner is None:
            self._runner = Runner()
        return self._runner

    def open(self):
        """Opens the connection.

        In other words: Creates the SSH controlmaster to be used by this
        connection, if it is not there already.
        """

        # set control options
        ctrl_options = ["-o", "ControlMaster=auto", "-o", "ControlPersist=yes"] + self.ctrl_options
        # create ssh control master command
        cmd = ["ssh"] + ctrl_options + [self.host, "exit"]

        # start control master:
        lgr.debug("Try starting control master by calling:\n%s" % cmd)
        proc = Popen(cmd)
        proc.communicate(input="\n")  # why the f.. this is necessary?

    def close(self):
        """Closes the connection.
        """

        # stop controlmaster:
        cmd = ["ssh", "-O", "stop", "-S", self.ctrl_path, self.host]
        try:
            self.runner.run(cmd, expect_stderr=True, expect_fail=True)
        except CommandError as e:
            if "No such file or directory" in e.stderr:
                # nothing to clean up
                pass
            else:
                raise

    def copy(self, source, destination, recursive=False, preserve_attrs=False):
        """Copies source file/folder to destination on the remote.

        Parameters
        ----------
        source: str or list
          file/folder path(s) to copy from on local
        destination: str
          file/folder path to copy to on remote

        Returns
        -------
        str
          stdout, stderr of the copy operation.
        """

        # add recursive, preserve_attributes flag if recursive, preserve_attrs set and create scp command
        scp_options = self.ctrl_options + ["-r"] if recursive else self.ctrl_options
        scp_options += ["-p"] if preserve_attrs else []
        scp_cmd = ["scp"] + scp_options

        # add source filepath(s) to scp command
        scp_cmd += source if isinstance(source, list) \
            else [source]

        # add destination path
        scp_cmd += [self.host + ":" + destination]
        return self.runner.run(scp_cmd)


@auto_repr
class SSHManager(object):
    """Keeps ssh connections to share. Serves singleton representation
    per connection.
    """

    def __init__(self):
        not_supported_on_windows("TODO: Make this an abstraction to "
                                 "interface platform dependent SSH")

        self._connections = dict()

        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir
        assure_dir(self.socket_dir)

    def get_connection(self, url):
        """Get a singleton, representing a shared ssh connection to `url`

        Parameters
        ----------
        url: str
          ssh url

        Returns
        -------
        SSHConnection
        """
        # parse url:
        sshri = RI(url)

        if not is_ssh(sshri):
            raise ValueError("Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax".format(url))

        # determine control master:
        ctrl_path = "%s/%s" % (self.socket_dir, sshri.hostname)
        if sshri.port:
            ctrl_path += ":%s" % sshri.port

        # do we know it already?
        if ctrl_path in self._connections:
            return self._connections[ctrl_path]
        else:
            c = SSHConnection(ctrl_path, sshri.hostname)
            self._connections[ctrl_path] = c
            return c

    def close(self):
        """Closes all connections, known to this instance.
        """
        if self._connections:
            lgr.debug("Closing %d SSH connections..." % len(self._connections))
            for cnct in self._connections:
                self._connections[cnct].close()

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** SSHManager.close Method Lacks 'allow_fail' Parameter Causing Unhandled Exceptions on Exit

**Description:**
When the application attempts to terminate, the `SSHManager`'s `close` method is responsible for closing all active SSH connections. However, if any connection fails to close properly, it raises an exception that is not handled, leading to abrupt termination of the program.

**Example Buggy Code:**
```python
manager = SSHManager()

class BogusConnection:
    def close(self):
        raise Exception("Failed to close connection")

manager._connections['bogus'] = BogusConnection()
manager.close(allow_fail=False)  # This raises TypeError
```

**Expected Behavior:**
The `close` method should accept an `allow_fail` parameter. When `allow_fail` is set to `False`, it should gracefully handle any exceptions raised by individual connections, log the errors appropriately, and allow the program to exit cleanly without interruption.

**Actual Behavior:**
Passing the `allow_fail` argument to the `close` method results in a `TypeError` because the method does not accept this parameter. Consequently, unhandled exceptions prevent the program from exiting gracefully.
{
    "test_ssh_manager_close_no_throw": "PASSED",
    "test_ssh_copy": "ERROR"
}
diff --git a/datalad/__init__.py b/datalad/__init__.py
index 66d61e716..588b85d19 100644
--- a/datalad/__init__.py
+++ b/datalad/__init__.py
@@ -23,7 +23,7 @@ cfg = ConfigManager()
 lgr.log(5, "Instantiating ssh manager")
 from .support.sshconnector import SSHManager
 ssh_manager = SSHManager()
-atexit.register(ssh_manager.close)
+atexit.register(ssh_manager.close, allow_fail=False)
 
 
 def test(package='datalad', **kwargs):
diff --git a/datalad/support/sshconnector.py b/datalad/support/sshconnector.py
index 69fe54e0a..13b15e80d 100644
--- a/datalad/support/sshconnector.py
+++ b/datalad/support/sshconnector.py
@@ -23,6 +23,7 @@ from datalad.support.network import RI, is_ssh
 # TODO: centralize AppDirs (=> datalad.config?)
 from appdirs import AppDirs
 from datalad.support.exceptions import CommandError
+from datalad.dochelpers import exc_str
 from datalad.utils import not_supported_on_windows
 from datalad.utils import on_windows
 from datalad.utils import assure_dir
@@ -200,10 +201,23 @@ class SSHManager(object):
             self._connections[ctrl_path] = c
             return c
 
-    def close(self):
+    def close(self, allow_fail=True):
         """Closes all connections, known to this instance.
+
+        Parameters
+        ----------
+        allow_fail: bool, optional
+          If True, swallow exceptions which might be thrown during
+          connection.close, and just log them at DEBUG level
         """
         if self._connections:
             lgr.debug("Closing %d SSH connections..." % len(self._connections))
             for cnct in self._connections:
-                self._connections[cnct].close()
+                f = self._connections[cnct].close
+                if allow_fail:
+                    f()
+                else:
+                    try:
+                        f()
+                    except Exception as exc:
+                        lgr.debug("Failed to close a connection: %s", exc_str(exc))
\ No newline at end of file

{"datalad/support/sshconnector.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface to an ssh connection.\n\nAllows for connecting via ssh and keeping the connection open\n(by using a controlmaster), in order to perform several ssh commands or\ngit calls to a ssh remote without the need to reauthenticate.\n\"\"\"\n\nimport logging\nfrom subprocess import Popen\nfrom shlex import split as sh_split\nfrom os.path import isdir\n\nfrom datalad.support.network import RI, is_ssh\n\n# TODO: centralize AppDirs (=> datalad.config?)\nfrom appdirs import AppDirs\nfrom datalad.support.exceptions import CommandError\nfrom datalad.utils import not_supported_on_windows\nfrom datalad.utils import on_windows\nfrom datalad.utils import assure_dir\nfrom datalad.utils import auto_repr\nfrom datalad.cmd import Runner\n\nlgr = logging.getLogger('datalad.ssh')\n\n\n@auto_repr\nclass SSHConnection(object):\n    \"\"\"Representation of a (shared) ssh connection.\n    \"\"\"\n\n    def __init__(self, ctrl_path, host, port=None):\n        \"\"\"Create the connection.\n\n        This does not actually open the connection.\n        It's just its representation.\n\n        Parameters\n        ----------\n        ctrl_path: str\n          path to SSH controlmaster\n        host: str\n          host to connect to. This may include the user ( [user@]host )\n        port: str\n          port to connect over\n        \"\"\"\n        self._runner = None\n\n        # TODO: This may actually also contain \"user@host\".\n        #       So, better name instead of 'host'?\n        self.host = host\n        self.ctrl_path = ctrl_path + \":\" + port if port else ctrl_path\n        self.port = port\n        self.ctrl_options = [\"-o\", \"ControlPath=\" + self.ctrl_path]\n\n    def __call__(self, cmd):\n        \"\"\"Executes a command on the remote.\n\n        Parameters\n        ----------\n        cmd: list or str\n          command to run on the remote\n\n        Returns\n        -------\n        tuple of str\n          stdout, stderr of the command run.\n        \"\"\"\n\n        # TODO: Do we need to check for the connection to be open or just rely\n        # on possible ssh failing?\n\n        ssh_cmd = [\"ssh\"] + self.ctrl_options + [self.host] + cmd if isinstance(cmd, list) \\\n            else sh_split(cmd, posix=not on_windows)\n            # windows check currently not needed, but keep it as a reminder\n\n        # TODO: pass expect parameters from above?\n        # Hard to explain to toplevel users ... So for now, just set True\n        return self.runner.run(ssh_cmd, expect_fail=True, expect_stderr=True)\n\n    @property\n    def runner(self):\n        if self._runner is None:\n            self._runner = Runner()\n        return self._runner\n\n    def open(self):\n        \"\"\"Opens the connection.\n\n        In other words: Creates the SSH controlmaster to be used by this\n        connection, if it is not there already.\n        \"\"\"\n\n        # set control options\n        ctrl_options = [\"-o\", \"ControlMaster=auto\", \"-o\", \"ControlPersist=yes\"] + self.ctrl_options\n        # create ssh control master command\n        cmd = [\"ssh\"] + ctrl_options + [self.host, \"exit\"]\n\n        # start control master:\n        lgr.debug(\"Try starting control master by calling:\\n%s\" % cmd)\n        proc = Popen(cmd)\n        proc.communicate(input=\"\\n\")  # why the f.. this is necessary?\n\n    def close(self):\n        \"\"\"Closes the connection.\n        \"\"\"\n\n        # stop controlmaster:\n        cmd = [\"ssh\", \"-O\", \"stop\", \"-S\", self.ctrl_path, self.host]\n        try:\n            self.runner.run(cmd, expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if \"No such file or directory\" in e.stderr:\n                # nothing to clean up\n                pass\n            else:\n                raise\n\n    def copy(self, source, destination, recursive=False, preserve_attrs=False):\n        \"\"\"Copies source file/folder to destination on the remote.\n\n        Parameters\n        ----------\n        source: str or list\n          file/folder path(s) to copy from on local\n        destination: str\n          file/folder path to copy to on remote\n\n        Returns\n        -------\n        str\n          stdout, stderr of the copy operation.\n        \"\"\"\n\n        # add recursive, preserve_attributes flag if recursive, preserve_attrs set and create scp command\n        scp_options = self.ctrl_options + [\"-r\"] if recursive else self.ctrl_options\n        scp_options += [\"-p\"] if preserve_attrs else []\n        scp_cmd = [\"scp\"] + scp_options\n\n        # add source filepath(s) to scp command\n        scp_cmd += source if isinstance(source, list) \\\n            else [source]\n\n        # add destination path\n        scp_cmd += [self.host + \":\" + destination]\n        return self.runner.run(scp_cmd)\n\n\n@auto_repr\nclass SSHManager(object):\n    \"\"\"Keeps ssh connections to share. Serves singleton representation\n    per connection.\n    \"\"\"\n\n    def __init__(self):\n        not_supported_on_windows(\"TODO: Make this an abstraction to \"\n                                 \"interface platform dependent SSH\")\n\n        self._connections = dict()\n\n        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir\n        assure_dir(self.socket_dir)\n\n    def get_connection(self, url):\n        \"\"\"Get a singleton, representing a shared ssh connection to `url`\n\n        Parameters\n        ----------\n        url: str\n          ssh url\n\n        Returns\n        -------\n        SSHConnection\n        \"\"\"\n        # parse url:\n        sshri = RI(url)\n\n        if not is_ssh(sshri):\n            raise ValueError(\"Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax\".format(url))\n\n        # determine control master:\n        ctrl_path = \"%s/%s\" % (self.socket_dir, sshri.hostname)\n        if sshri.port:\n            ctrl_path += \":%s\" % sshri.port\n\n        # do we know it already?\n        if ctrl_path in self._connections:\n            return self._connections[ctrl_path]\n        else:\n            c = SSHConnection(ctrl_path, sshri.hostname)\n            self._connections[ctrl_path] = c\n            return c\n\n    def close(self):\n        \"\"\"Closes all connections, known to this instance.\n        \"\"\"\n        if self._connections:\n            lgr.debug(\"Closing %d SSH connections...\" % len(self._connections))\n            for cnct in self._connections:\n                self._connections[cnct].close()\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** SSHManager.close Method Lacks 'allow_fail' Parameter Causing Unhandled Exceptions on Exit

**Description:**
When the application attempts to terminate, the `SSHManager`'s `close` method is responsible for closing all active SSH connections. However, if any connection fails to close properly, it raises an exception that is not handled, leading to abrupt termination of the program.

**Example Buggy Code:**
```python
manager = SSHManager()

class BogusConnection:
    def close(self):
        raise Exception("Failed to close connection")

manager._connections['bogus'] = BogusConnection()
manager.close(allow_fail=False)  # This raises TypeError
```

**Expected Behavior:**
The `close` method should accept an `allow_fail` parameter. When `allow_fail` is set to `False`, it should gracefully handle any exceptions raised by individual connections, log the errors appropriately, and allow the program to exit cleanly without interruption.

**Actual Behavior:**
Passing the `allow_fail` argument to the `close` method results in a `TypeError` because the method does not accept this parameter. Consequently, unhandled exceptions prevent the program from exiting gracefully.
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/support/sshconnector.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface to an ssh connection.

Allows for connecting via ssh and keeping the connection open
(by using a controlmaster), in order to perform several ssh commands or
git calls to a ssh remote without the need to reauthenticate.
"""

import logging
from subprocess import Popen
from shlex import split as sh_split
from os.path import isdir

from datalad.support.network import RI, is_ssh

# TODO: centralize AppDirs (=> datalad.config?)
from appdirs import AppDirs
from datalad.support.exceptions import CommandError
from datalad.utils import not_supported_on_windows
from datalad.utils import on_windows
from datalad.utils import assure_dir
from datalad.utils import auto_repr
from datalad.cmd import Runner

lgr = logging.getLogger('datalad.ssh')


@auto_repr
class SSHConnection(object):
    """Representation of a (shared) ssh connection.
    """

    def __init__(self, ctrl_path, host, port=None):
        """Create the connection.

        This does not actually open the connection.
        It's just its representation.

        Parameters
        ----------
        ctrl_path: str
          path to SSH controlmaster
        host: str
          host to connect to. This may include the user ( [user@]host )
        port: str
          port to connect over
        """
        self._runner = None

        # TODO: This may actually also contain "user@host".
        #       So, better name instead of 'host'?
        self.host = host
        self.ctrl_path = ctrl_path + ":" + port if port else ctrl_path
        self.port = port
        self.ctrl_options = ["-o", "ControlPath=" + self.ctrl_path]

    def __call__(self, cmd):
        """Executes a command on the remote.

        Parameters
        ----------
        cmd: list or str
          command to run on the remote

        Returns
        -------
        tuple of str
          stdout, stderr of the command run.
        """

        # TODO: Do we need to check for the connection to be open or just rely
        # on possible ssh failing?

        ssh_cmd = ["ssh"] + self.ctrl_options + [self.host] + cmd if isinstance(cmd, list) \
            else sh_split(cmd, posix=not on_windows)
            # windows check currently not needed, but keep it as a reminder

        # TODO: pass expect parameters from above?
        # Hard to explain to toplevel users ... So for now, just set True
        return self.runner.run(ssh_cmd, expect_fail=True, expect_stderr=True)

    @property
    def runner(self):
        if self._runner is None:
            self._runner = Runner()
        return self._runner

    def open(self):
        """Opens the connection.

        In other words: Creates the SSH controlmaster to be used by this
        connection, if it is not there already.
        """

        # set control options
        ctrl_options = ["-o", "ControlMaster=auto", "-o", "ControlPersist=yes"] + self.ctrl_options
        # create ssh control master command
        cmd = ["ssh"] + ctrl_options + [self.host, "exit"]

        # start control master:
        lgr.debug("Try starting control master by calling:\n%s" % cmd)
        proc = Popen(cmd)
        proc.communicate(input="\n")  # why the f.. this is necessary?

    def close(self):
        """Closes the connection.
        """

        # stop controlmaster:
        cmd = ["ssh", "-O", "stop", "-S", self.ctrl_path, self.host]
        try:
            self.runner.run(cmd, expect_stderr=True, expect_fail=True)
        except CommandError as e:
            if "No such file or directory" in e.stderr:
                # nothing to clean up
                pass
            else:
                raise

    def copy(self, source, destination, recursive=False, preserve_attrs=False):
        """Copies source file/folder to destination on the remote.

        Parameters
        ----------
        source: str or list
          file/folder path(s) to copy from on local
        destination: str
          file/folder path to copy to on remote

        Returns
        -------
        str
          stdout, stderr of the copy operation.
        """

        # add recursive, preserve_attributes flag if recursive, preserve_attrs set and create scp command
        scp_options = self.ctrl_options + ["-r"] if recursive else self.ctrl_options
        scp_options += ["-p"] if preserve_attrs else []
        scp_cmd = ["scp"] + scp_options

        # add source filepath(s) to scp command
        scp_cmd += source if isinstance(source, list) \
            else [source]

        # add destination path
        scp_cmd += [self.host + ":" + destination]
        return self.runner.run(scp_cmd)


@auto_repr
class SSHManager(object):
    """Keeps ssh connections to share. Serves singleton representation
    per connection.
    """

    def __init__(self):
        not_supported_on_windows("TODO: Make this an abstraction to "
                                 "interface platform dependent SSH")

        self._connections = dict()

        self.socket_dir = AppDirs('datalad', 'datalad.org').user_config_dir
        assure_dir(self.socket_dir)

    def get_connection(self, url):
        """Get a singleton, representing a shared ssh connection to `url`

        Parameters
        ----------
        url: str
          ssh url

        Returns
        -------
        SSHConnection
        """
        # parse url:
        sshri = RI(url)

        if not is_ssh(sshri):
            raise ValueError("Unsupported SSH URL: '{0}', use ssh://host/path or host:path syntax".format(url))

        # determine control master:
        ctrl_path = "%s/%s" % (self.socket_dir, sshri.hostname)
        if sshri.port:
            ctrl_path += ":%s" % sshri.port

        # do we know it already?
        if ctrl_path in self._connections:
            return self._connections[ctrl_path]
        else:
            c = SSHConnection(ctrl_path, sshri.hostname)
            self._connections[ctrl_path] = c
            return c

    def close(self):
        """Closes all connections, known to this instance.
        """
        if self._connections:
            lgr.debug("Closing %d SSH connections..." % len(self._connections))
            for cnct in self._connections:
                self._connections[cnct].close()

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user