{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"DataLad aims to expose (scientific) data available online as a unified data\ndistribution with the convenience of git-annex repositories as a backend.\"\"\"\n\n\n# Other imports are interspersed with lgr.debug to ease troubleshooting startup\n# delays etc.\n\n# If there is a bundled git, make sure GitPython uses it too:\nfrom datalad.cmd import GitRunner\nGitRunner._check_git_path()\nif GitRunner._GIT_PATH:\n    import os\n    os.environ['GIT_PYTHON_GIT_EXECUTABLE'] = \\\n        os.path.join(GitRunner._GIT_PATH, 'git')\n\nfrom .config import ConfigManager\ncfg = ConfigManager()\n\nfrom .log import lgr\nimport atexit\nfrom datalad.utils import on_windows\n\nif not on_windows:\n    lgr.log(5, \"Instantiating ssh manager\")\n    from .support.sshconnector import SSHManager\n    ssh_manager = SSHManager()\n    atexit.register(ssh_manager.close, allow_fail=False)\nelse:\n    ssh_manager = None\n\ntry:\n    # this will fix the rendering of ANSI escape sequences\n    # for colored terminal output on windows\n    # it will do nothing on any other platform, hence it\n    # is safe to call unconditionally\n    import colorama\n    colorama.init()\n    atexit.register(colorama.deinit)\nexcept ImportError as e:\n    if on_windows:\n        from datalad.dochelpers import exc_str\n        lgr.warning(\n            \"'colorama' Python module missing, terminal output may look garbled [%s]\",\n            exc_str(e))\n    pass\n\natexit.register(lgr.log, 5, \"Exiting\")\n\nfrom .version import __version__\n\n\ndef test(module='datalad', verbose=False, nocapture=False, pdb=False, stop=False):\n    \"\"\"A helper to run datalad's tests.  Requires nose\n    \"\"\"\n    argv = [] #module]\n    # could make it 'smarter' but decided to be explicit so later we could\n    # easily migrate to another runner without changing any API here\n    if verbose:\n        argv.append('-v')\n    if nocapture:\n        argv.append('-s')\n    if pdb:\n        argv.append('--pdb')\n    if stop:\n        argv.append('--stop')\n    from datalad.support.third.nosetester import NoseTester\n    tester = NoseTester(module)\n    tester.package_name = module.split('.', 1)[0]\n    tester.test(extra_argv=argv)\n\ntest.__test__ = False\n\n# Following fixtures are necessary at the top level __init__ for fixtures which\n# would cover all **/tests and not just datalad/tests/\n\n# To store settings which setup_package changes and teardown_package should return\n_test_states = {\n    'loglevel': None,\n    'DATALAD_LOG_LEVEL': None,\n}\n\n\ndef setup_package():\n    import os\n\n    # To overcome pybuild overriding HOME but us possibly wanting our\n    # own HOME where we pre-setup git for testing (name, email)\n    if 'GIT_HOME' in os.environ:\n        os.environ['HOME'] = os.environ['GIT_HOME']\n\n    # For now we will just verify that it is ready to run the tests\n    from datalad.support.gitrepo import check_git_configured\n    check_git_configured()\n\n    # To overcome pybuild by default defining http{,s}_proxy we would need\n    # to define them to e.g. empty value so it wouldn't bother touching them.\n    # But then haskell libraries do not digest empty value nicely, so we just\n    # pop them out from the environment\n    for ev in ('http_proxy', 'https_proxy'):\n        if ev in os.environ and not (os.environ[ev]):\n            lgr.debug(\"Removing %s from the environment since it is empty\", ev)\n            os.environ.pop(ev)\n\n    DATALAD_LOG_LEVEL = os.environ.get('DATALAD_LOG_LEVEL', None)\n    if DATALAD_LOG_LEVEL is None:\n        # very very silent.  Tests introspecting logs should use\n        # swallow_logs(new_level=...)\n        _test_states['loglevel'] = lgr.getEffectiveLevel()\n        lgr.setLevel(100)\n\n        # And we should also set it within environ so underlying commands also stay silent\n        _test_states['DATALAD_LOG_LEVEL'] = DATALAD_LOG_LEVEL\n        os.environ['DATALAD_LOG_LEVEL'] = '100'\n    else:\n        # We are not overriding them, since explicitly were asked to have some log level\n        _test_states['loglevel'] = None\n\n    # Set to non-interactive UI\n    from datalad.ui import ui\n    _test_states['ui_backend'] = ui.backend\n    # obtain() since that one consults for the default value\n    ui.set_backend(cfg.obtain('datalad.tests.ui.backend'))\n\n\ndef teardown_package():\n    import os\n    if os.environ.get('DATALAD_TESTS_NOTEARDOWN'):\n        return\n    from datalad.ui import ui\n    ui.set_backend(_test_states['ui_backend'])\n    if _test_states['loglevel'] is not None:\n        lgr.setLevel(_test_states['loglevel'])\n        if _test_states['DATALAD_LOG_LEVEL'] is None:\n            os.environ.pop('DATALAD_LOG_LEVEL')\n        else:\n            os.environ['DATALAD_LOG_LEVEL'] = _test_states['DATALAD_LOG_LEVEL']\n\n    from datalad.tests import _TEMP_PATHS_GENERATED\n    from datalad.tests.utils import rmtemp\n    if len(_TEMP_PATHS_GENERATED):\n        msg = \"Removing %d dirs/files: %s\" % (len(_TEMP_PATHS_GENERATED), ', '.join(_TEMP_PATHS_GENERATED))\n    else:\n        msg = \"Nothing to remove\"\n    lgr.debug(\"Teardown tests. \" + msg)\n    for path in _TEMP_PATHS_GENERATED:\n        rmtemp(path, ignore_errors=True)\n\n    lgr.debug(\"Printing versioning information collected so far\")\n    from datalad.support.external_versions import external_versions as ev\n    print(ev.dumps(query=True))\n\nlgr.log(5, \"Done importing main __init__\")\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"DataLad aims to expose (scientific) data available online as a unified data\ndistribution with the convenience of git-annex repositories as a backend.\"\"\"\n\n\n# Other imports are interspersed with lgr.debug to ease troubleshooting startup\n# delays etc.\n\n# If there is a bundled git, make sure GitPython uses it too:\nfrom datalad.cmd import GitRunner\nGitRunner._check_git_path()\nif GitRunner._GIT_PATH:\n    import os\n    os.environ['GIT_PYTHON_GIT_EXECUTABLE'] = \\\n        os.path.join(GitRunner._GIT_PATH, 'git')\n\nfrom .config import ConfigManager\ncfg = ConfigManager()\n\nfrom .log import lgr\nimport atexit\nfrom datalad.utils import on_windows\n\nif not on_windows:\n    lgr.log(5, \"Instantiating ssh manager\")\n    from .support.sshconnector import SSHManager\n    ssh_manager = SSHManager()\n    atexit.register(ssh_manager.close, allow_fail=False)\nelse:\n    ssh_manager = None\n\ntry:\n    # this will fix the rendering of ANSI escape sequences\n    # for colored terminal output on windows\n    # it will do nothing on any other platform, hence it\n    # is safe to call unconditionally\n    import colorama\n    colorama.init()\n    atexit.register(colorama.deinit)\nexcept ImportError as e:\n    if on_windows:\n        from datalad.dochelpers import exc_str\n        lgr.warning(\n            \"'colorama' Python module missing, terminal output may look garbled [%s]\",\n            exc_str(e))\n    pass\n\natexit.register(lgr.log, 5, \"Exiting\")\n\nfrom .version import __version__\n\n\ndef test(module='datalad', verbose=False, nocapture=False, pdb=False, stop=False):\n    \"\"\"A helper to run datalad's tests.  Requires nose\n    \"\"\"\n    argv = [] #module]\n    # could make it 'smarter' but decided to be explicit so later we could\n    # easily migrate to another runner without changing any API here\n    if verbose:\n        argv.append('-v')\n    if nocapture:\n        argv.append('-s')\n    if pdb:\n        argv.append('--pdb')\n    if stop:\n        argv.append('--stop')\n    from datalad.support.third.nosetester import NoseTester\n    tester = NoseTester(module)\n    tester.package_name = module.split('.', 1)[0]\n    tester.test(extra_argv=argv)\n\ntest.__test__ = False\n\n# Following fixtures are necessary at the top level __init__ for fixtures which\n# would cover all **/tests and not just datalad/tests/\n\n# To store settings which setup_package changes and teardown_package should return\n_test_states = {\n    'loglevel': None,\n    'DATALAD_LOG_LEVEL': None,\n    'HOME': None,\n}\n\n\ndef setup_package():\n    import os\n\n    _test_states['HOME'] = os.environ.get('HOME', None)\n\n    # To overcome pybuild overriding HOME but us possibly wanting our\n    # own HOME where we pre-setup git for testing (name, email)\n    if 'GIT_HOME' in os.environ:\n        os.environ['HOME'] = os.environ['GIT_HOME']\n    else:\n        # we setup our own new HOME, the BEST and HUGE one\n        from datalad.utils import make_tempfile\n        from datalad.tests import _TEMP_PATHS_GENERATED\n        # TODO: split into a function + context manager\n        with make_tempfile(mkdir=True) as new_home:\n            os.environ['HOME'] = new_home\n        os.makedirs(new_home)\n        with open(os.path.join(new_home, '.gitconfig'), 'w') as f:\n            f.write(\"\"\"\\\n[user]\n\tname = DataLad Tester\n\temail = test@example.com\n\"\"\")\n        _TEMP_PATHS_GENERATED.append(new_home)\n\n    # For now we will just verify that it is ready to run the tests\n    from datalad.support.gitrepo import check_git_configured\n    check_git_configured()\n\n    # To overcome pybuild by default defining http{,s}_proxy we would need\n    # to define them to e.g. empty value so it wouldn't bother touching them.\n    # But then haskell libraries do not digest empty value nicely, so we just\n    # pop them out from the environment\n    for ev in ('http_proxy', 'https_proxy'):\n        if ev in os.environ and not (os.environ[ev]):\n            lgr.debug(\"Removing %s from the environment since it is empty\", ev)\n            os.environ.pop(ev)\n\n    DATALAD_LOG_LEVEL = os.environ.get('DATALAD_LOG_LEVEL', None)\n    if DATALAD_LOG_LEVEL is None:\n        # very very silent.  Tests introspecting logs should use\n        # swallow_logs(new_level=...)\n        _test_states['loglevel'] = lgr.getEffectiveLevel()\n        lgr.setLevel(100)\n\n        # And we should also set it within environ so underlying commands also stay silent\n        _test_states['DATALAD_LOG_LEVEL'] = DATALAD_LOG_LEVEL\n        os.environ['DATALAD_LOG_LEVEL'] = '100'\n    else:\n        # We are not overriding them, since explicitly were asked to have some log level\n        _test_states['loglevel'] = None\n\n    # Set to non-interactive UI\n    from datalad.ui import ui\n    _test_states['ui_backend'] = ui.backend\n    # obtain() since that one consults for the default value\n    ui.set_backend(cfg.obtain('datalad.tests.ui.backend'))\n\n\ndef teardown_package():\n    import os\n    if os.environ.get('DATALAD_TESTS_NOTEARDOWN'):\n        return\n    from datalad.ui import ui\n    ui.set_backend(_test_states['ui_backend'])\n    if _test_states['loglevel'] is not None:\n        lgr.setLevel(_test_states['loglevel'])\n        if _test_states['DATALAD_LOG_LEVEL'] is None:\n            os.environ.pop('DATALAD_LOG_LEVEL')\n        else:\n            os.environ['DATALAD_LOG_LEVEL'] = _test_states['DATALAD_LOG_LEVEL']\n\n    from datalad.tests import _TEMP_PATHS_GENERATED\n    from datalad.tests.utils import rmtemp\n    if len(_TEMP_PATHS_GENERATED):\n        msg = \"Removing %d dirs/files: %s\" % (len(_TEMP_PATHS_GENERATED), ', '.join(_TEMP_PATHS_GENERATED))\n    else:\n        msg = \"Nothing to remove\"\n    lgr.debug(\"Teardown tests. \" + msg)\n    for path in _TEMP_PATHS_GENERATED:\n        rmtemp(path, ignore_errors=True)\n\n    if _test_states['HOME'] is not None:\n        os.environ['HOME'] = _test_states['HOME']\n\n    lgr.debug(\"Printing versioning information collected so far\")\n    from datalad.support.external_versions import external_versions as ev\n    print(ev.dumps(query=True))\n\nlgr.log(5, \"Done importing main __init__\")\n",
            "header": {
                "file": {
                    "path": "datalad/__init__.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "aa1647791",
                "new_commit_hash": "b9e7ff2c9",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/__init__.py"
            },
            "plus_file": {
                "path": "b/datalad/__init__.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 85,
                            "length": 16
                        },
                        "new_range": {
                            "start": 85,
                            "length": 34
                        },
                        "section": "test.__test__ = False"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "_test_states = {",
                                "type": "context"
                            },
                            {
                                "content": "    'loglevel': None,",
                                "type": "context"
                            },
                            {
                                "content": "    'DATALAD_LOG_LEVEL': None,",
                                "type": "context"
                            },
                            {
                                "content": "    'HOME': None,",
                                "type": "added"
                            },
                            {
                                "content": "}",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def setup_package():",
                                "type": "context"
                            },
                            {
                                "content": "    import os",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    _test_states['HOME'] = os.environ.get('HOME', None)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    # To overcome pybuild overriding HOME but us possibly wanting our",
                                "type": "context"
                            },
                            {
                                "content": "    # own HOME where we pre-setup git for testing (name, email)",
                                "type": "context"
                            },
                            {
                                "content": "    if 'GIT_HOME' in os.environ:",
                                "type": "context"
                            },
                            {
                                "content": "        os.environ['HOME'] = os.environ['GIT_HOME']",
                                "type": "context"
                            },
                            {
                                "content": "    else:",
                                "type": "added"
                            },
                            {
                                "content": "        # we setup our own new HOME, the BEST and HUGE one",
                                "type": "added"
                            },
                            {
                                "content": "        from datalad.utils import make_tempfile",
                                "type": "added"
                            },
                            {
                                "content": "        from datalad.tests import _TEMP_PATHS_GENERATED",
                                "type": "added"
                            },
                            {
                                "content": "        # TODO: split into a function + context manager",
                                "type": "added"
                            },
                            {
                                "content": "        with make_tempfile(mkdir=True) as new_home:",
                                "type": "added"
                            },
                            {
                                "content": "            os.environ['HOME'] = new_home",
                                "type": "added"
                            },
                            {
                                "content": "        os.makedirs(new_home)",
                                "type": "added"
                            },
                            {
                                "content": "        with open(os.path.join(new_home, '.gitconfig'), 'w') as f:",
                                "type": "added"
                            },
                            {
                                "content": "            f.write(\"\"\"\\",
                                "type": "added"
                            },
                            {
                                "content": "[user]",
                                "type": "added"
                            },
                            {
                                "content": "\tname = DataLad Tester",
                                "type": "added"
                            },
                            {
                                "content": "\temail = test@example.com",
                                "type": "added"
                            },
                            {
                                "content": "\"\"\")",
                                "type": "added"
                            },
                            {
                                "content": "        _TEMP_PATHS_GENERATED.append(new_home)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # For now we will just verify that it is ready to run the tests",
                                "type": "context"
                            },
                            {
                                "content": "    from datalad.support.gitrepo import check_git_configured",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/__init__.py",
                            "type": "function",
                            "name": "setup_package",
                            "content": "def setup_package():\n    import os\n\n    _test_states['HOME'] = os.environ.get('HOME', None)\n\n    # To overcome pybuild overriding HOME but us possibly wanting our\n    # own HOME where we pre-setup git for testing (name, email)\n    if 'GIT_HOME' in os.environ:\n        os.environ['HOME'] = os.environ['GIT_HOME']\n    else:\n        # we setup our own new HOME, the BEST and HUGE one\n        from datalad.utils import make_tempfile\n        from datalad.tests import _TEMP_PATHS_GENERATED\n        # TODO: split into a function + context manager\n        with make_tempfile(mkdir=True) as new_home:\n            os.environ['HOME'] = new_home\n        os.makedirs(new_home)\n        with open(os.path.join(new_home, '.gitconfig'), 'w') as f:\n            f.write(\"\"\"\\\n[user]\n\tname = DataLad Tester\n\temail = test@example.com\n\"\"\")\n        _TEMP_PATHS_GENERATED.append(new_home)\n\n    # For now we will just verify that it is ready to run the tests\n    from datalad.support.gitrepo import check_git_configured\n    check_git_configured()\n\n    # To overcome pybuild by default defining http{,s}_proxy we would need\n    # to define them to e.g. empty value so it wouldn't bother touching them.\n    # But then haskell libraries do not digest empty value nicely, so we just\n    # pop them out from the environment\n    for ev in ('http_proxy', 'https_proxy'):\n        if ev in os.environ and not (os.environ[ev]):\n            lgr.debug(\"Removing %s from the environment since it is empty\", ev)\n            os.environ.pop(ev)\n\n    DATALAD_LOG_LEVEL = os.environ.get('DATALAD_LOG_LEVEL', None)\n    if DATALAD_LOG_LEVEL is None:\n        # very very silent.  Tests introspecting logs should use\n        # swallow_logs(new_level=...)\n        _test_states['loglevel'] = lgr.getEffectiveLevel()\n        lgr.setLevel(100)\n\n        # And we should also set it within environ so underlying commands also stay silent\n        _test_states['DATALAD_LOG_LEVEL'] = DATALAD_LOG_LEVEL\n        os.environ['DATALAD_LOG_LEVEL'] = '100'\n    else:\n        # We are not overriding them, since explicitly were asked to have some log level\n        _test_states['loglevel'] = None\n\n    # Set to non-interactive UI\n    from datalad.ui import ui\n    _test_states['ui_backend'] = ui.backend\n    # obtain() since that one consults for the default value\n    ui.set_backend(cfg.obtain('datalad.tests.ui.backend'))",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 92,
                            "end_lineno": 148,
                            "parent": null
                        }
                    ],
                    "added_entities": [
                        {
                            "file_name": "datalad/__init__.py",
                            "type": "statement",
                            "name": "",
                            "content": "_test_states = {\n    'loglevel': None,\n    'DATALAD_LOG_LEVEL': None,\n    'HOME': None,\n}",
                            "ast_type_str": "Assign",
                            "start_lineno": 85,
                            "end_lineno": 89,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 153,
                            "length": 6
                        },
                        "new_range": {
                            "start": 171,
                            "length": 9
                        },
                        "section": "def teardown_package():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    for path in _TEMP_PATHS_GENERATED:",
                                "type": "context"
                            },
                            {
                                "content": "        rmtemp(path, ignore_errors=True)",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    if _test_states['HOME'] is not None:",
                                "type": "added"
                            },
                            {
                                "content": "        os.environ['HOME'] = _test_states['HOME']",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    lgr.debug(\"Printing versioning information collected so far\")",
                                "type": "context"
                            },
                            {
                                "content": "    from datalad.support.external_versions import external_versions as ev",
                                "type": "context"
                            },
                            {
                                "content": "    print(ev.dumps(query=True))",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/__init__.py",
                            "type": "function",
                            "name": "teardown_package",
                            "content": "def teardown_package():\n    import os\n    if os.environ.get('DATALAD_TESTS_NOTEARDOWN'):\n        return\n    from datalad.ui import ui\n    ui.set_backend(_test_states['ui_backend'])\n    if _test_states['loglevel'] is not None:\n        lgr.setLevel(_test_states['loglevel'])\n        if _test_states['DATALAD_LOG_LEVEL'] is None:\n            os.environ.pop('DATALAD_LOG_LEVEL')\n        else:\n            os.environ['DATALAD_LOG_LEVEL'] = _test_states['DATALAD_LOG_LEVEL']\n\n    from datalad.tests import _TEMP_PATHS_GENERATED\n    from datalad.tests.utils import rmtemp\n    if len(_TEMP_PATHS_GENERATED):\n        msg = \"Removing %d dirs/files: %s\" % (len(_TEMP_PATHS_GENERATED), ', '.join(_TEMP_PATHS_GENERATED))\n    else:\n        msg = \"Nothing to remove\"\n    lgr.debug(\"Teardown tests. \" + msg)\n    for path in _TEMP_PATHS_GENERATED:\n        rmtemp(path, ignore_errors=True)\n\n    if _test_states['HOME'] is not None:\n        os.environ['HOME'] = _test_states['HOME']\n\n    lgr.debug(\"Printing versioning information collected so far\")\n    from datalad.support.external_versions import external_versions as ev\n    print(ev.dumps(query=True))",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 151,
                            "end_lineno": 179,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface to Git via GitPython\n\nFor further information on GitPython see http://gitpython.readthedocs.org/\n\n\"\"\"\n\nimport logging\nimport re\nimport shlex\nimport os\nfrom os import linesep\nfrom os.path import join as opj\nfrom os.path import exists\nfrom os.path import normpath\nfrom os.path import isabs\nfrom os.path import commonprefix\nfrom os.path import relpath\nfrom os.path import realpath\nfrom os.path import dirname\nfrom os.path import basename\nfrom os.path import curdir\nfrom os.path import pardir\nfrom os.path import sep\nimport posixpath\nfrom weakref import WeakValueDictionary\n\n\nfrom six import string_types\nfrom six import add_metaclass\nfrom functools import wraps\nimport git as gitpy\nfrom git.exc import GitCommandError\nfrom git.exc import NoSuchPathError\nfrom git.exc import InvalidGitRepositoryError\nfrom git.objects.blob import Blob\n\nfrom datalad import ssh_manager\nfrom datalad.cmd import GitRunner\nfrom datalad.consts import GIT_SSH_COMMAND\nfrom datalad.dochelpers import exc_str\nfrom datalad.config import ConfigManager\nfrom datalad.utils import assure_list\nfrom datalad.utils import optional_args\nfrom datalad.utils import on_windows\nfrom datalad.utils import getpwd\nfrom datalad.utils import updated\nfrom datalad.utils import posix_relpath\n\n\n# imports from same module:\nfrom .external_versions import external_versions\nfrom .exceptions import CommandError\nfrom .exceptions import DeprecatedError\nfrom .exceptions import FileNotInRepositoryError\nfrom .exceptions import MissingBranchError\nfrom .network import RI\nfrom .network import is_ssh\nfrom .repo import Flyweight\nfrom .repo import RepoInterface\n\n# shortcuts\n_curdirsep = curdir + sep\n_pardirsep = pardir + sep\n\n\nlgr = logging.getLogger('datalad.gitrepo')\n\n# Override default GitPython's DB backend to talk directly to git so it doesn't\n# interfere with possible operations performed by gc/repack\ndefault_git_odbt = gitpy.GitCmdObjectDB\n\n# TODO: Figure out how GIT_PYTHON_TRACE ('full') is supposed to be used.\n# Didn't work as expected on a first try. Probably there is a neatier way to\n# log Exceptions from git commands.\n\n\n# TODO: ignore leading and/or trailing underscore to allow for\n# python-reserved words\n@optional_args\ndef kwargs_to_options(func, split_single_char_options=True,\n                      target_kw='options'):\n    \"\"\"Decorator to provide convenient way to pass options to command calls.\n\n    Parameters\n    ----------\n    func: Callable\n        function to decorate\n    split_single_char_options: bool\n        whether or not to split key and value of single char keyword arguments\n        into two subsequent entries of the list\n    target_kw: str\n        keyword argument to pass the generated list of cmdline arguments to\n\n    Returns\n    -------\n    Callable\n    \"\"\"\n\n    # TODO: don't overwrite options, but join\n\n    @wraps(func)\n    def newfunc(self, *args, **kwargs):\n        t_kwargs = dict()\n        t_kwargs[target_kw] = \\\n            gitpy.Git().transform_kwargs(\n                split_single_char_options=split_single_char_options,\n                **kwargs)\n        return func(self, *args, **t_kwargs)\n    return newfunc\n\n\ndef to_options(**kwargs):\n    \"\"\"Transform keyword arguments into a list of cmdline options\n\n    Parameters\n    ----------\n    split_single_char_options: bool\n\n    kwargs:\n\n    Returns\n    -------\n    list\n    \"\"\"\n    # TODO: borrow_docs!\n\n    return gitpy.Git().transform_kwargs(**kwargs)\n\n\ndef _normalize_path(base_dir, path):\n    \"\"\"Helper to check paths passed to methods of this class.\n\n    Checks whether `path` is beneath `base_dir` and normalizes it.\n    Additionally paths are converted into relative paths with respect to\n    `base_dir`, considering PWD in case of relative paths. This\n    is intended to be used in repository classes, which means that\n    `base_dir` usually will be the repository's base directory.\n\n    Parameters\n    ----------\n    path: str\n        path to be normalized\n    base_dir: str\n        directory to serve as base to normalized, relative paths\n\n    Returns\n    -------\n    str:\n        path, that is a relative path with respect to `base_dir`\n    \"\"\"\n    if not path:\n        return path\n\n    base_dir = realpath(base_dir)  # realpath OK\n    # path = normpath(path)\n    # Note: disabled normpath, because it may break paths containing symlinks;\n    # But we don't want to realpath relative paths, in case cwd isn't the\n    # correct base.\n\n    if isabs(path):\n        # path might already be a symlink pointing to annex etc,\n        # so realpath only its directory, to get \"inline\" with\n        # realpath(base_dir) above\n        path = opj(realpath(dirname(path)), basename(path))  # realpath OK\n    # Executive decision was made to not do this kind of magic!\n    #\n    # elif commonprefix([realpath(getpwd()), base_dir]) == base_dir:\n    #     # If we are inside repository, rebuilt relative paths.\n    #     path = opj(realpath(getpwd()), path)\n    #\n    # BUT with relative curdir/pardir start it would assume relative to curdir\n    #\n    elif path.startswith(_curdirsep) or path.startswith(_pardirsep):\n        path = normpath(opj(realpath(getpwd()), path))  # realpath OK\n    else:\n        # We were called from outside the repo. Therefore relative paths\n        # are interpreted as being relative to self.path already.\n        return path\n\n    if commonprefix([path, base_dir]) != base_dir:\n        raise FileNotInRepositoryError(msg=\"Path outside repository: %s\"\n                                           % path, filename=path)\n\n    return relpath(path, start=base_dir)\n\n\n@optional_args\ndef normalize_path(func):\n    \"\"\"Decorator to provide unified path conversion for a single file\n\n    Unlike normalize_paths, intended to be used for functions dealing with a\n    single filename at a time\n\n    Note\n    ----\n    This is intended to be used within the repository classes and therefore\n    returns a class method!\n\n    The decorated function is expected to take a path at\n    first positional argument (after 'self'). Additionally the class `func`\n    is a member of, is expected to have an attribute 'path'.\n    \"\"\"\n\n    @wraps(func)\n    def newfunc(self, file_, *args, **kwargs):\n        file_new = _normalize_path(self.path, file_)\n        return func(self, file_new, *args, **kwargs)\n\n    return newfunc\n\n\n@optional_args\ndef normalize_paths(func, match_return_type=True, map_filenames_back=False,\n                    serialize=False):\n    \"\"\"Decorator to provide unified path conversions.\n\n    Note\n    ----\n    This is intended to be used within the repository classes and therefore\n    returns a class method!\n\n    The decorated function is expected to take a path or a list of paths at\n    first positional argument (after 'self'). Additionally the class `func`\n    is a member of, is expected to have an attribute 'path'.\n\n    Accepts either a list of paths or a single path in a str. Passes a list\n    to decorated function either way, but would return based on the value of\n    match_return_type and possibly input argument.\n\n    If a call to the wrapped function includes normalize_path and it is False\n    no normalization happens for that function call (used for calls to wrapped\n    functions within wrapped functions, while possible CWD is within a\n    repository)\n\n    Parameters\n    ----------\n    match_return_type : bool, optional\n      If True, and a single string was passed in, it would return the first\n      element of the output (after verifying that it is a list of length 1).\n      It makes easier to work with single files input.\n    map_filenames_back : bool, optional\n      If True and returned value is a dictionary, it assumes to carry entries\n      one per file, and then filenames are mapped back to as provided from the\n      normalized (from the root of the repo) paths\n    serialize : bool, optional\n      Loop through files giving only a single one to the function one at a time.\n      This allows to simplify implementation and interface to annex commands\n      which do not take multiple args in the same call (e.g. checkpresentkey)\n    \"\"\"\n\n    @wraps(func)\n    def newfunc(self, files, *args, **kwargs):\n\n        normalize = _normalize_path if kwargs.pop('normalize_paths', True) \\\n            else lambda rpath, filepath: filepath\n\n        if files:\n            if isinstance(files, string_types) or not files:\n                files_new = [normalize(self.path, files)]\n                single_file = True\n            elif isinstance(files, list):\n                files_new = [normalize(self.path, path) for path in files]\n                single_file = False\n            else:\n                raise ValueError(\"_files_decorator: Don't know how to handle \"\n                                 \"instance of %s.\" % type(files))\n        else:\n            single_file = None\n            files_new = []\n\n        if map_filenames_back:\n            def remap_filenames(out):\n                \"\"\"Helper to map files back to non-normalized paths\"\"\"\n                if isinstance(out, dict):\n                    assert(len(out) == len(files_new))\n                    files_ = [files] if single_file else files\n                    mapped = out.__class__()\n                    for fin, fout in zip(files_, files_new):\n                        mapped[fin] = out[fout]\n                    return mapped\n                else:\n                    return out\n        else:\n            remap_filenames = lambda x: x\n\n        if serialize:  # and not single_file:\n            result = [\n                func(self, f, *args, **kwargs)\n                for f in files_new\n            ]\n        else:\n            result = func(self, files_new, *args, **kwargs)\n\n        if single_file is None:\n            # no files were provided, nothing we can do really\n            return result\n        elif (result is None) or not match_return_type or not single_file:\n            # If function doesn't return anything or no denormalization\n            # was requested or it was not a single file\n            return remap_filenames(result)\n        elif single_file:\n            if len(result) != 1:\n                # Magic doesn't apply\n                return remap_filenames(result)\n            elif isinstance(result, (list, tuple)):\n                return result[0]\n            elif isinstance(result, dict) and tuple(result)[0] == files_new[0]:\n                # assume that returned dictionary has files as keys.\n                return tuple(result.values())[0]\n            else:\n                # no magic can apply\n                return remap_filenames(result)\n        else:\n            return RuntimeError(\"should have not got here... check logic\")\n\n    return newfunc\n\n\ndef check_git_configured():\n    \"\"\"Do a check if git is configured (user.name and user.email are set)\n\n    Raises\n    ------\n    RuntimeError  if any of those two ariables are not set\n    \"\"\"\n\n    check_runner = GitRunner()\n    for c in 'user.name', 'user.email':\n        try:\n            check_runner.run(['git', 'config', '--global', c])\n        except CommandError as exc:\n            lgr.debug(\"Failed to verify that git is configured: %s\",\n                      exc_str(exc))\n            raise RuntimeError(\n                \"You must configure git first (set both user.name and \"\n                \"user.email) settings before using DataLad.\"\n            )\n\n\ndef _remove_empty_items(list_):\n    \"\"\"Remove empty entries from list\n\n    This is needed, since some functions of GitPython may convert\n    an empty entry to '.', when used with a list of paths.\n\n    Parameter:\n    ----------\n    list_: list of str\n\n    Returns\n    -------\n    list of str\n    \"\"\"\n    if not isinstance(list_, list):\n        lgr.warning(\n            \"_remove_empty_items() called with non-list type: %s\" % type(list_))\n        return list_\n    return [file_ for file_ in list_ if file_]\n\n\ndef Repo(*args, **kwargs):\n    \"\"\"Factory method around gitpy.Repo to consistently initiate with different\n    backend\n    \"\"\"\n    # TODO: This probably doesn't work as intended (or at least not as\n    #       consistently as intended). gitpy.Repo could be instantiated by\n    #       classmethods Repo.init or Repo.clone_from. In these cases 'odbt'\n    #       would be needed as a paramter to these methods instead of the\n    #       constructor.\n    if 'odbt' not in kwargs:\n        kwargs['odbt'] = default_git_odbt\n    return gitpy.Repo(*args, **kwargs)\n\n\ndef split_remote_branch(branch):\n    \"\"\"Splits a remote branch's name into the name of the remote and the name\n    of the branch.\n\n    Parameters\n    ----------\n    branch: str\n      the remote branch's name to split\n\n    Returns\n    -------\n    list of str\n    \"\"\"\n    assert '/' in branch, \\\n        \"remote branch %s must have had a /\" % branch\n    assert not branch.endswith('/'), \\\n        \"branch name with trailing / is invalid. (%s)\" % branch\n    return branch.split('/', 1)\n\n\n@add_metaclass(Flyweight)\nclass GitRepo(RepoInterface):\n    \"\"\"Representation of a git repository\n\n    \"\"\"\n\n    # We use our sshrun helper\n    GIT_SSH_ENV = {'GIT_SSH_COMMAND': GIT_SSH_COMMAND}\n\n    # Just a non-functional example:\n    # must be implemented, since abstract in RepoInterface:\n    def sth_like_file_has_content(self):\n        return \"Yes, if it's in the index\"\n\n    # We must check git config to have name and email set, but\n    # should do it once\n    _config_checked = False\n\n    # Begin Flyweight:\n\n    _unique_instances = WeakValueDictionary()\n\n    @classmethod\n    def _flyweight_id_from_args(cls, *args, **kwargs):\n\n        if args:\n            # to a certain degree we need to simulate an actual call to __init__\n            # and make sure, passed arguments are fitting:\n            # TODO: Figure out, whether there is a cleaner way to do this in a\n            # generic fashion\n            assert('path' not in kwargs)\n            path = args[0]\n            args = args[1:]\n        elif 'path' in kwargs:\n            path = kwargs.pop('path')\n        else:\n            raise TypeError(\"__init__() requires argument `path`\")\n\n        if path is None:\n            raise AttributeError\n\n        # Sanity check for argument `path`:\n        # raise if we cannot deal with `path` at all or\n        # if it is not a local thing:\n        path = RI(path).localpath\n        # resolve symlinks to make sure we have exactly one instance per\n        # physical repository at a time\n        path = realpath(path)\n        kwargs['path'] = path\n        return path, args, kwargs\n\n    @classmethod\n    def _flyweight_invalid(cls, id_):\n        return not cls.is_valid_repo(id_)\n\n    @classmethod\n    def _flyweight_reject(cls, id_, *args, **kwargs):\n        # TODO:\n        # This is a temporary approach. See PR # ...\n        # create = kwargs.pop('create', None)\n        # kwargs.pop('path', None)\n        # if create and kwargs:\n        #     # we have `create` plus options other than `path`\n        #     return \"Call to {0}() with args {1} and kwargs {2} conflicts \" \\\n        #            \"with existing instance {3}.\" \\\n        #            \"This is likely to be caused by inconsistent logic in \" \\\n        #            \"your code.\" \\\n        #            \"\".format(cls, args, kwargs, cls._unique_instances[id_])\n        pass\n\n    # End Flyweight\n\n    def __init__(self, path, url=None, runner=None, create=True,\n                 git_opts=None, repo=None, **kwargs):\n        \"\"\"Creates representation of git repository at `path`.\n\n        Can also be used to create a git repository at `path`.\n\n        Parameters\n        ----------\n        path: str\n          path to the git repository; In case it's not an absolute path,\n          it's relative to PWD\n        url: str, optional\n          DEPRECATED -- use .clone() class method\n          url to the to-be-cloned repository. Requires a valid git url\n          according to:\n          http://www.kernel.org/pub/software/scm/git/docs/git-clone.html#URLS .\n        create: bool, optional\n          if true, creates a git repository at `path` if there is none. Also\n          creates `path`, if it doesn't exist.\n          If set to false, an exception is raised in case `path` doesn't exist\n          or doesn't contain a git repository.\n        repo: git.Repo, optional\n          GitPython's Repo instance to (re)use if provided\n        kwargs:\n          keyword arguments serving as additional options to the git-init\n          command. Therefore, it makes sense only if called with `create`.\n\n          Generally, this way of passing options to the git executable is\n          (or will be) used a lot in this class. It's a transformation of\n          python-style keyword arguments (or a `dict`) to command line arguments,\n          provided by GitPython.\n\n          A single character keyword will be prefixed by '-', multiple characters\n          by '--'. An underscore in the keyword becomes a dash. The value of the\n          keyword argument is used as the value for the corresponding command\n          line argument. Assigning a boolean creates a flag.\n\n          Examples:\n          no_commit=True => --no-commit\n          C='/my/path'   => -C /my/path\n\n        \"\"\"\n\n        if url is not None:\n            raise DeprecatedError(\n                new=\".clone() class method\",\n                version=\"0.5.0\",\n                msg=\"RF: url passed to init()\"\n            )\n\n        # So that we \"share\" control paths with git/git-annex\n        if ssh_manager:\n            ssh_manager.assure_initialized()\n\n        if not GitRepo._config_checked:\n            check_git_configured()\n            GitRepo._config_checked = True\n\n        self.realpath = realpath(path)\n        # note: we may also want to distinguish between a path to the worktree\n        # and the actual repository\n\n        # Disable automatic garbage and autopacking\n        self._GIT_COMMON_OPTIONS = ['-c', 'receive.autogc=0', '-c', 'gc.auto=0']\n        # actually no need with default GitPython db backend not in memory\n        # default_git_odbt but still allows for faster testing etc.\n        # May be eventually we would make it switchable _GIT_COMMON_OPTIONS = []\n\n        if git_opts is None:\n            git_opts = {}\n        if kwargs:\n            git_opts.update(kwargs)\n\n        self.path = path\n        self.cmd_call_wrapper = runner or GitRunner(cwd=self.path)\n        self._repo = repo\n        self._cfg = None\n\n        _valid_repo = GitRepo.is_valid_repo(path)\n        if create and not _valid_repo:\n            if repo is not None:\n                # `repo` passed with `create`, which doesn't make sense\n                raise TypeError(\"argument 'repo' must not be used with 'create'\")\n\n            try:\n                lgr.debug(\n                    \"Initialize empty Git repository at '%s'%s\",\n                    path,\n                    ' %s' % git_opts if git_opts else '')\n                self._repo = self.cmd_call_wrapper(gitpy.Repo.init, path,\n                                                   mkdir=True,\n                                                   odbt=default_git_odbt,\n                                                   **git_opts)\n            except GitCommandError as e:\n                lgr.error(exc_str(e))\n                raise\n        else:\n            # Note: We used to call gitpy.Repo(path) here, which potentially\n            # raised NoSuchPathError or InvalidGitRepositoryError. This is\n            # used by callers of GitRepo.__init__() to detect whether we have a\n            # valid repo at `path`. Now, with switching to lazy loading property\n            # `repo`, we detect those cases without instantiating a\n            # gitpy.Repo().\n\n            if not exists(path):\n                raise NoSuchPathError(path)\n            if not _valid_repo:\n                raise InvalidGitRepositoryError(path)\n\n        # inject git options into GitPython's git call wrapper:\n        # Note: `None` currently can happen, when Runner's protocol prevents\n        # calls above from being actually executed (DryRunProtocol)\n        if self._repo is not None:\n            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS\n\n        # with DryRunProtocol path might still not exist\n        if exists(self.realpath):\n            self.inode = os.stat(self.realpath).st_ino\n        else:\n            self.inode = None\n\n    @property\n    def repo(self):\n        # with DryRunProtocol path not exist\n        if exists(self.realpath):\n            inode = os.stat(self.realpath).st_ino\n        else:\n            inode = None\n        if self.inode != inode:\n            # reset background processes invoked by GitPython:\n            self._repo.git.clear_cache()\n            self.inode = inode\n\n        if self._repo is None:\n            # Note, that this may raise GitCommandError, NoSuchPathError,\n            # InvalidGitRepositoryError:\n            self._repo = self.cmd_call_wrapper(Repo, self.path)\n            lgr.log(8, \"Using existing Git repository at %s\", self.path)\n\n        # inject git options into GitPython's git call wrapper:\n        # Note: `None` currently can happen, when Runner's protocol prevents\n        # call of Repo(path) above from being actually executed (DryRunProtocol)\n        if self._repo is not None:\n            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS\n\n        return self._repo\n\n    @classmethod\n    def clone(cls, url, path, *args, **kwargs):\n        \"\"\"Clone url into path\n\n        Provides workarounds for known issues (e.g.\n        https://github.com/datalad/datalad/issues/785)\n\n        Parameters\n        ----------\n        url : str\n        path : str\n        expect_fail : bool\n          Either expect that command might fail, so error should be logged then\n          at DEBUG level instead of ERROR\n        \"\"\"\n\n        if 'repo' in kwargs:\n            raise TypeError(\"argument 'repo' conflicts with cloning\")\n            # TODO: what about 'create'?\n\n        expect_fail = kwargs.pop('expect_fail', False)\n        # fail early on non-empty target:\n        from os import listdir\n        if exists(path) and listdir(path):\n            # simulate actual GitCommandError:\n            lgr.warning(\"destination path '%s' already exists and is not an \"\n                        \"empty directory.\" % path)\n            raise GitCommandError(\n                ['git', 'clone', '-v', url, path],\n                128,\n                \"fatal: destination path '%s' already exists and is not an \"\n                \"empty directory.\" % path)\n        else:\n            # protect against cloning into existing and obviously dangling\n            # instance for that location\n            try:\n                del cls._unique_instances[path]\n            except KeyError:\n                # didn't exist - all fine\n                pass\n\n        # try to get a local path from `url`:\n        try:\n            if not isinstance(url, RI):\n                url = RI(url).localpath\n            else:\n                url = url.localpath\n        except ValueError:\n            pass\n\n        if is_ssh(url):\n            ssh_manager.get_connection(url).open()\n            # TODO: with git <= 2.3 keep old mechanism:\n            #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n            env = GitRepo.GIT_SSH_ENV\n        else:\n            env = None\n        ntries = 5  # 3 is not enough for robust workaround\n        for trial in range(ntries):\n            try:\n                lgr.debug(\"Git clone from {0} to {1}\".format(url, path))\n                repo = gitpy.Repo.clone_from(url, path, env=env,\n                                             odbt=default_git_odbt)\n                # Note/TODO: signature for clone from:\n                # (url, to_path, progress=None, env=None, **kwargs)\n\n                lgr.debug(\"Git clone completed\")\n                break\n            except GitCommandError as e:\n                # log here but let caller decide what to do\n                e_str = exc_str(e)\n                # see https://github.com/datalad/datalad/issues/785\n                if re.search(\"Request for .*aborted.*Unable to find\", str(e),\n                             re.DOTALL) \\\n                        and trial < ntries - 1:\n                    lgr.info(\n                        \"Hit a known issue with Git (see GH#785). Trial #%d, \"\n                        \"retrying\",\n                        trial)\n                    continue\n                    (lgr.debug if expect_fail else lgr.error)(e_str)\n                raise\n            except ValueError as e:\n                if gitpy.__version__ == '1.0.2' \\\n                        and \"I/O operation on closed file\" in str(e):\n                    # bug https://github.com/gitpython-developers/GitPython\n                    # /issues/383\n                    raise GitCommandError(\n                        \"clone has failed, telling ya\",\n                        999,  # good number\n                        stdout=\"%s already exists\" if exists(path) else \"\")\n                raise  # reraise original\n\n        gr = cls(path, *args, repo=repo, **kwargs)\n        return gr\n\n    def __del__(self):\n        # unbind possibly bound ConfigManager, to prevent all kinds of weird\n        # stalls etc\n        self._cfg = None\n        # Make sure to flush pending changes, especially close batch processes\n        # (internal `git cat-file --batch` by GitPython)\n        try:\n            if hasattr(self, 'repo') and exists(self.path) \\\n                    and self.repo is not None:\n                # gc might be late, so the (temporary)\n                # repo doesn't exist on FS anymore\n\n                self.repo.git.clear_cache()\n                if exists(opj(self.path, '.git')):  # don't try to write otherwise\n                    self.repo.index.write()\n        except InvalidGitRepositoryError:\n            # might have being removed and no longer valid\n            pass\n\n    def __repr__(self):\n        return \"<GitRepo path=%s (%s)>\" % (self.path, type(self))\n\n    def __eq__(self, obj):\n        \"\"\"Decides whether or not two instances of this class are equal.\n\n        This is done by comparing the base repository path.\n        \"\"\"\n        return self.realpath == obj.realpath\n\n    @classmethod\n    def is_valid_repo(cls, path):\n        \"\"\"Returns if a given path points to a git repository\"\"\"\n        return exists(opj(path, '.git'))\n\n    @property\n    def config(self):\n        \"\"\"Get an instance of the parser for the persistent repository\n        configuration.\n\n        Note: This allows to also read/write .datalad/config,\n        not just .git/config\n\n        Returns\n        -------\n        ConfigManager\n        \"\"\"\n        if self._cfg is None:\n            # associate with this dataset and read the entire config hierarchy\n            self._cfg = ConfigManager(dataset=self, dataset_only=False)\n        return self._cfg\n\n    def is_with_annex(self, only_remote=False):\n        \"\"\"Return True if GitRepo (assumed) at the path has remotes with git-annex branch\n\n        Parameters\n        ----------\n        only_remote: bool, optional\n            Check only remote (no local branches) for having git-annex branch\n        \"\"\"\n        return any((b.endswith('/git-annex') or\n                    'annex/direct' in b\n                    for b in self.get_remote_branches())) or \\\n            ((not only_remote) and\n             any((b == 'git-annex' or 'annex/direct' in b\n                  for b in self.get_branches())))\n\n    @classmethod\n    def get_toppath(cls, path, follow_up=True, git_options=None):\n        \"\"\"Return top-level of a repository given the path.\n\n        Parameters\n        -----------\n        follow_up : bool\n          If path has symlinks -- they get resolved by git.  If follow_up is\n          True, we will follow original path up until we hit the same resolved\n          path.  If no such path found, resolved one would be returned.\n        git_options: list of str\n          options to be passed to the git rev-parse call\n\n        Return None if no parent directory contains a git repository.\n        \"\"\"\n        cmd = ['git']\n        if git_options:\n            cmd.extend(git_options)\n        cmd += [\"rev-parse\", \"--show-toplevel\"]\n        try:\n            toppath, err = GitRunner().run(\n                cmd,\n                cwd=path,\n                log_stdout=True, log_stderr=True,\n                expect_fail=True, expect_stderr=True)\n            toppath = toppath.rstrip('\\n\\r')\n        except CommandError:\n            return None\n        except OSError:\n            toppath = GitRepo.get_toppath(dirname(path), follow_up=follow_up,\n                                          git_options=git_options)\n\n        if follow_up:\n            path_ = path\n            path_prev = \"\"\n            while path_ and path_ != path_prev:  # on top /.. = /\n                if realpath(path_) == toppath:\n                    toppath = path_\n                    break\n                path_prev = path_\n                path_ = dirname(path_)\n\n        return toppath\n\n    # classmethod so behavior could be tuned in derived classes\n    @classmethod\n    def _get_added_files_commit_msg(cls, files):\n        if not files:\n            return \"No files were added\"\n        msg = \"Added %d file\" % len(files)\n        if len(files) > 1:\n            msg += \"s\"\n        return msg + '\\n\\nFiles:\\n' + '\\n'.join(files)\n\n    @normalize_paths\n    def add(self, files, commit=False, msg=None, git=True, git_options=None,\n            _datalad_msg=False, update=False):\n        \"\"\"Adds file(s) to the repository.\n\n        Parameters\n        ----------\n        files: list\n          list of paths to add\n        commit: bool\n          whether or not to directly commit\n        msg: str\n          commit message in case `commit=True`. A default message, containing\n          the list of files that were added, is created by default.\n        git: bool\n          somewhat ugly construction to be compatible with AnnexRepo.add();\n          has to be always true.\n        update: bool\n          --update option for git-add. From git's manpage:\n           Update the index just where it already has an entry matching\n           <pathspec>. This removes as well as modifies index entries to match\n           the working tree, but adds no new files.\n\n           If no <pathspec> is given when --update option is used, all tracked\n           files in the entire working tree are updated (old versions of Git\n           used to limit the update to the current directory and its\n           subdirectories).\n        \"\"\"\n\n        # TODO: git_options is used as options for the git-add here,\n        # instead of options to the git executable => rename for consistency\n\n        # needs to be True - see docstring:\n        assert(git)\n\n        files = _remove_empty_items(files)\n        out = []\n\n        if files or git_options or update:\n            try:\n                # without --verbose git 2.9.3  add does not return anything\n                add_out = self._git_custom_command(\n                    files,\n                    ['git', 'add'] + assure_list(git_options) +\n                    to_options(update=update) + ['--verbose']\n                )\n                # get all the entries\n                out = self._process_git_get_output(*add_out)\n                # Note: as opposed to git cmdline, force is True by default in\n                #       gitpython, which would lead to add things, that are\n                #       ignored or excluded otherwise\n                # 2. Note: There is an issue with globbing (like adding '.'),\n                #       which apparently doesn't care for 'force' and therefore\n                #       adds '.git/...'. May be it's expanded at the wrong\n                #       point in time or sth. like that.\n                # For now, use direct call to git add.\n                #self.cmd_call_wrapper(self.repo.index.add, files, write=True,\n                #                      force=False)\n                # TODO: May be make use of 'fprogress'-option to indicate\n                # progress\n                # But then, we don't have it for git-annex add, anyway.\n                #\n                # TODO: Is write=True a reasonable way to do it?\n                # May be should not write until success of operation is\n                # confirmed?\n                # What's best in case of a list of files?\n            except OSError as e:\n                lgr.error(\"add: %s\" % e)\n                raise\n\n        else:\n            lgr.warning(\"add was called with empty file list and no options.\")\n\n        if commit:\n            if msg is None:\n                msg = self._get_added_files_commit_msg(files)\n            self.commit(msg=msg, _datalad_msg=_datalad_msg)\n\n        # Make sure return value from GitRepo is consistent with AnnexRepo\n        # currently simulating similar return value, assuming success\n        # for all files:\n        # TODO: Make return values consistent across both *Repo classes!\n        return out\n\n    @staticmethod\n    def _process_git_get_output(stdout, stderr=None):\n        \"\"\"Given both outputs (stderr is ignored atm) of git add - process it\n\n        Primarily to centralize handling in both indirect annex and direct\n        modes when ran through proxy\n        \"\"\"\n        return [{u'file': f, u'success': True}\n                for f in re.findall(\"'(.*)'[\\n$]\", stdout)]\n\n    @normalize_paths(match_return_type=False)\n    def remove(self, files, recursive=False, **kwargs):\n        \"\"\"Remove files.\n\n        Calls git-rm.\n\n        Parameters\n        ----------\n        files: str\n          list of paths to remove\n        recursive: False\n          either to allow recursive removal from subdirectories\n        kwargs:\n          see `__init__`\n\n        Returns\n        -------\n        [str]\n          list of successfully removed files.\n        \"\"\"\n\n        files = _remove_empty_items(files)\n\n        if recursive:\n            kwargs['r'] = True\n        stdout, stderr = self._git_custom_command(\n            files, ['git', 'rm'] + to_options(**kwargs))\n\n        # output per removed file is expected to be \"rm 'PATH'\":\n        return [line.strip()[4:-1] for line in stdout.splitlines()]\n\n        #return self.repo.git.rm(files, cached=False, **kwargs)\n\n    def precommit(self):\n        \"\"\"Perform pre-commit maintenance tasks\n        \"\"\"\n        # All GitPython commands should take care about flushing index\n        # whenever they modify it, so we would not care to do anything\n        # if self.repo is not None and exists(opj(self.path, '.git')):  # don't try to write otherwise:\n        #     # flush possibly cached in GitPython changes to index:\n        #     # if self.repo.git:\n        #     #     sys.stderr.write(\"CLEARING\\n\")\n        #     #     self.repo.git.clear_cache()\n        #     self.repo.index.write()\n        pass\n\n    @staticmethod\n    def _get_prefixed_commit_msg(msg):\n        DATALAD_PREFIX = \"[DATALAD]\"\n        return DATALAD_PREFIX if not msg else \"%s %s\" % (DATALAD_PREFIX, msg)\n\n    def commit(self, msg=None, options=None, _datalad_msg=False, careless=True,\n               files=None):\n        \"\"\"Commit changes to git.\n\n        Parameters\n        ----------\n        msg: str, optional\n          commit-message\n        options: list of str, optional\n          cmdline options for git-commit\n        _datalad_msg: bool, optional\n          To signal that commit is automated commit by datalad, so\n          it would carry the [DATALAD] prefix\n        careless: bool, optional\n          if False, raise when there's nothing actually committed;\n          if True, don't care\n        files: list of str, optional\n          path(s) to commit\n        \"\"\"\n\n        self.precommit()\n\n        if _datalad_msg:\n            msg = self._get_prefixed_commit_msg(msg)\n\n        if not msg:\n            if options:\n                if \"--allow-empty-message\" not in options:\n                        options.append(\"--allow-empty-message\")\n            else:\n                options = [\"--allow-empty-message\"]\n\n        # Note: We used to use a direct call to git only if there were options,\n        # since we can't pass all possible options to gitpython's implementation\n        # of commit.\n        # But there's an additional issue. GitPython implements commit in a way,\n        # that it might create a new commit, when a direct call wouldn't. This\n        # was discovered with a modified (but unstaged) submodule, leading to a\n        # commit, that apparently did nothing - git status still showed the very\n        # same thing afterwards. But a commit was created nevertheless:\n        # diff --git a/sub b/sub\n        # --- a/sub\n        # +++ b/sub\n        # @@ -1 +1 @@\n        # -Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f\n        # +Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f-dirty\n        #\n        # Therefore, for now always use direct call.\n        # TODO: Figure out, what exactly is going on with gitpython here\n\n        cmd = ['git', 'commit'] + ([\"-m\", msg if msg else \"\"])\n        if options:\n            cmd.extend(options)\n        lgr.debug(\"Committing via direct call of git: %s\" % cmd)\n\n        try:\n            self._git_custom_command(files, cmd,\n                                     expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if 'nothing to commit' in e.stdout:\n                if careless:\n                    lgr.debug(\"nothing to commit in {}. \"\n                              \"Ignored.\".format(self))\n                else:\n                    raise\n            elif 'no changes added to commit' in e.stdout or \\\n                    'nothing added to commit' in e.stdout:\n                if careless:\n                    lgr.debug(\"no changes added to commit in {}. \"\n                              \"Ignored.\".format(self))\n                else:\n                    raise\n            elif \"did not match any file(s) known to git.\" in e.stderr:\n                # TODO: Improve FileNotInXXXXError classes to better deal with\n                # multiple files; Also consider PathOutsideRepositoryError\n                raise FileNotInRepositoryError(cmd=e.cmd,\n                                               msg=\"File(s) unknown to git\",\n                                               code=e.code,\n                                               filename=linesep.join(\n                                            [l for l in e.stderr.splitlines()\n                                             if l.startswith(\"pathspec\")]))\n            else:\n                raise\n\n    def get_indexed_files(self):\n        \"\"\"Get a list of files in git's index\n\n        Returns\n        -------\n        list\n            list of paths rooting in git's base dir\n        \"\"\"\n\n        return [x[0] for x in self.cmd_call_wrapper(\n            self.repo.index.entries.keys)]\n\n    def get_hexsha(self, object=None):\n        \"\"\"Return a hexsha for a given object. If None - of current HEAD\n\n        Parameters\n        ----------\n        object: str, optional\n          Any type of Git object identifier. See `git show`.\n        \"\"\"\n        cmd = ['git', 'show', '--no-patch', \"--format=%H\"]\n        if object:\n            cmd.append(object)\n        # make sure Git takes our argument as a revision\n        cmd.append('--')\n        try:\n            stdout, stderr = self._git_custom_command(\n                '', cmd, expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if 'bad revision' in e.stderr:\n                raise ValueError(\"Unknown object identifier: %s\" % object)\n            elif 'does not have any commits yet' in e.stderr:\n                return None\n            else:\n                raise e\n        stdout = stdout.splitlines()\n        assert(len(stdout) == 1)\n        return stdout[0]\n\n    def get_merge_base(self, treeishes):\n        \"\"\"Get a merge base hexsha\n\n        Parameters\n        ----------\n        treeishes: str or list of str\n          List of treeishes (branches, hexshas, etc) to determine the merge\n          base of. If a single value provided, returns merge_base with the\n          current branch.\n\n        Returns\n        -------\n        str or None\n          If no merge-base for given commits, or specified treeish doesn't\n          exist, None returned\n        \"\"\"\n        if isinstance(treeishes, string_types):\n            treeishes = [treeishes]\n        if not treeishes:\n            raise ValueError(\"Provide at least a single value\")\n        elif len(treeishes) == 1:\n            treeishes = treeishes + [self.get_active_branch()]\n\n        try:\n            bases = self.repo.merge_base(*treeishes)\n        except GitCommandError as exc:\n            if \"fatal: Not a valid object name\" in str(exc):\n                return None\n            raise\n\n        if not bases:\n            return None\n        assert(len(bases) == 1)  # we do not do 'all' yet\n        return bases[0].hexsha\n\n    def get_committed_date(self, branch=None):\n        \"\"\"Get the date stamp of the last commit (in a branch). None if no commit\"\"\"\n        try:\n            commit = next(\n                self.get_branch_commits(branch\n                                        or self.get_active_branch())\n            )\n        except Exception as exc:\n            lgr.debug(\"Got exception while trying to get last commit: %s\",\n                      exc_str(exc))\n            return None\n        return commit.committed_date\n\n    def get_active_branch(self):\n        try:\n            branch = self.repo.active_branch.name\n        except TypeError as e:\n            if \"HEAD is a detached symbolic reference\" in str(e):\n                lgr.debug(\"detached HEAD in {0}\".format(self))\n                return None\n            else:\n                raise\n        return branch\n\n    def get_branches(self):\n        \"\"\"Get all branches of the repo.\n\n        Returns\n        -------\n        [str]\n            Names of all branches of this repository.\n        \"\"\"\n\n        return [branch.name for branch in self.repo.branches]\n\n    def get_remote_branches(self):\n        \"\"\"Get all branches of all remotes of the repo.\n\n        Returns\n        -----------\n        [str]\n            Names of all remote branches.\n        \"\"\"\n        # TODO: Reconsider melting with get_branches()\n\n        # TODO: treat entries like this: origin/HEAD -> origin/master'\n        # currently this is done in collection\n\n        # For some reason, this is three times faster than the version below:\n        remote_branches = list()\n        for remote in self.repo.remotes:\n            try:\n                for ref in remote.refs:\n                    remote_branches.append(ref.name)\n            except AssertionError as e:\n                if str(e).endswith(\"did not have any references\"):\n                    # this will happen with git annex special remotes\n                    pass\n                else:\n                    raise e\n        return remote_branches\n        # return [branch.strip() for branch in\n        #         self.repo.git.branch(r=True).splitlines()]\n\n    def get_remotes(self, with_urls_only=False):\n        \"\"\"Get known remotes of the repository\n\n        Parameters\n        ----------\n        with_urls_only : bool, optional\n          return only remotes which have urls\n\n        Returns\n        -------\n        remotes : list of str\n          List of names of the remotes\n        \"\"\"\n\n        # Note: read directly from config and spare instantiation of gitpy.Repo\n        # since we need this in AnnexRepo constructor. Furthermore gitpy does it\n        # pretty much the same way and the use of a Repo instance seems to have\n        # no reason other than a nice object oriented look.\n        from datalad.utils import unique\n\n        self.config.reload()\n        remotes = unique([x[7:] for x in self.config.sections()\n                          if x.startswith(\"remote.\")])\n\n        if with_urls_only:\n            remotes = [\n                r for r in remotes\n                if self.config.get('remote.%s.url' % r)\n            ]\n        return remotes\n\n    def get_files(self, branch=None):\n        \"\"\"Get a list of files in git.\n\n        Lists the files in the (remote) branch.\n\n        Parameters\n        ----------\n        branch: str\n          Name of the branch to query. Default: active branch.\n\n        Returns\n        -------\n        [str]\n          list of files.\n        \"\"\"\n        # TODO: RF codes base and melt get_indexed_files() in\n\n        if branch is None:\n            # active branch can be queried way faster:\n            return self.get_indexed_files()\n        else:\n            return [item.path for item in self.repo.tree(branch).traverse()\n                    if isinstance(item, Blob)]\n\n    def get_file_content(self, file_, branch='HEAD'):\n        \"\"\"\n\n        Returns\n        -------\n        [str]\n          content of file_ as a list of lines.\n        \"\"\"\n        content_str = self.repo.commit(branch).tree[file_].data_stream.read()\n\n        # in python3 a byte string is returned. Need to convert it:\n        from six import PY3\n        if PY3:\n            conv_str = u''\n            for b in bytes(content_str):\n                conv_str += chr(b)\n            return conv_str.splitlines()\n        else:\n            return content_str.splitlines()\n        # TODO: keep splitlines?\n\n    def _get_files_history(self, files, branch='HEAD'):\n        \"\"\"\n\n        Parameters\n        ----------\n        files: list\n          list of files, only commits with queried files are considered\n        branch: str\n          Name of the branch to query. Default: HEAD.\n\n        Returns\n        -------\n        [iterator]\n        yielding Commit items generator from branch history associated with files\n        \"\"\"\n        return gitpy.objects.commit.Commit.iter_items(self.repo, branch, paths=files)\n\n    def _get_remotes_having_commit(self, commit_hexsha, with_urls_only=True):\n        \"\"\"Traverse all branches of the remote and check if commit in any of their ancestry\n\n        It is a generator yielding names of the remotes\n        \"\"\"\n        out, err = self._git_custom_command(\n            '', 'git branch -r --contains ' + commit_hexsha\n        )\n        # sanitize a bit (all the spaces and new lines)\n        remote_branches = [\n            b  # could be origin/HEAD -> origin/master, we just skip ->\n            for b in filter(bool, out.split())\n            if b != '->'\n        ]\n        return [\n            remote\n            for remote in self.get_remotes(with_urls_only=with_urls_only)\n            if any(rb.startswith(remote + '/') for rb in remote_branches)\n        ]\n\n    def _gitpy_custom_call(self, cmd, cmd_args=None, cmd_options=None,\n                           git_options=None, env=None,\n\n                           # 'old' options for Runner; not sure yet, which of\n                           # them are actually still needed:\n                           log_stdout=True, log_stderr=True, log_online=False,\n                           expect_stderr=True, cwd=None,\n                           shell=None, expect_fail=False):\n\n        \"\"\"Helper to call GitPython's wrapper for git calls.\n\n        The used instance of `gitpy.Git` is bound to the repository,\n        which determines its working directory.\n        This is used for adhoc implementation of a git command and to\n        demonstrate how to use it in more specific implementations.\n\n        Note\n        ----\n        Aims to replace the use of datalad's `Runner` class for direct git\n        calls. (Currently the `_git_custom_command()` method).\n        Therefore mimicking its behaviour during RF'ing.\n\n        Parameters\n        ----------\n        cmd: str\n          the native git command to call\n        cmd_args: list of str\n          arguments to the git command\n        cmd_options: dict\n          options for the command as key, value pair\n          (this transformation, needs some central place to document)\n        git_options: dict\n          options for the git executable as key, value pair\n          (see above)\n        env: dict\n          environment vaiables to temporarily set for this call\n\n        TODO\n        ----\n        Example\n\n        Returns\n        -------\n        (stdout, stderr)\n        \"\"\"\n\n        # TODO: Reconsider when to log/stream what (stdout, stderr) and/or\n        # fully implement the behaviour of `Runner`\n\n        if log_online:\n            raise NotImplementedError(\"option 'log_online' not implemented yet\")\n        with_exceptions = not expect_fail\n        if cwd:\n            # the gitpy.cmd.Git instance, bound to this repository doesn't allow\n            # to explicitly set the working dir, except for using os.getcwd\n            raise NotImplementedError(\"working dir is a read-only property\")\n\n        _tmp_shell = gitpy.cmd.Git.USE_SHELL\n        gitpy.cmd.Git.USE_SHELL = shell\n\n        if env is None:\n            env = {}\n        if git_options is None:\n            git_options = {}\n        if cmd_options is None:\n            cmd_options = {}\n        cmd_options.update({'with_exceptions': with_exceptions,\n                            'with_extended_output': True})\n\n        # TODO: _GIT_COMMON_OPTIONS!\n\n        with self.repo.git.custom_environment(**env):\n            try:\n                status, std_out, std_err = \\\n                    self.repo.git(**git_options).__getattr__(cmd)(\n                        cmd_args, **cmd_options)\n            except GitCommandError as e:\n                # For now just reraise. May be raise CommandError instead\n                raise\n            finally:\n                gitpy.cmd.Git.USE_SHELL = _tmp_shell\n\n        if not expect_stderr and std_err:\n            lgr.error(\"Unexpected output on stderr: %s\" % std_err)\n            raise CommandError\n        if log_stdout:\n            for line in std_out.splitlines():\n                lgr.debug(\"stdout| \" + line)\n        if log_stderr:\n            for line in std_err.splitlines():\n                lgr.log(level=logging.DEBUG if expect_stderr else logging.ERROR,\n                        msg=\"stderr| \" + line)\n\n        return std_out, std_err\n\n    @normalize_paths(match_return_type=False)\n    def _git_custom_command(self, files, cmd_str,\n                            log_stdout=True, log_stderr=True, log_online=False,\n                            expect_stderr=True, cwd=None, env=None,\n                            shell=None, expect_fail=False):\n        \"\"\"Allows for calling arbitrary commands.\n\n        Helper for developing purposes, i.e. to quickly implement git commands\n        for proof of concept without the need to figure out, how this is done\n        via GitPython.\n\n        Parameters\n        ----------\n        files: list of files\n        cmd_str: str or list\n            arbitrary command str. `files` is appended to that string.\n\n        Returns\n        -------\n        stdout, stderr\n        \"\"\"\n\n        # ensure cmd_str becomes a well-formed list:\n        if isinstance(cmd_str, string_types):\n            if files and not cmd_str.strip().endswith(\" --\"):\n                cmd_str += \" --\"\n            cmd_str = shlex.split(cmd_str, posix=not on_windows)\n        else:\n            if files and cmd_str[-1] != '--':\n                cmd_str.append('--')\n\n        cmd = cmd_str + files\n\n        assert(cmd[0] == 'git')\n        cmd = cmd[:1] + self._GIT_COMMON_OPTIONS + cmd[1:]\n\n        from .exceptions import GitIgnoreError\n\n        try:\n            out, err = self.cmd_call_wrapper.run(\n                cmd,\n                log_stderr=log_stderr,\n                log_stdout=log_stdout,\n                log_online=log_online,\n                expect_stderr=expect_stderr,\n                cwd=cwd,\n                env=env,\n                shell=shell,\n                expect_fail=expect_fail)\n        except CommandError as e:\n            ignored = re.search(GitIgnoreError.pattern, e.stderr)\n            if ignored:\n                raise GitIgnoreError(cmd=e.cmd, msg=e.stderr,\n                                     code=e.code, stdout=e.stdout,\n                                     stderr=e.stderr,\n                                     paths=ignored.groups()[0].splitlines())\n            raise\n        return out, err\n\n# TODO: --------------------------------------------------------------------\n\n    def add_remote(self, name, url, options=None):\n        \"\"\"Register remote pointing to a url\n        \"\"\"\n        cmd = ['git', 'remote', 'add']\n        if options:\n            cmd += options\n        cmd += [name, url]\n\n        result = self._git_custom_command('', cmd)\n        self.config.reload()\n        return result\n\n    def remove_remote(self, name):\n        \"\"\"Remove existing remote\n        \"\"\"\n\n        # TODO: testing and error handling!\n        from .exceptions import RemoteNotAvailableError\n        try:\n            out, err = self._git_custom_command(\n                '', ['git', 'remote', 'remove', name])\n        except CommandError as e:\n            if 'fatal: No such remote' in e.stderr:\n                raise RemoteNotAvailableError(name,\n                                              cmd=\"git remote remove\",\n                                              msg=\"No such remote\",\n                                              stdout=out,\n                                              stderr=err)\n            else:\n                raise e\n\n        # TODO: config.reload necessary?\n        self.config.reload()\n        return\n\n    def update_remote(self, name=None, verbose=False):\n        \"\"\"\n        \"\"\"\n        options = [\"-v\"] if verbose else []\n        name = [name] if name else []\n        self._git_custom_command(\n            '', ['git', 'remote'] + name + ['update'] + options,\n            expect_stderr=True\n        )\n\n    # TODO: centralize all the c&p code in fetch, pull, push\n    # TODO: document **kwargs passed to gitpython\n    def fetch(self, remote=None, refspec=None, progress=None, all_=False,\n              **kwargs):\n        \"\"\"Fetches changes from a remote (or all_ remotes).\n\n        Parameters\n        ----------\n        remote: str\n          (optional) name of the remote to fetch from. If no remote is given and\n          `all_` is not set, the tracking branch is fetched.\n        refspec: str\n          (optional) refspec to fetch.\n        progress:\n          passed to gitpython. TODO: Figure it out, make consistent use of it\n          and document it.\n        all_: bool\n          fetch all_ remotes (and all_ of their branches).\n          Fails if `remote` was given.\n        kwargs:\n          passed to gitpython. TODO: Figure it out, make consistent use of it\n          and document it.\n\n        Returns\n        -------\n        list\n            FetchInfo objects of the items fetched from remote\n        \"\"\"\n        # TODO: options=> **kwargs):\n        # Note: Apparently there is no explicit (fetch --all) in gitpython,\n        #       but fetch is always bound to a certain remote instead.\n        #       Therefore implement it on our own:\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            if all_:\n                remotes_to_fetch = [\n                    self.repo.remote(r)\n                    for r in self.get_remotes(with_urls_only=True)\n                ]\n            else:\n                # No explicit remote to fetch.\n                # => get tracking branch:\n                tb_remote, refspec = self.get_tracking_branch()\n                if tb_remote is not None:\n                    remotes_to_fetch = [self.repo.remote(tb_remote)]\n                else:\n                    # No remote, no tracking branch\n                    # => fail\n                    raise ValueError(\"Neither a remote is specified to fetch \"\n                                     \"from nor a tracking branch is set up.\")\n        else:\n            remotes_to_fetch = [self.repo.remote(remote)]\n\n        fi_list = []\n        for rm in remotes_to_fetch:\n            fetch_url = \\\n                self.config.get('remote.%s.fetchurl' % rm.name,\n                                self.config.get('remote.%s.url' % rm.name,\n                                                None))\n            if fetch_url is None:\n                lgr.debug(\"Remote %s has no URL\", rm)\n                return []\n\n            if is_ssh(fetch_url):\n                ssh_manager.get_connection(fetch_url).open()\n                # TODO: with git <= 2.3 keep old mechanism:\n                #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                    fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)\n                    # TODO: progress +kwargs\n            else:\n                fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n\n        # TODO: fetch returns a list of FetchInfo instances. Make use of it.\n        return fi_list\n\n    def pull(self, remote=None, refspec=None, progress=None, **kwargs):\n        \"\"\"See fetch\n        \"\"\"\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            # No explicit remote to pull from.\n            # => get tracking branch:\n            tb_remote, refspec = self.get_tracking_branch()\n            if tb_remote is not None:\n                remote = self.repo.remote(tb_remote)\n            else:\n                # No remote, no tracking branch\n                # => fail\n                raise ValueError(\"No remote specified to pull from nor a \"\n                                 \"tracking branch is set up.\")\n\n        else:\n            remote = self.repo.remote(remote)\n\n        fetch_url = \\\n            remote.config_reader.get(\n                'fetchurl' if remote.config_reader.has_option('fetchurl')\n                else 'url')\n        if is_ssh(fetch_url):\n            ssh_manager.get_connection(fetch_url).open()\n            # TODO: with git <= 2.3 keep old mechanism:\n            #       with remote.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n            with remote.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                return remote.pull(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n        else:\n            return remote.pull(refspec=refspec, progress=progress, **kwargs)\n            # TODO: progress +kwargs\n\n    def push(self, remote=None, refspec=None, progress=None, all_remotes=False,\n             **kwargs):\n        \"\"\"Push to remote repository\n\n        Parameters\n        ----------\n        remote: str\n          name of the remote to push to\n        refspec: str\n          specify what to push\n        progress:\n          TODO\n        all_remotes: bool\n          if set to True push to all remotes. Conflicts with `remote` not being\n          None.\n        kwargs: dict\n          options to pass to `git push`\n\n        Returns\n        -------\n        list\n            PushInfo objects of the items pushed to remote\n        \"\"\"\n\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            if all_remotes:\n                remotes_to_push = self.repo.remotes\n            else:\n                # Nothing explicitly specified. Just call `git push` and let git\n                # decide what to do would be an option. But:\n                # - without knowing the remote and its URL we cannot provide\n                #   shared SSH connection\n                # - we lose ability to use GitPython's progress info and return\n                #   values\n                #   (the latter would be solvable:\n                #    Provide a Repo.push() method for GitPython, copying\n                #    Remote.push() for similar return value and progress\n                #    (also: fetch, pull)\n\n                # Do what git would do:\n                # 1. branch.*.remote for current branch or 'origin' as default\n                #    if config is missing\n                # 2. remote.*.push or push.default\n\n                # TODO: check out \"same procedure\" for fetch/pull\n\n                tb_remote, refspec = self.get_tracking_branch()\n                if tb_remote is None:\n                    tb_remote = 'origin'\n                remotes_to_push = [self.repo.remote(tb_remote)]\n                # use no refspec; let git find remote.*.push or push.default on\n                # its own\n\n        else:\n            if all_remotes:\n                lgr.warning(\"Option 'all_remotes' conflicts with specified \"\n                            \"remote '%s'. Option ignored.\")\n            remotes_to_push = [self.repo.remote(remote)]\n\n        pi_list = []\n        for rm in remotes_to_push:\n            push_url = \\\n                rm.config_reader.get('pushurl'\n                                     if rm.config_reader.has_option('pushurl')\n                                     else 'url')\n            if is_ssh(push_url):\n                ssh_manager.get_connection(push_url).open()\n                # TODO: with git <= 2.3 keep old mechanism:\n                #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                    pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)\n                    # TODO: progress +kwargs\n            else:\n                pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n        return pi_list\n\n    def get_remote_url(self, name, push=False):\n        \"\"\"Get the url of a remote.\n\n        Reads the configuration of remote `name` and returns its url or None,\n        if there is no url configured.\n\n        Parameters\n        ----------\n        name: str\n          name of the remote\n        push: bool\n          if True, get the pushurl instead of the fetch url.\n        \"\"\"\n\n        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')\n        return self.config.get(var, None)\n\n    def set_remote_url(self, name, url, push=False):\n        \"\"\"Set the URL a remote is pointing to\n\n        Sets the URL of the remote `name`. Requires the remote to already exist.\n\n        Parameters\n        ----------\n        name: str\n          name of the remote\n        url: str\n        push: bool\n          if True, set the push URL, otherwise the fetch URL\n        \"\"\"\n\n        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')\n        self.config.set(var, url, where='local', reload=True)\n\n    def get_branch_commits(self, branch=None, limit=None, stop=None, value=None):\n        \"\"\"Return GitPython's commits for the branch\n\n        Pretty much similar to what 'git log <branch>' does.\n        It is a generator which returns top commits first\n\n        Parameters\n        ----------\n        branch: str, optional\n          If not provided, assumes current branch\n        limit: None | 'left-only', optional\n          Limit which commits to report.  If None -- all commits (merged or not),\n          if 'left-only' -- only the commits from the left side of the tree upon\n          merges\n        stop: str, optional\n          hexsha of the commit at which stop reporting (matched one is not\n          reported either)\n        value: None | 'hexsha', optional\n          What to yield.  If None - entire commit object is yielded, if 'hexsha'\n          only its hexsha\n        \"\"\"\n\n        if not branch:\n            branch = self.get_active_branch()\n\n        try:\n            _branch = self.repo.branches[branch]\n        except IndexError:\n            raise MissingBranchError(self, branch,\n                                     [b.name for b in self.repo.branches])\n\n        fvalue = {None: lambda x: x, 'hexsha': lambda x: x.hexsha}[value]\n\n        if not limit:\n            def gen():\n                # traverse doesn't yield original commit\n                co = _branch.commit\n                yield co\n                for co_ in co.traverse():\n                    yield co_\n        elif limit == 'left-only':\n            # we need a custom implementation since couldn't figure out how to\n            # do with .traversal\n            def gen():\n                co = _branch.commit\n                while co:\n                    yield co\n                    co = co.parents[0] if co.parents else None\n        else:\n            raise ValueError(limit)\n\n        for c in gen():\n            if stop and c.hexsha == stop:\n                return\n            yield fvalue(c)\n\n    def checkout(self, name, options=None):\n        \"\"\"\n        \"\"\"\n        # TODO: May be check for the need of -b options herein?\n        cmd = ['git', 'checkout']\n        if options:\n            cmd += options\n        cmd += [str(name)]\n\n        self._git_custom_command('', cmd, expect_stderr=True)\n\n    # TODO: Before implementing annex merge, find usages and check for a needed\n    # change to call super().merge\n    def merge(self, name, options=None, msg=None, allow_unrelated=False, **kwargs):\n        if options is None:\n            options = []\n        if msg:\n            options = options + [\"-m\", msg]\n        if allow_unrelated and external_versions['cmd:git'] >= '2.9':\n            options += ['--allow-unrelated-histories']\n        self._git_custom_command(\n            '', ['git', 'merge'] + options + [name],\n            **kwargs\n        )\n\n    def remove_branch(self, branch):\n        self._git_custom_command(\n            '', ['git', 'branch', '-D', branch]\n        )\n\n    def ls_remote(self, remote, options=None):\n        if options is None:\n            options = []\n        self._git_custom_command(\n            '', ['git', 'ls-remote'] + options + [remote]\n        )\n        # TODO: Return values?\n\n    def is_dirty(self, index=True, working_tree=True, untracked_files=True,\n                 submodules=True, path=None):\n        \"\"\"Returns true if the repo is considered to be dirty\n\n        Parameters\n        ----------\n        index: bool\n          if True, consider changes to the index\n        working_tree: bool\n          if True, consider changes to the working tree\n        untracked_files: bool\n          if True, consider untracked files\n        submodules: bool\n          if True, consider submodules\n        path: str or list of str\n          path(s) to consider only\n        Returns\n        -------\n          bool\n        \"\"\"\n\n        return self.repo.is_dirty(index=index, working_tree=working_tree,\n                                  untracked_files=untracked_files,\n                                  submodules=submodules, path=path)\n\n    @property\n    def dirty(self):\n        return self.is_dirty()\n\n    @property\n    def untracked_files(self):\n        return self.repo.untracked_files\n\n    def gc(self, allow_background=False, auto=False):\n        \"\"\"Perform house keeping (garbage collection, repacking)\"\"\"\n        cmd_options = ['git']\n        if not allow_background:\n            cmd_options += ['-c', 'gc.autodetach=0']\n        cmd_options += ['gc', '--aggressive']\n        if auto:\n            cmd_options += ['--auto']\n        self._git_custom_command('', cmd_options)\n\n    def get_submodules(self, sorted_=True):\n        \"\"\"Return a list of git.Submodule instances for all submodules\"\"\"\n        # check whether we have anything in the repo. if not go home early\n        if not self.repo.head.is_valid():\n            return []\n        submodules = self.repo.submodules\n        if sorted_:\n            submodules = sorted(submodules, key=lambda x: x.path)\n        return submodules\n\n    def is_submodule_modified(self, name, options=[]):\n        \"\"\"Whether a submodule has new commits\n\n        Note: This is an adhoc method. It parses output of\n        'git submodule summary' and currently is not able to distinguish whether\n        or not this change is staged in `self` and whether this would be\n        reported 'added' or 'modified' by 'git status'.\n        Parsing isn't heavily tested yet.\n\n        Parameters\n        ----------\n        name: str\n          the submodule's name\n        options: list\n          options to pass to 'git submodule summary'\n        Returns\n        -------\n        bool\n          True if there are commits in the submodule, differing from\n          what is registered in `self`\n        --------\n        \"\"\"\n\n        out, err = self._git_custom_command('',\n                                            ['git', 'submodule', 'summary'] + \\\n                                            options + ['--', name])\n        return any([line.split()[1] == name\n                    for line in out.splitlines()\n                    if line and len(line.split()) > 1])\n\n    def add_submodule(self, path, name=None, url=None, branch=None):\n        \"\"\"Add a new submodule to the repository.\n\n        This will alter the index as well as the .gitmodules file, but will not\n        create a new commit.  If the submodule already exists, no matter if the\n        configuration differs from the one provided, the existing submodule\n        is considered as already added and no further action is performed.\n\n        Parameters\n        ----------\n        path : str\n          repository-relative path at which the submodule should be located, and\n          which will be created as required during the repository initialization.\n        name : str or None\n          name/identifier for the submodule. If `None`, the `path` will be used\n          as name.\n        url : str or None\n          git-clone compatible URL. If `None`, the repository is assumed to\n          exist, and the url of the first remote is taken instead. This is\n          useful if you want to make an existing repository a submodule of\n          another one.\n        branch : str or None\n          name of branch to be checked out in the submodule. The given branch\n          must exist in the remote repository, and will be checked out locally\n          as a tracking branch. If `None`, remote HEAD will be checked out.\n        \"\"\"\n        if name is None:\n            name = path\n        # XXX the following should do it, but GitPython will refuse to add a submodule\n        # unless you specify a URL that is configured as one of its remotes, or you\n        # specify no URL, but the repo has at least one remote.\n        # this is stupid, as for us it is valid to not have any remote, because we can\n        # still obtain the submodule from a future publication location, based on the\n        # parent\n        # gitpy.Submodule.add(self.repo, name, path, url=url, branch=branch)\n        # going git native instead\n        cmd = ['git', 'submodule', 'add', '--name', name]\n        if branch is not None:\n            cmd += ['-b', branch]\n        if url is None:\n            if not isabs(path):\n                # need to recode into a relative path \"URL\" in POSIX\n                # style, even on windows\n                url = posixpath.join(curdir, posix_relpath(path))\n            else:\n                url = path\n        cmd += [url, path]\n        self._git_custom_command('', cmd)\n        # TODO: return value\n\n    def deinit_submodule(self, path, **kwargs):\n        \"\"\"Deinit a submodule\n\n        Parameters\n        ----------\n        path: str\n            path to the submodule; relative to `self.path`\n        kwargs:\n            see `__init__`\n        \"\"\"\n\n        self._git_custom_command(path,\n                                 ['git', 'submodule', 'deinit'] +\n                                 to_options(**kwargs))\n        # TODO: return value\n\n    def update_submodule(self, path, mode='checkout', init=False):\n        \"\"\"Update a registered submodule.\n\n        This will make the submodule match what the superproject expects by\n        cloning missing submodules and updating the working tree of the\n        submodules. The \"updating\" can be done in several ways depending\n        on the value of submodule.<name>.update configuration variable, or\n        the `mode` argument.\n\n        Parameters\n        ----------\n        path : str\n          Identifies which submodule to operate on by it's repository-relative\n          path.\n        mode : {checkout, rebase, merge}\n          Update procedure to perform. 'checkout': the commit recorded in the\n          superproject will be checked out in the submodule on a detached HEAD;\n          'rebase': the current branch of the submodule will be rebased onto\n          the commit recorded in the superproject; 'merge': the commit recorded\n          in the superproject will be merged into the current branch in the\n          submodule.\n        init : bool\n          If True, initialize all submodules for which \"git submodule init\" has\n          not been called so far before updating.\n          Primarily provided for internal purposes and should not be used directly\n          since would result in not so annex-friendly .git symlinks/references\n          instead of full featured .git/ directories in the submodules\n        \"\"\"\n        cmd = ['git', 'submodule', 'update', '--%s' % mode]\n        if init:\n            cmd.append('--init')\n            subgitpath = opj(self.path, path, '.git')\n            if not exists(subgitpath):\n                # TODO:  wouldn't with --init we get all those symlink'ed .git/?\n                # At least let's warn\n                lgr.warning(\n                    \"Do not use update_submodule with init=True to avoid git creating \"\n                    \"symlinked .git/ directories in submodules\"\n                )\n            #  yoh: I thought I saw one recently but thought it was some kind of\n            #  an artifact from running submodule update --init manually at\n            #  some point, but looking at this code now I worry that it was not\n        self._git_custom_command(path, cmd)\n        # TODO: return value\n\n    def update_ref(self, ref, value, symbolic=False):\n        \"\"\"Update the object name stored in a ref \"safely\".\n\n        Just a shim for `git update-ref` call if not symbolic, and\n        `git symbolic-ref` if symbolic\n\n        Parameters\n        ----------\n        ref : str\n          Reference, such as `ref/heads/BRANCHNAME` or HEAD.\n        value : str\n          Value to update to, e.g. hexsha of a commit when updating for a\n          branch ref, or branch ref if updating HEAD\n        symbolic : None\n          To instruct if ref is symbolic, e.g. should be used in case of\n          ref=HEAD\n        \"\"\"\n        self._git_custom_command(\n            '',\n            ['git', 'symbolic-ref' if symbolic else 'update-ref', ref, value]\n        )\n\n    def tag(self, tag):\n        \"\"\"Assign a tag to current commit\n\n        Parameters\n        ----------\n        tag : str\n          Custom tag label.\n        \"\"\"\n        # TODO later to be extended with tagging particular commits and signing\n        # TODO: call in save.py complains about extensive logging. When does it\n        # happen in what way? Figure out, whether to just silence it or raise or\n        # whatever else.\n        self._git_custom_command(\n            '', ['git', 'tag', str(tag)]\n        )\n\n    def get_tags(self, output=None):\n        \"\"\"Get list of tags\n\n        Parameters\n        ----------\n        output : str, optional\n          If given, limit the return value to a list of values matching that\n          particular key of the tag properties.\n\n        Returns\n        -------\n        list\n          Each item is a dictionary with information on a tag. At present\n          this includes 'hexsha', and 'name', where the latter is the string\n          label of the tag, and the format the hexsha of the object the tag\n          is attched to. The list is sorted by commit date, with the most\n          recent commit being the last element.\n        \"\"\"\n        # TODO it would be straightforward to add more info and tweak the\n        # sorting\n        stdout, stderr = self._git_custom_command(\n            '',\n            ['git', 'tag', '--format=%(refname:strip=2)%00%(object)',\n             '--sort=*committerdate'])\n        fields = ('name', 'hexsha')\n        tags = [dict(zip(fields, line.split('\\0'))) for line in stdout.splitlines()]\n        if output:\n            return [t[output] for t in tags]\n        else:\n            return tags\n\n    def describe(self, **kwargs):\n        \"\"\" Quick and dirty implementation to call git-describe\n\n        Parameters:\n        -----------\n        kwargs:\n            transformed to cmdline options for git-describe;\n            see __init__ for description of the transformation\n        \"\"\"\n        # TODO: be more precise what failure to expect when and raise actual\n        # errors\n        try:\n            describe, outerr = self._git_custom_command(\n                [],\n                ['git', 'describe'] + to_options(**kwargs),\n                expect_fail=True)\n            return describe.strip()\n        # TODO: WTF \"catch everything\"?\n        except:\n            return None\n\n    def get_tracking_branch(self, branch=None):\n        \"\"\"Get the tracking branch for `branch` if there is any.\n\n        Parameters\n        ----------\n        branch: str\n            local branch to look up. If none is given, active branch is used.\n\n        Returns\n        -------\n        tuple\n            (remote or None, refspec or None) of the tracking branch\n        \"\"\"\n        if branch is None:\n            branch = self.get_active_branch()\n            if branch is None:\n                return None, None\n\n        track_remote = self.config.get('branch.{0}.remote'.format(branch), None)\n        track_branch = self.config.get('branch.{0}.merge'.format(branch), None)\n        return track_remote, track_branch\n\n    @property\n    def count_objects(self):\n        \"\"\"return dictionary with count, size(in KiB) information of git objects\n        \"\"\"\n\n        count_cmd = ['git', 'count-objects', '-v']\n        count_str, err = self._git_custom_command('', count_cmd)\n        count = {key: int(value)\n                 for key, value in [item.split(': ')\n                                    for item in count_str.split('\\n')\n                                    if len(item.split(': ')) == 2]}\n        return count\n\n    def get_missing_files(self):\n        \"\"\"Return a list of paths with missing files (and no staged deletion)\"\"\"\n        return [f.split('\\t')[1]\n                for f in self.repo.git.diff('--raw', '--name-status').split('\\n')\n                if f.split('\\t')[0] == 'D']\n\n    def get_deleted_files(self):\n        \"\"\"Return a list of paths with deleted files (staged deletion)\"\"\"\n        return [f.split('\\t')[1]\n                for f in self.repo.git.diff('--raw', '--name-status', '--staged').split('\\n')\n                if f.split('\\t')[0] == 'D']\n\n    def get_git_attributes(self):\n        \"\"\"Check git attribute for the current repository (not per-file support for now)\n\n        Parameters\n        ----------\n        all_: bool\n          Adds --all to git check-attr call\n\n        Returns\n        -------\n        dict:\n          attribute: value pairs\n        \"\"\"\n        out, err = self._git_custom_command([\".\"], [\"git\", \"check-attr\", \"--all\"])\n        assert not err, \"no stderr output is expected\"\n        out_split = [\n            # splitting by : would leave leading space(s)\n            [e.lstrip(' ') for e in l.split(':', 2)]\n            for l in out.split('\\n') if l\n        ]\n        assert all(o[0] == '.' for o in out_split)  # for paranoid\n        return dict(o[1:] for o in out_split)\n\n\n# TODO\n# remove submodule: nope, this is just deinit_submodule + remove\n# status?\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface to Git via GitPython\n\nFor further information on GitPython see http://gitpython.readthedocs.org/\n\n\"\"\"\n\nimport logging\nimport re\nimport shlex\nimport os\nfrom os import linesep\nfrom os.path import join as opj\nfrom os.path import exists\nfrom os.path import normpath\nfrom os.path import isabs\nfrom os.path import commonprefix\nfrom os.path import relpath\nfrom os.path import realpath\nfrom os.path import dirname\nfrom os.path import basename\nfrom os.path import curdir\nfrom os.path import pardir\nfrom os.path import sep\nimport posixpath\nfrom weakref import WeakValueDictionary\n\n\nfrom six import string_types\nfrom six import add_metaclass\nfrom functools import wraps\nimport git as gitpy\nfrom git.exc import GitCommandError\nfrom git.exc import NoSuchPathError\nfrom git.exc import InvalidGitRepositoryError\nfrom git.objects.blob import Blob\n\nfrom datalad import ssh_manager\nfrom datalad.cmd import GitRunner\nfrom datalad.consts import GIT_SSH_COMMAND\nfrom datalad.dochelpers import exc_str\nfrom datalad.config import ConfigManager\nfrom datalad.utils import assure_list\nfrom datalad.utils import optional_args\nfrom datalad.utils import on_windows\nfrom datalad.utils import getpwd\nfrom datalad.utils import updated\nfrom datalad.utils import posix_relpath\n\n\n# imports from same module:\nfrom .external_versions import external_versions\nfrom .exceptions import CommandError\nfrom .exceptions import DeprecatedError\nfrom .exceptions import FileNotInRepositoryError\nfrom .exceptions import MissingBranchError\nfrom .network import RI\nfrom .network import is_ssh\nfrom .repo import Flyweight\nfrom .repo import RepoInterface\n\n# shortcuts\n_curdirsep = curdir + sep\n_pardirsep = pardir + sep\n\n\nlgr = logging.getLogger('datalad.gitrepo')\n\n# Override default GitPython's DB backend to talk directly to git so it doesn't\n# interfere with possible operations performed by gc/repack\ndefault_git_odbt = gitpy.GitCmdObjectDB\n\n# TODO: Figure out how GIT_PYTHON_TRACE ('full') is supposed to be used.\n# Didn't work as expected on a first try. Probably there is a neatier way to\n# log Exceptions from git commands.\n\n\n# TODO: ignore leading and/or trailing underscore to allow for\n# python-reserved words\n@optional_args\ndef kwargs_to_options(func, split_single_char_options=True,\n                      target_kw='options'):\n    \"\"\"Decorator to provide convenient way to pass options to command calls.\n\n    Parameters\n    ----------\n    func: Callable\n        function to decorate\n    split_single_char_options: bool\n        whether or not to split key and value of single char keyword arguments\n        into two subsequent entries of the list\n    target_kw: str\n        keyword argument to pass the generated list of cmdline arguments to\n\n    Returns\n    -------\n    Callable\n    \"\"\"\n\n    # TODO: don't overwrite options, but join\n\n    @wraps(func)\n    def newfunc(self, *args, **kwargs):\n        t_kwargs = dict()\n        t_kwargs[target_kw] = \\\n            gitpy.Git().transform_kwargs(\n                split_single_char_options=split_single_char_options,\n                **kwargs)\n        return func(self, *args, **t_kwargs)\n    return newfunc\n\n\ndef to_options(**kwargs):\n    \"\"\"Transform keyword arguments into a list of cmdline options\n\n    Parameters\n    ----------\n    split_single_char_options: bool\n\n    kwargs:\n\n    Returns\n    -------\n    list\n    \"\"\"\n    # TODO: borrow_docs!\n\n    return gitpy.Git().transform_kwargs(**kwargs)\n\n\ndef _normalize_path(base_dir, path):\n    \"\"\"Helper to check paths passed to methods of this class.\n\n    Checks whether `path` is beneath `base_dir` and normalizes it.\n    Additionally paths are converted into relative paths with respect to\n    `base_dir`, considering PWD in case of relative paths. This\n    is intended to be used in repository classes, which means that\n    `base_dir` usually will be the repository's base directory.\n\n    Parameters\n    ----------\n    path: str\n        path to be normalized\n    base_dir: str\n        directory to serve as base to normalized, relative paths\n\n    Returns\n    -------\n    str:\n        path, that is a relative path with respect to `base_dir`\n    \"\"\"\n    if not path:\n        return path\n\n    base_dir = realpath(base_dir)  # realpath OK\n    # path = normpath(path)\n    # Note: disabled normpath, because it may break paths containing symlinks;\n    # But we don't want to realpath relative paths, in case cwd isn't the\n    # correct base.\n\n    if isabs(path):\n        # path might already be a symlink pointing to annex etc,\n        # so realpath only its directory, to get \"inline\" with\n        # realpath(base_dir) above\n        path = opj(realpath(dirname(path)), basename(path))  # realpath OK\n    # Executive decision was made to not do this kind of magic!\n    #\n    # elif commonprefix([realpath(getpwd()), base_dir]) == base_dir:\n    #     # If we are inside repository, rebuilt relative paths.\n    #     path = opj(realpath(getpwd()), path)\n    #\n    # BUT with relative curdir/pardir start it would assume relative to curdir\n    #\n    elif path.startswith(_curdirsep) or path.startswith(_pardirsep):\n        path = normpath(opj(realpath(getpwd()), path))  # realpath OK\n    else:\n        # We were called from outside the repo. Therefore relative paths\n        # are interpreted as being relative to self.path already.\n        return path\n\n    if commonprefix([path, base_dir]) != base_dir:\n        raise FileNotInRepositoryError(msg=\"Path outside repository: %s\"\n                                           % path, filename=path)\n\n    return relpath(path, start=base_dir)\n\n\n@optional_args\ndef normalize_path(func):\n    \"\"\"Decorator to provide unified path conversion for a single file\n\n    Unlike normalize_paths, intended to be used for functions dealing with a\n    single filename at a time\n\n    Note\n    ----\n    This is intended to be used within the repository classes and therefore\n    returns a class method!\n\n    The decorated function is expected to take a path at\n    first positional argument (after 'self'). Additionally the class `func`\n    is a member of, is expected to have an attribute 'path'.\n    \"\"\"\n\n    @wraps(func)\n    def newfunc(self, file_, *args, **kwargs):\n        file_new = _normalize_path(self.path, file_)\n        return func(self, file_new, *args, **kwargs)\n\n    return newfunc\n\n\n@optional_args\ndef normalize_paths(func, match_return_type=True, map_filenames_back=False,\n                    serialize=False):\n    \"\"\"Decorator to provide unified path conversions.\n\n    Note\n    ----\n    This is intended to be used within the repository classes and therefore\n    returns a class method!\n\n    The decorated function is expected to take a path or a list of paths at\n    first positional argument (after 'self'). Additionally the class `func`\n    is a member of, is expected to have an attribute 'path'.\n\n    Accepts either a list of paths or a single path in a str. Passes a list\n    to decorated function either way, but would return based on the value of\n    match_return_type and possibly input argument.\n\n    If a call to the wrapped function includes normalize_path and it is False\n    no normalization happens for that function call (used for calls to wrapped\n    functions within wrapped functions, while possible CWD is within a\n    repository)\n\n    Parameters\n    ----------\n    match_return_type : bool, optional\n      If True, and a single string was passed in, it would return the first\n      element of the output (after verifying that it is a list of length 1).\n      It makes easier to work with single files input.\n    map_filenames_back : bool, optional\n      If True and returned value is a dictionary, it assumes to carry entries\n      one per file, and then filenames are mapped back to as provided from the\n      normalized (from the root of the repo) paths\n    serialize : bool, optional\n      Loop through files giving only a single one to the function one at a time.\n      This allows to simplify implementation and interface to annex commands\n      which do not take multiple args in the same call (e.g. checkpresentkey)\n    \"\"\"\n\n    @wraps(func)\n    def newfunc(self, files, *args, **kwargs):\n\n        normalize = _normalize_path if kwargs.pop('normalize_paths', True) \\\n            else lambda rpath, filepath: filepath\n\n        if files:\n            if isinstance(files, string_types) or not files:\n                files_new = [normalize(self.path, files)]\n                single_file = True\n            elif isinstance(files, list):\n                files_new = [normalize(self.path, path) for path in files]\n                single_file = False\n            else:\n                raise ValueError(\"_files_decorator: Don't know how to handle \"\n                                 \"instance of %s.\" % type(files))\n        else:\n            single_file = None\n            files_new = []\n\n        if map_filenames_back:\n            def remap_filenames(out):\n                \"\"\"Helper to map files back to non-normalized paths\"\"\"\n                if isinstance(out, dict):\n                    assert(len(out) == len(files_new))\n                    files_ = [files] if single_file else files\n                    mapped = out.__class__()\n                    for fin, fout in zip(files_, files_new):\n                        mapped[fin] = out[fout]\n                    return mapped\n                else:\n                    return out\n        else:\n            remap_filenames = lambda x: x\n\n        if serialize:  # and not single_file:\n            result = [\n                func(self, f, *args, **kwargs)\n                for f in files_new\n            ]\n        else:\n            result = func(self, files_new, *args, **kwargs)\n\n        if single_file is None:\n            # no files were provided, nothing we can do really\n            return result\n        elif (result is None) or not match_return_type or not single_file:\n            # If function doesn't return anything or no denormalization\n            # was requested or it was not a single file\n            return remap_filenames(result)\n        elif single_file:\n            if len(result) != 1:\n                # Magic doesn't apply\n                return remap_filenames(result)\n            elif isinstance(result, (list, tuple)):\n                return result[0]\n            elif isinstance(result, dict) and tuple(result)[0] == files_new[0]:\n                # assume that returned dictionary has files as keys.\n                return tuple(result.values())[0]\n            else:\n                # no magic can apply\n                return remap_filenames(result)\n        else:\n            return RuntimeError(\"should have not got here... check logic\")\n\n    return newfunc\n\n\ndef check_git_configured():\n    \"\"\"Do a check if git is configured (user.name and user.email are set)\n\n    Raises\n    ------\n    RuntimeError  if any of those two ariables are not set\n\n    Returns\n    -------\n    dict with user.name and user.email entries\n    \"\"\"\n\n    check_runner = GitRunner()\n    vals = {}\n    for c in 'user.name', 'user.email':\n        try:\n            v, err = check_runner.run(['git', 'config', '--global', c])\n            vals[c] = v.rstrip('\\n')\n        except CommandError as exc:\n            lgr.debug(\"Failed to verify that git is configured: %s\",\n                      exc_str(exc))\n            raise RuntimeError(\n                \"You must configure git first (set both user.name and \"\n                \"user.email) settings before using DataLad.\"\n            )\n    return vals\n\n\ndef _remove_empty_items(list_):\n    \"\"\"Remove empty entries from list\n\n    This is needed, since some functions of GitPython may convert\n    an empty entry to '.', when used with a list of paths.\n\n    Parameter:\n    ----------\n    list_: list of str\n\n    Returns\n    -------\n    list of str\n    \"\"\"\n    if not isinstance(list_, list):\n        lgr.warning(\n            \"_remove_empty_items() called with non-list type: %s\" % type(list_))\n        return list_\n    return [file_ for file_ in list_ if file_]\n\n\ndef Repo(*args, **kwargs):\n    \"\"\"Factory method around gitpy.Repo to consistently initiate with different\n    backend\n    \"\"\"\n    # TODO: This probably doesn't work as intended (or at least not as\n    #       consistently as intended). gitpy.Repo could be instantiated by\n    #       classmethods Repo.init or Repo.clone_from. In these cases 'odbt'\n    #       would be needed as a paramter to these methods instead of the\n    #       constructor.\n    if 'odbt' not in kwargs:\n        kwargs['odbt'] = default_git_odbt\n    return gitpy.Repo(*args, **kwargs)\n\n\ndef split_remote_branch(branch):\n    \"\"\"Splits a remote branch's name into the name of the remote and the name\n    of the branch.\n\n    Parameters\n    ----------\n    branch: str\n      the remote branch's name to split\n\n    Returns\n    -------\n    list of str\n    \"\"\"\n    assert '/' in branch, \\\n        \"remote branch %s must have had a /\" % branch\n    assert not branch.endswith('/'), \\\n        \"branch name with trailing / is invalid. (%s)\" % branch\n    return branch.split('/', 1)\n\n\n@add_metaclass(Flyweight)\nclass GitRepo(RepoInterface):\n    \"\"\"Representation of a git repository\n\n    \"\"\"\n\n    # We use our sshrun helper\n    GIT_SSH_ENV = {'GIT_SSH_COMMAND': GIT_SSH_COMMAND}\n\n    # Just a non-functional example:\n    # must be implemented, since abstract in RepoInterface:\n    def sth_like_file_has_content(self):\n        return \"Yes, if it's in the index\"\n\n    # We must check git config to have name and email set, but\n    # should do it once\n    _config_checked = False\n\n    # Begin Flyweight:\n\n    _unique_instances = WeakValueDictionary()\n\n    @classmethod\n    def _flyweight_id_from_args(cls, *args, **kwargs):\n\n        if args:\n            # to a certain degree we need to simulate an actual call to __init__\n            # and make sure, passed arguments are fitting:\n            # TODO: Figure out, whether there is a cleaner way to do this in a\n            # generic fashion\n            assert('path' not in kwargs)\n            path = args[0]\n            args = args[1:]\n        elif 'path' in kwargs:\n            path = kwargs.pop('path')\n        else:\n            raise TypeError(\"__init__() requires argument `path`\")\n\n        if path is None:\n            raise AttributeError\n\n        # Sanity check for argument `path`:\n        # raise if we cannot deal with `path` at all or\n        # if it is not a local thing:\n        path = RI(path).localpath\n        # resolve symlinks to make sure we have exactly one instance per\n        # physical repository at a time\n        path = realpath(path)\n        kwargs['path'] = path\n        return path, args, kwargs\n\n    @classmethod\n    def _flyweight_invalid(cls, id_):\n        return not cls.is_valid_repo(id_)\n\n    @classmethod\n    def _flyweight_reject(cls, id_, *args, **kwargs):\n        # TODO:\n        # This is a temporary approach. See PR # ...\n        # create = kwargs.pop('create', None)\n        # kwargs.pop('path', None)\n        # if create and kwargs:\n        #     # we have `create` plus options other than `path`\n        #     return \"Call to {0}() with args {1} and kwargs {2} conflicts \" \\\n        #            \"with existing instance {3}.\" \\\n        #            \"This is likely to be caused by inconsistent logic in \" \\\n        #            \"your code.\" \\\n        #            \"\".format(cls, args, kwargs, cls._unique_instances[id_])\n        pass\n\n    # End Flyweight\n\n    def __init__(self, path, url=None, runner=None, create=True,\n                 git_opts=None, repo=None, **kwargs):\n        \"\"\"Creates representation of git repository at `path`.\n\n        Can also be used to create a git repository at `path`.\n\n        Parameters\n        ----------\n        path: str\n          path to the git repository; In case it's not an absolute path,\n          it's relative to PWD\n        url: str, optional\n          DEPRECATED -- use .clone() class method\n          url to the to-be-cloned repository. Requires a valid git url\n          according to:\n          http://www.kernel.org/pub/software/scm/git/docs/git-clone.html#URLS .\n        create: bool, optional\n          if true, creates a git repository at `path` if there is none. Also\n          creates `path`, if it doesn't exist.\n          If set to false, an exception is raised in case `path` doesn't exist\n          or doesn't contain a git repository.\n        repo: git.Repo, optional\n          GitPython's Repo instance to (re)use if provided\n        kwargs:\n          keyword arguments serving as additional options to the git-init\n          command. Therefore, it makes sense only if called with `create`.\n\n          Generally, this way of passing options to the git executable is\n          (or will be) used a lot in this class. It's a transformation of\n          python-style keyword arguments (or a `dict`) to command line arguments,\n          provided by GitPython.\n\n          A single character keyword will be prefixed by '-', multiple characters\n          by '--'. An underscore in the keyword becomes a dash. The value of the\n          keyword argument is used as the value for the corresponding command\n          line argument. Assigning a boolean creates a flag.\n\n          Examples:\n          no_commit=True => --no-commit\n          C='/my/path'   => -C /my/path\n\n        \"\"\"\n\n        if url is not None:\n            raise DeprecatedError(\n                new=\".clone() class method\",\n                version=\"0.5.0\",\n                msg=\"RF: url passed to init()\"\n            )\n\n        # So that we \"share\" control paths with git/git-annex\n        if ssh_manager:\n            ssh_manager.assure_initialized()\n\n        if not GitRepo._config_checked:\n            check_git_configured()\n            GitRepo._config_checked = True\n\n        self.realpath = realpath(path)\n        # note: we may also want to distinguish between a path to the worktree\n        # and the actual repository\n\n        # Disable automatic garbage and autopacking\n        self._GIT_COMMON_OPTIONS = ['-c', 'receive.autogc=0', '-c', 'gc.auto=0']\n        # actually no need with default GitPython db backend not in memory\n        # default_git_odbt but still allows for faster testing etc.\n        # May be eventually we would make it switchable _GIT_COMMON_OPTIONS = []\n\n        if git_opts is None:\n            git_opts = {}\n        if kwargs:\n            git_opts.update(kwargs)\n\n        self.path = path\n        self.cmd_call_wrapper = runner or GitRunner(cwd=self.path)\n        self._repo = repo\n        self._cfg = None\n\n        _valid_repo = GitRepo.is_valid_repo(path)\n        if create and not _valid_repo:\n            if repo is not None:\n                # `repo` passed with `create`, which doesn't make sense\n                raise TypeError(\"argument 'repo' must not be used with 'create'\")\n\n            try:\n                lgr.debug(\n                    \"Initialize empty Git repository at '%s'%s\",\n                    path,\n                    ' %s' % git_opts if git_opts else '')\n                self._repo = self.cmd_call_wrapper(gitpy.Repo.init, path,\n                                                   mkdir=True,\n                                                   odbt=default_git_odbt,\n                                                   **git_opts)\n            except GitCommandError as e:\n                lgr.error(exc_str(e))\n                raise\n        else:\n            # Note: We used to call gitpy.Repo(path) here, which potentially\n            # raised NoSuchPathError or InvalidGitRepositoryError. This is\n            # used by callers of GitRepo.__init__() to detect whether we have a\n            # valid repo at `path`. Now, with switching to lazy loading property\n            # `repo`, we detect those cases without instantiating a\n            # gitpy.Repo().\n\n            if not exists(path):\n                raise NoSuchPathError(path)\n            if not _valid_repo:\n                raise InvalidGitRepositoryError(path)\n\n        # inject git options into GitPython's git call wrapper:\n        # Note: `None` currently can happen, when Runner's protocol prevents\n        # calls above from being actually executed (DryRunProtocol)\n        if self._repo is not None:\n            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS\n\n        # with DryRunProtocol path might still not exist\n        if exists(self.realpath):\n            self.inode = os.stat(self.realpath).st_ino\n        else:\n            self.inode = None\n\n    @property\n    def repo(self):\n        # with DryRunProtocol path not exist\n        if exists(self.realpath):\n            inode = os.stat(self.realpath).st_ino\n        else:\n            inode = None\n        if self.inode != inode:\n            # reset background processes invoked by GitPython:\n            self._repo.git.clear_cache()\n            self.inode = inode\n\n        if self._repo is None:\n            # Note, that this may raise GitCommandError, NoSuchPathError,\n            # InvalidGitRepositoryError:\n            self._repo = self.cmd_call_wrapper(Repo, self.path)\n            lgr.log(8, \"Using existing Git repository at %s\", self.path)\n\n        # inject git options into GitPython's git call wrapper:\n        # Note: `None` currently can happen, when Runner's protocol prevents\n        # call of Repo(path) above from being actually executed (DryRunProtocol)\n        if self._repo is not None:\n            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS\n\n        return self._repo\n\n    @classmethod\n    def clone(cls, url, path, *args, **kwargs):\n        \"\"\"Clone url into path\n\n        Provides workarounds for known issues (e.g.\n        https://github.com/datalad/datalad/issues/785)\n\n        Parameters\n        ----------\n        url : str\n        path : str\n        expect_fail : bool\n          Either expect that command might fail, so error should be logged then\n          at DEBUG level instead of ERROR\n        \"\"\"\n\n        if 'repo' in kwargs:\n            raise TypeError(\"argument 'repo' conflicts with cloning\")\n            # TODO: what about 'create'?\n\n        expect_fail = kwargs.pop('expect_fail', False)\n        # fail early on non-empty target:\n        from os import listdir\n        if exists(path) and listdir(path):\n            # simulate actual GitCommandError:\n            lgr.warning(\"destination path '%s' already exists and is not an \"\n                        \"empty directory.\" % path)\n            raise GitCommandError(\n                ['git', 'clone', '-v', url, path],\n                128,\n                \"fatal: destination path '%s' already exists and is not an \"\n                \"empty directory.\" % path)\n        else:\n            # protect against cloning into existing and obviously dangling\n            # instance for that location\n            try:\n                del cls._unique_instances[path]\n            except KeyError:\n                # didn't exist - all fine\n                pass\n\n        # try to get a local path from `url`:\n        try:\n            if not isinstance(url, RI):\n                url = RI(url).localpath\n            else:\n                url = url.localpath\n        except ValueError:\n            pass\n\n        if is_ssh(url):\n            ssh_manager.get_connection(url).open()\n            # TODO: with git <= 2.3 keep old mechanism:\n            #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n            env = GitRepo.GIT_SSH_ENV\n        else:\n            env = None\n        ntries = 5  # 3 is not enough for robust workaround\n        for trial in range(ntries):\n            try:\n                lgr.debug(\"Git clone from {0} to {1}\".format(url, path))\n                repo = gitpy.Repo.clone_from(url, path, env=env,\n                                             odbt=default_git_odbt)\n                # Note/TODO: signature for clone from:\n                # (url, to_path, progress=None, env=None, **kwargs)\n\n                lgr.debug(\"Git clone completed\")\n                break\n            except GitCommandError as e:\n                # log here but let caller decide what to do\n                e_str = exc_str(e)\n                # see https://github.com/datalad/datalad/issues/785\n                if re.search(\"Request for .*aborted.*Unable to find\", str(e),\n                             re.DOTALL) \\\n                        and trial < ntries - 1:\n                    lgr.info(\n                        \"Hit a known issue with Git (see GH#785). Trial #%d, \"\n                        \"retrying\",\n                        trial)\n                    continue\n                    (lgr.debug if expect_fail else lgr.error)(e_str)\n                raise\n            except ValueError as e:\n                if gitpy.__version__ == '1.0.2' \\\n                        and \"I/O operation on closed file\" in str(e):\n                    # bug https://github.com/gitpython-developers/GitPython\n                    # /issues/383\n                    raise GitCommandError(\n                        \"clone has failed, telling ya\",\n                        999,  # good number\n                        stdout=\"%s already exists\" if exists(path) else \"\")\n                raise  # reraise original\n\n        gr = cls(path, *args, repo=repo, **kwargs)\n        return gr\n\n    def __del__(self):\n        # unbind possibly bound ConfigManager, to prevent all kinds of weird\n        # stalls etc\n        self._cfg = None\n        # Make sure to flush pending changes, especially close batch processes\n        # (internal `git cat-file --batch` by GitPython)\n        try:\n            if hasattr(self, 'repo') and exists(self.path) \\\n                    and self.repo is not None:\n                # gc might be late, so the (temporary)\n                # repo doesn't exist on FS anymore\n\n                self.repo.git.clear_cache()\n                if exists(opj(self.path, '.git')):  # don't try to write otherwise\n                    self.repo.index.write()\n        except InvalidGitRepositoryError:\n            # might have being removed and no longer valid\n            pass\n\n    def __repr__(self):\n        return \"<GitRepo path=%s (%s)>\" % (self.path, type(self))\n\n    def __eq__(self, obj):\n        \"\"\"Decides whether or not two instances of this class are equal.\n\n        This is done by comparing the base repository path.\n        \"\"\"\n        return self.realpath == obj.realpath\n\n    @classmethod\n    def is_valid_repo(cls, path):\n        \"\"\"Returns if a given path points to a git repository\"\"\"\n        return exists(opj(path, '.git'))\n\n    @property\n    def config(self):\n        \"\"\"Get an instance of the parser for the persistent repository\n        configuration.\n\n        Note: This allows to also read/write .datalad/config,\n        not just .git/config\n\n        Returns\n        -------\n        ConfigManager\n        \"\"\"\n        if self._cfg is None:\n            # associate with this dataset and read the entire config hierarchy\n            self._cfg = ConfigManager(dataset=self, dataset_only=False)\n        return self._cfg\n\n    def is_with_annex(self, only_remote=False):\n        \"\"\"Return True if GitRepo (assumed) at the path has remotes with git-annex branch\n\n        Parameters\n        ----------\n        only_remote: bool, optional\n            Check only remote (no local branches) for having git-annex branch\n        \"\"\"\n        return any((b.endswith('/git-annex') or\n                    'annex/direct' in b\n                    for b in self.get_remote_branches())) or \\\n            ((not only_remote) and\n             any((b == 'git-annex' or 'annex/direct' in b\n                  for b in self.get_branches())))\n\n    @classmethod\n    def get_toppath(cls, path, follow_up=True, git_options=None):\n        \"\"\"Return top-level of a repository given the path.\n\n        Parameters\n        -----------\n        follow_up : bool\n          If path has symlinks -- they get resolved by git.  If follow_up is\n          True, we will follow original path up until we hit the same resolved\n          path.  If no such path found, resolved one would be returned.\n        git_options: list of str\n          options to be passed to the git rev-parse call\n\n        Return None if no parent directory contains a git repository.\n        \"\"\"\n        cmd = ['git']\n        if git_options:\n            cmd.extend(git_options)\n        cmd += [\"rev-parse\", \"--show-toplevel\"]\n        try:\n            toppath, err = GitRunner().run(\n                cmd,\n                cwd=path,\n                log_stdout=True, log_stderr=True,\n                expect_fail=True, expect_stderr=True)\n            toppath = toppath.rstrip('\\n\\r')\n        except CommandError:\n            return None\n        except OSError:\n            toppath = GitRepo.get_toppath(dirname(path), follow_up=follow_up,\n                                          git_options=git_options)\n\n        if follow_up:\n            path_ = path\n            path_prev = \"\"\n            while path_ and path_ != path_prev:  # on top /.. = /\n                if realpath(path_) == toppath:\n                    toppath = path_\n                    break\n                path_prev = path_\n                path_ = dirname(path_)\n\n        return toppath\n\n    # classmethod so behavior could be tuned in derived classes\n    @classmethod\n    def _get_added_files_commit_msg(cls, files):\n        if not files:\n            return \"No files were added\"\n        msg = \"Added %d file\" % len(files)\n        if len(files) > 1:\n            msg += \"s\"\n        return msg + '\\n\\nFiles:\\n' + '\\n'.join(files)\n\n    @normalize_paths\n    def add(self, files, commit=False, msg=None, git=True, git_options=None,\n            _datalad_msg=False, update=False):\n        \"\"\"Adds file(s) to the repository.\n\n        Parameters\n        ----------\n        files: list\n          list of paths to add\n        commit: bool\n          whether or not to directly commit\n        msg: str\n          commit message in case `commit=True`. A default message, containing\n          the list of files that were added, is created by default.\n        git: bool\n          somewhat ugly construction to be compatible with AnnexRepo.add();\n          has to be always true.\n        update: bool\n          --update option for git-add. From git's manpage:\n           Update the index just where it already has an entry matching\n           <pathspec>. This removes as well as modifies index entries to match\n           the working tree, but adds no new files.\n\n           If no <pathspec> is given when --update option is used, all tracked\n           files in the entire working tree are updated (old versions of Git\n           used to limit the update to the current directory and its\n           subdirectories).\n        \"\"\"\n\n        # TODO: git_options is used as options for the git-add here,\n        # instead of options to the git executable => rename for consistency\n\n        # needs to be True - see docstring:\n        assert(git)\n\n        files = _remove_empty_items(files)\n        out = []\n\n        if files or git_options or update:\n            try:\n                # without --verbose git 2.9.3  add does not return anything\n                add_out = self._git_custom_command(\n                    files,\n                    ['git', 'add'] + assure_list(git_options) +\n                    to_options(update=update) + ['--verbose']\n                )\n                # get all the entries\n                out = self._process_git_get_output(*add_out)\n                # Note: as opposed to git cmdline, force is True by default in\n                #       gitpython, which would lead to add things, that are\n                #       ignored or excluded otherwise\n                # 2. Note: There is an issue with globbing (like adding '.'),\n                #       which apparently doesn't care for 'force' and therefore\n                #       adds '.git/...'. May be it's expanded at the wrong\n                #       point in time or sth. like that.\n                # For now, use direct call to git add.\n                #self.cmd_call_wrapper(self.repo.index.add, files, write=True,\n                #                      force=False)\n                # TODO: May be make use of 'fprogress'-option to indicate\n                # progress\n                # But then, we don't have it for git-annex add, anyway.\n                #\n                # TODO: Is write=True a reasonable way to do it?\n                # May be should not write until success of operation is\n                # confirmed?\n                # What's best in case of a list of files?\n            except OSError as e:\n                lgr.error(\"add: %s\" % e)\n                raise\n\n        else:\n            lgr.warning(\"add was called with empty file list and no options.\")\n\n        if commit:\n            if msg is None:\n                msg = self._get_added_files_commit_msg(files)\n            self.commit(msg=msg, _datalad_msg=_datalad_msg)\n\n        # Make sure return value from GitRepo is consistent with AnnexRepo\n        # currently simulating similar return value, assuming success\n        # for all files:\n        # TODO: Make return values consistent across both *Repo classes!\n        return out\n\n    @staticmethod\n    def _process_git_get_output(stdout, stderr=None):\n        \"\"\"Given both outputs (stderr is ignored atm) of git add - process it\n\n        Primarily to centralize handling in both indirect annex and direct\n        modes when ran through proxy\n        \"\"\"\n        return [{u'file': f, u'success': True}\n                for f in re.findall(\"'(.*)'[\\n$]\", stdout)]\n\n    @normalize_paths(match_return_type=False)\n    def remove(self, files, recursive=False, **kwargs):\n        \"\"\"Remove files.\n\n        Calls git-rm.\n\n        Parameters\n        ----------\n        files: str\n          list of paths to remove\n        recursive: False\n          either to allow recursive removal from subdirectories\n        kwargs:\n          see `__init__`\n\n        Returns\n        -------\n        [str]\n          list of successfully removed files.\n        \"\"\"\n\n        files = _remove_empty_items(files)\n\n        if recursive:\n            kwargs['r'] = True\n        stdout, stderr = self._git_custom_command(\n            files, ['git', 'rm'] + to_options(**kwargs))\n\n        # output per removed file is expected to be \"rm 'PATH'\":\n        return [line.strip()[4:-1] for line in stdout.splitlines()]\n\n        #return self.repo.git.rm(files, cached=False, **kwargs)\n\n    def precommit(self):\n        \"\"\"Perform pre-commit maintenance tasks\n        \"\"\"\n        # All GitPython commands should take care about flushing index\n        # whenever they modify it, so we would not care to do anything\n        # if self.repo is not None and exists(opj(self.path, '.git')):  # don't try to write otherwise:\n        #     # flush possibly cached in GitPython changes to index:\n        #     # if self.repo.git:\n        #     #     sys.stderr.write(\"CLEARING\\n\")\n        #     #     self.repo.git.clear_cache()\n        #     self.repo.index.write()\n        pass\n\n    @staticmethod\n    def _get_prefixed_commit_msg(msg):\n        DATALAD_PREFIX = \"[DATALAD]\"\n        return DATALAD_PREFIX if not msg else \"%s %s\" % (DATALAD_PREFIX, msg)\n\n    def commit(self, msg=None, options=None, _datalad_msg=False, careless=True,\n               files=None):\n        \"\"\"Commit changes to git.\n\n        Parameters\n        ----------\n        msg: str, optional\n          commit-message\n        options: list of str, optional\n          cmdline options for git-commit\n        _datalad_msg: bool, optional\n          To signal that commit is automated commit by datalad, so\n          it would carry the [DATALAD] prefix\n        careless: bool, optional\n          if False, raise when there's nothing actually committed;\n          if True, don't care\n        files: list of str, optional\n          path(s) to commit\n        \"\"\"\n\n        self.precommit()\n\n        if _datalad_msg:\n            msg = self._get_prefixed_commit_msg(msg)\n\n        if not msg:\n            if options:\n                if \"--allow-empty-message\" not in options:\n                        options.append(\"--allow-empty-message\")\n            else:\n                options = [\"--allow-empty-message\"]\n\n        # Note: We used to use a direct call to git only if there were options,\n        # since we can't pass all possible options to gitpython's implementation\n        # of commit.\n        # But there's an additional issue. GitPython implements commit in a way,\n        # that it might create a new commit, when a direct call wouldn't. This\n        # was discovered with a modified (but unstaged) submodule, leading to a\n        # commit, that apparently did nothing - git status still showed the very\n        # same thing afterwards. But a commit was created nevertheless:\n        # diff --git a/sub b/sub\n        # --- a/sub\n        # +++ b/sub\n        # @@ -1 +1 @@\n        # -Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f\n        # +Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f-dirty\n        #\n        # Therefore, for now always use direct call.\n        # TODO: Figure out, what exactly is going on with gitpython here\n\n        cmd = ['git', 'commit'] + ([\"-m\", msg if msg else \"\"])\n        if options:\n            cmd.extend(options)\n        lgr.debug(\"Committing via direct call of git: %s\" % cmd)\n\n        try:\n            self._git_custom_command(files, cmd,\n                                     expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if 'nothing to commit' in e.stdout:\n                if careless:\n                    lgr.debug(\"nothing to commit in {}. \"\n                              \"Ignored.\".format(self))\n                else:\n                    raise\n            elif 'no changes added to commit' in e.stdout or \\\n                    'nothing added to commit' in e.stdout:\n                if careless:\n                    lgr.debug(\"no changes added to commit in {}. \"\n                              \"Ignored.\".format(self))\n                else:\n                    raise\n            elif \"did not match any file(s) known to git.\" in e.stderr:\n                # TODO: Improve FileNotInXXXXError classes to better deal with\n                # multiple files; Also consider PathOutsideRepositoryError\n                raise FileNotInRepositoryError(cmd=e.cmd,\n                                               msg=\"File(s) unknown to git\",\n                                               code=e.code,\n                                               filename=linesep.join(\n                                            [l for l in e.stderr.splitlines()\n                                             if l.startswith(\"pathspec\")]))\n            else:\n                raise\n\n    def get_indexed_files(self):\n        \"\"\"Get a list of files in git's index\n\n        Returns\n        -------\n        list\n            list of paths rooting in git's base dir\n        \"\"\"\n\n        return [x[0] for x in self.cmd_call_wrapper(\n            self.repo.index.entries.keys)]\n\n    def get_hexsha(self, object=None):\n        \"\"\"Return a hexsha for a given object. If None - of current HEAD\n\n        Parameters\n        ----------\n        object: str, optional\n          Any type of Git object identifier. See `git show`.\n        \"\"\"\n        cmd = ['git', 'show', '--no-patch', \"--format=%H\"]\n        if object:\n            cmd.append(object)\n        # make sure Git takes our argument as a revision\n        cmd.append('--')\n        try:\n            stdout, stderr = self._git_custom_command(\n                '', cmd, expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if 'bad revision' in e.stderr:\n                raise ValueError(\"Unknown object identifier: %s\" % object)\n            elif 'does not have any commits yet' in e.stderr:\n                return None\n            else:\n                raise e\n        stdout = stdout.splitlines()\n        assert(len(stdout) == 1)\n        return stdout[0]\n\n    def get_merge_base(self, treeishes):\n        \"\"\"Get a merge base hexsha\n\n        Parameters\n        ----------\n        treeishes: str or list of str\n          List of treeishes (branches, hexshas, etc) to determine the merge\n          base of. If a single value provided, returns merge_base with the\n          current branch.\n\n        Returns\n        -------\n        str or None\n          If no merge-base for given commits, or specified treeish doesn't\n          exist, None returned\n        \"\"\"\n        if isinstance(treeishes, string_types):\n            treeishes = [treeishes]\n        if not treeishes:\n            raise ValueError(\"Provide at least a single value\")\n        elif len(treeishes) == 1:\n            treeishes = treeishes + [self.get_active_branch()]\n\n        try:\n            bases = self.repo.merge_base(*treeishes)\n        except GitCommandError as exc:\n            if \"fatal: Not a valid object name\" in str(exc):\n                return None\n            raise\n\n        if not bases:\n            return None\n        assert(len(bases) == 1)  # we do not do 'all' yet\n        return bases[0].hexsha\n\n    def get_committed_date(self, branch=None):\n        \"\"\"Get the date stamp of the last commit (in a branch). None if no commit\"\"\"\n        try:\n            commit = next(\n                self.get_branch_commits(branch\n                                        or self.get_active_branch())\n            )\n        except Exception as exc:\n            lgr.debug(\"Got exception while trying to get last commit: %s\",\n                      exc_str(exc))\n            return None\n        return commit.committed_date\n\n    def get_active_branch(self):\n        try:\n            branch = self.repo.active_branch.name\n        except TypeError as e:\n            if \"HEAD is a detached symbolic reference\" in str(e):\n                lgr.debug(\"detached HEAD in {0}\".format(self))\n                return None\n            else:\n                raise\n        return branch\n\n    def get_branches(self):\n        \"\"\"Get all branches of the repo.\n\n        Returns\n        -------\n        [str]\n            Names of all branches of this repository.\n        \"\"\"\n\n        return [branch.name for branch in self.repo.branches]\n\n    def get_remote_branches(self):\n        \"\"\"Get all branches of all remotes of the repo.\n\n        Returns\n        -----------\n        [str]\n            Names of all remote branches.\n        \"\"\"\n        # TODO: Reconsider melting with get_branches()\n\n        # TODO: treat entries like this: origin/HEAD -> origin/master'\n        # currently this is done in collection\n\n        # For some reason, this is three times faster than the version below:\n        remote_branches = list()\n        for remote in self.repo.remotes:\n            try:\n                for ref in remote.refs:\n                    remote_branches.append(ref.name)\n            except AssertionError as e:\n                if str(e).endswith(\"did not have any references\"):\n                    # this will happen with git annex special remotes\n                    pass\n                else:\n                    raise e\n        return remote_branches\n        # return [branch.strip() for branch in\n        #         self.repo.git.branch(r=True).splitlines()]\n\n    def get_remotes(self, with_urls_only=False):\n        \"\"\"Get known remotes of the repository\n\n        Parameters\n        ----------\n        with_urls_only : bool, optional\n          return only remotes which have urls\n\n        Returns\n        -------\n        remotes : list of str\n          List of names of the remotes\n        \"\"\"\n\n        # Note: read directly from config and spare instantiation of gitpy.Repo\n        # since we need this in AnnexRepo constructor. Furthermore gitpy does it\n        # pretty much the same way and the use of a Repo instance seems to have\n        # no reason other than a nice object oriented look.\n        from datalad.utils import unique\n\n        self.config.reload()\n        remotes = unique([x[7:] for x in self.config.sections()\n                          if x.startswith(\"remote.\")])\n\n        if with_urls_only:\n            remotes = [\n                r for r in remotes\n                if self.config.get('remote.%s.url' % r)\n            ]\n        return remotes\n\n    def get_files(self, branch=None):\n        \"\"\"Get a list of files in git.\n\n        Lists the files in the (remote) branch.\n\n        Parameters\n        ----------\n        branch: str\n          Name of the branch to query. Default: active branch.\n\n        Returns\n        -------\n        [str]\n          list of files.\n        \"\"\"\n        # TODO: RF codes base and melt get_indexed_files() in\n\n        if branch is None:\n            # active branch can be queried way faster:\n            return self.get_indexed_files()\n        else:\n            return [item.path for item in self.repo.tree(branch).traverse()\n                    if isinstance(item, Blob)]\n\n    def get_file_content(self, file_, branch='HEAD'):\n        \"\"\"\n\n        Returns\n        -------\n        [str]\n          content of file_ as a list of lines.\n        \"\"\"\n        content_str = self.repo.commit(branch).tree[file_].data_stream.read()\n\n        # in python3 a byte string is returned. Need to convert it:\n        from six import PY3\n        if PY3:\n            conv_str = u''\n            for b in bytes(content_str):\n                conv_str += chr(b)\n            return conv_str.splitlines()\n        else:\n            return content_str.splitlines()\n        # TODO: keep splitlines?\n\n    def _get_files_history(self, files, branch='HEAD'):\n        \"\"\"\n\n        Parameters\n        ----------\n        files: list\n          list of files, only commits with queried files are considered\n        branch: str\n          Name of the branch to query. Default: HEAD.\n\n        Returns\n        -------\n        [iterator]\n        yielding Commit items generator from branch history associated with files\n        \"\"\"\n        return gitpy.objects.commit.Commit.iter_items(self.repo, branch, paths=files)\n\n    def _get_remotes_having_commit(self, commit_hexsha, with_urls_only=True):\n        \"\"\"Traverse all branches of the remote and check if commit in any of their ancestry\n\n        It is a generator yielding names of the remotes\n        \"\"\"\n        out, err = self._git_custom_command(\n            '', 'git branch -r --contains ' + commit_hexsha\n        )\n        # sanitize a bit (all the spaces and new lines)\n        remote_branches = [\n            b  # could be origin/HEAD -> origin/master, we just skip ->\n            for b in filter(bool, out.split())\n            if b != '->'\n        ]\n        return [\n            remote\n            for remote in self.get_remotes(with_urls_only=with_urls_only)\n            if any(rb.startswith(remote + '/') for rb in remote_branches)\n        ]\n\n    def _gitpy_custom_call(self, cmd, cmd_args=None, cmd_options=None,\n                           git_options=None, env=None,\n\n                           # 'old' options for Runner; not sure yet, which of\n                           # them are actually still needed:\n                           log_stdout=True, log_stderr=True, log_online=False,\n                           expect_stderr=True, cwd=None,\n                           shell=None, expect_fail=False):\n\n        \"\"\"Helper to call GitPython's wrapper for git calls.\n\n        The used instance of `gitpy.Git` is bound to the repository,\n        which determines its working directory.\n        This is used for adhoc implementation of a git command and to\n        demonstrate how to use it in more specific implementations.\n\n        Note\n        ----\n        Aims to replace the use of datalad's `Runner` class for direct git\n        calls. (Currently the `_git_custom_command()` method).\n        Therefore mimicking its behaviour during RF'ing.\n\n        Parameters\n        ----------\n        cmd: str\n          the native git command to call\n        cmd_args: list of str\n          arguments to the git command\n        cmd_options: dict\n          options for the command as key, value pair\n          (this transformation, needs some central place to document)\n        git_options: dict\n          options for the git executable as key, value pair\n          (see above)\n        env: dict\n          environment vaiables to temporarily set for this call\n\n        TODO\n        ----\n        Example\n\n        Returns\n        -------\n        (stdout, stderr)\n        \"\"\"\n\n        # TODO: Reconsider when to log/stream what (stdout, stderr) and/or\n        # fully implement the behaviour of `Runner`\n\n        if log_online:\n            raise NotImplementedError(\"option 'log_online' not implemented yet\")\n        with_exceptions = not expect_fail\n        if cwd:\n            # the gitpy.cmd.Git instance, bound to this repository doesn't allow\n            # to explicitly set the working dir, except for using os.getcwd\n            raise NotImplementedError(\"working dir is a read-only property\")\n\n        _tmp_shell = gitpy.cmd.Git.USE_SHELL\n        gitpy.cmd.Git.USE_SHELL = shell\n\n        if env is None:\n            env = {}\n        if git_options is None:\n            git_options = {}\n        if cmd_options is None:\n            cmd_options = {}\n        cmd_options.update({'with_exceptions': with_exceptions,\n                            'with_extended_output': True})\n\n        # TODO: _GIT_COMMON_OPTIONS!\n\n        with self.repo.git.custom_environment(**env):\n            try:\n                status, std_out, std_err = \\\n                    self.repo.git(**git_options).__getattr__(cmd)(\n                        cmd_args, **cmd_options)\n            except GitCommandError as e:\n                # For now just reraise. May be raise CommandError instead\n                raise\n            finally:\n                gitpy.cmd.Git.USE_SHELL = _tmp_shell\n\n        if not expect_stderr and std_err:\n            lgr.error(\"Unexpected output on stderr: %s\" % std_err)\n            raise CommandError\n        if log_stdout:\n            for line in std_out.splitlines():\n                lgr.debug(\"stdout| \" + line)\n        if log_stderr:\n            for line in std_err.splitlines():\n                lgr.log(level=logging.DEBUG if expect_stderr else logging.ERROR,\n                        msg=\"stderr| \" + line)\n\n        return std_out, std_err\n\n    @normalize_paths(match_return_type=False)\n    def _git_custom_command(self, files, cmd_str,\n                            log_stdout=True, log_stderr=True, log_online=False,\n                            expect_stderr=True, cwd=None, env=None,\n                            shell=None, expect_fail=False):\n        \"\"\"Allows for calling arbitrary commands.\n\n        Helper for developing purposes, i.e. to quickly implement git commands\n        for proof of concept without the need to figure out, how this is done\n        via GitPython.\n\n        Parameters\n        ----------\n        files: list of files\n        cmd_str: str or list\n            arbitrary command str. `files` is appended to that string.\n\n        Returns\n        -------\n        stdout, stderr\n        \"\"\"\n\n        # ensure cmd_str becomes a well-formed list:\n        if isinstance(cmd_str, string_types):\n            if files and not cmd_str.strip().endswith(\" --\"):\n                cmd_str += \" --\"\n            cmd_str = shlex.split(cmd_str, posix=not on_windows)\n        else:\n            if files and cmd_str[-1] != '--':\n                cmd_str.append('--')\n\n        cmd = cmd_str + files\n\n        assert(cmd[0] == 'git')\n        cmd = cmd[:1] + self._GIT_COMMON_OPTIONS + cmd[1:]\n\n        from .exceptions import GitIgnoreError\n\n        try:\n            out, err = self.cmd_call_wrapper.run(\n                cmd,\n                log_stderr=log_stderr,\n                log_stdout=log_stdout,\n                log_online=log_online,\n                expect_stderr=expect_stderr,\n                cwd=cwd,\n                env=env,\n                shell=shell,\n                expect_fail=expect_fail)\n        except CommandError as e:\n            ignored = re.search(GitIgnoreError.pattern, e.stderr)\n            if ignored:\n                raise GitIgnoreError(cmd=e.cmd, msg=e.stderr,\n                                     code=e.code, stdout=e.stdout,\n                                     stderr=e.stderr,\n                                     paths=ignored.groups()[0].splitlines())\n            raise\n        return out, err\n\n# TODO: --------------------------------------------------------------------\n\n    def add_remote(self, name, url, options=None):\n        \"\"\"Register remote pointing to a url\n        \"\"\"\n        cmd = ['git', 'remote', 'add']\n        if options:\n            cmd += options\n        cmd += [name, url]\n\n        result = self._git_custom_command('', cmd)\n        self.config.reload()\n        return result\n\n    def remove_remote(self, name):\n        \"\"\"Remove existing remote\n        \"\"\"\n\n        # TODO: testing and error handling!\n        from .exceptions import RemoteNotAvailableError\n        try:\n            out, err = self._git_custom_command(\n                '', ['git', 'remote', 'remove', name])\n        except CommandError as e:\n            if 'fatal: No such remote' in e.stderr:\n                raise RemoteNotAvailableError(name,\n                                              cmd=\"git remote remove\",\n                                              msg=\"No such remote\",\n                                              stdout=out,\n                                              stderr=err)\n            else:\n                raise e\n\n        # TODO: config.reload necessary?\n        self.config.reload()\n        return\n\n    def update_remote(self, name=None, verbose=False):\n        \"\"\"\n        \"\"\"\n        options = [\"-v\"] if verbose else []\n        name = [name] if name else []\n        self._git_custom_command(\n            '', ['git', 'remote'] + name + ['update'] + options,\n            expect_stderr=True\n        )\n\n    # TODO: centralize all the c&p code in fetch, pull, push\n    # TODO: document **kwargs passed to gitpython\n    def fetch(self, remote=None, refspec=None, progress=None, all_=False,\n              **kwargs):\n        \"\"\"Fetches changes from a remote (or all_ remotes).\n\n        Parameters\n        ----------\n        remote: str\n          (optional) name of the remote to fetch from. If no remote is given and\n          `all_` is not set, the tracking branch is fetched.\n        refspec: str\n          (optional) refspec to fetch.\n        progress:\n          passed to gitpython. TODO: Figure it out, make consistent use of it\n          and document it.\n        all_: bool\n          fetch all_ remotes (and all_ of their branches).\n          Fails if `remote` was given.\n        kwargs:\n          passed to gitpython. TODO: Figure it out, make consistent use of it\n          and document it.\n\n        Returns\n        -------\n        list\n            FetchInfo objects of the items fetched from remote\n        \"\"\"\n        # TODO: options=> **kwargs):\n        # Note: Apparently there is no explicit (fetch --all) in gitpython,\n        #       but fetch is always bound to a certain remote instead.\n        #       Therefore implement it on our own:\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            if all_:\n                remotes_to_fetch = [\n                    self.repo.remote(r)\n                    for r in self.get_remotes(with_urls_only=True)\n                ]\n            else:\n                # No explicit remote to fetch.\n                # => get tracking branch:\n                tb_remote, refspec = self.get_tracking_branch()\n                if tb_remote is not None:\n                    remotes_to_fetch = [self.repo.remote(tb_remote)]\n                else:\n                    # No remote, no tracking branch\n                    # => fail\n                    raise ValueError(\"Neither a remote is specified to fetch \"\n                                     \"from nor a tracking branch is set up.\")\n        else:\n            remotes_to_fetch = [self.repo.remote(remote)]\n\n        fi_list = []\n        for rm in remotes_to_fetch:\n            fetch_url = \\\n                self.config.get('remote.%s.fetchurl' % rm.name,\n                                self.config.get('remote.%s.url' % rm.name,\n                                                None))\n            if fetch_url is None:\n                lgr.debug(\"Remote %s has no URL\", rm)\n                return []\n\n            if is_ssh(fetch_url):\n                ssh_manager.get_connection(fetch_url).open()\n                # TODO: with git <= 2.3 keep old mechanism:\n                #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                    fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)\n                    # TODO: progress +kwargs\n            else:\n                fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n\n        # TODO: fetch returns a list of FetchInfo instances. Make use of it.\n        return fi_list\n\n    def pull(self, remote=None, refspec=None, progress=None, **kwargs):\n        \"\"\"See fetch\n        \"\"\"\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            # No explicit remote to pull from.\n            # => get tracking branch:\n            tb_remote, refspec = self.get_tracking_branch()\n            if tb_remote is not None:\n                remote = self.repo.remote(tb_remote)\n            else:\n                # No remote, no tracking branch\n                # => fail\n                raise ValueError(\"No remote specified to pull from nor a \"\n                                 \"tracking branch is set up.\")\n\n        else:\n            remote = self.repo.remote(remote)\n\n        fetch_url = \\\n            remote.config_reader.get(\n                'fetchurl' if remote.config_reader.has_option('fetchurl')\n                else 'url')\n        if is_ssh(fetch_url):\n            ssh_manager.get_connection(fetch_url).open()\n            # TODO: with git <= 2.3 keep old mechanism:\n            #       with remote.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n            with remote.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                return remote.pull(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n        else:\n            return remote.pull(refspec=refspec, progress=progress, **kwargs)\n            # TODO: progress +kwargs\n\n    def push(self, remote=None, refspec=None, progress=None, all_remotes=False,\n             **kwargs):\n        \"\"\"Push to remote repository\n\n        Parameters\n        ----------\n        remote: str\n          name of the remote to push to\n        refspec: str\n          specify what to push\n        progress:\n          TODO\n        all_remotes: bool\n          if set to True push to all remotes. Conflicts with `remote` not being\n          None.\n        kwargs: dict\n          options to pass to `git push`\n\n        Returns\n        -------\n        list\n            PushInfo objects of the items pushed to remote\n        \"\"\"\n\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            if all_remotes:\n                remotes_to_push = self.repo.remotes\n            else:\n                # Nothing explicitly specified. Just call `git push` and let git\n                # decide what to do would be an option. But:\n                # - without knowing the remote and its URL we cannot provide\n                #   shared SSH connection\n                # - we lose ability to use GitPython's progress info and return\n                #   values\n                #   (the latter would be solvable:\n                #    Provide a Repo.push() method for GitPython, copying\n                #    Remote.push() for similar return value and progress\n                #    (also: fetch, pull)\n\n                # Do what git would do:\n                # 1. branch.*.remote for current branch or 'origin' as default\n                #    if config is missing\n                # 2. remote.*.push or push.default\n\n                # TODO: check out \"same procedure\" for fetch/pull\n\n                tb_remote, refspec = self.get_tracking_branch()\n                if tb_remote is None:\n                    tb_remote = 'origin'\n                remotes_to_push = [self.repo.remote(tb_remote)]\n                # use no refspec; let git find remote.*.push or push.default on\n                # its own\n\n        else:\n            if all_remotes:\n                lgr.warning(\"Option 'all_remotes' conflicts with specified \"\n                            \"remote '%s'. Option ignored.\")\n            remotes_to_push = [self.repo.remote(remote)]\n\n        pi_list = []\n        for rm in remotes_to_push:\n            push_url = \\\n                rm.config_reader.get('pushurl'\n                                     if rm.config_reader.has_option('pushurl')\n                                     else 'url')\n            if is_ssh(push_url):\n                ssh_manager.get_connection(push_url).open()\n                # TODO: with git <= 2.3 keep old mechanism:\n                #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                    pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)\n                    # TODO: progress +kwargs\n            else:\n                pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n        return pi_list\n\n    def get_remote_url(self, name, push=False):\n        \"\"\"Get the url of a remote.\n\n        Reads the configuration of remote `name` and returns its url or None,\n        if there is no url configured.\n\n        Parameters\n        ----------\n        name: str\n          name of the remote\n        push: bool\n          if True, get the pushurl instead of the fetch url.\n        \"\"\"\n\n        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')\n        return self.config.get(var, None)\n\n    def set_remote_url(self, name, url, push=False):\n        \"\"\"Set the URL a remote is pointing to\n\n        Sets the URL of the remote `name`. Requires the remote to already exist.\n\n        Parameters\n        ----------\n        name: str\n          name of the remote\n        url: str\n        push: bool\n          if True, set the push URL, otherwise the fetch URL\n        \"\"\"\n\n        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')\n        self.config.set(var, url, where='local', reload=True)\n\n    def get_branch_commits(self, branch=None, limit=None, stop=None, value=None):\n        \"\"\"Return GitPython's commits for the branch\n\n        Pretty much similar to what 'git log <branch>' does.\n        It is a generator which returns top commits first\n\n        Parameters\n        ----------\n        branch: str, optional\n          If not provided, assumes current branch\n        limit: None | 'left-only', optional\n          Limit which commits to report.  If None -- all commits (merged or not),\n          if 'left-only' -- only the commits from the left side of the tree upon\n          merges\n        stop: str, optional\n          hexsha of the commit at which stop reporting (matched one is not\n          reported either)\n        value: None | 'hexsha', optional\n          What to yield.  If None - entire commit object is yielded, if 'hexsha'\n          only its hexsha\n        \"\"\"\n\n        if not branch:\n            branch = self.get_active_branch()\n\n        try:\n            _branch = self.repo.branches[branch]\n        except IndexError:\n            raise MissingBranchError(self, branch,\n                                     [b.name for b in self.repo.branches])\n\n        fvalue = {None: lambda x: x, 'hexsha': lambda x: x.hexsha}[value]\n\n        if not limit:\n            def gen():\n                # traverse doesn't yield original commit\n                co = _branch.commit\n                yield co\n                for co_ in co.traverse():\n                    yield co_\n        elif limit == 'left-only':\n            # we need a custom implementation since couldn't figure out how to\n            # do with .traversal\n            def gen():\n                co = _branch.commit\n                while co:\n                    yield co\n                    co = co.parents[0] if co.parents else None\n        else:\n            raise ValueError(limit)\n\n        for c in gen():\n            if stop and c.hexsha == stop:\n                return\n            yield fvalue(c)\n\n    def checkout(self, name, options=None):\n        \"\"\"\n        \"\"\"\n        # TODO: May be check for the need of -b options herein?\n        cmd = ['git', 'checkout']\n        if options:\n            cmd += options\n        cmd += [str(name)]\n\n        self._git_custom_command('', cmd, expect_stderr=True)\n\n    # TODO: Before implementing annex merge, find usages and check for a needed\n    # change to call super().merge\n    def merge(self, name, options=None, msg=None, allow_unrelated=False, **kwargs):\n        if options is None:\n            options = []\n        if msg:\n            options = options + [\"-m\", msg]\n        if allow_unrelated and external_versions['cmd:git'] >= '2.9':\n            options += ['--allow-unrelated-histories']\n        self._git_custom_command(\n            '', ['git', 'merge'] + options + [name],\n            **kwargs\n        )\n\n    def remove_branch(self, branch):\n        self._git_custom_command(\n            '', ['git', 'branch', '-D', branch]\n        )\n\n    def ls_remote(self, remote, options=None):\n        if options is None:\n            options = []\n        self._git_custom_command(\n            '', ['git', 'ls-remote'] + options + [remote]\n        )\n        # TODO: Return values?\n\n    def is_dirty(self, index=True, working_tree=True, untracked_files=True,\n                 submodules=True, path=None):\n        \"\"\"Returns true if the repo is considered to be dirty\n\n        Parameters\n        ----------\n        index: bool\n          if True, consider changes to the index\n        working_tree: bool\n          if True, consider changes to the working tree\n        untracked_files: bool\n          if True, consider untracked files\n        submodules: bool\n          if True, consider submodules\n        path: str or list of str\n          path(s) to consider only\n        Returns\n        -------\n          bool\n        \"\"\"\n\n        return self.repo.is_dirty(index=index, working_tree=working_tree,\n                                  untracked_files=untracked_files,\n                                  submodules=submodules, path=path)\n\n    @property\n    def dirty(self):\n        return self.is_dirty()\n\n    @property\n    def untracked_files(self):\n        return self.repo.untracked_files\n\n    def gc(self, allow_background=False, auto=False):\n        \"\"\"Perform house keeping (garbage collection, repacking)\"\"\"\n        cmd_options = ['git']\n        if not allow_background:\n            cmd_options += ['-c', 'gc.autodetach=0']\n        cmd_options += ['gc', '--aggressive']\n        if auto:\n            cmd_options += ['--auto']\n        self._git_custom_command('', cmd_options)\n\n    def get_submodules(self, sorted_=True):\n        \"\"\"Return a list of git.Submodule instances for all submodules\"\"\"\n        # check whether we have anything in the repo. if not go home early\n        if not self.repo.head.is_valid():\n            return []\n        submodules = self.repo.submodules\n        if sorted_:\n            submodules = sorted(submodules, key=lambda x: x.path)\n        return submodules\n\n    def is_submodule_modified(self, name, options=[]):\n        \"\"\"Whether a submodule has new commits\n\n        Note: This is an adhoc method. It parses output of\n        'git submodule summary' and currently is not able to distinguish whether\n        or not this change is staged in `self` and whether this would be\n        reported 'added' or 'modified' by 'git status'.\n        Parsing isn't heavily tested yet.\n\n        Parameters\n        ----------\n        name: str\n          the submodule's name\n        options: list\n          options to pass to 'git submodule summary'\n        Returns\n        -------\n        bool\n          True if there are commits in the submodule, differing from\n          what is registered in `self`\n        --------\n        \"\"\"\n\n        out, err = self._git_custom_command('',\n                                            ['git', 'submodule', 'summary'] + \\\n                                            options + ['--', name])\n        return any([line.split()[1] == name\n                    for line in out.splitlines()\n                    if line and len(line.split()) > 1])\n\n    def add_submodule(self, path, name=None, url=None, branch=None):\n        \"\"\"Add a new submodule to the repository.\n\n        This will alter the index as well as the .gitmodules file, but will not\n        create a new commit.  If the submodule already exists, no matter if the\n        configuration differs from the one provided, the existing submodule\n        is considered as already added and no further action is performed.\n\n        Parameters\n        ----------\n        path : str\n          repository-relative path at which the submodule should be located, and\n          which will be created as required during the repository initialization.\n        name : str or None\n          name/identifier for the submodule. If `None`, the `path` will be used\n          as name.\n        url : str or None\n          git-clone compatible URL. If `None`, the repository is assumed to\n          exist, and the url of the first remote is taken instead. This is\n          useful if you want to make an existing repository a submodule of\n          another one.\n        branch : str or None\n          name of branch to be checked out in the submodule. The given branch\n          must exist in the remote repository, and will be checked out locally\n          as a tracking branch. If `None`, remote HEAD will be checked out.\n        \"\"\"\n        if name is None:\n            name = path\n        # XXX the following should do it, but GitPython will refuse to add a submodule\n        # unless you specify a URL that is configured as one of its remotes, or you\n        # specify no URL, but the repo has at least one remote.\n        # this is stupid, as for us it is valid to not have any remote, because we can\n        # still obtain the submodule from a future publication location, based on the\n        # parent\n        # gitpy.Submodule.add(self.repo, name, path, url=url, branch=branch)\n        # going git native instead\n        cmd = ['git', 'submodule', 'add', '--name', name]\n        if branch is not None:\n            cmd += ['-b', branch]\n        if url is None:\n            if not isabs(path):\n                # need to recode into a relative path \"URL\" in POSIX\n                # style, even on windows\n                url = posixpath.join(curdir, posix_relpath(path))\n            else:\n                url = path\n        cmd += [url, path]\n        self._git_custom_command('', cmd)\n        # TODO: return value\n\n    def deinit_submodule(self, path, **kwargs):\n        \"\"\"Deinit a submodule\n\n        Parameters\n        ----------\n        path: str\n            path to the submodule; relative to `self.path`\n        kwargs:\n            see `__init__`\n        \"\"\"\n\n        self._git_custom_command(path,\n                                 ['git', 'submodule', 'deinit'] +\n                                 to_options(**kwargs))\n        # TODO: return value\n\n    def update_submodule(self, path, mode='checkout', init=False):\n        \"\"\"Update a registered submodule.\n\n        This will make the submodule match what the superproject expects by\n        cloning missing submodules and updating the working tree of the\n        submodules. The \"updating\" can be done in several ways depending\n        on the value of submodule.<name>.update configuration variable, or\n        the `mode` argument.\n\n        Parameters\n        ----------\n        path : str\n          Identifies which submodule to operate on by it's repository-relative\n          path.\n        mode : {checkout, rebase, merge}\n          Update procedure to perform. 'checkout': the commit recorded in the\n          superproject will be checked out in the submodule on a detached HEAD;\n          'rebase': the current branch of the submodule will be rebased onto\n          the commit recorded in the superproject; 'merge': the commit recorded\n          in the superproject will be merged into the current branch in the\n          submodule.\n        init : bool\n          If True, initialize all submodules for which \"git submodule init\" has\n          not been called so far before updating.\n          Primarily provided for internal purposes and should not be used directly\n          since would result in not so annex-friendly .git symlinks/references\n          instead of full featured .git/ directories in the submodules\n        \"\"\"\n        cmd = ['git', 'submodule', 'update', '--%s' % mode]\n        if init:\n            cmd.append('--init')\n            subgitpath = opj(self.path, path, '.git')\n            if not exists(subgitpath):\n                # TODO:  wouldn't with --init we get all those symlink'ed .git/?\n                # At least let's warn\n                lgr.warning(\n                    \"Do not use update_submodule with init=True to avoid git creating \"\n                    \"symlinked .git/ directories in submodules\"\n                )\n            #  yoh: I thought I saw one recently but thought it was some kind of\n            #  an artifact from running submodule update --init manually at\n            #  some point, but looking at this code now I worry that it was not\n        self._git_custom_command(path, cmd)\n        # TODO: return value\n\n    def update_ref(self, ref, value, symbolic=False):\n        \"\"\"Update the object name stored in a ref \"safely\".\n\n        Just a shim for `git update-ref` call if not symbolic, and\n        `git symbolic-ref` if symbolic\n\n        Parameters\n        ----------\n        ref : str\n          Reference, such as `ref/heads/BRANCHNAME` or HEAD.\n        value : str\n          Value to update to, e.g. hexsha of a commit when updating for a\n          branch ref, or branch ref if updating HEAD\n        symbolic : None\n          To instruct if ref is symbolic, e.g. should be used in case of\n          ref=HEAD\n        \"\"\"\n        self._git_custom_command(\n            '',\n            ['git', 'symbolic-ref' if symbolic else 'update-ref', ref, value]\n        )\n\n    def tag(self, tag):\n        \"\"\"Assign a tag to current commit\n\n        Parameters\n        ----------\n        tag : str\n          Custom tag label.\n        \"\"\"\n        # TODO later to be extended with tagging particular commits and signing\n        # TODO: call in save.py complains about extensive logging. When does it\n        # happen in what way? Figure out, whether to just silence it or raise or\n        # whatever else.\n        self._git_custom_command(\n            '', ['git', 'tag', str(tag)]\n        )\n\n    def get_tags(self, output=None):\n        \"\"\"Get list of tags\n\n        Parameters\n        ----------\n        output : str, optional\n          If given, limit the return value to a list of values matching that\n          particular key of the tag properties.\n\n        Returns\n        -------\n        list\n          Each item is a dictionary with information on a tag. At present\n          this includes 'hexsha', and 'name', where the latter is the string\n          label of the tag, and the format the hexsha of the object the tag\n          is attched to. The list is sorted by commit date, with the most\n          recent commit being the last element.\n        \"\"\"\n        # TODO it would be straightforward to add more info and tweak the\n        # sorting\n        stdout, stderr = self._git_custom_command(\n            '',\n            ['git', 'tag', '--format=%(refname:strip=2)%00%(object)',\n             '--sort=*committerdate'])\n        fields = ('name', 'hexsha')\n        tags = [dict(zip(fields, line.split('\\0'))) for line in stdout.splitlines()]\n        if output:\n            return [t[output] for t in tags]\n        else:\n            return tags\n\n    def describe(self, **kwargs):\n        \"\"\" Quick and dirty implementation to call git-describe\n\n        Parameters:\n        -----------\n        kwargs:\n            transformed to cmdline options for git-describe;\n            see __init__ for description of the transformation\n        \"\"\"\n        # TODO: be more precise what failure to expect when and raise actual\n        # errors\n        try:\n            describe, outerr = self._git_custom_command(\n                [],\n                ['git', 'describe'] + to_options(**kwargs),\n                expect_fail=True)\n            return describe.strip()\n        # TODO: WTF \"catch everything\"?\n        except:\n            return None\n\n    def get_tracking_branch(self, branch=None):\n        \"\"\"Get the tracking branch for `branch` if there is any.\n\n        Parameters\n        ----------\n        branch: str\n            local branch to look up. If none is given, active branch is used.\n\n        Returns\n        -------\n        tuple\n            (remote or None, refspec or None) of the tracking branch\n        \"\"\"\n        if branch is None:\n            branch = self.get_active_branch()\n            if branch is None:\n                return None, None\n\n        track_remote = self.config.get('branch.{0}.remote'.format(branch), None)\n        track_branch = self.config.get('branch.{0}.merge'.format(branch), None)\n        return track_remote, track_branch\n\n    @property\n    def count_objects(self):\n        \"\"\"return dictionary with count, size(in KiB) information of git objects\n        \"\"\"\n\n        count_cmd = ['git', 'count-objects', '-v']\n        count_str, err = self._git_custom_command('', count_cmd)\n        count = {key: int(value)\n                 for key, value in [item.split(': ')\n                                    for item in count_str.split('\\n')\n                                    if len(item.split(': ')) == 2]}\n        return count\n\n    def get_missing_files(self):\n        \"\"\"Return a list of paths with missing files (and no staged deletion)\"\"\"\n        return [f.split('\\t')[1]\n                for f in self.repo.git.diff('--raw', '--name-status').split('\\n')\n                if f.split('\\t')[0] == 'D']\n\n    def get_deleted_files(self):\n        \"\"\"Return a list of paths with deleted files (staged deletion)\"\"\"\n        return [f.split('\\t')[1]\n                for f in self.repo.git.diff('--raw', '--name-status', '--staged').split('\\n')\n                if f.split('\\t')[0] == 'D']\n\n    def get_git_attributes(self):\n        \"\"\"Check git attribute for the current repository (not per-file support for now)\n\n        Parameters\n        ----------\n        all_: bool\n          Adds --all to git check-attr call\n\n        Returns\n        -------\n        dict:\n          attribute: value pairs\n        \"\"\"\n        out, err = self._git_custom_command([\".\"], [\"git\", \"check-attr\", \"--all\"])\n        assert not err, \"no stderr output is expected\"\n        out_split = [\n            # splitting by : would leave leading space(s)\n            [e.lstrip(' ') for e in l.split(':', 2)]\n            for l in out.split('\\n') if l\n        ]\n        assert all(o[0] == '.' for o in out_split)  # for paranoid\n        return dict(o[1:] for o in out_split)\n\n\n# TODO\n# remove submodule: nope, this is just deinit_submodule + remove\n# status?\n",
            "header": {
                "file": {
                    "path": "datalad/support/gitrepo.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "ae9a9a81b",
                "new_commit_hash": "9b0f14cbf",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/support/gitrepo.py"
            },
            "plus_file": {
                "path": "b/datalad/support/gitrepo.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 330,
                            "length": 12
                        },
                        "new_range": {
                            "start": 330,
                            "length": 18
                        },
                        "section": "def check_git_configured():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    Raises",
                                "type": "context"
                            },
                            {
                                "content": "    ------",
                                "type": "context"
                            },
                            {
                                "content": "    RuntimeError  if any of those two ariables are not set",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    Returns",
                                "type": "added"
                            },
                            {
                                "content": "    -------",
                                "type": "added"
                            },
                            {
                                "content": "    dict with user.name and user.email entries",
                                "type": "added"
                            },
                            {
                                "content": "    \"\"\"",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    check_runner = GitRunner()",
                                "type": "context"
                            },
                            {
                                "content": "    vals = {}",
                                "type": "added"
                            },
                            {
                                "content": "    for c in 'user.name', 'user.email':",
                                "type": "context"
                            },
                            {
                                "content": "        try:",
                                "type": "context"
                            },
                            {
                                "content": "            check_runner.run(['git', 'config', '--global', c])",
                                "type": "deleted"
                            },
                            {
                                "content": "            v, err = check_runner.run(['git', 'config', '--global', c])",
                                "type": "added"
                            },
                            {
                                "content": "            vals[c] = v.rstrip('\\n')",
                                "type": "added"
                            },
                            {
                                "content": "        except CommandError as exc:",
                                "type": "context"
                            },
                            {
                                "content": "            lgr.debug(\"Failed to verify that git is configured: %s\",",
                                "type": "context"
                            },
                            {
                                "content": "                      exc_str(exc))",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/support/gitrepo.py",
                            "type": "function",
                            "name": "check_git_configured",
                            "content": "def check_git_configured():\n    \"\"\"Do a check if git is configured (user.name and user.email are set)\n\n    Raises\n    ------\n    RuntimeError  if any of those two ariables are not set\n    \"\"\"\n\n    check_runner = GitRunner()\n    for c in 'user.name', 'user.email':\n        try:\n            check_runner.run(['git', 'config', '--global', c])\n        except CommandError as exc:\n            lgr.debug(\"Failed to verify that git is configured: %s\",\n                      exc_str(exc))\n            raise RuntimeError(\n                \"You must configure git first (set both user.name and \"\n                \"user.email) settings before using DataLad.\"\n            )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 327,
                            "end_lineno": 345,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 343,
                            "length": 6
                        },
                        "new_range": {
                            "start": 349,
                            "length": 7
                        },
                        "section": "def check_git_configured():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                \"You must configure git first (set both user.name and \"",
                                "type": "context"
                            },
                            {
                                "content": "                \"user.email) settings before using DataLad.\"",
                                "type": "context"
                            },
                            {
                                "content": "            )",
                                "type": "context"
                            },
                            {
                                "content": "    return vals",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def _remove_empty_items(list_):",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/support/gitrepo.py",
                            "type": "function",
                            "name": "check_git_configured",
                            "content": "def check_git_configured():\n    \"\"\"Do a check if git is configured (user.name and user.email are set)\n\n    Raises\n    ------\n    RuntimeError  if any of those two ariables are not set\n\n    Returns\n    -------\n    dict with user.name and user.email entries\n    \"\"\"\n\n    check_runner = GitRunner()\n    vals = {}\n    for c in 'user.name', 'user.email':\n        try:\n            v, err = check_runner.run(['git', 'config', '--global', c])\n            vals[c] = v.rstrip('\\n')\n        except CommandError as exc:\n            lgr.debug(\"Failed to verify that git is configured: %s\",\n                      exc_str(exc))\n            raise RuntimeError(\n                \"You must configure git first (set both user.name and \"\n                \"user.email) settings before using DataLad.\"\n            )\n    return vals",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 327,
                            "end_lineno": 352,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\nfrom .utils import (\n    chpwd,\n    get_dataset_root,\n    with_tree,\n\n    assert_raises,\n)\n\n\n#@with_tempfile(mkdir=True)\n# with_tempfile dereferences tempdir, so does not trigger the failure\n# on Yarik's laptop where TMPDIR=~/.tmp and ~/.tmp -> /tmp.\n# with_tree in turn just passes that ~/.tmp/ directory\n@with_tree(tree={})\ndef test_not_under_git(path):\n    from datalad.distribution.dataset import require_dataset\n    dsroot = get_dataset_root(path)\n    assert dsroot is None, \"There must be no dataset above tmp %s. Got: %s\" % (path, dsroot)\n    with chpwd(path):\n        # And require_dataset must puke also\n        assert_raises(\n            Exception,\n            require_dataset,\n            None, check_installed=True, purpose='test'\n        )",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\nfrom .utils import (\n    chpwd,\n    get_dataset_root,\n    with_tree,\n\n    assert_raises,\n    assert_equal,\n)\n\n\n#@with_tempfile(mkdir=True)\n# with_tempfile dereferences tempdir, so does not trigger the failure\n# on Yarik's laptop where TMPDIR=~/.tmp and ~/.tmp -> /tmp.\n# with_tree in turn just passes that ~/.tmp/ directory\n@with_tree(tree={})\ndef test_not_under_git(path):\n    from datalad.distribution.dataset import require_dataset\n    dsroot = get_dataset_root(path)\n    assert dsroot is None, \"There must be no dataset above tmp %s. Got: %s\" % (path, dsroot)\n    with chpwd(path):\n        # And require_dataset must puke also\n        assert_raises(\n            Exception,\n            require_dataset,\n            None, check_installed=True, purpose='test'\n        )\n\n\ndef test_git_config_fixture():\n    # in the setup_package we setup a new HOME with custom config\n    from datalad.support.gitrepo import check_git_configured\n    assert_equal(\n        check_git_configured(),\n        {\n            'user.name': 'DataLad Tester',\n            'user.email': 'test@example.com'\n         }\n    )",
            "header": {
                "file": {
                    "path": "datalad/tests/test_base.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "5a5cf4ace",
                "new_commit_hash": "206b00117",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/tests/test_base.py"
            },
            "plus_file": {
                "path": "b/datalad/tests/test_base.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 13,
                            "length": 6
                        },
                        "new_range": {
                            "start": 13,
                            "length": 7
                        },
                        "section": "from .utils import ("
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    with_tree,",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    assert_raises,",
                                "type": "context"
                            },
                            {
                                "content": "    assert_equal,",
                                "type": "added"
                            },
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/tests/test_base.py",
                            "type": "statement",
                            "name": "",
                            "content": "from .utils import (\n    chpwd,\n    get_dataset_root,\n    with_tree,\n\n    assert_raises,\n    assert_equal,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 10,
                            "end_lineno": 17,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 31,
                            "length": 4
                        },
                        "new_range": {
                            "start": 32,
                            "length": 16
                        },
                        "section": "def test_not_under_git(path):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            Exception,",
                                "type": "context"
                            },
                            {
                                "content": "            require_dataset,",
                                "type": "context"
                            },
                            {
                                "content": "            None, check_installed=True, purpose='test'",
                                "type": "context"
                            },
                            {
                                "content": "        )",
                                "type": "deleted"
                            },
                            {
                                "content": "No newline at end of file",
                                "type": "note"
                            },
                            {
                                "content": "        )",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "def test_git_config_fixture():",
                                "type": "added"
                            },
                            {
                                "content": "    # in the setup_package we setup a new HOME with custom config",
                                "type": "added"
                            },
                            {
                                "content": "    from datalad.support.gitrepo import check_git_configured",
                                "type": "added"
                            },
                            {
                                "content": "    assert_equal(",
                                "type": "added"
                            },
                            {
                                "content": "        check_git_configured(),",
                                "type": "added"
                            },
                            {
                                "content": "        {",
                                "type": "added"
                            },
                            {
                                "content": "            'user.name': 'DataLad Tester',",
                                "type": "added"
                            },
                            {
                                "content": "            'user.email': 'test@example.com'",
                                "type": "added"
                            },
                            {
                                "content": "         }",
                                "type": "added"
                            },
                            {
                                "content": "    )",
                                "type": "added"
                            },
                            {
                                "content": "No newline at end of file",
                                "type": "note"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/tests/test_base.py",
                            "type": "function",
                            "name": "test_not_under_git",
                            "content": "def test_not_under_git(path):\n    from datalad.distribution.dataset import require_dataset\n    dsroot = get_dataset_root(path)\n    assert dsroot is None, \"There must be no dataset above tmp %s. Got: %s\" % (path, dsroot)\n    with chpwd(path):\n        # And require_dataset must puke also\n        assert_raises(\n            Exception,\n            require_dataset,\n            None, check_installed=True, purpose='test'\n        )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 24,
                            "end_lineno": 34,
                            "parent": null
                        }
                    ],
                    "added_entities": [
                        {
                            "file_name": "datalad/tests/test_base.py",
                            "type": "function",
                            "name": "test_git_config_fixture",
                            "content": "def test_git_config_fixture():\n    # in the setup_package we setup a new HOME with custom config\n    from datalad.support.gitrepo import check_git_configured\n    assert_equal(\n        check_git_configured(),\n        {\n            'user.name': 'DataLad Tester',\n            'user.email': 'test@example.com'\n         }\n    )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 38,
                            "end_lineno": 47,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "7ad7e433751fa9e2d68169b1382b92ed7855c5c9^",
    "new_commit_hash": "7ad7e433751fa9e2d68169b1382b92ed7855c5c9",
    "commit_message": "ENH(TST): setup a fake HOME for running tests without being affected by user (absent) config etc",
    "commit_date": "2017-12-07T22:16:53-05:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "7ad7e433751fa9e2d68169b1382b92ed7855c5c9",
    "test_file_codes": [
        "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\nfrom datalad.tests.utils import (\n    chpwd,\n    get_dataset_root,\n    with_tree,\n\n    assert_raises,\n    assert_equal,\n)\n\n\n#@with_tempfile(mkdir=True)\n# with_tempfile dereferences tempdir, so does not trigger the failure\n# on Yarik's laptop where TMPDIR=~/.tmp and ~/.tmp -> /tmp.\n# with_tree in turn just passes that ~/.tmp/ directory\n@with_tree(tree={})\ndef test_not_under_git(path):\n    from datalad.distribution.dataset import require_dataset\n    dsroot = get_dataset_root(path)\n    assert dsroot is None, \"There must be no dataset above tmp %s. Got: %s\" % (path, dsroot)\n    with chpwd(path):\n        # And require_dataset must puke also\n        assert_raises(\n            Exception,\n            require_dataset,\n            None, check_installed=True, purpose='test'\n        )\n\n\ndef test_git_config_fixture():\n    # in the setup_package we setup a new HOME with custom config\n    from datalad.support.gitrepo import check_git_configured\n    assert_equal(\n        check_git_configured(),\n        {\n            'user.name': 'DataLad Tester',\n            'user.email': 'test@example.com'\n         }\n    )",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\n❌ Verification failed\nPython 3.9 installation failed, trying Python 3.7...\nVerifying installation...\nDatalad version: 0.9.1.dev138\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.7\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 20ms\nInstalled 25 packages in 5.10s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpKjxMr3/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpKjxMr3/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpKjxMr3/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpKjxMr3/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpKjxMr3/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 48, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmp2EQ57C/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmp2EQ57C/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmp2EQ57C/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmp2EQ57C/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmp2EQ57C/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 48, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_7ad7e433751fa9e2d68169b1382b92ed7855c5c9/datalad/__init__.py\", line 17, in <module>\n    from datalad.cmd import GitRunner\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_7ad7e433751fa9e2d68169b1382b92ed7855c5c9/datalad/cmd.py\", line 34, in <module>\n    from .utils import on_windows\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_7ad7e433751fa9e2d68169b1382b92ed7855c5c9/datalad/utils.py\", line 24, in <module>\n    import wrapt\nModuleNotFoundError: No module named 'wrapt'\nUsing CPython 3.7.9\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 26 packages in 329ms\nInstalled 26 packages in 4.80s\n + bids==0.0\n + coverage==7.2.7\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + fasteners==0.19\n + grabbit==0.2.6\n + importlib-metadata==6.7.0\n + iniconfig==2.0.0\n + nibabel==4.0.2\n + num2words==0.5.14\n + numpy==1.21.6\n + packaging==24.0\n + pandas==1.3.5\n + patsy==1.0.1\n + pluggy==1.2.0\n + pybids==0.6.5\n + pytest==7.4.4\n + pytest-cov==4.1.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.7.3\n + setuptools==68.0.0\n + six==1.17.0\n + tomli==2.0.1\n + typing-extensions==4.7.1\n + zipp==3.15.0\nResolved 77 packages in 2.48s\nPrepared 1 package in 1.19s\nInstalled 71 packages in 54ms\n + appdirs==1.4.4\n + attrs==24.2.0\n + automat==22.10.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + cachetools==5.5.0\n + certifi==2024.12.14\n + cffi==1.15.1\n + charset-normalizer==3.4.1\n + constantly==15.1.0\n + cryptography==44.0.0\n + cssselect==1.2.0\n + datalad==0.9.1.dev1 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_7ad7e433751fa9e2d68169b1382b92ed7855c5c9)\n + deprecated==1.2.15\n + filelock==3.12.2\n + frozendict==2.4.6\n + gitdb==4.0.12\n + gitpython==3.1.44\n + httpretty==1.1.4\n + humanize==4.6.0\n + hyperlink==21.0.0\n + idna==3.10\n + importlib-resources==5.12.0\n + incremental==22.10.0\n + iso8601==2.1.0\n + itemadapter==0.8.0\n + itemloaders==1.1.0\n + jaraco-classes==3.2.3\n + jeepney==0.8.0\n + jmespath==1.0.1\n + jsmin==3.0.1\n + keyring==24.1.1\n + keyrings-alt==4.2.0\n + lxml==5.3.0\n + mock==5.1.0\n + more-itertools==9.1.0\n + msgpack-python==0.5.6\n + multidict==6.0.5\n + nose==1.3.7\n + parsel==1.8.1\n + patool==1.12\n + protego==0.3.0\n + pyasn1==0.5.1\n + pyasn1-modules==0.3.0\n + pycparser==2.21\n + pydispatcher==2.0.7\n + pygithub==2.3.0\n + pyjwt==2.8.0\n + pyld==2.0.4\n + pynacl==1.5.0\n + pyopenssl==24.3.0\n + pyyaml==6.0.1\n + queuelib==1.6.2\n + requests==2.31.0\n + requests-file==2.1.0\n + requests-ftp==0.3.1\n + scrapy==2.9.0\n + secretstorage==3.3.3\n + service-identity==21.1.0\n + simplejson==3.19.3\n + smmap==5.0.2\n + soupsieve==2.4.1\n + tldextract==4.0.0\n + tqdm==4.67.1\n + twisted==23.8.0\n + urllib3==2.0.7\n + vcrpy==4.3.0\n + w3lib==2.1.2\n + wrapt==1.16.0\n + yarl==1.9.4\n + zope-interface==6.4.post2\nResolved 99 packages in 128ms\nInstalled 19 packages in 53ms\n + alabaster==0.7.13\n + babel==2.14.0\n + docutils==0.19\n + imagesize==1.4.1\n + jinja2==3.1.5\n + markupsafe==2.1.5\n + nose-timer==1.0.1\n + pygments==2.17.2\n + pypandoc==1.14\n + snowballstemmer==2.2.0\n + sphinx==5.3.0\n + sphinx-rtd-theme==2.0.0\n + sphinxcontrib-applehelp==1.0.2\n + sphinxcontrib-devhelp==1.0.2\n + sphinxcontrib-htmlhelp==2.0.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==1.0.3\n + sphinxcontrib-serializinghtml==1.1.5\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_7ad7e433751fa9e2d68169b1382b92ed7855c5c9\nplugins: cov-4.1.0\ncollected 2 items\n\nr2e_tests/test_1.py F.                                                   [100%]\n\n=================================== FAILURES ===================================\n______________________________ test_not_under_git ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexql7qe6ry'}\ntkwargs_ = {'prefix': 'datalad_temp_tree_test_not_under_git'}\nd = '/tmp/datalad_temp_tree_test_not_under_git0xko9im4'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_not_under_git() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:489: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_git_config_fixture\nFAILED r2e_tests/test_1.py::test_not_under_git - TypeError: test_not_under_gi...\n========================= 1 failed, 1 passed in 0.77s ==========================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_7ad7e433751fa9e2d68169b1382b92ed7855c5c9\nplugins: cov-4.1.0\ncollected 2 items\n\nr2e_tests/test_1.py FF                                                   [100%]\n\n=================================== FAILURES ===================================\n______________________________ test_not_under_git ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex40uljt3g'}\ntkwargs_ = {'prefix': 'datalad_temp_tree_test_not_under_git'}\nd = '/tmp/datalad_temp_tree_test_not_under_githhz6bxry'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_not_under_git() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:489: TypeError\n___________________________ test_git_config_fixture ____________________________\n\n    def test_git_config_fixture():\n        # in the setup_package we setup a new HOME with custom config\n        from datalad.support.gitrepo import check_git_configured\n        assert_equal(\n            check_git_configured(),\n            {\n                'user.name': 'DataLad Tester',\n>               'user.email': 'test@example.com'\n             }\n        )\n\nr2e_tests/test_1.py:45: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/unittest/case.py:852: in assertEqual\n    assertion_func(first, second, msg=msg)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = <nose.tools.trivial.Dummy testMethod=nop>, first = None\nsecond = {'user.email': 'test@example.com', 'user.name': 'DataLad Tester'}\nmsg = \"None != {'user.name': 'DataLad Tester', 'user.email': 'test@example.com'}\"\n\n    def _baseAssertEqual(self, first, second, msg=None):\n        \"\"\"The default assertEqual implementation, not type specific.\"\"\"\n        if not first == second:\n            standardMsg = '%s != %s' % _common_shorten_repr(first, second)\n            msg = self._formatMessage(msg, standardMsg)\n>           raise self.failureException(msg)\nE           AssertionError: None != {'user.name': 'DataLad Tester', 'user.email': 'test@example.com'}\n\n../../../../.local/share/uv/python/cpython-3.7.9-linux-x86_64-gnu/lib/python3.7/unittest/case.py:845: AssertionError\n=========================== short test summary info ============================\nFAILED r2e_tests/test_1.py::test_not_under_git - TypeError: test_not_under_gi...\nFAILED r2e_tests/test_1.py::test_git_config_fixture - AssertionError: None !=...\n============================== 2 failed in 0.46s ===============================\n",
    "old_commit_res_stderr": ""
}
datalad/__init__.py datalad/support/gitrepo.py datalad/tests/test_base.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title**: Git Configuration Retrieval Fails When HOME Environment Variable Is Unset

**Description**:
When the `HOME` environment variable is not set, attempts to retrieve Git user configuration (`user.name` and `user.email`) fail, causing the configuration check to return `None` instead of the expected values.

For example, running the following code without a `HOME` environment variable:

```python
from datalad.support.gitrepo import check_git_configured

config = check_git_configured()
print(config)
```

**Expected Behavior**:
The `check_git_configured` function should return a dictionary with `'user.name'` and `'user.email'` entries set to predefined test values, enabling tests to run without relying on the user's actual Git configuration.

**Actual Behavior**:
The function returns `None` instead of the expected configuration dictionary, leading to failures when tests assert the presence of Git user settings.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/support/gitrepo.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface to Git via GitPython

For further information on GitPython see http://gitpython.readthedocs.org/

"""

import logging
import re
import shlex
import os
from os import linesep
from os.path import join as opj
from os.path import exists
from os.path import normpath
from os.path import isabs
from os.path import commonprefix
from os.path import relpath
from os.path import realpath
from os.path import dirname
from os.path import basename
from os.path import curdir
from os.path import pardir
from os.path import sep
import posixpath
from weakref import WeakValueDictionary


from six import string_types
from six import add_metaclass
from functools import wraps
import git as gitpy
from git.exc import GitCommandError
from git.exc import NoSuchPathError
from git.exc import InvalidGitRepositoryError
from git.objects.blob import Blob

from datalad import ssh_manager
from datalad.cmd import GitRunner
from datalad.consts import GIT_SSH_COMMAND
from datalad.dochelpers import exc_str
from datalad.config import ConfigManager
from datalad.utils import assure_list
from datalad.utils import optional_args
from datalad.utils import on_windows
from datalad.utils import getpwd
from datalad.utils import updated
from datalad.utils import posix_relpath


# imports from same module:
from .external_versions import external_versions
from .exceptions import CommandError
from .exceptions import DeprecatedError
from .exceptions import FileNotInRepositoryError
from .exceptions import MissingBranchError
from .network import RI
from .network import is_ssh
from .repo import Flyweight
from .repo import RepoInterface

# shortcuts
_curdirsep = curdir + sep
_pardirsep = pardir + sep


lgr = logging.getLogger('datalad.gitrepo')

# Override default GitPython's DB backend to talk directly to git so it doesn't
# interfere with possible operations performed by gc/repack
default_git_odbt = gitpy.GitCmdObjectDB

# TODO: Figure out how GIT_PYTHON_TRACE ('full') is supposed to be used.
# Didn't work as expected on a first try. Probably there is a neatier way to
# log Exceptions from git commands.


# TODO: ignore leading and/or trailing underscore to allow for
# python-reserved words
@optional_args
def kwargs_to_options(func, split_single_char_options=True,
                      target_kw='options'):
    """Decorator to provide convenient way to pass options to command calls.

    Parameters
    ----------
    func: Callable
        function to decorate
    split_single_char_options: bool
        whether or not to split key and value of single char keyword arguments
        into two subsequent entries of the list
    target_kw: str
        keyword argument to pass the generated list of cmdline arguments to

    Returns
    -------
    Callable
    """

    # TODO: don't overwrite options, but join

    @wraps(func)
    def newfunc(self, *args, **kwargs):
        t_kwargs = dict()
        t_kwargs[target_kw] = \
            gitpy.Git().transform_kwargs(
                split_single_char_options=split_single_char_options,
                **kwargs)
        return func(self, *args, **t_kwargs)
    return newfunc


def to_options(**kwargs):
    """Transform keyword arguments into a list of cmdline options

    Parameters
    ----------
    split_single_char_options: bool

    kwargs:

    Returns
    -------
    list
    """
    # TODO: borrow_docs!

    return gitpy.Git().transform_kwargs(**kwargs)


def _normalize_path(base_dir, path):
    """Helper to check paths passed to methods of this class.

    Checks whether `path` is beneath `base_dir` and normalizes it.
    Additionally paths are converted into relative paths with respect to
    `base_dir`, considering PWD in case of relative paths. This
    is intended to be used in repository classes, which means that
    `base_dir` usually will be the repository's base directory.

    Parameters
    ----------
    path: str
        path to be normalized
    base_dir: str
        directory to serve as base to normalized, relative paths

    Returns
    -------
    str:
        path, that is a relative path with respect to `base_dir`
    """
    if not path:
        return path

    base_dir = realpath(base_dir)  # realpath OK
    # path = normpath(path)
    # Note: disabled normpath, because it may break paths containing symlinks;
    # But we don't want to realpath relative paths, in case cwd isn't the
    # correct base.

    if isabs(path):
        # path might already be a symlink pointing to annex etc,
        # so realpath only its directory, to get "inline" with
        # realpath(base_dir) above
        path = opj(realpath(dirname(path)), basename(path))  # realpath OK
    # Executive decision was made to not do this kind of magic!
    #
    # elif commonprefix([realpath(getpwd()), base_dir]) == base_dir:
    #     # If we are inside repository, rebuilt relative paths.
    #     path = opj(realpath(getpwd()), path)
    #
    # BUT with relative curdir/pardir start it would assume relative to curdir
    #
    elif path.startswith(_curdirsep) or path.startswith(_pardirsep):
        path = normpath(opj(realpath(getpwd()), path))  # realpath OK
    else:
        # We were called from outside the repo. Therefore relative paths
        # are interpreted as being relative to self.path already.
        return path

    if commonprefix([path, base_dir]) != base_dir:
        raise FileNotInRepositoryError(msg="Path outside repository: %s"
                                           % path, filename=path)

    return relpath(path, start=base_dir)


@optional_args
def normalize_path(func):
    """Decorator to provide unified path conversion for a single file

    Unlike normalize_paths, intended to be used for functions dealing with a
    single filename at a time

    Note
    ----
    This is intended to be used within the repository classes and therefore
    returns a class method!

    The decorated function is expected to take a path at
    first positional argument (after 'self'). Additionally the class `func`
    is a member of, is expected to have an attribute 'path'.
    """

    @wraps(func)
    def newfunc(self, file_, *args, **kwargs):
        file_new = _normalize_path(self.path, file_)
        return func(self, file_new, *args, **kwargs)

    return newfunc


@optional_args
def normalize_paths(func, match_return_type=True, map_filenames_back=False,
                    serialize=False):
    """Decorator to provide unified path conversions.

    Note
    ----
    This is intended to be used within the repository classes and therefore
    returns a class method!

    The decorated function is expected to take a path or a list of paths at
    first positional argument (after 'self'). Additionally the class `func`
    is a member of, is expected to have an attribute 'path'.

    Accepts either a list of paths or a single path in a str. Passes a list
    to decorated function either way, but would return based on the value of
    match_return_type and possibly input argument.

    If a call to the wrapped function includes normalize_path and it is False
    no normalization happens for that function call (used for calls to wrapped
    functions within wrapped functions, while possible CWD is within a
    repository)

    Parameters
    ----------
    match_return_type : bool, optional
      If True, and a single string was passed in, it would return the first
      element of the output (after verifying that it is a list of length 1).
      It makes easier to work with single files input.
    map_filenames_back : bool, optional
      If True and returned value is a dictionary, it assumes to carry entries
      one per file, and then filenames are mapped back to as provided from the
      normalized (from the root of the repo) paths
    serialize : bool, optional
      Loop through files giving only a single one to the function one at a time.
      This allows to simplify implementation and interface to annex commands
      which do not take multiple args in the same call (e.g. checkpresentkey)
    """

    @wraps(func)
    def newfunc(self, files, *args, **kwargs):

        normalize = _normalize_path if kwargs.pop('normalize_paths', True) \
            else lambda rpath, filepath: filepath

        if files:
            if isinstance(files, string_types) or not files:
                files_new = [normalize(self.path, files)]
                single_file = True
            elif isinstance(files, list):
                files_new = [normalize(self.path, path) for path in files]
                single_file = False
            else:
                raise ValueError("_files_decorator: Don't know how to handle "
                                 "instance of %s." % type(files))
        else:
            single_file = None
            files_new = []

        if map_filenames_back:
            def remap_filenames(out):
                """Helper to map files back to non-normalized paths"""
                if isinstance(out, dict):
                    assert(len(out) == len(files_new))
                    files_ = [files] if single_file else files
                    mapped = out.__class__()
                    for fin, fout in zip(files_, files_new):
                        mapped[fin] = out[fout]
                    return mapped
                else:
                    return out
        else:
            remap_filenames = lambda x: x

        if serialize:  # and not single_file:
            result = [
                func(self, f, *args, **kwargs)
                for f in files_new
            ]
        else:
            result = func(self, files_new, *args, **kwargs)

        if single_file is None:
            # no files were provided, nothing we can do really
            return result
        elif (result is None) or not match_return_type or not single_file:
            # If function doesn't return anything or no denormalization
            # was requested or it was not a single file
            return remap_filenames(result)
        elif single_file:
            if len(result) != 1:
                # Magic doesn't apply
                return remap_filenames(result)
            elif isinstance(result, (list, tuple)):
                return result[0]
            elif isinstance(result, dict) and tuple(result)[0] == files_new[0]:
                # assume that returned dictionary has files as keys.
                return tuple(result.values())[0]
            else:
                # no magic can apply
                return remap_filenames(result)
        else:
            return RuntimeError("should have not got here... check logic")

    return newfunc


def check_git_configured():
    """Do a check if git is configured (user.name and user.email are set)

    Raises
    ------
    RuntimeError  if any of those two ariables are not set
    """

    check_runner = GitRunner()
    for c in 'user.name', 'user.email':
        try:
            check_runner.run(['git', 'config', '--global', c])
        except CommandError as exc:
            lgr.debug("Failed to verify that git is configured: %s",
                      exc_str(exc))
            raise RuntimeError(
                "You must configure git first (set both user.name and "
                "user.email) settings before using DataLad."
            )


def _remove_empty_items(list_):
    """Remove empty entries from list

    This is needed, since some functions of GitPython may convert
    an empty entry to '.', when used with a list of paths.

    Parameter:
    ----------
    list_: list of str

    Returns
    -------
    list of str
    """
    if not isinstance(list_, list):
        lgr.warning(
            "_remove_empty_items() called with non-list type: %s" % type(list_))
        return list_
    return [file_ for file_ in list_ if file_]


def Repo(*args, **kwargs):
    """Factory method around gitpy.Repo to consistently initiate with different
    backend
    """
    # TODO: This probably doesn't work as intended (or at least not as
    #       consistently as intended). gitpy.Repo could be instantiated by
    #       classmethods Repo.init or Repo.clone_from. In these cases 'odbt'
    #       would be needed as a paramter to these methods instead of the
    #       constructor.
    if 'odbt' not in kwargs:
        kwargs['odbt'] = default_git_odbt
    return gitpy.Repo(*args, **kwargs)


def split_remote_branch(branch):
    """Splits a remote branch's name into the name of the remote and the name
    of the branch.

    Parameters
    ----------
    branch: str
      the remote branch's name to split

    Returns
    -------
    list of str
    """
    assert '/' in branch, \
        "remote branch %s must have had a /" % branch
    assert not branch.endswith('/'), \
        "branch name with trailing / is invalid. (%s)" % branch
    return branch.split('/', 1)


@add_metaclass(Flyweight)
class GitRepo(RepoInterface):
    """Representation of a git repository

    """

    # We use our sshrun helper
    GIT_SSH_ENV = {'GIT_SSH_COMMAND': GIT_SSH_COMMAND}

    # Just a non-functional example:
    # must be implemented, since abstract in RepoInterface:
    def sth_like_file_has_content(self):
        return "Yes, if it's in the index"

    # We must check git config to have name and email set, but
    # should do it once
    _config_checked = False

    # Begin Flyweight:

    _unique_instances = WeakValueDictionary()

    @classmethod
    def _flyweight_id_from_args(cls, *args, **kwargs):

        if args:
            # to a certain degree we need to simulate an actual call to __init__
            # and make sure, passed arguments are fitting:
            # TODO: Figure out, whether there is a cleaner way to do this in a
            # generic fashion
            assert('path' not in kwargs)
            path = args[0]
            args = args[1:]
        elif 'path' in kwargs:
            path = kwargs.pop('path')
        else:
            raise TypeError("__init__() requires argument `path`")

        if path is None:
            raise AttributeError

        # Sanity check for argument `path`:
        # raise if we cannot deal with `path` at all or
        # if it is not a local thing:
        path = RI(path).localpath
        # resolve symlinks to make sure we have exactly one instance per
        # physical repository at a time
        path = realpath(path)
        kwargs['path'] = path
        return path, args, kwargs

    @classmethod
    def _flyweight_invalid(cls, id_):
        return not cls.is_valid_repo(id_)

    @classmethod
    def _flyweight_reject(cls, id_, *args, **kwargs):
        # TODO:
        # This is a temporary approach. See PR # ...
        # create = kwargs.pop('create', None)
        # kwargs.pop('path', None)
        # if create and kwargs:
        #     # we have `create` plus options other than `path`
        #     return "Call to {0}() with args {1} and kwargs {2} conflicts " \
        #            "with existing instance {3}." \
        #            "This is likely to be caused by inconsistent logic in " \
        #            "your code." \
        #            "".format(cls, args, kwargs, cls._unique_instances[id_])
        pass

    # End Flyweight

    def __init__(self, path, url=None, runner=None, create=True,
                 git_opts=None, repo=None, **kwargs):
        """Creates representation of git repository at `path`.

        Can also be used to create a git repository at `path`.

        Parameters
        ----------
        path: str
          path to the git repository; In case it's not an absolute path,
          it's relative to PWD
        url: str, optional
          DEPRECATED -- use .clone() class method
          url to the to-be-cloned repository. Requires a valid git url
          according to:
          http://www.kernel.org/pub/software/scm/git/docs/git-clone.html#URLS .
        create: bool, optional
          if true, creates a git repository at `path` if there is none. Also
          creates `path`, if it doesn't exist.
          If set to false, an exception is raised in case `path` doesn't exist
          or doesn't contain a git repository.
        repo: git.Repo, optional
          GitPython's Repo instance to (re)use if provided
        kwargs:
          keyword arguments serving as additional options to the git-init
          command. Therefore, it makes sense only if called with `create`.

          Generally, this way of passing options to the git executable is
          (or will be) used a lot in this class. It's a transformation of
          python-style keyword arguments (or a `dict`) to command line arguments,
          provided by GitPython.

          A single character keyword will be prefixed by '-', multiple characters
          by '--'. An underscore in the keyword becomes a dash. The value of the
          keyword argument is used as the value for the corresponding command
          line argument. Assigning a boolean creates a flag.

          Examples:
          no_commit=True => --no-commit
          C='/my/path'   => -C /my/path

        """

        if url is not None:
            raise DeprecatedError(
                new=".clone() class method",
                version="0.5.0",
                msg="RF: url passed to init()"
            )

        # So that we "share" control paths with git/git-annex
        if ssh_manager:
            ssh_manager.assure_initialized()

        if not GitRepo._config_checked:
            check_git_configured()
            GitRepo._config_checked = True

        self.realpath = realpath(path)
        # note: we may also want to distinguish between a path to the worktree
        # and the actual repository

        # Disable automatic garbage and autopacking
        self._GIT_COMMON_OPTIONS = ['-c', 'receive.autogc=0', '-c', 'gc.auto=0']
        # actually no need with default GitPython db backend not in memory
        # default_git_odbt but still allows for faster testing etc.
        # May be eventually we would make it switchable _GIT_COMMON_OPTIONS = []

        if git_opts is None:
            git_opts = {}
        if kwargs:
            git_opts.update(kwargs)

        self.path = path
        self.cmd_call_wrapper = runner or GitRunner(cwd=self.path)
        self._repo = repo
        self._cfg = None

        _valid_repo = GitRepo.is_valid_repo(path)
        if create and not _valid_repo:
            if repo is not None:
                # `repo` passed with `create`, which doesn't make sense
                raise TypeError("argument 'repo' must not be used with 'create'")

            try:
                lgr.debug(
                    "Initialize empty Git repository at '%s'%s",
                    path,
                    ' %s' % git_opts if git_opts else '')
                self._repo = self.cmd_call_wrapper(gitpy.Repo.init, path,
                                                   mkdir=True,
                                                   odbt=default_git_odbt,
                                                   **git_opts)
            except GitCommandError as e:
                lgr.error(exc_str(e))
                raise
        else:
            # Note: We used to call gitpy.Repo(path) here, which potentially
            # raised NoSuchPathError or InvalidGitRepositoryError. This is
            # used by callers of GitRepo.__init__() to detect whether we have a
            # valid repo at `path`. Now, with switching to lazy loading property
            # `repo`, we detect those cases without instantiating a
            # gitpy.Repo().

            if not exists(path):
                raise NoSuchPathError(path)
            if not _valid_repo:
                raise InvalidGitRepositoryError(path)

        # inject git options into GitPython's git call wrapper:
        # Note: `None` currently can happen, when Runner's protocol prevents
        # calls above from being actually executed (DryRunProtocol)
        if self._repo is not None:
            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS

        # with DryRunProtocol path might still not exist
        if exists(self.realpath):
            self.inode = os.stat(self.realpath).st_ino
        else:
            self.inode = None

    @property
    def repo(self):
        # with DryRunProtocol path not exist
        if exists(self.realpath):
            inode = os.stat(self.realpath).st_ino
        else:
            inode = None
        if self.inode != inode:
            # reset background processes invoked by GitPython:
            self._repo.git.clear_cache()
            self.inode = inode

        if self._repo is None:
            # Note, that this may raise GitCommandError, NoSuchPathError,
            # InvalidGitRepositoryError:
            self._repo = self.cmd_call_wrapper(Repo, self.path)
            lgr.log(8, "Using existing Git repository at %s", self.path)

        # inject git options into GitPython's git call wrapper:
        # Note: `None` currently can happen, when Runner's protocol prevents
        # call of Repo(path) above from being actually executed (DryRunProtocol)
        if self._repo is not None:
            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS

        return self._repo

    @classmethod
    def clone(cls, url, path, *args, **kwargs):
        """Clone url into path

        Provides workarounds for known issues (e.g.
        https://github.com/datalad/datalad/issues/785)

        Parameters
        ----------
        url : str
        path : str
        expect_fail : bool
          Either expect that command might fail, so error should be logged then
          at DEBUG level instead of ERROR
        """

        if 'repo' in kwargs:
            raise TypeError("argument 'repo' conflicts with cloning")
            # TODO: what about 'create'?

        expect_fail = kwargs.pop('expect_fail', False)
        # fail early on non-empty target:
        from os import listdir
        if exists(path) and listdir(path):
            # simulate actual GitCommandError:
            lgr.warning("destination path '%s' already exists and is not an "
                        "empty directory." % path)
            raise GitCommandError(
                ['git', 'clone', '-v', url, path],
                128,
                "fatal: destination path '%s' already exists and is not an "
                "empty directory." % path)
        else:
            # protect against cloning into existing and obviously dangling
            # instance for that location
            try:
                del cls._unique_instances[path]
            except KeyError:
                # didn't exist - all fine
                pass

        # try to get a local path from `url`:
        try:
            if not isinstance(url, RI):
                url = RI(url).localpath
            else:
                url = url.localpath
        except ValueError:
            pass

        if is_ssh(url):
            ssh_manager.get_connection(url).open()
            # TODO: with git <= 2.3 keep old mechanism:
            #       with rm.repo.git.custom_environment(GIT_SSH="wrapper_script"):
            env = GitRepo.GIT_SSH_ENV
        else:
            env = None
        ntries = 5  # 3 is not enough for robust workaround
        for trial in range(ntries):
            try:
                lgr.debug("Git clone from {0} to {1}".format(url, path))
                repo = gitpy.Repo.clone_from(url, path, env=env,
                                             odbt=default_git_odbt)
                # Note/TODO: signature for clone from:
                # (url, to_path, progress=None, env=None, **kwargs)

                lgr.debug("Git clone completed")
                break
            except GitCommandError as e:
                # log here but let caller decide what to do
                e_str = exc_str(e)
                # see https://github.com/datalad/datalad/issues/785
                if re.search("Request for .*aborted.*Unable to find", str(e),
                             re.DOTALL) \
                        and trial < ntries - 1:
                    lgr.info(
                        "Hit a known issue with Git (see GH#785). Trial #%d, "
                        "retrying",
                        trial)
                    continue
                    (lgr.debug if expect_fail else lgr.error)(e_str)
                raise
            except ValueError as e:
                if gitpy.__version__ == '1.0.2' \
                        and "I/O operation on closed file" in str(e):
                    # bug https://github.com/gitpython-developers/GitPython
                    # /issues/383
                    raise GitCommandError(
                        "clone has failed, telling ya",
                        999,  # good number
                        stdout="%s already exists" if exists(path) else "")
                raise  # reraise original

        gr = cls(path, *args, repo=repo, **kwargs)
        return gr

    def __del__(self):
        # unbind possibly bound ConfigManager, to prevent all kinds of weird
        # stalls etc
        self._cfg = None
        # Make sure to flush pending changes, especially close batch processes
        # (internal `git cat-file --batch` by GitPython)
        try:
            if hasattr(self, 'repo') and exists(self.path) \
                    and self.repo is not None:
                # gc might be late, so the (temporary)
                # repo doesn't exist on FS anymore

                self.repo.git.clear_cache()
                if exists(opj(self.path, '.git')):  # don't try to write otherwise
                    self.repo.index.write()
        except InvalidGitRepositoryError:
            # might have being removed and no longer valid
            pass

    def __repr__(self):
        return "<GitRepo path=%s (%s)>" % (self.path, type(self))

    def __eq__(self, obj):
        """Decides whether or not two instances of this class are equal.

        This is done by comparing the base repository path.
        """
        return self.realpath == obj.realpath

    @classmethod
    def is_valid_repo(cls, path):
        """Returns if a given path points to a git repository"""
        return exists(opj(path, '.git'))

    @property
    def config(self):
        """Get an instance of the parser for the persistent repository
        configuration.

        Note: This allows to also read/write .datalad/config,
        not just .git/config

        Returns
        -------
        ConfigManager
        """
        if self._cfg is None:
            # associate with this dataset and read the entire config hierarchy
            self._cfg = ConfigManager(dataset=self, dataset_only=False)
        return self._cfg

    def is_with_annex(self, only_remote=False):
        """Return True if GitRepo (assumed) at the path has remotes with git-annex branch

        Parameters
        ----------
        only_remote: bool, optional
            Check only remote (no local branches) for having git-annex branch
        """
        return any((b.endswith('/git-annex') or
                    'annex/direct' in b
                    for b in self.get_remote_branches())) or \
            ((not only_remote) and
             any((b == 'git-annex' or 'annex/direct' in b
                  for b in self.get_branches())))

    @classmethod
    def get_toppath(cls, path, follow_up=True, git_options=None):
        """Return top-level of a repository given the path.

        Parameters
        -----------
        follow_up : bool
          If path has symlinks -- they get resolved by git.  If follow_up is
          True, we will follow original path up until we hit the same resolved
          path.  If no such path found, resolved one would be returned.
        git_options: list of str
          options to be passed to the git rev-parse call

        Return None if no parent directory contains a git repository.
        """
        cmd = ['git']
        if git_options:
            cmd.extend(git_options)
        cmd += ["rev-parse", "--show-toplevel"]
        try:
            toppath, err = GitRunner().run(
                cmd,
                cwd=path,
                log_stdout=True, log_stderr=True,
                expect_fail=True, expect_stderr=True)
            toppath = toppath.rstrip('\n\r')
        except CommandError:
            return None
        except OSError:
            toppath = GitRepo.get_toppath(dirname(path), follow_up=follow_up,
                                          git_options=git_options)

        if follow_up:
            path_ = path
            path_prev = ""
            while path_ and path_ != path_prev:  # on top /.. = /
                if realpath(path_) == toppath:
                    toppath = path_
                    break
                path_prev = path_
                path_ = dirname(path_)

        return toppath

    # classmethod so behavior could be tuned in derived classes
    @classmethod
    def _get_added_files_commit_msg(cls, files):
        if not files:
            return "No files were added"
        msg = "Added %d file" % len(files)
        if len(files) > 1:
            msg += "s"
        return msg + '\n\nFiles:\n' + '\n'.join(files)

    @normalize_paths
    def add(self, files, commit=False, msg=None, git=True, git_options=None,
            _datalad_msg=False, update=False):
        """Adds file(s) to the repository.

        Parameters
        ----------
        files: list
          list of paths to add
        commit: bool
          whether or not to directly commit
        msg: str
          commit message in case `commit=True`. A default message, containing
          the list of files that were added, is created by default.
        git: bool
          somewhat ugly construction to be compatible with AnnexRepo.add();
          has to be always true.
        update: bool
          --update option for git-add. From git's manpage:
           Update the index just where it already has an entry matching
           <pathspec>. This removes as well as modifies index entries to match
           the working tree, but adds no new files.

           If no <pathspec> is given when --update option is used, all tracked
           files in the entire working tree are updated (old versions of Git
           used to limit the update to the current directory and its
           subdirectories).
        """

        # TODO: git_options is used as options for the git-add here,
        # instead of options to the git executable => rename for consistency

        # needs to be True - see docstring:
        assert(git)

        files = _remove_empty_items(files)
        out = []

        if files or git_options or update:
            try:
                # without --verbose git 2.9.3  add does not return anything
                add_out = self._git_custom_command(
                    files,
                    ['git', 'add'] + assure_list(git_options) +
                    to_options(update=update) + ['--verbose']
                )
                # get all the entries
                out = self._process_git_get_output(*add_out)
                # Note: as opposed to git cmdline, force is True by default in
                #       gitpython, which would lead to add things, that are
                #       ignored or excluded otherwise
                # 2. Note: There is an issue with globbing (like adding '.'),
                #       which apparently doesn't care for 'force' and therefore
                #       adds '.git/...'. May be it's expanded at the wrong
                #       point in time or sth. like that.
                # For now, use direct call to git add.
                #self.cmd_call_wrapper(self.repo.index.add, files, write=True,
                #                      force=False)
                # TODO: May be make use of 'fprogress'-option to indicate
                # progress
                # But then, we don't have it for git-annex add, anyway.
                #
                # TODO: Is write=True a reasonable way to do it?
                # May be should not write until success of operation is
                # confirmed?
                # What's best in case of a list of files?
            except OSError as e:
                lgr.error("add: %s" % e)
                raise

        else:
            lgr.warning("add was called with empty file list and no options.")

        if commit:
            if msg is None:
                msg = self._get_added_files_commit_msg(files)
            self.commit(msg=msg, _datalad_msg=_datalad_msg)

        # Make sure return value from GitRepo is consistent with AnnexRepo
        # currently simulating similar return value, assuming success
        # for all files:
        # TODO: Make return values consistent across both *Repo classes!
        return out

    @staticmethod
    def _process_git_get_output(stdout, stderr=None):
        """Given both outputs (stderr is ignored atm) of git add - process it

        Primarily to centralize handling in both indirect annex and direct
        modes when ran through proxy
        """
        return [{u'file': f, u'success': True}
                for f in re.findall("'(.*)'[\n$]", stdout)]

    @normalize_paths(match_return_type=False)
    def remove(self, files, recursive=False, **kwargs):
        """Remove files.

        Calls git-rm.

        Parameters
        ----------
        files: str
          list of paths to remove
        recursive: False
          either to allow recursive removal from subdirectories
        kwargs:
          see `__init__`

        Returns
        -------
        [str]
          list of successfully removed files.
        """

        files = _remove_empty_items(files)

        if recursive:
            kwargs['r'] = True
        stdout, stderr = self._git_custom_command(
            files, ['git', 'rm'] + to_options(**kwargs))

        # output per removed file is expected to be "rm 'PATH'":
        return [line.strip()[4:-1] for line in stdout.splitlines()]

        #return self.repo.git.rm(files, cached=False, **kwargs)

    def precommit(self):
        """Perform pre-commit maintenance tasks
        """
        # All GitPython commands should take care about flushing index
        # whenever they modify it, so we would not care to do anything
        # if self.repo is not None and exists(opj(self.path, '.git')):  # don't try to write otherwise:
        #     # flush possibly cached in GitPython changes to index:
        #     # if self.repo.git:
        #     #     sys.stderr.write("CLEARING\n")
        #     #     self.repo.git.clear_cache()
        #     self.repo.index.write()
        pass

    @staticmethod
    def _get_prefixed_commit_msg(msg):
        DATALAD_PREFIX = "[DATALAD]"
        return DATALAD_PREFIX if not msg else "%s %s" % (DATALAD_PREFIX, msg)

    def commit(self, msg=None, options=None, _datalad_msg=False, careless=True,
               files=None):
        """Commit changes to git.

        Parameters
        ----------
        msg: str, optional
          commit-message
        options: list of str, optional
          cmdline options for git-commit
        _datalad_msg: bool, optional
          To signal that commit is automated commit by datalad, so
          it would carry the [DATALAD] prefix
        careless: bool, optional
          if False, raise when there's nothing actually committed;
          if True, don't care
        files: list of str, optional
          path(s) to commit
        """

        self.precommit()

        if _datalad_msg:
            msg = self._get_prefixed_commit_msg(msg)

        if not msg:
            if options:
                if "--allow-empty-message" not in options:
                        options.append("--allow-empty-message")
            else:
                options = ["--allow-empty-message"]

        # Note: We used to use a direct call to git only if there were options,
        # since we can't pass all possible options to gitpython's implementation
        # of commit.
        # But there's an additional issue. GitPython implements commit in a way,
        # that it might create a new commit, when a direct call wouldn't. This
        # was discovered with a modified (but unstaged) submodule, leading to a
        # commit, that apparently did nothing - git status still showed the very
        # same thing afterwards. But a commit was created nevertheless:
        # diff --git a/sub b/sub
        # --- a/sub
        # +++ b/sub
        # @@ -1 +1 @@
        # -Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f
        # +Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f-dirty
        #
        # Therefore, for now always use direct call.
        # TODO: Figure out, what exactly is going on with gitpython here

        cmd = ['git', 'commit'] + (["-m", msg if msg else ""])
        if options:
            cmd.extend(options)
        lgr.debug("Committing via direct call of git: %s" % cmd)

        try:
            self._git_custom_command(files, cmd,
                                     expect_stderr=True, expect_fail=True)
        except CommandError as e:
            if 'nothing to commit' in e.stdout:
                if careless:
                    lgr.debug("nothing to commit in {}. "
                              "Ignored.".format(self))
                else:
                    raise
            elif 'no changes added to commit' in e.stdout or \
                    'nothing added to commit' in e.stdout:
                if careless:
                    lgr.debug("no changes added to commit in {}. "
                              "Ignored.".format(self))
                else:
                    raise
            elif "did not match any file(s) known to git." in e.stderr:
                # TODO: Improve FileNotInXXXXError classes to better deal with
                # multiple files; Also consider PathOutsideRepositoryError
                raise FileNotInRepositoryError(cmd=e.cmd,
                                               msg="File(s) unknown to git",
                                               code=e.code,
                                               filename=linesep.join(
                                            [l for l in e.stderr.splitlines()
                                             if l.startswith("pathspec")]))
            else:
                raise

    def get_indexed_files(self):
        """Get a list of files in git's index

        Returns
        -------
        list
            list of paths rooting in git's base dir
        """

        return [x[0] for x in self.cmd_call_wrapper(
            self.repo.index.entries.keys)]

    def get_hexsha(self, object=None):
        """Return a hexsha for a given object. If None - of current HEAD

        Parameters
        ----------
        object: str, optional
          Any type of Git object identifier. See `git show`.
        """
        cmd = ['git', 'show', '--no-patch', "--format=%H"]
        if object:
            cmd.append(object)
        # make sure Git takes our argument as a revision
        cmd.append('--')
        try:
            stdout, stderr = self._git_custom_command(
                '', cmd, expect_stderr=True, expect_fail=True)
        except CommandError as e:
            if 'bad revision' in e.stderr:
                raise ValueError("Unknown object identifier: %s" % object)
            elif 'does not have any commits yet' in e.stderr:
                return None
            else:
                raise e
        stdout = stdout.splitlines()
        assert(len(stdout) == 1)
        return stdout[0]

    def get_merge_base(self, treeishes):
        """Get a merge base hexsha

        Parameters
        ----------
        treeishes: str or list of str
          List of treeishes (branches, hexshas, etc) to determine the merge
          base of. If a single value provided, returns merge_base with the
          current branch.

        Returns
        -------
        str or None
          If no merge-base for given commits, or specified treeish doesn't
          exist, None returned
        """
        if isinstance(treeishes, string_types):
            treeishes = [treeishes]
        if not treeishes:
            raise ValueError("Provide at least a single value")
        elif len(treeishes) == 1:
            treeishes = treeishes + [self.get_active_branch()]

        try:
            bases = self.repo.merge_base(*treeishes)
        except GitCommandError as exc:
            if "fatal: Not a valid object name" in str(exc):
                return None
            raise

        if not bases:
            return None
        assert(len(bases) == 1)  # we do not do 'all' yet
        return bases[0].hexsha

    def get_committed_date(self, branch=None):
        """Get the date stamp of the last commit (in a branch). None if no commit"""
        try:
            commit = next(
                self.get_branch_commits(branch
                                        or self.get_active_branch())
            )
        except Exception as exc:
            lgr.debug("Got exception while trying to get last commit: %s",
                      exc_str(exc))
            return None
        return commit.committed_date

    def get_active_branch(self):
        try:
            branch = self.repo.active_branch.name
        except TypeError as e:
            if "HEAD is a detached symbolic reference" in str(e):
                lgr.debug("detached HEAD in {0}".format(self))
                return None
            else:
                raise
        return branch

    def get_branches(self):
        """Get all branches of the repo.

        Returns
        -------
        [str]
            Names of all branches of this repository.
        """

        return [branch.name for branch in self.repo.branches]

    def get_remote_branches(self):
        """Get all branches of all remotes of the repo.

        Returns
        -----------
        [str]
            Names of all remote branches.
        """
        # TODO: Reconsider melting with get_branches()

        # TODO: treat entries like this: origin/HEAD -> origin/master'
        # currently this is done in collection

        # For some reason, this is three times faster than the version below:
        remote_branches = list()
        for remote in self.repo.remotes:
            try:
                for ref in remote.refs:
                    remote_branches.append(ref.name)
            except AssertionError as e:
                if str(e).endswith("did not have any references"):
                    # this will happen with git annex special remotes
                    pass
                else:
                    raise e
        return remote_branches
        # return [branch.strip() for branch in
        #         self.repo.git.branch(r=True).splitlines()]

    def get_remotes(self, with_urls_only=False):
        """Get known remotes of the repository

        Parameters
        ----------
        with_urls_only : bool, optional
          return only remotes which have urls

        Returns
        -------
        remotes : list of str
          List of names of the remotes
        """

        # Note: read directly from config and spare instantiation of gitpy.Repo
        # since we need this in AnnexRepo constructor. Furthermore gitpy does it
        # pretty much the same way and the use of a Repo instance seems to have
        # no reason other than a nice object oriented look.
        from datalad.utils import unique

        self.config.reload()
        remotes = unique([x[7:] for x in self.config.sections()
                          if x.startswith("remote.")])

        if with_urls_only:
            remotes = [
                r for r in remotes
                if self.config.get('remote.%s.url' % r)
            ]
        return remotes

    def get_files(self, branch=None):
        """Get a list of files in git.

        Lists the files in the (remote) branch.

        Parameters
        ----------
        branch: str
          Name of the branch to query. Default: active branch.

        Returns
        -------
        [str]
          list of files.
        """
        # TODO: RF codes base and melt get_indexed_files() in

        if branch is None:
            # active branch can be queried way faster:
            return self.get_indexed_files()
        else:
            return [item.path for item in self.repo.tree(branch).traverse()
                    if isinstance(item, Blob)]

    def get_file_content(self, file_, branch='HEAD'):
        """

        Returns
        -------
        [str]
          content of file_ as a list of lines.
        """
        content_str = self.repo.commit(branch).tree[file_].data_stream.read()

        # in python3 a byte string is returned. Need to convert it:
        from six import PY3
        if PY3:
            conv_str = u''
            for b in bytes(content_str):
                conv_str += chr(b)
            return conv_str.splitlines()
        else:
            return content_str.splitlines()
        # TODO: keep splitlines?

    def _get_files_history(self, files, branch='HEAD'):
        """

        Parameters
        ----------
        files: list
          list of files, only commits with queried files are considered
        branch: str
          Name of the branch to query. Default: HEAD.

        Returns
        -------
        [iterator]
        yielding Commit items generator from branch history associated with files
        """
        return gitpy.objects.commit.Commit.iter_items(self.repo, branch, paths=files)

    def _get_remotes_having_commit(self, commit_hexsha, with_urls_only=True):
        """Traverse all branches of the remote and check if commit in any of their ancestry

        It is a generator yielding names of the remotes
        """
        out, err = self._git_custom_command(
            '', 'git branch -r --contains ' + commit_hexsha
        )
        # sanitize a bit (all the spaces and new lines)
        remote_branches = [
            b  # could be origin/HEAD -> origin/master, we just skip ->
            for b in filter(bool, out.split())
            if b != '->'
        ]
        return [
            remote
            for remote in self.get_remotes(with_urls_only=with_urls_only)
            if any(rb.startswith(remote + '/') for rb in remote_branches)
        ]

    def _gitpy_custom_call(self, cmd, cmd_args=None, cmd_options=None,
                           git_options=None, env=None,

                           # 'old' options for Runner; not sure yet, which of
                           # them are actually still needed:
                           log_stdout=True, log_stderr=True, log_online=False,
                           expect_stderr=True, cwd=None,
                           shell=None, expect_fail=False):

        """Helper to call GitPython's wrapper for git calls.

        The used instance of `gitpy.Git` is bound to the repository,
        which determines its working directory.
        This is used for adhoc implementation of a git command and to
        demonstrate how to use it in more specific implementations.

        Note
        ----
        Aims to replace the use of datalad's `Runner` class for direct git
        calls. (Currently the `_git_custom_command()` method).
        Therefore mimicking its behaviour during RF'ing.

        Parameters
        ----------
        cmd: str
          the native git command to call
        cmd_args: list of str
          arguments to the git command
        cmd_options: dict
          options for the command as key, value pair
          (this transformation, needs some central place to document)
        git_options: dict
          options for the git executable as key, value pair
          (see above)
        env: dict
          environment vaiables to temporarily set for this call

        TODO
        ----
        Example

        Returns
        -------
        (stdout, stderr)
        """

        # TODO: Reconsider when to log/stream what (stdout, stderr) and/or
        # fully implement the behaviour of `Runner`

        if log_online:
            raise NotImplementedError("option 'log_online' not implemented yet")
        with_exceptions = not expect_fail
        if cwd:
            # the gitpy.cmd.Git instance, bound to this repository doesn't allow
            # to explicitly set the working dir, except for using os.getcwd
            raise NotImplementedError("working dir is a read-only property")

        _tmp_shell = gitpy.cmd.Git.USE_SHELL
        gitpy.cmd.Git.USE_SHELL = shell

        if env is None:
            env = {}
        if git_options is None:
            git_options = {}
        if cmd_options is None:
            cmd_options = {}
        cmd_options.update({'with_exceptions': with_exceptions,
                            'with_extended_output': True})

        # TODO: _GIT_COMMON_OPTIONS!

        with self.repo.git.custom_environment(**env):
            try:
                status, std_out, std_err = \
                    self.repo.git(**git_options).__getattr__(cmd)(
                        cmd_args, **cmd_options)
            except GitCommandError as e:
                # For now just reraise. May be raise CommandError instead
                raise
            finally:
                gitpy.cmd.Git.USE_SHELL = _tmp_shell

        if not expect_stderr and std_err:
            lgr.error("Unexpected output on stderr: %s" % std_err)
            raise CommandError
        if log_stdout:
            for line in std_out.splitlines():
                lgr.debug("stdout| " + line)
        if log_stderr:
            for line in std_err.splitlines():
                lgr.log(level=logging.DEBUG if expect_stderr else logging.ERROR,
                        msg="stderr| " + line)

        return std_out, std_err

    @normalize_paths(match_return_type=False)
    def _git_custom_command(self, files, cmd_str,
                            log_stdout=True, log_stderr=True, log_online=False,
                            expect_stderr=True, cwd=None, env=None,
                            shell=None, expect_fail=False):
        """Allows for calling arbitrary commands.

        Helper for developing purposes, i.e. to quickly implement git commands
        for proof of concept without the need to figure out, how this is done
        via GitPython.

        Parameters
        ----------
        files: list of files
        cmd_str: str or list
            arbitrary command str. `files` is appended to that string.

        Returns
        -------
        stdout, stderr
        """

        # ensure cmd_str becomes a well-formed list:
        if isinstance(cmd_str, string_types):
            if files and not cmd_str.strip().endswith(" --"):
                cmd_str += " --"
            cmd_str = shlex.split(cmd_str, posix=not on_windows)
        else:
            if files and cmd_str[-1] != '--':
                cmd_str.append('--')

        cmd = cmd_str + files

        assert(cmd[0] == 'git')
        cmd = cmd[:1] + self._GIT_COMMON_OPTIONS + cmd[1:]

        from .exceptions import GitIgnoreError

        try:
            out, err = self.cmd_call_wrapper.run(
                cmd,
                log_stderr=log_stderr,
                log_stdout=log_stdout,
                log_online=log_online,
                expect_stderr=expect_stderr,
                cwd=cwd,
                env=env,
                shell=shell,
                expect_fail=expect_fail)
        except CommandError as e:
            ignored = re.search(GitIgnoreError.pattern, e.stderr)
            if ignored:
                raise GitIgnoreError(cmd=e.cmd, msg=e.stderr,
                                     code=e.code, stdout=e.stdout,
                                     stderr=e.stderr,
                                     paths=ignored.groups()[0].splitlines())
            raise
        return out, err

# TODO: --------------------------------------------------------------------

    def add_remote(self, name, url, options=None):
        """Register remote pointing to a url
        """
        cmd = ['git', 'remote', 'add']
        if options:
            cmd += options
        cmd += [name, url]

        result = self._git_custom_command('', cmd)
        self.config.reload()
        return result

    def remove_remote(self, name):
        """Remove existing remote
        """

        # TODO: testing and error handling!
        from .exceptions import RemoteNotAvailableError
        try:
            out, err = self._git_custom_command(
                '', ['git', 'remote', 'remove', name])
        except CommandError as e:
            if 'fatal: No such remote' in e.stderr:
                raise RemoteNotAvailableError(name,
                                              cmd="git remote remove",
                                              msg="No such remote",
                                              stdout=out,
                                              stderr=err)
            else:
                raise e

        # TODO: config.reload necessary?
        self.config.reload()
        return

    def update_remote(self, name=None, verbose=False):
        """
        """
        options = ["-v"] if verbose else []
        name = [name] if name else []
        self._git_custom_command(
            '', ['git', 'remote'] + name + ['update'] + options,
            expect_stderr=True
        )

    # TODO: centralize all the c&p code in fetch, pull, push
    # TODO: document **kwargs passed to gitpython
    def fetch(self, remote=None, refspec=None, progress=None, all_=False,
              **kwargs):
        """Fetches changes from a remote (or all_ remotes).

        Parameters
        ----------
        remote: str
          (optional) name of the remote to fetch from. If no remote is given and
          `all_` is not set, the tracking branch is fetched.
        refspec: str
          (optional) refspec to fetch.
        progress:
          passed to gitpython. TODO: Figure it out, make consistent use of it
          and document it.
        all_: bool
          fetch all_ remotes (and all_ of their branches).
          Fails if `remote` was given.
        kwargs:
          passed to gitpython. TODO: Figure it out, make consistent use of it
          and document it.

        Returns
        -------
        list
            FetchInfo objects of the items fetched from remote
        """
        # TODO: options=> **kwargs):
        # Note: Apparently there is no explicit (fetch --all) in gitpython,
        #       but fetch is always bound to a certain remote instead.
        #       Therefore implement it on our own:
        if remote is None:
            if refspec is not None:
                # conflicts with using tracking branch or fetch all remotes
                # For now: Just fail.
                # TODO: May be check whether it fits to tracking branch
                raise ValueError("refspec specified without a remote. (%s)" %
                                 refspec)
            if all_:
                remotes_to_fetch = [
                    self.repo.remote(r)
                    for r in self.get_remotes(with_urls_only=True)
                ]
            else:
                # No explicit remote to fetch.
                # => get tracking branch:
                tb_remote, refspec = self.get_tracking_branch()
                if tb_remote is not None:
                    remotes_to_fetch = [self.repo.remote(tb_remote)]
                else:
                    # No remote, no tracking branch
                    # => fail
                    raise ValueError("Neither a remote is specified to fetch "
                                     "from nor a tracking branch is set up.")
        else:
            remotes_to_fetch = [self.repo.remote(remote)]

        fi_list = []
        for rm in remotes_to_fetch:
            fetch_url = \
                self.config.get('remote.%s.fetchurl' % rm.name,
                                self.config.get('remote.%s.url' % rm.name,
                                                None))
            if fetch_url is None:
                lgr.debug("Remote %s has no URL", rm)
                return []

            if is_ssh(fetch_url):
                ssh_manager.get_connection(fetch_url).open()
                # TODO: with git <= 2.3 keep old mechanism:
                #       with rm.repo.git.custom_environment(GIT_SSH="wrapper_script"):
                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):
                    fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)
                    # TODO: progress +kwargs
            else:
                fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)
                # TODO: progress +kwargs

        # TODO: fetch returns a list of FetchInfo instances. Make use of it.
        return fi_list

    def pull(self, remote=None, refspec=None, progress=None, **kwargs):
        """See fetch
        """
        if remote is None:
            if refspec is not None:
                # conflicts with using tracking branch or fetch all remotes
                # For now: Just fail.
                # TODO: May be check whether it fits to tracking branch
                raise ValueError("refspec specified without a remote. (%s)" %
                                 refspec)
            # No explicit remote to pull from.
            # => get tracking branch:
            tb_remote, refspec = self.get_tracking_branch()
            if tb_remote is not None:
                remote = self.repo.remote(tb_remote)
            else:
                # No remote, no tracking branch
                # => fail
                raise ValueError("No remote specified to pull from nor a "
                                 "tracking branch is set up.")

        else:
            remote = self.repo.remote(remote)

        fetch_url = \
            remote.config_reader.get(
                'fetchurl' if remote.config_reader.has_option('fetchurl')
                else 'url')
        if is_ssh(fetch_url):
            ssh_manager.get_connection(fetch_url).open()
            # TODO: with git <= 2.3 keep old mechanism:
            #       with remote.repo.git.custom_environment(GIT_SSH="wrapper_script"):
            with remote.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):
                return remote.pull(refspec=refspec, progress=progress, **kwargs)
                # TODO: progress +kwargs
        else:
            return remote.pull(refspec=refspec, progress=progress, **kwargs)
            # TODO: progress +kwargs

    def push(self, remote=None, refspec=None, progress=None, all_remotes=False,
             **kwargs):
        """Push to remote repository

        Parameters
        ----------
        remote: str
          name of the remote to push to
        refspec: str
          specify what to push
        progress:
          TODO
        all_remotes: bool
          if set to True push to all remotes. Conflicts with `remote` not being
          None.
        kwargs: dict
          options to pass to `git push`

        Returns
        -------
        list
            PushInfo objects of the items pushed to remote
        """

        if remote is None:
            if refspec is not None:
                # conflicts with using tracking branch or fetch all remotes
                # For now: Just fail.
                # TODO: May be check whether it fits to tracking branch
                raise ValueError("refspec specified without a remote. (%s)" %
                                 refspec)
            if all_remotes:
                remotes_to_push = self.repo.remotes
            else:
                # Nothing explicitly specified. Just call `git push` and let git
                # decide what to do would be an option. But:
                # - without knowing the remote and its URL we cannot provide
                #   shared SSH connection
                # - we lose ability to use GitPython's progress info and return
                #   values
                #   (the latter would be solvable:
                #    Provide a Repo.push() method for GitPython, copying
                #    Remote.push() for similar return value and progress
                #    (also: fetch, pull)

                # Do what git would do:
                # 1. branch.*.remote for current branch or 'origin' as default
                #    if config is missing
                # 2. remote.*.push or push.default

                # TODO: check out "same procedure" for fetch/pull

                tb_remote, refspec = self.get_tracking_branch()
                if tb_remote is None:
                    tb_remote = 'origin'
                remotes_to_push = [self.repo.remote(tb_remote)]
                # use no refspec; let git find remote.*.push or push.default on
                # its own

        else:
            if all_remotes:
                lgr.warning("Option 'all_remotes' conflicts with specified "
                            "remote '%s'. Option ignored.")
            remotes_to_push = [self.repo.remote(remote)]

        pi_list = []
        for rm in remotes_to_push:
            push_url = \
                rm.config_reader.get('pushurl'
                                     if rm.config_reader.has_option('pushurl')
                                     else 'url')
            if is_ssh(push_url):
                ssh_manager.get_connection(push_url).open()
                # TODO: with git <= 2.3 keep old mechanism:
                #       with rm.repo.git.custom_environment(GIT_SSH="wrapper_script"):
                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):
                    pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)
                    # TODO: progress +kwargs
            else:
                pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)
                # TODO: progress +kwargs
        return pi_list

    def get_remote_url(self, name, push=False):
        """Get the url of a remote.

        Reads the configuration of remote `name` and returns its url or None,
        if there is no url configured.

        Parameters
        ----------
        name: str
          name of the remote
        push: bool
          if True, get the pushurl instead of the fetch url.
        """

        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')
        return self.config.get(var, None)

    def set_remote_url(self, name, url, push=False):
        """Set the URL a remote is pointing to

        Sets the URL of the remote `name`. Requires the remote to already exist.

        Parameters
        ----------
        name: str
          name of the remote
        url: str
        push: bool
          if True, set the push URL, otherwise the fetch URL
        """

        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')
        self.config.set(var, url, where='local', reload=True)

    def get_branch_commits(self, branch=None, limit=None, stop=None, value=None):
        """Return GitPython's commits for the branch

        Pretty much similar to what 'git log <branch>' does.
        It is a generator which returns top commits first

        Parameters
        ----------
        branch: str, optional
          If not provided, assumes current branch
        limit: None | 'left-only', optional
          Limit which commits to report.  If None -- all commits (merged or not),
          if 'left-only' -- only the commits from the left side of the tree upon
          merges
        stop: str, optional
          hexsha of the commit at which stop reporting (matched one is not
          reported either)
        value: None | 'hexsha', optional
          What to yield.  If None - entire commit object is yielded, if 'hexsha'
          only its hexsha
        """

        if not branch:
            branch = self.get_active_branch()

        try:
            _branch = self.repo.branches[branch]
        except IndexError:
            raise MissingBranchError(self, branch,
                                     [b.name for b in self.repo.branches])

        fvalue = {None: lambda x: x, 'hexsha': lambda x: x.hexsha}[value]

        if not limit:
            def gen():
                # traverse doesn't yield original commit
                co = _branch.commit
                yield co
                for co_ in co.traverse():
                    yield co_
        elif limit == 'left-only':
            # we need a custom implementation since couldn't figure out how to
            # do with .traversal
            def gen():
                co = _branch.commit
                while co:
                    yield co
                    co = co.parents[0] if co.parents else None
        else:
            raise ValueError(limit)

        for c in gen():
            if stop and c.hexsha == stop:
                return
            yield fvalue(c)

    def checkout(self, name, options=None):
        """
        """
        # TODO: May be check for the need of -b options herein?
        cmd = ['git', 'checkout']
        if options:
            cmd += options
        cmd += [str(name)]

        self._git_custom_command('', cmd, expect_stderr=True)

    # TODO: Before implementing annex merge, find usages and check for a needed
    # change to call super().merge
    def merge(self, name, options=None, msg=None, allow_unrelated=False, **kwargs):
        if options is None:
            options = []
        if msg:
            options = options + ["-m", msg]
        if allow_unrelated and external_versions['cmd:git'] >= '2.9':
            options += ['--allow-unrelated-histories']
        self._git_custom_command(
            '', ['git', 'merge'] + options + [name],
            **kwargs
        )

    def remove_branch(self, branch):
        self._git_custom_command(
            '', ['git', 'branch', '-D', branch]
        )

    def ls_remote(self, remote, options=None):
        if options is None:
            options = []
        self._git_custom_command(
            '', ['git', 'ls-remote'] + options + [remote]
        )
        # TODO: Return values?

    def is_dirty(self, index=True, working_tree=True, untracked_files=True,
                 submodules=True, path=None):
        """Returns true if the repo is considered to be dirty

        Parameters
        ----------
        index: bool
          if True, consider changes to the index
        working_tree: bool
          if True, consider changes to the working tree
        untracked_files: bool
          if True, consider untracked files
        submodules: bool
          if True, consider submodules
        path: str or list of str
          path(s) to consider only
        Returns
        -------
          bool
        """

        return self.repo.is_dirty(index=index, working_tree=working_tree,
                                  untracked_files=untracked_files,
                                  submodules=submodules, path=path)

    @property
    def dirty(self):
        return self.is_dirty()

    @property
    def untracked_files(self):
        return self.repo.untracked_files

    def gc(self, allow_background=False, auto=False):
        """Perform house keeping (garbage collection, repacking)"""
        cmd_options = ['git']
        if not allow_background:
            cmd_options += ['-c', 'gc.autodetach=0']
        cmd_options += ['gc', '--aggressive']
        if auto:
            cmd_options += ['--auto']
        self._git_custom_command('', cmd_options)

    def get_submodules(self, sorted_=True):
        """Return a list of git.Submodule instances for all submodules"""
        # check whether we have anything in the repo. if not go home early
        if not self.repo.head.is_valid():
            return []
        submodules = self.repo.submodules
        if sorted_:
            submodules = sorted(submodules, key=lambda x: x.path)
        return submodules

    def is_submodule_modified(self, name, options=[]):
        """Whether a submodule has new commits

        Note: This is an adhoc method. It parses output of
        'git submodule summary' and currently is not able to distinguish whether
        or not this change is staged in `self` and whether this would be
        reported 'added' or 'modified' by 'git status'.
        Parsing isn't heavily tested yet.

        Parameters
        ----------
        name: str
          the submodule's name
        options: list
          options to pass to 'git submodule summary'
        Returns
        -------
        bool
          True if there are commits in the submodule, differing from
          what is registered in `self`
        --------
        """

        out, err = self._git_custom_command('',
                                            ['git', 'submodule', 'summary'] + \
                                            options + ['--', name])
        return any([line.split()[1] == name
                    for line in out.splitlines()
                    if line and len(line.split()) > 1])

    def add_submodule(self, path, name=None, url=None, branch=None):
        """Add a new submodule to the repository.

        This will alter the index as well as the .gitmodules file, but will not
        create a new commit.  If the submodule already exists, no matter if the
        configuration differs from the one provided, the existing submodule
        is considered as already added and no further action is performed.

        Parameters
        ----------
        path : str
          repository-relative path at which the submodule should be located, and
          which will be created as required during the repository initialization.
        name : str or None
          name/identifier for the submodule. If `None`, the `path` will be used
          as name.
        url : str or None
          git-clone compatible URL. If `None`, the repository is assumed to
          exist, and the url of the first remote is taken instead. This is
          useful if you want to make an existing repository a submodule of
          another one.
        branch : str or None
          name of branch to be checked out in the submodule. The given branch
          must exist in the remote repository, and will be checked out locally
          as a tracking branch. If `None`, remote HEAD will be checked out.
        """
        if name is None:
            name = path
        # XXX the following should do it, but GitPython will refuse to add a submodule
        # unless you specify a URL that is configured as one of its remotes, or you
        # specify no URL, but the repo has at least one remote.
        # this is stupid, as for us it is valid to not have any remote, because we can
        # still obtain the submodule from a future publication location, based on the
        # parent
        # gitpy.Submodule.add(self.repo, name, path, url=url, branch=branch)
        # going git native instead
        cmd = ['git', 'submodule', 'add', '--name', name]
        if branch is not None:
            cmd += ['-b', branch]
        if url is None:
            if not isabs(path):
                # need to recode into a relative path "URL" in POSIX
                # style, even on windows
                url = posixpath.join(curdir, posix_relpath(path))
            else:
                url = path
        cmd += [url, path]
        self._git_custom_command('', cmd)
        # TODO: return value

    def deinit_submodule(self, path, **kwargs):
        """Deinit a submodule

        Parameters
        ----------
        path: str
            path to the submodule; relative to `self.path`
        kwargs:
            see `__init__`
        """

        self._git_custom_command(path,
                                 ['git', 'submodule', 'deinit'] +
                                 to_options(**kwargs))
        # TODO: return value

    def update_submodule(self, path, mode='checkout', init=False):
        """Update a registered submodule.

        This will make the submodule match what the superproject expects by
        cloning missing submodules and updating the working tree of the
        submodules. The "updating" can be done in several ways depending
        on the value of submodule.<name>.update configuration variable, or
        the `mode` argument.

        Parameters
        ----------
        path : str
          Identifies which submodule to operate on by it's repository-relative
          path.
        mode : {checkout, rebase, merge}
          Update procedure to perform. 'checkout': the commit recorded in the
          superproject will be checked out in the submodule on a detached HEAD;
          'rebase': the current branch of the submodule will be rebased onto
          the commit recorded in the superproject; 'merge': the commit recorded
          in the superproject will be merged into the current branch in the
          submodule.
        init : bool
          If True, initialize all submodules for which "git submodule init" has
          not been called so far before updating.
          Primarily provided for internal purposes and should not be used directly
          since would result in not so annex-friendly .git symlinks/references
          instead of full featured .git/ directories in the submodules
        """
        cmd = ['git', 'submodule', 'update', '--%s' % mode]
        if init:
            cmd.append('--init')
            subgitpath = opj(self.path, path, '.git')
            if not exists(subgitpath):
                # TODO:  wouldn't with --init we get all those symlink'ed .git/?
                # At least let's warn
                lgr.warning(
                    "Do not use update_submodule with init=True to avoid git creating "
                    "symlinked .git/ directories in submodules"
                )
            #  yoh: I thought I saw one recently but thought it was some kind of
            #  an artifact from running submodule update --init manually at
            #  some point, but looking at this code now I worry that it was not
        self._git_custom_command(path, cmd)
        # TODO: return value

    def update_ref(self, ref, value, symbolic=False):
        """Update the object name stored in a ref "safely".

        Just a shim for `git update-ref` call if not symbolic, and
        `git symbolic-ref` if symbolic

        Parameters
        ----------
        ref : str
          Reference, such as `ref/heads/BRANCHNAME` or HEAD.
        value : str
          Value to update to, e.g. hexsha of a commit when updating for a
          branch ref, or branch ref if updating HEAD
        symbolic : None
          To instruct if ref is symbolic, e.g. should be used in case of
          ref=HEAD
        """
        self._git_custom_command(
            '',
            ['git', 'symbolic-ref' if symbolic else 'update-ref', ref, value]
        )

    def tag(self, tag):
        """Assign a tag to current commit

        Parameters
        ----------
        tag : str
          Custom tag label.
        """
        # TODO later to be extended with tagging particular commits and signing
        # TODO: call in save.py complains about extensive logging. When does it
        # happen in what way? Figure out, whether to just silence it or raise or
        # whatever else.
        self._git_custom_command(
            '', ['git', 'tag', str(tag)]
        )

    def get_tags(self, output=None):
        """Get list of tags

        Parameters
        ----------
        output : str, optional
          If given, limit the return value to a list of values matching that
          particular key of the tag properties.

        Returns
        -------
        list
          Each item is a dictionary with information on a tag. At present
          this includes 'hexsha', and 'name', where the latter is the string
          label of the tag, and the format the hexsha of the object the tag
          is attched to. The list is sorted by commit date, with the most
          recent commit being the last element.
        """
        # TODO it would be straightforward to add more info and tweak the
        # sorting
        stdout, stderr = self._git_custom_command(
            '',
            ['git', 'tag', '--format=%(refname:strip=2)%00%(object)',
             '--sort=*committerdate'])
        fields = ('name', 'hexsha')
        tags = [dict(zip(fields, line.split('\0'))) for line in stdout.splitlines()]
        if output:
            return [t[output] for t in tags]
        else:
            return tags

    def describe(self, **kwargs):
        """ Quick and dirty implementation to call git-describe

        Parameters:
        -----------
        kwargs:
            transformed to cmdline options for git-describe;
            see __init__ for description of the transformation
        """
        # TODO: be more precise what failure to expect when and raise actual
        # errors
        try:
            describe, outerr = self._git_custom_command(
                [],
                ['git', 'describe'] + to_options(**kwargs),
                expect_fail=True)
            return describe.strip()
        # TODO: WTF "catch everything"?
        except:
            return None

    def get_tracking_branch(self, branch=None):
        """Get the tracking branch for `branch` if there is any.

        Parameters
        ----------
        branch: str
            local branch to look up. If none is given, active branch is used.

        Returns
        -------
        tuple
            (remote or None, refspec or None) of the tracking branch
        """
        if branch is None:
            branch = self.get_active_branch()
            if branch is None:
                return None, None

        track_remote = self.config.get('branch.{0}.remote'.format(branch), None)
        track_branch = self.config.get('branch.{0}.merge'.format(branch), None)
        return track_remote, track_branch

    @property
    def count_objects(self):
        """return dictionary with count, size(in KiB) information of git objects
        """

        count_cmd = ['git', 'count-objects', '-v']
        count_str, err = self._git_custom_command('', count_cmd)
        count = {key: int(value)
                 for key, value in [item.split(': ')
                                    for item in count_str.split('\n')
                                    if len(item.split(': ')) == 2]}
        return count

    def get_missing_files(self):
        """Return a list of paths with missing files (and no staged deletion)"""
        return [f.split('\t')[1]
                for f in self.repo.git.diff('--raw', '--name-status').split('\n')
                if f.split('\t')[0] == 'D']

    def get_deleted_files(self):
        """Return a list of paths with deleted files (staged deletion)"""
        return [f.split('\t')[1]
                for f in self.repo.git.diff('--raw', '--name-status', '--staged').split('\n')
                if f.split('\t')[0] == 'D']

    def get_git_attributes(self):
        """Check git attribute for the current repository (not per-file support for now)

        Parameters
        ----------
        all_: bool
          Adds --all to git check-attr call

        Returns
        -------
        dict:
          attribute: value pairs
        """
        out, err = self._git_custom_command(["."], ["git", "check-attr", "--all"])
        assert not err, "no stderr output is expected"
        out_split = [
            # splitting by : would leave leading space(s)
            [e.lstrip(' ') for e in l.split(':', 2)]
            for l in out.split('\n') if l
        ]
        assert all(o[0] == '.' for o in out_split)  # for paranoid
        return dict(o[1:] for o in out_split)


# TODO
# remove submodule: nope, this is just deinit_submodule + remove
# status?

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]

**Title**: Git Configuration Retrieval Fails When HOME Environment Variable Is Unset

**Description**:
When the `HOME` environment variable is not set, attempts to retrieve Git user configuration (`user.name` and `user.email`) fail, causing the configuration check to return `None` instead of the expected values.

For example, running the following code without a `HOME` environment variable:

```python
from datalad.support.gitrepo import check_git_configured

config = check_git_configured()
print(config)
```

**Expected Behavior**:
The `check_git_configured` function should return a dictionary with `'user.name'` and `'user.email'` entries set to predefined test values, enabling tests to run without relying on the user's actual Git configuration.

**Actual Behavior**:
The function returns `None` instead of the expected configuration dictionary, leading to failures when tests assert the presence of Git user settings.

[/ISSUE]
{
    "test_git_config_fixture": "PASSED",
    "test_not_under_git": "FAILED"
}
diff --git a/datalad/__init__.py b/datalad/__init__.py
index aa1647791..b9e7ff2c9 100644
--- a/datalad/__init__.py
+++ b/datalad/__init__.py
@@ -85,16 +85,34 @@ test.__test__ = False
 _test_states = {
     'loglevel': None,
     'DATALAD_LOG_LEVEL': None,
+    'HOME': None,
 }
 
 
 def setup_package():
     import os
 
+    _test_states['HOME'] = os.environ.get('HOME', None)
+
     # To overcome pybuild overriding HOME but us possibly wanting our
     # own HOME where we pre-setup git for testing (name, email)
     if 'GIT_HOME' in os.environ:
         os.environ['HOME'] = os.environ['GIT_HOME']
+    else:
+        # we setup our own new HOME, the BEST and HUGE one
+        from datalad.utils import make_tempfile
+        from datalad.tests import _TEMP_PATHS_GENERATED
+        # TODO: split into a function + context manager
+        with make_tempfile(mkdir=True) as new_home:
+            os.environ['HOME'] = new_home
+        os.makedirs(new_home)
+        with open(os.path.join(new_home, '.gitconfig'), 'w') as f:
+            f.write("""\
+[user]
+	name = DataLad Tester
+	email = test@example.com
+""")
+        _TEMP_PATHS_GENERATED.append(new_home)
 
     # For now we will just verify that it is ready to run the tests
     from datalad.support.gitrepo import check_git_configured
@@ -153,6 +171,9 @@ def teardown_package():
     for path in _TEMP_PATHS_GENERATED:
         rmtemp(path, ignore_errors=True)
 
+    if _test_states['HOME'] is not None:
+        os.environ['HOME'] = _test_states['HOME']
+
     lgr.debug("Printing versioning information collected so far")
     from datalad.support.external_versions import external_versions as ev
     print(ev.dumps(query=True))
diff --git a/datalad/support/gitrepo.py b/datalad/support/gitrepo.py
index ae9a9a81b..9b0f14cbf 100644
--- a/datalad/support/gitrepo.py
+++ b/datalad/support/gitrepo.py
@@ -330,12 +330,18 @@ def check_git_configured():
     Raises
     ------
     RuntimeError  if any of those two ariables are not set
+
+    Returns
+    -------
+    dict with user.name and user.email entries
     """
 
     check_runner = GitRunner()
+    vals = {}
     for c in 'user.name', 'user.email':
         try:
-            check_runner.run(['git', 'config', '--global', c])
+            v, err = check_runner.run(['git', 'config', '--global', c])
+            vals[c] = v.rstrip('\n')
         except CommandError as exc:
             lgr.debug("Failed to verify that git is configured: %s",
                       exc_str(exc))
@@ -343,6 +349,7 @@ def check_git_configured():
                 "You must configure git first (set both user.name and "
                 "user.email) settings before using DataLad."
             )
+    return vals
 
 
 def _remove_empty_items(list_):

{"datalad/support/gitrepo.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface to Git via GitPython\n\nFor further information on GitPython see http://gitpython.readthedocs.org/\n\n\"\"\"\n\nimport logging\nimport re\nimport shlex\nimport os\nfrom os import linesep\nfrom os.path import join as opj\nfrom os.path import exists\nfrom os.path import normpath\nfrom os.path import isabs\nfrom os.path import commonprefix\nfrom os.path import relpath\nfrom os.path import realpath\nfrom os.path import dirname\nfrom os.path import basename\nfrom os.path import curdir\nfrom os.path import pardir\nfrom os.path import sep\nimport posixpath\nfrom weakref import WeakValueDictionary\n\n\nfrom six import string_types\nfrom six import add_metaclass\nfrom functools import wraps\nimport git as gitpy\nfrom git.exc import GitCommandError\nfrom git.exc import NoSuchPathError\nfrom git.exc import InvalidGitRepositoryError\nfrom git.objects.blob import Blob\n\nfrom datalad import ssh_manager\nfrom datalad.cmd import GitRunner\nfrom datalad.consts import GIT_SSH_COMMAND\nfrom datalad.dochelpers import exc_str\nfrom datalad.config import ConfigManager\nfrom datalad.utils import assure_list\nfrom datalad.utils import optional_args\nfrom datalad.utils import on_windows\nfrom datalad.utils import getpwd\nfrom datalad.utils import updated\nfrom datalad.utils import posix_relpath\n\n\n# imports from same module:\nfrom .external_versions import external_versions\nfrom .exceptions import CommandError\nfrom .exceptions import DeprecatedError\nfrom .exceptions import FileNotInRepositoryError\nfrom .exceptions import MissingBranchError\nfrom .network import RI\nfrom .network import is_ssh\nfrom .repo import Flyweight\nfrom .repo import RepoInterface\n\n# shortcuts\n_curdirsep = curdir + sep\n_pardirsep = pardir + sep\n\n\nlgr = logging.getLogger('datalad.gitrepo')\n\n# Override default GitPython's DB backend to talk directly to git so it doesn't\n# interfere with possible operations performed by gc/repack\ndefault_git_odbt = gitpy.GitCmdObjectDB\n\n# TODO: Figure out how GIT_PYTHON_TRACE ('full') is supposed to be used.\n# Didn't work as expected on a first try. Probably there is a neatier way to\n# log Exceptions from git commands.\n\n\n# TODO: ignore leading and/or trailing underscore to allow for\n# python-reserved words\n@optional_args\ndef kwargs_to_options(func, split_single_char_options=True,\n                      target_kw='options'):\n    \"\"\"Decorator to provide convenient way to pass options to command calls.\n\n    Parameters\n    ----------\n    func: Callable\n        function to decorate\n    split_single_char_options: bool\n        whether or not to split key and value of single char keyword arguments\n        into two subsequent entries of the list\n    target_kw: str\n        keyword argument to pass the generated list of cmdline arguments to\n\n    Returns\n    -------\n    Callable\n    \"\"\"\n\n    # TODO: don't overwrite options, but join\n\n    @wraps(func)\n    def newfunc(self, *args, **kwargs):\n        t_kwargs = dict()\n        t_kwargs[target_kw] = \\\n            gitpy.Git().transform_kwargs(\n                split_single_char_options=split_single_char_options,\n                **kwargs)\n        return func(self, *args, **t_kwargs)\n    return newfunc\n\n\ndef to_options(**kwargs):\n    \"\"\"Transform keyword arguments into a list of cmdline options\n\n    Parameters\n    ----------\n    split_single_char_options: bool\n\n    kwargs:\n\n    Returns\n    -------\n    list\n    \"\"\"\n    # TODO: borrow_docs!\n\n    return gitpy.Git().transform_kwargs(**kwargs)\n\n\ndef _normalize_path(base_dir, path):\n    \"\"\"Helper to check paths passed to methods of this class.\n\n    Checks whether `path` is beneath `base_dir` and normalizes it.\n    Additionally paths are converted into relative paths with respect to\n    `base_dir`, considering PWD in case of relative paths. This\n    is intended to be used in repository classes, which means that\n    `base_dir` usually will be the repository's base directory.\n\n    Parameters\n    ----------\n    path: str\n        path to be normalized\n    base_dir: str\n        directory to serve as base to normalized, relative paths\n\n    Returns\n    -------\n    str:\n        path, that is a relative path with respect to `base_dir`\n    \"\"\"\n    if not path:\n        return path\n\n    base_dir = realpath(base_dir)  # realpath OK\n    # path = normpath(path)\n    # Note: disabled normpath, because it may break paths containing symlinks;\n    # But we don't want to realpath relative paths, in case cwd isn't the\n    # correct base.\n\n    if isabs(path):\n        # path might already be a symlink pointing to annex etc,\n        # so realpath only its directory, to get \"inline\" with\n        # realpath(base_dir) above\n        path = opj(realpath(dirname(path)), basename(path))  # realpath OK\n    # Executive decision was made to not do this kind of magic!\n    #\n    # elif commonprefix([realpath(getpwd()), base_dir]) == base_dir:\n    #     # If we are inside repository, rebuilt relative paths.\n    #     path = opj(realpath(getpwd()), path)\n    #\n    # BUT with relative curdir/pardir start it would assume relative to curdir\n    #\n    elif path.startswith(_curdirsep) or path.startswith(_pardirsep):\n        path = normpath(opj(realpath(getpwd()), path))  # realpath OK\n    else:\n        # We were called from outside the repo. Therefore relative paths\n        # are interpreted as being relative to self.path already.\n        return path\n\n    if commonprefix([path, base_dir]) != base_dir:\n        raise FileNotInRepositoryError(msg=\"Path outside repository: %s\"\n                                           % path, filename=path)\n\n    return relpath(path, start=base_dir)\n\n\n@optional_args\ndef normalize_path(func):\n    \"\"\"Decorator to provide unified path conversion for a single file\n\n    Unlike normalize_paths, intended to be used for functions dealing with a\n    single filename at a time\n\n    Note\n    ----\n    This is intended to be used within the repository classes and therefore\n    returns a class method!\n\n    The decorated function is expected to take a path at\n    first positional argument (after 'self'). Additionally the class `func`\n    is a member of, is expected to have an attribute 'path'.\n    \"\"\"\n\n    @wraps(func)\n    def newfunc(self, file_, *args, **kwargs):\n        file_new = _normalize_path(self.path, file_)\n        return func(self, file_new, *args, **kwargs)\n\n    return newfunc\n\n\n@optional_args\ndef normalize_paths(func, match_return_type=True, map_filenames_back=False,\n                    serialize=False):\n    \"\"\"Decorator to provide unified path conversions.\n\n    Note\n    ----\n    This is intended to be used within the repository classes and therefore\n    returns a class method!\n\n    The decorated function is expected to take a path or a list of paths at\n    first positional argument (after 'self'). Additionally the class `func`\n    is a member of, is expected to have an attribute 'path'.\n\n    Accepts either a list of paths or a single path in a str. Passes a list\n    to decorated function either way, but would return based on the value of\n    match_return_type and possibly input argument.\n\n    If a call to the wrapped function includes normalize_path and it is False\n    no normalization happens for that function call (used for calls to wrapped\n    functions within wrapped functions, while possible CWD is within a\n    repository)\n\n    Parameters\n    ----------\n    match_return_type : bool, optional\n      If True, and a single string was passed in, it would return the first\n      element of the output (after verifying that it is a list of length 1).\n      It makes easier to work with single files input.\n    map_filenames_back : bool, optional\n      If True and returned value is a dictionary, it assumes to carry entries\n      one per file, and then filenames are mapped back to as provided from the\n      normalized (from the root of the repo) paths\n    serialize : bool, optional\n      Loop through files giving only a single one to the function one at a time.\n      This allows to simplify implementation and interface to annex commands\n      which do not take multiple args in the same call (e.g. checkpresentkey)\n    \"\"\"\n\n    @wraps(func)\n    def newfunc(self, files, *args, **kwargs):\n\n        normalize = _normalize_path if kwargs.pop('normalize_paths', True) \\\n            else lambda rpath, filepath: filepath\n\n        if files:\n            if isinstance(files, string_types) or not files:\n                files_new = [normalize(self.path, files)]\n                single_file = True\n            elif isinstance(files, list):\n                files_new = [normalize(self.path, path) for path in files]\n                single_file = False\n            else:\n                raise ValueError(\"_files_decorator: Don't know how to handle \"\n                                 \"instance of %s.\" % type(files))\n        else:\n            single_file = None\n            files_new = []\n\n        if map_filenames_back:\n            def remap_filenames(out):\n                \"\"\"Helper to map files back to non-normalized paths\"\"\"\n                if isinstance(out, dict):\n                    assert(len(out) == len(files_new))\n                    files_ = [files] if single_file else files\n                    mapped = out.__class__()\n                    for fin, fout in zip(files_, files_new):\n                        mapped[fin] = out[fout]\n                    return mapped\n                else:\n                    return out\n        else:\n            remap_filenames = lambda x: x\n\n        if serialize:  # and not single_file:\n            result = [\n                func(self, f, *args, **kwargs)\n                for f in files_new\n            ]\n        else:\n            result = func(self, files_new, *args, **kwargs)\n\n        if single_file is None:\n            # no files were provided, nothing we can do really\n            return result\n        elif (result is None) or not match_return_type or not single_file:\n            # If function doesn't return anything or no denormalization\n            # was requested or it was not a single file\n            return remap_filenames(result)\n        elif single_file:\n            if len(result) != 1:\n                # Magic doesn't apply\n                return remap_filenames(result)\n            elif isinstance(result, (list, tuple)):\n                return result[0]\n            elif isinstance(result, dict) and tuple(result)[0] == files_new[0]:\n                # assume that returned dictionary has files as keys.\n                return tuple(result.values())[0]\n            else:\n                # no magic can apply\n                return remap_filenames(result)\n        else:\n            return RuntimeError(\"should have not got here... check logic\")\n\n    return newfunc\n\n\ndef check_git_configured():\n    \"\"\"Do a check if git is configured (user.name and user.email are set)\n\n    Raises\n    ------\n    RuntimeError  if any of those two ariables are not set\n    \"\"\"\n\n    check_runner = GitRunner()\n    for c in 'user.name', 'user.email':\n        try:\n            check_runner.run(['git', 'config', '--global', c])\n        except CommandError as exc:\n            lgr.debug(\"Failed to verify that git is configured: %s\",\n                      exc_str(exc))\n            raise RuntimeError(\n                \"You must configure git first (set both user.name and \"\n                \"user.email) settings before using DataLad.\"\n            )\n\n\ndef _remove_empty_items(list_):\n    \"\"\"Remove empty entries from list\n\n    This is needed, since some functions of GitPython may convert\n    an empty entry to '.', when used with a list of paths.\n\n    Parameter:\n    ----------\n    list_: list of str\n\n    Returns\n    -------\n    list of str\n    \"\"\"\n    if not isinstance(list_, list):\n        lgr.warning(\n            \"_remove_empty_items() called with non-list type: %s\" % type(list_))\n        return list_\n    return [file_ for file_ in list_ if file_]\n\n\ndef Repo(*args, **kwargs):\n    \"\"\"Factory method around gitpy.Repo to consistently initiate with different\n    backend\n    \"\"\"\n    # TODO: This probably doesn't work as intended (or at least not as\n    #       consistently as intended). gitpy.Repo could be instantiated by\n    #       classmethods Repo.init or Repo.clone_from. In these cases 'odbt'\n    #       would be needed as a paramter to these methods instead of the\n    #       constructor.\n    if 'odbt' not in kwargs:\n        kwargs['odbt'] = default_git_odbt\n    return gitpy.Repo(*args, **kwargs)\n\n\ndef split_remote_branch(branch):\n    \"\"\"Splits a remote branch's name into the name of the remote and the name\n    of the branch.\n\n    Parameters\n    ----------\n    branch: str\n      the remote branch's name to split\n\n    Returns\n    -------\n    list of str\n    \"\"\"\n    assert '/' in branch, \\\n        \"remote branch %s must have had a /\" % branch\n    assert not branch.endswith('/'), \\\n        \"branch name with trailing / is invalid. (%s)\" % branch\n    return branch.split('/', 1)\n\n\n@add_metaclass(Flyweight)\nclass GitRepo(RepoInterface):\n    \"\"\"Representation of a git repository\n\n    \"\"\"\n\n    # We use our sshrun helper\n    GIT_SSH_ENV = {'GIT_SSH_COMMAND': GIT_SSH_COMMAND}\n\n    # Just a non-functional example:\n    # must be implemented, since abstract in RepoInterface:\n    def sth_like_file_has_content(self):\n        return \"Yes, if it's in the index\"\n\n    # We must check git config to have name and email set, but\n    # should do it once\n    _config_checked = False\n\n    # Begin Flyweight:\n\n    _unique_instances = WeakValueDictionary()\n\n    @classmethod\n    def _flyweight_id_from_args(cls, *args, **kwargs):\n\n        if args:\n            # to a certain degree we need to simulate an actual call to __init__\n            # and make sure, passed arguments are fitting:\n            # TODO: Figure out, whether there is a cleaner way to do this in a\n            # generic fashion\n            assert('path' not in kwargs)\n            path = args[0]\n            args = args[1:]\n        elif 'path' in kwargs:\n            path = kwargs.pop('path')\n        else:\n            raise TypeError(\"__init__() requires argument `path`\")\n\n        if path is None:\n            raise AttributeError\n\n        # Sanity check for argument `path`:\n        # raise if we cannot deal with `path` at all or\n        # if it is not a local thing:\n        path = RI(path).localpath\n        # resolve symlinks to make sure we have exactly one instance per\n        # physical repository at a time\n        path = realpath(path)\n        kwargs['path'] = path\n        return path, args, kwargs\n\n    @classmethod\n    def _flyweight_invalid(cls, id_):\n        return not cls.is_valid_repo(id_)\n\n    @classmethod\n    def _flyweight_reject(cls, id_, *args, **kwargs):\n        # TODO:\n        # This is a temporary approach. See PR # ...\n        # create = kwargs.pop('create', None)\n        # kwargs.pop('path', None)\n        # if create and kwargs:\n        #     # we have `create` plus options other than `path`\n        #     return \"Call to {0}() with args {1} and kwargs {2} conflicts \" \\\n        #            \"with existing instance {3}.\" \\\n        #            \"This is likely to be caused by inconsistent logic in \" \\\n        #            \"your code.\" \\\n        #            \"\".format(cls, args, kwargs, cls._unique_instances[id_])\n        pass\n\n    # End Flyweight\n\n    def __init__(self, path, url=None, runner=None, create=True,\n                 git_opts=None, repo=None, **kwargs):\n        \"\"\"Creates representation of git repository at `path`.\n\n        Can also be used to create a git repository at `path`.\n\n        Parameters\n        ----------\n        path: str\n          path to the git repository; In case it's not an absolute path,\n          it's relative to PWD\n        url: str, optional\n          DEPRECATED -- use .clone() class method\n          url to the to-be-cloned repository. Requires a valid git url\n          according to:\n          http://www.kernel.org/pub/software/scm/git/docs/git-clone.html#URLS .\n        create: bool, optional\n          if true, creates a git repository at `path` if there is none. Also\n          creates `path`, if it doesn't exist.\n          If set to false, an exception is raised in case `path` doesn't exist\n          or doesn't contain a git repository.\n        repo: git.Repo, optional\n          GitPython's Repo instance to (re)use if provided\n        kwargs:\n          keyword arguments serving as additional options to the git-init\n          command. Therefore, it makes sense only if called with `create`.\n\n          Generally, this way of passing options to the git executable is\n          (or will be) used a lot in this class. It's a transformation of\n          python-style keyword arguments (or a `dict`) to command line arguments,\n          provided by GitPython.\n\n          A single character keyword will be prefixed by '-', multiple characters\n          by '--'. An underscore in the keyword becomes a dash. The value of the\n          keyword argument is used as the value for the corresponding command\n          line argument. Assigning a boolean creates a flag.\n\n          Examples:\n          no_commit=True => --no-commit\n          C='/my/path'   => -C /my/path\n\n        \"\"\"\n\n        if url is not None:\n            raise DeprecatedError(\n                new=\".clone() class method\",\n                version=\"0.5.0\",\n                msg=\"RF: url passed to init()\"\n            )\n\n        # So that we \"share\" control paths with git/git-annex\n        if ssh_manager:\n            ssh_manager.assure_initialized()\n\n        if not GitRepo._config_checked:\n            check_git_configured()\n            GitRepo._config_checked = True\n\n        self.realpath = realpath(path)\n        # note: we may also want to distinguish between a path to the worktree\n        # and the actual repository\n\n        # Disable automatic garbage and autopacking\n        self._GIT_COMMON_OPTIONS = ['-c', 'receive.autogc=0', '-c', 'gc.auto=0']\n        # actually no need with default GitPython db backend not in memory\n        # default_git_odbt but still allows for faster testing etc.\n        # May be eventually we would make it switchable _GIT_COMMON_OPTIONS = []\n\n        if git_opts is None:\n            git_opts = {}\n        if kwargs:\n            git_opts.update(kwargs)\n\n        self.path = path\n        self.cmd_call_wrapper = runner or GitRunner(cwd=self.path)\n        self._repo = repo\n        self._cfg = None\n\n        _valid_repo = GitRepo.is_valid_repo(path)\n        if create and not _valid_repo:\n            if repo is not None:\n                # `repo` passed with `create`, which doesn't make sense\n                raise TypeError(\"argument 'repo' must not be used with 'create'\")\n\n            try:\n                lgr.debug(\n                    \"Initialize empty Git repository at '%s'%s\",\n                    path,\n                    ' %s' % git_opts if git_opts else '')\n                self._repo = self.cmd_call_wrapper(gitpy.Repo.init, path,\n                                                   mkdir=True,\n                                                   odbt=default_git_odbt,\n                                                   **git_opts)\n            except GitCommandError as e:\n                lgr.error(exc_str(e))\n                raise\n        else:\n            # Note: We used to call gitpy.Repo(path) here, which potentially\n            # raised NoSuchPathError or InvalidGitRepositoryError. This is\n            # used by callers of GitRepo.__init__() to detect whether we have a\n            # valid repo at `path`. Now, with switching to lazy loading property\n            # `repo`, we detect those cases without instantiating a\n            # gitpy.Repo().\n\n            if not exists(path):\n                raise NoSuchPathError(path)\n            if not _valid_repo:\n                raise InvalidGitRepositoryError(path)\n\n        # inject git options into GitPython's git call wrapper:\n        # Note: `None` currently can happen, when Runner's protocol prevents\n        # calls above from being actually executed (DryRunProtocol)\n        if self._repo is not None:\n            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS\n\n        # with DryRunProtocol path might still not exist\n        if exists(self.realpath):\n            self.inode = os.stat(self.realpath).st_ino\n        else:\n            self.inode = None\n\n    @property\n    def repo(self):\n        # with DryRunProtocol path not exist\n        if exists(self.realpath):\n            inode = os.stat(self.realpath).st_ino\n        else:\n            inode = None\n        if self.inode != inode:\n            # reset background processes invoked by GitPython:\n            self._repo.git.clear_cache()\n            self.inode = inode\n\n        if self._repo is None:\n            # Note, that this may raise GitCommandError, NoSuchPathError,\n            # InvalidGitRepositoryError:\n            self._repo = self.cmd_call_wrapper(Repo, self.path)\n            lgr.log(8, \"Using existing Git repository at %s\", self.path)\n\n        # inject git options into GitPython's git call wrapper:\n        # Note: `None` currently can happen, when Runner's protocol prevents\n        # call of Repo(path) above from being actually executed (DryRunProtocol)\n        if self._repo is not None:\n            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS\n\n        return self._repo\n\n    @classmethod\n    def clone(cls, url, path, *args, **kwargs):\n        \"\"\"Clone url into path\n\n        Provides workarounds for known issues (e.g.\n        https://github.com/datalad/datalad/issues/785)\n\n        Parameters\n        ----------\n        url : str\n        path : str\n        expect_fail : bool\n          Either expect that command might fail, so error should be logged then\n          at DEBUG level instead of ERROR\n        \"\"\"\n\n        if 'repo' in kwargs:\n            raise TypeError(\"argument 'repo' conflicts with cloning\")\n            # TODO: what about 'create'?\n\n        expect_fail = kwargs.pop('expect_fail', False)\n        # fail early on non-empty target:\n        from os import listdir\n        if exists(path) and listdir(path):\n            # simulate actual GitCommandError:\n            lgr.warning(\"destination path '%s' already exists and is not an \"\n                        \"empty directory.\" % path)\n            raise GitCommandError(\n                ['git', 'clone', '-v', url, path],\n                128,\n                \"fatal: destination path '%s' already exists and is not an \"\n                \"empty directory.\" % path)\n        else:\n            # protect against cloning into existing and obviously dangling\n            # instance for that location\n            try:\n                del cls._unique_instances[path]\n            except KeyError:\n                # didn't exist - all fine\n                pass\n\n        # try to get a local path from `url`:\n        try:\n            if not isinstance(url, RI):\n                url = RI(url).localpath\n            else:\n                url = url.localpath\n        except ValueError:\n            pass\n\n        if is_ssh(url):\n            ssh_manager.get_connection(url).open()\n            # TODO: with git <= 2.3 keep old mechanism:\n            #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n            env = GitRepo.GIT_SSH_ENV\n        else:\n            env = None\n        ntries = 5  # 3 is not enough for robust workaround\n        for trial in range(ntries):\n            try:\n                lgr.debug(\"Git clone from {0} to {1}\".format(url, path))\n                repo = gitpy.Repo.clone_from(url, path, env=env,\n                                             odbt=default_git_odbt)\n                # Note/TODO: signature for clone from:\n                # (url, to_path, progress=None, env=None, **kwargs)\n\n                lgr.debug(\"Git clone completed\")\n                break\n            except GitCommandError as e:\n                # log here but let caller decide what to do\n                e_str = exc_str(e)\n                # see https://github.com/datalad/datalad/issues/785\n                if re.search(\"Request for .*aborted.*Unable to find\", str(e),\n                             re.DOTALL) \\\n                        and trial < ntries - 1:\n                    lgr.info(\n                        \"Hit a known issue with Git (see GH#785). Trial #%d, \"\n                        \"retrying\",\n                        trial)\n                    continue\n                    (lgr.debug if expect_fail else lgr.error)(e_str)\n                raise\n            except ValueError as e:\n                if gitpy.__version__ == '1.0.2' \\\n                        and \"I/O operation on closed file\" in str(e):\n                    # bug https://github.com/gitpython-developers/GitPython\n                    # /issues/383\n                    raise GitCommandError(\n                        \"clone has failed, telling ya\",\n                        999,  # good number\n                        stdout=\"%s already exists\" if exists(path) else \"\")\n                raise  # reraise original\n\n        gr = cls(path, *args, repo=repo, **kwargs)\n        return gr\n\n    def __del__(self):\n        # unbind possibly bound ConfigManager, to prevent all kinds of weird\n        # stalls etc\n        self._cfg = None\n        # Make sure to flush pending changes, especially close batch processes\n        # (internal `git cat-file --batch` by GitPython)\n        try:\n            if hasattr(self, 'repo') and exists(self.path) \\\n                    and self.repo is not None:\n                # gc might be late, so the (temporary)\n                # repo doesn't exist on FS anymore\n\n                self.repo.git.clear_cache()\n                if exists(opj(self.path, '.git')):  # don't try to write otherwise\n                    self.repo.index.write()\n        except InvalidGitRepositoryError:\n            # might have being removed and no longer valid\n            pass\n\n    def __repr__(self):\n        return \"<GitRepo path=%s (%s)>\" % (self.path, type(self))\n\n    def __eq__(self, obj):\n        \"\"\"Decides whether or not two instances of this class are equal.\n\n        This is done by comparing the base repository path.\n        \"\"\"\n        return self.realpath == obj.realpath\n\n    @classmethod\n    def is_valid_repo(cls, path):\n        \"\"\"Returns if a given path points to a git repository\"\"\"\n        return exists(opj(path, '.git'))\n\n    @property\n    def config(self):\n        \"\"\"Get an instance of the parser for the persistent repository\n        configuration.\n\n        Note: This allows to also read/write .datalad/config,\n        not just .git/config\n\n        Returns\n        -------\n        ConfigManager\n        \"\"\"\n        if self._cfg is None:\n            # associate with this dataset and read the entire config hierarchy\n            self._cfg = ConfigManager(dataset=self, dataset_only=False)\n        return self._cfg\n\n    def is_with_annex(self, only_remote=False):\n        \"\"\"Return True if GitRepo (assumed) at the path has remotes with git-annex branch\n\n        Parameters\n        ----------\n        only_remote: bool, optional\n            Check only remote (no local branches) for having git-annex branch\n        \"\"\"\n        return any((b.endswith('/git-annex') or\n                    'annex/direct' in b\n                    for b in self.get_remote_branches())) or \\\n            ((not only_remote) and\n             any((b == 'git-annex' or 'annex/direct' in b\n                  for b in self.get_branches())))\n\n    @classmethod\n    def get_toppath(cls, path, follow_up=True, git_options=None):\n        \"\"\"Return top-level of a repository given the path.\n\n        Parameters\n        -----------\n        follow_up : bool\n          If path has symlinks -- they get resolved by git.  If follow_up is\n          True, we will follow original path up until we hit the same resolved\n          path.  If no such path found, resolved one would be returned.\n        git_options: list of str\n          options to be passed to the git rev-parse call\n\n        Return None if no parent directory contains a git repository.\n        \"\"\"\n        cmd = ['git']\n        if git_options:\n            cmd.extend(git_options)\n        cmd += [\"rev-parse\", \"--show-toplevel\"]\n        try:\n            toppath, err = GitRunner().run(\n                cmd,\n                cwd=path,\n                log_stdout=True, log_stderr=True,\n                expect_fail=True, expect_stderr=True)\n            toppath = toppath.rstrip('\\n\\r')\n        except CommandError:\n            return None\n        except OSError:\n            toppath = GitRepo.get_toppath(dirname(path), follow_up=follow_up,\n                                          git_options=git_options)\n\n        if follow_up:\n            path_ = path\n            path_prev = \"\"\n            while path_ and path_ != path_prev:  # on top /.. = /\n                if realpath(path_) == toppath:\n                    toppath = path_\n                    break\n                path_prev = path_\n                path_ = dirname(path_)\n\n        return toppath\n\n    # classmethod so behavior could be tuned in derived classes\n    @classmethod\n    def _get_added_files_commit_msg(cls, files):\n        if not files:\n            return \"No files were added\"\n        msg = \"Added %d file\" % len(files)\n        if len(files) > 1:\n            msg += \"s\"\n        return msg + '\\n\\nFiles:\\n' + '\\n'.join(files)\n\n    @normalize_paths\n    def add(self, files, commit=False, msg=None, git=True, git_options=None,\n            _datalad_msg=False, update=False):\n        \"\"\"Adds file(s) to the repository.\n\n        Parameters\n        ----------\n        files: list\n          list of paths to add\n        commit: bool\n          whether or not to directly commit\n        msg: str\n          commit message in case `commit=True`. A default message, containing\n          the list of files that were added, is created by default.\n        git: bool\n          somewhat ugly construction to be compatible with AnnexRepo.add();\n          has to be always true.\n        update: bool\n          --update option for git-add. From git's manpage:\n           Update the index just where it already has an entry matching\n           <pathspec>. This removes as well as modifies index entries to match\n           the working tree, but adds no new files.\n\n           If no <pathspec> is given when --update option is used, all tracked\n           files in the entire working tree are updated (old versions of Git\n           used to limit the update to the current directory and its\n           subdirectories).\n        \"\"\"\n\n        # TODO: git_options is used as options for the git-add here,\n        # instead of options to the git executable => rename for consistency\n\n        # needs to be True - see docstring:\n        assert(git)\n\n        files = _remove_empty_items(files)\n        out = []\n\n        if files or git_options or update:\n            try:\n                # without --verbose git 2.9.3  add does not return anything\n                add_out = self._git_custom_command(\n                    files,\n                    ['git', 'add'] + assure_list(git_options) +\n                    to_options(update=update) + ['--verbose']\n                )\n                # get all the entries\n                out = self._process_git_get_output(*add_out)\n                # Note: as opposed to git cmdline, force is True by default in\n                #       gitpython, which would lead to add things, that are\n                #       ignored or excluded otherwise\n                # 2. Note: There is an issue with globbing (like adding '.'),\n                #       which apparently doesn't care for 'force' and therefore\n                #       adds '.git/...'. May be it's expanded at the wrong\n                #       point in time or sth. like that.\n                # For now, use direct call to git add.\n                #self.cmd_call_wrapper(self.repo.index.add, files, write=True,\n                #                      force=False)\n                # TODO: May be make use of 'fprogress'-option to indicate\n                # progress\n                # But then, we don't have it for git-annex add, anyway.\n                #\n                # TODO: Is write=True a reasonable way to do it?\n                # May be should not write until success of operation is\n                # confirmed?\n                # What's best in case of a list of files?\n            except OSError as e:\n                lgr.error(\"add: %s\" % e)\n                raise\n\n        else:\n            lgr.warning(\"add was called with empty file list and no options.\")\n\n        if commit:\n            if msg is None:\n                msg = self._get_added_files_commit_msg(files)\n            self.commit(msg=msg, _datalad_msg=_datalad_msg)\n\n        # Make sure return value from GitRepo is consistent with AnnexRepo\n        # currently simulating similar return value, assuming success\n        # for all files:\n        # TODO: Make return values consistent across both *Repo classes!\n        return out\n\n    @staticmethod\n    def _process_git_get_output(stdout, stderr=None):\n        \"\"\"Given both outputs (stderr is ignored atm) of git add - process it\n\n        Primarily to centralize handling in both indirect annex and direct\n        modes when ran through proxy\n        \"\"\"\n        return [{u'file': f, u'success': True}\n                for f in re.findall(\"'(.*)'[\\n$]\", stdout)]\n\n    @normalize_paths(match_return_type=False)\n    def remove(self, files, recursive=False, **kwargs):\n        \"\"\"Remove files.\n\n        Calls git-rm.\n\n        Parameters\n        ----------\n        files: str\n          list of paths to remove\n        recursive: False\n          either to allow recursive removal from subdirectories\n        kwargs:\n          see `__init__`\n\n        Returns\n        -------\n        [str]\n          list of successfully removed files.\n        \"\"\"\n\n        files = _remove_empty_items(files)\n\n        if recursive:\n            kwargs['r'] = True\n        stdout, stderr = self._git_custom_command(\n            files, ['git', 'rm'] + to_options(**kwargs))\n\n        # output per removed file is expected to be \"rm 'PATH'\":\n        return [line.strip()[4:-1] for line in stdout.splitlines()]\n\n        #return self.repo.git.rm(files, cached=False, **kwargs)\n\n    def precommit(self):\n        \"\"\"Perform pre-commit maintenance tasks\n        \"\"\"\n        # All GitPython commands should take care about flushing index\n        # whenever they modify it, so we would not care to do anything\n        # if self.repo is not None and exists(opj(self.path, '.git')):  # don't try to write otherwise:\n        #     # flush possibly cached in GitPython changes to index:\n        #     # if self.repo.git:\n        #     #     sys.stderr.write(\"CLEARING\\n\")\n        #     #     self.repo.git.clear_cache()\n        #     self.repo.index.write()\n        pass\n\n    @staticmethod\n    def _get_prefixed_commit_msg(msg):\n        DATALAD_PREFIX = \"[DATALAD]\"\n        return DATALAD_PREFIX if not msg else \"%s %s\" % (DATALAD_PREFIX, msg)\n\n    def commit(self, msg=None, options=None, _datalad_msg=False, careless=True,\n               files=None):\n        \"\"\"Commit changes to git.\n\n        Parameters\n        ----------\n        msg: str, optional\n          commit-message\n        options: list of str, optional\n          cmdline options for git-commit\n        _datalad_msg: bool, optional\n          To signal that commit is automated commit by datalad, so\n          it would carry the [DATALAD] prefix\n        careless: bool, optional\n          if False, raise when there's nothing actually committed;\n          if True, don't care\n        files: list of str, optional\n          path(s) to commit\n        \"\"\"\n\n        self.precommit()\n\n        if _datalad_msg:\n            msg = self._get_prefixed_commit_msg(msg)\n\n        if not msg:\n            if options:\n                if \"--allow-empty-message\" not in options:\n                        options.append(\"--allow-empty-message\")\n            else:\n                options = [\"--allow-empty-message\"]\n\n        # Note: We used to use a direct call to git only if there were options,\n        # since we can't pass all possible options to gitpython's implementation\n        # of commit.\n        # But there's an additional issue. GitPython implements commit in a way,\n        # that it might create a new commit, when a direct call wouldn't. This\n        # was discovered with a modified (but unstaged) submodule, leading to a\n        # commit, that apparently did nothing - git status still showed the very\n        # same thing afterwards. But a commit was created nevertheless:\n        # diff --git a/sub b/sub\n        # --- a/sub\n        # +++ b/sub\n        # @@ -1 +1 @@\n        # -Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f\n        # +Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f-dirty\n        #\n        # Therefore, for now always use direct call.\n        # TODO: Figure out, what exactly is going on with gitpython here\n\n        cmd = ['git', 'commit'] + ([\"-m\", msg if msg else \"\"])\n        if options:\n            cmd.extend(options)\n        lgr.debug(\"Committing via direct call of git: %s\" % cmd)\n\n        try:\n            self._git_custom_command(files, cmd,\n                                     expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if 'nothing to commit' in e.stdout:\n                if careless:\n                    lgr.debug(\"nothing to commit in {}. \"\n                              \"Ignored.\".format(self))\n                else:\n                    raise\n            elif 'no changes added to commit' in e.stdout or \\\n                    'nothing added to commit' in e.stdout:\n                if careless:\n                    lgr.debug(\"no changes added to commit in {}. \"\n                              \"Ignored.\".format(self))\n                else:\n                    raise\n            elif \"did not match any file(s) known to git.\" in e.stderr:\n                # TODO: Improve FileNotInXXXXError classes to better deal with\n                # multiple files; Also consider PathOutsideRepositoryError\n                raise FileNotInRepositoryError(cmd=e.cmd,\n                                               msg=\"File(s) unknown to git\",\n                                               code=e.code,\n                                               filename=linesep.join(\n                                            [l for l in e.stderr.splitlines()\n                                             if l.startswith(\"pathspec\")]))\n            else:\n                raise\n\n    def get_indexed_files(self):\n        \"\"\"Get a list of files in git's index\n\n        Returns\n        -------\n        list\n            list of paths rooting in git's base dir\n        \"\"\"\n\n        return [x[0] for x in self.cmd_call_wrapper(\n            self.repo.index.entries.keys)]\n\n    def get_hexsha(self, object=None):\n        \"\"\"Return a hexsha for a given object. If None - of current HEAD\n\n        Parameters\n        ----------\n        object: str, optional\n          Any type of Git object identifier. See `git show`.\n        \"\"\"\n        cmd = ['git', 'show', '--no-patch', \"--format=%H\"]\n        if object:\n            cmd.append(object)\n        # make sure Git takes our argument as a revision\n        cmd.append('--')\n        try:\n            stdout, stderr = self._git_custom_command(\n                '', cmd, expect_stderr=True, expect_fail=True)\n        except CommandError as e:\n            if 'bad revision' in e.stderr:\n                raise ValueError(\"Unknown object identifier: %s\" % object)\n            elif 'does not have any commits yet' in e.stderr:\n                return None\n            else:\n                raise e\n        stdout = stdout.splitlines()\n        assert(len(stdout) == 1)\n        return stdout[0]\n\n    def get_merge_base(self, treeishes):\n        \"\"\"Get a merge base hexsha\n\n        Parameters\n        ----------\n        treeishes: str or list of str\n          List of treeishes (branches, hexshas, etc) to determine the merge\n          base of. If a single value provided, returns merge_base with the\n          current branch.\n\n        Returns\n        -------\n        str or None\n          If no merge-base for given commits, or specified treeish doesn't\n          exist, None returned\n        \"\"\"\n        if isinstance(treeishes, string_types):\n            treeishes = [treeishes]\n        if not treeishes:\n            raise ValueError(\"Provide at least a single value\")\n        elif len(treeishes) == 1:\n            treeishes = treeishes + [self.get_active_branch()]\n\n        try:\n            bases = self.repo.merge_base(*treeishes)\n        except GitCommandError as exc:\n            if \"fatal: Not a valid object name\" in str(exc):\n                return None\n            raise\n\n        if not bases:\n            return None\n        assert(len(bases) == 1)  # we do not do 'all' yet\n        return bases[0].hexsha\n\n    def get_committed_date(self, branch=None):\n        \"\"\"Get the date stamp of the last commit (in a branch). None if no commit\"\"\"\n        try:\n            commit = next(\n                self.get_branch_commits(branch\n                                        or self.get_active_branch())\n            )\n        except Exception as exc:\n            lgr.debug(\"Got exception while trying to get last commit: %s\",\n                      exc_str(exc))\n            return None\n        return commit.committed_date\n\n    def get_active_branch(self):\n        try:\n            branch = self.repo.active_branch.name\n        except TypeError as e:\n            if \"HEAD is a detached symbolic reference\" in str(e):\n                lgr.debug(\"detached HEAD in {0}\".format(self))\n                return None\n            else:\n                raise\n        return branch\n\n    def get_branches(self):\n        \"\"\"Get all branches of the repo.\n\n        Returns\n        -------\n        [str]\n            Names of all branches of this repository.\n        \"\"\"\n\n        return [branch.name for branch in self.repo.branches]\n\n    def get_remote_branches(self):\n        \"\"\"Get all branches of all remotes of the repo.\n\n        Returns\n        -----------\n        [str]\n            Names of all remote branches.\n        \"\"\"\n        # TODO: Reconsider melting with get_branches()\n\n        # TODO: treat entries like this: origin/HEAD -> origin/master'\n        # currently this is done in collection\n\n        # For some reason, this is three times faster than the version below:\n        remote_branches = list()\n        for remote in self.repo.remotes:\n            try:\n                for ref in remote.refs:\n                    remote_branches.append(ref.name)\n            except AssertionError as e:\n                if str(e).endswith(\"did not have any references\"):\n                    # this will happen with git annex special remotes\n                    pass\n                else:\n                    raise e\n        return remote_branches\n        # return [branch.strip() for branch in\n        #         self.repo.git.branch(r=True).splitlines()]\n\n    def get_remotes(self, with_urls_only=False):\n        \"\"\"Get known remotes of the repository\n\n        Parameters\n        ----------\n        with_urls_only : bool, optional\n          return only remotes which have urls\n\n        Returns\n        -------\n        remotes : list of str\n          List of names of the remotes\n        \"\"\"\n\n        # Note: read directly from config and spare instantiation of gitpy.Repo\n        # since we need this in AnnexRepo constructor. Furthermore gitpy does it\n        # pretty much the same way and the use of a Repo instance seems to have\n        # no reason other than a nice object oriented look.\n        from datalad.utils import unique\n\n        self.config.reload()\n        remotes = unique([x[7:] for x in self.config.sections()\n                          if x.startswith(\"remote.\")])\n\n        if with_urls_only:\n            remotes = [\n                r for r in remotes\n                if self.config.get('remote.%s.url' % r)\n            ]\n        return remotes\n\n    def get_files(self, branch=None):\n        \"\"\"Get a list of files in git.\n\n        Lists the files in the (remote) branch.\n\n        Parameters\n        ----------\n        branch: str\n          Name of the branch to query. Default: active branch.\n\n        Returns\n        -------\n        [str]\n          list of files.\n        \"\"\"\n        # TODO: RF codes base and melt get_indexed_files() in\n\n        if branch is None:\n            # active branch can be queried way faster:\n            return self.get_indexed_files()\n        else:\n            return [item.path for item in self.repo.tree(branch).traverse()\n                    if isinstance(item, Blob)]\n\n    def get_file_content(self, file_, branch='HEAD'):\n        \"\"\"\n\n        Returns\n        -------\n        [str]\n          content of file_ as a list of lines.\n        \"\"\"\n        content_str = self.repo.commit(branch).tree[file_].data_stream.read()\n\n        # in python3 a byte string is returned. Need to convert it:\n        from six import PY3\n        if PY3:\n            conv_str = u''\n            for b in bytes(content_str):\n                conv_str += chr(b)\n            return conv_str.splitlines()\n        else:\n            return content_str.splitlines()\n        # TODO: keep splitlines?\n\n    def _get_files_history(self, files, branch='HEAD'):\n        \"\"\"\n\n        Parameters\n        ----------\n        files: list\n          list of files, only commits with queried files are considered\n        branch: str\n          Name of the branch to query. Default: HEAD.\n\n        Returns\n        -------\n        [iterator]\n        yielding Commit items generator from branch history associated with files\n        \"\"\"\n        return gitpy.objects.commit.Commit.iter_items(self.repo, branch, paths=files)\n\n    def _get_remotes_having_commit(self, commit_hexsha, with_urls_only=True):\n        \"\"\"Traverse all branches of the remote and check if commit in any of their ancestry\n\n        It is a generator yielding names of the remotes\n        \"\"\"\n        out, err = self._git_custom_command(\n            '', 'git branch -r --contains ' + commit_hexsha\n        )\n        # sanitize a bit (all the spaces and new lines)\n        remote_branches = [\n            b  # could be origin/HEAD -> origin/master, we just skip ->\n            for b in filter(bool, out.split())\n            if b != '->'\n        ]\n        return [\n            remote\n            for remote in self.get_remotes(with_urls_only=with_urls_only)\n            if any(rb.startswith(remote + '/') for rb in remote_branches)\n        ]\n\n    def _gitpy_custom_call(self, cmd, cmd_args=None, cmd_options=None,\n                           git_options=None, env=None,\n\n                           # 'old' options for Runner; not sure yet, which of\n                           # them are actually still needed:\n                           log_stdout=True, log_stderr=True, log_online=False,\n                           expect_stderr=True, cwd=None,\n                           shell=None, expect_fail=False):\n\n        \"\"\"Helper to call GitPython's wrapper for git calls.\n\n        The used instance of `gitpy.Git` is bound to the repository,\n        which determines its working directory.\n        This is used for adhoc implementation of a git command and to\n        demonstrate how to use it in more specific implementations.\n\n        Note\n        ----\n        Aims to replace the use of datalad's `Runner` class for direct git\n        calls. (Currently the `_git_custom_command()` method).\n        Therefore mimicking its behaviour during RF'ing.\n\n        Parameters\n        ----------\n        cmd: str\n          the native git command to call\n        cmd_args: list of str\n          arguments to the git command\n        cmd_options: dict\n          options for the command as key, value pair\n          (this transformation, needs some central place to document)\n        git_options: dict\n          options for the git executable as key, value pair\n          (see above)\n        env: dict\n          environment vaiables to temporarily set for this call\n\n        TODO\n        ----\n        Example\n\n        Returns\n        -------\n        (stdout, stderr)\n        \"\"\"\n\n        # TODO: Reconsider when to log/stream what (stdout, stderr) and/or\n        # fully implement the behaviour of `Runner`\n\n        if log_online:\n            raise NotImplementedError(\"option 'log_online' not implemented yet\")\n        with_exceptions = not expect_fail\n        if cwd:\n            # the gitpy.cmd.Git instance, bound to this repository doesn't allow\n            # to explicitly set the working dir, except for using os.getcwd\n            raise NotImplementedError(\"working dir is a read-only property\")\n\n        _tmp_shell = gitpy.cmd.Git.USE_SHELL\n        gitpy.cmd.Git.USE_SHELL = shell\n\n        if env is None:\n            env = {}\n        if git_options is None:\n            git_options = {}\n        if cmd_options is None:\n            cmd_options = {}\n        cmd_options.update({'with_exceptions': with_exceptions,\n                            'with_extended_output': True})\n\n        # TODO: _GIT_COMMON_OPTIONS!\n\n        with self.repo.git.custom_environment(**env):\n            try:\n                status, std_out, std_err = \\\n                    self.repo.git(**git_options).__getattr__(cmd)(\n                        cmd_args, **cmd_options)\n            except GitCommandError as e:\n                # For now just reraise. May be raise CommandError instead\n                raise\n            finally:\n                gitpy.cmd.Git.USE_SHELL = _tmp_shell\n\n        if not expect_stderr and std_err:\n            lgr.error(\"Unexpected output on stderr: %s\" % std_err)\n            raise CommandError\n        if log_stdout:\n            for line in std_out.splitlines():\n                lgr.debug(\"stdout| \" + line)\n        if log_stderr:\n            for line in std_err.splitlines():\n                lgr.log(level=logging.DEBUG if expect_stderr else logging.ERROR,\n                        msg=\"stderr| \" + line)\n\n        return std_out, std_err\n\n    @normalize_paths(match_return_type=False)\n    def _git_custom_command(self, files, cmd_str,\n                            log_stdout=True, log_stderr=True, log_online=False,\n                            expect_stderr=True, cwd=None, env=None,\n                            shell=None, expect_fail=False):\n        \"\"\"Allows for calling arbitrary commands.\n\n        Helper for developing purposes, i.e. to quickly implement git commands\n        for proof of concept without the need to figure out, how this is done\n        via GitPython.\n\n        Parameters\n        ----------\n        files: list of files\n        cmd_str: str or list\n            arbitrary command str. `files` is appended to that string.\n\n        Returns\n        -------\n        stdout, stderr\n        \"\"\"\n\n        # ensure cmd_str becomes a well-formed list:\n        if isinstance(cmd_str, string_types):\n            if files and not cmd_str.strip().endswith(\" --\"):\n                cmd_str += \" --\"\n            cmd_str = shlex.split(cmd_str, posix=not on_windows)\n        else:\n            if files and cmd_str[-1] != '--':\n                cmd_str.append('--')\n\n        cmd = cmd_str + files\n\n        assert(cmd[0] == 'git')\n        cmd = cmd[:1] + self._GIT_COMMON_OPTIONS + cmd[1:]\n\n        from .exceptions import GitIgnoreError\n\n        try:\n            out, err = self.cmd_call_wrapper.run(\n                cmd,\n                log_stderr=log_stderr,\n                log_stdout=log_stdout,\n                log_online=log_online,\n                expect_stderr=expect_stderr,\n                cwd=cwd,\n                env=env,\n                shell=shell,\n                expect_fail=expect_fail)\n        except CommandError as e:\n            ignored = re.search(GitIgnoreError.pattern, e.stderr)\n            if ignored:\n                raise GitIgnoreError(cmd=e.cmd, msg=e.stderr,\n                                     code=e.code, stdout=e.stdout,\n                                     stderr=e.stderr,\n                                     paths=ignored.groups()[0].splitlines())\n            raise\n        return out, err\n\n# TODO: --------------------------------------------------------------------\n\n    def add_remote(self, name, url, options=None):\n        \"\"\"Register remote pointing to a url\n        \"\"\"\n        cmd = ['git', 'remote', 'add']\n        if options:\n            cmd += options\n        cmd += [name, url]\n\n        result = self._git_custom_command('', cmd)\n        self.config.reload()\n        return result\n\n    def remove_remote(self, name):\n        \"\"\"Remove existing remote\n        \"\"\"\n\n        # TODO: testing and error handling!\n        from .exceptions import RemoteNotAvailableError\n        try:\n            out, err = self._git_custom_command(\n                '', ['git', 'remote', 'remove', name])\n        except CommandError as e:\n            if 'fatal: No such remote' in e.stderr:\n                raise RemoteNotAvailableError(name,\n                                              cmd=\"git remote remove\",\n                                              msg=\"No such remote\",\n                                              stdout=out,\n                                              stderr=err)\n            else:\n                raise e\n\n        # TODO: config.reload necessary?\n        self.config.reload()\n        return\n\n    def update_remote(self, name=None, verbose=False):\n        \"\"\"\n        \"\"\"\n        options = [\"-v\"] if verbose else []\n        name = [name] if name else []\n        self._git_custom_command(\n            '', ['git', 'remote'] + name + ['update'] + options,\n            expect_stderr=True\n        )\n\n    # TODO: centralize all the c&p code in fetch, pull, push\n    # TODO: document **kwargs passed to gitpython\n    def fetch(self, remote=None, refspec=None, progress=None, all_=False,\n              **kwargs):\n        \"\"\"Fetches changes from a remote (or all_ remotes).\n\n        Parameters\n        ----------\n        remote: str\n          (optional) name of the remote to fetch from. If no remote is given and\n          `all_` is not set, the tracking branch is fetched.\n        refspec: str\n          (optional) refspec to fetch.\n        progress:\n          passed to gitpython. TODO: Figure it out, make consistent use of it\n          and document it.\n        all_: bool\n          fetch all_ remotes (and all_ of their branches).\n          Fails if `remote` was given.\n        kwargs:\n          passed to gitpython. TODO: Figure it out, make consistent use of it\n          and document it.\n\n        Returns\n        -------\n        list\n            FetchInfo objects of the items fetched from remote\n        \"\"\"\n        # TODO: options=> **kwargs):\n        # Note: Apparently there is no explicit (fetch --all) in gitpython,\n        #       but fetch is always bound to a certain remote instead.\n        #       Therefore implement it on our own:\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            if all_:\n                remotes_to_fetch = [\n                    self.repo.remote(r)\n                    for r in self.get_remotes(with_urls_only=True)\n                ]\n            else:\n                # No explicit remote to fetch.\n                # => get tracking branch:\n                tb_remote, refspec = self.get_tracking_branch()\n                if tb_remote is not None:\n                    remotes_to_fetch = [self.repo.remote(tb_remote)]\n                else:\n                    # No remote, no tracking branch\n                    # => fail\n                    raise ValueError(\"Neither a remote is specified to fetch \"\n                                     \"from nor a tracking branch is set up.\")\n        else:\n            remotes_to_fetch = [self.repo.remote(remote)]\n\n        fi_list = []\n        for rm in remotes_to_fetch:\n            fetch_url = \\\n                self.config.get('remote.%s.fetchurl' % rm.name,\n                                self.config.get('remote.%s.url' % rm.name,\n                                                None))\n            if fetch_url is None:\n                lgr.debug(\"Remote %s has no URL\", rm)\n                return []\n\n            if is_ssh(fetch_url):\n                ssh_manager.get_connection(fetch_url).open()\n                # TODO: with git <= 2.3 keep old mechanism:\n                #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                    fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)\n                    # TODO: progress +kwargs\n            else:\n                fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n\n        # TODO: fetch returns a list of FetchInfo instances. Make use of it.\n        return fi_list\n\n    def pull(self, remote=None, refspec=None, progress=None, **kwargs):\n        \"\"\"See fetch\n        \"\"\"\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            # No explicit remote to pull from.\n            # => get tracking branch:\n            tb_remote, refspec = self.get_tracking_branch()\n            if tb_remote is not None:\n                remote = self.repo.remote(tb_remote)\n            else:\n                # No remote, no tracking branch\n                # => fail\n                raise ValueError(\"No remote specified to pull from nor a \"\n                                 \"tracking branch is set up.\")\n\n        else:\n            remote = self.repo.remote(remote)\n\n        fetch_url = \\\n            remote.config_reader.get(\n                'fetchurl' if remote.config_reader.has_option('fetchurl')\n                else 'url')\n        if is_ssh(fetch_url):\n            ssh_manager.get_connection(fetch_url).open()\n            # TODO: with git <= 2.3 keep old mechanism:\n            #       with remote.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n            with remote.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                return remote.pull(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n        else:\n            return remote.pull(refspec=refspec, progress=progress, **kwargs)\n            # TODO: progress +kwargs\n\n    def push(self, remote=None, refspec=None, progress=None, all_remotes=False,\n             **kwargs):\n        \"\"\"Push to remote repository\n\n        Parameters\n        ----------\n        remote: str\n          name of the remote to push to\n        refspec: str\n          specify what to push\n        progress:\n          TODO\n        all_remotes: bool\n          if set to True push to all remotes. Conflicts with `remote` not being\n          None.\n        kwargs: dict\n          options to pass to `git push`\n\n        Returns\n        -------\n        list\n            PushInfo objects of the items pushed to remote\n        \"\"\"\n\n        if remote is None:\n            if refspec is not None:\n                # conflicts with using tracking branch or fetch all remotes\n                # For now: Just fail.\n                # TODO: May be check whether it fits to tracking branch\n                raise ValueError(\"refspec specified without a remote. (%s)\" %\n                                 refspec)\n            if all_remotes:\n                remotes_to_push = self.repo.remotes\n            else:\n                # Nothing explicitly specified. Just call `git push` and let git\n                # decide what to do would be an option. But:\n                # - without knowing the remote and its URL we cannot provide\n                #   shared SSH connection\n                # - we lose ability to use GitPython's progress info and return\n                #   values\n                #   (the latter would be solvable:\n                #    Provide a Repo.push() method for GitPython, copying\n                #    Remote.push() for similar return value and progress\n                #    (also: fetch, pull)\n\n                # Do what git would do:\n                # 1. branch.*.remote for current branch or 'origin' as default\n                #    if config is missing\n                # 2. remote.*.push or push.default\n\n                # TODO: check out \"same procedure\" for fetch/pull\n\n                tb_remote, refspec = self.get_tracking_branch()\n                if tb_remote is None:\n                    tb_remote = 'origin'\n                remotes_to_push = [self.repo.remote(tb_remote)]\n                # use no refspec; let git find remote.*.push or push.default on\n                # its own\n\n        else:\n            if all_remotes:\n                lgr.warning(\"Option 'all_remotes' conflicts with specified \"\n                            \"remote '%s'. Option ignored.\")\n            remotes_to_push = [self.repo.remote(remote)]\n\n        pi_list = []\n        for rm in remotes_to_push:\n            push_url = \\\n                rm.config_reader.get('pushurl'\n                                     if rm.config_reader.has_option('pushurl')\n                                     else 'url')\n            if is_ssh(push_url):\n                ssh_manager.get_connection(push_url).open()\n                # TODO: with git <= 2.3 keep old mechanism:\n                #       with rm.repo.git.custom_environment(GIT_SSH=\"wrapper_script\"):\n                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):\n                    pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)\n                    # TODO: progress +kwargs\n            else:\n                pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)\n                # TODO: progress +kwargs\n        return pi_list\n\n    def get_remote_url(self, name, push=False):\n        \"\"\"Get the url of a remote.\n\n        Reads the configuration of remote `name` and returns its url or None,\n        if there is no url configured.\n\n        Parameters\n        ----------\n        name: str\n          name of the remote\n        push: bool\n          if True, get the pushurl instead of the fetch url.\n        \"\"\"\n\n        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')\n        return self.config.get(var, None)\n\n    def set_remote_url(self, name, url, push=False):\n        \"\"\"Set the URL a remote is pointing to\n\n        Sets the URL of the remote `name`. Requires the remote to already exist.\n\n        Parameters\n        ----------\n        name: str\n          name of the remote\n        url: str\n        push: bool\n          if True, set the push URL, otherwise the fetch URL\n        \"\"\"\n\n        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')\n        self.config.set(var, url, where='local', reload=True)\n\n    def get_branch_commits(self, branch=None, limit=None, stop=None, value=None):\n        \"\"\"Return GitPython's commits for the branch\n\n        Pretty much similar to what 'git log <branch>' does.\n        It is a generator which returns top commits first\n\n        Parameters\n        ----------\n        branch: str, optional\n          If not provided, assumes current branch\n        limit: None | 'left-only', optional\n          Limit which commits to report.  If None -- all commits (merged or not),\n          if 'left-only' -- only the commits from the left side of the tree upon\n          merges\n        stop: str, optional\n          hexsha of the commit at which stop reporting (matched one is not\n          reported either)\n        value: None | 'hexsha', optional\n          What to yield.  If None - entire commit object is yielded, if 'hexsha'\n          only its hexsha\n        \"\"\"\n\n        if not branch:\n            branch = self.get_active_branch()\n\n        try:\n            _branch = self.repo.branches[branch]\n        except IndexError:\n            raise MissingBranchError(self, branch,\n                                     [b.name for b in self.repo.branches])\n\n        fvalue = {None: lambda x: x, 'hexsha': lambda x: x.hexsha}[value]\n\n        if not limit:\n            def gen():\n                # traverse doesn't yield original commit\n                co = _branch.commit\n                yield co\n                for co_ in co.traverse():\n                    yield co_\n        elif limit == 'left-only':\n            # we need a custom implementation since couldn't figure out how to\n            # do with .traversal\n            def gen():\n                co = _branch.commit\n                while co:\n                    yield co\n                    co = co.parents[0] if co.parents else None\n        else:\n            raise ValueError(limit)\n\n        for c in gen():\n            if stop and c.hexsha == stop:\n                return\n            yield fvalue(c)\n\n    def checkout(self, name, options=None):\n        \"\"\"\n        \"\"\"\n        # TODO: May be check for the need of -b options herein?\n        cmd = ['git', 'checkout']\n        if options:\n            cmd += options\n        cmd += [str(name)]\n\n        self._git_custom_command('', cmd, expect_stderr=True)\n\n    # TODO: Before implementing annex merge, find usages and check for a needed\n    # change to call super().merge\n    def merge(self, name, options=None, msg=None, allow_unrelated=False, **kwargs):\n        if options is None:\n            options = []\n        if msg:\n            options = options + [\"-m\", msg]\n        if allow_unrelated and external_versions['cmd:git'] >= '2.9':\n            options += ['--allow-unrelated-histories']\n        self._git_custom_command(\n            '', ['git', 'merge'] + options + [name],\n            **kwargs\n        )\n\n    def remove_branch(self, branch):\n        self._git_custom_command(\n            '', ['git', 'branch', '-D', branch]\n        )\n\n    def ls_remote(self, remote, options=None):\n        if options is None:\n            options = []\n        self._git_custom_command(\n            '', ['git', 'ls-remote'] + options + [remote]\n        )\n        # TODO: Return values?\n\n    def is_dirty(self, index=True, working_tree=True, untracked_files=True,\n                 submodules=True, path=None):\n        \"\"\"Returns true if the repo is considered to be dirty\n\n        Parameters\n        ----------\n        index: bool\n          if True, consider changes to the index\n        working_tree: bool\n          if True, consider changes to the working tree\n        untracked_files: bool\n          if True, consider untracked files\n        submodules: bool\n          if True, consider submodules\n        path: str or list of str\n          path(s) to consider only\n        Returns\n        -------\n          bool\n        \"\"\"\n\n        return self.repo.is_dirty(index=index, working_tree=working_tree,\n                                  untracked_files=untracked_files,\n                                  submodules=submodules, path=path)\n\n    @property\n    def dirty(self):\n        return self.is_dirty()\n\n    @property\n    def untracked_files(self):\n        return self.repo.untracked_files\n\n    def gc(self, allow_background=False, auto=False):\n        \"\"\"Perform house keeping (garbage collection, repacking)\"\"\"\n        cmd_options = ['git']\n        if not allow_background:\n            cmd_options += ['-c', 'gc.autodetach=0']\n        cmd_options += ['gc', '--aggressive']\n        if auto:\n            cmd_options += ['--auto']\n        self._git_custom_command('', cmd_options)\n\n    def get_submodules(self, sorted_=True):\n        \"\"\"Return a list of git.Submodule instances for all submodules\"\"\"\n        # check whether we have anything in the repo. if not go home early\n        if not self.repo.head.is_valid():\n            return []\n        submodules = self.repo.submodules\n        if sorted_:\n            submodules = sorted(submodules, key=lambda x: x.path)\n        return submodules\n\n    def is_submodule_modified(self, name, options=[]):\n        \"\"\"Whether a submodule has new commits\n\n        Note: This is an adhoc method. It parses output of\n        'git submodule summary' and currently is not able to distinguish whether\n        or not this change is staged in `self` and whether this would be\n        reported 'added' or 'modified' by 'git status'.\n        Parsing isn't heavily tested yet.\n\n        Parameters\n        ----------\n        name: str\n          the submodule's name\n        options: list\n          options to pass to 'git submodule summary'\n        Returns\n        -------\n        bool\n          True if there are commits in the submodule, differing from\n          what is registered in `self`\n        --------\n        \"\"\"\n\n        out, err = self._git_custom_command('',\n                                            ['git', 'submodule', 'summary'] + \\\n                                            options + ['--', name])\n        return any([line.split()[1] == name\n                    for line in out.splitlines()\n                    if line and len(line.split()) > 1])\n\n    def add_submodule(self, path, name=None, url=None, branch=None):\n        \"\"\"Add a new submodule to the repository.\n\n        This will alter the index as well as the .gitmodules file, but will not\n        create a new commit.  If the submodule already exists, no matter if the\n        configuration differs from the one provided, the existing submodule\n        is considered as already added and no further action is performed.\n\n        Parameters\n        ----------\n        path : str\n          repository-relative path at which the submodule should be located, and\n          which will be created as required during the repository initialization.\n        name : str or None\n          name/identifier for the submodule. If `None`, the `path` will be used\n          as name.\n        url : str or None\n          git-clone compatible URL. If `None`, the repository is assumed to\n          exist, and the url of the first remote is taken instead. This is\n          useful if you want to make an existing repository a submodule of\n          another one.\n        branch : str or None\n          name of branch to be checked out in the submodule. The given branch\n          must exist in the remote repository, and will be checked out locally\n          as a tracking branch. If `None`, remote HEAD will be checked out.\n        \"\"\"\n        if name is None:\n            name = path\n        # XXX the following should do it, but GitPython will refuse to add a submodule\n        # unless you specify a URL that is configured as one of its remotes, or you\n        # specify no URL, but the repo has at least one remote.\n        # this is stupid, as for us it is valid to not have any remote, because we can\n        # still obtain the submodule from a future publication location, based on the\n        # parent\n        # gitpy.Submodule.add(self.repo, name, path, url=url, branch=branch)\n        # going git native instead\n        cmd = ['git', 'submodule', 'add', '--name', name]\n        if branch is not None:\n            cmd += ['-b', branch]\n        if url is None:\n            if not isabs(path):\n                # need to recode into a relative path \"URL\" in POSIX\n                # style, even on windows\n                url = posixpath.join(curdir, posix_relpath(path))\n            else:\n                url = path\n        cmd += [url, path]\n        self._git_custom_command('', cmd)\n        # TODO: return value\n\n    def deinit_submodule(self, path, **kwargs):\n        \"\"\"Deinit a submodule\n\n        Parameters\n        ----------\n        path: str\n            path to the submodule; relative to `self.path`\n        kwargs:\n            see `__init__`\n        \"\"\"\n\n        self._git_custom_command(path,\n                                 ['git', 'submodule', 'deinit'] +\n                                 to_options(**kwargs))\n        # TODO: return value\n\n    def update_submodule(self, path, mode='checkout', init=False):\n        \"\"\"Update a registered submodule.\n\n        This will make the submodule match what the superproject expects by\n        cloning missing submodules and updating the working tree of the\n        submodules. The \"updating\" can be done in several ways depending\n        on the value of submodule.<name>.update configuration variable, or\n        the `mode` argument.\n\n        Parameters\n        ----------\n        path : str\n          Identifies which submodule to operate on by it's repository-relative\n          path.\n        mode : {checkout, rebase, merge}\n          Update procedure to perform. 'checkout': the commit recorded in the\n          superproject will be checked out in the submodule on a detached HEAD;\n          'rebase': the current branch of the submodule will be rebased onto\n          the commit recorded in the superproject; 'merge': the commit recorded\n          in the superproject will be merged into the current branch in the\n          submodule.\n        init : bool\n          If True, initialize all submodules for which \"git submodule init\" has\n          not been called so far before updating.\n          Primarily provided for internal purposes and should not be used directly\n          since would result in not so annex-friendly .git symlinks/references\n          instead of full featured .git/ directories in the submodules\n        \"\"\"\n        cmd = ['git', 'submodule', 'update', '--%s' % mode]\n        if init:\n            cmd.append('--init')\n            subgitpath = opj(self.path, path, '.git')\n            if not exists(subgitpath):\n                # TODO:  wouldn't with --init we get all those symlink'ed .git/?\n                # At least let's warn\n                lgr.warning(\n                    \"Do not use update_submodule with init=True to avoid git creating \"\n                    \"symlinked .git/ directories in submodules\"\n                )\n            #  yoh: I thought I saw one recently but thought it was some kind of\n            #  an artifact from running submodule update --init manually at\n            #  some point, but looking at this code now I worry that it was not\n        self._git_custom_command(path, cmd)\n        # TODO: return value\n\n    def update_ref(self, ref, value, symbolic=False):\n        \"\"\"Update the object name stored in a ref \"safely\".\n\n        Just a shim for `git update-ref` call if not symbolic, and\n        `git symbolic-ref` if symbolic\n\n        Parameters\n        ----------\n        ref : str\n          Reference, such as `ref/heads/BRANCHNAME` or HEAD.\n        value : str\n          Value to update to, e.g. hexsha of a commit when updating for a\n          branch ref, or branch ref if updating HEAD\n        symbolic : None\n          To instruct if ref is symbolic, e.g. should be used in case of\n          ref=HEAD\n        \"\"\"\n        self._git_custom_command(\n            '',\n            ['git', 'symbolic-ref' if symbolic else 'update-ref', ref, value]\n        )\n\n    def tag(self, tag):\n        \"\"\"Assign a tag to current commit\n\n        Parameters\n        ----------\n        tag : str\n          Custom tag label.\n        \"\"\"\n        # TODO later to be extended with tagging particular commits and signing\n        # TODO: call in save.py complains about extensive logging. When does it\n        # happen in what way? Figure out, whether to just silence it or raise or\n        # whatever else.\n        self._git_custom_command(\n            '', ['git', 'tag', str(tag)]\n        )\n\n    def get_tags(self, output=None):\n        \"\"\"Get list of tags\n\n        Parameters\n        ----------\n        output : str, optional\n          If given, limit the return value to a list of values matching that\n          particular key of the tag properties.\n\n        Returns\n        -------\n        list\n          Each item is a dictionary with information on a tag. At present\n          this includes 'hexsha', and 'name', where the latter is the string\n          label of the tag, and the format the hexsha of the object the tag\n          is attched to. The list is sorted by commit date, with the most\n          recent commit being the last element.\n        \"\"\"\n        # TODO it would be straightforward to add more info and tweak the\n        # sorting\n        stdout, stderr = self._git_custom_command(\n            '',\n            ['git', 'tag', '--format=%(refname:strip=2)%00%(object)',\n             '--sort=*committerdate'])\n        fields = ('name', 'hexsha')\n        tags = [dict(zip(fields, line.split('\\0'))) for line in stdout.splitlines()]\n        if output:\n            return [t[output] for t in tags]\n        else:\n            return tags\n\n    def describe(self, **kwargs):\n        \"\"\" Quick and dirty implementation to call git-describe\n\n        Parameters:\n        -----------\n        kwargs:\n            transformed to cmdline options for git-describe;\n            see __init__ for description of the transformation\n        \"\"\"\n        # TODO: be more precise what failure to expect when and raise actual\n        # errors\n        try:\n            describe, outerr = self._git_custom_command(\n                [],\n                ['git', 'describe'] + to_options(**kwargs),\n                expect_fail=True)\n            return describe.strip()\n        # TODO: WTF \"catch everything\"?\n        except:\n            return None\n\n    def get_tracking_branch(self, branch=None):\n        \"\"\"Get the tracking branch for `branch` if there is any.\n\n        Parameters\n        ----------\n        branch: str\n            local branch to look up. If none is given, active branch is used.\n\n        Returns\n        -------\n        tuple\n            (remote or None, refspec or None) of the tracking branch\n        \"\"\"\n        if branch is None:\n            branch = self.get_active_branch()\n            if branch is None:\n                return None, None\n\n        track_remote = self.config.get('branch.{0}.remote'.format(branch), None)\n        track_branch = self.config.get('branch.{0}.merge'.format(branch), None)\n        return track_remote, track_branch\n\n    @property\n    def count_objects(self):\n        \"\"\"return dictionary with count, size(in KiB) information of git objects\n        \"\"\"\n\n        count_cmd = ['git', 'count-objects', '-v']\n        count_str, err = self._git_custom_command('', count_cmd)\n        count = {key: int(value)\n                 for key, value in [item.split(': ')\n                                    for item in count_str.split('\\n')\n                                    if len(item.split(': ')) == 2]}\n        return count\n\n    def get_missing_files(self):\n        \"\"\"Return a list of paths with missing files (and no staged deletion)\"\"\"\n        return [f.split('\\t')[1]\n                for f in self.repo.git.diff('--raw', '--name-status').split('\\n')\n                if f.split('\\t')[0] == 'D']\n\n    def get_deleted_files(self):\n        \"\"\"Return a list of paths with deleted files (staged deletion)\"\"\"\n        return [f.split('\\t')[1]\n                for f in self.repo.git.diff('--raw', '--name-status', '--staged').split('\\n')\n                if f.split('\\t')[0] == 'D']\n\n    def get_git_attributes(self):\n        \"\"\"Check git attribute for the current repository (not per-file support for now)\n\n        Parameters\n        ----------\n        all_: bool\n          Adds --all to git check-attr call\n\n        Returns\n        -------\n        dict:\n          attribute: value pairs\n        \"\"\"\n        out, err = self._git_custom_command([\".\"], [\"git\", \"check-attr\", \"--all\"])\n        assert not err, \"no stderr output is expected\"\n        out_split = [\n            # splitting by : would leave leading space(s)\n            [e.lstrip(' ') for e in l.split(':', 2)]\n            for l in out.split('\\n') if l\n        ]\n        assert all(o[0] == '.' for o in out_split)  # for paranoid\n        return dict(o[1:] for o in out_split)\n\n\n# TODO\n# remove submodule: nope, this is just deinit_submodule + remove\n# status?\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title**: Git Configuration Retrieval Fails When HOME Environment Variable Is Unset

**Description**:
When the `HOME` environment variable is not set, attempts to retrieve Git user configuration (`user.name` and `user.email`) fail, causing the configuration check to return `None` instead of the expected values.

For example, running the following code without a `HOME` environment variable:

```python
from datalad.support.gitrepo import check_git_configured

config = check_git_configured()
print(config)
```

**Expected Behavior**:
The `check_git_configured` function should return a dictionary with `'user.name'` and `'user.email'` entries set to predefined test values, enabling tests to run without relying on the user's actual Git configuration.

**Actual Behavior**:
The function returns `None` instead of the expected configuration dictionary, leading to failures when tests assert the presence of Git user settings.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/support/gitrepo.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface to Git via GitPython

For further information on GitPython see http://gitpython.readthedocs.org/

"""

import logging
import re
import shlex
import os
from os import linesep
from os.path import join as opj
from os.path import exists
from os.path import normpath
from os.path import isabs
from os.path import commonprefix
from os.path import relpath
from os.path import realpath
from os.path import dirname
from os.path import basename
from os.path import curdir
from os.path import pardir
from os.path import sep
import posixpath
from weakref import WeakValueDictionary


from six import string_types
from six import add_metaclass
from functools import wraps
import git as gitpy
from git.exc import GitCommandError
from git.exc import NoSuchPathError
from git.exc import InvalidGitRepositoryError
from git.objects.blob import Blob

from datalad import ssh_manager
from datalad.cmd import GitRunner
from datalad.consts import GIT_SSH_COMMAND
from datalad.dochelpers import exc_str
from datalad.config import ConfigManager
from datalad.utils import assure_list
from datalad.utils import optional_args
from datalad.utils import on_windows
from datalad.utils import getpwd
from datalad.utils import updated
from datalad.utils import posix_relpath


# imports from same module:
from .external_versions import external_versions
from .exceptions import CommandError
from .exceptions import DeprecatedError
from .exceptions import FileNotInRepositoryError
from .exceptions import MissingBranchError
from .network import RI
from .network import is_ssh
from .repo import Flyweight
from .repo import RepoInterface

# shortcuts
_curdirsep = curdir + sep
_pardirsep = pardir + sep


lgr = logging.getLogger('datalad.gitrepo')

# Override default GitPython's DB backend to talk directly to git so it doesn't
# interfere with possible operations performed by gc/repack
default_git_odbt = gitpy.GitCmdObjectDB

# TODO: Figure out how GIT_PYTHON_TRACE ('full') is supposed to be used.
# Didn't work as expected on a first try. Probably there is a neatier way to
# log Exceptions from git commands.


# TODO: ignore leading and/or trailing underscore to allow for
# python-reserved words
@optional_args
def kwargs_to_options(func, split_single_char_options=True,
                      target_kw='options'):
    """Decorator to provide convenient way to pass options to command calls.

    Parameters
    ----------
    func: Callable
        function to decorate
    split_single_char_options: bool
        whether or not to split key and value of single char keyword arguments
        into two subsequent entries of the list
    target_kw: str
        keyword argument to pass the generated list of cmdline arguments to

    Returns
    -------
    Callable
    """

    # TODO: don't overwrite options, but join

    @wraps(func)
    def newfunc(self, *args, **kwargs):
        t_kwargs = dict()
        t_kwargs[target_kw] = \
            gitpy.Git().transform_kwargs(
                split_single_char_options=split_single_char_options,
                **kwargs)
        return func(self, *args, **t_kwargs)
    return newfunc


def to_options(**kwargs):
    """Transform keyword arguments into a list of cmdline options

    Parameters
    ----------
    split_single_char_options: bool

    kwargs:

    Returns
    -------
    list
    """
    # TODO: borrow_docs!

    return gitpy.Git().transform_kwargs(**kwargs)


def _normalize_path(base_dir, path):
    """Helper to check paths passed to methods of this class.

    Checks whether `path` is beneath `base_dir` and normalizes it.
    Additionally paths are converted into relative paths with respect to
    `base_dir`, considering PWD in case of relative paths. This
    is intended to be used in repository classes, which means that
    `base_dir` usually will be the repository's base directory.

    Parameters
    ----------
    path: str
        path to be normalized
    base_dir: str
        directory to serve as base to normalized, relative paths

    Returns
    -------
    str:
        path, that is a relative path with respect to `base_dir`
    """
    if not path:
        return path

    base_dir = realpath(base_dir)  # realpath OK
    # path = normpath(path)
    # Note: disabled normpath, because it may break paths containing symlinks;
    # But we don't want to realpath relative paths, in case cwd isn't the
    # correct base.

    if isabs(path):
        # path might already be a symlink pointing to annex etc,
        # so realpath only its directory, to get "inline" with
        # realpath(base_dir) above
        path = opj(realpath(dirname(path)), basename(path))  # realpath OK
    # Executive decision was made to not do this kind of magic!
    #
    # elif commonprefix([realpath(getpwd()), base_dir]) == base_dir:
    #     # If we are inside repository, rebuilt relative paths.
    #     path = opj(realpath(getpwd()), path)
    #
    # BUT with relative curdir/pardir start it would assume relative to curdir
    #
    elif path.startswith(_curdirsep) or path.startswith(_pardirsep):
        path = normpath(opj(realpath(getpwd()), path))  # realpath OK
    else:
        # We were called from outside the repo. Therefore relative paths
        # are interpreted as being relative to self.path already.
        return path

    if commonprefix([path, base_dir]) != base_dir:
        raise FileNotInRepositoryError(msg="Path outside repository: %s"
                                           % path, filename=path)

    return relpath(path, start=base_dir)


@optional_args
def normalize_path(func):
    """Decorator to provide unified path conversion for a single file

    Unlike normalize_paths, intended to be used for functions dealing with a
    single filename at a time

    Note
    ----
    This is intended to be used within the repository classes and therefore
    returns a class method!

    The decorated function is expected to take a path at
    first positional argument (after 'self'). Additionally the class `func`
    is a member of, is expected to have an attribute 'path'.
    """

    @wraps(func)
    def newfunc(self, file_, *args, **kwargs):
        file_new = _normalize_path(self.path, file_)
        return func(self, file_new, *args, **kwargs)

    return newfunc


@optional_args
def normalize_paths(func, match_return_type=True, map_filenames_back=False,
                    serialize=False):
    """Decorator to provide unified path conversions.

    Note
    ----
    This is intended to be used within the repository classes and therefore
    returns a class method!

    The decorated function is expected to take a path or a list of paths at
    first positional argument (after 'self'). Additionally the class `func`
    is a member of, is expected to have an attribute 'path'.

    Accepts either a list of paths or a single path in a str. Passes a list
    to decorated function either way, but would return based on the value of
    match_return_type and possibly input argument.

    If a call to the wrapped function includes normalize_path and it is False
    no normalization happens for that function call (used for calls to wrapped
    functions within wrapped functions, while possible CWD is within a
    repository)

    Parameters
    ----------
    match_return_type : bool, optional
      If True, and a single string was passed in, it would return the first
      element of the output (after verifying that it is a list of length 1).
      It makes easier to work with single files input.
    map_filenames_back : bool, optional
      If True and returned value is a dictionary, it assumes to carry entries
      one per file, and then filenames are mapped back to as provided from the
      normalized (from the root of the repo) paths
    serialize : bool, optional
      Loop through files giving only a single one to the function one at a time.
      This allows to simplify implementation and interface to annex commands
      which do not take multiple args in the same call (e.g. checkpresentkey)
    """

    @wraps(func)
    def newfunc(self, files, *args, **kwargs):

        normalize = _normalize_path if kwargs.pop('normalize_paths', True) \
            else lambda rpath, filepath: filepath

        if files:
            if isinstance(files, string_types) or not files:
                files_new = [normalize(self.path, files)]
                single_file = True
            elif isinstance(files, list):
                files_new = [normalize(self.path, path) for path in files]
                single_file = False
            else:
                raise ValueError("_files_decorator: Don't know how to handle "
                                 "instance of %s." % type(files))
        else:
            single_file = None
            files_new = []

        if map_filenames_back:
            def remap_filenames(out):
                """Helper to map files back to non-normalized paths"""
                if isinstance(out, dict):
                    assert(len(out) == len(files_new))
                    files_ = [files] if single_file else files
                    mapped = out.__class__()
                    for fin, fout in zip(files_, files_new):
                        mapped[fin] = out[fout]
                    return mapped
                else:
                    return out
        else:
            remap_filenames = lambda x: x

        if serialize:  # and not single_file:
            result = [
                func(self, f, *args, **kwargs)
                for f in files_new
            ]
        else:
            result = func(self, files_new, *args, **kwargs)

        if single_file is None:
            # no files were provided, nothing we can do really
            return result
        elif (result is None) or not match_return_type or not single_file:
            # If function doesn't return anything or no denormalization
            # was requested or it was not a single file
            return remap_filenames(result)
        elif single_file:
            if len(result) != 1:
                # Magic doesn't apply
                return remap_filenames(result)
            elif isinstance(result, (list, tuple)):
                return result[0]
            elif isinstance(result, dict) and tuple(result)[0] == files_new[0]:
                # assume that returned dictionary has files as keys.
                return tuple(result.values())[0]
            else:
                # no magic can apply
                return remap_filenames(result)
        else:
            return RuntimeError("should have not got here... check logic")

    return newfunc


def check_git_configured():
    """Do a check if git is configured (user.name and user.email are set)

    Raises
    ------
    RuntimeError  if any of those two ariables are not set
    """

    check_runner = GitRunner()
    for c in 'user.name', 'user.email':
        try:
            check_runner.run(['git', 'config', '--global', c])
        except CommandError as exc:
            lgr.debug("Failed to verify that git is configured: %s",
                      exc_str(exc))
            raise RuntimeError(
                "You must configure git first (set both user.name and "
                "user.email) settings before using DataLad."
            )


def _remove_empty_items(list_):
    """Remove empty entries from list

    This is needed, since some functions of GitPython may convert
    an empty entry to '.', when used with a list of paths.

    Parameter:
    ----------
    list_: list of str

    Returns
    -------
    list of str
    """
    if not isinstance(list_, list):
        lgr.warning(
            "_remove_empty_items() called with non-list type: %s" % type(list_))
        return list_
    return [file_ for file_ in list_ if file_]


def Repo(*args, **kwargs):
    """Factory method around gitpy.Repo to consistently initiate with different
    backend
    """
    # TODO: This probably doesn't work as intended (or at least not as
    #       consistently as intended). gitpy.Repo could be instantiated by
    #       classmethods Repo.init or Repo.clone_from. In these cases 'odbt'
    #       would be needed as a paramter to these methods instead of the
    #       constructor.
    if 'odbt' not in kwargs:
        kwargs['odbt'] = default_git_odbt
    return gitpy.Repo(*args, **kwargs)


def split_remote_branch(branch):
    """Splits a remote branch's name into the name of the remote and the name
    of the branch.

    Parameters
    ----------
    branch: str
      the remote branch's name to split

    Returns
    -------
    list of str
    """
    assert '/' in branch, \
        "remote branch %s must have had a /" % branch
    assert not branch.endswith('/'), \
        "branch name with trailing / is invalid. (%s)" % branch
    return branch.split('/', 1)


@add_metaclass(Flyweight)
class GitRepo(RepoInterface):
    """Representation of a git repository

    """

    # We use our sshrun helper
    GIT_SSH_ENV = {'GIT_SSH_COMMAND': GIT_SSH_COMMAND}

    # Just a non-functional example:
    # must be implemented, since abstract in RepoInterface:
    def sth_like_file_has_content(self):
        return "Yes, if it's in the index"

    # We must check git config to have name and email set, but
    # should do it once
    _config_checked = False

    # Begin Flyweight:

    _unique_instances = WeakValueDictionary()

    @classmethod
    def _flyweight_id_from_args(cls, *args, **kwargs):

        if args:
            # to a certain degree we need to simulate an actual call to __init__
            # and make sure, passed arguments are fitting:
            # TODO: Figure out, whether there is a cleaner way to do this in a
            # generic fashion
            assert('path' not in kwargs)
            path = args[0]
            args = args[1:]
        elif 'path' in kwargs:
            path = kwargs.pop('path')
        else:
            raise TypeError("__init__() requires argument `path`")

        if path is None:
            raise AttributeError

        # Sanity check for argument `path`:
        # raise if we cannot deal with `path` at all or
        # if it is not a local thing:
        path = RI(path).localpath
        # resolve symlinks to make sure we have exactly one instance per
        # physical repository at a time
        path = realpath(path)
        kwargs['path'] = path
        return path, args, kwargs

    @classmethod
    def _flyweight_invalid(cls, id_):
        return not cls.is_valid_repo(id_)

    @classmethod
    def _flyweight_reject(cls, id_, *args, **kwargs):
        # TODO:
        # This is a temporary approach. See PR # ...
        # create = kwargs.pop('create', None)
        # kwargs.pop('path', None)
        # if create and kwargs:
        #     # we have `create` plus options other than `path`
        #     return "Call to {0}() with args {1} and kwargs {2} conflicts " \
        #            "with existing instance {3}." \
        #            "This is likely to be caused by inconsistent logic in " \
        #            "your code." \
        #            "".format(cls, args, kwargs, cls._unique_instances[id_])
        pass

    # End Flyweight

    def __init__(self, path, url=None, runner=None, create=True,
                 git_opts=None, repo=None, **kwargs):
        """Creates representation of git repository at `path`.

        Can also be used to create a git repository at `path`.

        Parameters
        ----------
        path: str
          path to the git repository; In case it's not an absolute path,
          it's relative to PWD
        url: str, optional
          DEPRECATED -- use .clone() class method
          url to the to-be-cloned repository. Requires a valid git url
          according to:
          http://www.kernel.org/pub/software/scm/git/docs/git-clone.html#URLS .
        create: bool, optional
          if true, creates a git repository at `path` if there is none. Also
          creates `path`, if it doesn't exist.
          If set to false, an exception is raised in case `path` doesn't exist
          or doesn't contain a git repository.
        repo: git.Repo, optional
          GitPython's Repo instance to (re)use if provided
        kwargs:
          keyword arguments serving as additional options to the git-init
          command. Therefore, it makes sense only if called with `create`.

          Generally, this way of passing options to the git executable is
          (or will be) used a lot in this class. It's a transformation of
          python-style keyword arguments (or a `dict`) to command line arguments,
          provided by GitPython.

          A single character keyword will be prefixed by '-', multiple characters
          by '--'. An underscore in the keyword becomes a dash. The value of the
          keyword argument is used as the value for the corresponding command
          line argument. Assigning a boolean creates a flag.

          Examples:
          no_commit=True => --no-commit
          C='/my/path'   => -C /my/path

        """

        if url is not None:
            raise DeprecatedError(
                new=".clone() class method",
                version="0.5.0",
                msg="RF: url passed to init()"
            )

        # So that we "share" control paths with git/git-annex
        if ssh_manager:
            ssh_manager.assure_initialized()

        if not GitRepo._config_checked:
            check_git_configured()
            GitRepo._config_checked = True

        self.realpath = realpath(path)
        # note: we may also want to distinguish between a path to the worktree
        # and the actual repository

        # Disable automatic garbage and autopacking
        self._GIT_COMMON_OPTIONS = ['-c', 'receive.autogc=0', '-c', 'gc.auto=0']
        # actually no need with default GitPython db backend not in memory
        # default_git_odbt but still allows for faster testing etc.
        # May be eventually we would make it switchable _GIT_COMMON_OPTIONS = []

        if git_opts is None:
            git_opts = {}
        if kwargs:
            git_opts.update(kwargs)

        self.path = path
        self.cmd_call_wrapper = runner or GitRunner(cwd=self.path)
        self._repo = repo
        self._cfg = None

        _valid_repo = GitRepo.is_valid_repo(path)
        if create and not _valid_repo:
            if repo is not None:
                # `repo` passed with `create`, which doesn't make sense
                raise TypeError("argument 'repo' must not be used with 'create'")

            try:
                lgr.debug(
                    "Initialize empty Git repository at '%s'%s",
                    path,
                    ' %s' % git_opts if git_opts else '')
                self._repo = self.cmd_call_wrapper(gitpy.Repo.init, path,
                                                   mkdir=True,
                                                   odbt=default_git_odbt,
                                                   **git_opts)
            except GitCommandError as e:
                lgr.error(exc_str(e))
                raise
        else:
            # Note: We used to call gitpy.Repo(path) here, which potentially
            # raised NoSuchPathError or InvalidGitRepositoryError. This is
            # used by callers of GitRepo.__init__() to detect whether we have a
            # valid repo at `path`. Now, with switching to lazy loading property
            # `repo`, we detect those cases without instantiating a
            # gitpy.Repo().

            if not exists(path):
                raise NoSuchPathError(path)
            if not _valid_repo:
                raise InvalidGitRepositoryError(path)

        # inject git options into GitPython's git call wrapper:
        # Note: `None` currently can happen, when Runner's protocol prevents
        # calls above from being actually executed (DryRunProtocol)
        if self._repo is not None:
            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS

        # with DryRunProtocol path might still not exist
        if exists(self.realpath):
            self.inode = os.stat(self.realpath).st_ino
        else:
            self.inode = None

    @property
    def repo(self):
        # with DryRunProtocol path not exist
        if exists(self.realpath):
            inode = os.stat(self.realpath).st_ino
        else:
            inode = None
        if self.inode != inode:
            # reset background processes invoked by GitPython:
            self._repo.git.clear_cache()
            self.inode = inode

        if self._repo is None:
            # Note, that this may raise GitCommandError, NoSuchPathError,
            # InvalidGitRepositoryError:
            self._repo = self.cmd_call_wrapper(Repo, self.path)
            lgr.log(8, "Using existing Git repository at %s", self.path)

        # inject git options into GitPython's git call wrapper:
        # Note: `None` currently can happen, when Runner's protocol prevents
        # call of Repo(path) above from being actually executed (DryRunProtocol)
        if self._repo is not None:
            self._repo.git._persistent_git_options = self._GIT_COMMON_OPTIONS

        return self._repo

    @classmethod
    def clone(cls, url, path, *args, **kwargs):
        """Clone url into path

        Provides workarounds for known issues (e.g.
        https://github.com/datalad/datalad/issues/785)

        Parameters
        ----------
        url : str
        path : str
        expect_fail : bool
          Either expect that command might fail, so error should be logged then
          at DEBUG level instead of ERROR
        """

        if 'repo' in kwargs:
            raise TypeError("argument 'repo' conflicts with cloning")
            # TODO: what about 'create'?

        expect_fail = kwargs.pop('expect_fail', False)
        # fail early on non-empty target:
        from os import listdir
        if exists(path) and listdir(path):
            # simulate actual GitCommandError:
            lgr.warning("destination path '%s' already exists and is not an "
                        "empty directory." % path)
            raise GitCommandError(
                ['git', 'clone', '-v', url, path],
                128,
                "fatal: destination path '%s' already exists and is not an "
                "empty directory." % path)
        else:
            # protect against cloning into existing and obviously dangling
            # instance for that location
            try:
                del cls._unique_instances[path]
            except KeyError:
                # didn't exist - all fine
                pass

        # try to get a local path from `url`:
        try:
            if not isinstance(url, RI):
                url = RI(url).localpath
            else:
                url = url.localpath
        except ValueError:
            pass

        if is_ssh(url):
            ssh_manager.get_connection(url).open()
            # TODO: with git <= 2.3 keep old mechanism:
            #       with rm.repo.git.custom_environment(GIT_SSH="wrapper_script"):
            env = GitRepo.GIT_SSH_ENV
        else:
            env = None
        ntries = 5  # 3 is not enough for robust workaround
        for trial in range(ntries):
            try:
                lgr.debug("Git clone from {0} to {1}".format(url, path))
                repo = gitpy.Repo.clone_from(url, path, env=env,
                                             odbt=default_git_odbt)
                # Note/TODO: signature for clone from:
                # (url, to_path, progress=None, env=None, **kwargs)

                lgr.debug("Git clone completed")
                break
            except GitCommandError as e:
                # log here but let caller decide what to do
                e_str = exc_str(e)
                # see https://github.com/datalad/datalad/issues/785
                if re.search("Request for .*aborted.*Unable to find", str(e),
                             re.DOTALL) \
                        and trial < ntries - 1:
                    lgr.info(
                        "Hit a known issue with Git (see GH#785). Trial #%d, "
                        "retrying",
                        trial)
                    continue
                    (lgr.debug if expect_fail else lgr.error)(e_str)
                raise
            except ValueError as e:
                if gitpy.__version__ == '1.0.2' \
                        and "I/O operation on closed file" in str(e):
                    # bug https://github.com/gitpython-developers/GitPython
                    # /issues/383
                    raise GitCommandError(
                        "clone has failed, telling ya",
                        999,  # good number
                        stdout="%s already exists" if exists(path) else "")
                raise  # reraise original

        gr = cls(path, *args, repo=repo, **kwargs)
        return gr

    def __del__(self):
        # unbind possibly bound ConfigManager, to prevent all kinds of weird
        # stalls etc
        self._cfg = None
        # Make sure to flush pending changes, especially close batch processes
        # (internal `git cat-file --batch` by GitPython)
        try:
            if hasattr(self, 'repo') and exists(self.path) \
                    and self.repo is not None:
                # gc might be late, so the (temporary)
                # repo doesn't exist on FS anymore

                self.repo.git.clear_cache()
                if exists(opj(self.path, '.git')):  # don't try to write otherwise
                    self.repo.index.write()
        except InvalidGitRepositoryError:
            # might have being removed and no longer valid
            pass

    def __repr__(self):
        return "<GitRepo path=%s (%s)>" % (self.path, type(self))

    def __eq__(self, obj):
        """Decides whether or not two instances of this class are equal.

        This is done by comparing the base repository path.
        """
        return self.realpath == obj.realpath

    @classmethod
    def is_valid_repo(cls, path):
        """Returns if a given path points to a git repository"""
        return exists(opj(path, '.git'))

    @property
    def config(self):
        """Get an instance of the parser for the persistent repository
        configuration.

        Note: This allows to also read/write .datalad/config,
        not just .git/config

        Returns
        -------
        ConfigManager
        """
        if self._cfg is None:
            # associate with this dataset and read the entire config hierarchy
            self._cfg = ConfigManager(dataset=self, dataset_only=False)
        return self._cfg

    def is_with_annex(self, only_remote=False):
        """Return True if GitRepo (assumed) at the path has remotes with git-annex branch

        Parameters
        ----------
        only_remote: bool, optional
            Check only remote (no local branches) for having git-annex branch
        """
        return any((b.endswith('/git-annex') or
                    'annex/direct' in b
                    for b in self.get_remote_branches())) or \
            ((not only_remote) and
             any((b == 'git-annex' or 'annex/direct' in b
                  for b in self.get_branches())))

    @classmethod
    def get_toppath(cls, path, follow_up=True, git_options=None):
        """Return top-level of a repository given the path.

        Parameters
        -----------
        follow_up : bool
          If path has symlinks -- they get resolved by git.  If follow_up is
          True, we will follow original path up until we hit the same resolved
          path.  If no such path found, resolved one would be returned.
        git_options: list of str
          options to be passed to the git rev-parse call

        Return None if no parent directory contains a git repository.
        """
        cmd = ['git']
        if git_options:
            cmd.extend(git_options)
        cmd += ["rev-parse", "--show-toplevel"]
        try:
            toppath, err = GitRunner().run(
                cmd,
                cwd=path,
                log_stdout=True, log_stderr=True,
                expect_fail=True, expect_stderr=True)
            toppath = toppath.rstrip('\n\r')
        except CommandError:
            return None
        except OSError:
            toppath = GitRepo.get_toppath(dirname(path), follow_up=follow_up,
                                          git_options=git_options)

        if follow_up:
            path_ = path
            path_prev = ""
            while path_ and path_ != path_prev:  # on top /.. = /
                if realpath(path_) == toppath:
                    toppath = path_
                    break
                path_prev = path_
                path_ = dirname(path_)

        return toppath

    # classmethod so behavior could be tuned in derived classes
    @classmethod
    def _get_added_files_commit_msg(cls, files):
        if not files:
            return "No files were added"
        msg = "Added %d file" % len(files)
        if len(files) > 1:
            msg += "s"
        return msg + '\n\nFiles:\n' + '\n'.join(files)

    @normalize_paths
    def add(self, files, commit=False, msg=None, git=True, git_options=None,
            _datalad_msg=False, update=False):
        """Adds file(s) to the repository.

        Parameters
        ----------
        files: list
          list of paths to add
        commit: bool
          whether or not to directly commit
        msg: str
          commit message in case `commit=True`. A default message, containing
          the list of files that were added, is created by default.
        git: bool
          somewhat ugly construction to be compatible with AnnexRepo.add();
          has to be always true.
        update: bool
          --update option for git-add. From git's manpage:
           Update the index just where it already has an entry matching
           <pathspec>. This removes as well as modifies index entries to match
           the working tree, but adds no new files.

           If no <pathspec> is given when --update option is used, all tracked
           files in the entire working tree are updated (old versions of Git
           used to limit the update to the current directory and its
           subdirectories).
        """

        # TODO: git_options is used as options for the git-add here,
        # instead of options to the git executable => rename for consistency

        # needs to be True - see docstring:
        assert(git)

        files = _remove_empty_items(files)
        out = []

        if files or git_options or update:
            try:
                # without --verbose git 2.9.3  add does not return anything
                add_out = self._git_custom_command(
                    files,
                    ['git', 'add'] + assure_list(git_options) +
                    to_options(update=update) + ['--verbose']
                )
                # get all the entries
                out = self._process_git_get_output(*add_out)
                # Note: as opposed to git cmdline, force is True by default in
                #       gitpython, which would lead to add things, that are
                #       ignored or excluded otherwise
                # 2. Note: There is an issue with globbing (like adding '.'),
                #       which apparently doesn't care for 'force' and therefore
                #       adds '.git/...'. May be it's expanded at the wrong
                #       point in time or sth. like that.
                # For now, use direct call to git add.
                #self.cmd_call_wrapper(self.repo.index.add, files, write=True,
                #                      force=False)
                # TODO: May be make use of 'fprogress'-option to indicate
                # progress
                # But then, we don't have it for git-annex add, anyway.
                #
                # TODO: Is write=True a reasonable way to do it?
                # May be should not write until success of operation is
                # confirmed?
                # What's best in case of a list of files?
            except OSError as e:
                lgr.error("add: %s" % e)
                raise

        else:
            lgr.warning("add was called with empty file list and no options.")

        if commit:
            if msg is None:
                msg = self._get_added_files_commit_msg(files)
            self.commit(msg=msg, _datalad_msg=_datalad_msg)

        # Make sure return value from GitRepo is consistent with AnnexRepo
        # currently simulating similar return value, assuming success
        # for all files:
        # TODO: Make return values consistent across both *Repo classes!
        return out

    @staticmethod
    def _process_git_get_output(stdout, stderr=None):
        """Given both outputs (stderr is ignored atm) of git add - process it

        Primarily to centralize handling in both indirect annex and direct
        modes when ran through proxy
        """
        return [{u'file': f, u'success': True}
                for f in re.findall("'(.*)'[\n$]", stdout)]

    @normalize_paths(match_return_type=False)
    def remove(self, files, recursive=False, **kwargs):
        """Remove files.

        Calls git-rm.

        Parameters
        ----------
        files: str
          list of paths to remove
        recursive: False
          either to allow recursive removal from subdirectories
        kwargs:
          see `__init__`

        Returns
        -------
        [str]
          list of successfully removed files.
        """

        files = _remove_empty_items(files)

        if recursive:
            kwargs['r'] = True
        stdout, stderr = self._git_custom_command(
            files, ['git', 'rm'] + to_options(**kwargs))

        # output per removed file is expected to be "rm 'PATH'":
        return [line.strip()[4:-1] for line in stdout.splitlines()]

        #return self.repo.git.rm(files, cached=False, **kwargs)

    def precommit(self):
        """Perform pre-commit maintenance tasks
        """
        # All GitPython commands should take care about flushing index
        # whenever they modify it, so we would not care to do anything
        # if self.repo is not None and exists(opj(self.path, '.git')):  # don't try to write otherwise:
        #     # flush possibly cached in GitPython changes to index:
        #     # if self.repo.git:
        #     #     sys.stderr.write("CLEARING\n")
        #     #     self.repo.git.clear_cache()
        #     self.repo.index.write()
        pass

    @staticmethod
    def _get_prefixed_commit_msg(msg):
        DATALAD_PREFIX = "[DATALAD]"
        return DATALAD_PREFIX if not msg else "%s %s" % (DATALAD_PREFIX, msg)

    def commit(self, msg=None, options=None, _datalad_msg=False, careless=True,
               files=None):
        """Commit changes to git.

        Parameters
        ----------
        msg: str, optional
          commit-message
        options: list of str, optional
          cmdline options for git-commit
        _datalad_msg: bool, optional
          To signal that commit is automated commit by datalad, so
          it would carry the [DATALAD] prefix
        careless: bool, optional
          if False, raise when there's nothing actually committed;
          if True, don't care
        files: list of str, optional
          path(s) to commit
        """

        self.precommit()

        if _datalad_msg:
            msg = self._get_prefixed_commit_msg(msg)

        if not msg:
            if options:
                if "--allow-empty-message" not in options:
                        options.append("--allow-empty-message")
            else:
                options = ["--allow-empty-message"]

        # Note: We used to use a direct call to git only if there were options,
        # since we can't pass all possible options to gitpython's implementation
        # of commit.
        # But there's an additional issue. GitPython implements commit in a way,
        # that it might create a new commit, when a direct call wouldn't. This
        # was discovered with a modified (but unstaged) submodule, leading to a
        # commit, that apparently did nothing - git status still showed the very
        # same thing afterwards. But a commit was created nevertheless:
        # diff --git a/sub b/sub
        # --- a/sub
        # +++ b/sub
        # @@ -1 +1 @@
        # -Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f
        # +Subproject commit d3935338a3b3735792de1078bbfb5e9913ef998f-dirty
        #
        # Therefore, for now always use direct call.
        # TODO: Figure out, what exactly is going on with gitpython here

        cmd = ['git', 'commit'] + (["-m", msg if msg else ""])
        if options:
            cmd.extend(options)
        lgr.debug("Committing via direct call of git: %s" % cmd)

        try:
            self._git_custom_command(files, cmd,
                                     expect_stderr=True, expect_fail=True)
        except CommandError as e:
            if 'nothing to commit' in e.stdout:
                if careless:
                    lgr.debug("nothing to commit in {}. "
                              "Ignored.".format(self))
                else:
                    raise
            elif 'no changes added to commit' in e.stdout or \
                    'nothing added to commit' in e.stdout:
                if careless:
                    lgr.debug("no changes added to commit in {}. "
                              "Ignored.".format(self))
                else:
                    raise
            elif "did not match any file(s) known to git." in e.stderr:
                # TODO: Improve FileNotInXXXXError classes to better deal with
                # multiple files; Also consider PathOutsideRepositoryError
                raise FileNotInRepositoryError(cmd=e.cmd,
                                               msg="File(s) unknown to git",
                                               code=e.code,
                                               filename=linesep.join(
                                            [l for l in e.stderr.splitlines()
                                             if l.startswith("pathspec")]))
            else:
                raise

    def get_indexed_files(self):
        """Get a list of files in git's index

        Returns
        -------
        list
            list of paths rooting in git's base dir
        """

        return [x[0] for x in self.cmd_call_wrapper(
            self.repo.index.entries.keys)]

    def get_hexsha(self, object=None):
        """Return a hexsha for a given object. If None - of current HEAD

        Parameters
        ----------
        object: str, optional
          Any type of Git object identifier. See `git show`.
        """
        cmd = ['git', 'show', '--no-patch', "--format=%H"]
        if object:
            cmd.append(object)
        # make sure Git takes our argument as a revision
        cmd.append('--')
        try:
            stdout, stderr = self._git_custom_command(
                '', cmd, expect_stderr=True, expect_fail=True)
        except CommandError as e:
            if 'bad revision' in e.stderr:
                raise ValueError("Unknown object identifier: %s" % object)
            elif 'does not have any commits yet' in e.stderr:
                return None
            else:
                raise e
        stdout = stdout.splitlines()
        assert(len(stdout) == 1)
        return stdout[0]

    def get_merge_base(self, treeishes):
        """Get a merge base hexsha

        Parameters
        ----------
        treeishes: str or list of str
          List of treeishes (branches, hexshas, etc) to determine the merge
          base of. If a single value provided, returns merge_base with the
          current branch.

        Returns
        -------
        str or None
          If no merge-base for given commits, or specified treeish doesn't
          exist, None returned
        """
        if isinstance(treeishes, string_types):
            treeishes = [treeishes]
        if not treeishes:
            raise ValueError("Provide at least a single value")
        elif len(treeishes) == 1:
            treeishes = treeishes + [self.get_active_branch()]

        try:
            bases = self.repo.merge_base(*treeishes)
        except GitCommandError as exc:
            if "fatal: Not a valid object name" in str(exc):
                return None
            raise

        if not bases:
            return None
        assert(len(bases) == 1)  # we do not do 'all' yet
        return bases[0].hexsha

    def get_committed_date(self, branch=None):
        """Get the date stamp of the last commit (in a branch). None if no commit"""
        try:
            commit = next(
                self.get_branch_commits(branch
                                        or self.get_active_branch())
            )
        except Exception as exc:
            lgr.debug("Got exception while trying to get last commit: %s",
                      exc_str(exc))
            return None
        return commit.committed_date

    def get_active_branch(self):
        try:
            branch = self.repo.active_branch.name
        except TypeError as e:
            if "HEAD is a detached symbolic reference" in str(e):
                lgr.debug("detached HEAD in {0}".format(self))
                return None
            else:
                raise
        return branch

    def get_branches(self):
        """Get all branches of the repo.

        Returns
        -------
        [str]
            Names of all branches of this repository.
        """

        return [branch.name for branch in self.repo.branches]

    def get_remote_branches(self):
        """Get all branches of all remotes of the repo.

        Returns
        -----------
        [str]
            Names of all remote branches.
        """
        # TODO: Reconsider melting with get_branches()

        # TODO: treat entries like this: origin/HEAD -> origin/master'
        # currently this is done in collection

        # For some reason, this is three times faster than the version below:
        remote_branches = list()
        for remote in self.repo.remotes:
            try:
                for ref in remote.refs:
                    remote_branches.append(ref.name)
            except AssertionError as e:
                if str(e).endswith("did not have any references"):
                    # this will happen with git annex special remotes
                    pass
                else:
                    raise e
        return remote_branches
        # return [branch.strip() for branch in
        #         self.repo.git.branch(r=True).splitlines()]

    def get_remotes(self, with_urls_only=False):
        """Get known remotes of the repository

        Parameters
        ----------
        with_urls_only : bool, optional
          return only remotes which have urls

        Returns
        -------
        remotes : list of str
          List of names of the remotes
        """

        # Note: read directly from config and spare instantiation of gitpy.Repo
        # since we need this in AnnexRepo constructor. Furthermore gitpy does it
        # pretty much the same way and the use of a Repo instance seems to have
        # no reason other than a nice object oriented look.
        from datalad.utils import unique

        self.config.reload()
        remotes = unique([x[7:] for x in self.config.sections()
                          if x.startswith("remote.")])

        if with_urls_only:
            remotes = [
                r for r in remotes
                if self.config.get('remote.%s.url' % r)
            ]
        return remotes

    def get_files(self, branch=None):
        """Get a list of files in git.

        Lists the files in the (remote) branch.

        Parameters
        ----------
        branch: str
          Name of the branch to query. Default: active branch.

        Returns
        -------
        [str]
          list of files.
        """
        # TODO: RF codes base and melt get_indexed_files() in

        if branch is None:
            # active branch can be queried way faster:
            return self.get_indexed_files()
        else:
            return [item.path for item in self.repo.tree(branch).traverse()
                    if isinstance(item, Blob)]

    def get_file_content(self, file_, branch='HEAD'):
        """

        Returns
        -------
        [str]
          content of file_ as a list of lines.
        """
        content_str = self.repo.commit(branch).tree[file_].data_stream.read()

        # in python3 a byte string is returned. Need to convert it:
        from six import PY3
        if PY3:
            conv_str = u''
            for b in bytes(content_str):
                conv_str += chr(b)
            return conv_str.splitlines()
        else:
            return content_str.splitlines()
        # TODO: keep splitlines?

    def _get_files_history(self, files, branch='HEAD'):
        """

        Parameters
        ----------
        files: list
          list of files, only commits with queried files are considered
        branch: str
          Name of the branch to query. Default: HEAD.

        Returns
        -------
        [iterator]
        yielding Commit items generator from branch history associated with files
        """
        return gitpy.objects.commit.Commit.iter_items(self.repo, branch, paths=files)

    def _get_remotes_having_commit(self, commit_hexsha, with_urls_only=True):
        """Traverse all branches of the remote and check if commit in any of their ancestry

        It is a generator yielding names of the remotes
        """
        out, err = self._git_custom_command(
            '', 'git branch -r --contains ' + commit_hexsha
        )
        # sanitize a bit (all the spaces and new lines)
        remote_branches = [
            b  # could be origin/HEAD -> origin/master, we just skip ->
            for b in filter(bool, out.split())
            if b != '->'
        ]
        return [
            remote
            for remote in self.get_remotes(with_urls_only=with_urls_only)
            if any(rb.startswith(remote + '/') for rb in remote_branches)
        ]

    def _gitpy_custom_call(self, cmd, cmd_args=None, cmd_options=None,
                           git_options=None, env=None,

                           # 'old' options for Runner; not sure yet, which of
                           # them are actually still needed:
                           log_stdout=True, log_stderr=True, log_online=False,
                           expect_stderr=True, cwd=None,
                           shell=None, expect_fail=False):

        """Helper to call GitPython's wrapper for git calls.

        The used instance of `gitpy.Git` is bound to the repository,
        which determines its working directory.
        This is used for adhoc implementation of a git command and to
        demonstrate how to use it in more specific implementations.

        Note
        ----
        Aims to replace the use of datalad's `Runner` class for direct git
        calls. (Currently the `_git_custom_command()` method).
        Therefore mimicking its behaviour during RF'ing.

        Parameters
        ----------
        cmd: str
          the native git command to call
        cmd_args: list of str
          arguments to the git command
        cmd_options: dict
          options for the command as key, value pair
          (this transformation, needs some central place to document)
        git_options: dict
          options for the git executable as key, value pair
          (see above)
        env: dict
          environment vaiables to temporarily set for this call

        TODO
        ----
        Example

        Returns
        -------
        (stdout, stderr)
        """

        # TODO: Reconsider when to log/stream what (stdout, stderr) and/or
        # fully implement the behaviour of `Runner`

        if log_online:
            raise NotImplementedError("option 'log_online' not implemented yet")
        with_exceptions = not expect_fail
        if cwd:
            # the gitpy.cmd.Git instance, bound to this repository doesn't allow
            # to explicitly set the working dir, except for using os.getcwd
            raise NotImplementedError("working dir is a read-only property")

        _tmp_shell = gitpy.cmd.Git.USE_SHELL
        gitpy.cmd.Git.USE_SHELL = shell

        if env is None:
            env = {}
        if git_options is None:
            git_options = {}
        if cmd_options is None:
            cmd_options = {}
        cmd_options.update({'with_exceptions': with_exceptions,
                            'with_extended_output': True})

        # TODO: _GIT_COMMON_OPTIONS!

        with self.repo.git.custom_environment(**env):
            try:
                status, std_out, std_err = \
                    self.repo.git(**git_options).__getattr__(cmd)(
                        cmd_args, **cmd_options)
            except GitCommandError as e:
                # For now just reraise. May be raise CommandError instead
                raise
            finally:
                gitpy.cmd.Git.USE_SHELL = _tmp_shell

        if not expect_stderr and std_err:
            lgr.error("Unexpected output on stderr: %s" % std_err)
            raise CommandError
        if log_stdout:
            for line in std_out.splitlines():
                lgr.debug("stdout| " + line)
        if log_stderr:
            for line in std_err.splitlines():
                lgr.log(level=logging.DEBUG if expect_stderr else logging.ERROR,
                        msg="stderr| " + line)

        return std_out, std_err

    @normalize_paths(match_return_type=False)
    def _git_custom_command(self, files, cmd_str,
                            log_stdout=True, log_stderr=True, log_online=False,
                            expect_stderr=True, cwd=None, env=None,
                            shell=None, expect_fail=False):
        """Allows for calling arbitrary commands.

        Helper for developing purposes, i.e. to quickly implement git commands
        for proof of concept without the need to figure out, how this is done
        via GitPython.

        Parameters
        ----------
        files: list of files
        cmd_str: str or list
            arbitrary command str. `files` is appended to that string.

        Returns
        -------
        stdout, stderr
        """

        # ensure cmd_str becomes a well-formed list:
        if isinstance(cmd_str, string_types):
            if files and not cmd_str.strip().endswith(" --"):
                cmd_str += " --"
            cmd_str = shlex.split(cmd_str, posix=not on_windows)
        else:
            if files and cmd_str[-1] != '--':
                cmd_str.append('--')

        cmd = cmd_str + files

        assert(cmd[0] == 'git')
        cmd = cmd[:1] + self._GIT_COMMON_OPTIONS + cmd[1:]

        from .exceptions import GitIgnoreError

        try:
            out, err = self.cmd_call_wrapper.run(
                cmd,
                log_stderr=log_stderr,
                log_stdout=log_stdout,
                log_online=log_online,
                expect_stderr=expect_stderr,
                cwd=cwd,
                env=env,
                shell=shell,
                expect_fail=expect_fail)
        except CommandError as e:
            ignored = re.search(GitIgnoreError.pattern, e.stderr)
            if ignored:
                raise GitIgnoreError(cmd=e.cmd, msg=e.stderr,
                                     code=e.code, stdout=e.stdout,
                                     stderr=e.stderr,
                                     paths=ignored.groups()[0].splitlines())
            raise
        return out, err

# TODO: --------------------------------------------------------------------

    def add_remote(self, name, url, options=None):
        """Register remote pointing to a url
        """
        cmd = ['git', 'remote', 'add']
        if options:
            cmd += options
        cmd += [name, url]

        result = self._git_custom_command('', cmd)
        self.config.reload()
        return result

    def remove_remote(self, name):
        """Remove existing remote
        """

        # TODO: testing and error handling!
        from .exceptions import RemoteNotAvailableError
        try:
            out, err = self._git_custom_command(
                '', ['git', 'remote', 'remove', name])
        except CommandError as e:
            if 'fatal: No such remote' in e.stderr:
                raise RemoteNotAvailableError(name,
                                              cmd="git remote remove",
                                              msg="No such remote",
                                              stdout=out,
                                              stderr=err)
            else:
                raise e

        # TODO: config.reload necessary?
        self.config.reload()
        return

    def update_remote(self, name=None, verbose=False):
        """
        """
        options = ["-v"] if verbose else []
        name = [name] if name else []
        self._git_custom_command(
            '', ['git', 'remote'] + name + ['update'] + options,
            expect_stderr=True
        )

    # TODO: centralize all the c&p code in fetch, pull, push
    # TODO: document **kwargs passed to gitpython
    def fetch(self, remote=None, refspec=None, progress=None, all_=False,
              **kwargs):
        """Fetches changes from a remote (or all_ remotes).

        Parameters
        ----------
        remote: str
          (optional) name of the remote to fetch from. If no remote is given and
          `all_` is not set, the tracking branch is fetched.
        refspec: str
          (optional) refspec to fetch.
        progress:
          passed to gitpython. TODO: Figure it out, make consistent use of it
          and document it.
        all_: bool
          fetch all_ remotes (and all_ of their branches).
          Fails if `remote` was given.
        kwargs:
          passed to gitpython. TODO: Figure it out, make consistent use of it
          and document it.

        Returns
        -------
        list
            FetchInfo objects of the items fetched from remote
        """
        # TODO: options=> **kwargs):
        # Note: Apparently there is no explicit (fetch --all) in gitpython,
        #       but fetch is always bound to a certain remote instead.
        #       Therefore implement it on our own:
        if remote is None:
            if refspec is not None:
                # conflicts with using tracking branch or fetch all remotes
                # For now: Just fail.
                # TODO: May be check whether it fits to tracking branch
                raise ValueError("refspec specified without a remote. (%s)" %
                                 refspec)
            if all_:
                remotes_to_fetch = [
                    self.repo.remote(r)
                    for r in self.get_remotes(with_urls_only=True)
                ]
            else:
                # No explicit remote to fetch.
                # => get tracking branch:
                tb_remote, refspec = self.get_tracking_branch()
                if tb_remote is not None:
                    remotes_to_fetch = [self.repo.remote(tb_remote)]
                else:
                    # No remote, no tracking branch
                    # => fail
                    raise ValueError("Neither a remote is specified to fetch "
                                     "from nor a tracking branch is set up.")
        else:
            remotes_to_fetch = [self.repo.remote(remote)]

        fi_list = []
        for rm in remotes_to_fetch:
            fetch_url = \
                self.config.get('remote.%s.fetchurl' % rm.name,
                                self.config.get('remote.%s.url' % rm.name,
                                                None))
            if fetch_url is None:
                lgr.debug("Remote %s has no URL", rm)
                return []

            if is_ssh(fetch_url):
                ssh_manager.get_connection(fetch_url).open()
                # TODO: with git <= 2.3 keep old mechanism:
                #       with rm.repo.git.custom_environment(GIT_SSH="wrapper_script"):
                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):
                    fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)
                    # TODO: progress +kwargs
            else:
                fi_list += rm.fetch(refspec=refspec, progress=progress, **kwargs)
                # TODO: progress +kwargs

        # TODO: fetch returns a list of FetchInfo instances. Make use of it.
        return fi_list

    def pull(self, remote=None, refspec=None, progress=None, **kwargs):
        """See fetch
        """
        if remote is None:
            if refspec is not None:
                # conflicts with using tracking branch or fetch all remotes
                # For now: Just fail.
                # TODO: May be check whether it fits to tracking branch
                raise ValueError("refspec specified without a remote. (%s)" %
                                 refspec)
            # No explicit remote to pull from.
            # => get tracking branch:
            tb_remote, refspec = self.get_tracking_branch()
            if tb_remote is not None:
                remote = self.repo.remote(tb_remote)
            else:
                # No remote, no tracking branch
                # => fail
                raise ValueError("No remote specified to pull from nor a "
                                 "tracking branch is set up.")

        else:
            remote = self.repo.remote(remote)

        fetch_url = \
            remote.config_reader.get(
                'fetchurl' if remote.config_reader.has_option('fetchurl')
                else 'url')
        if is_ssh(fetch_url):
            ssh_manager.get_connection(fetch_url).open()
            # TODO: with git <= 2.3 keep old mechanism:
            #       with remote.repo.git.custom_environment(GIT_SSH="wrapper_script"):
            with remote.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):
                return remote.pull(refspec=refspec, progress=progress, **kwargs)
                # TODO: progress +kwargs
        else:
            return remote.pull(refspec=refspec, progress=progress, **kwargs)
            # TODO: progress +kwargs

    def push(self, remote=None, refspec=None, progress=None, all_remotes=False,
             **kwargs):
        """Push to remote repository

        Parameters
        ----------
        remote: str
          name of the remote to push to
        refspec: str
          specify what to push
        progress:
          TODO
        all_remotes: bool
          if set to True push to all remotes. Conflicts with `remote` not being
          None.
        kwargs: dict
          options to pass to `git push`

        Returns
        -------
        list
            PushInfo objects of the items pushed to remote
        """

        if remote is None:
            if refspec is not None:
                # conflicts with using tracking branch or fetch all remotes
                # For now: Just fail.
                # TODO: May be check whether it fits to tracking branch
                raise ValueError("refspec specified without a remote. (%s)" %
                                 refspec)
            if all_remotes:
                remotes_to_push = self.repo.remotes
            else:
                # Nothing explicitly specified. Just call `git push` and let git
                # decide what to do would be an option. But:
                # - without knowing the remote and its URL we cannot provide
                #   shared SSH connection
                # - we lose ability to use GitPython's progress info and return
                #   values
                #   (the latter would be solvable:
                #    Provide a Repo.push() method for GitPython, copying
                #    Remote.push() for similar return value and progress
                #    (also: fetch, pull)

                # Do what git would do:
                # 1. branch.*.remote for current branch or 'origin' as default
                #    if config is missing
                # 2. remote.*.push or push.default

                # TODO: check out "same procedure" for fetch/pull

                tb_remote, refspec = self.get_tracking_branch()
                if tb_remote is None:
                    tb_remote = 'origin'
                remotes_to_push = [self.repo.remote(tb_remote)]
                # use no refspec; let git find remote.*.push or push.default on
                # its own

        else:
            if all_remotes:
                lgr.warning("Option 'all_remotes' conflicts with specified "
                            "remote '%s'. Option ignored.")
            remotes_to_push = [self.repo.remote(remote)]

        pi_list = []
        for rm in remotes_to_push:
            push_url = \
                rm.config_reader.get('pushurl'
                                     if rm.config_reader.has_option('pushurl')
                                     else 'url')
            if is_ssh(push_url):
                ssh_manager.get_connection(push_url).open()
                # TODO: with git <= 2.3 keep old mechanism:
                #       with rm.repo.git.custom_environment(GIT_SSH="wrapper_script"):
                with rm.repo.git.custom_environment(**GitRepo.GIT_SSH_ENV):
                    pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)
                    # TODO: progress +kwargs
            else:
                pi_list += rm.push(refspec=refspec, progress=progress, **kwargs)
                # TODO: progress +kwargs
        return pi_list

    def get_remote_url(self, name, push=False):
        """Get the url of a remote.

        Reads the configuration of remote `name` and returns its url or None,
        if there is no url configured.

        Parameters
        ----------
        name: str
          name of the remote
        push: bool
          if True, get the pushurl instead of the fetch url.
        """

        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')
        return self.config.get(var, None)

    def set_remote_url(self, name, url, push=False):
        """Set the URL a remote is pointing to

        Sets the URL of the remote `name`. Requires the remote to already exist.

        Parameters
        ----------
        name: str
          name of the remote
        url: str
        push: bool
          if True, set the push URL, otherwise the fetch URL
        """

        var = 'remote.{0}.{1}'.format(name, 'pushurl' if push else 'url')
        self.config.set(var, url, where='local', reload=True)

    def get_branch_commits(self, branch=None, limit=None, stop=None, value=None):
        """Return GitPython's commits for the branch

        Pretty much similar to what 'git log <branch>' does.
        It is a generator which returns top commits first

        Parameters
        ----------
        branch: str, optional
          If not provided, assumes current branch
        limit: None | 'left-only', optional
          Limit which commits to report.  If None -- all commits (merged or not),
          if 'left-only' -- only the commits from the left side of the tree upon
          merges
        stop: str, optional
          hexsha of the commit at which stop reporting (matched one is not
          reported either)
        value: None | 'hexsha', optional
          What to yield.  If None - entire commit object is yielded, if 'hexsha'
          only its hexsha
        """

        if not branch:
            branch = self.get_active_branch()

        try:
            _branch = self.repo.branches[branch]
        except IndexError:
            raise MissingBranchError(self, branch,
                                     [b.name for b in self.repo.branches])

        fvalue = {None: lambda x: x, 'hexsha': lambda x: x.hexsha}[value]

        if not limit:
            def gen():
                # traverse doesn't yield original commit
                co = _branch.commit
                yield co
                for co_ in co.traverse():
                    yield co_
        elif limit == 'left-only':
            # we need a custom implementation since couldn't figure out how to
            # do with .traversal
            def gen():
                co = _branch.commit
                while co:
                    yield co
                    co = co.parents[0] if co.parents else None
        else:
            raise ValueError(limit)

        for c in gen():
            if stop and c.hexsha == stop:
                return
            yield fvalue(c)

    def checkout(self, name, options=None):
        """
        """
        # TODO: May be check for the need of -b options herein?
        cmd = ['git', 'checkout']
        if options:
            cmd += options
        cmd += [str(name)]

        self._git_custom_command('', cmd, expect_stderr=True)

    # TODO: Before implementing annex merge, find usages and check for a needed
    # change to call super().merge
    def merge(self, name, options=None, msg=None, allow_unrelated=False, **kwargs):
        if options is None:
            options = []
        if msg:
            options = options + ["-m", msg]
        if allow_unrelated and external_versions['cmd:git'] >= '2.9':
            options += ['--allow-unrelated-histories']
        self._git_custom_command(
            '', ['git', 'merge'] + options + [name],
            **kwargs
        )

    def remove_branch(self, branch):
        self._git_custom_command(
            '', ['git', 'branch', '-D', branch]
        )

    def ls_remote(self, remote, options=None):
        if options is None:
            options = []
        self._git_custom_command(
            '', ['git', 'ls-remote'] + options + [remote]
        )
        # TODO: Return values?

    def is_dirty(self, index=True, working_tree=True, untracked_files=True,
                 submodules=True, path=None):
        """Returns true if the repo is considered to be dirty

        Parameters
        ----------
        index: bool
          if True, consider changes to the index
        working_tree: bool
          if True, consider changes to the working tree
        untracked_files: bool
          if True, consider untracked files
        submodules: bool
          if True, consider submodules
        path: str or list of str
          path(s) to consider only
        Returns
        -------
          bool
        """

        return self.repo.is_dirty(index=index, working_tree=working_tree,
                                  untracked_files=untracked_files,
                                  submodules=submodules, path=path)

    @property
    def dirty(self):
        return self.is_dirty()

    @property
    def untracked_files(self):
        return self.repo.untracked_files

    def gc(self, allow_background=False, auto=False):
        """Perform house keeping (garbage collection, repacking)"""
        cmd_options = ['git']
        if not allow_background:
            cmd_options += ['-c', 'gc.autodetach=0']
        cmd_options += ['gc', '--aggressive']
        if auto:
            cmd_options += ['--auto']
        self._git_custom_command('', cmd_options)

    def get_submodules(self, sorted_=True):
        """Return a list of git.Submodule instances for all submodules"""
        # check whether we have anything in the repo. if not go home early
        if not self.repo.head.is_valid():
            return []
        submodules = self.repo.submodules
        if sorted_:
            submodules = sorted(submodules, key=lambda x: x.path)
        return submodules

    def is_submodule_modified(self, name, options=[]):
        """Whether a submodule has new commits

        Note: This is an adhoc method. It parses output of
        'git submodule summary' and currently is not able to distinguish whether
        or not this change is staged in `self` and whether this would be
        reported 'added' or 'modified' by 'git status'.
        Parsing isn't heavily tested yet.

        Parameters
        ----------
        name: str
          the submodule's name
        options: list
          options to pass to 'git submodule summary'
        Returns
        -------
        bool
          True if there are commits in the submodule, differing from
          what is registered in `self`
        --------
        """

        out, err = self._git_custom_command('',
                                            ['git', 'submodule', 'summary'] + \
                                            options + ['--', name])
        return any([line.split()[1] == name
                    for line in out.splitlines()
                    if line and len(line.split()) > 1])

    def add_submodule(self, path, name=None, url=None, branch=None):
        """Add a new submodule to the repository.

        This will alter the index as well as the .gitmodules file, but will not
        create a new commit.  If the submodule already exists, no matter if the
        configuration differs from the one provided, the existing submodule
        is considered as already added and no further action is performed.

        Parameters
        ----------
        path : str
          repository-relative path at which the submodule should be located, and
          which will be created as required during the repository initialization.
        name : str or None
          name/identifier for the submodule. If `None`, the `path` will be used
          as name.
        url : str or None
          git-clone compatible URL. If `None`, the repository is assumed to
          exist, and the url of the first remote is taken instead. This is
          useful if you want to make an existing repository a submodule of
          another one.
        branch : str or None
          name of branch to be checked out in the submodule. The given branch
          must exist in the remote repository, and will be checked out locally
          as a tracking branch. If `None`, remote HEAD will be checked out.
        """
        if name is None:
            name = path
        # XXX the following should do it, but GitPython will refuse to add a submodule
        # unless you specify a URL that is configured as one of its remotes, or you
        # specify no URL, but the repo has at least one remote.
        # this is stupid, as for us it is valid to not have any remote, because we can
        # still obtain the submodule from a future publication location, based on the
        # parent
        # gitpy.Submodule.add(self.repo, name, path, url=url, branch=branch)
        # going git native instead
        cmd = ['git', 'submodule', 'add', '--name', name]
        if branch is not None:
            cmd += ['-b', branch]
        if url is None:
            if not isabs(path):
                # need to recode into a relative path "URL" in POSIX
                # style, even on windows
                url = posixpath.join(curdir, posix_relpath(path))
            else:
                url = path
        cmd += [url, path]
        self._git_custom_command('', cmd)
        # TODO: return value

    def deinit_submodule(self, path, **kwargs):
        """Deinit a submodule

        Parameters
        ----------
        path: str
            path to the submodule; relative to `self.path`
        kwargs:
            see `__init__`
        """

        self._git_custom_command(path,
                                 ['git', 'submodule', 'deinit'] +
                                 to_options(**kwargs))
        # TODO: return value

    def update_submodule(self, path, mode='checkout', init=False):
        """Update a registered submodule.

        This will make the submodule match what the superproject expects by
        cloning missing submodules and updating the working tree of the
        submodules. The "updating" can be done in several ways depending
        on the value of submodule.<name>.update configuration variable, or
        the `mode` argument.

        Parameters
        ----------
        path : str
          Identifies which submodule to operate on by it's repository-relative
          path.
        mode : {checkout, rebase, merge}
          Update procedure to perform. 'checkout': the commit recorded in the
          superproject will be checked out in the submodule on a detached HEAD;
          'rebase': the current branch of the submodule will be rebased onto
          the commit recorded in the superproject; 'merge': the commit recorded
          in the superproject will be merged into the current branch in the
          submodule.
        init : bool
          If True, initialize all submodules for which "git submodule init" has
          not been called so far before updating.
          Primarily provided for internal purposes and should not be used directly
          since would result in not so annex-friendly .git symlinks/references
          instead of full featured .git/ directories in the submodules
        """
        cmd = ['git', 'submodule', 'update', '--%s' % mode]
        if init:
            cmd.append('--init')
            subgitpath = opj(self.path, path, '.git')
            if not exists(subgitpath):
                # TODO:  wouldn't with --init we get all those symlink'ed .git/?
                # At least let's warn
                lgr.warning(
                    "Do not use update_submodule with init=True to avoid git creating "
                    "symlinked .git/ directories in submodules"
                )
            #  yoh: I thought I saw one recently but thought it was some kind of
            #  an artifact from running submodule update --init manually at
            #  some point, but looking at this code now I worry that it was not
        self._git_custom_command(path, cmd)
        # TODO: return value

    def update_ref(self, ref, value, symbolic=False):
        """Update the object name stored in a ref "safely".

        Just a shim for `git update-ref` call if not symbolic, and
        `git symbolic-ref` if symbolic

        Parameters
        ----------
        ref : str
          Reference, such as `ref/heads/BRANCHNAME` or HEAD.
        value : str
          Value to update to, e.g. hexsha of a commit when updating for a
          branch ref, or branch ref if updating HEAD
        symbolic : None
          To instruct if ref is symbolic, e.g. should be used in case of
          ref=HEAD
        """
        self._git_custom_command(
            '',
            ['git', 'symbolic-ref' if symbolic else 'update-ref', ref, value]
        )

    def tag(self, tag):
        """Assign a tag to current commit

        Parameters
        ----------
        tag : str
          Custom tag label.
        """
        # TODO later to be extended with tagging particular commits and signing
        # TODO: call in save.py complains about extensive logging. When does it
        # happen in what way? Figure out, whether to just silence it or raise or
        # whatever else.
        self._git_custom_command(
            '', ['git', 'tag', str(tag)]
        )

    def get_tags(self, output=None):
        """Get list of tags

        Parameters
        ----------
        output : str, optional
          If given, limit the return value to a list of values matching that
          particular key of the tag properties.

        Returns
        -------
        list
          Each item is a dictionary with information on a tag. At present
          this includes 'hexsha', and 'name', where the latter is the string
          label of the tag, and the format the hexsha of the object the tag
          is attched to. The list is sorted by commit date, with the most
          recent commit being the last element.
        """
        # TODO it would be straightforward to add more info and tweak the
        # sorting
        stdout, stderr = self._git_custom_command(
            '',
            ['git', 'tag', '--format=%(refname:strip=2)%00%(object)',
             '--sort=*committerdate'])
        fields = ('name', 'hexsha')
        tags = [dict(zip(fields, line.split('\0'))) for line in stdout.splitlines()]
        if output:
            return [t[output] for t in tags]
        else:
            return tags

    def describe(self, **kwargs):
        """ Quick and dirty implementation to call git-describe

        Parameters:
        -----------
        kwargs:
            transformed to cmdline options for git-describe;
            see __init__ for description of the transformation
        """
        # TODO: be more precise what failure to expect when and raise actual
        # errors
        try:
            describe, outerr = self._git_custom_command(
                [],
                ['git', 'describe'] + to_options(**kwargs),
                expect_fail=True)
            return describe.strip()
        # TODO: WTF "catch everything"?
        except:
            return None

    def get_tracking_branch(self, branch=None):
        """Get the tracking branch for `branch` if there is any.

        Parameters
        ----------
        branch: str
            local branch to look up. If none is given, active branch is used.

        Returns
        -------
        tuple
            (remote or None, refspec or None) of the tracking branch
        """
        if branch is None:
            branch = self.get_active_branch()
            if branch is None:
                return None, None

        track_remote = self.config.get('branch.{0}.remote'.format(branch), None)
        track_branch = self.config.get('branch.{0}.merge'.format(branch), None)
        return track_remote, track_branch

    @property
    def count_objects(self):
        """return dictionary with count, size(in KiB) information of git objects
        """

        count_cmd = ['git', 'count-objects', '-v']
        count_str, err = self._git_custom_command('', count_cmd)
        count = {key: int(value)
                 for key, value in [item.split(': ')
                                    for item in count_str.split('\n')
                                    if len(item.split(': ')) == 2]}
        return count

    def get_missing_files(self):
        """Return a list of paths with missing files (and no staged deletion)"""
        return [f.split('\t')[1]
                for f in self.repo.git.diff('--raw', '--name-status').split('\n')
                if f.split('\t')[0] == 'D']

    def get_deleted_files(self):
        """Return a list of paths with deleted files (staged deletion)"""
        return [f.split('\t')[1]
                for f in self.repo.git.diff('--raw', '--name-status', '--staged').split('\n')
                if f.split('\t')[0] == 'D']

    def get_git_attributes(self):
        """Check git attribute for the current repository (not per-file support for now)

        Parameters
        ----------
        all_: bool
          Adds --all to git check-attr call

        Returns
        -------
        dict:
          attribute: value pairs
        """
        out, err = self._git_custom_command(["."], ["git", "check-attr", "--all"])
        assert not err, "no stderr output is expected"
        out_split = [
            # splitting by : would leave leading space(s)
            [e.lstrip(' ') for e in l.split(':', 2)]
            for l in out.split('\n') if l
        ]
        assert all(o[0] == '.' for o in out_split)  # for paranoid
        return dict(o[1:] for o in out_split)


# TODO
# remove submodule: nope, this is just deinit_submodule + remove
# status?

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user