{
    "file_diffs": [
        {
            "old_file_content": "# vim ft=yaml\n# travis-ci.org definition for DataLad build\nlanguage: python\nservices:\n  - docker\n\npython:\n  - 3.6\n\ncache:\n  - apt\n\nenv:\n  global:\n    # will be used in the matrix, where neither other variable is used\n    - BOTO_CONFIG=/tmp/nowhere\n    - DATALAD_TESTS_SSH=1\n    - DATALAD_LOG_ENV=GIT_SSH_COMMAND\n    - TESTS_TO_PERFORM=datalad\n    # Should be an array, travis breaks on it, define/adjust in the \"before_install\"\n    #- NOSE_OPTS=( -v )\n    # Note, that there's \"turtle\" as well, which is always excluded from\n    # running on Travis.\n    - NOSE_SELECTION=\"integration or usecase or slow or network\"\n    - NOSE_SELECTION_OP=\"not \"   # so it would be \"not (integration or usecase)\"\n    - DATALAD_DATASETS_TOPURL=http://datasets-tests.datalad.org\n    # How/which git-annex we install.  conda's build would be the fastest, but it must not\n    # get ahead in PATH to not shadow travis' python\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20201007 -m conda\"\n\nmatrix:\n  include:\n  # Additional custom ones\n  - python: 3.7\n    # Single run for Python 3.7\n    env:\n    # Run all tests in a single whoop here\n    # We cannot have empty -A selector, so the one which always will be fulfilled\n    - NOSE_SELECTION=\n    - NOSE_SELECTION_OP=not\n    # To test https://github.com/datalad/datalad/pull/4342 fix.\n    # From our testing in that PR seems to have no effect, but kept around since should not hurt.\n    - LC_ALL=ru_RU.UTF-8\n  - python: 3.8\n    dist: bionic # Xenial doesn't have 3.8 pre-installed (only 3.8-dev).\n    env:\n    - NOSE_SELECTION=\n    - NOSE_SELECTION_OP=not\n  - if: type = cron\n    python: 3.6\n    # Single run for Python 3.6\n    env:\n    # Run all tests in a single whoop here\n    # We cannot have empty -A selector, so the one which always will be fulfilled\n    - NOSE_SELECTION=\n    - NOSE_SELECTION_OP=not\n  - python: 3.6\n    # Split runs for v6 since a single one is too long now\n    env:\n    - NOSE_SELECTION_OP=not\n    - DATALAD_SSH_MULTIPLEX__CONNECTIONS=0\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20210310 -m conda\"\n  - python: 3.6\n    env:\n    - NOSE_SELECTION_OP=\"\"\n    - DATALAD_SSH_MULTIPLEX__CONNECTIONS=0\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20210310 -m conda\"\n    # To test https://github.com/datalad/datalad/pull/4342 fix in case of no \"not\" for NOSE.\n    # From our testing in that PR seems to have no effect, but kept around since should not hurt.\n    - LANG=bg_BG.UTF-8\n  - python: 3.6\n    # Run slow etc tests under a single tricky scenario\n    env:\n    - _DL_TMPDIR=\"/var/tmp/sym link\"\n    - NOSE_SELECTION_OP=\"\"\n    # And the leading - in filenames for the most challenge\n    - DATALAD_TESTS_OBSCURE_PREFIX=-\n    - DATALAD_LOG_TRACEBACK=collide  # just a smoke test for now\n  - python: 3.6\n    # A run loaded with various customizations to smoke test those functionalities\n    # apparently moving symlink outside has different effects on abspath\n    # see  https://github.com/datalad/datalad/issues/878\n    env:\n    # eventually: - _DL_TMPDIR=\"/var/tmp/sym ссылка\"\n    - _DL_TMPDIR=\"/var/tmp/sym link\"\n    # and obscure the names more a bit\n    - DATALAD_TESTS_OBSCURE_PREFIX=-\n    # By default no logs will be output. This one is to test with log output at INFO level\n    - DATALAD_LOG_LEVEL=INFO\n    - DATALAD_LOG_TRACEBACK=1  # just a smoke test for now\n    - DATALAD_LOG_VMEM=1\n  - python: 3.6\n    # By default no logs will be output. This one is to test with low level but dumped to /dev/null\n    env:\n    - DATALAD_LOG_LEVEL=2\n    - DATALAD_LOG_TARGET=/dev/null\n    - DATALAD_TESTS_PROTOCOLREMOTE=1\n    - DATALAD_TESTS_DATALADREMOTE=1\n    - DATALAD_LOG_CWD=1\n    - DATALAD_LOG_OUTPUTS=1\n    - DATALAD_LOG_ENV=1\n    - DATALAD_LOG_STDIN=1\n    - DATALAD_TESTS_UI_BACKEND=console\n    - DATALAD_TESTS_OBSCURE_PREFIX=-\n    - DATALAD_SEED=1\n    - GIT_AUTHOR_DATE=\"Thu, 07 Apr 2005 22:13:13 +0200\"\n    - GIT_AUTHOR_NAME=blah\n    - GIT_AUTHOR_EMAIL=committer@example.com\n    - GIT_COMMITTER_DATE=\"Thu, 07 Apr 2005 22:13:13 +0200\"\n    - GIT_COMMITTER_NAME=blah\n    - GIT_COMMITTER_EMAIL=committer@example.com\n  - python: 3.6\n    # Test some under NFS mount  (only selected sub-set)\n    env:\n    # do not run SSH-based tests due to stall(s)\n    # https://github.com/datalad/datalad/pull/4172\n    - DATALAD_TESTS_SSH=0\n    - _DL_TMPDIR=\"/tmp/nfsmount\"\n    - TESTS_TO_PERFORM=\"datalad.tests datalad.support\"\n    #\n  # The ones to run only on weekends against master.\n  # They will not contribute to coverage etc, but might lead to failed status\n  #\n  # run with minimal supported git-annex version as defined in AnnexRepo.GIT_ANNEX_MIN_VERSION\n  # TODO: ATM we do not have that minimal version as a Debian package in\n  # snapshots!\n  - if: type = cron\n    python: 3.6\n    env:\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20200309 -m conda\"\n  # Run with git's master branch rather the default one on the system.\n  - if: type = cron\n    python: 3.6\n    env:\n    - DATALAD_USE_DEFAULT_GIT=1\n    - _DL_UPSTREAM_GIT=1\n  # Run with our reported minimum Git version.\n  - if: type = cron\n    python: 3.6\n    env:\n    - DATALAD_USE_DEFAULT_GIT=1\n    - _DL_MIN_GIT=1\n    - PATH=\"$PWD/git-src/bin-wrappers/:$PATH\"\n  - if: type = cron\n    python: 3.6\n    env:\n    # to test operation under root since also would consider FS \"crippled\" due to\n    # ability to rewrite R/O files\n    - NOSE_WRAPPER=\"sudo -E\"\n    # no key authentication for root:\n    - DATALAD_TESTS_SSH=0\n  - if: type = cron\n    python: 3.6\n    env:\n    - DATALAD_TESTS_NONETWORK=1\n    # must operate nicely with those env variables set\n    - http_proxy=\n    - https_proxy=\n  - if: type = cron\n    python: 3.6\n    # Test under NFS mount  (full, only in master)\n    env:\n    - _DL_TMPDIR=\"/tmp/nfsmount\"\n\n  allow_failures:\n  # Test under NFS mount  (full, only in master)\n  - if: type = cron\n    python: 3.6\n    env:\n    - _DL_TMPDIR=\"/tmp/nfsmount\"\n\n# Causes complete laptop or travis instance crash atm, but survives in a docker\n# need to figure it out (looks like some PID explosion)\n#  - python: 3.6\n#    # we would need to migrate to boto3 to test it fully, but SSH should work\n#    env:\n#    - DATALAD_TESTS_SSH=1\n#    - UNSET_S3_SECRETS=1\n\nbefore_install:\n  - NOSE_OPTS=( )\n  # If we requested to run only not slow (typically <10sec) tests, fail if a test\n  # takes 3x more than that - it needs to get @slow or @turtle annotation\n  - if echo \"$NOSE_SELECTION_OP($NOSE_SELECTION)\" | grep -q \"^not.*slow\"; then\n      NOSE_OPTS=( \"${NOSE_OPTS[@]}\" --with-doctest --with-timer --timer-ok 5 --timer-warning 30 --timer-fail error --timer-filter warning,error );\n      export DATALAD_TESTS_SETUP_TESTREPOS=1;\n    fi\n  # Just in case we need to check if nfs is there etc\n  - sudo lsmod\n  # The ultimate one-liner setup for NeuroDebian repository\n  - bash <(wget -q -O- http://neuro.debian.net/_files/neurodebian-travis.sh)\n  - travis_retry sudo apt-get update -qq\n  - travis_retry sudo apt-get install eatmydata  # to speedup some installations\n  - tools/ci/prep-travis-forssh.sh\n  # Install various basic depedencies\n  - travis_retry sudo eatmydata apt-get install zip pandoc p7zip-full\n  # needed for tests of patool compression fall-back solution\n  - travis_retry sudo eatmydata apt-get install xz-utils\n  - travis_retry sudo eatmydata apt-get install shunit2\n  # for metadata support\n  - travis_retry sudo eatmydata apt-get install exempi\n  # Configure _DL_TMPDIR before trying install git-annex -- script might use it\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/sym\\ link ]]; then echo \"Symlinking $_DL_TMPDIR\"; ln -s /tmp \"$_DL_TMPDIR\"; fi\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/d\\ i\\ r ]]; then echo \"mkdir $_DL_TMPDIR\"; mkdir -p \"$_DL_TMPDIR\"; fi\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/nfsmount ]]; then echo \"mkdir $_DL_TMPDIR\"; mkdir -p \"$_DL_TMPDIR\" \"${_DL_TMPDIR}_\"; echo \"/tmp/nfsmount_ localhost(rw)\" | sudo bash -c 'cat - > /etc/exports'; sudo apt-get install -y nfs-kernel-server; sudo exportfs -a; sudo mount -t nfs localhost:/tmp/nfsmount_ /tmp/nfsmount; fi\n  # Maybe build install custom git.\n  - if [ ! -z \"${_DL_UPSTREAM_GIT:-}\" ]; then source tools/ci/install-upstream-git.sh; fi\n  - if [ ! -z \"${_DL_MIN_GIT:-}\" ]; then tools/ci/install-minimum-git.sh; fi\n  # Install git-annex\n  - pip install datalad-installer\n  - eval datalad-installer --sudo ok -E new.env ${_DL_ANNEX_INSTALL_SCENARIO}\n  - source new.env && cat new.env >> ~/.bashrc\n  - pip install --upgrade pip\n\ninstall:\n  - git config --global user.email \"test@travis.land\"\n  - git config --global user.name \"Travis Almighty\"\n  - cd ..; pip install -q codecov; cd -\n  - pip install -r requirements-devel.txt\n  # So we could test under sudo -E with PATH pointing to installed location\n  - sudo sed -i -e 's/^Defaults.*secure_path.*$//' /etc/sudoers\n  # git-annex workaround. TODO: remove - should not be needed\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/nfsmount ]]; then sudo git config --system annex.pidlock true ; fi\n\nscript:\n  # Now it should be safe to point TMPDIR to a \"tricky\" setup just for the purpose of testing\n  - if [ -n \"${_DL_TMPDIR:-}\" ]; then export TMPDIR=\"${_DL_TMPDIR}\"; fi\n  # Test installation system-wide\n  - sudo pip install .\n  # Report WTF information using system wide installed version\n  - datalad wtf\n  - mkdir -p __testhome__\n  - cd __testhome__\n  # Run tests\n  - http_proxy=\n    PATH=$PWD/../tools/coverage-bin:$PATH\n    $NOSE_WRAPPER `which nosetests` \"${NOSE_OPTS[@]}\"\n      -A \"$NOSE_SELECTION_OP($NOSE_SELECTION) and not(turtle)\"\n      --with-doctest\n      --with-cov --cover-package datalad\n      --logging-level=INFO\n      $TESTS_TO_PERFORM\n  - cd ..\n\nafter_success:\n  # submit only what we have covered in the PRs\n  - if [ ! ${TRAVIS_EVENT_TYPE} = \"cron\" ]; then\n      cd __testhome__;\n      coverage combine -a /tmp/.coverage-entrypoints-*;\n      codecov;\n    fi\n\n# makes it only more difficult to comprehend the failing output.  Enable only when necessary\n# for a particular debugging\n#after_failure:\n#  - if [ ! -z \"$DATALAD_TESTS_NONETWORK\" ]; then sudo route add -net 0.0.0.0 netmask 0.0.0.0 dev lo; fi\n#  - DATALAD_LOG_LEVEL=DEBUG $NOSE_WRAPPER `which nosetests` -s -v --with-doctest --with-cov --cover-package datalad --logging-level=DEBUG\n#  - if [ ! -z \"$DATALAD_TESTS_NONETWORK\" ]; then sudo route del -net 0.0.0.0 netmask 0.0.0.0 dev lo; fi\n",
            "new_file_content": "# vim ft=yaml\n# travis-ci.org definition for DataLad build\nlanguage: python\nservices:\n  - docker\n\npython:\n  - 3.6\n\ncache:\n  - apt\n\nenv:\n  global:\n    # will be used in the matrix, where neither other variable is used\n    - BOTO_CONFIG=/tmp/nowhere\n    - DATALAD_TESTS_SSH=1\n    - DATALAD_LOG_ENV=GIT_SSH_COMMAND\n    - TESTS_TO_PERFORM=datalad\n    # Should be an array, travis breaks on it, define/adjust in the \"before_install\"\n    #- NOSE_OPTS=( -v )\n    # Note, that there's \"turtle\" as well, which is always excluded from\n    # running on Travis.\n    - NOSE_SELECTION=\"integration or usecase or slow or network\"\n    - NOSE_SELECTION_OP=\"not \"   # so it would be \"not (integration or usecase)\"\n    - DATALAD_DATASETS_TOPURL=http://datasets-tests.datalad.org\n    # How/which git-annex we install.  conda's build would be the fastest, but it must not\n    # get ahead in PATH to not shadow travis' python\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20201007 -m conda\"\n\nmatrix:\n  include:\n  # Additional custom ones\n  - python: 3.7\n    # Single run for Python 3.7\n    env:\n    # Run all tests in a single whoop here\n    # We cannot have empty -A selector, so the one which always will be fulfilled\n    - NOSE_SELECTION=\n    - NOSE_SELECTION_OP=not\n    # To test https://github.com/datalad/datalad/pull/4342 fix.\n    # From our testing in that PR seems to have no effect, but kept around since should not hurt.\n    - LC_ALL=ru_RU.UTF-8\n  - python: 3.8\n    dist: bionic # Xenial doesn't have 3.8 pre-installed (only 3.8-dev).\n    env:\n    - NOSE_SELECTION=\n    - NOSE_SELECTION_OP=not\n  - if: type = cron\n    python: 3.6\n    # Single run for Python 3.6\n    env:\n    # Run all tests in a single whoop here\n    # We cannot have empty -A selector, so the one which always will be fulfilled\n    - NOSE_SELECTION=\n    - NOSE_SELECTION_OP=not\n  - python: 3.6\n    # Split runs for v6 since a single one is too long now\n    env:\n    - NOSE_SELECTION_OP=not\n    - DATALAD_SSH_MULTIPLEX__CONNECTIONS=0\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20210310 -m conda\"\n  - python: 3.6\n    env:\n    - NOSE_SELECTION_OP=\"\"\n    - DATALAD_SSH_MULTIPLEX__CONNECTIONS=0\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20210310 -m conda\"\n    # To test https://github.com/datalad/datalad/pull/4342 fix in case of no \"not\" for NOSE.\n    # From our testing in that PR seems to have no effect, but kept around since should not hurt.\n    - LANG=bg_BG.UTF-8\n  - python: 3.6\n    # Run slow etc tests under a single tricky scenario\n    env:\n    - _DL_TMPDIR=\"/var/tmp/sym link\"\n    - NOSE_SELECTION_OP=\"\"\n    # And the leading - in filenames for the most challenge\n    - DATALAD_TESTS_OBSCURE_PREFIX=-\n    - DATALAD_LOG_TRACEBACK=collide  # just a smoke test for now\n  - python: 3.6\n    # A run loaded with various customizations to smoke test those functionalities\n    # apparently moving symlink outside has different effects on abspath\n    # see  https://github.com/datalad/datalad/issues/878\n    env:\n    # eventually: - _DL_TMPDIR=\"/var/tmp/sym ссылка\"\n    - _DL_TMPDIR=\"/var/tmp/sym link\"\n    # and obscure the names more a bit\n    - DATALAD_TESTS_OBSCURE_PREFIX=-\n    # By default no logs will be output. This one is to test with log output at INFO level\n    - DATALAD_LOG_LEVEL=INFO\n    - DATALAD_LOG_TRACEBACK=1  # just a smoke test for now\n    - DATALAD_LOG_VMEM=1\n  - python: 3.6\n    # By default no logs will be output. This one is to test with low level but dumped to /dev/null\n    env:\n    - DATALAD_LOG_LEVEL=2\n    - DATALAD_LOG_TARGET=/dev/null\n    - DATALAD_TESTS_PROTOCOLREMOTE=1\n    - DATALAD_TESTS_DATALADREMOTE=1\n    - DATALAD_LOG_CWD=1\n    - DATALAD_LOG_OUTPUTS=1\n    - DATALAD_LOG_ENV=1\n    - DATALAD_LOG_STDIN=1\n    - DATALAD_TESTS_UI_BACKEND=console\n    - DATALAD_TESTS_OBSCURE_PREFIX=-\n    - DATALAD_SEED=1\n    - GIT_AUTHOR_DATE=\"Thu, 07 Apr 2005 22:13:13 +0200\"\n    - GIT_AUTHOR_NAME=blah\n    - GIT_AUTHOR_EMAIL=committer@example.com\n    - GIT_COMMITTER_DATE=\"Thu, 07 Apr 2005 22:13:13 +0200\"\n    - GIT_COMMITTER_NAME=blah\n    - GIT_COMMITTER_EMAIL=committer@example.com\n  - python: 3.6\n    # Test some under NFS mount  (only selected sub-set)\n    env:\n    # do not run SSH-based tests due to stall(s)\n    # https://github.com/datalad/datalad/pull/4172\n    - DATALAD_TESTS_SSH=0\n    - _DL_TMPDIR=\"/tmp/nfsmount\"\n    - TESTS_TO_PERFORM=\"datalad.tests datalad.support\"\n    #\n  # The ones to run only on weekends against master.\n  # They will not contribute to coverage etc, but might lead to failed status\n  #\n  # run with minimal supported git-annex version as defined in AnnexRepo.GIT_ANNEX_MIN_VERSION\n  # TODO: ATM we do not have that minimal version as a Debian package in\n  # snapshots!\n  - if: type = cron\n    python: 3.6\n    env:\n    - _DL_ANNEX_INSTALL_SCENARIO=\"miniconda --batch git-annex=8.20200309 -m conda\"\n  # Run with git's master branch rather the default one on the system.\n  - if: type = cron\n    python: 3.6\n    env:\n    - DATALAD_USE_DEFAULT_GIT=1\n    - _DL_UPSTREAM_GIT=1\n  # Run with our reported minimum Git version.\n  - if: type = cron\n    python: 3.6\n    env:\n    - DATALAD_USE_DEFAULT_GIT=1\n    - _DL_MIN_GIT=1\n    - PATH=\"$PWD/git-src/bin-wrappers/:$PATH\"\n  - if: type = cron\n    python: 3.6\n    env:\n    # to test operation under root since also would consider FS \"crippled\" due to\n    # ability to rewrite R/O files\n    - NOSE_WRAPPER=\"sudo -E\"\n    # no key authentication for root:\n    - DATALAD_TESTS_SSH=0\n  - if: type = cron\n    python: 3.6\n    env:\n    - DATALAD_TESTS_NONETWORK=1\n    # must operate nicely with those env variables set\n    - http_proxy=\n    - https_proxy=\n  - if: type = cron\n    python: 3.6\n    # Test under NFS mount  (full, only in master)\n    env:\n    - _DL_TMPDIR=\"/tmp/nfsmount\"\n\n  allow_failures:\n  # Test under NFS mount  (full, only in master)\n  - if: type = cron\n    python: 3.6\n    env:\n    - _DL_TMPDIR=\"/tmp/nfsmount\"\n\n# Causes complete laptop or travis instance crash atm, but survives in a docker\n# need to figure it out (looks like some PID explosion)\n#  - python: 3.6\n#    # we would need to migrate to boto3 to test it fully, but SSH should work\n#    env:\n#    - DATALAD_TESTS_SSH=1\n#    - UNSET_S3_SECRETS=1\n\nbefore_install:\n  - NOSE_OPTS=( )\n  # If we requested to run only not slow (typically <10sec) tests, fail if a test\n  # takes 3x more than that - it needs to get @slow or @turtle annotation\n  - if echo \"$NOSE_SELECTION_OP($NOSE_SELECTION)\" | grep -q \"^not.*slow\"; then\n      NOSE_OPTS=( \"${NOSE_OPTS[@]}\" --with-doctest --with-timer --timer-ok 5 --timer-warning 30 --timer-fail error --timer-filter warning,error );\n      export DATALAD_TESTS_SETUP_TESTREPOS=1;\n    fi\n  # Just in case we need to check if nfs is there etc\n  - sudo lsmod\n  # The ultimate one-liner setup for NeuroDebian repository\n  - bash <(wget -q -O- http://neuro.debian.net/_files/neurodebian-travis.sh)\n  - travis_retry sudo apt-get update -qq\n  - travis_retry sudo apt-get install eatmydata  # to speedup some installations\n  - tools/ci/prep-travis-forssh.sh\n  # Install various basic depedencies\n  - travis_retry sudo eatmydata apt-get install zip pandoc p7zip-full\n  # needed for tests of patool compression fall-back solution\n  - travis_retry sudo eatmydata apt-get install xz-utils\n  - travis_retry sudo eatmydata apt-get install shunit2\n  # for metadata support\n  - travis_retry sudo eatmydata apt-get install exempi\n  # Configure _DL_TMPDIR before trying install git-annex -- script might use it\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/sym\\ link ]]; then echo \"Symlinking $_DL_TMPDIR\"; ln -s /tmp \"$_DL_TMPDIR\"; fi\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/d\\ i\\ r ]]; then echo \"mkdir $_DL_TMPDIR\"; mkdir -p \"$_DL_TMPDIR\"; fi\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/nfsmount ]]; then echo \"mkdir $_DL_TMPDIR\"; mkdir -p \"$_DL_TMPDIR\" \"${_DL_TMPDIR}_\"; echo \"/tmp/nfsmount_ localhost(rw)\" | sudo bash -c 'cat - > /etc/exports'; sudo apt-get install -y nfs-kernel-server; sudo exportfs -a; sudo mount -t nfs localhost:/tmp/nfsmount_ /tmp/nfsmount; fi\n  # Maybe build install custom git.\n  - if [ ! -z \"${_DL_UPSTREAM_GIT:-}\" ]; then source tools/ci/install-upstream-git.sh; fi\n  - if [ ! -z \"${_DL_MIN_GIT:-}\" ]; then tools/ci/install-minimum-git.sh; fi\n  # Install git-annex\n  - pip install datalad-installer\n  - eval datalad-installer --sudo ok -E new.env ${_DL_ANNEX_INSTALL_SCENARIO}\n  - source new.env && cat new.env >> ~/.bashrc\n  - pip install --upgrade pip\n\ninstall:\n  - git config --global user.email \"test@travis.land\"\n  - git config --global user.name \"Travis Almighty\"\n  - cd ..; pip install -q codecov; cd -\n  - pip install -r requirements-devel.txt\n  # So we could test under sudo -E with PATH pointing to installed location\n  - sudo sed -i -e 's/^Defaults.*secure_path.*$//' /etc/sudoers\n  # git-annex workaround. TODO: remove - should not be needed\n  - if [[ \"${_DL_TMPDIR:-}\" =~ .*/nfsmount ]]; then sudo git config --system annex.pidlock true ; fi\n\nscript:\n  # Now it should be safe to point TMPDIR to a \"tricky\" setup just for the purpose of testing\n  - if [ -n \"${_DL_TMPDIR:-}\" ]; then export TMPDIR=\"${_DL_TMPDIR}\"; fi\n  # Test installation system-wide\n  - sudo pip install .\n  # Report WTF information using system wide installed version\n  - datalad wtf\n  - mkdir -p __testhome__\n  - cd __testhome__\n  # Run tests\n  - http_proxy=\n    PATH=$PWD/../tools/coverage-bin:$PATH\n    $NOSE_WRAPPER python -m nose \"${NOSE_OPTS[@]}\"\n      -A \"$NOSE_SELECTION_OP($NOSE_SELECTION) and not(turtle)\"\n      --with-doctest\n      --with-cov --cover-package datalad\n      --logging-level=INFO\n      $TESTS_TO_PERFORM\n  - cd ..\n\nafter_success:\n  # cron jobs test more and then PRs will be falling behind since they would not\n  # trigger some codepaths.  So submit coverage only from non-cron jobs, but report for all\n  - cd __testhome__;\n  - python -m coverage combine -a /tmp/.coverage-entrypoints-*;\n  - python -m coverage report;\n  - if [ ! ${TRAVIS_EVENT_TYPE} = \"cron\" ]; then\n      codecov;\n    fi\n\n# makes it only more difficult to comprehend the failing output.  Enable only when necessary\n# for a particular debugging\n#after_failure:\n#  - if [ ! -z \"$DATALAD_TESTS_NONETWORK\" ]; then sudo route add -net 0.0.0.0 netmask 0.0.0.0 dev lo; fi\n#  - DATALAD_LOG_LEVEL=DEBUG $NOSE_WRAPPER `which nosetests` -s -v --with-doctest --with-cov --cover-package datalad --logging-level=DEBUG\n#  - if [ ! -z \"$DATALAD_TESTS_NONETWORK\" ]; then sudo route del -net 0.0.0.0 netmask 0.0.0.0 dev lo; fi\n",
            "header": {
                "file": {
                    "path": ".travis.yml"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "f29da154e",
                "new_commit_hash": "3fe97bcb8",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/.travis.yml"
            },
            "plus_file": {
                "path": "b/.travis.yml"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 234,
                            "length": 7
                        },
                        "new_range": {
                            "start": 234,
                            "length": 7
                        },
                        "section": "script:"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "  # Run tests",
                                "type": "context"
                            },
                            {
                                "content": "  - http_proxy=",
                                "type": "context"
                            },
                            {
                                "content": "    PATH=$PWD/../tools/coverage-bin:$PATH",
                                "type": "context"
                            },
                            {
                                "content": "    $NOSE_WRAPPER `which nosetests` \"${NOSE_OPTS[@]}\"",
                                "type": "deleted"
                            },
                            {
                                "content": "    $NOSE_WRAPPER python -m nose \"${NOSE_OPTS[@]}\"",
                                "type": "added"
                            },
                            {
                                "content": "      -A \"$NOSE_SELECTION_OP($NOSE_SELECTION) and not(turtle)\"",
                                "type": "context"
                            },
                            {
                                "content": "      --with-doctest",
                                "type": "context"
                            },
                            {
                                "content": "      --with-cov --cover-package datalad",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 243,
                            "length": 10
                        },
                        "new_range": {
                            "start": 243,
                            "length": 12
                        },
                        "section": "script:"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "  - cd ..",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "after_success:",
                                "type": "context"
                            },
                            {
                                "content": "  # submit only what we have covered in the PRs",
                                "type": "deleted"
                            },
                            {
                                "content": "  # cron jobs test more and then PRs will be falling behind since they would not",
                                "type": "added"
                            },
                            {
                                "content": "  # trigger some codepaths.  So submit coverage only from non-cron jobs, but report for all",
                                "type": "added"
                            },
                            {
                                "content": "  - cd __testhome__;",
                                "type": "added"
                            },
                            {
                                "content": "  - python -m coverage combine -a /tmp/.coverage-entrypoints-*;",
                                "type": "added"
                            },
                            {
                                "content": "  - python -m coverage report;",
                                "type": "added"
                            },
                            {
                                "content": "  - if [ ! ${TRAVIS_EVENT_TYPE} = \"cron\" ]; then",
                                "type": "context"
                            },
                            {
                                "content": "      cd __testhome__;",
                                "type": "deleted"
                            },
                            {
                                "content": "      coverage combine -a /tmp/.coverage-entrypoints-*;",
                                "type": "deleted"
                            },
                            {
                                "content": "      codecov;",
                                "type": "context"
                            },
                            {
                                "content": "    fi",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Plumbing command for dataset installation\"\"\"\n\n\nimport logging\nimport re\nfrom os.path import expanduser\nfrom collections import OrderedDict\nfrom urllib.parse import unquote as urlunquote\n\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import eval_results\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.common_opts import (\n    location_description,\n    reckless_opt,\n)\nfrom datalad.log import log_progress\nfrom datalad.support.gitrepo import (\n    GitRepo,\n)\nfrom datalad.cmd import (\n    CommandError,\n    GitWitlessRunner,\n    StdOutCapture,\n)\nfrom datalad.distributed.ora_remote import (\n    LocalIO,\n    RIARemoteError,\n    SSHRemoteIO,\n)\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.constraints import (\n    EnsureNone,\n    EnsureStr,\n    EnsureKeyChoice,\n)\nfrom datalad.support.exceptions import DownloadError\nfrom datalad.support.param import Parameter\nfrom datalad.support.strings import get_replacement_dict\nfrom datalad.support.network import (\n    get_local_file_url,\n    download_url,\n    is_url,\n    URL,\n    RI,\n    DataLadRI,\n    PathRI,\n)\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\nfrom datalad.utils import (\n    ensure_bool,\n    ensure_list,\n    knows_annex,\n    make_tempfile,\n    Path,\n    PurePosixPath,\n    rmtree,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    resolve_path,\n    require_dataset,\n    EnsureDataset,\n)\nfrom datalad.distribution.utils import (\n    _get_flexible_source_candidates,\n)\nfrom datalad.utils import (\n    check_symlink_capability\n)\n\n__docformat__ = 'restructuredtext'\n\nlgr = logging.getLogger('datalad.core.distributed.clone')\n\n\n@build_doc\nclass Clone(Interface):\n    \"\"\"Obtain a dataset (copy) from a URL or local directory\n\n    The purpose of this command is to obtain a new clone (copy) of a dataset\n    and place it into a not-yet-existing or empty directory. As such `clone`\n    provides a strict subset of the functionality offered by `install`. Only a\n    single dataset can be obtained, and immediate recursive installation of\n    subdatasets is not supported. However, once a (super)dataset is installed\n    via `clone`, any content, including subdatasets can be obtained by a\n    subsequent `get` command.\n\n    Primary differences over a direct `git clone` call are 1) the automatic\n    initialization of a dataset annex (pure Git repositories are equally\n    supported); 2) automatic registration of the newly obtained dataset as a\n    subdataset (submodule), if a parent dataset is specified; 3) support\n    for additional resource identifiers (DataLad resource identifiers as used\n    on datasets.datalad.org, and RIA store URLs as used for store.datalad.org\n    - optionally in specific versions as identified by a branch or a tag; see\n    examples); and 4) automatic configurable generation of alternative access\n    URL for common cases (such as appending '.git' to the URL in case the\n    accessing the base URL failed).\n\n    In case the clone is registered as a subdataset, the original URL passed to\n    `clone` is recorded in `.gitmodules` of the parent dataset in addition\n    to the resolved URL used internally for git-clone. This allows to preserve\n    datalad specific URLs like ria+ssh://... for subsequent calls to `get` if\n    the subdataset was locally removed later on.\n\n    || PYTHON >>By default, the command returns a single Dataset instance for\n    an installed dataset, regardless of whether it was newly installed ('ok'\n    result), or found already installed from the specified source ('notneeded'\n    result).<< PYTHON ||\n\n    URL mapping configuration\n\n    'clone' supports the transformation of URLs via (multi-part) substitution\n    specifications. A substitution specification is defined as a configuration\n    setting 'datalad.clone.url-substition.<seriesID>' with a string containing\n    a match and substitution expression, each following Python's regular\n    expression syntax. Both expressions are concatenated to a single string\n    with an arbitrary delimiter character. The delimiter is defined by\n    prefixing the string with the delimiter. Prefix and delimiter are stripped\n    from the expressions (Example: \",^http://(.*)$,https://\\\\1\").  This setting\n    can be defined multiple times, using the same '<seriesID>'.  Substitutions\n    in a series will be applied incrementally, in order of their definition.\n    The first substitution in such a series must match, otherwise no further\n    substitutions in a series will be considered. However, following the first\n    match all further substitutions in a series are processed, regardless\n    whether intermediate expressions match or not. Substitution series themselves\n    have no particular order, each matching series will result in a candidate\n    clone URL. Consequently, the initial match specification in a series should\n    be as precise as possible to prevent inflation of candidate URLs.\n\n    .. seealso::\n\n      :ref:`handbook:3-001`\n        More information on Remote Indexed Archive (RIA) stores\n    \"\"\"\n    # by default ignore everything but install results\n    # i.e. no \"add to super dataset\"\n    result_filter = EnsureKeyChoice('action', ('install',))\n    # very frequently this command will yield exactly one installed dataset\n    # spare people the pain of going through a list by default\n    return_type = 'item-or-list'\n    # as discussed in #1409 and #1470, we want to return dataset instances\n    # matching what is actually available after command completion (and\n    # None for any failed dataset installation)\n    result_xfm = 'successdatasets-or-none'\n\n    _examples_ = [\n        dict(text=\"Install a dataset from Github into the current directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install a dataset into a specific directory\",\n             code_py=\"\"\"\\\n             clone(source='https://github.com/datalad-datasets/longnow-podcasts.git',\n                   path='myfavpodcasts')\"\"\",\n             code_cmd=\"\"\"\\\n             datalad clone https://github.com/datalad-datasets/longnow-podcasts.git \\\\\n             myfavpodcasts\"\"\"),\n        dict(text=\"Install a dataset as a subdataset into the current dataset\",\n             code_py=\"\"\"\\\n             clone(dataset='.',\n                   source='https://github.com/datalad-datasets/longnow-podcasts.git')\"\"\",\n             code_cmd=\"datalad clone -d . \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install the main superdataset from datasets.datalad.org\",\n             code_py=\"clone(source='///')\",\n             code_cmd=\"datalad clone ///\"),\n        dict(text=\"Install a dataset identified by a literal alias from store.datalad.org\",\n             code_py=\"clone(source='ria+http://store.datalad.org#~hcp-openaccess')\",\n             code_cmd=\"datalad clone ria+http://store.datalad.org#~hcp-openaccess\"),\n        dict(\n            text=\"Install a dataset in a specific version as identified by a \"\n                 \"branch or tag name from store.datalad.org\",\n            code_py=\"clone(source='ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier')\",\n            code_cmd=\"datalad clone ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier\"),\n        dict(\n            text=\"Install a dataset with group-write access permissions\",\n            code_py=\\\n            \"clone(source='http://example.com/dataset', reckless='shared-group')\",\n            code_cmd=\\\n            \"datalad clone http://example.com/dataset --reckless shared-group\"),\n    ]\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"(parent) dataset to clone into. If given, the newly cloned\n            dataset is registered as a subdataset of the parent. Also, if given,\n            relative paths are interpreted as being relative to the parent\n            dataset, and not relative to the working directory.\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        source=Parameter(\n            args=(\"source\",),\n            metavar='SOURCE',\n            doc=\"\"\"URL, DataLad resource identifier, local path or instance of\n            dataset to be cloned\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            nargs=\"?\",\n            doc=\"\"\"path to clone into.  If no `path` is provided a\n            destination path will be derived from a source URL\n            similar to :command:`git clone`\"\"\"),\n        description=location_description,\n        reckless=reckless_opt,\n    )\n\n    @staticmethod\n    @datasetmethod(name='clone')\n    @eval_results\n    def __call__(\n            source,\n            path=None,\n            dataset=None,\n            description=None,\n            reckless=None):\n        # did we explicitly get a dataset to install into?\n        # if we got a dataset, path will be resolved against it.\n        # Otherwise path will be resolved first.\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='clone') \\\n            if dataset is not None else dataset\n        refds_path = ds.path if ds else None\n\n        # legacy compatibility\n        if reckless is True:\n            # so that we can forget about how things used to be\n            reckless = 'auto'\n\n        if isinstance(source, Dataset):\n            source = source.path\n\n        if source == path:\n            # even if they turn out to be identical after resolving symlinks\n            # and more sophisticated witchcraft, it would still happily say\n            # \"it appears to be already installed\", so we just catch an\n            # obviously pointless input combination\n            raise ValueError(\n                \"clone `source` and destination `path` are identical [{}]. \"\n                \"If you are trying to add a subdataset simply use `save`\".format(\n                    path))\n\n        if path is not None:\n            path = resolve_path(path, dataset)\n\n        # derive target from source:\n        if path is None:\n            # we got nothing but a source. do something similar to git clone\n            # and derive the path from the source and continue\n            # since this is a relative `path`, resolve it:\n            # we are not going to reuse the decoded URL, as this is done for\n            # all source candidates in clone_dataset(), we just use to determine\n            # a destination path here in order to perform a bunch of additional\n            # checks that shall not pollute the helper function\n            source_ = decode_source_spec(\n                source, cfg=None if ds is None else ds.config)\n            path = resolve_path(source_['default_destpath'], dataset)\n            lgr.debug(\"Determined clone target path from source\")\n        lgr.debug(\"Resolved clone target path to: '%s'\", path)\n\n        # there is no other way -- my intoxicated brain tells me\n        assert(path is not None)\n\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            refds=refds_path,\n            source_url=source)\n\n        try:\n            # this will implicitly cause pathlib to run a bunch of checks\n            # whether the present path makes any sense on the platform\n            # we are running on -- we don't care if the path actually\n            # exists at this point, but we want to abort early if the path\n            # spec is determined to be useless\n            path.exists()\n        except OSError as e:\n            yield get_status_dict(\n                status='error',\n                path=path,\n                message=('cannot handle target path: %s', exc_str(e)),\n                **result_props)\n            return\n\n        destination_dataset = Dataset(path)\n        result_props['ds'] = destination_dataset\n\n        if ds is not None and ds.pathobj not in path.parents:\n            yield get_status_dict(\n                status='error',\n                message=(\"clone target path '%s' not in specified target dataset '%s'\",\n                         path, ds),\n                **result_props)\n            return\n\n        # perform the actual cloning operation\n        yield from clone_dataset(\n            [source],\n            destination_dataset,\n            reckless,\n            description,\n            result_props,\n            cfg=None if ds is None else ds.config,\n        )\n\n        # TODO handle any 'version' property handling and verification using a\n        # dedicated public helper\n\n        if ds is not None:\n            # we created a dataset in another dataset\n            # -> make submodule\n            actually_saved_subds = False\n            for r in ds.save(\n                    path,\n                    # Note, that here we know we don't save anything but a new\n                    # subdataset. Hence, don't go with default commit message,\n                    # but be more specific.\n                    message=\"[DATALAD] Added subdataset\",\n                    return_type='generator',\n                    result_filter=None,\n                    result_xfm=None,\n                    on_failure='ignore'):\n                actually_saved_subds = actually_saved_subds or (\n                        r['action'] == 'save' and\n                        r['type'] == 'dataset' and\n                        r['refds'] == ds.path and\n                        r['status'] == 'ok')\n                yield r\n\n            # Modify .gitmodules to contain originally given url. This is\n            # particularly relevant for postclone routines on a later `get`\n            # for that subdataset. See gh-5256.\n            if actually_saved_subds:\n                # New subdataset actually saved. Amend the modification\n                # of .gitmodules. Note, that we didn't allow to deviate\n                # from git default behavior WRT a submodule's name vs\n                # its path when we made this a new subdataset.\n                subds_name = path.relative_to(ds.pathobj)\n                ds.repo.call_git(\n                    ['config',\n                     '--file',\n                     '.gitmodules',\n                     '--replace-all',\n                     'submodule.{}.{}'.format(subds_name,\n                                              \"datalad-url\"),\n                     source]\n                )\n                yield from ds.save('.gitmodules',\n                                   amend=True, to_git=True)\n            else:\n                # We didn't really commit. Just call `subdatasets`\n                # in that case to have the modification included in the\n                # post-clone state (whatever that may be).\n                ds.subdatasets(path, set_property=[(\"datalad-url\", source)])\n\n\ndef _get_url_mappings(cfg):\n    cfg_prefix = 'datalad.clone.url-substitute.'\n    # figure out which keys we should be looking for\n    # in the active config\n    subst_keys = set(k for k in cfg.keys() if k.startswith(cfg_prefix))\n    # and in the common config specs\n    from datalad.interface.common_cfg import definitions\n    subst_keys.update(k for k in definitions if k.startswith(cfg_prefix))\n    # TODO a potential sorting of substitution series could be implemented\n    # here\n    return [\n        # decode the rule specifications\n        get_replacement_dict(\n            # one or more could come out\n            ensure_list(\n                cfg.get(\n                    k,\n                    # make sure to pull the default from the common config\n                    default=cfg.obtain(k),\n                    # we specifically support declaration of multiple\n                    # settings to build replacement chains\n                    get_all=True)))\n        for k in subst_keys\n    ]\n\n\ndef _map_urls(cfg, urls):\n    mapping_specs = _get_url_mappings(cfg)\n    if not mapping_specs:\n        return urls\n\n    mapped = []\n    # we process the candidate in order to maintain any prioritization\n    # encoded in it (e.g. _get_flexible_source_candidates_for_submodule)\n    # if we have a matching mapping replace the URL in its position\n    for u in urls:\n        # we only permit a single match\n        # TODO we likely want to RF this to pick the longest match\n        mapping_applied = False\n        # try one mapping set at a time\n        for mapping_spec in mapping_specs:\n            # process all substitution patterns in the specification\n            # always operate on strings (could be a Path instance too)\n            mu = str(u)\n            matched = False\n            for match_ex, subst_ex in mapping_spec.items():\n                if not matched:\n                    matched = re.match(match_ex, mu) is not None\n                if not matched:\n                    break\n                # try to map, would return unchanged, if there is no match\n                mu = re.sub(match_ex, subst_ex, mu)\n            if mu != u:\n                lgr.debug(\"URL substitution: '%s' -> '%s'\", u, mu)\n                mapped.append(mu)\n                # we could consider breaking after the for effective mapping\n                # specification. however, that would mean any generic\n                # definition of a broadly matching substitution would derail\n                # the entroe system. moreover, suddently order would matter\n                # substantially\n                mapping_applied = True\n        if not mapping_applied:\n            # none of the mappings matches, go with the original URL\n            # (really original, not the stringified one)\n            mapped.append(u)\n    return mapped\n\n\ndef clone_dataset(\n        srcs,\n        destds,\n        reckless=None,\n        description=None,\n        result_props=None,\n        cfg=None,\n        checkout_gitsha=None):\n    \"\"\"Internal helper to perform cloning without sanity checks (assumed done)\n\n    This helper does not handle any saving of subdataset modification or adding\n    in a superdataset.\n\n    Parameters\n    ----------\n    srcs : list\n      Any suitable clone source specifications (paths, URLs)\n    destds : Dataset\n      Dataset instance for the clone destination\n    reckless : {None, 'auto', 'ephemeral', 'shared-...'}, optional\n      Mode switch to put cloned dataset into unsafe/throw-away configurations, i.e.\n      sacrifice data safety for performance or resource footprint. When None\n      and `cfg` is specified, use the value of `datalad.clone.reckless`.\n    description : str, optional\n      Location description for the annex of the dataset clone (if there is any).\n    result_props : dict, optional\n      Default properties for any yielded result, passed on to get_status_dict().\n    cfg : ConfigManager, optional\n      Configuration for parent dataset. This will be queried instead\n      of the global DataLad configuration.\n    checkout_gitsha : str, optional\n      If given, a specific commit, identified by shasum, will be checked out after\n      cloning. A dedicated follow-up fetch will be performed, if the initial clone\n      did not obtain the commit object. Should the checkout of the target commit\n      cause a detached HEAD, the previously active branch will be reset to the\n      target commit.\n\n    Yields\n    ------\n    dict\n      DataLad result records\n    \"\"\"\n    if not result_props:\n        # in case the caller had no specific idea on how results should look\n        # like, provide sensible defaults\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            ds=destds,\n        )\n\n    if reckless is None and cfg:\n        # if reckless is not explicitly given, but we operate on a\n        # superdataset, query whether it has been instructed to operate\n        # in a reckless mode, and inherit it for the coming clone\n        reckless = cfg.get('datalad.clone.reckless', None)\n\n    dest_path = destds.pathobj\n\n    # check for configured URL mappings, either in the given config manager\n    # or in the one of the destination dataset, which is typically not existent\n    # yet and the process config manager is then used effectively\n    srcs = _map_urls(cfg or destds.config, srcs)\n\n    # decode all source candidate specifications\n    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]\n\n    # now expand the candidate sources with additional variants of the decoded\n    # giturl, while duplicating the other properties in the additional records\n    # for simplicity. The hope is to overcome a few corner cases and be more\n    # robust than git clone\n    candidate_sources = [\n        dict(props, giturl=s) for props in candidate_sources\n        for s in _get_flexible_source_candidates(props['giturl'])\n    ]\n\n    # important test! based on this `rmtree` will happen below after failed clone\n    dest_path_existed = dest_path.exists()\n    if dest_path_existed and any(dest_path.iterdir()):\n        if destds.is_installed():\n            # check if dest was cloned from the given source before\n            # this is where we would have installed this from\n            # this is where it was actually installed from\n            track_name, track_url = _get_tracking_source(destds)\n            try:\n                # this will get us track_url in system native path conventions,\n                # whenever it is a path (and not a URL)\n                # this is needed to match it to any potentially incoming local\n                # source path in the 'notneeded' test below\n                track_path = str(Path(track_url))\n            except Exception:\n                # this should never happen, because Path() will let any non-path stringification\n                # pass through unmodified, but we do not want any potential crash due to\n                # pathlib behavior changes\n                lgr.debug(\"Unexpected behavior of pathlib!\")\n                track_path = None\n            for cand in candidate_sources:\n                src = cand['giturl']\n                if track_url == src \\\n                        or (not is_url(track_url)\n                            and get_local_file_url(track_url, compatibility='git') == src) \\\n                        or track_path == expanduser(src):\n                    yield get_status_dict(\n                        status='notneeded',\n                        message=(\"dataset %s was already cloned from '%s'\",\n                                 destds,\n                                 src),\n                        **result_props)\n                    return\n        # anything else is an error\n        yield get_status_dict(\n            status='error',\n            message='target path already exists and not empty, refuse to clone into target path',\n            **result_props)\n        return\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Cloning dataset to %s', destds,\n        total=len(candidate_sources),\n        label='Clone attempt',\n        unit=' Candidate locations',\n    )\n    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url\n    for cand in candidate_sources:\n        log_progress(\n            lgr.info,\n            'cloneds',\n            'Attempting to clone from %s to %s', cand['giturl'], dest_path,\n            update=1,\n            increment=True)\n\n        clone_opts = {}\n\n        if cand.get('version', None):\n            clone_opts['branch'] = cand['version']\n        try:\n            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35\n            # doesn't make it happen seamlessly\n            GitRepo.clone(\n                path=str(dest_path),\n                url=cand['giturl'],\n                clone_options=clone_opts,\n                create=True)\n\n        except CommandError as e:\n            e_stderr = e.stderr\n\n            error_msgs[cand['giturl']] = e\n            lgr.debug(\"Failed to clone from URL: %s (%s)\",\n                      cand['giturl'], exc_str(e))\n            if dest_path.exists():\n                lgr.debug(\"Wiping out unsuccessful clone attempt at: %s\",\n                          dest_path)\n                # We must not just rmtree since it might be curdir etc\n                # we should remove all files/directories under it\n                # TODO stringification can be removed once patlib compatible\n                # or if PY35 is no longer supported\n                rmtree(str(dest_path), children_only=dest_path_existed)\n\n            if e_stderr and 'could not create work tree' in e_stderr.lower():\n                # this cannot be fixed by trying another URL\n                re_match = re.match(r\".*fatal: (.*)$\", e_stderr,\n                                    flags=re.MULTILINE | re.DOTALL)\n                # cancel progress bar\n                log_progress(\n                    lgr.info,\n                    'cloneds',\n                    'Completed clone attempts for %s', destds\n                )\n                yield get_status_dict(\n                    status='error',\n                    message=re_match.group(1).strip()\n                    if re_match else \"stderr: \" + e_stderr,\n                    **result_props)\n                return\n            # next candidate\n            continue\n\n        result_props['source'] = cand\n        # do not bother with other sources if succeeded\n        break\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Completed clone attempts for %s', destds\n    )\n\n    if not destds.is_installed():\n        if len(error_msgs):\n            if all(not e.stdout and not e.stderr for e in error_msgs.values()):\n                # there is nothing we can learn from the actual exception,\n                # the exit code is uninformative, the command is predictable\n                error_msg = \"Failed to clone from all attempted sources: %s\"\n                error_args = list(error_msgs.keys())\n            else:\n                error_msg = \"Failed to clone from any candidate source URL. \" \\\n                            \"Encountered errors per each url were:\\n- %s\"\n                error_args = '\\n- '.join(\n                    '{}\\n  {}'.format(url, exc_str(exc))\n                    for url, exc in error_msgs.items()\n                )\n        else:\n            # yoh: Not sure if we ever get here but I felt that there could\n            #      be a case when this might happen and original error would\n            #      not be sufficient to troubleshoot what is going on.\n            error_msg = \"Awkward error -- we failed to clone properly. \" \\\n                        \"Although no errors were encountered, target \" \\\n                        \"dataset at %s seems to be not fully installed. \" \\\n                        \"The 'succesful' source was: %s\"\n            error_args = (destds.path, cand['giturl'])\n        yield get_status_dict(\n            status='error',\n            message=(error_msg, error_args),\n            **result_props)\n        return\n\n    dest_repo = destds.repo\n\n    remotes = dest_repo.get_remotes(with_urls_only=True)\n    nremotes = len(remotes)\n    if nremotes == 1:\n        remote = remotes[0]\n        lgr.debug(\"Determined %s to be remote of %s\", remote, destds)\n    elif remotes > 1:\n        lgr.warning(\n            \"Fresh clone %s unexpected has multiple remotes: %s. Using %s\",\n            destds.path, remotes, remotes[0])\n        remote = remotes[0]\n    else:\n        raise RuntimeError(\"bug: fresh clone has zero remotes\")\n\n    if not cand.get(\"version\"):\n        postclone_check_head(destds, remote=remote)\n\n    # act on --reckless=shared-...\n    # must happen prior git-annex-init, where we can cheaply alter the repo\n    # setup through safe re-init'ing\n    if reckless and reckless.startswith('shared-'):\n        lgr.debug('Reinit %s to enable shared access permissions', destds)\n        destds.repo.call_git(['init', '--shared={}'.format(reckless[7:])])\n\n    # In case of RIA stores we need to prepare *before* annex is called at all\n    if result_props['source']['type'] == 'ria':\n        postclone_preannex_cfg_ria(destds, remote=remote)\n\n    yield from postclonecfg_annexdataset(\n        destds,\n        reckless,\n        description,\n        remote=remote)\n\n    if checkout_gitsha and \\\n       dest_repo.get_hexsha(dest_repo.get_corresponding_branch()) != checkout_gitsha:\n        try:\n            postclone_checkout_commit(dest_repo, checkout_gitsha,\n                                      remote=remote)\n        except Exception as e:\n            yield get_status_dict(\n                status='error',\n                message=str(e),\n                **result_props,\n            )\n\n            # We were supposed to clone a particular version but failed to.\n            # This is particularly pointless in case of subdatasets and\n            # potentially fatal with current implementation of recursion.\n            # see gh-5387\n            lgr.debug(\"Failed to checkout %s, removing this clone attempt at %s\", checkout_gitsha, dest_path)\n            # TODO stringification can be removed once pathlib compatible\n            # or if PY35 is no longer supported\n            rmtree(str(dest_path), children_only=dest_path_existed)\n            return\n\n    # perform any post-processing that needs to know details of the clone\n    # source\n    if result_props['source']['type'] == 'ria':\n        yield from postclonecfg_ria(destds, result_props['source'],\n                                    remote=remote)\n\n    if reckless:\n        # store the reckless setting in the dataset to make it\n        # known to later clones of subdatasets via get()\n        destds.config.set(\n            'datalad.clone.reckless', reckless,\n            where='local',\n            reload=True)\n    else:\n        # We would still want to reload configuration to ensure that any of the\n        # above git invocations could have potentially changed the config\n        # TODO: might no longer be necessary if 0.14.0 adds reloading upon\n        # non-readonly commands invocation\n        destds.config.reload()\n\n    # yield successful clone of the base dataset now, as any possible\n    # subdataset clone down below will not alter the Git-state of the\n    # parent\n    yield get_status_dict(status='ok', **result_props)\n\n\ndef postclone_checkout_commit(repo, target_commit, remote=\"origin\"):\n    \"\"\"Helper to check out a specific target commit in a fresh clone.\n\n    Will not check (again) if current commit and target commit are already\n    the same!\n    \"\"\"\n    # record what branch we were on right after the clone\n    active_branch = repo.get_active_branch()\n    corr_branch = repo.get_corresponding_branch(branch=active_branch)\n    was_adjusted = bool(corr_branch)\n    repo_orig_branch = corr_branch or active_branch\n    # if we are on a branch this hexsha will be the tip of that branch\n    repo_orig_hexsha = repo.get_hexsha(repo_orig_branch)\n    # make sure we have the desired commit locally\n    # expensive and possibly error-prone fetch conditional on cheap\n    # local check\n    if not repo.commit_exists(target_commit):\n        try:\n            repo.fetch(remote=remote, refspec=target_commit)\n        except CommandError:\n            pass\n        # instead of inspecting the fetch results for possible ways\n        # with which it could failed to produced the desired result\n        # let's verify the presence of the commit directly, we are in\n        # expensive-land already anyways\n        if not repo.commit_exists(target_commit):\n            # there is nothing we can do about this\n            # MIH thinks that removing the clone is not needed, as a likely\n            # next step will have to be a manual recovery intervention\n            # and not another blind attempt\n            raise ValueError(\n                'Target commit %s does not exist in the clone, and '\n                'a fetch that commit from remote failed'\n                % target_commit[:8])\n    # checkout the desired commit\n    repo.call_git(['checkout', target_commit])\n    # did we detach?\n    if repo_orig_branch and not repo.get_active_branch():\n        # trace if current state is a predecessor of the branch_hexsha\n        lgr.debug(\n            \"Detached HEAD after resetting worktree of %s \"\n            \"(original branch: %s)\", repo, repo_orig_branch)\n        if repo.get_merge_base(\n                [repo_orig_hexsha, target_commit]) == target_commit:\n            # we assume the target_commit to be from the same branch,\n            # because it is an ancestor -- update that original branch\n            # to point to the target_commit, and update HEAD to point to\n            # that location\n            lgr.info(\n                \"Reset branch '%s' to %s (from %s) to \"\n                \"avoid a detached HEAD\",\n                repo_orig_branch, target_commit[:8], repo_orig_hexsha[:8])\n            branch_ref = 'refs/heads/%s' % repo_orig_branch\n            repo.update_ref(branch_ref, target_commit)\n            repo.update_ref('HEAD', branch_ref, symbolic=True)\n            if was_adjusted:\n                # Note: The --force is needed because the adjust branch already\n                # exists.\n                repo.adjust(options=[\"--unlock\", \"--force\"])\n        else:\n            lgr.warning(\n                \"%s has a detached HEAD, because the target commit \"\n                \"%s has no unique ancestor with branch '%s'\",\n                repo, target_commit[:8], repo_orig_branch)\n\n\ndef postclone_check_head(ds, remote=\"origin\"):\n    repo = ds.repo\n    if not repo.commit_exists(\"HEAD\"):\n        # HEAD points to an unborn branch. A likely cause of this is that the\n        # remote's main branch is something other than master but HEAD wasn't\n        # adjusted accordingly.\n        #\n        # Let's choose the most recently updated remote ref (according to\n        # commit date). In the case of a submodule, switching to a ref with\n        # commits prevents .update_submodule() from failing. It is likely that\n        # the ref includes the registered commit, but we don't have the\n        # information here to know for sure. If it doesn't, .update_submodule()\n        # will check out a detached HEAD.\n        remote_branches = (\n            b[\"refname:strip=2\"] for b in repo.for_each_ref_(\n                fields=\"refname:strip=2\", sort=\"-committerdate\",\n                pattern=\"refs/remotes/\" + remote))\n        for rbranch in remote_branches:\n            if rbranch in [remote + \"/git-annex\", \"HEAD\"]:\n                continue\n            if rbranch.startswith(remote + \"/adjusted/\"):\n                # If necessary for this file system, a downstream\n                # git-annex-init call will handle moving into an\n                # adjusted state.\n                continue\n            repo.call_git([\"checkout\", \"-b\",\n                           rbranch[len(remote) + 1:],  # drop \"<remote>/\"\n                           \"--track\", rbranch])\n            lgr.debug(\"Checked out local branch from %s\", rbranch)\n            return\n        lgr.warning(\"Cloned %s but could not find a branch \"\n                    \"with commits\", ds.path)\n\n\ndef postclone_preannex_cfg_ria(ds, remote=\"origin\"):\n\n    # We need to annex-ignore the remote before annex-init is called on the clone,\n    # due to issues 5186 and 5253 (and we would have done it afterwards anyway).\n    # annex/objects in RIA stores is special for several reasons.\n    # 1. the remote doesn't know about it (no actual local annex for the remote)\n    # 2. RIA may use hashdir mixed, copying data to it via git-annex (if cloned\n    #    via ssh or local) would make it see a bare repo and establish a\n    #    hashdir lower annex object tree.\n    # 3. We want the ORA remote to receive all data for the store, so its\n    #    objects could be moved into archives (the main point of a RIA store).\n\n    # Note, that this function might need an enhancement as theoretically a RIA\n    # store could also hold simple standard annexes w/o an intended ORA remote.\n    # This needs the introduction of a new version label in RIA datasets, making\n    # the following call conditional.\n    ds.config.set(f'remote.{remote}.annex-ignore', 'true', where='local')\n\n\ndef postclonecfg_ria(ds, props, remote=\"origin\"):\n    \"\"\"Configure a dataset freshly cloned from a RIA store\"\"\"\n    repo = ds.repo\n\n    def get_uuid_from_store(store_url):\n        # First figure whether we cloned via SSH, HTTP or local path and then\n        # get that config file the same way:\n        config_content = None\n        scheme = store_url.split(':', 1)[0]\n        if scheme in ['http', 'https']:\n            try:\n                config_content = download_url(\n                    \"{}{}config\".format(\n                        store_url,\n                        '/' if not store_url.endswith('/') else ''))\n            except DownloadError as e:\n                lgr.debug(\"Failed to get config file from source:\\n%s\",\n                          exc_str(e))\n        elif scheme == 'ssh':\n            # TODO: switch the following to proper command abstraction:\n            # SSHRemoteIO ignores the path part ATM. No remote CWD! (To be\n            # changed with command abstractions). So we need to get that part to\n            # have a valid path to the remote's config file:\n            cfg_path = PurePosixPath(URL(store_url).path) / 'config'\n            op = SSHRemoteIO(store_url)\n            try:\n                config_content = op.read_file(cfg_path)\n            except RIARemoteError as e:\n                lgr.debug(\"Failed to get config file from source: %s\",\n                          exc_str(e))\n\n        elif scheme == 'file':\n            # TODO: switch the following to proper command abstraction:\n            op = LocalIO()\n            cfg_path = Path(URL(store_url).localpath) / 'config'\n            try:\n                config_content = op.read_file(cfg_path)\n            except (RIARemoteError, OSError) as e:\n                lgr.debug(\"Failed to get config file from source: %s\",\n                          exc_str(e))\n        else:\n            lgr.debug(\"Unknown URL-Scheme %s in %s. Can handle SSH, HTTP or \"\n                      \"FILE scheme URLs.\", scheme, props['source'])\n\n        # And read it\n        uuid = None\n        if config_content:\n            # TODO: We might be able to spare the saving to a file.\n            #       \"git config -f -\" is not explicitly documented but happens\n            #       to work and would read from stdin. Make sure we know this\n            #       works for required git versions and on all platforms.\n            with make_tempfile(content=config_content) as cfg_file:\n                runner = GitWitlessRunner()\n                try:\n                    result = runner.run(\n                        ['git', 'config', '-f', cfg_file,\n                         'datalad.ora-remote.uuid'],\n                        protocol=StdOutCapture\n                    )\n                    uuid = result['stdout'].strip()\n                except CommandError as e:\n                    # doesn't contain what we are looking for\n                    lgr.debug(\"Found no UUID for ORA special remote at \"\n                              \"'%s' (%s)\", remote, exc_str(e))\n\n        return uuid\n\n\n\n\n    # chances are that if this dataset came from a RIA store, its subdatasets\n    # may live there too. Place a subdataset source candidate config that makes\n    # get probe this RIA store when obtaining subdatasets\n    ria_store_url = props['source'].split('#', maxsplit=1)[0]\n    ds.config.set(\n        # we use the label 'origin' for this candidate in order to not have to\n        # generate a complicated name from the actual source specification.\n        # we pick a cost of 200 to sort it before datalad's default candidates\n        # for non-RIA URLs, because they prioritize hierarchical layouts that\n        # cannot be found in a RIA store\n        'datalad.get.subdataset-source-candidate-200origin',\n        # use the entire original URL, up to the fragment + plus dataset ID\n        # placeholder, this should make things work with any store setup we\n        # support (paths, ports, ...)\n        ria_store_url + '#{id}',\n        where='local')\n\n    # setup publication dependency, if a corresponding special remote exists\n    # and was enabled (there could be RIA stores that actually only have repos)\n    # make this function be a generator\n    ora_remotes = [s for s in ds.siblings('query', result_renderer='disabled')\n                   if s.get('annex-externaltype') == 'ora']\n    # get full special remotes' config for access to stored URL\n    srs = repo.get_special_remotes() \\\n        if hasattr(repo, 'get_special_remotes') else dict()\n\n    has_only_disabled_ora = \\\n        not ora_remotes and \\\n        any(r.get('externaltype') == 'ora' for r in srs.values())\n\n    def match_in_urls(special_remote_cfg, url_to_match):\n        # Figure whether either `url` or `push-url` in an ORA remote's config\n        # match a given URL (to a RIA store).\n        return special_remote_cfg['url'].startswith(url_to_match) or \\\n               (special_remote_cfg['push-url'].startswith(url_to_match)\n                if 'push-url' in special_remote_cfg else False)\n\n    no_enabled_ora_matches_url = \\\n        all(not match_in_urls(srs[r['annex-uuid']], ria_store_url)\n            for r in ora_remotes)\n\n    if has_only_disabled_ora or no_enabled_ora_matches_url:\n\n        # No ORA remote autoenabled, but configuration known about at least one,\n        # or enabled ORA remotes seem to not match clone URL.\n        # Let's check the remote's config for datalad.ora-remote.uuid as stored\n        # by create-sibling-ria and try enabling that one.\n        lgr.debug(\"Found no autoenabled ORA special remote. Trying to look it \"\n                  \"up in source config ...\")\n\n        org_uuid = get_uuid_from_store(props['giturl'])\n\n        # Now, enable it. If annex-init didn't fail to enable it as stored, we\n        # wouldn't end up here, so enable with store URL as suggested by the URL\n        # we cloned from.\n        if org_uuid:\n            if org_uuid in srs.keys():\n                # TODO: - Double-check autoenable value and only do this when\n                #         true?\n                #       - What if still fails? -> Annex shouldn't change config\n                #         in that case\n\n                # we only need the store:\n                new_url = props['source'].split('#')[0]\n                try:\n                    repo.enable_remote(srs[org_uuid]['name'],\n                                       options=['url={}'.format(new_url)]\n                                       )\n                    lgr.info(\"Reconfigured %s for %s\",\n                             srs[org_uuid]['name'], new_url)\n                    # update ora_remotes for considering publication dependency\n                    # below\n                    ora_remotes = [s for s in\n                                   ds.siblings('query',\n                                               result_renderer='disabled')\n                                   if s.get('annex-externaltype', None) ==\n                                   'ora']\n                except CommandError as e:\n                    lgr.debug(\"Failed to reconfigure ORA special remote: %s\",\n                              exc_str(e))\n            else:\n                lgr.debug(\"Unknown ORA special remote uuid at '%s': %s\",\n                          remote, org_uuid)\n\n    # Set publication dependency for `remote` on the respective ORA remote:\n    if ora_remotes:\n        url_matching_remotes = [r for r in ora_remotes\n                                if srs[r['annex-uuid']]['url'] == ria_store_url]\n\n        if len(url_matching_remotes) == 1:\n            # We have exactly one ORA remote with the same store URL we used for\n            # cloning (includes previously reconfigured remote).\n            # Set publication dependency:\n            yield from ds.siblings('configure',\n                                   name=remote,\n                                   publish_depends=url_matching_remotes[0]['name'],\n                                   result_filter=None,\n                                   result_renderer='disabled')\n\n        elif not url_matching_remotes:\n            # No matches but we have successfully autoenabled ORA remotes. Could\n            # be the same store accessed by different method (cloning via HTTP\n            # but special remote access via SSH). We can confidently set\n            # publication dependency if the store knows the UUID.\n            org_uuid = get_uuid_from_store(props['giturl'])\n            uuid_matching_remotes = [r for r in ora_remotes\n                                     if r['annex-uuid'] == org_uuid]\n            if uuid_matching_remotes:\n                # Multiple uuid matches are actually possible via same-as.\n                # However, in that case we can't decide which one is supposed to\n                # be used with publishing to `remote`.\n                if len(uuid_matching_remotes) == 1:\n                    yield from ds.siblings(\n                        'configure',\n                        name=remote,\n                        publish_depends=uuid_matching_remotes[0]['name'],\n                        result_filter=None,\n                        result_renderer='disabled')\n                else:\n                    lgr.warning(\n                        \"Found multiple matching ORA remotes. Couldn't decide \"\n                        \"which one publishing to '%s' should depend on: %s.\"\n                        \" Consider running 'datalad siblings configure -s \"\n                        \"%s --publish-depends ORAREMOTENAME' to set \"\n                        \"publication dependency manually.\",\n                        remote,\n                        [r['name'] for r in uuid_matching_remotes],\n                        remote)\n\n        else:\n            # We have multiple ORA remotes with the same store URL we cloned\n            # from.\n            lgr.warning(\"Found multiple matching ORA remotes. Couldn't decide \"\n                        \"which one publishing to '%s' should depend on: %s.\"\n                        \" Consider running 'datalad siblings configure -s \"\n                        \"%s --publish-depends ORAREMOTENAME' to set \"\n                        \"publication dependency manually.\",\n                        remote,\n                        [r['name'] for r in url_matching_remotes],\n                        remote)\n\n\ndef postclonecfg_annexdataset(ds, reckless, description=None, remote=\"origin\"):\n    \"\"\"If ds \"knows annex\" -- annex init it, set into reckless etc\n\n    Provides additional tune up to a possibly an annex repo, e.g.\n    \"enables\" reckless mode, sets up description\n    \"\"\"\n    # in any case check whether we need to annex-init the installed thing:\n    if not knows_annex(ds.path):\n        # not for us\n        return\n\n    # init annex when traces of a remote annex can be detected\n    if reckless == 'auto':\n        lgr.debug(\n            \"Instruct annex to hardlink content in %s from local \"\n            \"sources, if possible (reckless)\", ds.path)\n        ds.config.set(\n            'annex.hardlink', 'true', where='local', reload=True)\n\n    lgr.debug(\"Initializing annex repo at %s\", ds.path)\n    # Note, that we cannot enforce annex-init via AnnexRepo().\n    # If such an instance already exists, its __init__ will not be executed.\n    # Therefore do quick test once we have an object and decide whether to call\n    # its _init().\n    #\n    # Additionally, call init if we need to add a description (see #1403),\n    # since AnnexRepo.__init__ can only do it with create=True\n    repo = AnnexRepo(ds.path, init=True)\n    if not repo.is_initialized() or description:\n        repo._init(description=description)\n    if reckless == 'auto' or (reckless and reckless.startswith('shared-')):\n        repo.call_annex(['untrust', 'here'])\n\n    elif reckless == 'ephemeral':\n        # with ephemeral we declare 'here' as 'dead' right away, whenever\n        # we symlink the remote's annex, since availability from 'here' should\n        # not be propagated for an ephemeral clone when we publish back to\n        # the remote.\n        # This will cause stuff like this for a locally present annexed file:\n        # % git annex whereis d1\n        # whereis d1 (0 copies) failed\n        # BUT this works:\n        # % git annex find . --not --in here\n        # % git annex find . --in here\n        # d1\n\n        # we don't want annex copy-to <remote>\n        ds.config.set(\n            f'remote.{remote}.annex-ignore', 'true',\n            where='local')\n\n        ds.repo.set_remote_dead('here')\n\n        if check_symlink_capability(ds.repo.dot_git / 'dl_link_test',\n                                    ds.repo.dot_git / 'dl_target_test'):\n            # symlink the annex to avoid needless copies in an ephemeral clone\n            annex_dir = ds.repo.dot_git / 'annex'\n            origin_annex_url = ds.config.get(f\"remote.{remote}.url\", None)\n            origin_git_path = None\n            if origin_annex_url:\n                try:\n                    # Deal with file:// scheme URLs as well as plain paths.\n                    # If origin isn't local, we have nothing to do.\n                    origin_git_path = Path(RI(origin_annex_url).localpath)\n\n                    # we are local; check for a bare repo first to not mess w/\n                    # the path\n                    if GitRepo(origin_git_path, create=False).bare:\n                        # origin is a bare repo -> use path as is\n                        pass\n                    elif origin_git_path.name != '.git':\n                        origin_git_path /= '.git'\n                except ValueError:\n                    # Note, that accessing localpath on a non-local RI throws\n                    # ValueError rather than resulting in an AttributeError.\n                    # TODO: Warning level okay or is info level sufficient?\n                    # Note, that setting annex-dead is independent of\n                    # symlinking .git/annex. It might still make sense to\n                    # have an ephemeral clone that doesn't propagate its avail.\n                    # info. Therefore don't fail altogether.\n                    lgr.warning(\"reckless=ephemeral mode: %s doesn't seem \"\n                                \"local: %s\\nno symlinks being used\",\n                                remote, origin_annex_url)\n            if origin_git_path:\n                # TODO make sure that we do not delete any unique data\n                rmtree(str(annex_dir)) \\\n                    if not annex_dir.is_symlink() else annex_dir.unlink()\n                annex_dir.symlink_to(origin_git_path / 'annex',\n                                     target_is_directory=True)\n        else:\n            # TODO: What level? + note, that annex-dead is independent\n            lgr.warning(\"reckless=ephemeral mode: Unable to create symlinks on \"\n                        \"this file system.\")\n\n    srs = {True: [], False: []}  # special remotes by \"autoenable\" key\n    remote_uuids = None  # might be necessary to discover known UUIDs\n\n    repo_config = repo.config\n    # Note: The purpose of this function is to inform the user. So if something\n    # looks misconfigured, we'll warn and move on to the next item.\n    for uuid, config in repo.get_special_remotes().items():\n        sr_name = config.get('name', None)\n        if sr_name is None:\n            lgr.warning(\n                'Ignoring special remote %s because it does not have a name. '\n                'Known information: %s',\n                uuid, config)\n            continue\n        sr_autoenable = config.get('autoenable', False)\n        try:\n            sr_autoenable = ensure_bool(sr_autoenable)\n        except ValueError:\n            lgr.warning(\n                'Failed to process \"autoenable\" value %r for sibling %s in '\n                'dataset %s as bool.'\n                'You might need to enable it later manually and/or fix it up to'\n                ' avoid this message in the future.',\n                sr_autoenable, sr_name, ds.path)\n            continue\n\n        # If it looks like a type=git special remote, make sure we have up to\n        # date information. See gh-2897.\n        if sr_autoenable and repo_config.get(\"remote.{}.fetch\".format(sr_name)):\n            try:\n                repo.fetch(remote=sr_name)\n            except CommandError as exc:\n                lgr.warning(\"Failed to fetch type=git special remote %s: %s\",\n                            sr_name, exc_str(exc))\n\n        # determine whether there is a registered remote with matching UUID\n        if uuid:\n            if remote_uuids is None:\n                remote_uuids = {\n                    # Check annex-config-uuid first. For sameas annex remotes,\n                    # this will point to the UUID for the configuration (i.e.\n                    # the key returned by get_special_remotes) rather than the\n                    # shared UUID.\n                    (repo_config.get('remote.%s.annex-config-uuid' % r) or\n                     repo_config.get('remote.%s.annex-uuid' % r))\n                    for r in repo.get_remotes()\n                }\n            if uuid not in remote_uuids:\n                srs[sr_autoenable].append(sr_name)\n\n    if srs[True]:\n        lgr.debug(\n            \"configuration for %s %s added because of autoenable,\"\n            \" but no UUIDs for them yet known for dataset %s\",\n            # since we are only at debug level, we could call things their\n            # proper names\n            single_or_plural(\"special remote\",\n                             \"special remotes\", len(srs[True]), True),\n            \", \".join(srs[True]),\n            ds.path\n        )\n\n    if srs[False]:\n        # if has no auto-enable special remotes\n        lgr.info(\n            'access to %s %s not auto-enabled, enable with:\\n'\n            '\\t\\tdatalad siblings -d \"%s\" enable -s %s',\n            # but since humans might read it, we better confuse them with our\n            # own terms!\n            single_or_plural(\"dataset sibling\",\n                             \"dataset siblings\", len(srs[False]), True),\n            \", \".join(srs[False]),\n            ds.path,\n            srs[False][0] if len(srs[False]) == 1 else \"SIBLING\",\n        )\n\n    # we have just cloned the repo, so it has a remote `remote`, configure any\n    # reachable origin of origins\n    yield from configure_origins(ds, ds, remote=remote)\n\n\n_handle_possible_annex_dataset = postclonecfg_annexdataset\n\n\ndef configure_origins(cfgds, probeds, label=None, remote=\"origin\"):\n    \"\"\"Configure any discoverable local dataset sibling as a remote\n\n    Parameters\n    ----------\n    cfgds : Dataset\n      Dataset to receive the remote configurations\n    probeds : Dataset\n      Dataset to start looking for `remote` remotes. May be identical with\n      `cfgds`.\n    label : int, optional\n      Each discovered remote will be configured as a remote under the name\n      '<remote>-<label>'. If no label is given, '2' will be used by default,\n      given that there is typically a remote named `remote` already.\n    remote : str, optional\n      Name of the default remote on clone.\n    \"\"\"\n    if label is None:\n        label = 1\n    # let's look at the URL for that remote and see if it is a local\n    # dataset\n    origin_url = probeds.config.get(f'remote.{remote}.url')\n    if not origin_url:\n        # no remote with default name, nothing to do\n        return\n    if not cfgds.config.obtain(\n            'datalad.install.inherit-local-origin',\n            default=True):\n        # no inheritance wanted\n        return\n    if not isinstance(RI(origin_url), PathRI):\n        # not local path\n        return\n\n    # no need to reconfigure original/direct remote again\n    if cfgds != probeds:\n        # prevent duplicates\n        known_remote_urls = set(\n            cfgds.config.get(r + '.url', None)\n            for r in cfgds.config.sections()\n            if r.startswith('remote.')\n        )\n        if origin_url not in known_remote_urls:\n            yield from cfgds.siblings(\n                'configure',\n                # no chance for conflict, can only be the second configured\n                # remote\n                name='{}-{}'.format(remote, label),\n                url=origin_url,\n                # fetch to get all annex info\n                fetch=True,\n                result_renderer='disabled',\n                on_failure='ignore',\n            )\n    # and dive deeper\n    # given the clone source is a local dataset, we can have a\n    # cheap look at it, and configure its own `remote` as a remote\n    # (if there is any), and benefit from additional annex availability\n    yield from configure_origins(\n        cfgds,\n        Dataset(probeds.pathobj / origin_url),\n        label=label + 1,\n        remote=remote)\n\n\ndef _get_tracking_source(ds):\n    \"\"\"Returns name and url of a potential configured source\n    tracking remote\"\"\"\n    vcs = ds.repo\n    # if we have a remote, let's check the location of that remote\n    # for the presence of the desired submodule\n\n    remote_name, tracking_branch = vcs.get_tracking_branch()\n    if not remote_name and isinstance(vcs, AnnexRepo):\n        # maybe cloned from a source repo that was in adjusted mode\n        # https://github.com/datalad/datalad/issues/3969\n        remote_name, tracking_branch = vcs.get_tracking_branch(\n            corresponding=False)\n    # TODO: better default `None`? Check where we might rely on '':\n    remote_url = ''\n    if remote_name:\n        remote_url = vcs.get_remote_url(remote_name, push=False)\n\n    return remote_name, remote_url\n\n\ndef _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path\n\n\ndef decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n\n    # Git never gets to see these URLs, so let's manually apply any\n    # rewrite configuration Git might know about.\n    # Note: We need to rewrite before parsing, otherwise parsing might go wrong.\n    # This is particularly true for insteadOf labels replacing even the URL\n    # scheme.\n    spec = cfg.rewrite_url(spec)\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if re.match(uuid_regex, dsid):\n            trace = '{}/{}'.format(dsid[:3], dsid[3:])\n            default_destpath = dsid\n        elif dsid.startswith('~'):\n            trace = 'alias/{}'.format(dsid[1:])\n            default_destpath = dsid[1:]\n        else:\n            raise ValueError(\n                'RIA URI not recognized, no valid dataset ID or other supported '\n                'scheme: {}'.format(spec))\n        # now we cancel the fragment in the original URL, but keep everything else\n        # in order to be able to support the various combinations of ports, paths,\n        # and everything else\n        source_ri.fragment = ''\n        # strip the custom protocol and go with standard one\n        source_ri.scheme = source_ri.scheme[4:]\n        # take any existing path, and add trace to dataset within the store\n        source_ri.path = '{urlpath}{urldelim}{trace}'.format(\n            urlpath=source_ri.path if source_ri.path else '',\n            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',\n            trace=trace,\n        )\n        props.update(\n            type='ria',\n            giturl=str(source_ri),\n            version=version,\n            default_destpath=default_destpath,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        props['type'] = 'giturl'\n        # use original input verbatim\n        props['giturl'] = spec\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Plumbing command for dataset installation\"\"\"\n\n\nimport logging\nimport re\nfrom os.path import expanduser\nfrom collections import OrderedDict\nfrom urllib.parse import unquote as urlunquote\n\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import eval_results\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.common_opts import (\n    location_description,\n    reckless_opt,\n)\nfrom datalad.log import log_progress\nfrom datalad.support.gitrepo import (\n    GitRepo,\n)\nfrom datalad.cmd import (\n    CommandError,\n    GitWitlessRunner,\n    StdOutCapture,\n)\nfrom datalad.distributed.ora_remote import (\n    LocalIO,\n    RIARemoteError,\n    SSHRemoteIO,\n)\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.constraints import (\n    EnsureNone,\n    EnsureStr,\n    EnsureKeyChoice,\n)\nfrom datalad.support.exceptions import DownloadError\nfrom datalad.support.param import Parameter\nfrom datalad.support.strings import get_replacement_dict\nfrom datalad.support.network import (\n    DataLadRI,\n    PathRI,\n    RI,\n    SSHRI,\n    URL,\n    download_url,\n    get_local_file_url,\n    is_url,\n)\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\nfrom datalad.utils import (\n    ensure_bool,\n    ensure_list,\n    knows_annex,\n    make_tempfile,\n    Path,\n    PurePosixPath,\n    rmtree,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    resolve_path,\n    require_dataset,\n    EnsureDataset,\n)\nfrom datalad.distribution.utils import (\n    _get_flexible_source_candidates,\n)\nfrom datalad.utils import (\n    check_symlink_capability\n)\n\n__docformat__ = 'restructuredtext'\n\nlgr = logging.getLogger('datalad.core.distributed.clone')\n\n\n@build_doc\nclass Clone(Interface):\n    \"\"\"Obtain a dataset (copy) from a URL or local directory\n\n    The purpose of this command is to obtain a new clone (copy) of a dataset\n    and place it into a not-yet-existing or empty directory. As such `clone`\n    provides a strict subset of the functionality offered by `install`. Only a\n    single dataset can be obtained, and immediate recursive installation of\n    subdatasets is not supported. However, once a (super)dataset is installed\n    via `clone`, any content, including subdatasets can be obtained by a\n    subsequent `get` command.\n\n    Primary differences over a direct `git clone` call are 1) the automatic\n    initialization of a dataset annex (pure Git repositories are equally\n    supported); 2) automatic registration of the newly obtained dataset as a\n    subdataset (submodule), if a parent dataset is specified; 3) support\n    for additional resource identifiers (DataLad resource identifiers as used\n    on datasets.datalad.org, and RIA store URLs as used for store.datalad.org\n    - optionally in specific versions as identified by a branch or a tag; see\n    examples); and 4) automatic configurable generation of alternative access\n    URL for common cases (such as appending '.git' to the URL in case the\n    accessing the base URL failed).\n\n    In case the clone is registered as a subdataset, the original URL passed to\n    `clone` is recorded in `.gitmodules` of the parent dataset in addition\n    to the resolved URL used internally for git-clone. This allows to preserve\n    datalad specific URLs like ria+ssh://... for subsequent calls to `get` if\n    the subdataset was locally removed later on.\n\n    || PYTHON >>By default, the command returns a single Dataset instance for\n    an installed dataset, regardless of whether it was newly installed ('ok'\n    result), or found already installed from the specified source ('notneeded'\n    result).<< PYTHON ||\n\n    URL mapping configuration\n\n    'clone' supports the transformation of URLs via (multi-part) substitution\n    specifications. A substitution specification is defined as a configuration\n    setting 'datalad.clone.url-substition.<seriesID>' with a string containing\n    a match and substitution expression, each following Python's regular\n    expression syntax. Both expressions are concatenated to a single string\n    with an arbitrary delimiter character. The delimiter is defined by\n    prefixing the string with the delimiter. Prefix and delimiter are stripped\n    from the expressions (Example: \",^http://(.*)$,https://\\\\1\").  This setting\n    can be defined multiple times, using the same '<seriesID>'.  Substitutions\n    in a series will be applied incrementally, in order of their definition.\n    The first substitution in such a series must match, otherwise no further\n    substitutions in a series will be considered. However, following the first\n    match all further substitutions in a series are processed, regardless\n    whether intermediate expressions match or not. Substitution series themselves\n    have no particular order, each matching series will result in a candidate\n    clone URL. Consequently, the initial match specification in a series should\n    be as precise as possible to prevent inflation of candidate URLs.\n\n    .. seealso::\n\n      :ref:`handbook:3-001`\n        More information on Remote Indexed Archive (RIA) stores\n    \"\"\"\n    # by default ignore everything but install results\n    # i.e. no \"add to super dataset\"\n    result_filter = EnsureKeyChoice('action', ('install',))\n    # very frequently this command will yield exactly one installed dataset\n    # spare people the pain of going through a list by default\n    return_type = 'item-or-list'\n    # as discussed in #1409 and #1470, we want to return dataset instances\n    # matching what is actually available after command completion (and\n    # None for any failed dataset installation)\n    result_xfm = 'successdatasets-or-none'\n\n    _examples_ = [\n        dict(text=\"Install a dataset from Github into the current directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install a dataset into a specific directory\",\n             code_py=\"\"\"\\\n             clone(source='https://github.com/datalad-datasets/longnow-podcasts.git',\n                   path='myfavpodcasts')\"\"\",\n             code_cmd=\"\"\"\\\n             datalad clone https://github.com/datalad-datasets/longnow-podcasts.git \\\\\n             myfavpodcasts\"\"\"),\n        dict(text=\"Install a dataset as a subdataset into the current dataset\",\n             code_py=\"\"\"\\\n             clone(dataset='.',\n                   source='https://github.com/datalad-datasets/longnow-podcasts.git')\"\"\",\n             code_cmd=\"datalad clone -d . \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install the main superdataset from datasets.datalad.org\",\n             code_py=\"clone(source='///')\",\n             code_cmd=\"datalad clone ///\"),\n        dict(text=\"Install a dataset identified by a literal alias from store.datalad.org\",\n             code_py=\"clone(source='ria+http://store.datalad.org#~hcp-openaccess')\",\n             code_cmd=\"datalad clone ria+http://store.datalad.org#~hcp-openaccess\"),\n        dict(\n            text=\"Install a dataset in a specific version as identified by a \"\n                 \"branch or tag name from store.datalad.org\",\n            code_py=\"clone(source='ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier')\",\n            code_cmd=\"datalad clone ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier\"),\n        dict(\n            text=\"Install a dataset with group-write access permissions\",\n            code_py=\\\n            \"clone(source='http://example.com/dataset', reckless='shared-group')\",\n            code_cmd=\\\n            \"datalad clone http://example.com/dataset --reckless shared-group\"),\n    ]\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"(parent) dataset to clone into. If given, the newly cloned\n            dataset is registered as a subdataset of the parent. Also, if given,\n            relative paths are interpreted as being relative to the parent\n            dataset, and not relative to the working directory.\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        source=Parameter(\n            args=(\"source\",),\n            metavar='SOURCE',\n            doc=\"\"\"URL, DataLad resource identifier, local path or instance of\n            dataset to be cloned\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            nargs=\"?\",\n            doc=\"\"\"path to clone into.  If no `path` is provided a\n            destination path will be derived from a source URL\n            similar to :command:`git clone`\"\"\"),\n        description=location_description,\n        reckless=reckless_opt,\n    )\n\n    @staticmethod\n    @datasetmethod(name='clone')\n    @eval_results\n    def __call__(\n            source,\n            path=None,\n            dataset=None,\n            description=None,\n            reckless=None):\n        # did we explicitly get a dataset to install into?\n        # if we got a dataset, path will be resolved against it.\n        # Otherwise path will be resolved first.\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='clone') \\\n            if dataset is not None else dataset\n        refds_path = ds.path if ds else None\n\n        # legacy compatibility\n        if reckless is True:\n            # so that we can forget about how things used to be\n            reckless = 'auto'\n\n        if isinstance(source, Dataset):\n            source = source.path\n\n        if source == path:\n            # even if they turn out to be identical after resolving symlinks\n            # and more sophisticated witchcraft, it would still happily say\n            # \"it appears to be already installed\", so we just catch an\n            # obviously pointless input combination\n            raise ValueError(\n                \"clone `source` and destination `path` are identical [{}]. \"\n                \"If you are trying to add a subdataset simply use `save`\".format(\n                    path))\n\n        if path is not None:\n            path = resolve_path(path, dataset)\n\n        # derive target from source:\n        if path is None:\n            # we got nothing but a source. do something similar to git clone\n            # and derive the path from the source and continue\n            # since this is a relative `path`, resolve it:\n            # we are not going to reuse the decoded URL, as this is done for\n            # all source candidates in clone_dataset(), we just use to determine\n            # a destination path here in order to perform a bunch of additional\n            # checks that shall not pollute the helper function\n            source_ = decode_source_spec(\n                source, cfg=None if ds is None else ds.config)\n            path = resolve_path(source_['default_destpath'], dataset)\n            lgr.debug(\"Determined clone target path from source\")\n        lgr.debug(\"Resolved clone target path to: '%s'\", path)\n\n        # there is no other way -- my intoxicated brain tells me\n        assert(path is not None)\n\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            refds=refds_path,\n            source_url=source)\n\n        try:\n            # this will implicitly cause pathlib to run a bunch of checks\n            # whether the present path makes any sense on the platform\n            # we are running on -- we don't care if the path actually\n            # exists at this point, but we want to abort early if the path\n            # spec is determined to be useless\n            path.exists()\n        except OSError as e:\n            yield get_status_dict(\n                status='error',\n                path=path,\n                message=('cannot handle target path: %s', exc_str(e)),\n                **result_props)\n            return\n\n        destination_dataset = Dataset(path)\n        result_props['ds'] = destination_dataset\n\n        if ds is not None and ds.pathobj not in path.parents:\n            yield get_status_dict(\n                status='error',\n                message=(\"clone target path '%s' not in specified target dataset '%s'\",\n                         path, ds),\n                **result_props)\n            return\n\n        # perform the actual cloning operation\n        yield from clone_dataset(\n            [source],\n            destination_dataset,\n            reckless,\n            description,\n            result_props,\n            cfg=None if ds is None else ds.config,\n        )\n\n        # TODO handle any 'version' property handling and verification using a\n        # dedicated public helper\n\n        if ds is not None:\n            # we created a dataset in another dataset\n            # -> make submodule\n            actually_saved_subds = False\n            for r in ds.save(\n                    path,\n                    # Note, that here we know we don't save anything but a new\n                    # subdataset. Hence, don't go with default commit message,\n                    # but be more specific.\n                    message=\"[DATALAD] Added subdataset\",\n                    return_type='generator',\n                    result_filter=None,\n                    result_xfm=None,\n                    on_failure='ignore'):\n                actually_saved_subds = actually_saved_subds or (\n                        r['action'] == 'save' and\n                        r['type'] == 'dataset' and\n                        r['refds'] == ds.path and\n                        r['status'] == 'ok')\n                yield r\n\n            # Modify .gitmodules to contain originally given url. This is\n            # particularly relevant for postclone routines on a later `get`\n            # for that subdataset. See gh-5256.\n            if actually_saved_subds:\n                # New subdataset actually saved. Amend the modification\n                # of .gitmodules. Note, that we didn't allow to deviate\n                # from git default behavior WRT a submodule's name vs\n                # its path when we made this a new subdataset.\n                subds_name = path.relative_to(ds.pathobj)\n                ds.repo.call_git(\n                    ['config',\n                     '--file',\n                     '.gitmodules',\n                     '--replace-all',\n                     'submodule.{}.{}'.format(subds_name,\n                                              \"datalad-url\"),\n                     source]\n                )\n                yield from ds.save('.gitmodules',\n                                   amend=True, to_git=True)\n            else:\n                # We didn't really commit. Just call `subdatasets`\n                # in that case to have the modification included in the\n                # post-clone state (whatever that may be).\n                ds.subdatasets(path, set_property=[(\"datalad-url\", source)])\n\n\ndef _get_url_mappings(cfg):\n    cfg_prefix = 'datalad.clone.url-substitute.'\n    # figure out which keys we should be looking for\n    # in the active config\n    subst_keys = set(k for k in cfg.keys() if k.startswith(cfg_prefix))\n    # and in the common config specs\n    from datalad.interface.common_cfg import definitions\n    subst_keys.update(k for k in definitions if k.startswith(cfg_prefix))\n    # TODO a potential sorting of substitution series could be implemented\n    # here\n    return [\n        # decode the rule specifications\n        get_replacement_dict(\n            # one or more could come out\n            ensure_list(\n                cfg.get(\n                    k,\n                    # make sure to pull the default from the common config\n                    default=cfg.obtain(k),\n                    # we specifically support declaration of multiple\n                    # settings to build replacement chains\n                    get_all=True)))\n        for k in subst_keys\n    ]\n\n\ndef _map_urls(cfg, urls):\n    mapping_specs = _get_url_mappings(cfg)\n    if not mapping_specs:\n        return urls\n\n    mapped = []\n    # we process the candidate in order to maintain any prioritization\n    # encoded in it (e.g. _get_flexible_source_candidates_for_submodule)\n    # if we have a matching mapping replace the URL in its position\n    for u in urls:\n        # we only permit a single match\n        # TODO we likely want to RF this to pick the longest match\n        mapping_applied = False\n        # try one mapping set at a time\n        for mapping_spec in mapping_specs:\n            # process all substitution patterns in the specification\n            # always operate on strings (could be a Path instance too)\n            mu = str(u)\n            matched = False\n            for match_ex, subst_ex in mapping_spec.items():\n                if not matched:\n                    matched = re.match(match_ex, mu) is not None\n                if not matched:\n                    break\n                # try to map, would return unchanged, if there is no match\n                mu = re.sub(match_ex, subst_ex, mu)\n            if mu != u:\n                lgr.debug(\"URL substitution: '%s' -> '%s'\", u, mu)\n                mapped.append(mu)\n                # we could consider breaking after the for effective mapping\n                # specification. however, that would mean any generic\n                # definition of a broadly matching substitution would derail\n                # the entroe system. moreover, suddently order would matter\n                # substantially\n                mapping_applied = True\n        if not mapping_applied:\n            # none of the mappings matches, go with the original URL\n            # (really original, not the stringified one)\n            mapped.append(u)\n    return mapped\n\n\ndef clone_dataset(\n        srcs,\n        destds,\n        reckless=None,\n        description=None,\n        result_props=None,\n        cfg=None,\n        checkout_gitsha=None):\n    \"\"\"Internal helper to perform cloning without sanity checks (assumed done)\n\n    This helper does not handle any saving of subdataset modification or adding\n    in a superdataset.\n\n    Parameters\n    ----------\n    srcs : list\n      Any suitable clone source specifications (paths, URLs)\n    destds : Dataset\n      Dataset instance for the clone destination\n    reckless : {None, 'auto', 'ephemeral', 'shared-...'}, optional\n      Mode switch to put cloned dataset into unsafe/throw-away configurations, i.e.\n      sacrifice data safety for performance or resource footprint. When None\n      and `cfg` is specified, use the value of `datalad.clone.reckless`.\n    description : str, optional\n      Location description for the annex of the dataset clone (if there is any).\n    result_props : dict, optional\n      Default properties for any yielded result, passed on to get_status_dict().\n    cfg : ConfigManager, optional\n      Configuration for parent dataset. This will be queried instead\n      of the global DataLad configuration.\n    checkout_gitsha : str, optional\n      If given, a specific commit, identified by shasum, will be checked out after\n      cloning. A dedicated follow-up fetch will be performed, if the initial clone\n      did not obtain the commit object. Should the checkout of the target commit\n      cause a detached HEAD, the previously active branch will be reset to the\n      target commit.\n\n    Yields\n    ------\n    dict\n      DataLad result records\n    \"\"\"\n    if not result_props:\n        # in case the caller had no specific idea on how results should look\n        # like, provide sensible defaults\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            ds=destds,\n        )\n\n    if reckless is None and cfg:\n        # if reckless is not explicitly given, but we operate on a\n        # superdataset, query whether it has been instructed to operate\n        # in a reckless mode, and inherit it for the coming clone\n        reckless = cfg.get('datalad.clone.reckless', None)\n\n    dest_path = destds.pathobj\n\n    # check for configured URL mappings, either in the given config manager\n    # or in the one of the destination dataset, which is typically not existent\n    # yet and the process config manager is then used effectively\n    srcs = _map_urls(cfg or destds.config, srcs)\n\n    # decode all source candidate specifications\n    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]\n\n    # now expand the candidate sources with additional variants of the decoded\n    # giturl, while duplicating the other properties in the additional records\n    # for simplicity. The hope is to overcome a few corner cases and be more\n    # robust than git clone\n    candidate_sources = [\n        dict(props, giturl=s) for props in candidate_sources\n        for s in _get_flexible_source_candidates(props['giturl'])\n    ]\n\n    # important test! based on this `rmtree` will happen below after failed clone\n    dest_path_existed = dest_path.exists()\n    if dest_path_existed and any(dest_path.iterdir()):\n        if destds.is_installed():\n            # check if dest was cloned from the given source before\n            # this is where we would have installed this from\n            # this is where it was actually installed from\n            track_name, track_url = _get_tracking_source(destds)\n            try:\n                # this will get us track_url in system native path conventions,\n                # whenever it is a path (and not a URL)\n                # this is needed to match it to any potentially incoming local\n                # source path in the 'notneeded' test below\n                track_path = str(Path(track_url))\n            except Exception:\n                # this should never happen, because Path() will let any non-path stringification\n                # pass through unmodified, but we do not want any potential crash due to\n                # pathlib behavior changes\n                lgr.debug(\"Unexpected behavior of pathlib!\")\n                track_path = None\n            for cand in candidate_sources:\n                src = cand['giturl']\n                if track_url == src \\\n                        or (not is_url(track_url)\n                            and get_local_file_url(track_url, compatibility='git') == src) \\\n                        or track_path == expanduser(src):\n                    yield get_status_dict(\n                        status='notneeded',\n                        message=(\"dataset %s was already cloned from '%s'\",\n                                 destds,\n                                 src),\n                        **result_props)\n                    return\n        # anything else is an error\n        yield get_status_dict(\n            status='error',\n            message='target path already exists and not empty, refuse to clone into target path',\n            **result_props)\n        return\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Cloning dataset to %s', destds,\n        total=len(candidate_sources),\n        label='Clone attempt',\n        unit=' Candidate locations',\n    )\n    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url\n    for cand in candidate_sources:\n        log_progress(\n            lgr.info,\n            'cloneds',\n            'Attempting to clone from %s to %s', cand['giturl'], dest_path,\n            update=1,\n            increment=True)\n\n        clone_opts = {}\n\n        if cand.get('version', None):\n            clone_opts['branch'] = cand['version']\n        try:\n            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35\n            # doesn't make it happen seamlessly\n            GitRepo.clone(\n                path=str(dest_path),\n                url=cand['giturl'],\n                clone_options=clone_opts,\n                create=True)\n\n        except CommandError as e:\n            e_stderr = e.stderr\n\n            error_msgs[cand['giturl']] = e\n            lgr.debug(\"Failed to clone from URL: %s (%s)\",\n                      cand['giturl'], exc_str(e))\n            if dest_path.exists():\n                lgr.debug(\"Wiping out unsuccessful clone attempt at: %s\",\n                          dest_path)\n                # We must not just rmtree since it might be curdir etc\n                # we should remove all files/directories under it\n                # TODO stringification can be removed once patlib compatible\n                # or if PY35 is no longer supported\n                rmtree(str(dest_path), children_only=dest_path_existed)\n\n            if e_stderr and 'could not create work tree' in e_stderr.lower():\n                # this cannot be fixed by trying another URL\n                re_match = re.match(r\".*fatal: (.*)$\", e_stderr,\n                                    flags=re.MULTILINE | re.DOTALL)\n                # cancel progress bar\n                log_progress(\n                    lgr.info,\n                    'cloneds',\n                    'Completed clone attempts for %s', destds\n                )\n                yield get_status_dict(\n                    status='error',\n                    message=re_match.group(1).strip()\n                    if re_match else \"stderr: \" + e_stderr,\n                    **result_props)\n                return\n            # next candidate\n            continue\n\n        result_props['source'] = cand\n        # do not bother with other sources if succeeded\n        break\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Completed clone attempts for %s', destds\n    )\n\n    if not destds.is_installed():\n        if len(error_msgs):\n            if all(not e.stdout and not e.stderr for e in error_msgs.values()):\n                # there is nothing we can learn from the actual exception,\n                # the exit code is uninformative, the command is predictable\n                error_msg = \"Failed to clone from all attempted sources: %s\"\n                error_args = list(error_msgs.keys())\n            else:\n                error_msg = \"Failed to clone from any candidate source URL. \" \\\n                            \"Encountered errors per each url were:\\n- %s\"\n                error_args = '\\n- '.join(\n                    '{}\\n  {}'.format(url, exc_str(exc))\n                    for url, exc in error_msgs.items()\n                )\n        else:\n            # yoh: Not sure if we ever get here but I felt that there could\n            #      be a case when this might happen and original error would\n            #      not be sufficient to troubleshoot what is going on.\n            error_msg = \"Awkward error -- we failed to clone properly. \" \\\n                        \"Although no errors were encountered, target \" \\\n                        \"dataset at %s seems to be not fully installed. \" \\\n                        \"The 'succesful' source was: %s\"\n            error_args = (destds.path, cand['giturl'])\n        yield get_status_dict(\n            status='error',\n            message=(error_msg, error_args),\n            **result_props)\n        return\n\n    dest_repo = destds.repo\n\n    remotes = dest_repo.get_remotes(with_urls_only=True)\n    nremotes = len(remotes)\n    if nremotes == 1:\n        remote = remotes[0]\n        lgr.debug(\"Determined %s to be remote of %s\", remote, destds)\n    elif remotes > 1:\n        lgr.warning(\n            \"Fresh clone %s unexpected has multiple remotes: %s. Using %s\",\n            destds.path, remotes, remotes[0])\n        remote = remotes[0]\n    else:\n        raise RuntimeError(\"bug: fresh clone has zero remotes\")\n\n    if not cand.get(\"version\"):\n        postclone_check_head(destds, remote=remote)\n\n    # act on --reckless=shared-...\n    # must happen prior git-annex-init, where we can cheaply alter the repo\n    # setup through safe re-init'ing\n    if reckless and reckless.startswith('shared-'):\n        lgr.debug('Reinit %s to enable shared access permissions', destds)\n        destds.repo.call_git(['init', '--shared={}'.format(reckless[7:])])\n\n    # In case of RIA stores we need to prepare *before* annex is called at all\n    if result_props['source']['type'] == 'ria':\n        postclone_preannex_cfg_ria(destds, remote=remote)\n\n    yield from postclonecfg_annexdataset(\n        destds,\n        reckless,\n        description,\n        remote=remote)\n\n    if checkout_gitsha and \\\n       dest_repo.get_hexsha(dest_repo.get_corresponding_branch()) != checkout_gitsha:\n        try:\n            postclone_checkout_commit(dest_repo, checkout_gitsha,\n                                      remote=remote)\n        except Exception as e:\n            yield get_status_dict(\n                status='error',\n                message=str(e),\n                **result_props,\n            )\n\n            # We were supposed to clone a particular version but failed to.\n            # This is particularly pointless in case of subdatasets and\n            # potentially fatal with current implementation of recursion.\n            # see gh-5387\n            lgr.debug(\"Failed to checkout %s, removing this clone attempt at %s\", checkout_gitsha, dest_path)\n            # TODO stringification can be removed once pathlib compatible\n            # or if PY35 is no longer supported\n            rmtree(str(dest_path), children_only=dest_path_existed)\n            return\n\n    # perform any post-processing that needs to know details of the clone\n    # source\n    if result_props['source']['type'] == 'ria':\n        yield from postclonecfg_ria(destds, result_props['source'],\n                                    remote=remote)\n\n    if reckless:\n        # store the reckless setting in the dataset to make it\n        # known to later clones of subdatasets via get()\n        destds.config.set(\n            'datalad.clone.reckless', reckless,\n            where='local',\n            reload=True)\n    else:\n        # We would still want to reload configuration to ensure that any of the\n        # above git invocations could have potentially changed the config\n        # TODO: might no longer be necessary if 0.14.0 adds reloading upon\n        # non-readonly commands invocation\n        destds.config.reload()\n\n    # yield successful clone of the base dataset now, as any possible\n    # subdataset clone down below will not alter the Git-state of the\n    # parent\n    yield get_status_dict(status='ok', **result_props)\n\n\ndef postclone_checkout_commit(repo, target_commit, remote=\"origin\"):\n    \"\"\"Helper to check out a specific target commit in a fresh clone.\n\n    Will not check (again) if current commit and target commit are already\n    the same!\n    \"\"\"\n    # record what branch we were on right after the clone\n    active_branch = repo.get_active_branch()\n    corr_branch = repo.get_corresponding_branch(branch=active_branch)\n    was_adjusted = bool(corr_branch)\n    repo_orig_branch = corr_branch or active_branch\n    # if we are on a branch this hexsha will be the tip of that branch\n    repo_orig_hexsha = repo.get_hexsha(repo_orig_branch)\n    # make sure we have the desired commit locally\n    # expensive and possibly error-prone fetch conditional on cheap\n    # local check\n    if not repo.commit_exists(target_commit):\n        try:\n            repo.fetch(remote=remote, refspec=target_commit)\n        except CommandError:\n            pass\n        # instead of inspecting the fetch results for possible ways\n        # with which it could failed to produced the desired result\n        # let's verify the presence of the commit directly, we are in\n        # expensive-land already anyways\n        if not repo.commit_exists(target_commit):\n            # there is nothing we can do about this\n            # MIH thinks that removing the clone is not needed, as a likely\n            # next step will have to be a manual recovery intervention\n            # and not another blind attempt\n            raise ValueError(\n                'Target commit %s does not exist in the clone, and '\n                'a fetch that commit from remote failed'\n                % target_commit[:8])\n    # checkout the desired commit\n    repo.call_git(['checkout', target_commit])\n    # did we detach?\n    if repo_orig_branch and not repo.get_active_branch():\n        # trace if current state is a predecessor of the branch_hexsha\n        lgr.debug(\n            \"Detached HEAD after resetting worktree of %s \"\n            \"(original branch: %s)\", repo, repo_orig_branch)\n        if repo.get_merge_base(\n                [repo_orig_hexsha, target_commit]) == target_commit:\n            # we assume the target_commit to be from the same branch,\n            # because it is an ancestor -- update that original branch\n            # to point to the target_commit, and update HEAD to point to\n            # that location\n            lgr.info(\n                \"Reset branch '%s' to %s (from %s) to \"\n                \"avoid a detached HEAD\",\n                repo_orig_branch, target_commit[:8], repo_orig_hexsha[:8])\n            branch_ref = 'refs/heads/%s' % repo_orig_branch\n            repo.update_ref(branch_ref, target_commit)\n            repo.update_ref('HEAD', branch_ref, symbolic=True)\n            if was_adjusted:\n                # Note: The --force is needed because the adjust branch already\n                # exists.\n                repo.adjust(options=[\"--unlock\", \"--force\"])\n        else:\n            lgr.warning(\n                \"%s has a detached HEAD, because the target commit \"\n                \"%s has no unique ancestor with branch '%s'\",\n                repo, target_commit[:8], repo_orig_branch)\n\n\ndef postclone_check_head(ds, remote=\"origin\"):\n    repo = ds.repo\n    if not repo.commit_exists(\"HEAD\"):\n        # HEAD points to an unborn branch. A likely cause of this is that the\n        # remote's main branch is something other than master but HEAD wasn't\n        # adjusted accordingly.\n        #\n        # Let's choose the most recently updated remote ref (according to\n        # commit date). In the case of a submodule, switching to a ref with\n        # commits prevents .update_submodule() from failing. It is likely that\n        # the ref includes the registered commit, but we don't have the\n        # information here to know for sure. If it doesn't, .update_submodule()\n        # will check out a detached HEAD.\n        remote_branches = (\n            b[\"refname:strip=2\"] for b in repo.for_each_ref_(\n                fields=\"refname:strip=2\", sort=\"-committerdate\",\n                pattern=\"refs/remotes/\" + remote))\n        for rbranch in remote_branches:\n            if rbranch in [remote + \"/git-annex\", \"HEAD\"]:\n                continue\n            if rbranch.startswith(remote + \"/adjusted/\"):\n                # If necessary for this file system, a downstream\n                # git-annex-init call will handle moving into an\n                # adjusted state.\n                continue\n            repo.call_git([\"checkout\", \"-b\",\n                           rbranch[len(remote) + 1:],  # drop \"<remote>/\"\n                           \"--track\", rbranch])\n            lgr.debug(\"Checked out local branch from %s\", rbranch)\n            return\n        lgr.warning(\"Cloned %s but could not find a branch \"\n                    \"with commits\", ds.path)\n\n\ndef postclone_preannex_cfg_ria(ds, remote=\"origin\"):\n\n    # We need to annex-ignore the remote before annex-init is called on the clone,\n    # due to issues 5186 and 5253 (and we would have done it afterwards anyway).\n    # annex/objects in RIA stores is special for several reasons.\n    # 1. the remote doesn't know about it (no actual local annex for the remote)\n    # 2. RIA may use hashdir mixed, copying data to it via git-annex (if cloned\n    #    via ssh or local) would make it see a bare repo and establish a\n    #    hashdir lower annex object tree.\n    # 3. We want the ORA remote to receive all data for the store, so its\n    #    objects could be moved into archives (the main point of a RIA store).\n\n    # Note, that this function might need an enhancement as theoretically a RIA\n    # store could also hold simple standard annexes w/o an intended ORA remote.\n    # This needs the introduction of a new version label in RIA datasets, making\n    # the following call conditional.\n    ds.config.set(f'remote.{remote}.annex-ignore', 'true', where='local')\n\n\ndef postclonecfg_ria(ds, props, remote=\"origin\"):\n    \"\"\"Configure a dataset freshly cloned from a RIA store\"\"\"\n    repo = ds.repo\n\n    def get_uuid_from_store(store_url):\n        # First figure whether we cloned via SSH, HTTP or local path and then\n        # get that config file the same way:\n        config_content = None\n        scheme = store_url.split(':', 1)[0]\n        if scheme in ['http', 'https']:\n            try:\n                config_content = download_url(\n                    \"{}{}config\".format(\n                        store_url,\n                        '/' if not store_url.endswith('/') else ''))\n            except DownloadError as e:\n                lgr.debug(\"Failed to get config file from source:\\n%s\",\n                          exc_str(e))\n        elif scheme == 'ssh':\n            # TODO: switch the following to proper command abstraction:\n            # SSHRemoteIO ignores the path part ATM. No remote CWD! (To be\n            # changed with command abstractions). So we need to get that part to\n            # have a valid path to the remote's config file:\n            cfg_path = PurePosixPath(URL(store_url).path) / 'config'\n            op = SSHRemoteIO(store_url)\n            try:\n                config_content = op.read_file(cfg_path)\n            except RIARemoteError as e:\n                lgr.debug(\"Failed to get config file from source: %s\",\n                          exc_str(e))\n\n        elif scheme == 'file':\n            # TODO: switch the following to proper command abstraction:\n            op = LocalIO()\n            cfg_path = Path(URL(store_url).localpath) / 'config'\n            try:\n                config_content = op.read_file(cfg_path)\n            except (RIARemoteError, OSError) as e:\n                lgr.debug(\"Failed to get config file from source: %s\",\n                          exc_str(e))\n        else:\n            lgr.debug(\"Unknown URL-Scheme %s in %s. Can handle SSH, HTTP or \"\n                      \"FILE scheme URLs.\", scheme, props['source'])\n\n        # And read it\n        uuid = None\n        if config_content:\n            # TODO: We might be able to spare the saving to a file.\n            #       \"git config -f -\" is not explicitly documented but happens\n            #       to work and would read from stdin. Make sure we know this\n            #       works for required git versions and on all platforms.\n            with make_tempfile(content=config_content) as cfg_file:\n                runner = GitWitlessRunner()\n                try:\n                    result = runner.run(\n                        ['git', 'config', '-f', cfg_file,\n                         'datalad.ora-remote.uuid'],\n                        protocol=StdOutCapture\n                    )\n                    uuid = result['stdout'].strip()\n                except CommandError as e:\n                    # doesn't contain what we are looking for\n                    lgr.debug(\"Found no UUID for ORA special remote at \"\n                              \"'%s' (%s)\", remote, exc_str(e))\n\n        return uuid\n\n\n\n\n    # chances are that if this dataset came from a RIA store, its subdatasets\n    # may live there too. Place a subdataset source candidate config that makes\n    # get probe this RIA store when obtaining subdatasets\n    ria_store_url = props['source'].split('#', maxsplit=1)[0]\n    ds.config.set(\n        # we use the label 'origin' for this candidate in order to not have to\n        # generate a complicated name from the actual source specification.\n        # we pick a cost of 200 to sort it before datalad's default candidates\n        # for non-RIA URLs, because they prioritize hierarchical layouts that\n        # cannot be found in a RIA store\n        'datalad.get.subdataset-source-candidate-200origin',\n        # use the entire original URL, up to the fragment + plus dataset ID\n        # placeholder, this should make things work with any store setup we\n        # support (paths, ports, ...)\n        ria_store_url + '#{id}',\n        where='local')\n\n    # setup publication dependency, if a corresponding special remote exists\n    # and was enabled (there could be RIA stores that actually only have repos)\n    # make this function be a generator\n    ora_remotes = [s for s in ds.siblings('query', result_renderer='disabled')\n                   if s.get('annex-externaltype') == 'ora']\n    # get full special remotes' config for access to stored URL\n    srs = repo.get_special_remotes() \\\n        if hasattr(repo, 'get_special_remotes') else dict()\n\n    has_only_disabled_ora = \\\n        not ora_remotes and \\\n        any(r.get('externaltype') == 'ora' for r in srs.values())\n\n    def match_in_urls(special_remote_cfg, url_to_match):\n        # Figure whether either `url` or `push-url` in an ORA remote's config\n        # match a given URL (to a RIA store).\n        return special_remote_cfg['url'].startswith(url_to_match) or \\\n               (special_remote_cfg['push-url'].startswith(url_to_match)\n                if 'push-url' in special_remote_cfg else False)\n\n    no_enabled_ora_matches_url = \\\n        all(not match_in_urls(srs[r['annex-uuid']], ria_store_url)\n            for r in ora_remotes)\n\n    if has_only_disabled_ora or no_enabled_ora_matches_url:\n\n        # No ORA remote autoenabled, but configuration known about at least one,\n        # or enabled ORA remotes seem to not match clone URL.\n        # Let's check the remote's config for datalad.ora-remote.uuid as stored\n        # by create-sibling-ria and try enabling that one.\n        lgr.debug(\"Found no autoenabled ORA special remote. Trying to look it \"\n                  \"up in source config ...\")\n\n        org_uuid = get_uuid_from_store(props['giturl'])\n\n        # Now, enable it. If annex-init didn't fail to enable it as stored, we\n        # wouldn't end up here, so enable with store URL as suggested by the URL\n        # we cloned from.\n        if org_uuid:\n            if org_uuid in srs.keys():\n                # TODO: - Double-check autoenable value and only do this when\n                #         true?\n                #       - What if still fails? -> Annex shouldn't change config\n                #         in that case\n\n                # we only need the store:\n                new_url = props['source'].split('#')[0]\n                try:\n                    repo.enable_remote(srs[org_uuid]['name'],\n                                       options=['url={}'.format(new_url)]\n                                       )\n                    lgr.info(\"Reconfigured %s for %s\",\n                             srs[org_uuid]['name'], new_url)\n                    # update ora_remotes for considering publication dependency\n                    # below\n                    ora_remotes = [s for s in\n                                   ds.siblings('query',\n                                               result_renderer='disabled')\n                                   if s.get('annex-externaltype', None) ==\n                                   'ora']\n                except CommandError as e:\n                    lgr.debug(\"Failed to reconfigure ORA special remote: %s\",\n                              exc_str(e))\n            else:\n                lgr.debug(\"Unknown ORA special remote uuid at '%s': %s\",\n                          remote, org_uuid)\n\n    # Set publication dependency for `remote` on the respective ORA remote:\n    if ora_remotes:\n        url_matching_remotes = [r for r in ora_remotes\n                                if srs[r['annex-uuid']]['url'] == ria_store_url]\n\n        if len(url_matching_remotes) == 1:\n            # We have exactly one ORA remote with the same store URL we used for\n            # cloning (includes previously reconfigured remote).\n            # Set publication dependency:\n            yield from ds.siblings('configure',\n                                   name=remote,\n                                   publish_depends=url_matching_remotes[0]['name'],\n                                   result_filter=None,\n                                   result_renderer='disabled')\n\n        elif not url_matching_remotes:\n            # No matches but we have successfully autoenabled ORA remotes. Could\n            # be the same store accessed by different method (cloning via HTTP\n            # but special remote access via SSH). We can confidently set\n            # publication dependency if the store knows the UUID.\n            org_uuid = get_uuid_from_store(props['giturl'])\n            uuid_matching_remotes = [r for r in ora_remotes\n                                     if r['annex-uuid'] == org_uuid]\n            if uuid_matching_remotes:\n                # Multiple uuid matches are actually possible via same-as.\n                # However, in that case we can't decide which one is supposed to\n                # be used with publishing to `remote`.\n                if len(uuid_matching_remotes) == 1:\n                    yield from ds.siblings(\n                        'configure',\n                        name=remote,\n                        publish_depends=uuid_matching_remotes[0]['name'],\n                        result_filter=None,\n                        result_renderer='disabled')\n                else:\n                    lgr.warning(\n                        \"Found multiple matching ORA remotes. Couldn't decide \"\n                        \"which one publishing to '%s' should depend on: %s.\"\n                        \" Consider running 'datalad siblings configure -s \"\n                        \"%s --publish-depends ORAREMOTENAME' to set \"\n                        \"publication dependency manually.\",\n                        remote,\n                        [r['name'] for r in uuid_matching_remotes],\n                        remote)\n\n        else:\n            # We have multiple ORA remotes with the same store URL we cloned\n            # from.\n            lgr.warning(\"Found multiple matching ORA remotes. Couldn't decide \"\n                        \"which one publishing to '%s' should depend on: %s.\"\n                        \" Consider running 'datalad siblings configure -s \"\n                        \"%s --publish-depends ORAREMOTENAME' to set \"\n                        \"publication dependency manually.\",\n                        remote,\n                        [r['name'] for r in url_matching_remotes],\n                        remote)\n\n\ndef postclonecfg_annexdataset(ds, reckless, description=None, remote=\"origin\"):\n    \"\"\"If ds \"knows annex\" -- annex init it, set into reckless etc\n\n    Provides additional tune up to a possibly an annex repo, e.g.\n    \"enables\" reckless mode, sets up description\n    \"\"\"\n    # in any case check whether we need to annex-init the installed thing:\n    if not knows_annex(ds.path):\n        # not for us\n        return\n\n    # init annex when traces of a remote annex can be detected\n    if reckless == 'auto':\n        lgr.debug(\n            \"Instruct annex to hardlink content in %s from local \"\n            \"sources, if possible (reckless)\", ds.path)\n        ds.config.set(\n            'annex.hardlink', 'true', where='local', reload=True)\n\n    lgr.debug(\"Initializing annex repo at %s\", ds.path)\n    # Note, that we cannot enforce annex-init via AnnexRepo().\n    # If such an instance already exists, its __init__ will not be executed.\n    # Therefore do quick test once we have an object and decide whether to call\n    # its _init().\n    #\n    # Additionally, call init if we need to add a description (see #1403),\n    # since AnnexRepo.__init__ can only do it with create=True\n    repo = AnnexRepo(ds.path, init=True)\n    if not repo.is_initialized() or description:\n        repo._init(description=description)\n    if reckless == 'auto' or (reckless and reckless.startswith('shared-')):\n        repo.call_annex(['untrust', 'here'])\n\n    elif reckless == 'ephemeral':\n        # with ephemeral we declare 'here' as 'dead' right away, whenever\n        # we symlink the remote's annex, since availability from 'here' should\n        # not be propagated for an ephemeral clone when we publish back to\n        # the remote.\n        # This will cause stuff like this for a locally present annexed file:\n        # % git annex whereis d1\n        # whereis d1 (0 copies) failed\n        # BUT this works:\n        # % git annex find . --not --in here\n        # % git annex find . --in here\n        # d1\n\n        # we don't want annex copy-to <remote>\n        ds.config.set(\n            f'remote.{remote}.annex-ignore', 'true',\n            where='local')\n\n        ds.repo.set_remote_dead('here')\n\n        if check_symlink_capability(ds.repo.dot_git / 'dl_link_test',\n                                    ds.repo.dot_git / 'dl_target_test'):\n            # symlink the annex to avoid needless copies in an ephemeral clone\n            annex_dir = ds.repo.dot_git / 'annex'\n            origin_annex_url = ds.config.get(f\"remote.{remote}.url\", None)\n            origin_git_path = None\n            if origin_annex_url:\n                try:\n                    # Deal with file:// scheme URLs as well as plain paths.\n                    # If origin isn't local, we have nothing to do.\n                    origin_git_path = Path(RI(origin_annex_url).localpath)\n\n                    # we are local; check for a bare repo first to not mess w/\n                    # the path\n                    if GitRepo(origin_git_path, create=False).bare:\n                        # origin is a bare repo -> use path as is\n                        pass\n                    elif origin_git_path.name != '.git':\n                        origin_git_path /= '.git'\n                except ValueError:\n                    # Note, that accessing localpath on a non-local RI throws\n                    # ValueError rather than resulting in an AttributeError.\n                    # TODO: Warning level okay or is info level sufficient?\n                    # Note, that setting annex-dead is independent of\n                    # symlinking .git/annex. It might still make sense to\n                    # have an ephemeral clone that doesn't propagate its avail.\n                    # info. Therefore don't fail altogether.\n                    lgr.warning(\"reckless=ephemeral mode: %s doesn't seem \"\n                                \"local: %s\\nno symlinks being used\",\n                                remote, origin_annex_url)\n            if origin_git_path:\n                # TODO make sure that we do not delete any unique data\n                rmtree(str(annex_dir)) \\\n                    if not annex_dir.is_symlink() else annex_dir.unlink()\n                annex_dir.symlink_to(origin_git_path / 'annex',\n                                     target_is_directory=True)\n        else:\n            # TODO: What level? + note, that annex-dead is independent\n            lgr.warning(\"reckless=ephemeral mode: Unable to create symlinks on \"\n                        \"this file system.\")\n\n    srs = {True: [], False: []}  # special remotes by \"autoenable\" key\n    remote_uuids = None  # might be necessary to discover known UUIDs\n\n    repo_config = repo.config\n    # Note: The purpose of this function is to inform the user. So if something\n    # looks misconfigured, we'll warn and move on to the next item.\n    for uuid, config in repo.get_special_remotes().items():\n        sr_name = config.get('name', None)\n        if sr_name is None:\n            lgr.warning(\n                'Ignoring special remote %s because it does not have a name. '\n                'Known information: %s',\n                uuid, config)\n            continue\n        sr_autoenable = config.get('autoenable', False)\n        try:\n            sr_autoenable = ensure_bool(sr_autoenable)\n        except ValueError:\n            lgr.warning(\n                'Failed to process \"autoenable\" value %r for sibling %s in '\n                'dataset %s as bool.'\n                'You might need to enable it later manually and/or fix it up to'\n                ' avoid this message in the future.',\n                sr_autoenable, sr_name, ds.path)\n            continue\n\n        # If it looks like a type=git special remote, make sure we have up to\n        # date information. See gh-2897.\n        if sr_autoenable and repo_config.get(\"remote.{}.fetch\".format(sr_name)):\n            try:\n                repo.fetch(remote=sr_name)\n            except CommandError as exc:\n                lgr.warning(\"Failed to fetch type=git special remote %s: %s\",\n                            sr_name, exc_str(exc))\n\n        # determine whether there is a registered remote with matching UUID\n        if uuid:\n            if remote_uuids is None:\n                remote_uuids = {\n                    # Check annex-config-uuid first. For sameas annex remotes,\n                    # this will point to the UUID for the configuration (i.e.\n                    # the key returned by get_special_remotes) rather than the\n                    # shared UUID.\n                    (repo_config.get('remote.%s.annex-config-uuid' % r) or\n                     repo_config.get('remote.%s.annex-uuid' % r))\n                    for r in repo.get_remotes()\n                }\n            if uuid not in remote_uuids:\n                srs[sr_autoenable].append(sr_name)\n\n    if srs[True]:\n        lgr.debug(\n            \"configuration for %s %s added because of autoenable,\"\n            \" but no UUIDs for them yet known for dataset %s\",\n            # since we are only at debug level, we could call things their\n            # proper names\n            single_or_plural(\"special remote\",\n                             \"special remotes\", len(srs[True]), True),\n            \", \".join(srs[True]),\n            ds.path\n        )\n\n    if srs[False]:\n        # if has no auto-enable special remotes\n        lgr.info(\n            'access to %s %s not auto-enabled, enable with:\\n'\n            '\\t\\tdatalad siblings -d \"%s\" enable -s %s',\n            # but since humans might read it, we better confuse them with our\n            # own terms!\n            single_or_plural(\"dataset sibling\",\n                             \"dataset siblings\", len(srs[False]), True),\n            \", \".join(srs[False]),\n            ds.path,\n            srs[False][0] if len(srs[False]) == 1 else \"SIBLING\",\n        )\n\n    # we have just cloned the repo, so it has a remote `remote`, configure any\n    # reachable origin of origins\n    yield from configure_origins(ds, ds, remote=remote)\n\n\n_handle_possible_annex_dataset = postclonecfg_annexdataset\n\n\ndef configure_origins(cfgds, probeds, label=None, remote=\"origin\"):\n    \"\"\"Configure any discoverable local dataset sibling as a remote\n\n    Parameters\n    ----------\n    cfgds : Dataset\n      Dataset to receive the remote configurations\n    probeds : Dataset\n      Dataset to start looking for `remote` remotes. May be identical with\n      `cfgds`.\n    label : int, optional\n      Each discovered remote will be configured as a remote under the name\n      '<remote>-<label>'. If no label is given, '2' will be used by default,\n      given that there is typically a remote named `remote` already.\n    remote : str, optional\n      Name of the default remote on clone.\n    \"\"\"\n    if label is None:\n        label = 1\n    # let's look at the URL for that remote and see if it is a local\n    # dataset\n    origin_url = probeds.config.get(f'remote.{remote}.url')\n    if not origin_url:\n        # no remote with default name, nothing to do\n        return\n    if not cfgds.config.obtain(\n            'datalad.install.inherit-local-origin',\n            default=True):\n        # no inheritance wanted\n        return\n    if not isinstance(RI(origin_url), PathRI):\n        # not local path\n        return\n\n    # no need to reconfigure original/direct remote again\n    if cfgds != probeds:\n        # prevent duplicates\n        known_remote_urls = set(\n            cfgds.config.get(r + '.url', None)\n            for r in cfgds.config.sections()\n            if r.startswith('remote.')\n        )\n        if origin_url not in known_remote_urls:\n            yield from cfgds.siblings(\n                'configure',\n                # no chance for conflict, can only be the second configured\n                # remote\n                name='{}-{}'.format(remote, label),\n                url=origin_url,\n                # fetch to get all annex info\n                fetch=True,\n                result_renderer='disabled',\n                on_failure='ignore',\n            )\n    # and dive deeper\n    # given the clone source is a local dataset, we can have a\n    # cheap look at it, and configure its own `remote` as a remote\n    # (if there is any), and benefit from additional annex availability\n    yield from configure_origins(\n        cfgds,\n        Dataset(probeds.pathobj / origin_url),\n        label=label + 1,\n        remote=remote)\n\n\ndef _get_tracking_source(ds):\n    \"\"\"Returns name and url of a potential configured source\n    tracking remote\"\"\"\n    vcs = ds.repo\n    # if we have a remote, let's check the location of that remote\n    # for the presence of the desired submodule\n\n    remote_name, tracking_branch = vcs.get_tracking_branch()\n    if not remote_name and isinstance(vcs, AnnexRepo):\n        # maybe cloned from a source repo that was in adjusted mode\n        # https://github.com/datalad/datalad/issues/3969\n        remote_name, tracking_branch = vcs.get_tracking_branch(\n            corresponding=False)\n    # TODO: better default `None`? Check where we might rely on '':\n    remote_url = ''\n    if remote_name:\n        remote_url = vcs.get_remote_url(remote_name, push=False)\n\n    return remote_name, remote_url\n\n\ndef _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI, SSHRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path\n\n\ndef decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n\n    # Git never gets to see these URLs, so let's manually apply any\n    # rewrite configuration Git might know about.\n    # Note: We need to rewrite before parsing, otherwise parsing might go wrong.\n    # This is particularly true for insteadOf labels replacing even the URL\n    # scheme.\n    spec = cfg.rewrite_url(spec)\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if re.match(uuid_regex, dsid):\n            trace = '{}/{}'.format(dsid[:3], dsid[3:])\n            default_destpath = dsid\n        elif dsid.startswith('~'):\n            trace = 'alias/{}'.format(dsid[1:])\n            default_destpath = dsid[1:]\n        else:\n            raise ValueError(\n                'RIA URI not recognized, no valid dataset ID or other supported '\n                'scheme: {}'.format(spec))\n        # now we cancel the fragment in the original URL, but keep everything else\n        # in order to be able to support the various combinations of ports, paths,\n        # and everything else\n        source_ri.fragment = ''\n        # strip the custom protocol and go with standard one\n        source_ri.scheme = source_ri.scheme[4:]\n        # take any existing path, and add trace to dataset within the store\n        source_ri.path = '{urlpath}{urldelim}{trace}'.format(\n            urlpath=source_ri.path if source_ri.path else '',\n            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',\n            trace=trace,\n        )\n        props.update(\n            type='ria',\n            giturl=str(source_ri),\n            version=version,\n            default_destpath=default_destpath,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        props['type'] = 'giturl'\n        # use original input verbatim\n        props['giturl'] = spec\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props\n",
            "header": {
                "file": {
                    "path": "datalad/core/distributed/clone.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "6f1ff6778",
                "new_commit_hash": "a603a8681",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/core/distributed/clone.py"
            },
            "plus_file": {
                "path": "b/datalad/core/distributed/clone.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 47,
                            "length": 13
                        },
                        "new_range": {
                            "start": 47,
                            "length": 14
                        },
                        "section": "from datalad.support.exceptions import DownloadError"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "from datalad.support.param import Parameter",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.support.strings import get_replacement_dict",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.support.network import (",
                                "type": "context"
                            },
                            {
                                "content": "    get_local_file_url,",
                                "type": "deleted"
                            },
                            {
                                "content": "    download_url,",
                                "type": "deleted"
                            },
                            {
                                "content": "    is_url,",
                                "type": "deleted"
                            },
                            {
                                "content": "    URL,",
                                "type": "deleted"
                            },
                            {
                                "content": "    RI,",
                                "type": "deleted"
                            },
                            {
                                "content": "    DataLadRI,",
                                "type": "context"
                            },
                            {
                                "content": "    PathRI,",
                                "type": "context"
                            },
                            {
                                "content": "    RI,",
                                "type": "added"
                            },
                            {
                                "content": "    SSHRI,",
                                "type": "added"
                            },
                            {
                                "content": "    URL,",
                                "type": "added"
                            },
                            {
                                "content": "    download_url,",
                                "type": "added"
                            },
                            {
                                "content": "    get_local_file_url,",
                                "type": "added"
                            },
                            {
                                "content": "    is_url,",
                                "type": "added"
                            },
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.dochelpers import (",
                                "type": "context"
                            },
                            {
                                "content": "    exc_str,",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/core/distributed/clone.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.support.network import (\n    DataLadRI,\n    PathRI,\n    RI,\n    SSHRI,\n    URL,\n    download_url,\n    get_local_file_url,\n    is_url,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 49,
                            "end_lineno": 58,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/core/distributed/clone.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.support.network import (\n    get_local_file_url,\n    download_url,\n    is_url,\n    URL,\n    RI,\n    DataLadRI,\n    PathRI,\n)",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 49,
                            "end_lineno": 57,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 1343,
                            "length": 7
                        },
                        "new_range": {
                            "start": 1344,
                            "length": 7
                        },
                        "section": "def _get_installationpath_from_url(url):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    from a URL, analog to what `git clone` does.",
                                "type": "context"
                            },
                            {
                                "content": "    \"\"\"",
                                "type": "context"
                            },
                            {
                                "content": "    ri = RI(url)",
                                "type": "context"
                            },
                            {
                                "content": "    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL",
                                "type": "deleted"
                            },
                            {
                                "content": "    if isinstance(ri, (URL, DataLadRI, SSHRI)):  # decode only if URL",
                                "type": "added"
                            },
                            {
                                "content": "        path = ri.path.rstrip('/')",
                                "type": "context"
                            },
                            {
                                "content": "        path = urlunquote(path) if path else ri.hostname",
                                "type": "context"
                            },
                            {
                                "content": "        if '/' in path:",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/core/distributed/clone.py",
                            "type": "function",
                            "name": "_get_installationpath_from_url",
                            "content": "def _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 1339,
                            "end_lineno": 1359,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test clone action\n\n\"\"\"\n\nimport logging\nimport os.path as op\nimport stat\n\nfrom unittest.mock import patch\n\nfrom datalad.config import ConfigManager\nfrom datalad import consts\nfrom datalad.api import (\n    clone,\n    create,\n    remove,\n)\nfrom datalad.utils import (\n    chpwd,\n    get_home_envvars,\n    Path,\n    on_windows,\n    rmtree\n)\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.cmd import (\n    GitWitlessRunner,\n    WitlessRunner as Runner,\n)\nfrom datalad.tests.utils import (\n    assert_false,\n    assert_in,\n    assert_in_results,\n    assert_not_is_instance,\n    assert_message,\n    assert_not_in,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_result_values_equal,\n    assert_status,\n    create_tree,\n    DEFAULT_BRANCH,\n    DEFAULT_REMOTE,\n    eq_,\n    get_datasets_topdir,\n    integration,\n    known_failure,\n    known_failure_githubci_win,\n    known_failure_windows,\n    neq_,\n    nok_,\n    ok_,\n    ok_file_has_content,\n    has_symlink_capability,\n    ok_startswith,\n    patch_config,\n    set_date,\n    serve_path_via_http,\n    skip_if_adjusted_branch,\n    skip_if_no_network,\n    skip_if_on_windows,\n    skip_ssh,\n    slow,\n    swallow_logs,\n    use_cassette,\n    with_sameas_remote,\n    with_tempfile,\n    with_testrepos,\n    with_tree,\n    SkipTest,\n)\nfrom datalad.support.network import get_local_file_url\nfrom datalad.core.distributed.clone import (\n    _get_installationpath_from_url,\n    decode_source_spec,\n)\nfrom datalad.distribution.dataset import Dataset\n\n# this is the dataset ID of our test dataset in the main datalad RIA store\ndatalad_store_testds_id = '76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561'\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_args(path, otherpath, alienpath):\n    # source == path\n    assert_raises(ValueError, clone, 'Zoidberg', path='Zoidberg')\n    assert_raises(ValueError, clone, 'ssh://mars/Zoidberg', path='ssh://mars/Zoidberg')\n\n    # \"invalid URL\" is a valid filepath... and since no clone to remote\n    # is possible - we can just assume that it is the (legit) file path\n    # which is provided, not a URL.  So both below should fail as any\n    # other clone from a non-existing source and not for the reason of\n    # \"invalid something\".  Behavior is similar to how Git performs - can\n    # clone into a URL-like path.\n\n    # install to an \"invalid URL\" path\n    res = clone('Zoidberg', path='ssh://mars:Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # install to a \"remote location\" path\n    res = clone('Zoidberg', path='ssh://mars/Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # make fake dataset\n    ds = create(path)\n    assert_raises(IncompleteResultsError, ds.clone, '/higherup.', 'Zoidberg')\n    # make real dataset, try to install outside\n    ds_target = create(Path(otherpath) / 'target')\n    assert_raises(ValueError, ds_target.clone, ds.path, path=ds.path)\n    assert_status('error', ds_target.clone(ds.path, path=alienpath, on_failure='ignore'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_crcns(tdir, ds_path):\n    with chpwd(tdir):\n        res = clone('///', path=\"all-nonrecursive\", on_failure='ignore',\n                    result_xfm=None, return_type='list')\n        assert_status('ok', res)\n\n    # again, but into existing dataset:\n    ds = create(ds_path)\n    crcns = ds.clone(\"///crcns\", result_xfm='datasets', return_type='item-or-list')\n    ok_(crcns.is_installed())\n    eq_(crcns.pathobj, ds.pathobj / \"crcns\")\n    assert_in(crcns.path, ds.subdatasets(result_xfm='paths'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tree(tree={'sub': {}})\ndef test_clone_datasets_root(tdir):\n    tdir = Path(tdir)\n    with chpwd(tdir):\n        ds = clone(\"///\")\n        ok_(ds.is_installed())\n        eq_(ds.pathobj, tdir / get_datasets_topdir())\n\n        # do it a second time:\n        res = clone(\"///\", on_failure='ignore', result_xfm=None, return_type='list')\n        assert_message(\n            \"dataset %s was already cloned from '%s'\",\n            res)\n        assert_status('notneeded', res)\n\n        # and a third time into an existing something, that is not a dataset:\n        (tdir / 'sub' / 'a_file.txt').write_text(\"something\")\n\n        res = clone('///', path=\"sub\", on_failure='ignore')\n        assert_message(\n            'target path already exists and not empty, refuse to clone into target path',\n            res)\n        assert_status('error', res)\n\n\n@with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n@with_tempfile(mkdir=True)\ndef test_clone_simple_local(src, path):\n    origin = Dataset(path)\n\n    # now install it somewhere else\n    ds = clone(src, path, description='mydummy',\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    ok_(ds.is_installed())\n    if not isinstance(origin.repo, AnnexRepo):\n        # this means it is a GitRepo\n        ok_(isinstance(origin.repo, GitRepo))\n        # stays plain Git repo\n        ok_(isinstance(ds.repo, GitRepo))\n        ok_(not isinstance(ds.repo, AnnexRepo))\n        ok_(GitRepo.is_valid_repo(ds.path))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt'})\n        assert_repo_status(path, annex=False)\n    else:\n        # must be an annex\n        ok_(isinstance(ds.repo, AnnexRepo))\n        ok_(AnnexRepo.is_valid_repo(ds.path, allow_noninitialized=False))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt', 'test-annex.dat'})\n        assert_repo_status(path, annex=True)\n        # no content was installed:\n        ok_(not ds.repo.file_has_content('test-annex.dat'))\n        uuid_before = ds.repo.uuid\n        ok_(uuid_before)  # make sure we actually have an uuid\n        eq_(ds.repo.get_description(), 'mydummy')\n\n    # installing it again, shouldn't matter:\n    res = clone(src, path, result_xfm=None, return_type='list')\n    assert_result_values_equal(res, 'source_url', [src])\n    assert_status('notneeded', res)\n    assert_message(\"dataset %s was already cloned from '%s'\", res)\n    ok_(ds.is_installed())\n    if isinstance(origin.repo, AnnexRepo):\n        eq_(uuid_before, ds.repo.uuid)\n\n\n\n# AssertionError: unexpected content of state \"deleted\": [WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/datalad_temp_gzegy3hf/testrepo--basic--r1/test-annex.dat')] != []\n@known_failure_githubci_win\n@with_testrepos(flavors=['local-url', 'network', 'local'])\n@with_tempfile\ndef test_clone_dataset_from_just_source(url, path):\n    with chpwd(path, mkdir=True):\n        ds = clone(url, result_xfm='datasets', return_type='item-or-list')\n\n    ok_startswith(ds.path, path)\n    ok_(ds.is_installed())\n    ok_(GitRepo.is_valid_repo(ds.path))\n    assert_repo_status(ds.path, annex=None)\n    assert_in('INFO.txt', ds.repo.get_indexed_files())\n\n\n# test fails randomly, likely a bug in one of the employed test helpers\n# https://github.com/datalad/datalad/pull/3966#issuecomment-571267932\n@known_failure\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_clone_dataladri(src, topurl, path):\n    # make plain git repo\n    ds_path = Path(src) / 'ds'\n    gr = GitRepo(ds_path, create=True)\n    gr.add('test.txt')\n    gr.commit('demo')\n    Runner(cwd=gr.path).run(['git', 'update-server-info'])\n    # now install it somewhere else\n    with patch('datalad.consts.DATASETS_TOPURL', topurl):\n        ds = clone('///ds', path, result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    assert_repo_status(path, annex=False)\n    ok_file_has_content(ds.pathobj / 'test.txt', 'some')\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_isnot_recursive(path_src, path_nr, path_r):\n    src = Dataset(path_src).create()\n    src.create('subm 1')\n    src.create('2')\n\n    ds = clone(src, path_nr, result_xfm='datasets', return_type='item-or-list')\n    ok_(ds.is_installed())\n    # check nothing is unintentionally installed\n    subdss = ds.subdatasets(recursive=True)\n    assert_result_count(subdss, len(subdss), state='absent')\n    # this also means, subdatasets to be listed as not fulfilled:\n    eq_(set(ds.subdatasets(recursive=True, fulfilled=False, result_xfm='relpaths')),\n        {'subm 1', '2'})\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_into_dataset(source_path, top_path):\n    source = Dataset(source_path).create()\n    ds = create(top_path)\n    assert_repo_status(ds.path)\n    # Note, we test against the produced history in DEFAULT_BRANCH, not what it\n    # turns into in an adjusted branch!\n    hexsha_before = ds.repo.get_hexsha(DEFAULT_BRANCH)\n    subds = ds.clone(source, \"sub\",\n                     result_xfm='datasets', return_type='item-or-list')\n    ok_((subds.pathobj / '.git').is_dir())\n    ok_(subds.is_installed())\n    assert_in('sub', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # sub is clean:\n    assert_repo_status(subds.path, annex=None)\n    # top is clean:\n    assert_repo_status(ds.path, annex=None)\n    # source is recorded in .gitmodules:\n    sds = ds.subdatasets(\"sub\")\n    assert_result_count(sds, 1, action='subdataset')\n    eq_(sds[0]['gitmodule_datalad-url'], source.path)\n    # Clone produced one commit including the addition to .gitmodule:\n    commits = list(ds.repo.get_branch_commits_(\n        branch=DEFAULT_BRANCH,\n        stop=hexsha_before\n    ))\n    assert_not_in(hexsha_before, commits)\n    eq_(len(commits), 1)\n\n    # but we could also save while installing and there should be no side-effect\n    # of saving any other changes if we state to not auto-save changes\n    # Create a dummy change\n    create_tree(ds.path, {'dummy.txt': 'buga'})\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n    subds_ = ds.clone(source, \"sub2\",\n                      result_xfm='datasets', return_type='item-or-list')\n    eq_(subds_.pathobj, ds.pathobj / \"sub2\")  # for paranoid yoh ;)\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_notclone_known_subdataset(src_path, path):\n    src = Dataset(src_path).create()\n    sub = src.create('subm 1')\n    sub_id = sub.id\n    # get the superdataset:\n    ds = clone(src, path,\n               result_xfm='datasets', return_type='item-or-list')\n\n    # subdataset not installed:\n    subds = Dataset(ds.pathobj / 'subm 1')\n    assert_false(subds.is_installed())\n    assert_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # clone is not meaningful\n    res = ds.clone('subm 1', on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n    # get does the job\n    res = ds.get(path='subm 1', get_data=False)\n    assert_status('ok', res)\n    ok_(subds.is_installed())\n    ok_(AnnexRepo.is_valid_repo(subds.path, allow_noninitialized=False))\n    # Verify that it is the correct submodule installed and not\n    # new repository initiated\n    eq_(subds.id, sub_id)\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n\n\n@with_tempfile(mkdir=True)\ndef test_failed_clone(dspath):\n    ds = create(dspath)\n    res = ds.clone(\"http://nonexistingreallyanything.datalad.org/bla\", \"sub\",\n                   on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n\n\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@with_tempfile\ndef test_clone_missing_commit(source, clone):\n\n    from datalad.core.distributed.clone import clone_dataset\n\n    source = Path(source)\n    clone = Path(clone)\n\n    # Commit SHA from another repository - should never be recreated in a fresh\n    # dataset:\n    commit_sha = \"c29691b37b05b78ffa76e5fdf0044e9df673e8f1\"\n\n    origin = Dataset(source).create(force=True)\n    origin.save()\n\n    # clone origin but request commit_sha to be checked out:\n\n    results = [x for x in\n               clone_dataset(srcs=[source], destds=Dataset(clone),\n                             checkout_gitsha=commit_sha)\n               ]\n    # expected error result:\n    assert_result_count(results, 1)\n    assert_in_results(results, status='error', action='install',\n                      path=str(clone), type='dataset')\n    assert_in(\"Target commit c29691b3 does not exist in the clone\",\n              results[0]['message'])\n    # failed attempt was removed:\n    assert_false(clone.exists())\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef check_reckless(annex, src_path, top_path, sharedpath):\n    # super with or without annex\n    src = Dataset(src_path).create(annex=annex)\n    # sub always with annex\n    srcsub = src.create('sub')\n\n    # and for the actual test\n    ds = clone(src.path, top_path, reckless=True,\n               result_xfm='datasets', return_type='item-or-list')\n\n    is_crippled = srcsub.repo.is_managed_branch()\n\n    if annex and not is_crippled:\n        eq_(ds.config.get('annex.hardlink', None), 'true')\n\n    # actual value is 'auto', because True is a legacy value and we map it\n    eq_(ds.config.get('datalad.clone.reckless', None), 'auto')\n    if annex:\n        eq_(ds.repo.repo_info()['untrusted repositories'][0]['here'], True)\n    # now, if we clone another repo into this one, it will inherit the setting\n    # without having to provide it explicitly\n    newsub = ds.clone(srcsub, 'newsub', result_xfm='datasets', return_type='item-or-list')\n    # and `get` the original subdataset\n    origsub = ds.get('sub', result_xfm='datasets', return_type='item-or-list')\n    for sds in (newsub, origsub):\n        eq_(sds.config.get('datalad.clone.reckless', None), 'auto')\n        if not is_crippled:\n            eq_(sds.config.get('annex.hardlink', None), 'true')\n\n    if is_crippled:\n        raise SkipTest(\"Remainder of test needs proper filesystem permissions\")\n\n    if annex:\n        if ds.repo.git_annex_version < \"8.20200908\":\n            # TODO: Drop when GIT_ANNEX_MIN_VERSION is at least 8.20200908.\n\n            # the standard setup keeps the annex locks accessible to the user only\n            nok_((ds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n                 & stat.S_IWGRP)\n        else:\n            # umask might be such (e.g. 002) that group write permissions are inherited, so\n            # for the next test we should check if that is the case on some sample file\n            dltmp_path = ds.pathobj / '.git' / \"dltmp\"\n            dltmp_path.write_text('')\n            default_grp_write_perms = dltmp_path.stat().st_mode & stat.S_IWGRP\n            dltmp_path.unlink()\n            # the standard setup keeps the annex locks following umask inheritance\n            eq_((ds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n                 & stat.S_IWGRP, default_grp_write_perms)\n\n        # but we can set it up for group-shared access too\n        sharedds = clone(\n            src, sharedpath,\n            reckless='shared-group',\n            result_xfm='datasets',\n            return_type='item-or-list')\n        ok_((sharedds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n            & stat.S_IWGRP)\n\n\ndef test_reckless():\n    yield check_reckless, True\n    yield check_reckless, False\n\n@with_tempfile\n@with_tempfile\ndef test_install_source_relpath(src, dest):\n    src = Path(src)\n    create(src)\n    src_ = src.name\n    with chpwd(src.parent):\n        clone(src_, dest)\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_isnt_a_smartass(origin_path, path):\n    origin = create(origin_path)\n    cloned = clone(origin, path,\n                   result_xfm='datasets', return_type='item-or-list')\n    with chpwd(path):\n        # no were are inside a dataset clone, and we make another one\n        # we do not want automatic subdatasetification without given a dataset\n        # explicitly\n        clonedsub = clone(origin, 'testsub',\n                          result_xfm='datasets', return_type='item-or-list')\n    # correct destination\n    assert clonedsub.path.startswith(path)\n    # no subdataset relation\n    eq_(cloned.subdatasets(), [])\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_report_permission_issue(tdir):\n    pdir = Path(tdir) / 'protected'\n    pdir.mkdir()\n    # make it read-only\n    pdir.chmod(0o555)\n    with chpwd(pdir):\n        # first check the premise of the test. If we can write (strangely\n        # mounted/crippled file system, subsequent assumptions are violated\n        # and we can stop\n        probe = Path('probe')\n        try:\n            probe.write_text('should not work')\n            raise SkipTest\n        except PermissionError:\n            # we are indeed in a read-only situation\n            pass\n        res = clone('///', result_xfm=None, return_type='list', on_failure='ignore')\n        assert_status('error', res)\n        assert_result_count(\n            res, 1, status='error',\n            message=\"could not create work tree dir '%s/%s': Permission denied\"\n                    % (pdir, get_datasets_topdir())\n        )\n\n\n@skip_if_no_network\n@with_tempfile\ndef test_autoenabled_remote_msg(path):\n    # Verify that no message about a remote not been enabled is displayed\n    # whenever the remote we clone is the  type=git special remote, so the name\n    # of the remote might not match\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone('///repronim/containers', path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_not_in(\"not auto-enabled\", cml.out)\n\n\n@with_sameas_remote(autoenabled=True)\n@with_tempfile(mkdir=True)\ndef test_clone_autoenable_msg_handles_sameas(repo, clone_path):\n    ds = Dataset(repo.path)\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone(ds, clone_path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_in(\"r_dir\", cml.out)\n        assert_in(\"not auto-enabled\", cml.out)\n        # The rsyncurl remote was enabled.\n        assert_not_in(\"r_rsync\", cml.out)\n    ds_cloned = Dataset(clone_path)\n    remotes = ds_cloned.repo.get_remotes()\n    assert_in(\"r_rsync\", remotes)\n    assert_not_in(\"r_dir\", remotes)\n\n\ndef test_installationpath_from_url():\n    cases = (\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n    ) + (\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ) if on_windows else (\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    )\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')\n\n\n# https://github.com/datalad/datalad/issues/3958\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_expanduser(srcpath, destpath):\n    src = Dataset(Path(srcpath) / 'src').create()\n    dest = Dataset(Path(destpath) / 'dest').create()\n\n    # We switch away from home set up in datalad.setup_package(), so make sure\n    # we have a valid identity.\n    with open(op.join(srcpath, \".gitconfig\"), \"w\") as fh:\n        fh.write(\"[user]\\n\"\n                 \"name = DataLad oooooTester\\n\"\n                 \"email = test@example.com\\n\")\n\n    with chpwd(destpath), patch.dict('os.environ', get_home_envvars(srcpath)):\n        res = clone(op.join('~', 'src'), 'dest', result_xfm=None, return_type='list',\n                    on_failure='ignore')\n        assert_result_count(res, 1)\n        assert_result_count(\n            res, 1, action='install', status='error', path=dest.path,\n            message='target path already exists and not empty, refuse to '\n            'clone into target path')\n        # wipe out destination, and try again\n        assert_status('ok', remove(dataset=dest, check=False))\n        # now it should do it, and clone the right one\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n        eq_(src.id, cloneds.id)\n        # and it shouldn't fail when doing it again, because it detects\n        # the re-clone\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n\n\n@with_tempfile(mkdir=True)\ndef test_cfg_originorigin(path):\n    path = Path(path)\n    origin = Dataset(path / 'origin').create()\n    (origin.pathobj / 'file1.txt').write_text('content')\n    origin.save()\n    clone_lev1 = clone(origin, path / 'clone_lev1')\n    clone_lev2 = clone(clone_lev1, path / 'clone_lev2')\n    # the goal is to be able to get file content from origin without\n    # the need to configure it manually\n    assert_result_count(\n        clone_lev2.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_lev2.pathobj / 'file1.txt'),\n    )\n    eq_((clone_lev2.pathobj / 'file1.txt').read_text(), 'content')\n    eq_(\n        Path(clone_lev2.siblings(\n            'query',\n            name=DEFAULT_REMOTE + '-2',\n            return_type='item-or-list')['url']),\n        origin.pathobj\n    )\n\n    # Clone another level, this time with a relative path. Drop content from\n    # lev2 so that origin is the only place that the file is available from.\n    clone_lev2.drop(\"file1.txt\")\n    with chpwd(path), swallow_logs(new_level=logging.DEBUG) as cml:\n        clone_lev3 = clone('clone_lev2', 'clone_lev3')\n        # we called git-annex-init; see gh-4367:\n        cml.assert_logged(msg=r\"[^[]*Run \\[('git',.*'annex'|'git-annex'), 'init'\",\n                          match=False,\n                          level='DEBUG')\n    assert_result_count(\n        clone_lev3.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_lev3.pathobj / 'file1.txt'))\n\n\n# test fix for gh-2601/gh-3538\n@known_failure\n@with_tempfile()\ndef test_relative_submodule_url(path):\n    Dataset(op.join(path, 'origin')).create()\n    ds = Dataset(op.join(path, 'ds')).create()\n    with chpwd(ds.path):\n        ds_cloned = ds.clone(\n            source=op.join(op.pardir, 'origin'),\n            path='sources')\n\n    # Check that a simple fetch call does not fail.\n    ds_cloned.repo.fetch()\n\n    subinfo = ds.subdatasets(return_type='item-or-list')\n    eq_(subinfo['gitmodule_url'],\n        # must be a relative URL, not platform-specific relpath!\n        '../../origin')\n\n\n@with_tree(tree={\"subdir\": {}})\n@with_tempfile(mkdir=True)\ndef test_local_url_with_fetch(path, path_other):\n    path = Path(path)\n    path_other = Path(path_other)\n    Dataset(path / \"source\").create()\n\n    for where, source, path in [\n            (path, \"source\", \"a\"),\n            (path / \"subdir\", op.join(op.pardir, \"source\"), \"a\"),\n            (path, \"source\", path_other / \"a\")]:\n        with chpwd(where):\n            ds_cloned = clone(source=source, path=path)\n            # Perform a fetch to check that the URL points to a valid location.\n            ds_cloned.repo.fetch()\n\n\ndef test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}/{}/{}'.format(\n                    proto,\n                    loc,\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')\n\n    # literal dataset name/location\n    eq_(decode_source_spec('ria+http://example.com#~rootds'),\n        {'source': 'ria+http://example.com#~rootds',\n         'version': None, 'type': 'ria',\n         'giturl': 'http://example.com/alias/rootds',\n         'default_destpath': 'rootds'})\n    # version etc still works\n    eq_(decode_source_spec('ria+http://example.com#~rootds@specialbranch'),\n        {'source': 'ria+http://example.com#~rootds@specialbranch',\n         'version': 'specialbranch', 'type': 'ria',\n         'giturl': 'http://example.com/alias/rootds',\n         'default_destpath': 'rootds'})\n\n\ndef _move2store(storepath, d):\n    # make a bare clone of it into a local that matches the organization\n    # of a ria dataset store\n    store_loc = str(storepath / d.id[:3] / d.id[3:])\n    d.repo.call_git(['clone', '--bare', d.path, store_loc])\n    d.siblings('configure', name='store', url=str(store_loc),\n               result_renderer='disabled')\n    Runner(cwd=store_loc).run(['git', 'update-server-info'])\n\n\n@slow  # 12sec on Yarik's laptop\n@with_tree(tree={\n    'ds': {\n        'test.txt': 'some',\n        'subdir': {\n            'subds': {'testsub.txt': 'somemore'},\n        },\n    },\n})\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_ria_http(lcl, storepath, url):\n    # create a local dataset with a subdataset\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    subds = Dataset(lcl / 'ds' / 'subdir' / 'subds').create(force=True)\n    subds.save()\n    ds = Dataset(lcl / 'ds').create(force=True)\n    ds.save(version_tag='original')\n    assert_repo_status(ds.path)\n    for d in (ds, subds):\n        _move2store(storepath, d)\n    # location of superds in store\n    storeds_loc = str(storepath / ds.id[:3] / ds.id[3:])\n    # now we should be able to clone from a ria+http url\n    # the super\n    riaclone = clone(\n        'ria+{}#{}'.format(url, ds.id),\n        lcl / 'clone',\n    )\n\n    # due to default configuration, clone() should automatically look for the\n    # subdataset in the store, too -- if not the following would fail, because\n    # we never configured a proper submodule URL\n    riaclonesub = riaclone.get(\n        op.join('subdir', 'subds'), get_data=False,\n        result_xfm='datasets', return_type='item-or-list')\n\n    # both datasets came from the store and must be set up in an identical\n    # fashion\n    for origds, cloneds in ((ds, riaclone), (subds, riaclonesub)):\n        eq_(origds.id, cloneds.id)\n        if not ds.repo.is_managed_branch():\n            # test logic cannot handle adjusted branches\n            eq_(origds.repo.get_hexsha(), cloneds.repo.get_hexsha())\n        ok_(cloneds.config.get(f'remote.{DEFAULT_REMOTE}.url').startswith(url))\n        eq_(cloneds.config.get(f'remote.{DEFAULT_REMOTE}.annex-ignore'), 'true')\n        eq_(cloneds.config.get('datalad.get.subdataset-source-candidate-200origin'),\n            'ria+%s#{id}' % url)\n\n    # now advance the source dataset\n    (ds.pathobj / 'newfile.txt').write_text('new')\n    ds.save()\n    ds.publish(to='store')\n    Runner(cwd=storeds_loc).run(['git', 'update-server-info'])\n    # re-clone as before\n    riaclone2 = clone(\n        'ria+{}#{}'.format(url, ds.id),\n        lcl / 'clone2',\n    )\n    # and now clone a specific version, here given be the tag name\n    riaclone_orig = clone(\n        'ria+{}#{}@{}'.format(url, ds.id, 'original'),\n        lcl / 'clone_orig',\n    )\n    if not ds.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        # we got the precise version we wanted\n        eq_(riaclone.repo.get_hexsha(), riaclone_orig.repo.get_hexsha())\n        # and not the latest\n        eq_(riaclone2.repo.get_hexsha(), ds.repo.get_hexsha())\n        neq_(riaclone2.repo.get_hexsha(), riaclone_orig.repo.get_hexsha())\n\n    # attempt to clone a version that doesn't exist\n    with swallow_logs():\n        with assert_raises(IncompleteResultsError) as cme:\n            clone('ria+{}#{}@impossible'.format(url, ds.id),\n                  lcl / 'clone_failed')\n        assert_in(\"not found in upstream\", str(cme.exception))\n\n    # lastly test if URL rewriting is in effect\n    # on the surface we clone from an SSH source identified by some custom\n    # label, no full URL, but URL rewriting setup maps it back to the\n    # HTTP URL used above\n    with patch_config({\n            'url.ria+{}#.insteadof'.format(url): 'ria+ssh://somelabel#'}):\n        cloned_by_label = clone(\n            'ria+ssh://somelabel#{}'.format(origds.id),\n            lcl / 'cloned_by_label',\n        )\n    # so we get the same setup as above, but....\n    eq_(origds.id, cloned_by_label.id)\n    if not ds.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        eq_(origds.repo.get_hexsha(), cloned_by_label.repo.get_hexsha())\n    ok_(cloned_by_label.config.get(\n        f'remote.{DEFAULT_REMOTE}.url').startswith(url))\n    eq_(cloned_by_label.config.get(f'remote.{DEFAULT_REMOTE}.annex-ignore'),\n        'true')\n    # ... the clone candidates go with the label-based URL such that\n    # future get() requests acknowledge a (system-wide) configuration\n    # update\n    eq_(cloned_by_label.config.get('datalad.get.subdataset-source-candidate-200origin'),\n        'ria+ssh://somelabel#{id}')\n\n    if not has_symlink_capability():\n        return\n    # place a symlink in the store to serve as a dataset alias\n    (storepath / 'alias').mkdir()\n    (storepath / 'alias' / 'myname').symlink_to(storeds_loc)\n    with chpwd(lcl):\n        cloned_by_alias = clone('ria+{}#~{}'.format(url, 'myname'))\n    # still get the same data\n    eq_(cloned_by_alias.id, ds.id)\n    # more sensible default install path\n    eq_(cloned_by_alias.pathobj.name, 'myname')\n\n\n@with_tempfile\n@with_tempfile\ndef _test_ria_postclonecfg(url, dsid, clone_path, superds):\n    # Test cloning from RIA store while ORA special remote autoenabling failed\n    # due to an invalid URL from the POV of the cloner.\n    # Origin's git-config-file should contain the UUID to enable. This needs to\n    # work via HTTP, SSH and local cloning.\n\n    # Autoenabling should fail initially by git-annex-init and we would report\n    # on INFO level. Only postclone routine would deal with it.\n    with swallow_logs(new_level=logging.INFO) as cml:\n        # First, the super ds:\n        riaclone = clone('ria+{}#{}'.format(url, dsid), clone_path)\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n\n    # However, we now can retrieve content since clone should have enabled the\n    # special remote with new URL (or origin in case of HTTP).\n    res = riaclone.get('test.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(riaclone.pathobj / 'test.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n    # Second ORA remote is enabled and not reconfigured:\n    untouched_remote = riaclone.siblings(name='anotherstore-storage',\n                                         return_type='item-or-list')\n    assert_not_is_instance(untouched_remote, list)\n    untouched_url = riaclone.repo.get_special_remotes()[\n        untouched_remote['annex-uuid']]['url']\n    ok_(untouched_url.startswith(\"ria+file://\"))\n    ok_(not untouched_url.startswith(\"ria+{}\".format(url)))\n\n    # publication dependency was set for store-storage but not for\n    # anotherstore-storage:\n    eq_(riaclone.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                            get_all=True),\n        \"store-storage\")\n\n    # same thing for the sub ds (we don't need a store-url and id - get should\n    # figure those itself):\n    with swallow_logs(new_level=logging.INFO) as cml:\n        riaclonesub = riaclone.get(\n            op.join('subdir', 'subds'), get_data=False,\n            result_xfm='datasets', return_type='item-or-list')\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n    res = riaclonesub.get('testsub.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(riaclonesub.pathobj / 'testsub.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n    # publication dependency was set for store-storage but not for\n    # anotherstore-storage:\n    eq_(riaclonesub.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                               get_all=True),\n        \"store-storage\")\n\n    # finally get the plain git subdataset.\n    # Clone should figure to also clone it from a ria+ URL\n    # (subdataset-source-candidate), notice that there wasn't an autoenabled ORA\n    # remote, but shouldn't stumble upon it, since it's a plain git.\n    res = riaclone.get(op.join('subdir', 'subgit', 'testgit.txt'))\n    assert_result_count(res, 1, status='ok', type='dataset', action='install')\n    assert_result_count(res, 1, status='notneeded', type='file')\n    assert_result_count(res, 2)\n    # no ORA remote, no publication dependency:\n    riaclonesubgit = Dataset(riaclone.pathobj / 'subdir' / 'subgit')\n    eq_(riaclonesubgit.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                                  get_all=True),\n        None)\n\n    # Now, test that if cloning into a dataset, ria-URL is preserved and\n    # post-clone configuration is triggered again, when we remove the subds and\n    # retrieve it again via `get`:\n    ds = Dataset(superds).create()\n    ria_url = 'ria+{}#{}'.format(url, dsid)\n    ds.clone(ria_url, 'sub')\n    sds = ds.subdatasets('sub')\n    eq_(len(sds), 1)\n    eq_(sds[0]['gitmodule_datalad-url'], ria_url)\n    assert_repo_status(ds.path)\n    ds.uninstall('sub', check=False)\n    assert_repo_status(ds.path)\n\n    # .gitmodules still there:\n    sds = ds.subdatasets('sub')\n    eq_(len(sds), 1)\n    eq_(sds[0]['gitmodule_datalad-url'], ria_url)\n    # get it again:\n\n    # Autoenabling should fail initially by git-annex-init and we would report\n    # on INFO level. Only postclone routine would deal with it.\n    with swallow_logs(new_level=logging.INFO) as cml:\n        ds.get('sub', get_data=False)\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n\n    subds = Dataset(ds.pathobj / 'sub')\n    # special remote is fine:\n    res = subds.get('test.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(subds.pathobj / 'test.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n\n@with_tempfile\ndef _postclonetest_prepare(lcl, storepath, storepath2, link):\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n        create_ds_in_store,\n        get_layout_locations\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n    create_tree(lcl,\n                tree={\n                        'ds': {\n                            'test.txt': 'some',\n                            'subdir': {\n                                'subds': {'testsub.txt': 'somemore'},\n                                'subgit': {'testgit.txt': 'even more'}\n                            },\n                        },\n                      })\n\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    storepath2 = Path(storepath2)\n    link = Path(link)\n    link.symlink_to(storepath)\n\n    # create a local dataset with a subdataset\n    subds = Dataset(lcl / 'ds' / 'subdir' / 'subds').create(force=True)\n    subds.save()\n    # add a plain git dataset as well\n    subgit = Dataset(lcl / 'ds' / 'subdir' / 'subgit').create(force=True,\n                                                              annex=False)\n    subgit.save()\n    ds = Dataset(lcl / 'ds').create(force=True)\n    ds.save(version_tag='original')\n    assert_repo_status(ds.path)\n\n    io = LocalIO()\n\n    # Have a second store with valid ORA remote. This should not interfere with\n    # reconfiguration of the first one, when that second store is not the one we\n    # clone from. However, don't push data into it for easier get-based testing\n    # later on.\n    # Doing this first, so datasets in \"first\"/primary store know about this.\n    create_store(io, storepath2, '1')\n    url2 = \"ria+{}\".format(get_local_file_url(str(storepath2)))\n    for d in (ds, subds, subgit):\n        create_ds_in_store(io, storepath2, d.id, '2', '1')\n        d.create_sibling_ria(url2, \"anotherstore\")\n        d.push('.', to='anotherstore', data='nothing')\n        store2_loc, _, _ = get_layout_locations(1, storepath2, d.id)\n        Runner(cwd=str(store2_loc)).run(['git', 'update-server-info'])\n\n    # Now the store to clone from:\n    create_store(io, storepath, '1')\n\n    # URL to use for upload. Point is, that this should be invalid for the clone\n    # so that autoenable would fail. Therefore let it be based on a to be\n    # deleted symlink\n    upl_url = \"ria+{}\".format(get_local_file_url(str(link)))\n\n    for d in (ds, subds, subgit):\n\n        # TODO: create-sibling-ria required for config! => adapt to RF'd\n        #       creation (missed on rebase?)\n        create_ds_in_store(io, storepath, d.id, '2', '1')\n        d.create_sibling_ria(upl_url, \"store\")\n\n        if d is not subgit:\n            # Now, simulate the problem by reconfiguring the special remote to\n            # not be autoenabled.\n            # Note, however, that the actual intention is a URL, that isn't\n            # valid from the point of view of the clone (doesn't resolve, no\n            # credentials, etc.) and therefore autoenabling on git-annex-init\n            # when datalad-cloning would fail to succeed.\n            Runner(cwd=d.path).run(['git', 'annex', 'enableremote',\n                                    'store-storage',\n                                    'autoenable=false'])\n        d.push('.', to='store')\n        store_loc, _, _ = get_layout_locations(1, storepath, d.id)\n        Runner(cwd=str(store_loc)).run(['git', 'update-server-info'])\n\n    link.unlink()\n    # We should now have a store with datasets that have an autoenabled ORA\n    # remote relying on an inaccessible URL.\n    # datalad-clone is supposed to reconfigure based on the URL we cloned from.\n    # Test this feature for cloning via HTTP, SSH and FILE URLs.\n\n    return ds.id\n\n\n@known_failure_windows  # https://github.com/datalad/datalad/issues/5134\n@slow  # 14 sec on travis\ndef test_ria_postclonecfg():\n\n    if not has_symlink_capability():\n        # This is needed to create an ORA remote using an URL for upload,\n        # that is then invalidated later on (delete the symlink it's based on).\n        raise SkipTest(\"Can't create symlinks\")\n\n    from datalad.utils import make_tempfile\n\n    with make_tempfile(mkdir=True) as lcl, make_tempfile(mkdir=True) as store, \\\n            make_tempfile(mkdir=True) as store2:\n        id = _postclonetest_prepare(lcl, store, store2)\n\n        # test cloning via ria+file://\n        yield _test_ria_postclonecfg, \\\n              get_local_file_url(store, compatibility='git'), id\n\n        # Note: HTTP disabled for now. Requires proper implementation in ORA\n        #       remote. See\n        # https://github.com/datalad/datalad/pull/4203#discussion_r410284649\n\n        # # test cloning via ria+http://\n        # with HTTPPath(store) as url:\n        #     yield _test_ria_postclonecfg, url, id\n\n        # test cloning via ria+ssh://\n        yield skip_ssh(_test_ria_postclonecfg), \\\n            \"ssh://datalad-test:{}\".format(Path(store).as_posix()), id\n\n\n@known_failure_windows\n@skip_ssh\n@with_tree(tree={'somefile.txt': 'some content'})\n@with_tempfile\n@with_tempfile\ndef test_no_ria_postclonecfg(dspath, storepath, clonepath):\n\n    dspath = Path(dspath)\n    storepath = Path(storepath)\n    clonepath = Path(clonepath)\n\n    # Test that particular configuration(s) do NOT lead to a reconfiguration\n    # upon clone. (See gh-5628)\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n    ds = Dataset(dspath).create(force=True)\n    ds.save()\n    assert_repo_status(ds.path)\n\n    io = LocalIO()\n    create_store(io, storepath, '1')\n    file_url = \"ria+{}\".format(get_local_file_url(str(storepath)))\n    ssh_url = \"ria+ssh://datalad-test:{}\".format(storepath.as_posix())\n    ds.create_sibling_ria(file_url, \"teststore\",\n                          push_url=ssh_url, alias=\"testds\")\n    ds.push('.', to='teststore')\n\n    # Now clone via SSH. Should not reconfigure although `url` doesn't match the\n    # URL we cloned from. However, `push-url` does.\n    riaclone = clone('{}#{}'.format(ssh_url, ds.id), clonepath)\n\n    # ORA remote is enabled (since URL still valid) but not reconfigured:\n    untouched_remote = riaclone.siblings(name='teststore-storage',\n                                         return_type='item-or-list')\n    assert_not_is_instance(untouched_remote, list)\n    ora_cfg = riaclone.repo.get_special_remotes()[\n        untouched_remote['annex-uuid']]\n    ok_(ora_cfg['url'] == file_url)\n    ok_(ora_cfg['push-url'] == ssh_url)\n\n    # publication dependency was still set (and it's the only one that was set):\n    eq_(riaclone.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                            get_all=True),\n        \"teststore-storage\")\n\n    # we can still get the content\n    ds.get(\"somefile.txt\")\n\n\n# fatal: Could not read from remote repository.\n@known_failure_githubci_win  # in datalad/git-annex as e.g. of 20201218\n@with_tempfile(mkdir=True)\n@with_tempfile\n@with_tempfile\ndef test_ria_postclone_noannex(dspath, storepath, clonepath):\n\n    # Test for gh-5186: Cloning from local FS, shouldn't lead to annex\n    # initializing origin.\n\n    dspath = Path(dspath)\n    storepath = Path(storepath)\n    clonepath = Path(clonepath)\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n        create_ds_in_store,\n        get_layout_locations\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n\n\n    # First create a dataset in a RIA store the standard way\n    somefile = dspath / 'a_file.txt'\n    somefile.write_text('irrelevant')\n    ds = Dataset(dspath).create(force=True)\n\n    io = LocalIO()\n    create_store(io, storepath, '1')\n    lcl_url = \"ria+{}\".format(get_local_file_url(str(storepath)))\n    create_ds_in_store(io, storepath, ds.id, '2', '1')\n    ds.create_sibling_ria(lcl_url, \"store\")\n    ds.push('.', to='store')\n\n\n    # now, remove annex/ tree from store in order to see, that clone\n    # doesn't cause annex to recreate it.\n    store_loc, _, _ = get_layout_locations(1, storepath, ds.id)\n    annex = store_loc / 'annex'\n    rmtree(str(annex))\n    assert_false(annex.exists())\n\n    clone_url = get_local_file_url(str(storepath), compatibility='git') + \\\n                '#{}'.format(ds.id)\n    clone(\"ria+{}\".format(clone_url), clonepath)\n\n    # no need to test the cloning itself - we do that over and over in here\n\n    # bare repo in store still has no local annex:\n    assert_false(annex.exists())\n\n\n@slow  # 17sec on Yarik's laptop\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_inherit_src_candidates(lcl, storepath, url):\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    # dataset with a subdataset\n    ds1 = Dataset(lcl / 'ds1').create()\n    ds1sub = ds1.create('sub')\n    # a different dataset into which we install ds1, but do not touch its subds\n    ds2 = Dataset(lcl / 'ds2').create()\n    ds2.clone(source=ds1.path, path='mysub')\n\n    # we give no dataset a source candidate config!\n    # move all dataset into the store\n    for d in (ds1, ds1sub, ds2):\n        _move2store(storepath, d)\n\n    # now we must be able to obtain all three datasets from the store\n    riaclone = clone(\n        'ria+{}#{}'.format(\n            # store URL\n            url,\n            # ID of the root dataset\n            ds2.id),\n        lcl / 'clone',\n    )\n    # what happens is the the initial clone call sets a source candidate\n    # config, because it sees the dataset coming from a store\n    # all obtained subdatasets get the config inherited on-clone\n    datasets = riaclone.get('.', get_data=False, recursive=True, result_xfm='datasets')\n    # we get two subdatasets\n    eq_(len(datasets), 2)\n    for ds in datasets:\n        eq_(ConfigManager(dataset=ds, source='dataset-local').get(\n            'datalad.get.subdataset-source-candidate-200origin'),\n            'ria+%s#{id}' % url)\n\n\n@skip_if_no_network\n@with_tempfile()\ndef test_ria_http_storedataladorg(path):\n    # can we clone from the store w/o any dedicated config\n    ds = clone('ria+http://store.datalad.org#{}'.format(datalad_store_testds_id), path)\n    ok_(ds.is_installed())\n    eq_(ds.id, datalad_store_testds_id)\n\n\n@skip_if_on_windows  # see gh-4131\n# Ephemeral clones cannot use adjusted mode repos\n@skip_if_adjusted_branch\n@with_tree(tree={\n    'ds': {\n        'test.txt': 'some',\n        'subdir': {'testsub.txt': 'somemore'},\n    },\n})\n@with_tempfile\n@with_tempfile\n@with_tempfile\n@with_tempfile\ndef test_ephemeral(origin_path, bare_path,\n                   clone1_path, clone2_path, clone3_path):\n\n    file_test = Path('ds') / 'test.txt'\n    file_testsub = Path('ds') / 'subdir' / 'testsub.txt'\n\n    origin = Dataset(origin_path).create(force=True)\n    origin.save()\n    # 1. clone via path\n    clone1 = clone(origin_path, clone1_path, reckless='ephemeral')\n\n    can_symlink = has_symlink_capability()\n\n    if can_symlink:\n        clone1_annex = (clone1.repo.dot_git / 'annex')\n        ok_(clone1_annex.is_symlink())\n        ok_(clone1_annex.resolve().samefile(origin.repo.dot_git / 'annex'))\n        if not clone1.repo.is_managed_branch():\n            # TODO: We can't properly handle adjusted branch yet\n            eq_((clone1.pathobj / file_test).read_text(), 'some')\n            eq_((clone1.pathobj / file_testsub).read_text(), 'somemore')\n\n    # 2. clone via file-scheme URL\n    clone2 = clone('file://' + Path(origin_path).as_posix(), clone2_path,\n                   reckless='ephemeral')\n\n    if can_symlink:\n        clone2_annex = (clone2.repo.dot_git / 'annex')\n        ok_(clone2_annex.is_symlink())\n        ok_(clone2_annex.resolve().samefile(origin.repo.dot_git / 'annex'))\n        if not clone2.repo.is_managed_branch():\n            # TODO: We can't properly handle adjusted branch yet\n            eq_((clone2.pathobj / file_test).read_text(), 'some')\n            eq_((clone2.pathobj / file_testsub).read_text(), 'somemore')\n\n    # 3. add something to clone1 and push back to origin availability from\n    # clone1 should not be propagated (we declared 'here' dead to that end)\n\n    (clone1.pathobj / 'addition.txt').write_text(\"even more\")\n    clone1.save()\n    origin.config.set(\"receive.denyCurrentBranch\", \"updateInstead\",\n                      where=\"local\")\n    # Note, that the only thing to test is git-annex-dead here,\n    # if we couldn't symlink:\n    clone1.publish(to=DEFAULT_REMOTE,\n                   transfer_data='none' if can_symlink else 'auto')\n    if not origin.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        eq_(origin.repo.get_hexsha(), clone1.repo.get_hexsha())\n    res = origin.repo.whereis(\"addition.txt\")\n    if can_symlink:\n        # obv. present in origin, but this is not yet known to origin:\n        eq_(res, [])\n        res = origin.repo.fsck()\n        assert_result_count(res, 3, success=True)\n        # TODO: Double check whether annex reports POSIX paths o windows!\n        eq_({str(file_test), str(file_testsub), \"addition.txt\"},\n            {r['file'] for r in res})\n        # now origin knows:\n    res = origin.repo.whereis(\"addition.txt\")\n    eq_(res, [origin.config.get(\"annex.uuid\")])\n\n    # 4. ephemeral clone from a bare repo\n    runner = GitWitlessRunner()\n    runner.run(['git', 'clone', '--bare', origin_path, bare_path])\n    runner.run(['git', 'annex', 'init'], cwd=bare_path)\n\n    eph_from_bare = clone(bare_path, clone3_path, reckless='ephemeral')\n    can_symlink = has_symlink_capability()\n\n    if can_symlink:\n        # Bare repo uses dirhashlower by default, while a standard repo uses\n        # dirhashmixed. Symlinking different object trees doesn't really work.\n        # Don't test that here, since this is not a matter of the \"ephemeral\"\n        # option alone. We should have such a setup in the RIA tests and test\n        # for data access there.\n        # Here we only test for the correct linking.\n        eph_annex = eph_from_bare.repo.dot_git / 'annex'\n        ok_(eph_annex.is_symlink())\n        ok_(eph_annex.resolve().samefile(Path(bare_path) / 'annex'))\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create()\n    repo = ds_origin.repo\n    managed = repo.is_managed_branch()\n\n    # The setup below is involved, mostly because it's accounting for adjusted\n    # branches. The scenario itself isn't so complicated, though:\n    #\n    #   * a checked out default branch with no commits\n    #   * a (potentially adjusted) \"abc\" branch with commits.\n    #   * a (potentially adjusted) \"chooseme\" branch whose tip commit has a\n    #     more recent commit than any in \"abc\".\n    (ds_origin.pathobj / \"foo\").write_text(\"foo content\")\n    ds_origin.save(message=\"foo\")\n    for res in repo.for_each_ref_(fields=\"refname\"):\n        ref = res[\"refname\"]\n        if DEFAULT_BRANCH in ref:\n            repo.update_ref(ref.replace(DEFAULT_BRANCH, \"abc\"), ref)\n            repo.call_git([\"update-ref\", \"-d\", ref])\n    repo.update_ref(\"HEAD\",\n                    \"refs/heads/{}\".format(\n                        \"adjusted/abc(unlocked)\" if managed else \"abc\"),\n                    symbolic=True)\n    abc_ts = int(repo.format_commit(\"%ct\"))\n    repo.call_git([\"checkout\", \"-b\", \"chooseme\", \"abc~1\"])\n    if managed:\n        repo.adjust()\n    (ds_origin.pathobj / \"bar\").write_text(\"bar content\")\n    with set_date(abc_ts + 1):\n        ds_origin.save(message=\"bar\")\n    # Make the git-annex branch the most recently updated ref so that we test\n    # that it is skipped.\n    with set_date(abc_ts + 2):\n        ds_origin.drop(\"bar\", check=False)\n    ds_origin.repo.checkout(DEFAULT_BRANCH, options=[\"--orphan\"])\n\n    ds = clone(ds_origin.path, op.join(path, \"b\"))\n    # We landed on the branch with the most recent commit, ignoring the\n    # git-annex branch.\n    branch = ds.repo.get_active_branch()\n    eq_(ds.repo.get_corresponding_branch(branch) or branch,\n        \"chooseme\")\n    eq_(ds_origin.repo.get_hexsha(\"chooseme\"),\n        ds.repo.get_hexsha(\"chooseme\"))\n    # In the context of this test, the clone should be on an adjusted branch if\n    # the source landed there initially because we're on the same file system.\n    eq_(managed, ds.repo.is_managed_branch())\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head_no_other_ref(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create(annex=False)\n    ds_origin.repo.call_git([\"update-ref\", \"-d\",\n                             \"refs/heads/\" + DEFAULT_BRANCH])\n    with swallow_logs(new_level=logging.WARNING) as cml:\n        clone(source=ds_origin.path, path=op.join(path, \"b\"))\n        assert_in(\"could not find a branch with commits\", cml.out)\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head_sub(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create()\n    ds_origin_sub = Dataset(op.join(path, \"a\", \"sub\")).create()\n    managed = ds_origin_sub.repo.is_managed_branch()\n    ds_origin.save(message=\"foo\")\n    sub_repo = ds_origin_sub.repo\n    # As with test_clone_unborn_head(), the setup below is complicated mostly\n    # because it's accounting for adjusted branches, but the scenario itself\n    # isn't too complicated:\n    #\n    #   * a submodule's HEAD points to a checked out branch with no commits\n    #     while a (potentially adjusted) \"other\" branch has commits\n    #\n    #   * the parent repo has the tip of \"other\" as the last recorded state\n    for res in sub_repo.for_each_ref_(fields=\"refname\"):\n        ref = res[\"refname\"]\n        if DEFAULT_BRANCH in ref:\n            sub_repo.update_ref(ref.replace(DEFAULT_BRANCH, \"other\"), ref)\n            sub_repo.call_git([\"update-ref\", \"-d\", ref])\n    sub_repo.update_ref(\n        \"HEAD\",\n        \"refs/heads/{}\".format(\n            \"adjusted/other(unlocked)\" if managed else \"other\"),\n        symbolic=True)\n    # END complicated handling for adjusted branches\n    ds_origin.save()\n    ds_origin_sub.repo.checkout(DEFAULT_BRANCH, options=[\"--orphan\"])\n\n    ds_cloned = clone(source=ds_origin.path, path=op.join(path, \"b\"))\n    ds_cloned_sub = ds_cloned.get(\n        \"sub\", result_xfm=\"datasets\", return_type=\"item-or-list\")\n\n    branch = ds_cloned_sub.repo.get_active_branch()\n    eq_(ds_cloned_sub.repo.get_corresponding_branch(branch) or branch,\n        \"other\")\n    # In the context of this test, the clone should be on an adjusted branch if\n    # the source landed there initially because we're on the same file system.\n    eq_(managed, ds_cloned_sub.repo.is_managed_branch())\n\n\n@skip_if_no_network\n@with_tempfile\ndef test_gin_cloning(path):\n    # can we clone a public ds anoynmously from gin and retrieve content\n    ds = clone('https://gin.g-node.org/datalad/datalad-ci-target', path)\n    ok_(ds.is_installed())\n    annex_path = op.join('annex', 'two')\n    git_path = op.join('git', 'one')\n    eq_(ds.repo.file_has_content(annex_path), False)\n    eq_(ds.repo.is_under_annex(git_path), False)\n    result = ds.get(annex_path)\n    assert_result_count(result, 1)\n    assert_status('ok', result)\n    eq_(result[0]['path'], op.join(ds.path, annex_path))\n    ok_file_has_content(op.join(ds.path, annex_path), 'two\\n')\n    ok_file_has_content(op.join(ds.path, git_path), 'one\\n')\n\n\n# TODO: git-annex-init fails in the second clone call below when this is\n# executed under ./tools/eval_under_testloopfs.\n@skip_if_adjusted_branch\n@with_tree(tree={\"special\": {\"f0\": \"0\"}})\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_fetch_git_special_remote(url_path, url, path):\n    url_path = Path(url_path)\n    path = Path(path)\n    ds_special = Dataset(url_path / \"special\").create(force=True)\n    ds_special.save()\n    ds_special.repo.call_git([\"update-server-info\"])\n\n    clone_url = url + \"special/.git\"\n    ds_a = clone(clone_url, path / \"a\")\n    ds_a.repo.call_annex(\n        [\"initremote\", \"special\", \"type=git\", \"autoenable=true\",\n         \"location=\" + clone_url])\n\n    # Set up a situation where a file is present only on the special remote,\n    # and its existence is known only to the special remote's git-annex branch.\n    (ds_special.pathobj / \"f1\").write_text(\"1\")\n    ds_special.save()\n    ds_special.repo.call_git([\"update-server-info\"])\n\n    ds_a.repo.fetch(DEFAULT_REMOTE)\n    ds_a.repo.merge(f\"{DEFAULT_REMOTE}/{DEFAULT_BRANCH}\")\n\n    ds_b = clone(ds_a.path, path / \"other\")\n    ds_b.get(\"f1\")\n    ok_(ds_b.repo.file_has_content(\"f1\"))\n\n\n@skip_if_adjusted_branch\n@skip_if_no_network\n@with_tempfile(mkdir=True)\ndef test_nonuniform_adjusted_subdataset(path):\n    # https://github.com/datalad/datalad/issues/5107\n    topds = Dataset(Path(path) / \"top\").create()\n    subds_url = 'git://github.com/datalad/testrepo--basic--r1'\n    topds.clone(\n        source='git://github.com/datalad/testrepo--basic--r1',\n        path='subds')\n    eq_(topds.subdatasets(return_type='item-or-list')['gitmodule_url'],\n        subds_url)\n\n\n@with_tempfile\ndef test_clone_recorded_subds_reset(path):\n    path = Path(path)\n    ds_a = create(path / \"ds_a\")\n    ds_a_sub = ds_a.create(\"sub\")\n    (ds_a_sub.pathobj / \"foo\").write_text(\"foo\")\n    ds_a.save(recursive=True)\n    (ds_a_sub.pathobj / \"bar\").write_text(\"bar\")\n    ds_a_sub.save()\n\n    ds_b = clone(ds_a.path, path / \"ds_b\")\n    ds_b.get(\"sub\")\n    assert_repo_status(ds_b.path)\n    sub_repo = Dataset(path / \"ds_b\" / \"sub\").repo\n    branch = sub_repo.get_active_branch()\n    eq_(ds_b.subdatasets()[0][\"gitshasum\"],\n        sub_repo.get_hexsha(\n            sub_repo.get_corresponding_branch(branch) or branch))\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_url_mapping(src_path, dest_path):\n    src = create(src_path)\n    dest = Dataset(dest_path)\n    # check that the impossible doesn't work\n    assert_raises(IncompleteResultsError, clone, 'rambo', dest_path)\n    # rather than adding test URL mapping here, consider\n    # test_url_mapping_specs(), it is cheaper there\n  \n    # anticipate windows test paths and escape them\n    escaped_subst = (r',rambo,%s' % src_path).replace('\\\\', '\\\\\\\\')\n    for specs in (\n            # we can clone with a simple substitution\n            {'datalad.clone.url-substitute.mike': escaped_subst},\n            # a prior match to a dysfunctional URL doesn't impact success\n            {\n                'datalad.clone.url-substitute.no': ',rambo,picknick',\n                'datalad.clone.url-substitute.mike': escaped_subst,\n            }):\n        try:\n            with patch.dict(dest.config._merged_store, specs):\n                clone('rambo', dest_path)\n        finally:\n            dest.remove(check=False)\n\n    # check submodule config impact\n    dest.create()\n    with patch.dict(dest.config._merged_store,\n                    {'datalad.clone.url-substitute.mike': escaped_subst}):\n        dest.clone('rambo', 'subds')\n    submod_rec = dest.repo.get_submodules()[0]\n    # we record the original-original URL\n    eq_(submod_rec['gitmodule_datalad-url'], 'rambo')\n    # and put the effective one as the primary URL\n    eq_(submod_rec['gitmodule_url'], src_path)\n\n\n_nomatch_map = {\n    'datalad.clone.url-substitute.nomatch': (\n        ',nomatch,NULL',\n    )\n}\n_windows_map = {\n    'datalad.clone.url-substitute.win': (\n        r',C:\\\\Users\\\\datalad\\\\from,D:\\\\to',\n    )\n}\n\n\ndef test_url_mapping_specs():\n    from datalad.core.distributed.clone import _map_urls\n    cfg = ConfigManager()\n    for m, i, o in (\n            # path redirect on windows\n            (_windows_map,\n             r'C:\\Users\\datalad\\from',\n             r'D:\\to'),\n            # test standard github mapping, no pathc needed\n            ({},\n             'https://github.com/datalad/testrepo_gh/sub _1',\n             'https://github.com/datalad/testrepo_gh-sub__1'),\n            # and on deep subdataset too\n            ({},\n             'https://github.com/datalad/testrepo_gh/sub _1/d/sub_-  1',\n             'https://github.com/datalad/testrepo_gh-sub__1-d-sub_-_1'),\n            # test that the presence of another mapping spec doesn't ruin\n            # the outcome\n            (_nomatch_map,\n             'https://github.com/datalad/testrepo_gh/sub _1',\n             'https://github.com/datalad/testrepo_gh-sub__1'),\n            # verify OSF mapping, but see\n            # https://github.com/datalad/datalad/issues/5769 for future\n            # implications\n            ({},\n             'https://osf.io/q8xnk/',\n             'osf://q8xnk'),\n            ):\n        with patch.dict(cfg._merged_store, m):\n            eq_(_map_urls(cfg, [i]), [o])\n",
            "new_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test clone action\n\n\"\"\"\n\nimport logging\nimport os.path as op\nimport stat\n\nfrom unittest.mock import patch\n\nfrom datalad.config import ConfigManager\nfrom datalad import consts\nfrom datalad.api import (\n    clone,\n    create,\n    remove,\n)\nfrom datalad.utils import (\n    chpwd,\n    get_home_envvars,\n    Path,\n    on_windows,\n    rmtree\n)\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.cmd import (\n    GitWitlessRunner,\n    WitlessRunner as Runner,\n)\nfrom datalad.tests.utils import (\n    assert_false,\n    assert_in,\n    assert_in_results,\n    assert_not_is_instance,\n    assert_message,\n    assert_not_in,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_result_values_equal,\n    assert_status,\n    create_tree,\n    DEFAULT_BRANCH,\n    DEFAULT_REMOTE,\n    eq_,\n    get_datasets_topdir,\n    integration,\n    known_failure,\n    known_failure_githubci_win,\n    known_failure_windows,\n    neq_,\n    nok_,\n    ok_,\n    ok_file_has_content,\n    has_symlink_capability,\n    ok_startswith,\n    patch_config,\n    set_date,\n    serve_path_via_http,\n    skip_if_adjusted_branch,\n    skip_if_no_network,\n    skip_if_on_windows,\n    skip_ssh,\n    slow,\n    swallow_logs,\n    use_cassette,\n    with_sameas_remote,\n    with_tempfile,\n    with_testrepos,\n    with_tree,\n    SkipTest,\n)\nfrom datalad.support.network import get_local_file_url\nfrom datalad.core.distributed.clone import (\n    _get_installationpath_from_url,\n    decode_source_spec,\n)\nfrom datalad.distribution.dataset import Dataset\n\n# this is the dataset ID of our test dataset in the main datalad RIA store\ndatalad_store_testds_id = '76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561'\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_args(path, otherpath, alienpath):\n    # source == path\n    assert_raises(ValueError, clone, 'Zoidberg', path='Zoidberg')\n    assert_raises(ValueError, clone, 'ssh://mars/Zoidberg', path='ssh://mars/Zoidberg')\n\n    # \"invalid URL\" is a valid filepath... and since no clone to remote\n    # is possible - we can just assume that it is the (legit) file path\n    # which is provided, not a URL.  So both below should fail as any\n    # other clone from a non-existing source and not for the reason of\n    # \"invalid something\".  Behavior is similar to how Git performs - can\n    # clone into a URL-like path.\n\n    # install to an \"invalid URL\" path\n    res = clone('Zoidberg', path='ssh://mars:Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # install to a \"remote location\" path\n    res = clone('Zoidberg', path='ssh://mars/Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # make fake dataset\n    ds = create(path)\n    assert_raises(IncompleteResultsError, ds.clone, '/higherup.', 'Zoidberg')\n    # make real dataset, try to install outside\n    ds_target = create(Path(otherpath) / 'target')\n    assert_raises(ValueError, ds_target.clone, ds.path, path=ds.path)\n    assert_status('error', ds_target.clone(ds.path, path=alienpath, on_failure='ignore'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_crcns(tdir, ds_path):\n    with chpwd(tdir):\n        res = clone('///', path=\"all-nonrecursive\", on_failure='ignore',\n                    result_xfm=None, return_type='list')\n        assert_status('ok', res)\n\n    # again, but into existing dataset:\n    ds = create(ds_path)\n    crcns = ds.clone(\"///crcns\", result_xfm='datasets', return_type='item-or-list')\n    ok_(crcns.is_installed())\n    eq_(crcns.pathobj, ds.pathobj / \"crcns\")\n    assert_in(crcns.path, ds.subdatasets(result_xfm='paths'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tree(tree={'sub': {}})\ndef test_clone_datasets_root(tdir):\n    tdir = Path(tdir)\n    with chpwd(tdir):\n        ds = clone(\"///\")\n        ok_(ds.is_installed())\n        eq_(ds.pathobj, tdir / get_datasets_topdir())\n\n        # do it a second time:\n        res = clone(\"///\", on_failure='ignore', result_xfm=None, return_type='list')\n        assert_message(\n            \"dataset %s was already cloned from '%s'\",\n            res)\n        assert_status('notneeded', res)\n\n        # and a third time into an existing something, that is not a dataset:\n        (tdir / 'sub' / 'a_file.txt').write_text(\"something\")\n\n        res = clone('///', path=\"sub\", on_failure='ignore')\n        assert_message(\n            'target path already exists and not empty, refuse to clone into target path',\n            res)\n        assert_status('error', res)\n\n\n@with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n@with_tempfile(mkdir=True)\ndef test_clone_simple_local(src, path):\n    origin = Dataset(path)\n\n    # now install it somewhere else\n    ds = clone(src, path, description='mydummy',\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    ok_(ds.is_installed())\n    if not isinstance(origin.repo, AnnexRepo):\n        # this means it is a GitRepo\n        ok_(isinstance(origin.repo, GitRepo))\n        # stays plain Git repo\n        ok_(isinstance(ds.repo, GitRepo))\n        ok_(not isinstance(ds.repo, AnnexRepo))\n        ok_(GitRepo.is_valid_repo(ds.path))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt'})\n        assert_repo_status(path, annex=False)\n    else:\n        # must be an annex\n        ok_(isinstance(ds.repo, AnnexRepo))\n        ok_(AnnexRepo.is_valid_repo(ds.path, allow_noninitialized=False))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt', 'test-annex.dat'})\n        assert_repo_status(path, annex=True)\n        # no content was installed:\n        ok_(not ds.repo.file_has_content('test-annex.dat'))\n        uuid_before = ds.repo.uuid\n        ok_(uuid_before)  # make sure we actually have an uuid\n        eq_(ds.repo.get_description(), 'mydummy')\n\n    # installing it again, shouldn't matter:\n    res = clone(src, path, result_xfm=None, return_type='list')\n    assert_result_values_equal(res, 'source_url', [src])\n    assert_status('notneeded', res)\n    assert_message(\"dataset %s was already cloned from '%s'\", res)\n    ok_(ds.is_installed())\n    if isinstance(origin.repo, AnnexRepo):\n        eq_(uuid_before, ds.repo.uuid)\n\n\n\n# AssertionError: unexpected content of state \"deleted\": [WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/datalad_temp_gzegy3hf/testrepo--basic--r1/test-annex.dat')] != []\n@known_failure_githubci_win\n@with_testrepos(flavors=['local-url', 'network', 'local'])\n@with_tempfile\ndef test_clone_dataset_from_just_source(url, path):\n    with chpwd(path, mkdir=True):\n        ds = clone(url, result_xfm='datasets', return_type='item-or-list')\n\n    ok_startswith(ds.path, path)\n    ok_(ds.is_installed())\n    ok_(GitRepo.is_valid_repo(ds.path))\n    assert_repo_status(ds.path, annex=None)\n    assert_in('INFO.txt', ds.repo.get_indexed_files())\n\n\n# test fails randomly, likely a bug in one of the employed test helpers\n# https://github.com/datalad/datalad/pull/3966#issuecomment-571267932\n@known_failure\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_clone_dataladri(src, topurl, path):\n    # make plain git repo\n    ds_path = Path(src) / 'ds'\n    gr = GitRepo(ds_path, create=True)\n    gr.add('test.txt')\n    gr.commit('demo')\n    Runner(cwd=gr.path).run(['git', 'update-server-info'])\n    # now install it somewhere else\n    with patch('datalad.consts.DATASETS_TOPURL', topurl):\n        ds = clone('///ds', path, result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    assert_repo_status(path, annex=False)\n    ok_file_has_content(ds.pathobj / 'test.txt', 'some')\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_isnot_recursive(path_src, path_nr, path_r):\n    src = Dataset(path_src).create()\n    src.create('subm 1')\n    src.create('2')\n\n    ds = clone(src, path_nr, result_xfm='datasets', return_type='item-or-list')\n    ok_(ds.is_installed())\n    # check nothing is unintentionally installed\n    subdss = ds.subdatasets(recursive=True)\n    assert_result_count(subdss, len(subdss), state='absent')\n    # this also means, subdatasets to be listed as not fulfilled:\n    eq_(set(ds.subdatasets(recursive=True, fulfilled=False, result_xfm='relpaths')),\n        {'subm 1', '2'})\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_into_dataset(source_path, top_path):\n    source = Dataset(source_path).create()\n    ds = create(top_path)\n    assert_repo_status(ds.path)\n    # Note, we test against the produced history in DEFAULT_BRANCH, not what it\n    # turns into in an adjusted branch!\n    hexsha_before = ds.repo.get_hexsha(DEFAULT_BRANCH)\n    subds = ds.clone(source, \"sub\",\n                     result_xfm='datasets', return_type='item-or-list')\n    ok_((subds.pathobj / '.git').is_dir())\n    ok_(subds.is_installed())\n    assert_in('sub', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # sub is clean:\n    assert_repo_status(subds.path, annex=None)\n    # top is clean:\n    assert_repo_status(ds.path, annex=None)\n    # source is recorded in .gitmodules:\n    sds = ds.subdatasets(\"sub\")\n    assert_result_count(sds, 1, action='subdataset')\n    eq_(sds[0]['gitmodule_datalad-url'], source.path)\n    # Clone produced one commit including the addition to .gitmodule:\n    commits = list(ds.repo.get_branch_commits_(\n        branch=DEFAULT_BRANCH,\n        stop=hexsha_before\n    ))\n    assert_not_in(hexsha_before, commits)\n    eq_(len(commits), 1)\n\n    # but we could also save while installing and there should be no side-effect\n    # of saving any other changes if we state to not auto-save changes\n    # Create a dummy change\n    create_tree(ds.path, {'dummy.txt': 'buga'})\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n    subds_ = ds.clone(source, \"sub2\",\n                      result_xfm='datasets', return_type='item-or-list')\n    eq_(subds_.pathobj, ds.pathobj / \"sub2\")  # for paranoid yoh ;)\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_notclone_known_subdataset(src_path, path):\n    src = Dataset(src_path).create()\n    sub = src.create('subm 1')\n    sub_id = sub.id\n    # get the superdataset:\n    ds = clone(src, path,\n               result_xfm='datasets', return_type='item-or-list')\n\n    # subdataset not installed:\n    subds = Dataset(ds.pathobj / 'subm 1')\n    assert_false(subds.is_installed())\n    assert_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # clone is not meaningful\n    res = ds.clone('subm 1', on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n    # get does the job\n    res = ds.get(path='subm 1', get_data=False)\n    assert_status('ok', res)\n    ok_(subds.is_installed())\n    ok_(AnnexRepo.is_valid_repo(subds.path, allow_noninitialized=False))\n    # Verify that it is the correct submodule installed and not\n    # new repository initiated\n    eq_(subds.id, sub_id)\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n\n\n@with_tempfile(mkdir=True)\ndef test_failed_clone(dspath):\n    ds = create(dspath)\n    res = ds.clone(\"http://nonexistingreallyanything.datalad.org/bla\", \"sub\",\n                   on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n\n\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@with_tempfile\ndef test_clone_missing_commit(source, clone):\n\n    from datalad.core.distributed.clone import clone_dataset\n\n    source = Path(source)\n    clone = Path(clone)\n\n    # Commit SHA from another repository - should never be recreated in a fresh\n    # dataset:\n    commit_sha = \"c29691b37b05b78ffa76e5fdf0044e9df673e8f1\"\n\n    origin = Dataset(source).create(force=True)\n    origin.save()\n\n    # clone origin but request commit_sha to be checked out:\n\n    results = [x for x in\n               clone_dataset(srcs=[source], destds=Dataset(clone),\n                             checkout_gitsha=commit_sha)\n               ]\n    # expected error result:\n    assert_result_count(results, 1)\n    assert_in_results(results, status='error', action='install',\n                      path=str(clone), type='dataset')\n    assert_in(\"Target commit c29691b3 does not exist in the clone\",\n              results[0]['message'])\n    # failed attempt was removed:\n    assert_false(clone.exists())\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef check_reckless(annex, src_path, top_path, sharedpath):\n    # super with or without annex\n    src = Dataset(src_path).create(annex=annex)\n    # sub always with annex\n    srcsub = src.create('sub')\n\n    # and for the actual test\n    ds = clone(src.path, top_path, reckless=True,\n               result_xfm='datasets', return_type='item-or-list')\n\n    is_crippled = srcsub.repo.is_managed_branch()\n\n    if annex and not is_crippled:\n        eq_(ds.config.get('annex.hardlink', None), 'true')\n\n    # actual value is 'auto', because True is a legacy value and we map it\n    eq_(ds.config.get('datalad.clone.reckless', None), 'auto')\n    if annex:\n        eq_(ds.repo.repo_info()['untrusted repositories'][0]['here'], True)\n    # now, if we clone another repo into this one, it will inherit the setting\n    # without having to provide it explicitly\n    newsub = ds.clone(srcsub, 'newsub', result_xfm='datasets', return_type='item-or-list')\n    # and `get` the original subdataset\n    origsub = ds.get('sub', result_xfm='datasets', return_type='item-or-list')\n    for sds in (newsub, origsub):\n        eq_(sds.config.get('datalad.clone.reckless', None), 'auto')\n        if not is_crippled:\n            eq_(sds.config.get('annex.hardlink', None), 'true')\n\n    if is_crippled:\n        raise SkipTest(\"Remainder of test needs proper filesystem permissions\")\n\n    if annex:\n        if ds.repo.git_annex_version < \"8.20200908\":\n            # TODO: Drop when GIT_ANNEX_MIN_VERSION is at least 8.20200908.\n\n            # the standard setup keeps the annex locks accessible to the user only\n            nok_((ds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n                 & stat.S_IWGRP)\n        else:\n            # umask might be such (e.g. 002) that group write permissions are inherited, so\n            # for the next test we should check if that is the case on some sample file\n            dltmp_path = ds.pathobj / '.git' / \"dltmp\"\n            dltmp_path.write_text('')\n            default_grp_write_perms = dltmp_path.stat().st_mode & stat.S_IWGRP\n            dltmp_path.unlink()\n            # the standard setup keeps the annex locks following umask inheritance\n            eq_((ds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n                 & stat.S_IWGRP, default_grp_write_perms)\n\n        # but we can set it up for group-shared access too\n        sharedds = clone(\n            src, sharedpath,\n            reckless='shared-group',\n            result_xfm='datasets',\n            return_type='item-or-list')\n        ok_((sharedds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n            & stat.S_IWGRP)\n\n\ndef test_reckless():\n    yield check_reckless, True\n    yield check_reckless, False\n\n@with_tempfile\n@with_tempfile\ndef test_install_source_relpath(src, dest):\n    src = Path(src)\n    create(src)\n    src_ = src.name\n    with chpwd(src.parent):\n        clone(src_, dest)\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_isnt_a_smartass(origin_path, path):\n    origin = create(origin_path)\n    cloned = clone(origin, path,\n                   result_xfm='datasets', return_type='item-or-list')\n    with chpwd(path):\n        # no were are inside a dataset clone, and we make another one\n        # we do not want automatic subdatasetification without given a dataset\n        # explicitly\n        clonedsub = clone(origin, 'testsub',\n                          result_xfm='datasets', return_type='item-or-list')\n    # correct destination\n    assert clonedsub.path.startswith(path)\n    # no subdataset relation\n    eq_(cloned.subdatasets(), [])\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_report_permission_issue(tdir):\n    pdir = Path(tdir) / 'protected'\n    pdir.mkdir()\n    # make it read-only\n    pdir.chmod(0o555)\n    with chpwd(pdir):\n        # first check the premise of the test. If we can write (strangely\n        # mounted/crippled file system, subsequent assumptions are violated\n        # and we can stop\n        probe = Path('probe')\n        try:\n            probe.write_text('should not work')\n            raise SkipTest\n        except PermissionError:\n            # we are indeed in a read-only situation\n            pass\n        res = clone('///', result_xfm=None, return_type='list', on_failure='ignore')\n        assert_status('error', res)\n        assert_result_count(\n            res, 1, status='error',\n            message=\"could not create work tree dir '%s/%s': Permission denied\"\n                    % (pdir, get_datasets_topdir())\n        )\n\n\n@skip_if_no_network\n@with_tempfile\ndef test_autoenabled_remote_msg(path):\n    # Verify that no message about a remote not been enabled is displayed\n    # whenever the remote we clone is the  type=git special remote, so the name\n    # of the remote might not match\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone('///repronim/containers', path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_not_in(\"not auto-enabled\", cml.out)\n\n\n@with_sameas_remote(autoenabled=True)\n@with_tempfile(mkdir=True)\ndef test_clone_autoenable_msg_handles_sameas(repo, clone_path):\n    ds = Dataset(repo.path)\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone(ds, clone_path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_in(\"r_dir\", cml.out)\n        assert_in(\"not auto-enabled\", cml.out)\n        # The rsyncurl remote was enabled.\n        assert_not_in(\"r_rsync\", cml.out)\n    ds_cloned = Dataset(clone_path)\n    remotes = ds_cloned.repo.get_remotes()\n    assert_in(\"r_rsync\", remotes)\n    assert_not_in(\"r_dir\", remotes)\n\n\ndef test_installationpath_from_url():\n    # cases for all OSes\n    cases = [\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n        # SSH\n        'hostname:lastbit',\n        'hostname:lastbit/',\n        'hostname:subd/lastbit',\n        'hostname:/full/path/lastbit',\n        'hostname:lastbit/.git',\n        'hostname:lastbit/.git/',\n        'hostname:/full/path/lastbit/.git',\n        'full.hostname.com:lastbit/.git',\n        'user@full.hostname.com:lastbit/.git',\n        'ssh://user:passw@full.hostname.com/full/path/lastbit',\n        'ssh://user:passw@full.hostname.com/full/path/lastbit/',\n        'ssh://user:passw@full.hostname.com/full/path/lastbit/.git',\n    ]\n    # OS specific cases\n    cases += [\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ] if on_windows else [\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    ]\n\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')\n\n\n# https://github.com/datalad/datalad/issues/3958\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_expanduser(srcpath, destpath):\n    src = Dataset(Path(srcpath) / 'src').create()\n    dest = Dataset(Path(destpath) / 'dest').create()\n\n    # We switch away from home set up in datalad.setup_package(), so make sure\n    # we have a valid identity.\n    with open(op.join(srcpath, \".gitconfig\"), \"w\") as fh:\n        fh.write(\"[user]\\n\"\n                 \"name = DataLad oooooTester\\n\"\n                 \"email = test@example.com\\n\")\n\n    with chpwd(destpath), patch.dict('os.environ', get_home_envvars(srcpath)):\n        res = clone(op.join('~', 'src'), 'dest', result_xfm=None, return_type='list',\n                    on_failure='ignore')\n        assert_result_count(res, 1)\n        assert_result_count(\n            res, 1, action='install', status='error', path=dest.path,\n            message='target path already exists and not empty, refuse to '\n            'clone into target path')\n        # wipe out destination, and try again\n        assert_status('ok', remove(dataset=dest, check=False))\n        # now it should do it, and clone the right one\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n        eq_(src.id, cloneds.id)\n        # and it shouldn't fail when doing it again, because it detects\n        # the re-clone\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n\n\n@with_tempfile(mkdir=True)\ndef test_cfg_originorigin(path):\n    path = Path(path)\n    origin = Dataset(path / 'origin').create()\n    (origin.pathobj / 'file1.txt').write_text('content')\n    origin.save()\n    clone_lev1 = clone(origin, path / 'clone_lev1')\n    clone_lev2 = clone(clone_lev1, path / 'clone_lev2')\n    # the goal is to be able to get file content from origin without\n    # the need to configure it manually\n    assert_result_count(\n        clone_lev2.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_lev2.pathobj / 'file1.txt'),\n    )\n    eq_((clone_lev2.pathobj / 'file1.txt').read_text(), 'content')\n    eq_(\n        Path(clone_lev2.siblings(\n            'query',\n            name=DEFAULT_REMOTE + '-2',\n            return_type='item-or-list')['url']),\n        origin.pathobj\n    )\n\n    # Clone another level, this time with a relative path. Drop content from\n    # lev2 so that origin is the only place that the file is available from.\n    clone_lev2.drop(\"file1.txt\")\n    with chpwd(path), swallow_logs(new_level=logging.DEBUG) as cml:\n        clone_lev3 = clone('clone_lev2', 'clone_lev3')\n        # we called git-annex-init; see gh-4367:\n        cml.assert_logged(msg=r\"[^[]*Run \\[('git',.*'annex'|'git-annex'), 'init'\",\n                          match=False,\n                          level='DEBUG')\n    assert_result_count(\n        clone_lev3.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_lev3.pathobj / 'file1.txt'))\n\n\n# test fix for gh-2601/gh-3538\n@known_failure\n@with_tempfile()\ndef test_relative_submodule_url(path):\n    Dataset(op.join(path, 'origin')).create()\n    ds = Dataset(op.join(path, 'ds')).create()\n    with chpwd(ds.path):\n        ds_cloned = ds.clone(\n            source=op.join(op.pardir, 'origin'),\n            path='sources')\n\n    # Check that a simple fetch call does not fail.\n    ds_cloned.repo.fetch()\n\n    subinfo = ds.subdatasets(return_type='item-or-list')\n    eq_(subinfo['gitmodule_url'],\n        # must be a relative URL, not platform-specific relpath!\n        '../../origin')\n\n\n@with_tree(tree={\"subdir\": {}})\n@with_tempfile(mkdir=True)\ndef test_local_url_with_fetch(path, path_other):\n    path = Path(path)\n    path_other = Path(path_other)\n    Dataset(path / \"source\").create()\n\n    for where, source, path in [\n            (path, \"source\", \"a\"),\n            (path / \"subdir\", op.join(op.pardir, \"source\"), \"a\"),\n            (path, \"source\", path_other / \"a\")]:\n        with chpwd(where):\n            ds_cloned = clone(source=source, path=path)\n            # Perform a fetch to check that the URL points to a valid location.\n            ds_cloned.repo.fetch()\n\n\ndef test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}/{}/{}'.format(\n                    proto,\n                    loc,\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')\n\n    # literal dataset name/location\n    eq_(decode_source_spec('ria+http://example.com#~rootds'),\n        {'source': 'ria+http://example.com#~rootds',\n         'version': None, 'type': 'ria',\n         'giturl': 'http://example.com/alias/rootds',\n         'default_destpath': 'rootds'})\n    # version etc still works\n    eq_(decode_source_spec('ria+http://example.com#~rootds@specialbranch'),\n        {'source': 'ria+http://example.com#~rootds@specialbranch',\n         'version': 'specialbranch', 'type': 'ria',\n         'giturl': 'http://example.com/alias/rootds',\n         'default_destpath': 'rootds'})\n\n\ndef _move2store(storepath, d):\n    # make a bare clone of it into a local that matches the organization\n    # of a ria dataset store\n    store_loc = str(storepath / d.id[:3] / d.id[3:])\n    d.repo.call_git(['clone', '--bare', d.path, store_loc])\n    d.siblings('configure', name='store', url=str(store_loc),\n               result_renderer='disabled')\n    Runner(cwd=store_loc).run(['git', 'update-server-info'])\n\n\n@slow  # 12sec on Yarik's laptop\n@with_tree(tree={\n    'ds': {\n        'test.txt': 'some',\n        'subdir': {\n            'subds': {'testsub.txt': 'somemore'},\n        },\n    },\n})\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_ria_http(lcl, storepath, url):\n    # create a local dataset with a subdataset\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    subds = Dataset(lcl / 'ds' / 'subdir' / 'subds').create(force=True)\n    subds.save()\n    ds = Dataset(lcl / 'ds').create(force=True)\n    ds.save(version_tag='original')\n    assert_repo_status(ds.path)\n    for d in (ds, subds):\n        _move2store(storepath, d)\n    # location of superds in store\n    storeds_loc = str(storepath / ds.id[:3] / ds.id[3:])\n    # now we should be able to clone from a ria+http url\n    # the super\n    riaclone = clone(\n        'ria+{}#{}'.format(url, ds.id),\n        lcl / 'clone',\n    )\n\n    # due to default configuration, clone() should automatically look for the\n    # subdataset in the store, too -- if not the following would fail, because\n    # we never configured a proper submodule URL\n    riaclonesub = riaclone.get(\n        op.join('subdir', 'subds'), get_data=False,\n        result_xfm='datasets', return_type='item-or-list')\n\n    # both datasets came from the store and must be set up in an identical\n    # fashion\n    for origds, cloneds in ((ds, riaclone), (subds, riaclonesub)):\n        eq_(origds.id, cloneds.id)\n        if not ds.repo.is_managed_branch():\n            # test logic cannot handle adjusted branches\n            eq_(origds.repo.get_hexsha(), cloneds.repo.get_hexsha())\n        ok_(cloneds.config.get(f'remote.{DEFAULT_REMOTE}.url').startswith(url))\n        eq_(cloneds.config.get(f'remote.{DEFAULT_REMOTE}.annex-ignore'), 'true')\n        eq_(cloneds.config.get('datalad.get.subdataset-source-candidate-200origin'),\n            'ria+%s#{id}' % url)\n\n    # now advance the source dataset\n    (ds.pathobj / 'newfile.txt').write_text('new')\n    ds.save()\n    ds.publish(to='store')\n    Runner(cwd=storeds_loc).run(['git', 'update-server-info'])\n    # re-clone as before\n    riaclone2 = clone(\n        'ria+{}#{}'.format(url, ds.id),\n        lcl / 'clone2',\n    )\n    # and now clone a specific version, here given be the tag name\n    riaclone_orig = clone(\n        'ria+{}#{}@{}'.format(url, ds.id, 'original'),\n        lcl / 'clone_orig',\n    )\n    if not ds.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        # we got the precise version we wanted\n        eq_(riaclone.repo.get_hexsha(), riaclone_orig.repo.get_hexsha())\n        # and not the latest\n        eq_(riaclone2.repo.get_hexsha(), ds.repo.get_hexsha())\n        neq_(riaclone2.repo.get_hexsha(), riaclone_orig.repo.get_hexsha())\n\n    # attempt to clone a version that doesn't exist\n    with swallow_logs():\n        with assert_raises(IncompleteResultsError) as cme:\n            clone('ria+{}#{}@impossible'.format(url, ds.id),\n                  lcl / 'clone_failed')\n        assert_in(\"not found in upstream\", str(cme.exception))\n\n    # lastly test if URL rewriting is in effect\n    # on the surface we clone from an SSH source identified by some custom\n    # label, no full URL, but URL rewriting setup maps it back to the\n    # HTTP URL used above\n    with patch_config({\n            'url.ria+{}#.insteadof'.format(url): 'ria+ssh://somelabel#'}):\n        cloned_by_label = clone(\n            'ria+ssh://somelabel#{}'.format(origds.id),\n            lcl / 'cloned_by_label',\n        )\n    # so we get the same setup as above, but....\n    eq_(origds.id, cloned_by_label.id)\n    if not ds.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        eq_(origds.repo.get_hexsha(), cloned_by_label.repo.get_hexsha())\n    ok_(cloned_by_label.config.get(\n        f'remote.{DEFAULT_REMOTE}.url').startswith(url))\n    eq_(cloned_by_label.config.get(f'remote.{DEFAULT_REMOTE}.annex-ignore'),\n        'true')\n    # ... the clone candidates go with the label-based URL such that\n    # future get() requests acknowledge a (system-wide) configuration\n    # update\n    eq_(cloned_by_label.config.get('datalad.get.subdataset-source-candidate-200origin'),\n        'ria+ssh://somelabel#{id}')\n\n    if not has_symlink_capability():\n        return\n    # place a symlink in the store to serve as a dataset alias\n    (storepath / 'alias').mkdir()\n    (storepath / 'alias' / 'myname').symlink_to(storeds_loc)\n    with chpwd(lcl):\n        cloned_by_alias = clone('ria+{}#~{}'.format(url, 'myname'))\n    # still get the same data\n    eq_(cloned_by_alias.id, ds.id)\n    # more sensible default install path\n    eq_(cloned_by_alias.pathobj.name, 'myname')\n\n\n@with_tempfile\n@with_tempfile\ndef _test_ria_postclonecfg(url, dsid, clone_path, superds):\n    # Test cloning from RIA store while ORA special remote autoenabling failed\n    # due to an invalid URL from the POV of the cloner.\n    # Origin's git-config-file should contain the UUID to enable. This needs to\n    # work via HTTP, SSH and local cloning.\n\n    # Autoenabling should fail initially by git-annex-init and we would report\n    # on INFO level. Only postclone routine would deal with it.\n    with swallow_logs(new_level=logging.INFO) as cml:\n        # First, the super ds:\n        riaclone = clone('ria+{}#{}'.format(url, dsid), clone_path)\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n\n    # However, we now can retrieve content since clone should have enabled the\n    # special remote with new URL (or origin in case of HTTP).\n    res = riaclone.get('test.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(riaclone.pathobj / 'test.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n    # Second ORA remote is enabled and not reconfigured:\n    untouched_remote = riaclone.siblings(name='anotherstore-storage',\n                                         return_type='item-or-list')\n    assert_not_is_instance(untouched_remote, list)\n    untouched_url = riaclone.repo.get_special_remotes()[\n        untouched_remote['annex-uuid']]['url']\n    ok_(untouched_url.startswith(\"ria+file://\"))\n    ok_(not untouched_url.startswith(\"ria+{}\".format(url)))\n\n    # publication dependency was set for store-storage but not for\n    # anotherstore-storage:\n    eq_(riaclone.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                            get_all=True),\n        \"store-storage\")\n\n    # same thing for the sub ds (we don't need a store-url and id - get should\n    # figure those itself):\n    with swallow_logs(new_level=logging.INFO) as cml:\n        riaclonesub = riaclone.get(\n            op.join('subdir', 'subds'), get_data=False,\n            result_xfm='datasets', return_type='item-or-list')\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n    res = riaclonesub.get('testsub.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(riaclonesub.pathobj / 'testsub.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n    # publication dependency was set for store-storage but not for\n    # anotherstore-storage:\n    eq_(riaclonesub.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                               get_all=True),\n        \"store-storage\")\n\n    # finally get the plain git subdataset.\n    # Clone should figure to also clone it from a ria+ URL\n    # (subdataset-source-candidate), notice that there wasn't an autoenabled ORA\n    # remote, but shouldn't stumble upon it, since it's a plain git.\n    res = riaclone.get(op.join('subdir', 'subgit', 'testgit.txt'))\n    assert_result_count(res, 1, status='ok', type='dataset', action='install')\n    assert_result_count(res, 1, status='notneeded', type='file')\n    assert_result_count(res, 2)\n    # no ORA remote, no publication dependency:\n    riaclonesubgit = Dataset(riaclone.pathobj / 'subdir' / 'subgit')\n    eq_(riaclonesubgit.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                                  get_all=True),\n        None)\n\n    # Now, test that if cloning into a dataset, ria-URL is preserved and\n    # post-clone configuration is triggered again, when we remove the subds and\n    # retrieve it again via `get`:\n    ds = Dataset(superds).create()\n    ria_url = 'ria+{}#{}'.format(url, dsid)\n    ds.clone(ria_url, 'sub')\n    sds = ds.subdatasets('sub')\n    eq_(len(sds), 1)\n    eq_(sds[0]['gitmodule_datalad-url'], ria_url)\n    assert_repo_status(ds.path)\n    ds.uninstall('sub', check=False)\n    assert_repo_status(ds.path)\n\n    # .gitmodules still there:\n    sds = ds.subdatasets('sub')\n    eq_(len(sds), 1)\n    eq_(sds[0]['gitmodule_datalad-url'], ria_url)\n    # get it again:\n\n    # Autoenabling should fail initially by git-annex-init and we would report\n    # on INFO level. Only postclone routine would deal with it.\n    with swallow_logs(new_level=logging.INFO) as cml:\n        ds.get('sub', get_data=False)\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n\n    subds = Dataset(ds.pathobj / 'sub')\n    # special remote is fine:\n    res = subds.get('test.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(subds.pathobj / 'test.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n\n@with_tempfile\ndef _postclonetest_prepare(lcl, storepath, storepath2, link):\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n        create_ds_in_store,\n        get_layout_locations\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n    create_tree(lcl,\n                tree={\n                        'ds': {\n                            'test.txt': 'some',\n                            'subdir': {\n                                'subds': {'testsub.txt': 'somemore'},\n                                'subgit': {'testgit.txt': 'even more'}\n                            },\n                        },\n                      })\n\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    storepath2 = Path(storepath2)\n    link = Path(link)\n    link.symlink_to(storepath)\n\n    # create a local dataset with a subdataset\n    subds = Dataset(lcl / 'ds' / 'subdir' / 'subds').create(force=True)\n    subds.save()\n    # add a plain git dataset as well\n    subgit = Dataset(lcl / 'ds' / 'subdir' / 'subgit').create(force=True,\n                                                              annex=False)\n    subgit.save()\n    ds = Dataset(lcl / 'ds').create(force=True)\n    ds.save(version_tag='original')\n    assert_repo_status(ds.path)\n\n    io = LocalIO()\n\n    # Have a second store with valid ORA remote. This should not interfere with\n    # reconfiguration of the first one, when that second store is not the one we\n    # clone from. However, don't push data into it for easier get-based testing\n    # later on.\n    # Doing this first, so datasets in \"first\"/primary store know about this.\n    create_store(io, storepath2, '1')\n    url2 = \"ria+{}\".format(get_local_file_url(str(storepath2)))\n    for d in (ds, subds, subgit):\n        create_ds_in_store(io, storepath2, d.id, '2', '1')\n        d.create_sibling_ria(url2, \"anotherstore\")\n        d.push('.', to='anotherstore', data='nothing')\n        store2_loc, _, _ = get_layout_locations(1, storepath2, d.id)\n        Runner(cwd=str(store2_loc)).run(['git', 'update-server-info'])\n\n    # Now the store to clone from:\n    create_store(io, storepath, '1')\n\n    # URL to use for upload. Point is, that this should be invalid for the clone\n    # so that autoenable would fail. Therefore let it be based on a to be\n    # deleted symlink\n    upl_url = \"ria+{}\".format(get_local_file_url(str(link)))\n\n    for d in (ds, subds, subgit):\n\n        # TODO: create-sibling-ria required for config! => adapt to RF'd\n        #       creation (missed on rebase?)\n        create_ds_in_store(io, storepath, d.id, '2', '1')\n        d.create_sibling_ria(upl_url, \"store\")\n\n        if d is not subgit:\n            # Now, simulate the problem by reconfiguring the special remote to\n            # not be autoenabled.\n            # Note, however, that the actual intention is a URL, that isn't\n            # valid from the point of view of the clone (doesn't resolve, no\n            # credentials, etc.) and therefore autoenabling on git-annex-init\n            # when datalad-cloning would fail to succeed.\n            Runner(cwd=d.path).run(['git', 'annex', 'enableremote',\n                                    'store-storage',\n                                    'autoenable=false'])\n        d.push('.', to='store')\n        store_loc, _, _ = get_layout_locations(1, storepath, d.id)\n        Runner(cwd=str(store_loc)).run(['git', 'update-server-info'])\n\n    link.unlink()\n    # We should now have a store with datasets that have an autoenabled ORA\n    # remote relying on an inaccessible URL.\n    # datalad-clone is supposed to reconfigure based on the URL we cloned from.\n    # Test this feature for cloning via HTTP, SSH and FILE URLs.\n\n    return ds.id\n\n\n@known_failure_windows  # https://github.com/datalad/datalad/issues/5134\n@slow  # 14 sec on travis\ndef test_ria_postclonecfg():\n\n    if not has_symlink_capability():\n        # This is needed to create an ORA remote using an URL for upload,\n        # that is then invalidated later on (delete the symlink it's based on).\n        raise SkipTest(\"Can't create symlinks\")\n\n    from datalad.utils import make_tempfile\n\n    with make_tempfile(mkdir=True) as lcl, make_tempfile(mkdir=True) as store, \\\n            make_tempfile(mkdir=True) as store2:\n        id = _postclonetest_prepare(lcl, store, store2)\n\n        # test cloning via ria+file://\n        yield _test_ria_postclonecfg, \\\n              get_local_file_url(store, compatibility='git'), id\n\n        # Note: HTTP disabled for now. Requires proper implementation in ORA\n        #       remote. See\n        # https://github.com/datalad/datalad/pull/4203#discussion_r410284649\n\n        # # test cloning via ria+http://\n        # with HTTPPath(store) as url:\n        #     yield _test_ria_postclonecfg, url, id\n\n        # test cloning via ria+ssh://\n        yield skip_ssh(_test_ria_postclonecfg), \\\n            \"ssh://datalad-test:{}\".format(Path(store).as_posix()), id\n\n\n@known_failure_windows\n@skip_ssh\n@with_tree(tree={'somefile.txt': 'some content'})\n@with_tempfile\n@with_tempfile\ndef test_no_ria_postclonecfg(dspath, storepath, clonepath):\n\n    dspath = Path(dspath)\n    storepath = Path(storepath)\n    clonepath = Path(clonepath)\n\n    # Test that particular configuration(s) do NOT lead to a reconfiguration\n    # upon clone. (See gh-5628)\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n    ds = Dataset(dspath).create(force=True)\n    ds.save()\n    assert_repo_status(ds.path)\n\n    io = LocalIO()\n    create_store(io, storepath, '1')\n    file_url = \"ria+{}\".format(get_local_file_url(str(storepath)))\n    ssh_url = \"ria+ssh://datalad-test:{}\".format(storepath.as_posix())\n    ds.create_sibling_ria(file_url, \"teststore\",\n                          push_url=ssh_url, alias=\"testds\")\n    ds.push('.', to='teststore')\n\n    # Now clone via SSH. Should not reconfigure although `url` doesn't match the\n    # URL we cloned from. However, `push-url` does.\n    riaclone = clone('{}#{}'.format(ssh_url, ds.id), clonepath)\n\n    # ORA remote is enabled (since URL still valid) but not reconfigured:\n    untouched_remote = riaclone.siblings(name='teststore-storage',\n                                         return_type='item-or-list')\n    assert_not_is_instance(untouched_remote, list)\n    ora_cfg = riaclone.repo.get_special_remotes()[\n        untouched_remote['annex-uuid']]\n    ok_(ora_cfg['url'] == file_url)\n    ok_(ora_cfg['push-url'] == ssh_url)\n\n    # publication dependency was still set (and it's the only one that was set):\n    eq_(riaclone.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                            get_all=True),\n        \"teststore-storage\")\n\n    # we can still get the content\n    ds.get(\"somefile.txt\")\n\n\n# fatal: Could not read from remote repository.\n@known_failure_githubci_win  # in datalad/git-annex as e.g. of 20201218\n@with_tempfile(mkdir=True)\n@with_tempfile\n@with_tempfile\ndef test_ria_postclone_noannex(dspath, storepath, clonepath):\n\n    # Test for gh-5186: Cloning from local FS, shouldn't lead to annex\n    # initializing origin.\n\n    dspath = Path(dspath)\n    storepath = Path(storepath)\n    clonepath = Path(clonepath)\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n        create_ds_in_store,\n        get_layout_locations\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n\n\n    # First create a dataset in a RIA store the standard way\n    somefile = dspath / 'a_file.txt'\n    somefile.write_text('irrelevant')\n    ds = Dataset(dspath).create(force=True)\n\n    io = LocalIO()\n    create_store(io, storepath, '1')\n    lcl_url = \"ria+{}\".format(get_local_file_url(str(storepath)))\n    create_ds_in_store(io, storepath, ds.id, '2', '1')\n    ds.create_sibling_ria(lcl_url, \"store\")\n    ds.push('.', to='store')\n\n\n    # now, remove annex/ tree from store in order to see, that clone\n    # doesn't cause annex to recreate it.\n    store_loc, _, _ = get_layout_locations(1, storepath, ds.id)\n    annex = store_loc / 'annex'\n    rmtree(str(annex))\n    assert_false(annex.exists())\n\n    clone_url = get_local_file_url(str(storepath), compatibility='git') + \\\n                '#{}'.format(ds.id)\n    clone(\"ria+{}\".format(clone_url), clonepath)\n\n    # no need to test the cloning itself - we do that over and over in here\n\n    # bare repo in store still has no local annex:\n    assert_false(annex.exists())\n\n\n@slow  # 17sec on Yarik's laptop\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_inherit_src_candidates(lcl, storepath, url):\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    # dataset with a subdataset\n    ds1 = Dataset(lcl / 'ds1').create()\n    ds1sub = ds1.create('sub')\n    # a different dataset into which we install ds1, but do not touch its subds\n    ds2 = Dataset(lcl / 'ds2').create()\n    ds2.clone(source=ds1.path, path='mysub')\n\n    # we give no dataset a source candidate config!\n    # move all dataset into the store\n    for d in (ds1, ds1sub, ds2):\n        _move2store(storepath, d)\n\n    # now we must be able to obtain all three datasets from the store\n    riaclone = clone(\n        'ria+{}#{}'.format(\n            # store URL\n            url,\n            # ID of the root dataset\n            ds2.id),\n        lcl / 'clone',\n    )\n    # what happens is the the initial clone call sets a source candidate\n    # config, because it sees the dataset coming from a store\n    # all obtained subdatasets get the config inherited on-clone\n    datasets = riaclone.get('.', get_data=False, recursive=True, result_xfm='datasets')\n    # we get two subdatasets\n    eq_(len(datasets), 2)\n    for ds in datasets:\n        eq_(ConfigManager(dataset=ds, source='dataset-local').get(\n            'datalad.get.subdataset-source-candidate-200origin'),\n            'ria+%s#{id}' % url)\n\n\n@skip_if_no_network\n@with_tempfile()\ndef test_ria_http_storedataladorg(path):\n    # can we clone from the store w/o any dedicated config\n    ds = clone('ria+http://store.datalad.org#{}'.format(datalad_store_testds_id), path)\n    ok_(ds.is_installed())\n    eq_(ds.id, datalad_store_testds_id)\n\n\n@skip_if_on_windows  # see gh-4131\n# Ephemeral clones cannot use adjusted mode repos\n@skip_if_adjusted_branch\n@with_tree(tree={\n    'ds': {\n        'test.txt': 'some',\n        'subdir': {'testsub.txt': 'somemore'},\n    },\n})\n@with_tempfile\n@with_tempfile\n@with_tempfile\n@with_tempfile\ndef test_ephemeral(origin_path, bare_path,\n                   clone1_path, clone2_path, clone3_path):\n\n    file_test = Path('ds') / 'test.txt'\n    file_testsub = Path('ds') / 'subdir' / 'testsub.txt'\n\n    origin = Dataset(origin_path).create(force=True)\n    origin.save()\n    # 1. clone via path\n    clone1 = clone(origin_path, clone1_path, reckless='ephemeral')\n\n    can_symlink = has_symlink_capability()\n\n    if can_symlink:\n        clone1_annex = (clone1.repo.dot_git / 'annex')\n        ok_(clone1_annex.is_symlink())\n        ok_(clone1_annex.resolve().samefile(origin.repo.dot_git / 'annex'))\n        if not clone1.repo.is_managed_branch():\n            # TODO: We can't properly handle adjusted branch yet\n            eq_((clone1.pathobj / file_test).read_text(), 'some')\n            eq_((clone1.pathobj / file_testsub).read_text(), 'somemore')\n\n    # 2. clone via file-scheme URL\n    clone2 = clone('file://' + Path(origin_path).as_posix(), clone2_path,\n                   reckless='ephemeral')\n\n    if can_symlink:\n        clone2_annex = (clone2.repo.dot_git / 'annex')\n        ok_(clone2_annex.is_symlink())\n        ok_(clone2_annex.resolve().samefile(origin.repo.dot_git / 'annex'))\n        if not clone2.repo.is_managed_branch():\n            # TODO: We can't properly handle adjusted branch yet\n            eq_((clone2.pathobj / file_test).read_text(), 'some')\n            eq_((clone2.pathobj / file_testsub).read_text(), 'somemore')\n\n    # 3. add something to clone1 and push back to origin availability from\n    # clone1 should not be propagated (we declared 'here' dead to that end)\n\n    (clone1.pathobj / 'addition.txt').write_text(\"even more\")\n    clone1.save()\n    origin.config.set(\"receive.denyCurrentBranch\", \"updateInstead\",\n                      where=\"local\")\n    # Note, that the only thing to test is git-annex-dead here,\n    # if we couldn't symlink:\n    clone1.publish(to=DEFAULT_REMOTE,\n                   transfer_data='none' if can_symlink else 'auto')\n    if not origin.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        eq_(origin.repo.get_hexsha(), clone1.repo.get_hexsha())\n    res = origin.repo.whereis(\"addition.txt\")\n    if can_symlink:\n        # obv. present in origin, but this is not yet known to origin:\n        eq_(res, [])\n        res = origin.repo.fsck()\n        assert_result_count(res, 3, success=True)\n        # TODO: Double check whether annex reports POSIX paths o windows!\n        eq_({str(file_test), str(file_testsub), \"addition.txt\"},\n            {r['file'] for r in res})\n        # now origin knows:\n    res = origin.repo.whereis(\"addition.txt\")\n    eq_(res, [origin.config.get(\"annex.uuid\")])\n\n    # 4. ephemeral clone from a bare repo\n    runner = GitWitlessRunner()\n    runner.run(['git', 'clone', '--bare', origin_path, bare_path])\n    runner.run(['git', 'annex', 'init'], cwd=bare_path)\n\n    eph_from_bare = clone(bare_path, clone3_path, reckless='ephemeral')\n    can_symlink = has_symlink_capability()\n\n    if can_symlink:\n        # Bare repo uses dirhashlower by default, while a standard repo uses\n        # dirhashmixed. Symlinking different object trees doesn't really work.\n        # Don't test that here, since this is not a matter of the \"ephemeral\"\n        # option alone. We should have such a setup in the RIA tests and test\n        # for data access there.\n        # Here we only test for the correct linking.\n        eph_annex = eph_from_bare.repo.dot_git / 'annex'\n        ok_(eph_annex.is_symlink())\n        ok_(eph_annex.resolve().samefile(Path(bare_path) / 'annex'))\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create()\n    repo = ds_origin.repo\n    managed = repo.is_managed_branch()\n\n    # The setup below is involved, mostly because it's accounting for adjusted\n    # branches. The scenario itself isn't so complicated, though:\n    #\n    #   * a checked out default branch with no commits\n    #   * a (potentially adjusted) \"abc\" branch with commits.\n    #   * a (potentially adjusted) \"chooseme\" branch whose tip commit has a\n    #     more recent commit than any in \"abc\".\n    (ds_origin.pathobj / \"foo\").write_text(\"foo content\")\n    ds_origin.save(message=\"foo\")\n    for res in repo.for_each_ref_(fields=\"refname\"):\n        ref = res[\"refname\"]\n        if DEFAULT_BRANCH in ref:\n            repo.update_ref(ref.replace(DEFAULT_BRANCH, \"abc\"), ref)\n            repo.call_git([\"update-ref\", \"-d\", ref])\n    repo.update_ref(\"HEAD\",\n                    \"refs/heads/{}\".format(\n                        \"adjusted/abc(unlocked)\" if managed else \"abc\"),\n                    symbolic=True)\n    abc_ts = int(repo.format_commit(\"%ct\"))\n    repo.call_git([\"checkout\", \"-b\", \"chooseme\", \"abc~1\"])\n    if managed:\n        repo.adjust()\n    (ds_origin.pathobj / \"bar\").write_text(\"bar content\")\n    with set_date(abc_ts + 1):\n        ds_origin.save(message=\"bar\")\n    # Make the git-annex branch the most recently updated ref so that we test\n    # that it is skipped.\n    with set_date(abc_ts + 2):\n        ds_origin.drop(\"bar\", check=False)\n    ds_origin.repo.checkout(DEFAULT_BRANCH, options=[\"--orphan\"])\n\n    ds = clone(ds_origin.path, op.join(path, \"b\"))\n    # We landed on the branch with the most recent commit, ignoring the\n    # git-annex branch.\n    branch = ds.repo.get_active_branch()\n    eq_(ds.repo.get_corresponding_branch(branch) or branch,\n        \"chooseme\")\n    eq_(ds_origin.repo.get_hexsha(\"chooseme\"),\n        ds.repo.get_hexsha(\"chooseme\"))\n    # In the context of this test, the clone should be on an adjusted branch if\n    # the source landed there initially because we're on the same file system.\n    eq_(managed, ds.repo.is_managed_branch())\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head_no_other_ref(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create(annex=False)\n    ds_origin.repo.call_git([\"update-ref\", \"-d\",\n                             \"refs/heads/\" + DEFAULT_BRANCH])\n    with swallow_logs(new_level=logging.WARNING) as cml:\n        clone(source=ds_origin.path, path=op.join(path, \"b\"))\n        assert_in(\"could not find a branch with commits\", cml.out)\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head_sub(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create()\n    ds_origin_sub = Dataset(op.join(path, \"a\", \"sub\")).create()\n    managed = ds_origin_sub.repo.is_managed_branch()\n    ds_origin.save(message=\"foo\")\n    sub_repo = ds_origin_sub.repo\n    # As with test_clone_unborn_head(), the setup below is complicated mostly\n    # because it's accounting for adjusted branches, but the scenario itself\n    # isn't too complicated:\n    #\n    #   * a submodule's HEAD points to a checked out branch with no commits\n    #     while a (potentially adjusted) \"other\" branch has commits\n    #\n    #   * the parent repo has the tip of \"other\" as the last recorded state\n    for res in sub_repo.for_each_ref_(fields=\"refname\"):\n        ref = res[\"refname\"]\n        if DEFAULT_BRANCH in ref:\n            sub_repo.update_ref(ref.replace(DEFAULT_BRANCH, \"other\"), ref)\n            sub_repo.call_git([\"update-ref\", \"-d\", ref])\n    sub_repo.update_ref(\n        \"HEAD\",\n        \"refs/heads/{}\".format(\n            \"adjusted/other(unlocked)\" if managed else \"other\"),\n        symbolic=True)\n    # END complicated handling for adjusted branches\n    ds_origin.save()\n    ds_origin_sub.repo.checkout(DEFAULT_BRANCH, options=[\"--orphan\"])\n\n    ds_cloned = clone(source=ds_origin.path, path=op.join(path, \"b\"))\n    ds_cloned_sub = ds_cloned.get(\n        \"sub\", result_xfm=\"datasets\", return_type=\"item-or-list\")\n\n    branch = ds_cloned_sub.repo.get_active_branch()\n    eq_(ds_cloned_sub.repo.get_corresponding_branch(branch) or branch,\n        \"other\")\n    # In the context of this test, the clone should be on an adjusted branch if\n    # the source landed there initially because we're on the same file system.\n    eq_(managed, ds_cloned_sub.repo.is_managed_branch())\n\n\n@skip_if_no_network\n@with_tempfile\ndef test_gin_cloning(path):\n    # can we clone a public ds anoynmously from gin and retrieve content\n    ds = clone('https://gin.g-node.org/datalad/datalad-ci-target', path)\n    ok_(ds.is_installed())\n    annex_path = op.join('annex', 'two')\n    git_path = op.join('git', 'one')\n    eq_(ds.repo.file_has_content(annex_path), False)\n    eq_(ds.repo.is_under_annex(git_path), False)\n    result = ds.get(annex_path)\n    assert_result_count(result, 1)\n    assert_status('ok', result)\n    eq_(result[0]['path'], op.join(ds.path, annex_path))\n    ok_file_has_content(op.join(ds.path, annex_path), 'two\\n')\n    ok_file_has_content(op.join(ds.path, git_path), 'one\\n')\n\n\n# TODO: git-annex-init fails in the second clone call below when this is\n# executed under ./tools/eval_under_testloopfs.\n@skip_if_adjusted_branch\n@with_tree(tree={\"special\": {\"f0\": \"0\"}})\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_fetch_git_special_remote(url_path, url, path):\n    url_path = Path(url_path)\n    path = Path(path)\n    ds_special = Dataset(url_path / \"special\").create(force=True)\n    ds_special.save()\n    ds_special.repo.call_git([\"update-server-info\"])\n\n    clone_url = url + \"special/.git\"\n    ds_a = clone(clone_url, path / \"a\")\n    ds_a.repo.call_annex(\n        [\"initremote\", \"special\", \"type=git\", \"autoenable=true\",\n         \"location=\" + clone_url])\n\n    # Set up a situation where a file is present only on the special remote,\n    # and its existence is known only to the special remote's git-annex branch.\n    (ds_special.pathobj / \"f1\").write_text(\"1\")\n    ds_special.save()\n    ds_special.repo.call_git([\"update-server-info\"])\n\n    ds_a.repo.fetch(DEFAULT_REMOTE)\n    ds_a.repo.merge(f\"{DEFAULT_REMOTE}/{DEFAULT_BRANCH}\")\n\n    ds_b = clone(ds_a.path, path / \"other\")\n    ds_b.get(\"f1\")\n    ok_(ds_b.repo.file_has_content(\"f1\"))\n\n\n@skip_if_adjusted_branch\n@skip_if_no_network\n@with_tempfile(mkdir=True)\ndef test_nonuniform_adjusted_subdataset(path):\n    # https://github.com/datalad/datalad/issues/5107\n    topds = Dataset(Path(path) / \"top\").create()\n    subds_url = 'git://github.com/datalad/testrepo--basic--r1'\n    topds.clone(\n        source='git://github.com/datalad/testrepo--basic--r1',\n        path='subds')\n    eq_(topds.subdatasets(return_type='item-or-list')['gitmodule_url'],\n        subds_url)\n\n\n@with_tempfile\ndef test_clone_recorded_subds_reset(path):\n    path = Path(path)\n    ds_a = create(path / \"ds_a\")\n    ds_a_sub = ds_a.create(\"sub\")\n    (ds_a_sub.pathobj / \"foo\").write_text(\"foo\")\n    ds_a.save(recursive=True)\n    (ds_a_sub.pathobj / \"bar\").write_text(\"bar\")\n    ds_a_sub.save()\n\n    ds_b = clone(ds_a.path, path / \"ds_b\")\n    ds_b.get(\"sub\")\n    assert_repo_status(ds_b.path)\n    sub_repo = Dataset(path / \"ds_b\" / \"sub\").repo\n    branch = sub_repo.get_active_branch()\n    eq_(ds_b.subdatasets()[0][\"gitshasum\"],\n        sub_repo.get_hexsha(\n            sub_repo.get_corresponding_branch(branch) or branch))\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_url_mapping(src_path, dest_path):\n    src = create(src_path)\n    dest = Dataset(dest_path)\n    # check that the impossible doesn't work\n    assert_raises(IncompleteResultsError, clone, 'rambo', dest_path)\n    # rather than adding test URL mapping here, consider\n    # test_url_mapping_specs(), it is cheaper there\n  \n    # anticipate windows test paths and escape them\n    escaped_subst = (r',rambo,%s' % src_path).replace('\\\\', '\\\\\\\\')\n    for specs in (\n            # we can clone with a simple substitution\n            {'datalad.clone.url-substitute.mike': escaped_subst},\n            # a prior match to a dysfunctional URL doesn't impact success\n            {\n                'datalad.clone.url-substitute.no': ',rambo,picknick',\n                'datalad.clone.url-substitute.mike': escaped_subst,\n            }):\n        try:\n            with patch.dict(dest.config._merged_store, specs):\n                clone('rambo', dest_path)\n        finally:\n            dest.remove(check=False)\n\n    # check submodule config impact\n    dest.create()\n    with patch.dict(dest.config._merged_store,\n                    {'datalad.clone.url-substitute.mike': escaped_subst}):\n        dest.clone('rambo', 'subds')\n    submod_rec = dest.repo.get_submodules()[0]\n    # we record the original-original URL\n    eq_(submod_rec['gitmodule_datalad-url'], 'rambo')\n    # and put the effective one as the primary URL\n    eq_(submod_rec['gitmodule_url'], src_path)\n\n\n_nomatch_map = {\n    'datalad.clone.url-substitute.nomatch': (\n        ',nomatch,NULL',\n    )\n}\n_windows_map = {\n    'datalad.clone.url-substitute.win': (\n        r',C:\\\\Users\\\\datalad\\\\from,D:\\\\to',\n    )\n}\n\n\ndef test_url_mapping_specs():\n    from datalad.core.distributed.clone import _map_urls\n    cfg = ConfigManager()\n    for m, i, o in (\n            # path redirect on windows\n            (_windows_map,\n             r'C:\\Users\\datalad\\from',\n             r'D:\\to'),\n            # test standard github mapping, no pathc needed\n            ({},\n             'https://github.com/datalad/testrepo_gh/sub _1',\n             'https://github.com/datalad/testrepo_gh-sub__1'),\n            # and on deep subdataset too\n            ({},\n             'https://github.com/datalad/testrepo_gh/sub _1/d/sub_-  1',\n             'https://github.com/datalad/testrepo_gh-sub__1-d-sub_-_1'),\n            # test that the presence of another mapping spec doesn't ruin\n            # the outcome\n            (_nomatch_map,\n             'https://github.com/datalad/testrepo_gh/sub _1',\n             'https://github.com/datalad/testrepo_gh-sub__1'),\n            # verify OSF mapping, but see\n            # https://github.com/datalad/datalad/issues/5769 for future\n            # implications\n            ({},\n             'https://osf.io/q8xnk/',\n             'osf://q8xnk'),\n            ):\n        with patch.dict(cfg._merged_store, m):\n            eq_(_map_urls(cfg, [i]), [o])\n",
            "header": {
                "file": {
                    "path": "datalad/core/distributed/tests/test_clone.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "1f2583c13",
                "new_commit_hash": "97b843e49",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/core/distributed/tests/test_clone.py"
            },
            "plus_file": {
                "path": "b/datalad/core/distributed/tests/test_clone.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 537,
                            "length": 24
                        },
                        "new_range": {
                            "start": 537,
                            "length": 41
                        },
                        "section": "def test_clone_autoenable_msg_handles_sameas(repo, clone_path):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_installationpath_from_url():",
                                "type": "context"
                            },
                            {
                                "content": "    cases = (",
                                "type": "deleted"
                            },
                            {
                                "content": "    # cases for all OSes",
                                "type": "added"
                            },
                            {
                                "content": "    cases = [",
                                "type": "added"
                            },
                            {
                                "content": "        'http://example.com/lastbit',",
                                "type": "context"
                            },
                            {
                                "content": "        'http://example.com/lastbit.git',",
                                "type": "context"
                            },
                            {
                                "content": "        'http://lastbit:8000',",
                                "type": "context"
                            },
                            {
                                "content": "    ) + (",
                                "type": "deleted"
                            },
                            {
                                "content": "        # SSH",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:lastbit',",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:lastbit/',",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:subd/lastbit',",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:/full/path/lastbit',",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:lastbit/.git',",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:lastbit/.git/',",
                                "type": "added"
                            },
                            {
                                "content": "        'hostname:/full/path/lastbit/.git',",
                                "type": "added"
                            },
                            {
                                "content": "        'full.hostname.com:lastbit/.git',",
                                "type": "added"
                            },
                            {
                                "content": "        'user@full.hostname.com:lastbit/.git',",
                                "type": "added"
                            },
                            {
                                "content": "        'ssh://user:passw@full.hostname.com/full/path/lastbit',",
                                "type": "added"
                            },
                            {
                                "content": "        'ssh://user:passw@full.hostname.com/full/path/lastbit/',",
                                "type": "added"
                            },
                            {
                                "content": "        'ssh://user:passw@full.hostname.com/full/path/lastbit/.git',",
                                "type": "added"
                            },
                            {
                                "content": "    ]",
                                "type": "added"
                            },
                            {
                                "content": "    # OS specific cases",
                                "type": "added"
                            },
                            {
                                "content": "    cases += [",
                                "type": "added"
                            },
                            {
                                "content": "        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',",
                                "type": "context"
                            },
                            {
                                "content": "        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',",
                                "type": "context"
                            },
                            {
                                "content": "        'Temp\\\\lastbit',",
                                "type": "context"
                            },
                            {
                                "content": "        'Temp\\\\lastbit\\\\',",
                                "type": "context"
                            },
                            {
                                "content": "        'lastbit.git',",
                                "type": "context"
                            },
                            {
                                "content": "        'lastbit.git\\\\',",
                                "type": "context"
                            },
                            {
                                "content": "    ) if on_windows else (",
                                "type": "deleted"
                            },
                            {
                                "content": "    ] if on_windows else [",
                                "type": "added"
                            },
                            {
                                "content": "        'lastbit',",
                                "type": "context"
                            },
                            {
                                "content": "        'lastbit/',",
                                "type": "context"
                            },
                            {
                                "content": "        '/lastbit',",
                                "type": "context"
                            },
                            {
                                "content": "        'lastbit.git',",
                                "type": "context"
                            },
                            {
                                "content": "        'lastbit.git/',",
                                "type": "context"
                            },
                            {
                                "content": "    )",
                                "type": "deleted"
                            },
                            {
                                "content": "    ]",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    for p in cases:",
                                "type": "context"
                            },
                            {
                                "content": "        eq_(_get_installationpath_from_url(p), 'lastbit')",
                                "type": "context"
                            },
                            {
                                "content": "    # we need to deal with quoted urls",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/core/distributed/tests/test_clone.py",
                            "type": "function",
                            "name": "test_installationpath_from_url",
                            "content": "def test_installationpath_from_url():\n    cases = (\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n    ) + (\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ) if on_windows else (\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    )\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 539,
                            "end_lineno": 572,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "b4fafb048489019fc165e4e1f8b69e5bb40df008^",
    "new_commit_hash": "b4fafb048489019fc165e4e1f8b69e5bb40df008",
    "commit_message": "Merge remote-tracking branch 'origin/maint'\n\n* origin/maint:\n  RF: use python -m nose (and coverage) instead of direct\n  RF: always combine/report coverage, submit still only for non-cron\n  ENH: provide coverage report after merging\n  BF: take path from SSHRI, test URLs not only on Windows",
    "commit_date": "2021-08-13T12:03:20-04:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "b4fafb048489019fc165e4e1f8b69e5bb40df008",
    "test_file_codes": [
        "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test clone action\n\n\"\"\"\n\nimport logging\nimport os.path as op\nimport stat\n\nfrom unittest.mock import patch\n\nfrom datalad.config import ConfigManager\nfrom datalad import consts\nfrom datalad.api import (\n    clone,\n    create,\n    remove,\n)\nfrom datalad.utils import (\n    chpwd,\n    get_home_envvars,\n    Path,\n    on_windows,\n    rmtree\n)\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.cmd import (\n    GitWitlessRunner,\n    WitlessRunner as Runner,\n)\nfrom datalad.tests.utils import (\n    assert_false,\n    assert_in,\n    assert_in_results,\n    assert_not_is_instance,\n    assert_message,\n    assert_not_in,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_result_values_equal,\n    assert_status,\n    create_tree,\n    DEFAULT_BRANCH,\n    DEFAULT_REMOTE,\n    eq_,\n    get_datasets_topdir,\n    integration,\n    known_failure,\n    known_failure_githubci_win,\n    known_failure_windows,\n    neq_,\n    nok_,\n    ok_,\n    ok_file_has_content,\n    has_symlink_capability,\n    ok_startswith,\n    patch_config,\n    set_date,\n    serve_path_via_http,\n    skip_if_adjusted_branch,\n    skip_if_no_network,\n    skip_if_on_windows,\n    skip_ssh,\n    slow,\n    swallow_logs,\n    use_cassette,\n    with_sameas_remote,\n    with_tempfile,\n    with_testrepos,\n    with_tree,\n    SkipTest,\n)\nfrom datalad.support.network import get_local_file_url\nfrom datalad.core.distributed.clone import (\n    _get_installationpath_from_url,\n    decode_source_spec,\n)\nfrom datalad.distribution.dataset import Dataset\n\n# this is the dataset ID of our test dataset in the main datalad RIA store\ndatalad_store_testds_id = '76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561'\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_args(path, otherpath, alienpath):\n    # source == path\n    assert_raises(ValueError, clone, 'Zoidberg', path='Zoidberg')\n    assert_raises(ValueError, clone, 'ssh://mars/Zoidberg', path='ssh://mars/Zoidberg')\n\n    # \"invalid URL\" is a valid filepath... and since no clone to remote\n    # is possible - we can just assume that it is the (legit) file path\n    # which is provided, not a URL.  So both below should fail as any\n    # other clone from a non-existing source and not for the reason of\n    # \"invalid something\".  Behavior is similar to how Git performs - can\n    # clone into a URL-like path.\n\n    # install to an \"invalid URL\" path\n    res = clone('Zoidberg', path='ssh://mars:Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # install to a \"remote location\" path\n    res = clone('Zoidberg', path='ssh://mars/Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # make fake dataset\n    ds = create(path)\n    assert_raises(IncompleteResultsError, ds.clone, '/higherup.', 'Zoidberg')\n    # make real dataset, try to install outside\n    ds_target = create(Path(otherpath) / 'target')\n    assert_raises(ValueError, ds_target.clone, ds.path, path=ds.path)\n    assert_status('error', ds_target.clone(ds.path, path=alienpath, on_failure='ignore'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_crcns(tdir, ds_path):\n    with chpwd(tdir):\n        res = clone('///', path=\"all-nonrecursive\", on_failure='ignore',\n                    result_xfm=None, return_type='list')\n        assert_status('ok', res)\n\n    # again, but into existing dataset:\n    ds = create(ds_path)\n    crcns = ds.clone(\"///crcns\", result_xfm='datasets', return_type='item-or-list')\n    ok_(crcns.is_installed())\n    eq_(crcns.pathobj, ds.pathobj / \"crcns\")\n    assert_in(crcns.path, ds.subdatasets(result_xfm='paths'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tree(tree={'sub': {}})\ndef test_clone_datasets_root(tdir):\n    tdir = Path(tdir)\n    with chpwd(tdir):\n        ds = clone(\"///\")\n        ok_(ds.is_installed())\n        eq_(ds.pathobj, tdir / get_datasets_topdir())\n\n        # do it a second time:\n        res = clone(\"///\", on_failure='ignore', result_xfm=None, return_type='list')\n        assert_message(\n            \"dataset %s was already cloned from '%s'\",\n            res)\n        assert_status('notneeded', res)\n\n        # and a third time into an existing something, that is not a dataset:\n        (tdir / 'sub' / 'a_file.txt').write_text(\"something\")\n\n        res = clone('///', path=\"sub\", on_failure='ignore')\n        assert_message(\n            'target path already exists and not empty, refuse to clone into target path',\n            res)\n        assert_status('error', res)\n\n\n@with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n@with_tempfile(mkdir=True)\ndef test_clone_simple_local(src, path):\n    origin = Dataset(path)\n\n    # now install it somewhere else\n    ds = clone(src, path, description='mydummy',\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    ok_(ds.is_installed())\n    if not isinstance(origin.repo, AnnexRepo):\n        # this means it is a GitRepo\n        ok_(isinstance(origin.repo, GitRepo))\n        # stays plain Git repo\n        ok_(isinstance(ds.repo, GitRepo))\n        ok_(not isinstance(ds.repo, AnnexRepo))\n        ok_(GitRepo.is_valid_repo(ds.path))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt'})\n        assert_repo_status(path, annex=False)\n    else:\n        # must be an annex\n        ok_(isinstance(ds.repo, AnnexRepo))\n        ok_(AnnexRepo.is_valid_repo(ds.path, allow_noninitialized=False))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt', 'test-annex.dat'})\n        assert_repo_status(path, annex=True)\n        # no content was installed:\n        ok_(not ds.repo.file_has_content('test-annex.dat'))\n        uuid_before = ds.repo.uuid\n        ok_(uuid_before)  # make sure we actually have an uuid\n        eq_(ds.repo.get_description(), 'mydummy')\n\n    # installing it again, shouldn't matter:\n    res = clone(src, path, result_xfm=None, return_type='list')\n    assert_result_values_equal(res, 'source_url', [src])\n    assert_status('notneeded', res)\n    assert_message(\"dataset %s was already cloned from '%s'\", res)\n    ok_(ds.is_installed())\n    if isinstance(origin.repo, AnnexRepo):\n        eq_(uuid_before, ds.repo.uuid)\n\n\n\n# AssertionError: unexpected content of state \"deleted\": [WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/datalad_temp_gzegy3hf/testrepo--basic--r1/test-annex.dat')] != []\n@known_failure_githubci_win\n@with_testrepos(flavors=['local-url', 'network', 'local'])\n@with_tempfile\ndef test_clone_dataset_from_just_source(url, path):\n    with chpwd(path, mkdir=True):\n        ds = clone(url, result_xfm='datasets', return_type='item-or-list')\n\n    ok_startswith(ds.path, path)\n    ok_(ds.is_installed())\n    ok_(GitRepo.is_valid_repo(ds.path))\n    assert_repo_status(ds.path, annex=None)\n    assert_in('INFO.txt', ds.repo.get_indexed_files())\n\n\n# test fails randomly, likely a bug in one of the employed test helpers\n# https://github.com/datalad/datalad/pull/3966#issuecomment-571267932\n@known_failure\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_clone_dataladri(src, topurl, path):\n    # make plain git repo\n    ds_path = Path(src) / 'ds'\n    gr = GitRepo(ds_path, create=True)\n    gr.add('test.txt')\n    gr.commit('demo')\n    Runner(cwd=gr.path).run(['git', 'update-server-info'])\n    # now install it somewhere else\n    with patch('datalad.consts.DATASETS_TOPURL', topurl):\n        ds = clone('///ds', path, result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    assert_repo_status(path, annex=False)\n    ok_file_has_content(ds.pathobj / 'test.txt', 'some')\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_isnot_recursive(path_src, path_nr, path_r):\n    src = Dataset(path_src).create()\n    src.create('subm 1')\n    src.create('2')\n\n    ds = clone(src, path_nr, result_xfm='datasets', return_type='item-or-list')\n    ok_(ds.is_installed())\n    # check nothing is unintentionally installed\n    subdss = ds.subdatasets(recursive=True)\n    assert_result_count(subdss, len(subdss), state='absent')\n    # this also means, subdatasets to be listed as not fulfilled:\n    eq_(set(ds.subdatasets(recursive=True, fulfilled=False, result_xfm='relpaths')),\n        {'subm 1', '2'})\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_into_dataset(source_path, top_path):\n    source = Dataset(source_path).create()\n    ds = create(top_path)\n    assert_repo_status(ds.path)\n    # Note, we test against the produced history in DEFAULT_BRANCH, not what it\n    # turns into in an adjusted branch!\n    hexsha_before = ds.repo.get_hexsha(DEFAULT_BRANCH)\n    subds = ds.clone(source, \"sub\",\n                     result_xfm='datasets', return_type='item-or-list')\n    ok_((subds.pathobj / '.git').is_dir())\n    ok_(subds.is_installed())\n    assert_in('sub', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # sub is clean:\n    assert_repo_status(subds.path, annex=None)\n    # top is clean:\n    assert_repo_status(ds.path, annex=None)\n    # source is recorded in .gitmodules:\n    sds = ds.subdatasets(\"sub\")\n    assert_result_count(sds, 1, action='subdataset')\n    eq_(sds[0]['gitmodule_datalad-url'], source.path)\n    # Clone produced one commit including the addition to .gitmodule:\n    commits = list(ds.repo.get_branch_commits_(\n        branch=DEFAULT_BRANCH,\n        stop=hexsha_before\n    ))\n    assert_not_in(hexsha_before, commits)\n    eq_(len(commits), 1)\n\n    # but we could also save while installing and there should be no side-effect\n    # of saving any other changes if we state to not auto-save changes\n    # Create a dummy change\n    create_tree(ds.path, {'dummy.txt': 'buga'})\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n    subds_ = ds.clone(source, \"sub2\",\n                      result_xfm='datasets', return_type='item-or-list')\n    eq_(subds_.pathobj, ds.pathobj / \"sub2\")  # for paranoid yoh ;)\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_notclone_known_subdataset(src_path, path):\n    src = Dataset(src_path).create()\n    sub = src.create('subm 1')\n    sub_id = sub.id\n    # get the superdataset:\n    ds = clone(src, path,\n               result_xfm='datasets', return_type='item-or-list')\n\n    # subdataset not installed:\n    subds = Dataset(ds.pathobj / 'subm 1')\n    assert_false(subds.is_installed())\n    assert_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # clone is not meaningful\n    res = ds.clone('subm 1', on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n    # get does the job\n    res = ds.get(path='subm 1', get_data=False)\n    assert_status('ok', res)\n    ok_(subds.is_installed())\n    ok_(AnnexRepo.is_valid_repo(subds.path, allow_noninitialized=False))\n    # Verify that it is the correct submodule installed and not\n    # new repository initiated\n    eq_(subds.id, sub_id)\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n\n\n@with_tempfile(mkdir=True)\ndef test_failed_clone(dspath):\n    ds = create(dspath)\n    res = ds.clone(\"http://nonexistingreallyanything.datalad.org/bla\", \"sub\",\n                   on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n\n\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@with_tempfile\ndef test_clone_missing_commit(source, clone):\n\n    from datalad.core.distributed.clone import clone_dataset\n\n    source = Path(source)\n    clone = Path(clone)\n\n    # Commit SHA from another repository - should never be recreated in a fresh\n    # dataset:\n    commit_sha = \"c29691b37b05b78ffa76e5fdf0044e9df673e8f1\"\n\n    origin = Dataset(source).create(force=True)\n    origin.save()\n\n    # clone origin but request commit_sha to be checked out:\n\n    results = [x for x in\n               clone_dataset(srcs=[source], destds=Dataset(clone),\n                             checkout_gitsha=commit_sha)\n               ]\n    # expected error result:\n    assert_result_count(results, 1)\n    assert_in_results(results, status='error', action='install',\n                      path=str(clone), type='dataset')\n    assert_in(\"Target commit c29691b3 does not exist in the clone\",\n              results[0]['message'])\n    # failed attempt was removed:\n    assert_false(clone.exists())\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef check_reckless(annex, src_path, top_path, sharedpath):\n    # super with or without annex\n    src = Dataset(src_path).create(annex=annex)\n    # sub always with annex\n    srcsub = src.create('sub')\n\n    # and for the actual test\n    ds = clone(src.path, top_path, reckless=True,\n               result_xfm='datasets', return_type='item-or-list')\n\n    is_crippled = srcsub.repo.is_managed_branch()\n\n    if annex and not is_crippled:\n        eq_(ds.config.get('annex.hardlink', None), 'true')\n\n    # actual value is 'auto', because True is a legacy value and we map it\n    eq_(ds.config.get('datalad.clone.reckless', None), 'auto')\n    if annex:\n        eq_(ds.repo.repo_info()['untrusted repositories'][0]['here'], True)\n    # now, if we clone another repo into this one, it will inherit the setting\n    # without having to provide it explicitly\n    newsub = ds.clone(srcsub, 'newsub', result_xfm='datasets', return_type='item-or-list')\n    # and `get` the original subdataset\n    origsub = ds.get('sub', result_xfm='datasets', return_type='item-or-list')\n    for sds in (newsub, origsub):\n        eq_(sds.config.get('datalad.clone.reckless', None), 'auto')\n        if not is_crippled:\n            eq_(sds.config.get('annex.hardlink', None), 'true')\n\n    if is_crippled:\n        raise SkipTest(\"Remainder of test needs proper filesystem permissions\")\n\n    if annex:\n        if ds.repo.git_annex_version < \"8.20200908\":\n            # TODO: Drop when GIT_ANNEX_MIN_VERSION is at least 8.20200908.\n\n            # the standard setup keeps the annex locks accessible to the user only\n            nok_((ds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n                 & stat.S_IWGRP)\n        else:\n            # umask might be such (e.g. 002) that group write permissions are inherited, so\n            # for the next test we should check if that is the case on some sample file\n            dltmp_path = ds.pathobj / '.git' / \"dltmp\"\n            dltmp_path.write_text('')\n            default_grp_write_perms = dltmp_path.stat().st_mode & stat.S_IWGRP\n            dltmp_path.unlink()\n            # the standard setup keeps the annex locks following umask inheritance\n            eq_((ds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n                 & stat.S_IWGRP, default_grp_write_perms)\n\n        # but we can set it up for group-shared access too\n        sharedds = clone(\n            src, sharedpath,\n            reckless='shared-group',\n            result_xfm='datasets',\n            return_type='item-or-list')\n        ok_((sharedds.pathobj / '.git' / 'annex' / 'index.lck').stat().st_mode \\\n            & stat.S_IWGRP)\n\n\ndef test_reckless():\n    yield check_reckless, True\n    yield check_reckless, False\n\n@with_tempfile\n@with_tempfile\ndef test_install_source_relpath(src, dest):\n    src = Path(src)\n    create(src)\n    src_ = src.name\n    with chpwd(src.parent):\n        clone(src_, dest)\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_isnt_a_smartass(origin_path, path):\n    origin = create(origin_path)\n    cloned = clone(origin, path,\n                   result_xfm='datasets', return_type='item-or-list')\n    with chpwd(path):\n        # no were are inside a dataset clone, and we make another one\n        # we do not want automatic subdatasetification without given a dataset\n        # explicitly\n        clonedsub = clone(origin, 'testsub',\n                          result_xfm='datasets', return_type='item-or-list')\n    # correct destination\n    assert clonedsub.path.startswith(path)\n    # no subdataset relation\n    eq_(cloned.subdatasets(), [])\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_report_permission_issue(tdir):\n    pdir = Path(tdir) / 'protected'\n    pdir.mkdir()\n    # make it read-only\n    pdir.chmod(0o555)\n    with chpwd(pdir):\n        # first check the premise of the test. If we can write (strangely\n        # mounted/crippled file system, subsequent assumptions are violated\n        # and we can stop\n        probe = Path('probe')\n        try:\n            probe.write_text('should not work')\n            raise SkipTest\n        except PermissionError:\n            # we are indeed in a read-only situation\n            pass\n        res = clone('///', result_xfm=None, return_type='list', on_failure='ignore')\n        assert_status('error', res)\n        assert_result_count(\n            res, 1, status='error',\n            message=\"could not create work tree dir '%s/%s': Permission denied\"\n                    % (pdir, get_datasets_topdir())\n        )\n\n\n@skip_if_no_network\n@with_tempfile\ndef test_autoenabled_remote_msg(path):\n    # Verify that no message about a remote not been enabled is displayed\n    # whenever the remote we clone is the  type=git special remote, so the name\n    # of the remote might not match\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone('///repronim/containers', path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_not_in(\"not auto-enabled\", cml.out)\n\n\n@with_sameas_remote(autoenabled=True)\n@with_tempfile(mkdir=True)\ndef test_clone_autoenable_msg_handles_sameas(repo, clone_path):\n    ds = Dataset(repo.path)\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone(ds, clone_path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_in(\"r_dir\", cml.out)\n        assert_in(\"not auto-enabled\", cml.out)\n        # The rsyncurl remote was enabled.\n        assert_not_in(\"r_rsync\", cml.out)\n    ds_cloned = Dataset(clone_path)\n    remotes = ds_cloned.repo.get_remotes()\n    assert_in(\"r_rsync\", remotes)\n    assert_not_in(\"r_dir\", remotes)\n\n\ndef test_installationpath_from_url():\n    # cases for all OSes\n    cases = [\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n        # SSH\n        'hostname:lastbit',\n        'hostname:lastbit/',\n        'hostname:subd/lastbit',\n        'hostname:/full/path/lastbit',\n        'hostname:lastbit/.git',\n        'hostname:lastbit/.git/',\n        'hostname:/full/path/lastbit/.git',\n        'full.hostname.com:lastbit/.git',\n        'user@full.hostname.com:lastbit/.git',\n        'ssh://user:passw@full.hostname.com/full/path/lastbit',\n        'ssh://user:passw@full.hostname.com/full/path/lastbit/',\n        'ssh://user:passw@full.hostname.com/full/path/lastbit/.git',\n    ]\n    # OS specific cases\n    cases += [\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ] if on_windows else [\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    ]\n\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')\n\n\n# https://github.com/datalad/datalad/issues/3958\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_expanduser(srcpath, destpath):\n    src = Dataset(Path(srcpath) / 'src').create()\n    dest = Dataset(Path(destpath) / 'dest').create()\n\n    # We switch away from home set up in datalad.setup_package(), so make sure\n    # we have a valid identity.\n    with open(op.join(srcpath, \".gitconfig\"), \"w\") as fh:\n        fh.write(\"[user]\\n\"\n                 \"name = DataLad oooooTester\\n\"\n                 \"email = test@example.com\\n\")\n\n    with chpwd(destpath), patch.dict('os.environ', get_home_envvars(srcpath)):\n        res = clone(op.join('~', 'src'), 'dest', result_xfm=None, return_type='list',\n                    on_failure='ignore')\n        assert_result_count(res, 1)\n        assert_result_count(\n            res, 1, action='install', status='error', path=dest.path,\n            message='target path already exists and not empty, refuse to '\n            'clone into target path')\n        # wipe out destination, and try again\n        assert_status('ok', remove(dataset=dest, check=False))\n        # now it should do it, and clone the right one\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n        eq_(src.id, cloneds.id)\n        # and it shouldn't fail when doing it again, because it detects\n        # the re-clone\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n\n\n@with_tempfile(mkdir=True)\ndef test_cfg_originorigin(path):\n    path = Path(path)\n    origin = Dataset(path / 'origin').create()\n    (origin.pathobj / 'file1.txt').write_text('content')\n    origin.save()\n    clone_lev1 = clone(origin, path / 'clone_lev1')\n    clone_lev2 = clone(clone_lev1, path / 'clone_lev2')\n    # the goal is to be able to get file content from origin without\n    # the need to configure it manually\n    assert_result_count(\n        clone_lev2.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_lev2.pathobj / 'file1.txt'),\n    )\n    eq_((clone_lev2.pathobj / 'file1.txt').read_text(), 'content')\n    eq_(\n        Path(clone_lev2.siblings(\n            'query',\n            name=DEFAULT_REMOTE + '-2',\n            return_type='item-or-list')['url']),\n        origin.pathobj\n    )\n\n    # Clone another level, this time with a relative path. Drop content from\n    # lev2 so that origin is the only place that the file is available from.\n    clone_lev2.drop(\"file1.txt\")\n    with chpwd(path), swallow_logs(new_level=logging.DEBUG) as cml:\n        clone_lev3 = clone('clone_lev2', 'clone_lev3')\n        # we called git-annex-init; see gh-4367:\n        cml.assert_logged(msg=r\"[^[]*Run \\[('git',.*'annex'|'git-annex'), 'init'\",\n                          match=False,\n                          level='DEBUG')\n    assert_result_count(\n        clone_lev3.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_lev3.pathobj / 'file1.txt'))\n\n\n# test fix for gh-2601/gh-3538\n@known_failure\n@with_tempfile()\ndef test_relative_submodule_url(path):\n    Dataset(op.join(path, 'origin')).create()\n    ds = Dataset(op.join(path, 'ds')).create()\n    with chpwd(ds.path):\n        ds_cloned = ds.clone(\n            source=op.join(op.pardir, 'origin'),\n            path='sources')\n\n    # Check that a simple fetch call does not fail.\n    ds_cloned.repo.fetch()\n\n    subinfo = ds.subdatasets(return_type='item-or-list')\n    eq_(subinfo['gitmodule_url'],\n        # must be a relative URL, not platform-specific relpath!\n        '../../origin')\n\n\n@with_tree(tree={\"subdir\": {}})\n@with_tempfile(mkdir=True)\ndef test_local_url_with_fetch(path, path_other):\n    path = Path(path)\n    path_other = Path(path_other)\n    Dataset(path / \"source\").create()\n\n    for where, source, path in [\n            (path, \"source\", \"a\"),\n            (path / \"subdir\", op.join(op.pardir, \"source\"), \"a\"),\n            (path, \"source\", path_other / \"a\")]:\n        with chpwd(where):\n            ds_cloned = clone(source=source, path=path)\n            # Perform a fetch to check that the URL points to a valid location.\n            ds_cloned.repo.fetch()\n\n\ndef test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}/{}/{}'.format(\n                    proto,\n                    loc,\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')\n\n    # literal dataset name/location\n    eq_(decode_source_spec('ria+http://example.com#~rootds'),\n        {'source': 'ria+http://example.com#~rootds',\n         'version': None, 'type': 'ria',\n         'giturl': 'http://example.com/alias/rootds',\n         'default_destpath': 'rootds'})\n    # version etc still works\n    eq_(decode_source_spec('ria+http://example.com#~rootds@specialbranch'),\n        {'source': 'ria+http://example.com#~rootds@specialbranch',\n         'version': 'specialbranch', 'type': 'ria',\n         'giturl': 'http://example.com/alias/rootds',\n         'default_destpath': 'rootds'})\n\n\ndef _move2store(storepath, d):\n    # make a bare clone of it into a local that matches the organization\n    # of a ria dataset store\n    store_loc = str(storepath / d.id[:3] / d.id[3:])\n    d.repo.call_git(['clone', '--bare', d.path, store_loc])\n    d.siblings('configure', name='store', url=str(store_loc),\n               result_renderer='disabled')\n    Runner(cwd=store_loc).run(['git', 'update-server-info'])\n\n\n@slow  # 12sec on Yarik's laptop\n@with_tree(tree={\n    'ds': {\n        'test.txt': 'some',\n        'subdir': {\n            'subds': {'testsub.txt': 'somemore'},\n        },\n    },\n})\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_ria_http(lcl, storepath, url):\n    # create a local dataset with a subdataset\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    subds = Dataset(lcl / 'ds' / 'subdir' / 'subds').create(force=True)\n    subds.save()\n    ds = Dataset(lcl / 'ds').create(force=True)\n    ds.save(version_tag='original')\n    assert_repo_status(ds.path)\n    for d in (ds, subds):\n        _move2store(storepath, d)\n    # location of superds in store\n    storeds_loc = str(storepath / ds.id[:3] / ds.id[3:])\n    # now we should be able to clone from a ria+http url\n    # the super\n    riaclone = clone(\n        'ria+{}#{}'.format(url, ds.id),\n        lcl / 'clone',\n    )\n\n    # due to default configuration, clone() should automatically look for the\n    # subdataset in the store, too -- if not the following would fail, because\n    # we never configured a proper submodule URL\n    riaclonesub = riaclone.get(\n        op.join('subdir', 'subds'), get_data=False,\n        result_xfm='datasets', return_type='item-or-list')\n\n    # both datasets came from the store and must be set up in an identical\n    # fashion\n    for origds, cloneds in ((ds, riaclone), (subds, riaclonesub)):\n        eq_(origds.id, cloneds.id)\n        if not ds.repo.is_managed_branch():\n            # test logic cannot handle adjusted branches\n            eq_(origds.repo.get_hexsha(), cloneds.repo.get_hexsha())\n        ok_(cloneds.config.get(f'remote.{DEFAULT_REMOTE}.url').startswith(url))\n        eq_(cloneds.config.get(f'remote.{DEFAULT_REMOTE}.annex-ignore'), 'true')\n        eq_(cloneds.config.get('datalad.get.subdataset-source-candidate-200origin'),\n            'ria+%s#{id}' % url)\n\n    # now advance the source dataset\n    (ds.pathobj / 'newfile.txt').write_text('new')\n    ds.save()\n    ds.publish(to='store')\n    Runner(cwd=storeds_loc).run(['git', 'update-server-info'])\n    # re-clone as before\n    riaclone2 = clone(\n        'ria+{}#{}'.format(url, ds.id),\n        lcl / 'clone2',\n    )\n    # and now clone a specific version, here given be the tag name\n    riaclone_orig = clone(\n        'ria+{}#{}@{}'.format(url, ds.id, 'original'),\n        lcl / 'clone_orig',\n    )\n    if not ds.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        # we got the precise version we wanted\n        eq_(riaclone.repo.get_hexsha(), riaclone_orig.repo.get_hexsha())\n        # and not the latest\n        eq_(riaclone2.repo.get_hexsha(), ds.repo.get_hexsha())\n        neq_(riaclone2.repo.get_hexsha(), riaclone_orig.repo.get_hexsha())\n\n    # attempt to clone a version that doesn't exist\n    with swallow_logs():\n        with assert_raises(IncompleteResultsError) as cme:\n            clone('ria+{}#{}@impossible'.format(url, ds.id),\n                  lcl / 'clone_failed')\n        assert_in(\"not found in upstream\", str(cme.exception))\n\n    # lastly test if URL rewriting is in effect\n    # on the surface we clone from an SSH source identified by some custom\n    # label, no full URL, but URL rewriting setup maps it back to the\n    # HTTP URL used above\n    with patch_config({\n            'url.ria+{}#.insteadof'.format(url): 'ria+ssh://somelabel#'}):\n        cloned_by_label = clone(\n            'ria+ssh://somelabel#{}'.format(origds.id),\n            lcl / 'cloned_by_label',\n        )\n    # so we get the same setup as above, but....\n    eq_(origds.id, cloned_by_label.id)\n    if not ds.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        eq_(origds.repo.get_hexsha(), cloned_by_label.repo.get_hexsha())\n    ok_(cloned_by_label.config.get(\n        f'remote.{DEFAULT_REMOTE}.url').startswith(url))\n    eq_(cloned_by_label.config.get(f'remote.{DEFAULT_REMOTE}.annex-ignore'),\n        'true')\n    # ... the clone candidates go with the label-based URL such that\n    # future get() requests acknowledge a (system-wide) configuration\n    # update\n    eq_(cloned_by_label.config.get('datalad.get.subdataset-source-candidate-200origin'),\n        'ria+ssh://somelabel#{id}')\n\n    if not has_symlink_capability():\n        return\n    # place a symlink in the store to serve as a dataset alias\n    (storepath / 'alias').mkdir()\n    (storepath / 'alias' / 'myname').symlink_to(storeds_loc)\n    with chpwd(lcl):\n        cloned_by_alias = clone('ria+{}#~{}'.format(url, 'myname'))\n    # still get the same data\n    eq_(cloned_by_alias.id, ds.id)\n    # more sensible default install path\n    eq_(cloned_by_alias.pathobj.name, 'myname')\n\n\n@with_tempfile\n@with_tempfile\ndef _test_ria_postclonecfg(url, dsid, clone_path, superds):\n    # Test cloning from RIA store while ORA special remote autoenabling failed\n    # due to an invalid URL from the POV of the cloner.\n    # Origin's git-config-file should contain the UUID to enable. This needs to\n    # work via HTTP, SSH and local cloning.\n\n    # Autoenabling should fail initially by git-annex-init and we would report\n    # on INFO level. Only postclone routine would deal with it.\n    with swallow_logs(new_level=logging.INFO) as cml:\n        # First, the super ds:\n        riaclone = clone('ria+{}#{}'.format(url, dsid), clone_path)\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n\n    # However, we now can retrieve content since clone should have enabled the\n    # special remote with new URL (or origin in case of HTTP).\n    res = riaclone.get('test.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(riaclone.pathobj / 'test.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n    # Second ORA remote is enabled and not reconfigured:\n    untouched_remote = riaclone.siblings(name='anotherstore-storage',\n                                         return_type='item-or-list')\n    assert_not_is_instance(untouched_remote, list)\n    untouched_url = riaclone.repo.get_special_remotes()[\n        untouched_remote['annex-uuid']]['url']\n    ok_(untouched_url.startswith(\"ria+file://\"))\n    ok_(not untouched_url.startswith(\"ria+{}\".format(url)))\n\n    # publication dependency was set for store-storage but not for\n    # anotherstore-storage:\n    eq_(riaclone.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                            get_all=True),\n        \"store-storage\")\n\n    # same thing for the sub ds (we don't need a store-url and id - get should\n    # figure those itself):\n    with swallow_logs(new_level=logging.INFO) as cml:\n        riaclonesub = riaclone.get(\n            op.join('subdir', 'subds'), get_data=False,\n            result_xfm='datasets', return_type='item-or-list')\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n    res = riaclonesub.get('testsub.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(riaclonesub.pathobj / 'testsub.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n    # publication dependency was set for store-storage but not for\n    # anotherstore-storage:\n    eq_(riaclonesub.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                               get_all=True),\n        \"store-storage\")\n\n    # finally get the plain git subdataset.\n    # Clone should figure to also clone it from a ria+ URL\n    # (subdataset-source-candidate), notice that there wasn't an autoenabled ORA\n    # remote, but shouldn't stumble upon it, since it's a plain git.\n    res = riaclone.get(op.join('subdir', 'subgit', 'testgit.txt'))\n    assert_result_count(res, 1, status='ok', type='dataset', action='install')\n    assert_result_count(res, 1, status='notneeded', type='file')\n    assert_result_count(res, 2)\n    # no ORA remote, no publication dependency:\n    riaclonesubgit = Dataset(riaclone.pathobj / 'subdir' / 'subgit')\n    eq_(riaclonesubgit.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                                  get_all=True),\n        None)\n\n    # Now, test that if cloning into a dataset, ria-URL is preserved and\n    # post-clone configuration is triggered again, when we remove the subds and\n    # retrieve it again via `get`:\n    ds = Dataset(superds).create()\n    ria_url = 'ria+{}#{}'.format(url, dsid)\n    ds.clone(ria_url, 'sub')\n    sds = ds.subdatasets('sub')\n    eq_(len(sds), 1)\n    eq_(sds[0]['gitmodule_datalad-url'], ria_url)\n    assert_repo_status(ds.path)\n    ds.uninstall('sub', check=False)\n    assert_repo_status(ds.path)\n\n    # .gitmodules still there:\n    sds = ds.subdatasets('sub')\n    eq_(len(sds), 1)\n    eq_(sds[0]['gitmodule_datalad-url'], ria_url)\n    # get it again:\n\n    # Autoenabling should fail initially by git-annex-init and we would report\n    # on INFO level. Only postclone routine would deal with it.\n    with swallow_logs(new_level=logging.INFO) as cml:\n        ds.get('sub', get_data=False)\n        cml.assert_logged(msg=\"access to 1 dataset sibling store-storage not \"\n                              \"auto-enabled\",\n                          level=\"INFO\",\n                          regex=False)\n\n    subds = Dataset(ds.pathobj / 'sub')\n    # special remote is fine:\n    res = subds.get('test.txt')\n    assert_result_count(res, 1,\n                        status='ok',\n                        path=str(subds.pathobj / 'test.txt'),\n                        message=\"from {}...\".format(DEFAULT_REMOTE\n                                                    if url.startswith('http')\n                                                    else \"store-storage\"))\n\n\n@with_tempfile\ndef _postclonetest_prepare(lcl, storepath, storepath2, link):\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n        create_ds_in_store,\n        get_layout_locations\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n    create_tree(lcl,\n                tree={\n                        'ds': {\n                            'test.txt': 'some',\n                            'subdir': {\n                                'subds': {'testsub.txt': 'somemore'},\n                                'subgit': {'testgit.txt': 'even more'}\n                            },\n                        },\n                      })\n\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    storepath2 = Path(storepath2)\n    link = Path(link)\n    link.symlink_to(storepath)\n\n    # create a local dataset with a subdataset\n    subds = Dataset(lcl / 'ds' / 'subdir' / 'subds').create(force=True)\n    subds.save()\n    # add a plain git dataset as well\n    subgit = Dataset(lcl / 'ds' / 'subdir' / 'subgit').create(force=True,\n                                                              annex=False)\n    subgit.save()\n    ds = Dataset(lcl / 'ds').create(force=True)\n    ds.save(version_tag='original')\n    assert_repo_status(ds.path)\n\n    io = LocalIO()\n\n    # Have a second store with valid ORA remote. This should not interfere with\n    # reconfiguration of the first one, when that second store is not the one we\n    # clone from. However, don't push data into it for easier get-based testing\n    # later on.\n    # Doing this first, so datasets in \"first\"/primary store know about this.\n    create_store(io, storepath2, '1')\n    url2 = \"ria+{}\".format(get_local_file_url(str(storepath2)))\n    for d in (ds, subds, subgit):\n        create_ds_in_store(io, storepath2, d.id, '2', '1')\n        d.create_sibling_ria(url2, \"anotherstore\")\n        d.push('.', to='anotherstore', data='nothing')\n        store2_loc, _, _ = get_layout_locations(1, storepath2, d.id)\n        Runner(cwd=str(store2_loc)).run(['git', 'update-server-info'])\n\n    # Now the store to clone from:\n    create_store(io, storepath, '1')\n\n    # URL to use for upload. Point is, that this should be invalid for the clone\n    # so that autoenable would fail. Therefore let it be based on a to be\n    # deleted symlink\n    upl_url = \"ria+{}\".format(get_local_file_url(str(link)))\n\n    for d in (ds, subds, subgit):\n\n        # TODO: create-sibling-ria required for config! => adapt to RF'd\n        #       creation (missed on rebase?)\n        create_ds_in_store(io, storepath, d.id, '2', '1')\n        d.create_sibling_ria(upl_url, \"store\")\n\n        if d is not subgit:\n            # Now, simulate the problem by reconfiguring the special remote to\n            # not be autoenabled.\n            # Note, however, that the actual intention is a URL, that isn't\n            # valid from the point of view of the clone (doesn't resolve, no\n            # credentials, etc.) and therefore autoenabling on git-annex-init\n            # when datalad-cloning would fail to succeed.\n            Runner(cwd=d.path).run(['git', 'annex', 'enableremote',\n                                    'store-storage',\n                                    'autoenable=false'])\n        d.push('.', to='store')\n        store_loc, _, _ = get_layout_locations(1, storepath, d.id)\n        Runner(cwd=str(store_loc)).run(['git', 'update-server-info'])\n\n    link.unlink()\n    # We should now have a store with datasets that have an autoenabled ORA\n    # remote relying on an inaccessible URL.\n    # datalad-clone is supposed to reconfigure based on the URL we cloned from.\n    # Test this feature for cloning via HTTP, SSH and FILE URLs.\n\n    return ds.id\n\n\n@known_failure_windows  # https://github.com/datalad/datalad/issues/5134\n@slow  # 14 sec on travis\ndef test_ria_postclonecfg():\n\n    if not has_symlink_capability():\n        # This is needed to create an ORA remote using an URL for upload,\n        # that is then invalidated later on (delete the symlink it's based on).\n        raise SkipTest(\"Can't create symlinks\")\n\n    from datalad.utils import make_tempfile\n\n    with make_tempfile(mkdir=True) as lcl, make_tempfile(mkdir=True) as store, \\\n            make_tempfile(mkdir=True) as store2:\n        id = _postclonetest_prepare(lcl, store, store2)\n\n        # test cloning via ria+file://\n        yield _test_ria_postclonecfg, \\\n              get_local_file_url(store, compatibility='git'), id\n\n        # Note: HTTP disabled for now. Requires proper implementation in ORA\n        #       remote. See\n        # https://github.com/datalad/datalad/pull/4203#discussion_r410284649\n\n        # # test cloning via ria+http://\n        # with HTTPPath(store) as url:\n        #     yield _test_ria_postclonecfg, url, id\n\n        # test cloning via ria+ssh://\n        yield skip_ssh(_test_ria_postclonecfg), \\\n            \"ssh://datalad-test:{}\".format(Path(store).as_posix()), id\n\n\n@known_failure_windows\n@skip_ssh\n@with_tree(tree={'somefile.txt': 'some content'})\n@with_tempfile\n@with_tempfile\ndef test_no_ria_postclonecfg(dspath, storepath, clonepath):\n\n    dspath = Path(dspath)\n    storepath = Path(storepath)\n    clonepath = Path(clonepath)\n\n    # Test that particular configuration(s) do NOT lead to a reconfiguration\n    # upon clone. (See gh-5628)\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n    ds = Dataset(dspath).create(force=True)\n    ds.save()\n    assert_repo_status(ds.path)\n\n    io = LocalIO()\n    create_store(io, storepath, '1')\n    file_url = \"ria+{}\".format(get_local_file_url(str(storepath)))\n    ssh_url = \"ria+ssh://datalad-test:{}\".format(storepath.as_posix())\n    ds.create_sibling_ria(file_url, \"teststore\",\n                          push_url=ssh_url, alias=\"testds\")\n    ds.push('.', to='teststore')\n\n    # Now clone via SSH. Should not reconfigure although `url` doesn't match the\n    # URL we cloned from. However, `push-url` does.\n    riaclone = clone('{}#{}'.format(ssh_url, ds.id), clonepath)\n\n    # ORA remote is enabled (since URL still valid) but not reconfigured:\n    untouched_remote = riaclone.siblings(name='teststore-storage',\n                                         return_type='item-or-list')\n    assert_not_is_instance(untouched_remote, list)\n    ora_cfg = riaclone.repo.get_special_remotes()[\n        untouched_remote['annex-uuid']]\n    ok_(ora_cfg['url'] == file_url)\n    ok_(ora_cfg['push-url'] == ssh_url)\n\n    # publication dependency was still set (and it's the only one that was set):\n    eq_(riaclone.config.get(f\"remote.{DEFAULT_REMOTE}.datalad-publish-depends\",\n                            get_all=True),\n        \"teststore-storage\")\n\n    # we can still get the content\n    ds.get(\"somefile.txt\")\n\n\n# fatal: Could not read from remote repository.\n@known_failure_githubci_win  # in datalad/git-annex as e.g. of 20201218\n@with_tempfile(mkdir=True)\n@with_tempfile\n@with_tempfile\ndef test_ria_postclone_noannex(dspath, storepath, clonepath):\n\n    # Test for gh-5186: Cloning from local FS, shouldn't lead to annex\n    # initializing origin.\n\n    dspath = Path(dspath)\n    storepath = Path(storepath)\n    clonepath = Path(clonepath)\n\n    from datalad.customremotes.ria_utils import (\n        create_store,\n        create_ds_in_store,\n        get_layout_locations\n    )\n    from datalad.distributed.ora_remote import (\n        LocalIO,\n    )\n\n\n\n    # First create a dataset in a RIA store the standard way\n    somefile = dspath / 'a_file.txt'\n    somefile.write_text('irrelevant')\n    ds = Dataset(dspath).create(force=True)\n\n    io = LocalIO()\n    create_store(io, storepath, '1')\n    lcl_url = \"ria+{}\".format(get_local_file_url(str(storepath)))\n    create_ds_in_store(io, storepath, ds.id, '2', '1')\n    ds.create_sibling_ria(lcl_url, \"store\")\n    ds.push('.', to='store')\n\n\n    # now, remove annex/ tree from store in order to see, that clone\n    # doesn't cause annex to recreate it.\n    store_loc, _, _ = get_layout_locations(1, storepath, ds.id)\n    annex = store_loc / 'annex'\n    rmtree(str(annex))\n    assert_false(annex.exists())\n\n    clone_url = get_local_file_url(str(storepath), compatibility='git') + \\\n                '#{}'.format(ds.id)\n    clone(\"ria+{}\".format(clone_url), clonepath)\n\n    # no need to test the cloning itself - we do that over and over in here\n\n    # bare repo in store still has no local annex:\n    assert_false(annex.exists())\n\n\n@slow  # 17sec on Yarik's laptop\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@serve_path_via_http\ndef test_inherit_src_candidates(lcl, storepath, url):\n    lcl = Path(lcl)\n    storepath = Path(storepath)\n    # dataset with a subdataset\n    ds1 = Dataset(lcl / 'ds1').create()\n    ds1sub = ds1.create('sub')\n    # a different dataset into which we install ds1, but do not touch its subds\n    ds2 = Dataset(lcl / 'ds2').create()\n    ds2.clone(source=ds1.path, path='mysub')\n\n    # we give no dataset a source candidate config!\n    # move all dataset into the store\n    for d in (ds1, ds1sub, ds2):\n        _move2store(storepath, d)\n\n    # now we must be able to obtain all three datasets from the store\n    riaclone = clone(\n        'ria+{}#{}'.format(\n            # store URL\n            url,\n            # ID of the root dataset\n            ds2.id),\n        lcl / 'clone',\n    )\n    # what happens is the the initial clone call sets a source candidate\n    # config, because it sees the dataset coming from a store\n    # all obtained subdatasets get the config inherited on-clone\n    datasets = riaclone.get('.', get_data=False, recursive=True, result_xfm='datasets')\n    # we get two subdatasets\n    eq_(len(datasets), 2)\n    for ds in datasets:\n        eq_(ConfigManager(dataset=ds, source='dataset-local').get(\n            'datalad.get.subdataset-source-candidate-200origin'),\n            'ria+%s#{id}' % url)\n\n\n@skip_if_no_network\n@with_tempfile()\ndef test_ria_http_storedataladorg(path):\n    # can we clone from the store w/o any dedicated config\n    ds = clone('ria+http://store.datalad.org#{}'.format(datalad_store_testds_id), path)\n    ok_(ds.is_installed())\n    eq_(ds.id, datalad_store_testds_id)\n\n\n@skip_if_on_windows  # see gh-4131\n# Ephemeral clones cannot use adjusted mode repos\n@skip_if_adjusted_branch\n@with_tree(tree={\n    'ds': {\n        'test.txt': 'some',\n        'subdir': {'testsub.txt': 'somemore'},\n    },\n})\n@with_tempfile\n@with_tempfile\n@with_tempfile\n@with_tempfile\ndef test_ephemeral(origin_path, bare_path,\n                   clone1_path, clone2_path, clone3_path):\n\n    file_test = Path('ds') / 'test.txt'\n    file_testsub = Path('ds') / 'subdir' / 'testsub.txt'\n\n    origin = Dataset(origin_path).create(force=True)\n    origin.save()\n    # 1. clone via path\n    clone1 = clone(origin_path, clone1_path, reckless='ephemeral')\n\n    can_symlink = has_symlink_capability()\n\n    if can_symlink:\n        clone1_annex = (clone1.repo.dot_git / 'annex')\n        ok_(clone1_annex.is_symlink())\n        ok_(clone1_annex.resolve().samefile(origin.repo.dot_git / 'annex'))\n        if not clone1.repo.is_managed_branch():\n            # TODO: We can't properly handle adjusted branch yet\n            eq_((clone1.pathobj / file_test).read_text(), 'some')\n            eq_((clone1.pathobj / file_testsub).read_text(), 'somemore')\n\n    # 2. clone via file-scheme URL\n    clone2 = clone('file://' + Path(origin_path).as_posix(), clone2_path,\n                   reckless='ephemeral')\n\n    if can_symlink:\n        clone2_annex = (clone2.repo.dot_git / 'annex')\n        ok_(clone2_annex.is_symlink())\n        ok_(clone2_annex.resolve().samefile(origin.repo.dot_git / 'annex'))\n        if not clone2.repo.is_managed_branch():\n            # TODO: We can't properly handle adjusted branch yet\n            eq_((clone2.pathobj / file_test).read_text(), 'some')\n            eq_((clone2.pathobj / file_testsub).read_text(), 'somemore')\n\n    # 3. add something to clone1 and push back to origin availability from\n    # clone1 should not be propagated (we declared 'here' dead to that end)\n\n    (clone1.pathobj / 'addition.txt').write_text(\"even more\")\n    clone1.save()\n    origin.config.set(\"receive.denyCurrentBranch\", \"updateInstead\",\n                      where=\"local\")\n    # Note, that the only thing to test is git-annex-dead here,\n    # if we couldn't symlink:\n    clone1.publish(to=DEFAULT_REMOTE,\n                   transfer_data='none' if can_symlink else 'auto')\n    if not origin.repo.is_managed_branch():\n        # test logic cannot handle adjusted branches\n        eq_(origin.repo.get_hexsha(), clone1.repo.get_hexsha())\n    res = origin.repo.whereis(\"addition.txt\")\n    if can_symlink:\n        # obv. present in origin, but this is not yet known to origin:\n        eq_(res, [])\n        res = origin.repo.fsck()\n        assert_result_count(res, 3, success=True)\n        # TODO: Double check whether annex reports POSIX paths o windows!\n        eq_({str(file_test), str(file_testsub), \"addition.txt\"},\n            {r['file'] for r in res})\n        # now origin knows:\n    res = origin.repo.whereis(\"addition.txt\")\n    eq_(res, [origin.config.get(\"annex.uuid\")])\n\n    # 4. ephemeral clone from a bare repo\n    runner = GitWitlessRunner()\n    runner.run(['git', 'clone', '--bare', origin_path, bare_path])\n    runner.run(['git', 'annex', 'init'], cwd=bare_path)\n\n    eph_from_bare = clone(bare_path, clone3_path, reckless='ephemeral')\n    can_symlink = has_symlink_capability()\n\n    if can_symlink:\n        # Bare repo uses dirhashlower by default, while a standard repo uses\n        # dirhashmixed. Symlinking different object trees doesn't really work.\n        # Don't test that here, since this is not a matter of the \"ephemeral\"\n        # option alone. We should have such a setup in the RIA tests and test\n        # for data access there.\n        # Here we only test for the correct linking.\n        eph_annex = eph_from_bare.repo.dot_git / 'annex'\n        ok_(eph_annex.is_symlink())\n        ok_(eph_annex.resolve().samefile(Path(bare_path) / 'annex'))\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create()\n    repo = ds_origin.repo\n    managed = repo.is_managed_branch()\n\n    # The setup below is involved, mostly because it's accounting for adjusted\n    # branches. The scenario itself isn't so complicated, though:\n    #\n    #   * a checked out default branch with no commits\n    #   * a (potentially adjusted) \"abc\" branch with commits.\n    #   * a (potentially adjusted) \"chooseme\" branch whose tip commit has a\n    #     more recent commit than any in \"abc\".\n    (ds_origin.pathobj / \"foo\").write_text(\"foo content\")\n    ds_origin.save(message=\"foo\")\n    for res in repo.for_each_ref_(fields=\"refname\"):\n        ref = res[\"refname\"]\n        if DEFAULT_BRANCH in ref:\n            repo.update_ref(ref.replace(DEFAULT_BRANCH, \"abc\"), ref)\n            repo.call_git([\"update-ref\", \"-d\", ref])\n    repo.update_ref(\"HEAD\",\n                    \"refs/heads/{}\".format(\n                        \"adjusted/abc(unlocked)\" if managed else \"abc\"),\n                    symbolic=True)\n    abc_ts = int(repo.format_commit(\"%ct\"))\n    repo.call_git([\"checkout\", \"-b\", \"chooseme\", \"abc~1\"])\n    if managed:\n        repo.adjust()\n    (ds_origin.pathobj / \"bar\").write_text(\"bar content\")\n    with set_date(abc_ts + 1):\n        ds_origin.save(message=\"bar\")\n    # Make the git-annex branch the most recently updated ref so that we test\n    # that it is skipped.\n    with set_date(abc_ts + 2):\n        ds_origin.drop(\"bar\", check=False)\n    ds_origin.repo.checkout(DEFAULT_BRANCH, options=[\"--orphan\"])\n\n    ds = clone(ds_origin.path, op.join(path, \"b\"))\n    # We landed on the branch with the most recent commit, ignoring the\n    # git-annex branch.\n    branch = ds.repo.get_active_branch()\n    eq_(ds.repo.get_corresponding_branch(branch) or branch,\n        \"chooseme\")\n    eq_(ds_origin.repo.get_hexsha(\"chooseme\"),\n        ds.repo.get_hexsha(\"chooseme\"))\n    # In the context of this test, the clone should be on an adjusted branch if\n    # the source landed there initially because we're on the same file system.\n    eq_(managed, ds.repo.is_managed_branch())\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head_no_other_ref(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create(annex=False)\n    ds_origin.repo.call_git([\"update-ref\", \"-d\",\n                             \"refs/heads/\" + DEFAULT_BRANCH])\n    with swallow_logs(new_level=logging.WARNING) as cml:\n        clone(source=ds_origin.path, path=op.join(path, \"b\"))\n        assert_in(\"could not find a branch with commits\", cml.out)\n\n\n@with_tempfile(mkdir=True)\ndef test_clone_unborn_head_sub(path):\n    ds_origin = Dataset(op.join(path, \"a\")).create()\n    ds_origin_sub = Dataset(op.join(path, \"a\", \"sub\")).create()\n    managed = ds_origin_sub.repo.is_managed_branch()\n    ds_origin.save(message=\"foo\")\n    sub_repo = ds_origin_sub.repo\n    # As with test_clone_unborn_head(), the setup below is complicated mostly\n    # because it's accounting for adjusted branches, but the scenario itself\n    # isn't too complicated:\n    #\n    #   * a submodule's HEAD points to a checked out branch with no commits\n    #     while a (potentially adjusted) \"other\" branch has commits\n    #\n    #   * the parent repo has the tip of \"other\" as the last recorded state\n    for res in sub_repo.for_each_ref_(fields=\"refname\"):\n        ref = res[\"refname\"]\n        if DEFAULT_BRANCH in ref:\n            sub_repo.update_ref(ref.replace(DEFAULT_BRANCH, \"other\"), ref)\n            sub_repo.call_git([\"update-ref\", \"-d\", ref])\n    sub_repo.update_ref(\n        \"HEAD\",\n        \"refs/heads/{}\".format(\n            \"adjusted/other(unlocked)\" if managed else \"other\"),\n        symbolic=True)\n    # END complicated handling for adjusted branches\n    ds_origin.save()\n    ds_origin_sub.repo.checkout(DEFAULT_BRANCH, options=[\"--orphan\"])\n\n    ds_cloned = clone(source=ds_origin.path, path=op.join(path, \"b\"))\n    ds_cloned_sub = ds_cloned.get(\n        \"sub\", result_xfm=\"datasets\", return_type=\"item-or-list\")\n\n    branch = ds_cloned_sub.repo.get_active_branch()\n    eq_(ds_cloned_sub.repo.get_corresponding_branch(branch) or branch,\n        \"other\")\n    # In the context of this test, the clone should be on an adjusted branch if\n    # the source landed there initially because we're on the same file system.\n    eq_(managed, ds_cloned_sub.repo.is_managed_branch())\n\n\n@skip_if_no_network\n@with_tempfile\ndef test_gin_cloning(path):\n    # can we clone a public ds anoynmously from gin and retrieve content\n    ds = clone('https://gin.g-node.org/datalad/datalad-ci-target', path)\n    ok_(ds.is_installed())\n    annex_path = op.join('annex', 'two')\n    git_path = op.join('git', 'one')\n    eq_(ds.repo.file_has_content(annex_path), False)\n    eq_(ds.repo.is_under_annex(git_path), False)\n    result = ds.get(annex_path)\n    assert_result_count(result, 1)\n    assert_status('ok', result)\n    eq_(result[0]['path'], op.join(ds.path, annex_path))\n    ok_file_has_content(op.join(ds.path, annex_path), 'two\\n')\n    ok_file_has_content(op.join(ds.path, git_path), 'one\\n')\n\n\n# TODO: git-annex-init fails in the second clone call below when this is\n# executed under ./tools/eval_under_testloopfs.\n@skip_if_adjusted_branch\n@with_tree(tree={\"special\": {\"f0\": \"0\"}})\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_fetch_git_special_remote(url_path, url, path):\n    url_path = Path(url_path)\n    path = Path(path)\n    ds_special = Dataset(url_path / \"special\").create(force=True)\n    ds_special.save()\n    ds_special.repo.call_git([\"update-server-info\"])\n\n    clone_url = url + \"special/.git\"\n    ds_a = clone(clone_url, path / \"a\")\n    ds_a.repo.call_annex(\n        [\"initremote\", \"special\", \"type=git\", \"autoenable=true\",\n         \"location=\" + clone_url])\n\n    # Set up a situation where a file is present only on the special remote,\n    # and its existence is known only to the special remote's git-annex branch.\n    (ds_special.pathobj / \"f1\").write_text(\"1\")\n    ds_special.save()\n    ds_special.repo.call_git([\"update-server-info\"])\n\n    ds_a.repo.fetch(DEFAULT_REMOTE)\n    ds_a.repo.merge(f\"{DEFAULT_REMOTE}/{DEFAULT_BRANCH}\")\n\n    ds_b = clone(ds_a.path, path / \"other\")\n    ds_b.get(\"f1\")\n    ok_(ds_b.repo.file_has_content(\"f1\"))\n\n\n@skip_if_adjusted_branch\n@skip_if_no_network\n@with_tempfile(mkdir=True)\ndef test_nonuniform_adjusted_subdataset(path):\n    # https://github.com/datalad/datalad/issues/5107\n    topds = Dataset(Path(path) / \"top\").create()\n    subds_url = 'git://github.com/datalad/testrepo--basic--r1'\n    topds.clone(\n        source='git://github.com/datalad/testrepo--basic--r1',\n        path='subds')\n    eq_(topds.subdatasets(return_type='item-or-list')['gitmodule_url'],\n        subds_url)\n\n\n@with_tempfile\ndef test_clone_recorded_subds_reset(path):\n    path = Path(path)\n    ds_a = create(path / \"ds_a\")\n    ds_a_sub = ds_a.create(\"sub\")\n    (ds_a_sub.pathobj / \"foo\").write_text(\"foo\")\n    ds_a.save(recursive=True)\n    (ds_a_sub.pathobj / \"bar\").write_text(\"bar\")\n    ds_a_sub.save()\n\n    ds_b = clone(ds_a.path, path / \"ds_b\")\n    ds_b.get(\"sub\")\n    assert_repo_status(ds_b.path)\n    sub_repo = Dataset(path / \"ds_b\" / \"sub\").repo\n    branch = sub_repo.get_active_branch()\n    eq_(ds_b.subdatasets()[0][\"gitshasum\"],\n        sub_repo.get_hexsha(\n            sub_repo.get_corresponding_branch(branch) or branch))\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_url_mapping(src_path, dest_path):\n    src = create(src_path)\n    dest = Dataset(dest_path)\n    # check that the impossible doesn't work\n    assert_raises(IncompleteResultsError, clone, 'rambo', dest_path)\n    # rather than adding test URL mapping here, consider\n    # test_url_mapping_specs(), it is cheaper there\n  \n    # anticipate windows test paths and escape them\n    escaped_subst = (r',rambo,%s' % src_path).replace('\\\\', '\\\\\\\\')\n    for specs in (\n            # we can clone with a simple substitution\n            {'datalad.clone.url-substitute.mike': escaped_subst},\n            # a prior match to a dysfunctional URL doesn't impact success\n            {\n                'datalad.clone.url-substitute.no': ',rambo,picknick',\n                'datalad.clone.url-substitute.mike': escaped_subst,\n            }):\n        try:\n            with patch.dict(dest.config._merged_store, specs):\n                clone('rambo', dest_path)\n        finally:\n            dest.remove(check=False)\n\n    # check submodule config impact\n    dest.create()\n    with patch.dict(dest.config._merged_store,\n                    {'datalad.clone.url-substitute.mike': escaped_subst}):\n        dest.clone('rambo', 'subds')\n    submod_rec = dest.repo.get_submodules()[0]\n    # we record the original-original URL\n    eq_(submod_rec['gitmodule_datalad-url'], 'rambo')\n    # and put the effective one as the primary URL\n    eq_(submod_rec['gitmodule_url'], src_path)\n\n\n_nomatch_map = {\n    'datalad.clone.url-substitute.nomatch': (\n        ',nomatch,NULL',\n    )\n}\n_windows_map = {\n    'datalad.clone.url-substitute.win': (\n        r',C:\\\\Users\\\\datalad\\\\from,D:\\\\to',\n    )\n}\n\n\ndef test_url_mapping_specs():\n    from datalad.core.distributed.clone import _map_urls\n    cfg = ConfigManager()\n    for m, i, o in (\n            # path redirect on windows\n            (_windows_map,\n             r'C:\\Users\\datalad\\from',\n             r'D:\\to'),\n            # test standard github mapping, no pathc needed\n            ({},\n             'https://github.com/datalad/testrepo_gh/sub _1',\n             'https://github.com/datalad/testrepo_gh-sub__1'),\n            # and on deep subdataset too\n            ({},\n             'https://github.com/datalad/testrepo_gh/sub _1/d/sub_-  1',\n             'https://github.com/datalad/testrepo_gh-sub__1-d-sub_-_1'),\n            # test that the presence of another mapping spec doesn't ruin\n            # the outcome\n            (_nomatch_map,\n             'https://github.com/datalad/testrepo_gh/sub _1',\n             'https://github.com/datalad/testrepo_gh-sub__1'),\n            # verify OSF mapping, but see\n            # https://github.com/datalad/datalad/issues/5769 for future\n            # implications\n            ({},\n             'https://osf.io/q8xnk/',\n             'osf://q8xnk'),\n            ):\n        with patch.dict(cfg._merged_store, m):\n            eq_(_map_urls(cfg, [i]), [o])\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.14.7+532.gb4fafb048\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 101ms\nInstalled 25 packages in 5.10s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 65 packages in 5.10s\nPrepared 1 package in 5.79s\nInstalled 59 packages in 599ms\n + annexremote==1.6.6\n + appdirs==1.4.4\n + argcomplete==3.5.3\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.14.7+532.gb4fafb048 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.0.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-gitlab==5.3.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + requests-toolbelt==1.0.0\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 99 packages in 288ms\nInstalled 32 packages in 607ms\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 39 items\n\nr2e_tests/test_1.py EEEEEEEEEEExEEEFE.EFsE.ExEEEFEFFFFEFFE.              [100%]\n\n==================================== ERRORS ====================================\n_____________________ ERROR at setup of test_invalid_args ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 93\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_invalid_args(path, otherpath, alienpath):\nE       fixture 'otherpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:93\n______________________ ERROR at setup of test_clone_crcns ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 125\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_crcns(tdir, ds_path):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:125\n__________________ ERROR at setup of test_clone_datasets_root __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 144\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tree(tree={'sub': {}})\n  def test_clone_datasets_root(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:144\n__________________ ERROR at setup of test_clone_simple_local ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 172\n  @with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n  @with_tempfile(mkdir=True)\n  def test_clone_simple_local(src, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:172\n____________ ERROR at setup of test_clone_dataset_from_just_source _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 217\n  @known_failure_githubci_win\n  @with_testrepos(flavors=['local-url', 'network', 'local'])\n  @with_tempfile\n  def test_clone_dataset_from_just_source(url, path):\nE       fixture 'url' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:217\n____________________ ERROR at setup of test_clone_dataladri ____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 233\n  @known_failure\n  @with_tree(tree={\n      'ds': {'test.txt': 'some'},\n      })\n  @serve_path_via_http\n  @with_tempfile(mkdir=True)\n  def test_clone_dataladri(src, topurl, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:233\n_________________ ERROR at setup of test_clone_isnot_recursive _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 254\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_isnot_recursive(path_src, path_nr, path_r):\nE       fixture 'path_src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:254\n__________________ ERROR at setup of test_clone_into_dataset ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 272\n  @with_tempfile\n  @with_tempfile\n  def test_clone_into_dataset(source_path, top_path):\nE       fixture 'source_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:272\n_______________ ERROR at setup of test_notclone_known_subdataset _______________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 313\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_notclone_known_subdataset(src_path, path):\nE       fixture 'src_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:313\n_____________________ ERROR at setup of test_failed_clone ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 345\n  @with_tempfile(mkdir=True)\n  def test_failed_clone(dspath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:345\n_________________ ERROR at setup of test_clone_missing_commit __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 355\n  @with_tree(tree={\n      'ds': {'test.txt': 'some'},\n      })\n  @with_tempfile\n  def test_clone_missing_commit(source, clone):\nE       fixture 'source' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:355\n________________ ERROR at setup of test_install_source_relpath _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 456\n  @with_tempfile\n  @with_tempfile\n  def test_install_source_relpath(src, dest):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:456\n_________________ ERROR at setup of test_clone_isnt_a_smartass _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 466\n  @with_tempfile\n  @with_tempfile\n  def test_clone_isnt_a_smartass(origin_path, path):\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:466\n_____________ ERROR at setup of test_clone_report_permission_issue _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 484\n  @with_tempfile(mkdir=True)\n  def test_clone_report_permission_issue(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:484\n__________ ERROR at setup of test_clone_autoenable_msg_handles_sameas __________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 522\n  @with_sameas_remote(autoenabled=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_autoenable_msg_handles_sameas(repo, clone_path):\nE       fixture 'repo' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:522\n______________________ ERROR at setup of test_expanduser _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 593\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_expanduser(srcpath, destpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:593\n_________________ ERROR at setup of test_local_url_with_fetch __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 689\n  @with_tree(tree={\"subdir\": {}})\n  @with_tempfile(mkdir=True)\n  def test_local_url_with_fetch(path, path_other):\nE       fixture 'path_other' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:689\n_______________________ ERROR at setup of test_ria_http ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 780\n  @slow  # 12sec on Yarik's laptop\n  @with_tree(tree={\n      'ds': {\n          'test.txt': 'some',\n          'subdir': {\n              'subds': {'testsub.txt': 'somemore'},\n          },\n      },\n  })\n  @with_tempfile(mkdir=True)\n  @serve_path_via_http\n  def test_ria_http(lcl, storepath, url):\nE       fixture 'lcl' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:780\n__________________ ERROR at setup of test_no_ria_postclonecfg __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1144\n  @known_failure_windows\n  @skip_ssh\n  @with_tree(tree={'somefile.txt': 'some content'})\n  @with_tempfile\n  @with_tempfile\n  def test_no_ria_postclonecfg(dspath, storepath, clonepath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1144\n_________________ ERROR at setup of test_ria_postclone_noannex _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1200\n  @known_failure_githubci_win  # in datalad/git-annex as e.g. of 20201218\n  @with_tempfile(mkdir=True)\n  @with_tempfile\n  @with_tempfile\n  def test_ria_postclone_noannex(dspath, storepath, clonepath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1200\n________________ ERROR at setup of test_inherit_src_candidates _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1254\n  @slow  # 17sec on Yarik's laptop\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @serve_path_via_http\n  def test_inherit_src_candidates(lcl, storepath, url):\nE       fixture 'lcl' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1254\n_______________________ ERROR at setup of test_ephemeral _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1303\n  @skip_if_on_windows  # see gh-4131\n  # Ephemeral clones cannot use adjusted mode repos\n  @skip_if_adjusted_branch\n  @with_tree(tree={\n      'ds': {\n          'test.txt': 'some',\n          'subdir': {'testsub.txt': 'somemore'},\n      },\n  })\n  @with_tempfile\n  @with_tempfile\n  @with_tempfile\n  @with_tempfile\n  def test_ephemeral(origin_path, bare_path,\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1303\n_______________ ERROR at setup of test_fetch_git_special_remote ________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1519\n  @skip_if_adjusted_branch\n  @with_tree(tree={\"special\": {\"f0\": \"0\"}})\n  @serve_path_via_http\n  @with_tempfile(mkdir=True)\n  def test_fetch_git_special_remote(url_path, url, path):\nE       fixture 'url_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1519\n___________________ ERROR at setup of test_clone_url_mapping ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1584\n  @with_tempfile\n  @with_tempfile\n  def test_clone_url_mapping(src_path, dest_path):\nE       fixture 'src_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1584\n=================================== FAILURES ===================================\n_________________________ test_autoenabled_remote_msg __________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annex079cwu5w'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def  _wrap_skip_if_no_network(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:189: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annex079cwu5w'}\nfilename = '/tmp/datalad_temp_test_autoenabled_remote_msgvw3xiu_x'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_autoenabled_remote_msg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n____________________________ test_cfg_originorigin _____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexuu94q5sf'}\nfilename = '/tmp/datalad_temp_test_cfg_originoriginmxjfycqi'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_cfg_originorigin() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n________________________ test_ria_http_storedataladorg _________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annex6nlfg7kr'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def  _wrap_skip_if_no_network(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:189: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annex6nlfg7kr'}\nfilename = '/tmp/datalad_temp_test_ria_http_storedataladorgmp738ed1'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_ria_http_storedataladorg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n____________________________ test_clone_unborn_head ____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexmbjn_bf9'}\nfilename = '/tmp/datalad_temp_test_clone_unborn_headnql3988d'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_unborn_head() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_____________________ test_clone_unborn_head_no_other_ref ______________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexn9xt42tv'}\nfilename = '/tmp/datalad_temp_test_clone_unborn_head_no_other_refmlwpk4rj'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_unborn_head_no_other_ref() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n__________________________ test_clone_unborn_head_sub __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexy8m44_7c'}\nfilename = '/tmp/datalad_temp_test_clone_unborn_head_subtd_ku8fl'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_unborn_head_sub() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_______________________________ test_gin_cloning _______________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexajdgz_pp'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def  _wrap_skip_if_no_network(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:189: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexajdgz_pp'}\nfilename = '/tmp/datalad_temp_test_gin_cloningsab9_z8s'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_gin_cloning() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_____________________ test_nonuniform_adjusted_subdataset ______________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annex00isin8o'}\n_check = <function skip_if_adjusted_branch.<locals>._wrap_skip_if_adjusted_branch.<locals>._check at 0x7f46dc3fac10>\n\n    @wraps(func)\n    @attr('skip_if_adjusted_branch')\n    def _wrap_skip_if_adjusted_branch(*args, **kwargs):\n        global _TESTS_ADJUSTED_TMPDIR\n        if _TESTS_ADJUSTED_TMPDIR is None:\n            @with_tempfile\n            def _check(path):\n                ds = Dataset(path).create(force=True)\n                return ds.repo.is_managed_branch()\n            _TESTS_ADJUSTED_TMPDIR = _check()\n    \n        if _TESTS_ADJUSTED_TMPDIR:\n            raise SkipTest(\"Test incompatible with adjusted branch default\")\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:1897: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ndatalad/tests/utils.py:189: in _wrap_skip_if_no_network\n    return func(*args, **kwargs)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annex00isin8o'}\nfilename = '/tmp/datalad_temp_test_nonuniform_adjusted_subdatasetkt5gnc60'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_nonuniform_adjusted_subdataset() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n----------------------------- Captured stderr call -----------------------------\n[INFO] Creating a new annex repo at /tmp/datalad_temp__checka42nh0al \n[INFO] scanning for unlocked files (this may take some time) \n------------------------------ Captured log call -------------------------------\nINFO     datalad.core.local.create:create.py:519 Creating a new annex repo at /tmp/datalad_temp__checka42nh0al\nINFO     datalad.annex:annexrepo.py:3647 scanning for unlocked files (this may take some time)\n_______________________ test_clone_recorded_subds_reset ________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexo6b858qt'}\nfilename = '/tmp/datalad_temp_test_clone_recorded_subds_resetxoci7a67'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_recorded_subds_reset() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_installationpath_from_url\nPASSED r2e_tests/test_1.py::test_decode_source_spec\nPASSED r2e_tests/test_1.py::test_url_mapping_specs\nSKIPPED [1] .venv/lib/python3.9/site-packages/_pytest/unittest.py:385: Skip test known to fail\nXFAIL r2e_tests/test_1.py::test_reckless - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_reckless will be ignored\nXFAIL r2e_tests/test_1.py::test_ria_postclonecfg - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_ria_postclonecfg will be ignored\nERROR r2e_tests/test_1.py::test_invalid_args\nERROR r2e_tests/test_1.py::test_clone_crcns\nERROR r2e_tests/test_1.py::test_clone_datasets_root\nERROR r2e_tests/test_1.py::test_clone_simple_local\nERROR r2e_tests/test_1.py::test_clone_dataset_from_just_source\nERROR r2e_tests/test_1.py::test_clone_dataladri\nERROR r2e_tests/test_1.py::test_clone_isnot_recursive\nERROR r2e_tests/test_1.py::test_clone_into_dataset\nERROR r2e_tests/test_1.py::test_notclone_known_subdataset\nERROR r2e_tests/test_1.py::test_failed_clone\nERROR r2e_tests/test_1.py::test_clone_missing_commit\nERROR r2e_tests/test_1.py::test_install_source_relpath\nERROR r2e_tests/test_1.py::test_clone_isnt_a_smartass\nERROR r2e_tests/test_1.py::test_clone_report_permission_issue\nERROR r2e_tests/test_1.py::test_clone_autoenable_msg_handles_sameas\nERROR r2e_tests/test_1.py::test_expanduser\nERROR r2e_tests/test_1.py::test_local_url_with_fetch\nERROR r2e_tests/test_1.py::test_ria_http\nERROR r2e_tests/test_1.py::test_no_ria_postclonecfg\nERROR r2e_tests/test_1.py::test_ria_postclone_noannex\nERROR r2e_tests/test_1.py::test_inherit_src_candidates\nERROR r2e_tests/test_1.py::test_ephemeral\nERROR r2e_tests/test_1.py::test_fetch_git_special_remote\nERROR r2e_tests/test_1.py::test_clone_url_mapping\nFAILED r2e_tests/test_1.py::test_autoenabled_remote_msg - TypeError: test_aut...\nFAILED r2e_tests/test_1.py::test_cfg_originorigin - TypeError: test_cfg_origi...\nFAILED r2e_tests/test_1.py::test_ria_http_storedataladorg - TypeError: test_r...\nFAILED r2e_tests/test_1.py::test_clone_unborn_head - TypeError: test_clone_un...\nFAILED r2e_tests/test_1.py::test_clone_unborn_head_no_other_ref - TypeError: ...\nFAILED r2e_tests/test_1.py::test_clone_unborn_head_sub - TypeError: test_clon...\nFAILED r2e_tests/test_1.py::test_gin_cloning - TypeError: test_gin_cloning() ...\nFAILED r2e_tests/test_1.py::test_nonuniform_adjusted_subdataset - TypeError: ...\nFAILED r2e_tests/test_1.py::test_clone_recorded_subds_reset - TypeError: test...\n========= 9 failed, 3 passed, 1 skipped, 2 xfailed, 24 errors in 3.59s =========\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 39 items\n\nr2e_tests/test_1.py EEEEEEEEEEExEEEFEFEFsE.ExEEEFEFFFFEFFE.              [100%]\n\n==================================== ERRORS ====================================\n_____________________ ERROR at setup of test_invalid_args ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 93\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_invalid_args(path, otherpath, alienpath):\nE       fixture 'otherpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:93\n______________________ ERROR at setup of test_clone_crcns ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 125\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_crcns(tdir, ds_path):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:125\n__________________ ERROR at setup of test_clone_datasets_root __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 144\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tree(tree={'sub': {}})\n  def test_clone_datasets_root(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:144\n__________________ ERROR at setup of test_clone_simple_local ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 172\n  @with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n  @with_tempfile(mkdir=True)\n  def test_clone_simple_local(src, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:172\n____________ ERROR at setup of test_clone_dataset_from_just_source _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 217\n  @known_failure_githubci_win\n  @with_testrepos(flavors=['local-url', 'network', 'local'])\n  @with_tempfile\n  def test_clone_dataset_from_just_source(url, path):\nE       fixture 'url' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:217\n____________________ ERROR at setup of test_clone_dataladri ____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 233\n  @known_failure\n  @with_tree(tree={\n      'ds': {'test.txt': 'some'},\n      })\n  @serve_path_via_http\n  @with_tempfile(mkdir=True)\n  def test_clone_dataladri(src, topurl, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:233\n_________________ ERROR at setup of test_clone_isnot_recursive _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 254\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_isnot_recursive(path_src, path_nr, path_r):\nE       fixture 'path_src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:254\n__________________ ERROR at setup of test_clone_into_dataset ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 272\n  @with_tempfile\n  @with_tempfile\n  def test_clone_into_dataset(source_path, top_path):\nE       fixture 'source_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:272\n_______________ ERROR at setup of test_notclone_known_subdataset _______________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 313\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_notclone_known_subdataset(src_path, path):\nE       fixture 'src_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:313\n_____________________ ERROR at setup of test_failed_clone ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 345\n  @with_tempfile(mkdir=True)\n  def test_failed_clone(dspath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:345\n_________________ ERROR at setup of test_clone_missing_commit __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 355\n  @with_tree(tree={\n      'ds': {'test.txt': 'some'},\n      })\n  @with_tempfile\n  def test_clone_missing_commit(source, clone):\nE       fixture 'source' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:355\n________________ ERROR at setup of test_install_source_relpath _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 456\n  @with_tempfile\n  @with_tempfile\n  def test_install_source_relpath(src, dest):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:456\n_________________ ERROR at setup of test_clone_isnt_a_smartass _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 466\n  @with_tempfile\n  @with_tempfile\n  def test_clone_isnt_a_smartass(origin_path, path):\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:466\n_____________ ERROR at setup of test_clone_report_permission_issue _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 484\n  @with_tempfile(mkdir=True)\n  def test_clone_report_permission_issue(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:484\n__________ ERROR at setup of test_clone_autoenable_msg_handles_sameas __________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 522\n  @with_sameas_remote(autoenabled=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_autoenable_msg_handles_sameas(repo, clone_path):\nE       fixture 'repo' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:522\n______________________ ERROR at setup of test_expanduser _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 593\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_expanduser(srcpath, destpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:593\n_________________ ERROR at setup of test_local_url_with_fetch __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 689\n  @with_tree(tree={\"subdir\": {}})\n  @with_tempfile(mkdir=True)\n  def test_local_url_with_fetch(path, path_other):\nE       fixture 'path_other' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:689\n_______________________ ERROR at setup of test_ria_http ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 780\n  @slow  # 12sec on Yarik's laptop\n  @with_tree(tree={\n      'ds': {\n          'test.txt': 'some',\n          'subdir': {\n              'subds': {'testsub.txt': 'somemore'},\n          },\n      },\n  })\n  @with_tempfile(mkdir=True)\n  @serve_path_via_http\n  def test_ria_http(lcl, storepath, url):\nE       fixture 'lcl' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:780\n__________________ ERROR at setup of test_no_ria_postclonecfg __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1144\n  @known_failure_windows\n  @skip_ssh\n  @with_tree(tree={'somefile.txt': 'some content'})\n  @with_tempfile\n  @with_tempfile\n  def test_no_ria_postclonecfg(dspath, storepath, clonepath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1144\n_________________ ERROR at setup of test_ria_postclone_noannex _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1200\n  @known_failure_githubci_win  # in datalad/git-annex as e.g. of 20201218\n  @with_tempfile(mkdir=True)\n  @with_tempfile\n  @with_tempfile\n  def test_ria_postclone_noannex(dspath, storepath, clonepath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1200\n________________ ERROR at setup of test_inherit_src_candidates _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1254\n  @slow  # 17sec on Yarik's laptop\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @serve_path_via_http\n  def test_inherit_src_candidates(lcl, storepath, url):\nE       fixture 'lcl' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1254\n_______________________ ERROR at setup of test_ephemeral _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1303\n  @skip_if_on_windows  # see gh-4131\n  # Ephemeral clones cannot use adjusted mode repos\n  @skip_if_adjusted_branch\n  @with_tree(tree={\n      'ds': {\n          'test.txt': 'some',\n          'subdir': {'testsub.txt': 'somemore'},\n      },\n  })\n  @with_tempfile\n  @with_tempfile\n  @with_tempfile\n  @with_tempfile\n  def test_ephemeral(origin_path, bare_path,\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1303\n_______________ ERROR at setup of test_fetch_git_special_remote ________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1519\n  @skip_if_adjusted_branch\n  @with_tree(tree={\"special\": {\"f0\": \"0\"}})\n  @serve_path_via_http\n  @with_tempfile(mkdir=True)\n  def test_fetch_git_special_remote(url_path, url, path):\nE       fixture 'url_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1519\n___________________ ERROR at setup of test_clone_url_mapping ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py, line 1584\n  @with_tempfile\n  @with_tempfile\n  def test_clone_url_mapping(src_path, dest_path):\nE       fixture 'src_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b4fafb048489019fc165e4e1f8b69e5bb40df008/r2e_tests/test_1.py:1584\n=================================== FAILURES ===================================\n_________________________ test_autoenabled_remote_msg __________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annex_653ucie'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def  _wrap_skip_if_no_network(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:189: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annex_653ucie'}\nfilename = '/tmp/datalad_temp_test_autoenabled_remote_msgqtfnndmh'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_autoenabled_remote_msg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n________________________ test_installationpath_from_url ________________________\n\n    def test_installationpath_from_url():\n        # cases for all OSes\n        cases = [\n            'http://example.com/lastbit',\n            'http://example.com/lastbit.git',\n            'http://lastbit:8000',\n            # SSH\n            'hostname:lastbit',\n            'hostname:lastbit/',\n            'hostname:subd/lastbit',\n            'hostname:/full/path/lastbit',\n            'hostname:lastbit/.git',\n            'hostname:lastbit/.git/',\n            'hostname:/full/path/lastbit/.git',\n            'full.hostname.com:lastbit/.git',\n            'user@full.hostname.com:lastbit/.git',\n            'ssh://user:passw@full.hostname.com/full/path/lastbit',\n            'ssh://user:passw@full.hostname.com/full/path/lastbit/',\n            'ssh://user:passw@full.hostname.com/full/path/lastbit/.git',\n        ]\n        # OS specific cases\n        cases += [\n            'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n            'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n            'Temp\\\\lastbit',\n            'Temp\\\\lastbit\\\\',\n            'lastbit.git',\n            'lastbit.git\\\\',\n        ] if on_windows else [\n            'lastbit',\n            'lastbit/',\n            '/lastbit',\n            'lastbit.git',\n            'lastbit.git/',\n        ]\n    \n        for p in cases:\n>           eq_(_get_installationpath_from_url(p), 'lastbit')\n\nr2e_tests/test_1.py:576: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\na = 'hostname:lastbit', b = 'lastbit', msg = None\n\n    def eq_(a, b, msg=None):\n        \"\"\"Shorthand for 'assert a == b, \"%r != %r\" % (a, b)\n        \"\"\"\n        if not a == b:\n>           raise AssertionError(msg or \"%r != %r\" % (a, b))\nE           AssertionError: 'hostname:lastbit' != 'lastbit'\n\n.venv/lib/python3.9/site-packages/nose/tools/trivial.py:29: AssertionError\n____________________________ test_cfg_originorigin _____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexp0h212bo'}\nfilename = '/tmp/datalad_temp_test_cfg_originoriginuedq9591'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_cfg_originorigin() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n________________________ test_ria_http_storedataladorg _________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexul_idinb'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def  _wrap_skip_if_no_network(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:189: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexul_idinb'}\nfilename = '/tmp/datalad_temp_test_ria_http_storedataladorges5w7oto'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_ria_http_storedataladorg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n____________________________ test_clone_unborn_head ____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexod90aey_'}\nfilename = '/tmp/datalad_temp_test_clone_unborn_headbwjmchxq'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_unborn_head() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_____________________ test_clone_unborn_head_no_other_ref ______________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexbrp7j3h3'}\nfilename = '/tmp/datalad_temp_test_clone_unborn_head_no_other_refsmj5g369'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_unborn_head_no_other_ref() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n__________________________ test_clone_unborn_head_sub __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexs5687huw'}\nfilename = '/tmp/datalad_temp_test_clone_unborn_head_subtobu0fr2'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_unborn_head_sub() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_______________________________ test_gin_cloning _______________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexanxbht3o'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def  _wrap_skip_if_no_network(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:189: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexanxbht3o'}\nfilename = '/tmp/datalad_temp_test_gin_cloningp620uo0z'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_gin_cloning() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n_____________________ test_nonuniform_adjusted_subdataset ______________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexh98g028n'}\n_check = <function skip_if_adjusted_branch.<locals>._wrap_skip_if_adjusted_branch.<locals>._check at 0x77d9707520d0>\n\n    @wraps(func)\n    @attr('skip_if_adjusted_branch')\n    def _wrap_skip_if_adjusted_branch(*args, **kwargs):\n        global _TESTS_ADJUSTED_TMPDIR\n        if _TESTS_ADJUSTED_TMPDIR is None:\n            @with_tempfile\n            def _check(path):\n                ds = Dataset(path).create(force=True)\n                return ds.repo.is_managed_branch()\n            _TESTS_ADJUSTED_TMPDIR = _check()\n    \n        if _TESTS_ADJUSTED_TMPDIR:\n            raise SkipTest(\"Test incompatible with adjusted branch default\")\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:1897: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \ndatalad/tests/utils.py:189: in _wrap_skip_if_no_network\n    return func(*args, **kwargs)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexh98g028n'}\nfilename = '/tmp/datalad_temp_test_nonuniform_adjusted_subdatasetj805b6kl'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_nonuniform_adjusted_subdataset() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n----------------------------- Captured stderr call -----------------------------\n[INFO] Creating a new annex repo at /tmp/datalad_temp__check5v6jd82g \n[INFO] scanning for unlocked files (this may take some time) \n------------------------------ Captured log call -------------------------------\nINFO     datalad.core.local.create:create.py:519 Creating a new annex repo at /tmp/datalad_temp__check5v6jd82g\nINFO     datalad.annex:annexrepo.py:3647 scanning for unlocked files (this may take some time)\n_______________________ test_clone_recorded_subds_reset ________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex8vx_6mja'}\nfilename = '/tmp/datalad_temp_test_clone_recorded_subds_resetgjhz156n'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_clone_recorded_subds_reset() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:739: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_decode_source_spec\nPASSED r2e_tests/test_1.py::test_url_mapping_specs\nSKIPPED [1] .venv/lib/python3.9/site-packages/_pytest/unittest.py:385: Skip test known to fail\nXFAIL r2e_tests/test_1.py::test_reckless - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_reckless will be ignored\nXFAIL r2e_tests/test_1.py::test_ria_postclonecfg - reason: [NOTRUN] yield tests were removed in pytest 4.0 - test_ria_postclonecfg will be ignored\nERROR r2e_tests/test_1.py::test_invalid_args\nERROR r2e_tests/test_1.py::test_clone_crcns\nERROR r2e_tests/test_1.py::test_clone_datasets_root\nERROR r2e_tests/test_1.py::test_clone_simple_local\nERROR r2e_tests/test_1.py::test_clone_dataset_from_just_source\nERROR r2e_tests/test_1.py::test_clone_dataladri\nERROR r2e_tests/test_1.py::test_clone_isnot_recursive\nERROR r2e_tests/test_1.py::test_clone_into_dataset\nERROR r2e_tests/test_1.py::test_notclone_known_subdataset\nERROR r2e_tests/test_1.py::test_failed_clone\nERROR r2e_tests/test_1.py::test_clone_missing_commit\nERROR r2e_tests/test_1.py::test_install_source_relpath\nERROR r2e_tests/test_1.py::test_clone_isnt_a_smartass\nERROR r2e_tests/test_1.py::test_clone_report_permission_issue\nERROR r2e_tests/test_1.py::test_clone_autoenable_msg_handles_sameas\nERROR r2e_tests/test_1.py::test_expanduser\nERROR r2e_tests/test_1.py::test_local_url_with_fetch\nERROR r2e_tests/test_1.py::test_ria_http\nERROR r2e_tests/test_1.py::test_no_ria_postclonecfg\nERROR r2e_tests/test_1.py::test_ria_postclone_noannex\nERROR r2e_tests/test_1.py::test_inherit_src_candidates\nERROR r2e_tests/test_1.py::test_ephemeral\nERROR r2e_tests/test_1.py::test_fetch_git_special_remote\nERROR r2e_tests/test_1.py::test_clone_url_mapping\nFAILED r2e_tests/test_1.py::test_autoenabled_remote_msg - TypeError: test_aut...\nFAILED r2e_tests/test_1.py::test_installationpath_from_url - AssertionError: ...\nFAILED r2e_tests/test_1.py::test_cfg_originorigin - TypeError: test_cfg_origi...\nFAILED r2e_tests/test_1.py::test_ria_http_storedataladorg - TypeError: test_r...\nFAILED r2e_tests/test_1.py::test_clone_unborn_head - TypeError: test_clone_un...\nFAILED r2e_tests/test_1.py::test_clone_unborn_head_no_other_ref - TypeError: ...\nFAILED r2e_tests/test_1.py::test_clone_unborn_head_sub - TypeError: test_clon...\nFAILED r2e_tests/test_1.py::test_gin_cloning - TypeError: test_gin_cloning() ...\nFAILED r2e_tests/test_1.py::test_nonuniform_adjusted_subdataset - TypeError: ...\nFAILED r2e_tests/test_1.py::test_clone_recorded_subds_reset - TypeError: test...\n======== 10 failed, 2 passed, 1 skipped, 2 xfailed, 24 errors in 1.82s =========\n",
    "old_commit_res_stderr": ""
}
.travis.yml datalad/core/distributed/clone.py datalad/core/distributed/tests/test_clone.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Incorrect extraction of installation path from SSH URLs

**Description:**
When providing an SSH URL, the installation path is not correctly extracted, resulting in the path including the hostname instead of just the repository name.

**Example Code:**
```python
url = 'hostname:lastbit'
path = _get_installationpath_from_url(url)
print(path)  # Outputs: 'hostname:lastbit'
```

**Expected Behavior:**
The installation path should be extracted as `'lastbit'` from the SSH URL.

**Actual Behavior:**
The function returns `'hostname:lastbit'` instead of the expected `'lastbit'`.
[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/core/distributed/clone.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Plumbing command for dataset installation"""


import logging
import re
from os.path import expanduser
from collections import OrderedDict
from urllib.parse import unquote as urlunquote

from datalad.interface.base import Interface
from datalad.interface.utils import eval_results
from datalad.interface.base import build_doc
from datalad.interface.results import get_status_dict
from datalad.interface.common_opts import (
    location_description,
    reckless_opt,
)
from datalad.log import log_progress
from datalad.support.gitrepo import (
    GitRepo,
)
from datalad.cmd import (
    CommandError,
    GitWitlessRunner,
    StdOutCapture,
)
from datalad.distributed.ora_remote import (
    LocalIO,
    RIARemoteError,
    SSHRemoteIO,
)
from datalad.support.annexrepo import AnnexRepo
from datalad.support.constraints import (
    EnsureNone,
    EnsureStr,
    EnsureKeyChoice,
)
from datalad.support.exceptions import DownloadError
from datalad.support.param import Parameter
from datalad.support.strings import get_replacement_dict
from datalad.support.network import (
    get_local_file_url,
    download_url,
    is_url,
    URL,
    RI,
    DataLadRI,
    PathRI,
)
from datalad.dochelpers import (
    exc_str,
    single_or_plural,
)
from datalad.utils import (
    ensure_bool,
    ensure_list,
    knows_annex,
    make_tempfile,
    Path,
    PurePosixPath,
    rmtree,
)

from datalad.distribution.dataset import (
    Dataset,
    datasetmethod,
    resolve_path,
    require_dataset,
    EnsureDataset,
)
from datalad.distribution.utils import (
    _get_flexible_source_candidates,
)
from datalad.utils import (
    check_symlink_capability
)

__docformat__ = 'restructuredtext'

lgr = logging.getLogger('datalad.core.distributed.clone')


@build_doc
class Clone(Interface):
    """Obtain a dataset (copy) from a URL or local directory

    The purpose of this command is to obtain a new clone (copy) of a dataset
    and place it into a not-yet-existing or empty directory. As such `clone`
    provides a strict subset of the functionality offered by `install`. Only a
    single dataset can be obtained, and immediate recursive installation of
    subdatasets is not supported. However, once a (super)dataset is installed
    via `clone`, any content, including subdatasets can be obtained by a
    subsequent `get` command.

    Primary differences over a direct `git clone` call are 1) the automatic
    initialization of a dataset annex (pure Git repositories are equally
    supported); 2) automatic registration of the newly obtained dataset as a
    subdataset (submodule), if a parent dataset is specified; 3) support
    for additional resource identifiers (DataLad resource identifiers as used
    on datasets.datalad.org, and RIA store URLs as used for store.datalad.org
    - optionally in specific versions as identified by a branch or a tag; see
    examples); and 4) automatic configurable generation of alternative access
    URL for common cases (such as appending '.git' to the URL in case the
    accessing the base URL failed).

    In case the clone is registered as a subdataset, the original URL passed to
    `clone` is recorded in `.gitmodules` of the parent dataset in addition
    to the resolved URL used internally for git-clone. This allows to preserve
    datalad specific URLs like ria+ssh://... for subsequent calls to `get` if
    the subdataset was locally removed later on.

    || PYTHON >>By default, the command returns a single Dataset instance for
    an installed dataset, regardless of whether it was newly installed ('ok'
    result), or found already installed from the specified source ('notneeded'
    result).<< PYTHON ||

    URL mapping configuration

    'clone' supports the transformation of URLs via (multi-part) substitution
    specifications. A substitution specification is defined as a configuration
    setting 'datalad.clone.url-substition.<seriesID>' with a string containing
    a match and substitution expression, each following Python's regular
    expression syntax. Both expressions are concatenated to a single string
    with an arbitrary delimiter character. The delimiter is defined by
    prefixing the string with the delimiter. Prefix and delimiter are stripped
    from the expressions (Example: ",^http://(.*)$,https://\\1").  This setting
    can be defined multiple times, using the same '<seriesID>'.  Substitutions
    in a series will be applied incrementally, in order of their definition.
    The first substitution in such a series must match, otherwise no further
    substitutions in a series will be considered. However, following the first
    match all further substitutions in a series are processed, regardless
    whether intermediate expressions match or not. Substitution series themselves
    have no particular order, each matching series will result in a candidate
    clone URL. Consequently, the initial match specification in a series should
    be as precise as possible to prevent inflation of candidate URLs.

    .. seealso::

      :ref:`handbook:3-001`
        More information on Remote Indexed Archive (RIA) stores
    """
    # by default ignore everything but install results
    # i.e. no "add to super dataset"
    result_filter = EnsureKeyChoice('action', ('install',))
    # very frequently this command will yield exactly one installed dataset
    # spare people the pain of going through a list by default
    return_type = 'item-or-list'
    # as discussed in #1409 and #1470, we want to return dataset instances
    # matching what is actually available after command completion (and
    # None for any failed dataset installation)
    result_xfm = 'successdatasets-or-none'

    _examples_ = [
        dict(text="Install a dataset from Github into the current directory",
             code_py="clone("
             "source='https://github.com/datalad-datasets/longnow"
             "-podcasts.git')",
             code_cmd="datalad clone "
             "https://github.com/datalad-datasets/longnow-podcasts.git"),
        dict(text="Install a dataset into a specific directory",
             code_py="""\
             clone(source='https://github.com/datalad-datasets/longnow-podcasts.git',
                   path='myfavpodcasts')""",
             code_cmd="""\
             datalad clone https://github.com/datalad-datasets/longnow-podcasts.git \\
             myfavpodcasts"""),
        dict(text="Install a dataset as a subdataset into the current dataset",
             code_py="""\
             clone(dataset='.',
                   source='https://github.com/datalad-datasets/longnow-podcasts.git')""",
             code_cmd="datalad clone -d . "
             "https://github.com/datalad-datasets/longnow-podcasts.git"),
        dict(text="Install the main superdataset from datasets.datalad.org",
             code_py="clone(source='///')",
             code_cmd="datalad clone ///"),
        dict(text="Install a dataset identified by a literal alias from store.datalad.org",
             code_py="clone(source='ria+http://store.datalad.org#~hcp-openaccess')",
             code_cmd="datalad clone ria+http://store.datalad.org#~hcp-openaccess"),
        dict(
            text="Install a dataset in a specific version as identified by a "
                 "branch or tag name from store.datalad.org",
            code_py="clone(source='ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier')",
            code_cmd="datalad clone ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier"),
        dict(
            text="Install a dataset with group-write access permissions",
            code_py=\
            "clone(source='http://example.com/dataset', reckless='shared-group')",
            code_cmd=\
            "datalad clone http://example.com/dataset --reckless shared-group"),
    ]

    _params_ = dict(
        dataset=Parameter(
            args=("-d", "--dataset"),
            doc="""(parent) dataset to clone into. If given, the newly cloned
            dataset is registered as a subdataset of the parent. Also, if given,
            relative paths are interpreted as being relative to the parent
            dataset, and not relative to the working directory.""",
            constraints=EnsureDataset() | EnsureNone()),
        source=Parameter(
            args=("source",),
            metavar='SOURCE',
            doc="""URL, DataLad resource identifier, local path or instance of
            dataset to be cloned""",
            constraints=EnsureStr() | EnsureNone()),
        path=Parameter(
            args=("path",),
            metavar='PATH',
            nargs="?",
            doc="""path to clone into.  If no `path` is provided a
            destination path will be derived from a source URL
            similar to :command:`git clone`"""),
        description=location_description,
        reckless=reckless_opt,
    )

    @staticmethod
    @datasetmethod(name='clone')
    @eval_results
    def __call__(
            source,
            path=None,
            dataset=None,
            description=None,
            reckless=None):
        # did we explicitly get a dataset to install into?
        # if we got a dataset, path will be resolved against it.
        # Otherwise path will be resolved first.
        ds = require_dataset(
            dataset, check_installed=True, purpose='clone') \
            if dataset is not None else dataset
        refds_path = ds.path if ds else None

        # legacy compatibility
        if reckless is True:
            # so that we can forget about how things used to be
            reckless = 'auto'

        if isinstance(source, Dataset):
            source = source.path

        if source == path:
            # even if they turn out to be identical after resolving symlinks
            # and more sophisticated witchcraft, it would still happily say
            # "it appears to be already installed", so we just catch an
            # obviously pointless input combination
            raise ValueError(
                "clone `source` and destination `path` are identical [{}]. "
                "If you are trying to add a subdataset simply use `save`".format(
                    path))

        if path is not None:
            path = resolve_path(path, dataset)

        # derive target from source:
        if path is None:
            # we got nothing but a source. do something similar to git clone
            # and derive the path from the source and continue
            # since this is a relative `path`, resolve it:
            # we are not going to reuse the decoded URL, as this is done for
            # all source candidates in clone_dataset(), we just use to determine
            # a destination path here in order to perform a bunch of additional
            # checks that shall not pollute the helper function
            source_ = decode_source_spec(
                source, cfg=None if ds is None else ds.config)
            path = resolve_path(source_['default_destpath'], dataset)
            lgr.debug("Determined clone target path from source")
        lgr.debug("Resolved clone target path to: '%s'", path)

        # there is no other way -- my intoxicated brain tells me
        assert(path is not None)

        result_props = dict(
            action='install',
            logger=lgr,
            refds=refds_path,
            source_url=source)

        try:
            # this will implicitly cause pathlib to run a bunch of checks
            # whether the present path makes any sense on the platform
            # we are running on -- we don't care if the path actually
            # exists at this point, but we want to abort early if the path
            # spec is determined to be useless
            path.exists()
        except OSError as e:
            yield get_status_dict(
                status='error',
                path=path,
                message=('cannot handle target path: %s', exc_str(e)),
                **result_props)
            return

        destination_dataset = Dataset(path)
        result_props['ds'] = destination_dataset

        if ds is not None and ds.pathobj not in path.parents:
            yield get_status_dict(
                status='error',
                message=("clone target path '%s' not in specified target dataset '%s'",
                         path, ds),
                **result_props)
            return

        # perform the actual cloning operation
        yield from clone_dataset(
            [source],
            destination_dataset,
            reckless,
            description,
            result_props,
            cfg=None if ds is None else ds.config,
        )

        # TODO handle any 'version' property handling and verification using a
        # dedicated public helper

        if ds is not None:
            # we created a dataset in another dataset
            # -> make submodule
            actually_saved_subds = False
            for r in ds.save(
                    path,
                    # Note, that here we know we don't save anything but a new
                    # subdataset. Hence, don't go with default commit message,
                    # but be more specific.
                    message="[DATALAD] Added subdataset",
                    return_type='generator',
                    result_filter=None,
                    result_xfm=None,
                    on_failure='ignore'):
                actually_saved_subds = actually_saved_subds or (
                        r['action'] == 'save' and
                        r['type'] == 'dataset' and
                        r['refds'] == ds.path and
                        r['status'] == 'ok')
                yield r

            # Modify .gitmodules to contain originally given url. This is
            # particularly relevant for postclone routines on a later `get`
            # for that subdataset. See gh-5256.
            if actually_saved_subds:
                # New subdataset actually saved. Amend the modification
                # of .gitmodules. Note, that we didn't allow to deviate
                # from git default behavior WRT a submodule's name vs
                # its path when we made this a new subdataset.
                subds_name = path.relative_to(ds.pathobj)
                ds.repo.call_git(
                    ['config',
                     '--file',
                     '.gitmodules',
                     '--replace-all',
                     'submodule.{}.{}'.format(subds_name,
                                              "datalad-url"),
                     source]
                )
                yield from ds.save('.gitmodules',
                                   amend=True, to_git=True)
            else:
                # We didn't really commit. Just call `subdatasets`
                # in that case to have the modification included in the
                # post-clone state (whatever that may be).
                ds.subdatasets(path, set_property=[("datalad-url", source)])


def _get_url_mappings(cfg):
    cfg_prefix = 'datalad.clone.url-substitute.'
    # figure out which keys we should be looking for
    # in the active config
    subst_keys = set(k for k in cfg.keys() if k.startswith(cfg_prefix))
    # and in the common config specs
    from datalad.interface.common_cfg import definitions
    subst_keys.update(k for k in definitions if k.startswith(cfg_prefix))
    # TODO a potential sorting of substitution series could be implemented
    # here
    return [
        # decode the rule specifications
        get_replacement_dict(
            # one or more could come out
            ensure_list(
                cfg.get(
                    k,
                    # make sure to pull the default from the common config
                    default=cfg.obtain(k),
                    # we specifically support declaration of multiple
                    # settings to build replacement chains
                    get_all=True)))
        for k in subst_keys
    ]


def _map_urls(cfg, urls):
    mapping_specs = _get_url_mappings(cfg)
    if not mapping_specs:
        return urls

    mapped = []
    # we process the candidate in order to maintain any prioritization
    # encoded in it (e.g. _get_flexible_source_candidates_for_submodule)
    # if we have a matching mapping replace the URL in its position
    for u in urls:
        # we only permit a single match
        # TODO we likely want to RF this to pick the longest match
        mapping_applied = False
        # try one mapping set at a time
        for mapping_spec in mapping_specs:
            # process all substitution patterns in the specification
            # always operate on strings (could be a Path instance too)
            mu = str(u)
            matched = False
            for match_ex, subst_ex in mapping_spec.items():
                if not matched:
                    matched = re.match(match_ex, mu) is not None
                if not matched:
                    break
                # try to map, would return unchanged, if there is no match
                mu = re.sub(match_ex, subst_ex, mu)
            if mu != u:
                lgr.debug("URL substitution: '%s' -> '%s'", u, mu)
                mapped.append(mu)
                # we could consider breaking after the for effective mapping
                # specification. however, that would mean any generic
                # definition of a broadly matching substitution would derail
                # the entroe system. moreover, suddently order would matter
                # substantially
                mapping_applied = True
        if not mapping_applied:
            # none of the mappings matches, go with the original URL
            # (really original, not the stringified one)
            mapped.append(u)
    return mapped


def clone_dataset(
        srcs,
        destds,
        reckless=None,
        description=None,
        result_props=None,
        cfg=None,
        checkout_gitsha=None):
    """Internal helper to perform cloning without sanity checks (assumed done)

    This helper does not handle any saving of subdataset modification or adding
    in a superdataset.

    Parameters
    ----------
    srcs : list
      Any suitable clone source specifications (paths, URLs)
    destds : Dataset
      Dataset instance for the clone destination
    reckless : {None, 'auto', 'ephemeral', 'shared-...'}, optional
      Mode switch to put cloned dataset into unsafe/throw-away configurations, i.e.
      sacrifice data safety for performance or resource footprint. When None
      and `cfg` is specified, use the value of `datalad.clone.reckless`.
    description : str, optional
      Location description for the annex of the dataset clone (if there is any).
    result_props : dict, optional
      Default properties for any yielded result, passed on to get_status_dict().
    cfg : ConfigManager, optional
      Configuration for parent dataset. This will be queried instead
      of the global DataLad configuration.
    checkout_gitsha : str, optional
      If given, a specific commit, identified by shasum, will be checked out after
      cloning. A dedicated follow-up fetch will be performed, if the initial clone
      did not obtain the commit object. Should the checkout of the target commit
      cause a detached HEAD, the previously active branch will be reset to the
      target commit.

    Yields
    ------
    dict
      DataLad result records
    """
    if not result_props:
        # in case the caller had no specific idea on how results should look
        # like, provide sensible defaults
        result_props = dict(
            action='install',
            logger=lgr,
            ds=destds,
        )

    if reckless is None and cfg:
        # if reckless is not explicitly given, but we operate on a
        # superdataset, query whether it has been instructed to operate
        # in a reckless mode, and inherit it for the coming clone
        reckless = cfg.get('datalad.clone.reckless', None)

    dest_path = destds.pathobj

    # check for configured URL mappings, either in the given config manager
    # or in the one of the destination dataset, which is typically not existent
    # yet and the process config manager is then used effectively
    srcs = _map_urls(cfg or destds.config, srcs)

    # decode all source candidate specifications
    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]

    # now expand the candidate sources with additional variants of the decoded
    # giturl, while duplicating the other properties in the additional records
    # for simplicity. The hope is to overcome a few corner cases and be more
    # robust than git clone
    candidate_sources = [
        dict(props, giturl=s) for props in candidate_sources
        for s in _get_flexible_source_candidates(props['giturl'])
    ]

    # important test! based on this `rmtree` will happen below after failed clone
    dest_path_existed = dest_path.exists()
    if dest_path_existed and any(dest_path.iterdir()):
        if destds.is_installed():
            # check if dest was cloned from the given source before
            # this is where we would have installed this from
            # this is where it was actually installed from
            track_name, track_url = _get_tracking_source(destds)
            try:
                # this will get us track_url in system native path conventions,
                # whenever it is a path (and not a URL)
                # this is needed to match it to any potentially incoming local
                # source path in the 'notneeded' test below
                track_path = str(Path(track_url))
            except Exception:
                # this should never happen, because Path() will let any non-path stringification
                # pass through unmodified, but we do not want any potential crash due to
                # pathlib behavior changes
                lgr.debug("Unexpected behavior of pathlib!")
                track_path = None
            for cand in candidate_sources:
                src = cand['giturl']
                if track_url == src \
                        or (not is_url(track_url)
                            and get_local_file_url(track_url, compatibility='git') == src) \
                        or track_path == expanduser(src):
                    yield get_status_dict(
                        status='notneeded',
                        message=("dataset %s was already cloned from '%s'",
                                 destds,
                                 src),
                        **result_props)
                    return
        # anything else is an error
        yield get_status_dict(
            status='error',
            message='target path already exists and not empty, refuse to clone into target path',
            **result_props)
        return

    log_progress(
        lgr.info,
        'cloneds',
        'Cloning dataset to %s', destds,
        total=len(candidate_sources),
        label='Clone attempt',
        unit=' Candidate locations',
    )
    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url
    for cand in candidate_sources:
        log_progress(
            lgr.info,
            'cloneds',
            'Attempting to clone from %s to %s', cand['giturl'], dest_path,
            update=1,
            increment=True)

        clone_opts = {}

        if cand.get('version', None):
            clone_opts['branch'] = cand['version']
        try:
            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35
            # doesn't make it happen seamlessly
            GitRepo.clone(
                path=str(dest_path),
                url=cand['giturl'],
                clone_options=clone_opts,
                create=True)

        except CommandError as e:
            e_stderr = e.stderr

            error_msgs[cand['giturl']] = e
            lgr.debug("Failed to clone from URL: %s (%s)",
                      cand['giturl'], exc_str(e))
            if dest_path.exists():
                lgr.debug("Wiping out unsuccessful clone attempt at: %s",
                          dest_path)
                # We must not just rmtree since it might be curdir etc
                # we should remove all files/directories under it
                # TODO stringification can be removed once patlib compatible
                # or if PY35 is no longer supported
                rmtree(str(dest_path), children_only=dest_path_existed)

            if e_stderr and 'could not create work tree' in e_stderr.lower():
                # this cannot be fixed by trying another URL
                re_match = re.match(r".*fatal: (.*)$", e_stderr,
                                    flags=re.MULTILINE | re.DOTALL)
                # cancel progress bar
                log_progress(
                    lgr.info,
                    'cloneds',
                    'Completed clone attempts for %s', destds
                )
                yield get_status_dict(
                    status='error',
                    message=re_match.group(1).strip()
                    if re_match else "stderr: " + e_stderr,
                    **result_props)
                return
            # next candidate
            continue

        result_props['source'] = cand
        # do not bother with other sources if succeeded
        break

    log_progress(
        lgr.info,
        'cloneds',
        'Completed clone attempts for %s', destds
    )

    if not destds.is_installed():
        if len(error_msgs):
            if all(not e.stdout and not e.stderr for e in error_msgs.values()):
                # there is nothing we can learn from the actual exception,
                # the exit code is uninformative, the command is predictable
                error_msg = "Failed to clone from all attempted sources: %s"
                error_args = list(error_msgs.keys())
            else:
                error_msg = "Failed to clone from any candidate source URL. " \
                            "Encountered errors per each url were:\n- %s"
                error_args = '\n- '.join(
                    '{}\n  {}'.format(url, exc_str(exc))
                    for url, exc in error_msgs.items()
                )
        else:
            # yoh: Not sure if we ever get here but I felt that there could
            #      be a case when this might happen and original error would
            #      not be sufficient to troubleshoot what is going on.
            error_msg = "Awkward error -- we failed to clone properly. " \
                        "Although no errors were encountered, target " \
                        "dataset at %s seems to be not fully installed. " \
                        "The 'succesful' source was: %s"
            error_args = (destds.path, cand['giturl'])
        yield get_status_dict(
            status='error',
            message=(error_msg, error_args),
            **result_props)
        return

    dest_repo = destds.repo

    remotes = dest_repo.get_remotes(with_urls_only=True)
    nremotes = len(remotes)
    if nremotes == 1:
        remote = remotes[0]
        lgr.debug("Determined %s to be remote of %s", remote, destds)
    elif remotes > 1:
        lgr.warning(
            "Fresh clone %s unexpected has multiple remotes: %s. Using %s",
            destds.path, remotes, remotes[0])
        remote = remotes[0]
    else:
        raise RuntimeError("bug: fresh clone has zero remotes")

    if not cand.get("version"):
        postclone_check_head(destds, remote=remote)

    # act on --reckless=shared-...
    # must happen prior git-annex-init, where we can cheaply alter the repo
    # setup through safe re-init'ing
    if reckless and reckless.startswith('shared-'):
        lgr.debug('Reinit %s to enable shared access permissions', destds)
        destds.repo.call_git(['init', '--shared={}'.format(reckless[7:])])

    # In case of RIA stores we need to prepare *before* annex is called at all
    if result_props['source']['type'] == 'ria':
        postclone_preannex_cfg_ria(destds, remote=remote)

    yield from postclonecfg_annexdataset(
        destds,
        reckless,
        description,
        remote=remote)

    if checkout_gitsha and \
       dest_repo.get_hexsha(dest_repo.get_corresponding_branch()) != checkout_gitsha:
        try:
            postclone_checkout_commit(dest_repo, checkout_gitsha,
                                      remote=remote)
        except Exception as e:
            yield get_status_dict(
                status='error',
                message=str(e),
                **result_props,
            )

            # We were supposed to clone a particular version but failed to.
            # This is particularly pointless in case of subdatasets and
            # potentially fatal with current implementation of recursion.
            # see gh-5387
            lgr.debug("Failed to checkout %s, removing this clone attempt at %s", checkout_gitsha, dest_path)
            # TODO stringification can be removed once pathlib compatible
            # or if PY35 is no longer supported
            rmtree(str(dest_path), children_only=dest_path_existed)
            return

    # perform any post-processing that needs to know details of the clone
    # source
    if result_props['source']['type'] == 'ria':
        yield from postclonecfg_ria(destds, result_props['source'],
                                    remote=remote)

    if reckless:
        # store the reckless setting in the dataset to make it
        # known to later clones of subdatasets via get()
        destds.config.set(
            'datalad.clone.reckless', reckless,
            where='local',
            reload=True)
    else:
        # We would still want to reload configuration to ensure that any of the
        # above git invocations could have potentially changed the config
        # TODO: might no longer be necessary if 0.14.0 adds reloading upon
        # non-readonly commands invocation
        destds.config.reload()

    # yield successful clone of the base dataset now, as any possible
    # subdataset clone down below will not alter the Git-state of the
    # parent
    yield get_status_dict(status='ok', **result_props)


def postclone_checkout_commit(repo, target_commit, remote="origin"):
    """Helper to check out a specific target commit in a fresh clone.

    Will not check (again) if current commit and target commit are already
    the same!
    """
    # record what branch we were on right after the clone
    active_branch = repo.get_active_branch()
    corr_branch = repo.get_corresponding_branch(branch=active_branch)
    was_adjusted = bool(corr_branch)
    repo_orig_branch = corr_branch or active_branch
    # if we are on a branch this hexsha will be the tip of that branch
    repo_orig_hexsha = repo.get_hexsha(repo_orig_branch)
    # make sure we have the desired commit locally
    # expensive and possibly error-prone fetch conditional on cheap
    # local check
    if not repo.commit_exists(target_commit):
        try:
            repo.fetch(remote=remote, refspec=target_commit)
        except CommandError:
            pass
        # instead of inspecting the fetch results for possible ways
        # with which it could failed to produced the desired result
        # let's verify the presence of the commit directly, we are in
        # expensive-land already anyways
        if not repo.commit_exists(target_commit):
            # there is nothing we can do about this
            # MIH thinks that removing the clone is not needed, as a likely
            # next step will have to be a manual recovery intervention
            # and not another blind attempt
            raise ValueError(
                'Target commit %s does not exist in the clone, and '
                'a fetch that commit from remote failed'
                % target_commit[:8])
    # checkout the desired commit
    repo.call_git(['checkout', target_commit])
    # did we detach?
    if repo_orig_branch and not repo.get_active_branch():
        # trace if current state is a predecessor of the branch_hexsha
        lgr.debug(
            "Detached HEAD after resetting worktree of %s "
            "(original branch: %s)", repo, repo_orig_branch)
        if repo.get_merge_base(
                [repo_orig_hexsha, target_commit]) == target_commit:
            # we assume the target_commit to be from the same branch,
            # because it is an ancestor -- update that original branch
            # to point to the target_commit, and update HEAD to point to
            # that location
            lgr.info(
                "Reset branch '%s' to %s (from %s) to "
                "avoid a detached HEAD",
                repo_orig_branch, target_commit[:8], repo_orig_hexsha[:8])
            branch_ref = 'refs/heads/%s' % repo_orig_branch
            repo.update_ref(branch_ref, target_commit)
            repo.update_ref('HEAD', branch_ref, symbolic=True)
            if was_adjusted:
                # Note: The --force is needed because the adjust branch already
                # exists.
                repo.adjust(options=["--unlock", "--force"])
        else:
            lgr.warning(
                "%s has a detached HEAD, because the target commit "
                "%s has no unique ancestor with branch '%s'",
                repo, target_commit[:8], repo_orig_branch)


def postclone_check_head(ds, remote="origin"):
    repo = ds.repo
    if not repo.commit_exists("HEAD"):
        # HEAD points to an unborn branch. A likely cause of this is that the
        # remote's main branch is something other than master but HEAD wasn't
        # adjusted accordingly.
        #
        # Let's choose the most recently updated remote ref (according to
        # commit date). In the case of a submodule, switching to a ref with
        # commits prevents .update_submodule() from failing. It is likely that
        # the ref includes the registered commit, but we don't have the
        # information here to know for sure. If it doesn't, .update_submodule()
        # will check out a detached HEAD.
        remote_branches = (
            b["refname:strip=2"] for b in repo.for_each_ref_(
                fields="refname:strip=2", sort="-committerdate",
                pattern="refs/remotes/" + remote))
        for rbranch in remote_branches:
            if rbranch in [remote + "/git-annex", "HEAD"]:
                continue
            if rbranch.startswith(remote + "/adjusted/"):
                # If necessary for this file system, a downstream
                # git-annex-init call will handle moving into an
                # adjusted state.
                continue
            repo.call_git(["checkout", "-b",
                           rbranch[len(remote) + 1:],  # drop "<remote>/"
                           "--track", rbranch])
            lgr.debug("Checked out local branch from %s", rbranch)
            return
        lgr.warning("Cloned %s but could not find a branch "
                    "with commits", ds.path)


def postclone_preannex_cfg_ria(ds, remote="origin"):

    # We need to annex-ignore the remote before annex-init is called on the clone,
    # due to issues 5186 and 5253 (and we would have done it afterwards anyway).
    # annex/objects in RIA stores is special for several reasons.
    # 1. the remote doesn't know about it (no actual local annex for the remote)
    # 2. RIA may use hashdir mixed, copying data to it via git-annex (if cloned
    #    via ssh or local) would make it see a bare repo and establish a
    #    hashdir lower annex object tree.
    # 3. We want the ORA remote to receive all data for the store, so its
    #    objects could be moved into archives (the main point of a RIA store).

    # Note, that this function might need an enhancement as theoretically a RIA
    # store could also hold simple standard annexes w/o an intended ORA remote.
    # This needs the introduction of a new version label in RIA datasets, making
    # the following call conditional.
    ds.config.set(f'remote.{remote}.annex-ignore', 'true', where='local')


def postclonecfg_ria(ds, props, remote="origin"):
    """Configure a dataset freshly cloned from a RIA store"""
    repo = ds.repo

    def get_uuid_from_store(store_url):
        # First figure whether we cloned via SSH, HTTP or local path and then
        # get that config file the same way:
        config_content = None
        scheme = store_url.split(':', 1)[0]
        if scheme in ['http', 'https']:
            try:
                config_content = download_url(
                    "{}{}config".format(
                        store_url,
                        '/' if not store_url.endswith('/') else ''))
            except DownloadError as e:
                lgr.debug("Failed to get config file from source:\n%s",
                          exc_str(e))
        elif scheme == 'ssh':
            # TODO: switch the following to proper command abstraction:
            # SSHRemoteIO ignores the path part ATM. No remote CWD! (To be
            # changed with command abstractions). So we need to get that part to
            # have a valid path to the remote's config file:
            cfg_path = PurePosixPath(URL(store_url).path) / 'config'
            op = SSHRemoteIO(store_url)
            try:
                config_content = op.read_file(cfg_path)
            except RIARemoteError as e:
                lgr.debug("Failed to get config file from source: %s",
                          exc_str(e))

        elif scheme == 'file':
            # TODO: switch the following to proper command abstraction:
            op = LocalIO()
            cfg_path = Path(URL(store_url).localpath) / 'config'
            try:
                config_content = op.read_file(cfg_path)
            except (RIARemoteError, OSError) as e:
                lgr.debug("Failed to get config file from source: %s",
                          exc_str(e))
        else:
            lgr.debug("Unknown URL-Scheme %s in %s. Can handle SSH, HTTP or "
                      "FILE scheme URLs.", scheme, props['source'])

        # And read it
        uuid = None
        if config_content:
            # TODO: We might be able to spare the saving to a file.
            #       "git config -f -" is not explicitly documented but happens
            #       to work and would read from stdin. Make sure we know this
            #       works for required git versions and on all platforms.
            with make_tempfile(content=config_content) as cfg_file:
                runner = GitWitlessRunner()
                try:
                    result = runner.run(
                        ['git', 'config', '-f', cfg_file,
                         'datalad.ora-remote.uuid'],
                        protocol=StdOutCapture
                    )
                    uuid = result['stdout'].strip()
                except CommandError as e:
                    # doesn't contain what we are looking for
                    lgr.debug("Found no UUID for ORA special remote at "
                              "'%s' (%s)", remote, exc_str(e))

        return uuid




    # chances are that if this dataset came from a RIA store, its subdatasets
    # may live there too. Place a subdataset source candidate config that makes
    # get probe this RIA store when obtaining subdatasets
    ria_store_url = props['source'].split('#', maxsplit=1)[0]
    ds.config.set(
        # we use the label 'origin' for this candidate in order to not have to
        # generate a complicated name from the actual source specification.
        # we pick a cost of 200 to sort it before datalad's default candidates
        # for non-RIA URLs, because they prioritize hierarchical layouts that
        # cannot be found in a RIA store
        'datalad.get.subdataset-source-candidate-200origin',
        # use the entire original URL, up to the fragment + plus dataset ID
        # placeholder, this should make things work with any store setup we
        # support (paths, ports, ...)
        ria_store_url + '#{id}',
        where='local')

    # setup publication dependency, if a corresponding special remote exists
    # and was enabled (there could be RIA stores that actually only have repos)
    # make this function be a generator
    ora_remotes = [s for s in ds.siblings('query', result_renderer='disabled')
                   if s.get('annex-externaltype') == 'ora']
    # get full special remotes' config for access to stored URL
    srs = repo.get_special_remotes() \
        if hasattr(repo, 'get_special_remotes') else dict()

    has_only_disabled_ora = \
        not ora_remotes and \
        any(r.get('externaltype') == 'ora' for r in srs.values())

    def match_in_urls(special_remote_cfg, url_to_match):
        # Figure whether either `url` or `push-url` in an ORA remote's config
        # match a given URL (to a RIA store).
        return special_remote_cfg['url'].startswith(url_to_match) or \
               (special_remote_cfg['push-url'].startswith(url_to_match)
                if 'push-url' in special_remote_cfg else False)

    no_enabled_ora_matches_url = \
        all(not match_in_urls(srs[r['annex-uuid']], ria_store_url)
            for r in ora_remotes)

    if has_only_disabled_ora or no_enabled_ora_matches_url:

        # No ORA remote autoenabled, but configuration known about at least one,
        # or enabled ORA remotes seem to not match clone URL.
        # Let's check the remote's config for datalad.ora-remote.uuid as stored
        # by create-sibling-ria and try enabling that one.
        lgr.debug("Found no autoenabled ORA special remote. Trying to look it "
                  "up in source config ...")

        org_uuid = get_uuid_from_store(props['giturl'])

        # Now, enable it. If annex-init didn't fail to enable it as stored, we
        # wouldn't end up here, so enable with store URL as suggested by the URL
        # we cloned from.
        if org_uuid:
            if org_uuid in srs.keys():
                # TODO: - Double-check autoenable value and only do this when
                #         true?
                #       - What if still fails? -> Annex shouldn't change config
                #         in that case

                # we only need the store:
                new_url = props['source'].split('#')[0]
                try:
                    repo.enable_remote(srs[org_uuid]['name'],
                                       options=['url={}'.format(new_url)]
                                       )
                    lgr.info("Reconfigured %s for %s",
                             srs[org_uuid]['name'], new_url)
                    # update ora_remotes for considering publication dependency
                    # below
                    ora_remotes = [s for s in
                                   ds.siblings('query',
                                               result_renderer='disabled')
                                   if s.get('annex-externaltype', None) ==
                                   'ora']
                except CommandError as e:
                    lgr.debug("Failed to reconfigure ORA special remote: %s",
                              exc_str(e))
            else:
                lgr.debug("Unknown ORA special remote uuid at '%s': %s",
                          remote, org_uuid)

    # Set publication dependency for `remote` on the respective ORA remote:
    if ora_remotes:
        url_matching_remotes = [r for r in ora_remotes
                                if srs[r['annex-uuid']]['url'] == ria_store_url]

        if len(url_matching_remotes) == 1:
            # We have exactly one ORA remote with the same store URL we used for
            # cloning (includes previously reconfigured remote).
            # Set publication dependency:
            yield from ds.siblings('configure',
                                   name=remote,
                                   publish_depends=url_matching_remotes[0]['name'],
                                   result_filter=None,
                                   result_renderer='disabled')

        elif not url_matching_remotes:
            # No matches but we have successfully autoenabled ORA remotes. Could
            # be the same store accessed by different method (cloning via HTTP
            # but special remote access via SSH). We can confidently set
            # publication dependency if the store knows the UUID.
            org_uuid = get_uuid_from_store(props['giturl'])
            uuid_matching_remotes = [r for r in ora_remotes
                                     if r['annex-uuid'] == org_uuid]
            if uuid_matching_remotes:
                # Multiple uuid matches are actually possible via same-as.
                # However, in that case we can't decide which one is supposed to
                # be used with publishing to `remote`.
                if len(uuid_matching_remotes) == 1:
                    yield from ds.siblings(
                        'configure',
                        name=remote,
                        publish_depends=uuid_matching_remotes[0]['name'],
                        result_filter=None,
                        result_renderer='disabled')
                else:
                    lgr.warning(
                        "Found multiple matching ORA remotes. Couldn't decide "
                        "which one publishing to '%s' should depend on: %s."
                        " Consider running 'datalad siblings configure -s "
                        "%s --publish-depends ORAREMOTENAME' to set "
                        "publication dependency manually.",
                        remote,
                        [r['name'] for r in uuid_matching_remotes],
                        remote)

        else:
            # We have multiple ORA remotes with the same store URL we cloned
            # from.
            lgr.warning("Found multiple matching ORA remotes. Couldn't decide "
                        "which one publishing to '%s' should depend on: %s."
                        " Consider running 'datalad siblings configure -s "
                        "%s --publish-depends ORAREMOTENAME' to set "
                        "publication dependency manually.",
                        remote,
                        [r['name'] for r in url_matching_remotes],
                        remote)


def postclonecfg_annexdataset(ds, reckless, description=None, remote="origin"):
    """If ds "knows annex" -- annex init it, set into reckless etc

    Provides additional tune up to a possibly an annex repo, e.g.
    "enables" reckless mode, sets up description
    """
    # in any case check whether we need to annex-init the installed thing:
    if not knows_annex(ds.path):
        # not for us
        return

    # init annex when traces of a remote annex can be detected
    if reckless == 'auto':
        lgr.debug(
            "Instruct annex to hardlink content in %s from local "
            "sources, if possible (reckless)", ds.path)
        ds.config.set(
            'annex.hardlink', 'true', where='local', reload=True)

    lgr.debug("Initializing annex repo at %s", ds.path)
    # Note, that we cannot enforce annex-init via AnnexRepo().
    # If such an instance already exists, its __init__ will not be executed.
    # Therefore do quick test once we have an object and decide whether to call
    # its _init().
    #
    # Additionally, call init if we need to add a description (see #1403),
    # since AnnexRepo.__init__ can only do it with create=True
    repo = AnnexRepo(ds.path, init=True)
    if not repo.is_initialized() or description:
        repo._init(description=description)
    if reckless == 'auto' or (reckless and reckless.startswith('shared-')):
        repo.call_annex(['untrust', 'here'])

    elif reckless == 'ephemeral':
        # with ephemeral we declare 'here' as 'dead' right away, whenever
        # we symlink the remote's annex, since availability from 'here' should
        # not be propagated for an ephemeral clone when we publish back to
        # the remote.
        # This will cause stuff like this for a locally present annexed file:
        # % git annex whereis d1
        # whereis d1 (0 copies) failed
        # BUT this works:
        # % git annex find . --not --in here
        # % git annex find . --in here
        # d1

        # we don't want annex copy-to <remote>
        ds.config.set(
            f'remote.{remote}.annex-ignore', 'true',
            where='local')

        ds.repo.set_remote_dead('here')

        if check_symlink_capability(ds.repo.dot_git / 'dl_link_test',
                                    ds.repo.dot_git / 'dl_target_test'):
            # symlink the annex to avoid needless copies in an ephemeral clone
            annex_dir = ds.repo.dot_git / 'annex'
            origin_annex_url = ds.config.get(f"remote.{remote}.url", None)
            origin_git_path = None
            if origin_annex_url:
                try:
                    # Deal with file:// scheme URLs as well as plain paths.
                    # If origin isn't local, we have nothing to do.
                    origin_git_path = Path(RI(origin_annex_url).localpath)

                    # we are local; check for a bare repo first to not mess w/
                    # the path
                    if GitRepo(origin_git_path, create=False).bare:
                        # origin is a bare repo -> use path as is
                        pass
                    elif origin_git_path.name != '.git':
                        origin_git_path /= '.git'
                except ValueError:
                    # Note, that accessing localpath on a non-local RI throws
                    # ValueError rather than resulting in an AttributeError.
                    # TODO: Warning level okay or is info level sufficient?
                    # Note, that setting annex-dead is independent of
                    # symlinking .git/annex. It might still make sense to
                    # have an ephemeral clone that doesn't propagate its avail.
                    # info. Therefore don't fail altogether.
                    lgr.warning("reckless=ephemeral mode: %s doesn't seem "
                                "local: %s\nno symlinks being used",
                                remote, origin_annex_url)
            if origin_git_path:
                # TODO make sure that we do not delete any unique data
                rmtree(str(annex_dir)) \
                    if not annex_dir.is_symlink() else annex_dir.unlink()
                annex_dir.symlink_to(origin_git_path / 'annex',
                                     target_is_directory=True)
        else:
            # TODO: What level? + note, that annex-dead is independent
            lgr.warning("reckless=ephemeral mode: Unable to create symlinks on "
                        "this file system.")

    srs = {True: [], False: []}  # special remotes by "autoenable" key
    remote_uuids = None  # might be necessary to discover known UUIDs

    repo_config = repo.config
    # Note: The purpose of this function is to inform the user. So if something
    # looks misconfigured, we'll warn and move on to the next item.
    for uuid, config in repo.get_special_remotes().items():
        sr_name = config.get('name', None)
        if sr_name is None:
            lgr.warning(
                'Ignoring special remote %s because it does not have a name. '
                'Known information: %s',
                uuid, config)
            continue
        sr_autoenable = config.get('autoenable', False)
        try:
            sr_autoenable = ensure_bool(sr_autoenable)
        except ValueError:
            lgr.warning(
                'Failed to process "autoenable" value %r for sibling %s in '
                'dataset %s as bool.'
                'You might need to enable it later manually and/or fix it up to'
                ' avoid this message in the future.',
                sr_autoenable, sr_name, ds.path)
            continue

        # If it looks like a type=git special remote, make sure we have up to
        # date information. See gh-2897.
        if sr_autoenable and repo_config.get("remote.{}.fetch".format(sr_name)):
            try:
                repo.fetch(remote=sr_name)
            except CommandError as exc:
                lgr.warning("Failed to fetch type=git special remote %s: %s",
                            sr_name, exc_str(exc))

        # determine whether there is a registered remote with matching UUID
        if uuid:
            if remote_uuids is None:
                remote_uuids = {
                    # Check annex-config-uuid first. For sameas annex remotes,
                    # this will point to the UUID for the configuration (i.e.
                    # the key returned by get_special_remotes) rather than the
                    # shared UUID.
                    (repo_config.get('remote.%s.annex-config-uuid' % r) or
                     repo_config.get('remote.%s.annex-uuid' % r))
                    for r in repo.get_remotes()
                }
            if uuid not in remote_uuids:
                srs[sr_autoenable].append(sr_name)

    if srs[True]:
        lgr.debug(
            "configuration for %s %s added because of autoenable,"
            " but no UUIDs for them yet known for dataset %s",
            # since we are only at debug level, we could call things their
            # proper names
            single_or_plural("special remote",
                             "special remotes", len(srs[True]), True),
            ", ".join(srs[True]),
            ds.path
        )

    if srs[False]:
        # if has no auto-enable special remotes
        lgr.info(
            'access to %s %s not auto-enabled, enable with:\n'
            '\t\tdatalad siblings -d "%s" enable -s %s',
            # but since humans might read it, we better confuse them with our
            # own terms!
            single_or_plural("dataset sibling",
                             "dataset siblings", len(srs[False]), True),
            ", ".join(srs[False]),
            ds.path,
            srs[False][0] if len(srs[False]) == 1 else "SIBLING",
        )

    # we have just cloned the repo, so it has a remote `remote`, configure any
    # reachable origin of origins
    yield from configure_origins(ds, ds, remote=remote)


_handle_possible_annex_dataset = postclonecfg_annexdataset


def configure_origins(cfgds, probeds, label=None, remote="origin"):
    """Configure any discoverable local dataset sibling as a remote

    Parameters
    ----------
    cfgds : Dataset
      Dataset to receive the remote configurations
    probeds : Dataset
      Dataset to start looking for `remote` remotes. May be identical with
      `cfgds`.
    label : int, optional
      Each discovered remote will be configured as a remote under the name
      '<remote>-<label>'. If no label is given, '2' will be used by default,
      given that there is typically a remote named `remote` already.
    remote : str, optional
      Name of the default remote on clone.
    """
    if label is None:
        label = 1
    # let's look at the URL for that remote and see if it is a local
    # dataset
    origin_url = probeds.config.get(f'remote.{remote}.url')
    if not origin_url:
        # no remote with default name, nothing to do
        return
    if not cfgds.config.obtain(
            'datalad.install.inherit-local-origin',
            default=True):
        # no inheritance wanted
        return
    if not isinstance(RI(origin_url), PathRI):
        # not local path
        return

    # no need to reconfigure original/direct remote again
    if cfgds != probeds:
        # prevent duplicates
        known_remote_urls = set(
            cfgds.config.get(r + '.url', None)
            for r in cfgds.config.sections()
            if r.startswith('remote.')
        )
        if origin_url not in known_remote_urls:
            yield from cfgds.siblings(
                'configure',
                # no chance for conflict, can only be the second configured
                # remote
                name='{}-{}'.format(remote, label),
                url=origin_url,
                # fetch to get all annex info
                fetch=True,
                result_renderer='disabled',
                on_failure='ignore',
            )
    # and dive deeper
    # given the clone source is a local dataset, we can have a
    # cheap look at it, and configure its own `remote` as a remote
    # (if there is any), and benefit from additional annex availability
    yield from configure_origins(
        cfgds,
        Dataset(probeds.pathobj / origin_url),
        label=label + 1,
        remote=remote)


def _get_tracking_source(ds):
    """Returns name and url of a potential configured source
    tracking remote"""
    vcs = ds.repo
    # if we have a remote, let's check the location of that remote
    # for the presence of the desired submodule

    remote_name, tracking_branch = vcs.get_tracking_branch()
    if not remote_name and isinstance(vcs, AnnexRepo):
        # maybe cloned from a source repo that was in adjusted mode
        # https://github.com/datalad/datalad/issues/3969
        remote_name, tracking_branch = vcs.get_tracking_branch(
            corresponding=False)
    # TODO: better default `None`? Check where we might rely on '':
    remote_url = ''
    if remote_name:
        remote_url = vcs.get_remote_url(remote_name, push=False)

    return remote_name, remote_url


def _get_installationpath_from_url(url):
    """Returns a relative path derived from the trailing end of a URL

    This can be used to determine an installation path of a Dataset
    from a URL, analog to what `git clone` does.
    """
    ri = RI(url)
    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL
        path = ri.path.rstrip('/')
        path = urlunquote(path) if path else ri.hostname
        if '/' in path:
            path = path.split('/')
            if path[-1] == '.git':
                path = path[-2]
            else:
                path = path[-1]
    else:
        path = Path(url).parts[-1]
    if path.endswith('.git'):
        path = path[:-4]
    return path


def decode_source_spec(spec, cfg=None):
    """Decode information from a clone source specification

    Parameters
    ----------
    spec : str
      Any supported clone source specification
    cfg : ConfigManager, optional
      Configuration will be queried from the instance (i.e. from a particular
      dataset). If None is given, the global DataLad configuration will be
      queried.

    Returns
    -------
    dict
      The value of each decoded property is stored under its own key in this
      dict. By default the following keys are return: 'type', a specification
      type label {'giturl', 'dataladri', 'ria'}; 'source' the original
      source specification; 'giturl' a URL for the source that is a suitable
      source argument for git-clone; 'version' a version-identifer, if present
      (None else); 'default_destpath' a relative path that that can be used as
      a clone destination.
    """
    if cfg is None:
        from datalad import cfg
    # standard property dict composition
    props = dict(
        source=spec,
        version=None,
    )

    # Git never gets to see these URLs, so let's manually apply any
    # rewrite configuration Git might know about.
    # Note: We need to rewrite before parsing, otherwise parsing might go wrong.
    # This is particularly true for insteadOf labels replacing even the URL
    # scheme.
    spec = cfg.rewrite_url(spec)
    # common starting point is a RI instance, support for accepting an RI
    # instance is kept for backward-compatibility reasons
    source_ri = RI(spec) if not isinstance(spec, RI) else spec

    # scenario switch, each case must set 'giturl' at the very minimum
    if isinstance(source_ri, DataLadRI):
        # we have got our DataLadRI as the source, so expand it
        props['type'] = 'dataladri'
        props['giturl'] = source_ri.as_git_url()
    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):
        # parse a RIA URI
        dsid, version = source_ri.fragment.split('@', maxsplit=1) \
            if '@' in source_ri.fragment else (source_ri.fragment, None)
        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'
        if re.match(uuid_regex, dsid):
            trace = '{}/{}'.format(dsid[:3], dsid[3:])
            default_destpath = dsid
        elif dsid.startswith('~'):
            trace = 'alias/{}'.format(dsid[1:])
            default_destpath = dsid[1:]
        else:
            raise ValueError(
                'RIA URI not recognized, no valid dataset ID or other supported '
                'scheme: {}'.format(spec))
        # now we cancel the fragment in the original URL, but keep everything else
        # in order to be able to support the various combinations of ports, paths,
        # and everything else
        source_ri.fragment = ''
        # strip the custom protocol and go with standard one
        source_ri.scheme = source_ri.scheme[4:]
        # take any existing path, and add trace to dataset within the store
        source_ri.path = '{urlpath}{urldelim}{trace}'.format(
            urlpath=source_ri.path if source_ri.path else '',
            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',
            trace=trace,
        )
        props.update(
            type='ria',
            giturl=str(source_ri),
            version=version,
            default_destpath=default_destpath,
        )
    else:
        # let's assume that anything else is a URI that Git can handle
        props['type'] = 'giturl'
        # use original input verbatim
        props['giturl'] = spec

    if 'default_destpath' not in props:
        # if we still have no good idea on where a dataset could be cloned to if no
        # path was given, do something similar to git clone and derive the path from
        # the source
        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])

    return props

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** Incorrect extraction of installation path from SSH URLs

**Description:**
When providing an SSH URL, the installation path is not correctly extracted, resulting in the path including the hostname instead of just the repository name.

**Example Code:**
```python
url = 'hostname:lastbit'
path = _get_installationpath_from_url(url)
print(path)  # Outputs: 'hostname:lastbit'
```

**Expected Behavior:**
The installation path should be extracted as `'lastbit'` from the SSH URL.

**Actual Behavior:**
The function returns `'hostname:lastbit'` instead of the expected `'lastbit'`.
[/ISSUE]
{
    "test_installationpath_from_url": "PASSED",
    "test_decode_source_spec": "PASSED",
    "test_url_mapping_specs": "PASSED",
    "test_invalid_args": "ERROR",
    "test_clone_crcns": "ERROR",
    "test_clone_datasets_root": "ERROR",
    "test_clone_simple_local": "ERROR",
    "test_clone_dataset_from_just_source": "ERROR",
    "test_clone_dataladri": "ERROR",
    "test_clone_isnot_recursive": "ERROR",
    "test_clone_into_dataset": "ERROR",
    "test_notclone_known_subdataset": "ERROR",
    "test_failed_clone": "ERROR",
    "test_clone_missing_commit": "ERROR",
    "test_install_source_relpath": "ERROR",
    "test_clone_isnt_a_smartass": "ERROR",
    "test_clone_report_permission_issue": "ERROR",
    "test_clone_autoenable_msg_handles_sameas": "ERROR",
    "test_expanduser": "ERROR",
    "test_local_url_with_fetch": "ERROR",
    "test_ria_http": "ERROR",
    "test_no_ria_postclonecfg": "ERROR",
    "test_ria_postclone_noannex": "ERROR",
    "test_inherit_src_candidates": "ERROR",
    "test_ephemeral": "ERROR",
    "test_fetch_git_special_remote": "ERROR",
    "test_clone_url_mapping": "ERROR",
    "test_autoenabled_remote_msg": "FAILED",
    "test_cfg_originorigin": "FAILED",
    "test_ria_http_storedataladorg": "FAILED",
    "test_clone_unborn_head": "FAILED",
    "test_clone_unborn_head_no_other_ref": "FAILED",
    "test_clone_unborn_head_sub": "FAILED",
    "test_gin_cloning": "FAILED",
    "test_nonuniform_adjusted_subdataset": "FAILED",
    "test_clone_recorded_subds_reset": "FAILED"
}
diff --git a/datalad/core/distributed/clone.py b/datalad/core/distributed/clone.py
index 6f1ff6778..a603a8681 100644
--- a/datalad/core/distributed/clone.py
+++ b/datalad/core/distributed/clone.py
@@ -47,13 +47,14 @@ from datalad.support.exceptions import DownloadError
 from datalad.support.param import Parameter
 from datalad.support.strings import get_replacement_dict
 from datalad.support.network import (
-    get_local_file_url,
-    download_url,
-    is_url,
-    URL,
-    RI,
     DataLadRI,
     PathRI,
+    RI,
+    SSHRI,
+    URL,
+    download_url,
+    get_local_file_url,
+    is_url,
 )
 from datalad.dochelpers import (
     exc_str,
@@ -1343,7 +1344,7 @@ def _get_installationpath_from_url(url):
     from a URL, analog to what `git clone` does.
     """
     ri = RI(url)
-    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL
+    if isinstance(ri, (URL, DataLadRI, SSHRI)):  # decode only if URL
         path = ri.path.rstrip('/')
         path = urlunquote(path) if path else ri.hostname
         if '/' in path:

{"datalad/core/distributed/clone.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Plumbing command for dataset installation\"\"\"\n\n\nimport logging\nimport re\nfrom os.path import expanduser\nfrom collections import OrderedDict\nfrom urllib.parse import unquote as urlunquote\n\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import eval_results\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.common_opts import (\n    location_description,\n    reckless_opt,\n)\nfrom datalad.log import log_progress\nfrom datalad.support.gitrepo import (\n    GitRepo,\n)\nfrom datalad.cmd import (\n    CommandError,\n    GitWitlessRunner,\n    StdOutCapture,\n)\nfrom datalad.distributed.ora_remote import (\n    LocalIO,\n    RIARemoteError,\n    SSHRemoteIO,\n)\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.constraints import (\n    EnsureNone,\n    EnsureStr,\n    EnsureKeyChoice,\n)\nfrom datalad.support.exceptions import DownloadError\nfrom datalad.support.param import Parameter\nfrom datalad.support.strings import get_replacement_dict\nfrom datalad.support.network import (\n    get_local_file_url,\n    download_url,\n    is_url,\n    URL,\n    RI,\n    DataLadRI,\n    PathRI,\n)\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\nfrom datalad.utils import (\n    ensure_bool,\n    ensure_list,\n    knows_annex,\n    make_tempfile,\n    Path,\n    PurePosixPath,\n    rmtree,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    resolve_path,\n    require_dataset,\n    EnsureDataset,\n)\nfrom datalad.distribution.utils import (\n    _get_flexible_source_candidates,\n)\nfrom datalad.utils import (\n    check_symlink_capability\n)\n\n__docformat__ = 'restructuredtext'\n\nlgr = logging.getLogger('datalad.core.distributed.clone')\n\n\n@build_doc\nclass Clone(Interface):\n    \"\"\"Obtain a dataset (copy) from a URL or local directory\n\n    The purpose of this command is to obtain a new clone (copy) of a dataset\n    and place it into a not-yet-existing or empty directory. As such `clone`\n    provides a strict subset of the functionality offered by `install`. Only a\n    single dataset can be obtained, and immediate recursive installation of\n    subdatasets is not supported. However, once a (super)dataset is installed\n    via `clone`, any content, including subdatasets can be obtained by a\n    subsequent `get` command.\n\n    Primary differences over a direct `git clone` call are 1) the automatic\n    initialization of a dataset annex (pure Git repositories are equally\n    supported); 2) automatic registration of the newly obtained dataset as a\n    subdataset (submodule), if a parent dataset is specified; 3) support\n    for additional resource identifiers (DataLad resource identifiers as used\n    on datasets.datalad.org, and RIA store URLs as used for store.datalad.org\n    - optionally in specific versions as identified by a branch or a tag; see\n    examples); and 4) automatic configurable generation of alternative access\n    URL for common cases (such as appending '.git' to the URL in case the\n    accessing the base URL failed).\n\n    In case the clone is registered as a subdataset, the original URL passed to\n    `clone` is recorded in `.gitmodules` of the parent dataset in addition\n    to the resolved URL used internally for git-clone. This allows to preserve\n    datalad specific URLs like ria+ssh://... for subsequent calls to `get` if\n    the subdataset was locally removed later on.\n\n    || PYTHON >>By default, the command returns a single Dataset instance for\n    an installed dataset, regardless of whether it was newly installed ('ok'\n    result), or found already installed from the specified source ('notneeded'\n    result).<< PYTHON ||\n\n    URL mapping configuration\n\n    'clone' supports the transformation of URLs via (multi-part) substitution\n    specifications. A substitution specification is defined as a configuration\n    setting 'datalad.clone.url-substition.<seriesID>' with a string containing\n    a match and substitution expression, each following Python's regular\n    expression syntax. Both expressions are concatenated to a single string\n    with an arbitrary delimiter character. The delimiter is defined by\n    prefixing the string with the delimiter. Prefix and delimiter are stripped\n    from the expressions (Example: \",^http://(.*)$,https://\\\\1\").  This setting\n    can be defined multiple times, using the same '<seriesID>'.  Substitutions\n    in a series will be applied incrementally, in order of their definition.\n    The first substitution in such a series must match, otherwise no further\n    substitutions in a series will be considered. However, following the first\n    match all further substitutions in a series are processed, regardless\n    whether intermediate expressions match or not. Substitution series themselves\n    have no particular order, each matching series will result in a candidate\n    clone URL. Consequently, the initial match specification in a series should\n    be as precise as possible to prevent inflation of candidate URLs.\n\n    .. seealso::\n\n      :ref:`handbook:3-001`\n        More information on Remote Indexed Archive (RIA) stores\n    \"\"\"\n    # by default ignore everything but install results\n    # i.e. no \"add to super dataset\"\n    result_filter = EnsureKeyChoice('action', ('install',))\n    # very frequently this command will yield exactly one installed dataset\n    # spare people the pain of going through a list by default\n    return_type = 'item-or-list'\n    # as discussed in #1409 and #1470, we want to return dataset instances\n    # matching what is actually available after command completion (and\n    # None for any failed dataset installation)\n    result_xfm = 'successdatasets-or-none'\n\n    _examples_ = [\n        dict(text=\"Install a dataset from Github into the current directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install a dataset into a specific directory\",\n             code_py=\"\"\"\\\n             clone(source='https://github.com/datalad-datasets/longnow-podcasts.git',\n                   path='myfavpodcasts')\"\"\",\n             code_cmd=\"\"\"\\\n             datalad clone https://github.com/datalad-datasets/longnow-podcasts.git \\\\\n             myfavpodcasts\"\"\"),\n        dict(text=\"Install a dataset as a subdataset into the current dataset\",\n             code_py=\"\"\"\\\n             clone(dataset='.',\n                   source='https://github.com/datalad-datasets/longnow-podcasts.git')\"\"\",\n             code_cmd=\"datalad clone -d . \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install the main superdataset from datasets.datalad.org\",\n             code_py=\"clone(source='///')\",\n             code_cmd=\"datalad clone ///\"),\n        dict(text=\"Install a dataset identified by a literal alias from store.datalad.org\",\n             code_py=\"clone(source='ria+http://store.datalad.org#~hcp-openaccess')\",\n             code_cmd=\"datalad clone ria+http://store.datalad.org#~hcp-openaccess\"),\n        dict(\n            text=\"Install a dataset in a specific version as identified by a \"\n                 \"branch or tag name from store.datalad.org\",\n            code_py=\"clone(source='ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier')\",\n            code_cmd=\"datalad clone ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier\"),\n        dict(\n            text=\"Install a dataset with group-write access permissions\",\n            code_py=\\\n            \"clone(source='http://example.com/dataset', reckless='shared-group')\",\n            code_cmd=\\\n            \"datalad clone http://example.com/dataset --reckless shared-group\"),\n    ]\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"(parent) dataset to clone into. If given, the newly cloned\n            dataset is registered as a subdataset of the parent. Also, if given,\n            relative paths are interpreted as being relative to the parent\n            dataset, and not relative to the working directory.\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        source=Parameter(\n            args=(\"source\",),\n            metavar='SOURCE',\n            doc=\"\"\"URL, DataLad resource identifier, local path or instance of\n            dataset to be cloned\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            nargs=\"?\",\n            doc=\"\"\"path to clone into.  If no `path` is provided a\n            destination path will be derived from a source URL\n            similar to :command:`git clone`\"\"\"),\n        description=location_description,\n        reckless=reckless_opt,\n    )\n\n    @staticmethod\n    @datasetmethod(name='clone')\n    @eval_results\n    def __call__(\n            source,\n            path=None,\n            dataset=None,\n            description=None,\n            reckless=None):\n        # did we explicitly get a dataset to install into?\n        # if we got a dataset, path will be resolved against it.\n        # Otherwise path will be resolved first.\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='clone') \\\n            if dataset is not None else dataset\n        refds_path = ds.path if ds else None\n\n        # legacy compatibility\n        if reckless is True:\n            # so that we can forget about how things used to be\n            reckless = 'auto'\n\n        if isinstance(source, Dataset):\n            source = source.path\n\n        if source == path:\n            # even if they turn out to be identical after resolving symlinks\n            # and more sophisticated witchcraft, it would still happily say\n            # \"it appears to be already installed\", so we just catch an\n            # obviously pointless input combination\n            raise ValueError(\n                \"clone `source` and destination `path` are identical [{}]. \"\n                \"If you are trying to add a subdataset simply use `save`\".format(\n                    path))\n\n        if path is not None:\n            path = resolve_path(path, dataset)\n\n        # derive target from source:\n        if path is None:\n            # we got nothing but a source. do something similar to git clone\n            # and derive the path from the source and continue\n            # since this is a relative `path`, resolve it:\n            # we are not going to reuse the decoded URL, as this is done for\n            # all source candidates in clone_dataset(), we just use to determine\n            # a destination path here in order to perform a bunch of additional\n            # checks that shall not pollute the helper function\n            source_ = decode_source_spec(\n                source, cfg=None if ds is None else ds.config)\n            path = resolve_path(source_['default_destpath'], dataset)\n            lgr.debug(\"Determined clone target path from source\")\n        lgr.debug(\"Resolved clone target path to: '%s'\", path)\n\n        # there is no other way -- my intoxicated brain tells me\n        assert(path is not None)\n\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            refds=refds_path,\n            source_url=source)\n\n        try:\n            # this will implicitly cause pathlib to run a bunch of checks\n            # whether the present path makes any sense on the platform\n            # we are running on -- we don't care if the path actually\n            # exists at this point, but we want to abort early if the path\n            # spec is determined to be useless\n            path.exists()\n        except OSError as e:\n            yield get_status_dict(\n                status='error',\n                path=path,\n                message=('cannot handle target path: %s', exc_str(e)),\n                **result_props)\n            return\n\n        destination_dataset = Dataset(path)\n        result_props['ds'] = destination_dataset\n\n        if ds is not None and ds.pathobj not in path.parents:\n            yield get_status_dict(\n                status='error',\n                message=(\"clone target path '%s' not in specified target dataset '%s'\",\n                         path, ds),\n                **result_props)\n            return\n\n        # perform the actual cloning operation\n        yield from clone_dataset(\n            [source],\n            destination_dataset,\n            reckless,\n            description,\n            result_props,\n            cfg=None if ds is None else ds.config,\n        )\n\n        # TODO handle any 'version' property handling and verification using a\n        # dedicated public helper\n\n        if ds is not None:\n            # we created a dataset in another dataset\n            # -> make submodule\n            actually_saved_subds = False\n            for r in ds.save(\n                    path,\n                    # Note, that here we know we don't save anything but a new\n                    # subdataset. Hence, don't go with default commit message,\n                    # but be more specific.\n                    message=\"[DATALAD] Added subdataset\",\n                    return_type='generator',\n                    result_filter=None,\n                    result_xfm=None,\n                    on_failure='ignore'):\n                actually_saved_subds = actually_saved_subds or (\n                        r['action'] == 'save' and\n                        r['type'] == 'dataset' and\n                        r['refds'] == ds.path and\n                        r['status'] == 'ok')\n                yield r\n\n            # Modify .gitmodules to contain originally given url. This is\n            # particularly relevant for postclone routines on a later `get`\n            # for that subdataset. See gh-5256.\n            if actually_saved_subds:\n                # New subdataset actually saved. Amend the modification\n                # of .gitmodules. Note, that we didn't allow to deviate\n                # from git default behavior WRT a submodule's name vs\n                # its path when we made this a new subdataset.\n                subds_name = path.relative_to(ds.pathobj)\n                ds.repo.call_git(\n                    ['config',\n                     '--file',\n                     '.gitmodules',\n                     '--replace-all',\n                     'submodule.{}.{}'.format(subds_name,\n                                              \"datalad-url\"),\n                     source]\n                )\n                yield from ds.save('.gitmodules',\n                                   amend=True, to_git=True)\n            else:\n                # We didn't really commit. Just call `subdatasets`\n                # in that case to have the modification included in the\n                # post-clone state (whatever that may be).\n                ds.subdatasets(path, set_property=[(\"datalad-url\", source)])\n\n\ndef _get_url_mappings(cfg):\n    cfg_prefix = 'datalad.clone.url-substitute.'\n    # figure out which keys we should be looking for\n    # in the active config\n    subst_keys = set(k for k in cfg.keys() if k.startswith(cfg_prefix))\n    # and in the common config specs\n    from datalad.interface.common_cfg import definitions\n    subst_keys.update(k for k in definitions if k.startswith(cfg_prefix))\n    # TODO a potential sorting of substitution series could be implemented\n    # here\n    return [\n        # decode the rule specifications\n        get_replacement_dict(\n            # one or more could come out\n            ensure_list(\n                cfg.get(\n                    k,\n                    # make sure to pull the default from the common config\n                    default=cfg.obtain(k),\n                    # we specifically support declaration of multiple\n                    # settings to build replacement chains\n                    get_all=True)))\n        for k in subst_keys\n    ]\n\n\ndef _map_urls(cfg, urls):\n    mapping_specs = _get_url_mappings(cfg)\n    if not mapping_specs:\n        return urls\n\n    mapped = []\n    # we process the candidate in order to maintain any prioritization\n    # encoded in it (e.g. _get_flexible_source_candidates_for_submodule)\n    # if we have a matching mapping replace the URL in its position\n    for u in urls:\n        # we only permit a single match\n        # TODO we likely want to RF this to pick the longest match\n        mapping_applied = False\n        # try one mapping set at a time\n        for mapping_spec in mapping_specs:\n            # process all substitution patterns in the specification\n            # always operate on strings (could be a Path instance too)\n            mu = str(u)\n            matched = False\n            for match_ex, subst_ex in mapping_spec.items():\n                if not matched:\n                    matched = re.match(match_ex, mu) is not None\n                if not matched:\n                    break\n                # try to map, would return unchanged, if there is no match\n                mu = re.sub(match_ex, subst_ex, mu)\n            if mu != u:\n                lgr.debug(\"URL substitution: '%s' -> '%s'\", u, mu)\n                mapped.append(mu)\n                # we could consider breaking after the for effective mapping\n                # specification. however, that would mean any generic\n                # definition of a broadly matching substitution would derail\n                # the entroe system. moreover, suddently order would matter\n                # substantially\n                mapping_applied = True\n        if not mapping_applied:\n            # none of the mappings matches, go with the original URL\n            # (really original, not the stringified one)\n            mapped.append(u)\n    return mapped\n\n\ndef clone_dataset(\n        srcs,\n        destds,\n        reckless=None,\n        description=None,\n        result_props=None,\n        cfg=None,\n        checkout_gitsha=None):\n    \"\"\"Internal helper to perform cloning without sanity checks (assumed done)\n\n    This helper does not handle any saving of subdataset modification or adding\n    in a superdataset.\n\n    Parameters\n    ----------\n    srcs : list\n      Any suitable clone source specifications (paths, URLs)\n    destds : Dataset\n      Dataset instance for the clone destination\n    reckless : {None, 'auto', 'ephemeral', 'shared-...'}, optional\n      Mode switch to put cloned dataset into unsafe/throw-away configurations, i.e.\n      sacrifice data safety for performance or resource footprint. When None\n      and `cfg` is specified, use the value of `datalad.clone.reckless`.\n    description : str, optional\n      Location description for the annex of the dataset clone (if there is any).\n    result_props : dict, optional\n      Default properties for any yielded result, passed on to get_status_dict().\n    cfg : ConfigManager, optional\n      Configuration for parent dataset. This will be queried instead\n      of the global DataLad configuration.\n    checkout_gitsha : str, optional\n      If given, a specific commit, identified by shasum, will be checked out after\n      cloning. A dedicated follow-up fetch will be performed, if the initial clone\n      did not obtain the commit object. Should the checkout of the target commit\n      cause a detached HEAD, the previously active branch will be reset to the\n      target commit.\n\n    Yields\n    ------\n    dict\n      DataLad result records\n    \"\"\"\n    if not result_props:\n        # in case the caller had no specific idea on how results should look\n        # like, provide sensible defaults\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            ds=destds,\n        )\n\n    if reckless is None and cfg:\n        # if reckless is not explicitly given, but we operate on a\n        # superdataset, query whether it has been instructed to operate\n        # in a reckless mode, and inherit it for the coming clone\n        reckless = cfg.get('datalad.clone.reckless', None)\n\n    dest_path = destds.pathobj\n\n    # check for configured URL mappings, either in the given config manager\n    # or in the one of the destination dataset, which is typically not existent\n    # yet and the process config manager is then used effectively\n    srcs = _map_urls(cfg or destds.config, srcs)\n\n    # decode all source candidate specifications\n    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]\n\n    # now expand the candidate sources with additional variants of the decoded\n    # giturl, while duplicating the other properties in the additional records\n    # for simplicity. The hope is to overcome a few corner cases and be more\n    # robust than git clone\n    candidate_sources = [\n        dict(props, giturl=s) for props in candidate_sources\n        for s in _get_flexible_source_candidates(props['giturl'])\n    ]\n\n    # important test! based on this `rmtree` will happen below after failed clone\n    dest_path_existed = dest_path.exists()\n    if dest_path_existed and any(dest_path.iterdir()):\n        if destds.is_installed():\n            # check if dest was cloned from the given source before\n            # this is where we would have installed this from\n            # this is where it was actually installed from\n            track_name, track_url = _get_tracking_source(destds)\n            try:\n                # this will get us track_url in system native path conventions,\n                # whenever it is a path (and not a URL)\n                # this is needed to match it to any potentially incoming local\n                # source path in the 'notneeded' test below\n                track_path = str(Path(track_url))\n            except Exception:\n                # this should never happen, because Path() will let any non-path stringification\n                # pass through unmodified, but we do not want any potential crash due to\n                # pathlib behavior changes\n                lgr.debug(\"Unexpected behavior of pathlib!\")\n                track_path = None\n            for cand in candidate_sources:\n                src = cand['giturl']\n                if track_url == src \\\n                        or (not is_url(track_url)\n                            and get_local_file_url(track_url, compatibility='git') == src) \\\n                        or track_path == expanduser(src):\n                    yield get_status_dict(\n                        status='notneeded',\n                        message=(\"dataset %s was already cloned from '%s'\",\n                                 destds,\n                                 src),\n                        **result_props)\n                    return\n        # anything else is an error\n        yield get_status_dict(\n            status='error',\n            message='target path already exists and not empty, refuse to clone into target path',\n            **result_props)\n        return\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Cloning dataset to %s', destds,\n        total=len(candidate_sources),\n        label='Clone attempt',\n        unit=' Candidate locations',\n    )\n    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url\n    for cand in candidate_sources:\n        log_progress(\n            lgr.info,\n            'cloneds',\n            'Attempting to clone from %s to %s', cand['giturl'], dest_path,\n            update=1,\n            increment=True)\n\n        clone_opts = {}\n\n        if cand.get('version', None):\n            clone_opts['branch'] = cand['version']\n        try:\n            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35\n            # doesn't make it happen seamlessly\n            GitRepo.clone(\n                path=str(dest_path),\n                url=cand['giturl'],\n                clone_options=clone_opts,\n                create=True)\n\n        except CommandError as e:\n            e_stderr = e.stderr\n\n            error_msgs[cand['giturl']] = e\n            lgr.debug(\"Failed to clone from URL: %s (%s)\",\n                      cand['giturl'], exc_str(e))\n            if dest_path.exists():\n                lgr.debug(\"Wiping out unsuccessful clone attempt at: %s\",\n                          dest_path)\n                # We must not just rmtree since it might be curdir etc\n                # we should remove all files/directories under it\n                # TODO stringification can be removed once patlib compatible\n                # or if PY35 is no longer supported\n                rmtree(str(dest_path), children_only=dest_path_existed)\n\n            if e_stderr and 'could not create work tree' in e_stderr.lower():\n                # this cannot be fixed by trying another URL\n                re_match = re.match(r\".*fatal: (.*)$\", e_stderr,\n                                    flags=re.MULTILINE | re.DOTALL)\n                # cancel progress bar\n                log_progress(\n                    lgr.info,\n                    'cloneds',\n                    'Completed clone attempts for %s', destds\n                )\n                yield get_status_dict(\n                    status='error',\n                    message=re_match.group(1).strip()\n                    if re_match else \"stderr: \" + e_stderr,\n                    **result_props)\n                return\n            # next candidate\n            continue\n\n        result_props['source'] = cand\n        # do not bother with other sources if succeeded\n        break\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Completed clone attempts for %s', destds\n    )\n\n    if not destds.is_installed():\n        if len(error_msgs):\n            if all(not e.stdout and not e.stderr for e in error_msgs.values()):\n                # there is nothing we can learn from the actual exception,\n                # the exit code is uninformative, the command is predictable\n                error_msg = \"Failed to clone from all attempted sources: %s\"\n                error_args = list(error_msgs.keys())\n            else:\n                error_msg = \"Failed to clone from any candidate source URL. \" \\\n                            \"Encountered errors per each url were:\\n- %s\"\n                error_args = '\\n- '.join(\n                    '{}\\n  {}'.format(url, exc_str(exc))\n                    for url, exc in error_msgs.items()\n                )\n        else:\n            # yoh: Not sure if we ever get here but I felt that there could\n            #      be a case when this might happen and original error would\n            #      not be sufficient to troubleshoot what is going on.\n            error_msg = \"Awkward error -- we failed to clone properly. \" \\\n                        \"Although no errors were encountered, target \" \\\n                        \"dataset at %s seems to be not fully installed. \" \\\n                        \"The 'succesful' source was: %s\"\n            error_args = (destds.path, cand['giturl'])\n        yield get_status_dict(\n            status='error',\n            message=(error_msg, error_args),\n            **result_props)\n        return\n\n    dest_repo = destds.repo\n\n    remotes = dest_repo.get_remotes(with_urls_only=True)\n    nremotes = len(remotes)\n    if nremotes == 1:\n        remote = remotes[0]\n        lgr.debug(\"Determined %s to be remote of %s\", remote, destds)\n    elif remotes > 1:\n        lgr.warning(\n            \"Fresh clone %s unexpected has multiple remotes: %s. Using %s\",\n            destds.path, remotes, remotes[0])\n        remote = remotes[0]\n    else:\n        raise RuntimeError(\"bug: fresh clone has zero remotes\")\n\n    if not cand.get(\"version\"):\n        postclone_check_head(destds, remote=remote)\n\n    # act on --reckless=shared-...\n    # must happen prior git-annex-init, where we can cheaply alter the repo\n    # setup through safe re-init'ing\n    if reckless and reckless.startswith('shared-'):\n        lgr.debug('Reinit %s to enable shared access permissions', destds)\n        destds.repo.call_git(['init', '--shared={}'.format(reckless[7:])])\n\n    # In case of RIA stores we need to prepare *before* annex is called at all\n    if result_props['source']['type'] == 'ria':\n        postclone_preannex_cfg_ria(destds, remote=remote)\n\n    yield from postclonecfg_annexdataset(\n        destds,\n        reckless,\n        description,\n        remote=remote)\n\n    if checkout_gitsha and \\\n       dest_repo.get_hexsha(dest_repo.get_corresponding_branch()) != checkout_gitsha:\n        try:\n            postclone_checkout_commit(dest_repo, checkout_gitsha,\n                                      remote=remote)\n        except Exception as e:\n            yield get_status_dict(\n                status='error',\n                message=str(e),\n                **result_props,\n            )\n\n            # We were supposed to clone a particular version but failed to.\n            # This is particularly pointless in case of subdatasets and\n            # potentially fatal with current implementation of recursion.\n            # see gh-5387\n            lgr.debug(\"Failed to checkout %s, removing this clone attempt at %s\", checkout_gitsha, dest_path)\n            # TODO stringification can be removed once pathlib compatible\n            # or if PY35 is no longer supported\n            rmtree(str(dest_path), children_only=dest_path_existed)\n            return\n\n    # perform any post-processing that needs to know details of the clone\n    # source\n    if result_props['source']['type'] == 'ria':\n        yield from postclonecfg_ria(destds, result_props['source'],\n                                    remote=remote)\n\n    if reckless:\n        # store the reckless setting in the dataset to make it\n        # known to later clones of subdatasets via get()\n        destds.config.set(\n            'datalad.clone.reckless', reckless,\n            where='local',\n            reload=True)\n    else:\n        # We would still want to reload configuration to ensure that any of the\n        # above git invocations could have potentially changed the config\n        # TODO: might no longer be necessary if 0.14.0 adds reloading upon\n        # non-readonly commands invocation\n        destds.config.reload()\n\n    # yield successful clone of the base dataset now, as any possible\n    # subdataset clone down below will not alter the Git-state of the\n    # parent\n    yield get_status_dict(status='ok', **result_props)\n\n\ndef postclone_checkout_commit(repo, target_commit, remote=\"origin\"):\n    \"\"\"Helper to check out a specific target commit in a fresh clone.\n\n    Will not check (again) if current commit and target commit are already\n    the same!\n    \"\"\"\n    # record what branch we were on right after the clone\n    active_branch = repo.get_active_branch()\n    corr_branch = repo.get_corresponding_branch(branch=active_branch)\n    was_adjusted = bool(corr_branch)\n    repo_orig_branch = corr_branch or active_branch\n    # if we are on a branch this hexsha will be the tip of that branch\n    repo_orig_hexsha = repo.get_hexsha(repo_orig_branch)\n    # make sure we have the desired commit locally\n    # expensive and possibly error-prone fetch conditional on cheap\n    # local check\n    if not repo.commit_exists(target_commit):\n        try:\n            repo.fetch(remote=remote, refspec=target_commit)\n        except CommandError:\n            pass\n        # instead of inspecting the fetch results for possible ways\n        # with which it could failed to produced the desired result\n        # let's verify the presence of the commit directly, we are in\n        # expensive-land already anyways\n        if not repo.commit_exists(target_commit):\n            # there is nothing we can do about this\n            # MIH thinks that removing the clone is not needed, as a likely\n            # next step will have to be a manual recovery intervention\n            # and not another blind attempt\n            raise ValueError(\n                'Target commit %s does not exist in the clone, and '\n                'a fetch that commit from remote failed'\n                % target_commit[:8])\n    # checkout the desired commit\n    repo.call_git(['checkout', target_commit])\n    # did we detach?\n    if repo_orig_branch and not repo.get_active_branch():\n        # trace if current state is a predecessor of the branch_hexsha\n        lgr.debug(\n            \"Detached HEAD after resetting worktree of %s \"\n            \"(original branch: %s)\", repo, repo_orig_branch)\n        if repo.get_merge_base(\n                [repo_orig_hexsha, target_commit]) == target_commit:\n            # we assume the target_commit to be from the same branch,\n            # because it is an ancestor -- update that original branch\n            # to point to the target_commit, and update HEAD to point to\n            # that location\n            lgr.info(\n                \"Reset branch '%s' to %s (from %s) to \"\n                \"avoid a detached HEAD\",\n                repo_orig_branch, target_commit[:8], repo_orig_hexsha[:8])\n            branch_ref = 'refs/heads/%s' % repo_orig_branch\n            repo.update_ref(branch_ref, target_commit)\n            repo.update_ref('HEAD', branch_ref, symbolic=True)\n            if was_adjusted:\n                # Note: The --force is needed because the adjust branch already\n                # exists.\n                repo.adjust(options=[\"--unlock\", \"--force\"])\n        else:\n            lgr.warning(\n                \"%s has a detached HEAD, because the target commit \"\n                \"%s has no unique ancestor with branch '%s'\",\n                repo, target_commit[:8], repo_orig_branch)\n\n\ndef postclone_check_head(ds, remote=\"origin\"):\n    repo = ds.repo\n    if not repo.commit_exists(\"HEAD\"):\n        # HEAD points to an unborn branch. A likely cause of this is that the\n        # remote's main branch is something other than master but HEAD wasn't\n        # adjusted accordingly.\n        #\n        # Let's choose the most recently updated remote ref (according to\n        # commit date). In the case of a submodule, switching to a ref with\n        # commits prevents .update_submodule() from failing. It is likely that\n        # the ref includes the registered commit, but we don't have the\n        # information here to know for sure. If it doesn't, .update_submodule()\n        # will check out a detached HEAD.\n        remote_branches = (\n            b[\"refname:strip=2\"] for b in repo.for_each_ref_(\n                fields=\"refname:strip=2\", sort=\"-committerdate\",\n                pattern=\"refs/remotes/\" + remote))\n        for rbranch in remote_branches:\n            if rbranch in [remote + \"/git-annex\", \"HEAD\"]:\n                continue\n            if rbranch.startswith(remote + \"/adjusted/\"):\n                # If necessary for this file system, a downstream\n                # git-annex-init call will handle moving into an\n                # adjusted state.\n                continue\n            repo.call_git([\"checkout\", \"-b\",\n                           rbranch[len(remote) + 1:],  # drop \"<remote>/\"\n                           \"--track\", rbranch])\n            lgr.debug(\"Checked out local branch from %s\", rbranch)\n            return\n        lgr.warning(\"Cloned %s but could not find a branch \"\n                    \"with commits\", ds.path)\n\n\ndef postclone_preannex_cfg_ria(ds, remote=\"origin\"):\n\n    # We need to annex-ignore the remote before annex-init is called on the clone,\n    # due to issues 5186 and 5253 (and we would have done it afterwards anyway).\n    # annex/objects in RIA stores is special for several reasons.\n    # 1. the remote doesn't know about it (no actual local annex for the remote)\n    # 2. RIA may use hashdir mixed, copying data to it via git-annex (if cloned\n    #    via ssh or local) would make it see a bare repo and establish a\n    #    hashdir lower annex object tree.\n    # 3. We want the ORA remote to receive all data for the store, so its\n    #    objects could be moved into archives (the main point of a RIA store).\n\n    # Note, that this function might need an enhancement as theoretically a RIA\n    # store could also hold simple standard annexes w/o an intended ORA remote.\n    # This needs the introduction of a new version label in RIA datasets, making\n    # the following call conditional.\n    ds.config.set(f'remote.{remote}.annex-ignore', 'true', where='local')\n\n\ndef postclonecfg_ria(ds, props, remote=\"origin\"):\n    \"\"\"Configure a dataset freshly cloned from a RIA store\"\"\"\n    repo = ds.repo\n\n    def get_uuid_from_store(store_url):\n        # First figure whether we cloned via SSH, HTTP or local path and then\n        # get that config file the same way:\n        config_content = None\n        scheme = store_url.split(':', 1)[0]\n        if scheme in ['http', 'https']:\n            try:\n                config_content = download_url(\n                    \"{}{}config\".format(\n                        store_url,\n                        '/' if not store_url.endswith('/') else ''))\n            except DownloadError as e:\n                lgr.debug(\"Failed to get config file from source:\\n%s\",\n                          exc_str(e))\n        elif scheme == 'ssh':\n            # TODO: switch the following to proper command abstraction:\n            # SSHRemoteIO ignores the path part ATM. No remote CWD! (To be\n            # changed with command abstractions). So we need to get that part to\n            # have a valid path to the remote's config file:\n            cfg_path = PurePosixPath(URL(store_url).path) / 'config'\n            op = SSHRemoteIO(store_url)\n            try:\n                config_content = op.read_file(cfg_path)\n            except RIARemoteError as e:\n                lgr.debug(\"Failed to get config file from source: %s\",\n                          exc_str(e))\n\n        elif scheme == 'file':\n            # TODO: switch the following to proper command abstraction:\n            op = LocalIO()\n            cfg_path = Path(URL(store_url).localpath) / 'config'\n            try:\n                config_content = op.read_file(cfg_path)\n            except (RIARemoteError, OSError) as e:\n                lgr.debug(\"Failed to get config file from source: %s\",\n                          exc_str(e))\n        else:\n            lgr.debug(\"Unknown URL-Scheme %s in %s. Can handle SSH, HTTP or \"\n                      \"FILE scheme URLs.\", scheme, props['source'])\n\n        # And read it\n        uuid = None\n        if config_content:\n            # TODO: We might be able to spare the saving to a file.\n            #       \"git config -f -\" is not explicitly documented but happens\n            #       to work and would read from stdin. Make sure we know this\n            #       works for required git versions and on all platforms.\n            with make_tempfile(content=config_content) as cfg_file:\n                runner = GitWitlessRunner()\n                try:\n                    result = runner.run(\n                        ['git', 'config', '-f', cfg_file,\n                         'datalad.ora-remote.uuid'],\n                        protocol=StdOutCapture\n                    )\n                    uuid = result['stdout'].strip()\n                except CommandError as e:\n                    # doesn't contain what we are looking for\n                    lgr.debug(\"Found no UUID for ORA special remote at \"\n                              \"'%s' (%s)\", remote, exc_str(e))\n\n        return uuid\n\n\n\n\n    # chances are that if this dataset came from a RIA store, its subdatasets\n    # may live there too. Place a subdataset source candidate config that makes\n    # get probe this RIA store when obtaining subdatasets\n    ria_store_url = props['source'].split('#', maxsplit=1)[0]\n    ds.config.set(\n        # we use the label 'origin' for this candidate in order to not have to\n        # generate a complicated name from the actual source specification.\n        # we pick a cost of 200 to sort it before datalad's default candidates\n        # for non-RIA URLs, because they prioritize hierarchical layouts that\n        # cannot be found in a RIA store\n        'datalad.get.subdataset-source-candidate-200origin',\n        # use the entire original URL, up to the fragment + plus dataset ID\n        # placeholder, this should make things work with any store setup we\n        # support (paths, ports, ...)\n        ria_store_url + '#{id}',\n        where='local')\n\n    # setup publication dependency, if a corresponding special remote exists\n    # and was enabled (there could be RIA stores that actually only have repos)\n    # make this function be a generator\n    ora_remotes = [s for s in ds.siblings('query', result_renderer='disabled')\n                   if s.get('annex-externaltype') == 'ora']\n    # get full special remotes' config for access to stored URL\n    srs = repo.get_special_remotes() \\\n        if hasattr(repo, 'get_special_remotes') else dict()\n\n    has_only_disabled_ora = \\\n        not ora_remotes and \\\n        any(r.get('externaltype') == 'ora' for r in srs.values())\n\n    def match_in_urls(special_remote_cfg, url_to_match):\n        # Figure whether either `url` or `push-url` in an ORA remote's config\n        # match a given URL (to a RIA store).\n        return special_remote_cfg['url'].startswith(url_to_match) or \\\n               (special_remote_cfg['push-url'].startswith(url_to_match)\n                if 'push-url' in special_remote_cfg else False)\n\n    no_enabled_ora_matches_url = \\\n        all(not match_in_urls(srs[r['annex-uuid']], ria_store_url)\n            for r in ora_remotes)\n\n    if has_only_disabled_ora or no_enabled_ora_matches_url:\n\n        # No ORA remote autoenabled, but configuration known about at least one,\n        # or enabled ORA remotes seem to not match clone URL.\n        # Let's check the remote's config for datalad.ora-remote.uuid as stored\n        # by create-sibling-ria and try enabling that one.\n        lgr.debug(\"Found no autoenabled ORA special remote. Trying to look it \"\n                  \"up in source config ...\")\n\n        org_uuid = get_uuid_from_store(props['giturl'])\n\n        # Now, enable it. If annex-init didn't fail to enable it as stored, we\n        # wouldn't end up here, so enable with store URL as suggested by the URL\n        # we cloned from.\n        if org_uuid:\n            if org_uuid in srs.keys():\n                # TODO: - Double-check autoenable value and only do this when\n                #         true?\n                #       - What if still fails? -> Annex shouldn't change config\n                #         in that case\n\n                # we only need the store:\n                new_url = props['source'].split('#')[0]\n                try:\n                    repo.enable_remote(srs[org_uuid]['name'],\n                                       options=['url={}'.format(new_url)]\n                                       )\n                    lgr.info(\"Reconfigured %s for %s\",\n                             srs[org_uuid]['name'], new_url)\n                    # update ora_remotes for considering publication dependency\n                    # below\n                    ora_remotes = [s for s in\n                                   ds.siblings('query',\n                                               result_renderer='disabled')\n                                   if s.get('annex-externaltype', None) ==\n                                   'ora']\n                except CommandError as e:\n                    lgr.debug(\"Failed to reconfigure ORA special remote: %s\",\n                              exc_str(e))\n            else:\n                lgr.debug(\"Unknown ORA special remote uuid at '%s': %s\",\n                          remote, org_uuid)\n\n    # Set publication dependency for `remote` on the respective ORA remote:\n    if ora_remotes:\n        url_matching_remotes = [r for r in ora_remotes\n                                if srs[r['annex-uuid']]['url'] == ria_store_url]\n\n        if len(url_matching_remotes) == 1:\n            # We have exactly one ORA remote with the same store URL we used for\n            # cloning (includes previously reconfigured remote).\n            # Set publication dependency:\n            yield from ds.siblings('configure',\n                                   name=remote,\n                                   publish_depends=url_matching_remotes[0]['name'],\n                                   result_filter=None,\n                                   result_renderer='disabled')\n\n        elif not url_matching_remotes:\n            # No matches but we have successfully autoenabled ORA remotes. Could\n            # be the same store accessed by different method (cloning via HTTP\n            # but special remote access via SSH). We can confidently set\n            # publication dependency if the store knows the UUID.\n            org_uuid = get_uuid_from_store(props['giturl'])\n            uuid_matching_remotes = [r for r in ora_remotes\n                                     if r['annex-uuid'] == org_uuid]\n            if uuid_matching_remotes:\n                # Multiple uuid matches are actually possible via same-as.\n                # However, in that case we can't decide which one is supposed to\n                # be used with publishing to `remote`.\n                if len(uuid_matching_remotes) == 1:\n                    yield from ds.siblings(\n                        'configure',\n                        name=remote,\n                        publish_depends=uuid_matching_remotes[0]['name'],\n                        result_filter=None,\n                        result_renderer='disabled')\n                else:\n                    lgr.warning(\n                        \"Found multiple matching ORA remotes. Couldn't decide \"\n                        \"which one publishing to '%s' should depend on: %s.\"\n                        \" Consider running 'datalad siblings configure -s \"\n                        \"%s --publish-depends ORAREMOTENAME' to set \"\n                        \"publication dependency manually.\",\n                        remote,\n                        [r['name'] for r in uuid_matching_remotes],\n                        remote)\n\n        else:\n            # We have multiple ORA remotes with the same store URL we cloned\n            # from.\n            lgr.warning(\"Found multiple matching ORA remotes. Couldn't decide \"\n                        \"which one publishing to '%s' should depend on: %s.\"\n                        \" Consider running 'datalad siblings configure -s \"\n                        \"%s --publish-depends ORAREMOTENAME' to set \"\n                        \"publication dependency manually.\",\n                        remote,\n                        [r['name'] for r in url_matching_remotes],\n                        remote)\n\n\ndef postclonecfg_annexdataset(ds, reckless, description=None, remote=\"origin\"):\n    \"\"\"If ds \"knows annex\" -- annex init it, set into reckless etc\n\n    Provides additional tune up to a possibly an annex repo, e.g.\n    \"enables\" reckless mode, sets up description\n    \"\"\"\n    # in any case check whether we need to annex-init the installed thing:\n    if not knows_annex(ds.path):\n        # not for us\n        return\n\n    # init annex when traces of a remote annex can be detected\n    if reckless == 'auto':\n        lgr.debug(\n            \"Instruct annex to hardlink content in %s from local \"\n            \"sources, if possible (reckless)\", ds.path)\n        ds.config.set(\n            'annex.hardlink', 'true', where='local', reload=True)\n\n    lgr.debug(\"Initializing annex repo at %s\", ds.path)\n    # Note, that we cannot enforce annex-init via AnnexRepo().\n    # If such an instance already exists, its __init__ will not be executed.\n    # Therefore do quick test once we have an object and decide whether to call\n    # its _init().\n    #\n    # Additionally, call init if we need to add a description (see #1403),\n    # since AnnexRepo.__init__ can only do it with create=True\n    repo = AnnexRepo(ds.path, init=True)\n    if not repo.is_initialized() or description:\n        repo._init(description=description)\n    if reckless == 'auto' or (reckless and reckless.startswith('shared-')):\n        repo.call_annex(['untrust', 'here'])\n\n    elif reckless == 'ephemeral':\n        # with ephemeral we declare 'here' as 'dead' right away, whenever\n        # we symlink the remote's annex, since availability from 'here' should\n        # not be propagated for an ephemeral clone when we publish back to\n        # the remote.\n        # This will cause stuff like this for a locally present annexed file:\n        # % git annex whereis d1\n        # whereis d1 (0 copies) failed\n        # BUT this works:\n        # % git annex find . --not --in here\n        # % git annex find . --in here\n        # d1\n\n        # we don't want annex copy-to <remote>\n        ds.config.set(\n            f'remote.{remote}.annex-ignore', 'true',\n            where='local')\n\n        ds.repo.set_remote_dead('here')\n\n        if check_symlink_capability(ds.repo.dot_git / 'dl_link_test',\n                                    ds.repo.dot_git / 'dl_target_test'):\n            # symlink the annex to avoid needless copies in an ephemeral clone\n            annex_dir = ds.repo.dot_git / 'annex'\n            origin_annex_url = ds.config.get(f\"remote.{remote}.url\", None)\n            origin_git_path = None\n            if origin_annex_url:\n                try:\n                    # Deal with file:// scheme URLs as well as plain paths.\n                    # If origin isn't local, we have nothing to do.\n                    origin_git_path = Path(RI(origin_annex_url).localpath)\n\n                    # we are local; check for a bare repo first to not mess w/\n                    # the path\n                    if GitRepo(origin_git_path, create=False).bare:\n                        # origin is a bare repo -> use path as is\n                        pass\n                    elif origin_git_path.name != '.git':\n                        origin_git_path /= '.git'\n                except ValueError:\n                    # Note, that accessing localpath on a non-local RI throws\n                    # ValueError rather than resulting in an AttributeError.\n                    # TODO: Warning level okay or is info level sufficient?\n                    # Note, that setting annex-dead is independent of\n                    # symlinking .git/annex. It might still make sense to\n                    # have an ephemeral clone that doesn't propagate its avail.\n                    # info. Therefore don't fail altogether.\n                    lgr.warning(\"reckless=ephemeral mode: %s doesn't seem \"\n                                \"local: %s\\nno symlinks being used\",\n                                remote, origin_annex_url)\n            if origin_git_path:\n                # TODO make sure that we do not delete any unique data\n                rmtree(str(annex_dir)) \\\n                    if not annex_dir.is_symlink() else annex_dir.unlink()\n                annex_dir.symlink_to(origin_git_path / 'annex',\n                                     target_is_directory=True)\n        else:\n            # TODO: What level? + note, that annex-dead is independent\n            lgr.warning(\"reckless=ephemeral mode: Unable to create symlinks on \"\n                        \"this file system.\")\n\n    srs = {True: [], False: []}  # special remotes by \"autoenable\" key\n    remote_uuids = None  # might be necessary to discover known UUIDs\n\n    repo_config = repo.config\n    # Note: The purpose of this function is to inform the user. So if something\n    # looks misconfigured, we'll warn and move on to the next item.\n    for uuid, config in repo.get_special_remotes().items():\n        sr_name = config.get('name', None)\n        if sr_name is None:\n            lgr.warning(\n                'Ignoring special remote %s because it does not have a name. '\n                'Known information: %s',\n                uuid, config)\n            continue\n        sr_autoenable = config.get('autoenable', False)\n        try:\n            sr_autoenable = ensure_bool(sr_autoenable)\n        except ValueError:\n            lgr.warning(\n                'Failed to process \"autoenable\" value %r for sibling %s in '\n                'dataset %s as bool.'\n                'You might need to enable it later manually and/or fix it up to'\n                ' avoid this message in the future.',\n                sr_autoenable, sr_name, ds.path)\n            continue\n\n        # If it looks like a type=git special remote, make sure we have up to\n        # date information. See gh-2897.\n        if sr_autoenable and repo_config.get(\"remote.{}.fetch\".format(sr_name)):\n            try:\n                repo.fetch(remote=sr_name)\n            except CommandError as exc:\n                lgr.warning(\"Failed to fetch type=git special remote %s: %s\",\n                            sr_name, exc_str(exc))\n\n        # determine whether there is a registered remote with matching UUID\n        if uuid:\n            if remote_uuids is None:\n                remote_uuids = {\n                    # Check annex-config-uuid first. For sameas annex remotes,\n                    # this will point to the UUID for the configuration (i.e.\n                    # the key returned by get_special_remotes) rather than the\n                    # shared UUID.\n                    (repo_config.get('remote.%s.annex-config-uuid' % r) or\n                     repo_config.get('remote.%s.annex-uuid' % r))\n                    for r in repo.get_remotes()\n                }\n            if uuid not in remote_uuids:\n                srs[sr_autoenable].append(sr_name)\n\n    if srs[True]:\n        lgr.debug(\n            \"configuration for %s %s added because of autoenable,\"\n            \" but no UUIDs for them yet known for dataset %s\",\n            # since we are only at debug level, we could call things their\n            # proper names\n            single_or_plural(\"special remote\",\n                             \"special remotes\", len(srs[True]), True),\n            \", \".join(srs[True]),\n            ds.path\n        )\n\n    if srs[False]:\n        # if has no auto-enable special remotes\n        lgr.info(\n            'access to %s %s not auto-enabled, enable with:\\n'\n            '\\t\\tdatalad siblings -d \"%s\" enable -s %s',\n            # but since humans might read it, we better confuse them with our\n            # own terms!\n            single_or_plural(\"dataset sibling\",\n                             \"dataset siblings\", len(srs[False]), True),\n            \", \".join(srs[False]),\n            ds.path,\n            srs[False][0] if len(srs[False]) == 1 else \"SIBLING\",\n        )\n\n    # we have just cloned the repo, so it has a remote `remote`, configure any\n    # reachable origin of origins\n    yield from configure_origins(ds, ds, remote=remote)\n\n\n_handle_possible_annex_dataset = postclonecfg_annexdataset\n\n\ndef configure_origins(cfgds, probeds, label=None, remote=\"origin\"):\n    \"\"\"Configure any discoverable local dataset sibling as a remote\n\n    Parameters\n    ----------\n    cfgds : Dataset\n      Dataset to receive the remote configurations\n    probeds : Dataset\n      Dataset to start looking for `remote` remotes. May be identical with\n      `cfgds`.\n    label : int, optional\n      Each discovered remote will be configured as a remote under the name\n      '<remote>-<label>'. If no label is given, '2' will be used by default,\n      given that there is typically a remote named `remote` already.\n    remote : str, optional\n      Name of the default remote on clone.\n    \"\"\"\n    if label is None:\n        label = 1\n    # let's look at the URL for that remote and see if it is a local\n    # dataset\n    origin_url = probeds.config.get(f'remote.{remote}.url')\n    if not origin_url:\n        # no remote with default name, nothing to do\n        return\n    if not cfgds.config.obtain(\n            'datalad.install.inherit-local-origin',\n            default=True):\n        # no inheritance wanted\n        return\n    if not isinstance(RI(origin_url), PathRI):\n        # not local path\n        return\n\n    # no need to reconfigure original/direct remote again\n    if cfgds != probeds:\n        # prevent duplicates\n        known_remote_urls = set(\n            cfgds.config.get(r + '.url', None)\n            for r in cfgds.config.sections()\n            if r.startswith('remote.')\n        )\n        if origin_url not in known_remote_urls:\n            yield from cfgds.siblings(\n                'configure',\n                # no chance for conflict, can only be the second configured\n                # remote\n                name='{}-{}'.format(remote, label),\n                url=origin_url,\n                # fetch to get all annex info\n                fetch=True,\n                result_renderer='disabled',\n                on_failure='ignore',\n            )\n    # and dive deeper\n    # given the clone source is a local dataset, we can have a\n    # cheap look at it, and configure its own `remote` as a remote\n    # (if there is any), and benefit from additional annex availability\n    yield from configure_origins(\n        cfgds,\n        Dataset(probeds.pathobj / origin_url),\n        label=label + 1,\n        remote=remote)\n\n\ndef _get_tracking_source(ds):\n    \"\"\"Returns name and url of a potential configured source\n    tracking remote\"\"\"\n    vcs = ds.repo\n    # if we have a remote, let's check the location of that remote\n    # for the presence of the desired submodule\n\n    remote_name, tracking_branch = vcs.get_tracking_branch()\n    if not remote_name and isinstance(vcs, AnnexRepo):\n        # maybe cloned from a source repo that was in adjusted mode\n        # https://github.com/datalad/datalad/issues/3969\n        remote_name, tracking_branch = vcs.get_tracking_branch(\n            corresponding=False)\n    # TODO: better default `None`? Check where we might rely on '':\n    remote_url = ''\n    if remote_name:\n        remote_url = vcs.get_remote_url(remote_name, push=False)\n\n    return remote_name, remote_url\n\n\ndef _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path\n\n\ndef decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n\n    # Git never gets to see these URLs, so let's manually apply any\n    # rewrite configuration Git might know about.\n    # Note: We need to rewrite before parsing, otherwise parsing might go wrong.\n    # This is particularly true for insteadOf labels replacing even the URL\n    # scheme.\n    spec = cfg.rewrite_url(spec)\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if re.match(uuid_regex, dsid):\n            trace = '{}/{}'.format(dsid[:3], dsid[3:])\n            default_destpath = dsid\n        elif dsid.startswith('~'):\n            trace = 'alias/{}'.format(dsid[1:])\n            default_destpath = dsid[1:]\n        else:\n            raise ValueError(\n                'RIA URI not recognized, no valid dataset ID or other supported '\n                'scheme: {}'.format(spec))\n        # now we cancel the fragment in the original URL, but keep everything else\n        # in order to be able to support the various combinations of ports, paths,\n        # and everything else\n        source_ri.fragment = ''\n        # strip the custom protocol and go with standard one\n        source_ri.scheme = source_ri.scheme[4:]\n        # take any existing path, and add trace to dataset within the store\n        source_ri.path = '{urlpath}{urldelim}{trace}'.format(\n            urlpath=source_ri.path if source_ri.path else '',\n            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',\n            trace=trace,\n        )\n        props.update(\n            type='ria',\n            giturl=str(source_ri),\n            version=version,\n            default_destpath=default_destpath,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        props['type'] = 'giturl'\n        # use original input verbatim\n        props['giturl'] = spec\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Incorrect extraction of installation path from SSH URLs

**Description:**
When providing an SSH URL, the installation path is not correctly extracted, resulting in the path including the hostname instead of just the repository name.

**Example Code:**
```python
url = 'hostname:lastbit'
path = _get_installationpath_from_url(url)
print(path)  # Outputs: 'hostname:lastbit'
```

**Expected Behavior:**
The installation path should be extracted as `'lastbit'` from the SSH URL.

**Actual Behavior:**
The function returns `'hostname:lastbit'` instead of the expected `'lastbit'`.
[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/core/distributed/clone.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Plumbing command for dataset installation"""


import logging
import re
from os.path import expanduser
from collections import OrderedDict
from urllib.parse import unquote as urlunquote

from datalad.interface.base import Interface
from datalad.interface.utils import eval_results
from datalad.interface.base import build_doc
from datalad.interface.results import get_status_dict
from datalad.interface.common_opts import (
    location_description,
    reckless_opt,
)
from datalad.log import log_progress
from datalad.support.gitrepo import (
    GitRepo,
)
from datalad.cmd import (
    CommandError,
    GitWitlessRunner,
    StdOutCapture,
)
from datalad.distributed.ora_remote import (
    LocalIO,
    RIARemoteError,
    SSHRemoteIO,
)
from datalad.support.annexrepo import AnnexRepo
from datalad.support.constraints import (
    EnsureNone,
    EnsureStr,
    EnsureKeyChoice,
)
from datalad.support.exceptions import DownloadError
from datalad.support.param import Parameter
from datalad.support.strings import get_replacement_dict
from datalad.support.network import (
    get_local_file_url,
    download_url,
    is_url,
    URL,
    RI,
    DataLadRI,
    PathRI,
)
from datalad.dochelpers import (
    exc_str,
    single_or_plural,
)
from datalad.utils import (
    ensure_bool,
    ensure_list,
    knows_annex,
    make_tempfile,
    Path,
    PurePosixPath,
    rmtree,
)

from datalad.distribution.dataset import (
    Dataset,
    datasetmethod,
    resolve_path,
    require_dataset,
    EnsureDataset,
)
from datalad.distribution.utils import (
    _get_flexible_source_candidates,
)
from datalad.utils import (
    check_symlink_capability
)

__docformat__ = 'restructuredtext'

lgr = logging.getLogger('datalad.core.distributed.clone')


@build_doc
class Clone(Interface):
    """Obtain a dataset (copy) from a URL or local directory

    The purpose of this command is to obtain a new clone (copy) of a dataset
    and place it into a not-yet-existing or empty directory. As such `clone`
    provides a strict subset of the functionality offered by `install`. Only a
    single dataset can be obtained, and immediate recursive installation of
    subdatasets is not supported. However, once a (super)dataset is installed
    via `clone`, any content, including subdatasets can be obtained by a
    subsequent `get` command.

    Primary differences over a direct `git clone` call are 1) the automatic
    initialization of a dataset annex (pure Git repositories are equally
    supported); 2) automatic registration of the newly obtained dataset as a
    subdataset (submodule), if a parent dataset is specified; 3) support
    for additional resource identifiers (DataLad resource identifiers as used
    on datasets.datalad.org, and RIA store URLs as used for store.datalad.org
    - optionally in specific versions as identified by a branch or a tag; see
    examples); and 4) automatic configurable generation of alternative access
    URL for common cases (such as appending '.git' to the URL in case the
    accessing the base URL failed).

    In case the clone is registered as a subdataset, the original URL passed to
    `clone` is recorded in `.gitmodules` of the parent dataset in addition
    to the resolved URL used internally for git-clone. This allows to preserve
    datalad specific URLs like ria+ssh://... for subsequent calls to `get` if
    the subdataset was locally removed later on.

    || PYTHON >>By default, the command returns a single Dataset instance for
    an installed dataset, regardless of whether it was newly installed ('ok'
    result), or found already installed from the specified source ('notneeded'
    result).<< PYTHON ||

    URL mapping configuration

    'clone' supports the transformation of URLs via (multi-part) substitution
    specifications. A substitution specification is defined as a configuration
    setting 'datalad.clone.url-substition.<seriesID>' with a string containing
    a match and substitution expression, each following Python's regular
    expression syntax. Both expressions are concatenated to a single string
    with an arbitrary delimiter character. The delimiter is defined by
    prefixing the string with the delimiter. Prefix and delimiter are stripped
    from the expressions (Example: ",^http://(.*)$,https://\\1").  This setting
    can be defined multiple times, using the same '<seriesID>'.  Substitutions
    in a series will be applied incrementally, in order of their definition.
    The first substitution in such a series must match, otherwise no further
    substitutions in a series will be considered. However, following the first
    match all further substitutions in a series are processed, regardless
    whether intermediate expressions match or not. Substitution series themselves
    have no particular order, each matching series will result in a candidate
    clone URL. Consequently, the initial match specification in a series should
    be as precise as possible to prevent inflation of candidate URLs.

    .. seealso::

      :ref:`handbook:3-001`
        More information on Remote Indexed Archive (RIA) stores
    """
    # by default ignore everything but install results
    # i.e. no "add to super dataset"
    result_filter = EnsureKeyChoice('action', ('install',))
    # very frequently this command will yield exactly one installed dataset
    # spare people the pain of going through a list by default
    return_type = 'item-or-list'
    # as discussed in #1409 and #1470, we want to return dataset instances
    # matching what is actually available after command completion (and
    # None for any failed dataset installation)
    result_xfm = 'successdatasets-or-none'

    _examples_ = [
        dict(text="Install a dataset from Github into the current directory",
             code_py="clone("
             "source='https://github.com/datalad-datasets/longnow"
             "-podcasts.git')",
             code_cmd="datalad clone "
             "https://github.com/datalad-datasets/longnow-podcasts.git"),
        dict(text="Install a dataset into a specific directory",
             code_py="""\
             clone(source='https://github.com/datalad-datasets/longnow-podcasts.git',
                   path='myfavpodcasts')""",
             code_cmd="""\
             datalad clone https://github.com/datalad-datasets/longnow-podcasts.git \\
             myfavpodcasts"""),
        dict(text="Install a dataset as a subdataset into the current dataset",
             code_py="""\
             clone(dataset='.',
                   source='https://github.com/datalad-datasets/longnow-podcasts.git')""",
             code_cmd="datalad clone -d . "
             "https://github.com/datalad-datasets/longnow-podcasts.git"),
        dict(text="Install the main superdataset from datasets.datalad.org",
             code_py="clone(source='///')",
             code_cmd="datalad clone ///"),
        dict(text="Install a dataset identified by a literal alias from store.datalad.org",
             code_py="clone(source='ria+http://store.datalad.org#~hcp-openaccess')",
             code_cmd="datalad clone ria+http://store.datalad.org#~hcp-openaccess"),
        dict(
            text="Install a dataset in a specific version as identified by a "
                 "branch or tag name from store.datalad.org",
            code_py="clone(source='ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier')",
            code_cmd="datalad clone ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier"),
        dict(
            text="Install a dataset with group-write access permissions",
            code_py=\
            "clone(source='http://example.com/dataset', reckless='shared-group')",
            code_cmd=\
            "datalad clone http://example.com/dataset --reckless shared-group"),
    ]

    _params_ = dict(
        dataset=Parameter(
            args=("-d", "--dataset"),
            doc="""(parent) dataset to clone into. If given, the newly cloned
            dataset is registered as a subdataset of the parent. Also, if given,
            relative paths are interpreted as being relative to the parent
            dataset, and not relative to the working directory.""",
            constraints=EnsureDataset() | EnsureNone()),
        source=Parameter(
            args=("source",),
            metavar='SOURCE',
            doc="""URL, DataLad resource identifier, local path or instance of
            dataset to be cloned""",
            constraints=EnsureStr() | EnsureNone()),
        path=Parameter(
            args=("path",),
            metavar='PATH',
            nargs="?",
            doc="""path to clone into.  If no `path` is provided a
            destination path will be derived from a source URL
            similar to :command:`git clone`"""),
        description=location_description,
        reckless=reckless_opt,
    )

    @staticmethod
    @datasetmethod(name='clone')
    @eval_results
    def __call__(
            source,
            path=None,
            dataset=None,
            description=None,
            reckless=None):
        # did we explicitly get a dataset to install into?
        # if we got a dataset, path will be resolved against it.
        # Otherwise path will be resolved first.
        ds = require_dataset(
            dataset, check_installed=True, purpose='clone') \
            if dataset is not None else dataset
        refds_path = ds.path if ds else None

        # legacy compatibility
        if reckless is True:
            # so that we can forget about how things used to be
            reckless = 'auto'

        if isinstance(source, Dataset):
            source = source.path

        if source == path:
            # even if they turn out to be identical after resolving symlinks
            # and more sophisticated witchcraft, it would still happily say
            # "it appears to be already installed", so we just catch an
            # obviously pointless input combination
            raise ValueError(
                "clone `source` and destination `path` are identical [{}]. "
                "If you are trying to add a subdataset simply use `save`".format(
                    path))

        if path is not None:
            path = resolve_path(path, dataset)

        # derive target from source:
        if path is None:
            # we got nothing but a source. do something similar to git clone
            # and derive the path from the source and continue
            # since this is a relative `path`, resolve it:
            # we are not going to reuse the decoded URL, as this is done for
            # all source candidates in clone_dataset(), we just use to determine
            # a destination path here in order to perform a bunch of additional
            # checks that shall not pollute the helper function
            source_ = decode_source_spec(
                source, cfg=None if ds is None else ds.config)
            path = resolve_path(source_['default_destpath'], dataset)
            lgr.debug("Determined clone target path from source")
        lgr.debug("Resolved clone target path to: '%s'", path)

        # there is no other way -- my intoxicated brain tells me
        assert(path is not None)

        result_props = dict(
            action='install',
            logger=lgr,
            refds=refds_path,
            source_url=source)

        try:
            # this will implicitly cause pathlib to run a bunch of checks
            # whether the present path makes any sense on the platform
            # we are running on -- we don't care if the path actually
            # exists at this point, but we want to abort early if the path
            # spec is determined to be useless
            path.exists()
        except OSError as e:
            yield get_status_dict(
                status='error',
                path=path,
                message=('cannot handle target path: %s', exc_str(e)),
                **result_props)
            return

        destination_dataset = Dataset(path)
        result_props['ds'] = destination_dataset

        if ds is not None and ds.pathobj not in path.parents:
            yield get_status_dict(
                status='error',
                message=("clone target path '%s' not in specified target dataset '%s'",
                         path, ds),
                **result_props)
            return

        # perform the actual cloning operation
        yield from clone_dataset(
            [source],
            destination_dataset,
            reckless,
            description,
            result_props,
            cfg=None if ds is None else ds.config,
        )

        # TODO handle any 'version' property handling and verification using a
        # dedicated public helper

        if ds is not None:
            # we created a dataset in another dataset
            # -> make submodule
            actually_saved_subds = False
            for r in ds.save(
                    path,
                    # Note, that here we know we don't save anything but a new
                    # subdataset. Hence, don't go with default commit message,
                    # but be more specific.
                    message="[DATALAD] Added subdataset",
                    return_type='generator',
                    result_filter=None,
                    result_xfm=None,
                    on_failure='ignore'):
                actually_saved_subds = actually_saved_subds or (
                        r['action'] == 'save' and
                        r['type'] == 'dataset' and
                        r['refds'] == ds.path and
                        r['status'] == 'ok')
                yield r

            # Modify .gitmodules to contain originally given url. This is
            # particularly relevant for postclone routines on a later `get`
            # for that subdataset. See gh-5256.
            if actually_saved_subds:
                # New subdataset actually saved. Amend the modification
                # of .gitmodules. Note, that we didn't allow to deviate
                # from git default behavior WRT a submodule's name vs
                # its path when we made this a new subdataset.
                subds_name = path.relative_to(ds.pathobj)
                ds.repo.call_git(
                    ['config',
                     '--file',
                     '.gitmodules',
                     '--replace-all',
                     'submodule.{}.{}'.format(subds_name,
                                              "datalad-url"),
                     source]
                )
                yield from ds.save('.gitmodules',
                                   amend=True, to_git=True)
            else:
                # We didn't really commit. Just call `subdatasets`
                # in that case to have the modification included in the
                # post-clone state (whatever that may be).
                ds.subdatasets(path, set_property=[("datalad-url", source)])


def _get_url_mappings(cfg):
    cfg_prefix = 'datalad.clone.url-substitute.'
    # figure out which keys we should be looking for
    # in the active config
    subst_keys = set(k for k in cfg.keys() if k.startswith(cfg_prefix))
    # and in the common config specs
    from datalad.interface.common_cfg import definitions
    subst_keys.update(k for k in definitions if k.startswith(cfg_prefix))
    # TODO a potential sorting of substitution series could be implemented
    # here
    return [
        # decode the rule specifications
        get_replacement_dict(
            # one or more could come out
            ensure_list(
                cfg.get(
                    k,
                    # make sure to pull the default from the common config
                    default=cfg.obtain(k),
                    # we specifically support declaration of multiple
                    # settings to build replacement chains
                    get_all=True)))
        for k in subst_keys
    ]


def _map_urls(cfg, urls):
    mapping_specs = _get_url_mappings(cfg)
    if not mapping_specs:
        return urls

    mapped = []
    # we process the candidate in order to maintain any prioritization
    # encoded in it (e.g. _get_flexible_source_candidates_for_submodule)
    # if we have a matching mapping replace the URL in its position
    for u in urls:
        # we only permit a single match
        # TODO we likely want to RF this to pick the longest match
        mapping_applied = False
        # try one mapping set at a time
        for mapping_spec in mapping_specs:
            # process all substitution patterns in the specification
            # always operate on strings (could be a Path instance too)
            mu = str(u)
            matched = False
            for match_ex, subst_ex in mapping_spec.items():
                if not matched:
                    matched = re.match(match_ex, mu) is not None
                if not matched:
                    break
                # try to map, would return unchanged, if there is no match
                mu = re.sub(match_ex, subst_ex, mu)
            if mu != u:
                lgr.debug("URL substitution: '%s' -> '%s'", u, mu)
                mapped.append(mu)
                # we could consider breaking after the for effective mapping
                # specification. however, that would mean any generic
                # definition of a broadly matching substitution would derail
                # the entroe system. moreover, suddently order would matter
                # substantially
                mapping_applied = True
        if not mapping_applied:
            # none of the mappings matches, go with the original URL
            # (really original, not the stringified one)
            mapped.append(u)
    return mapped


def clone_dataset(
        srcs,
        destds,
        reckless=None,
        description=None,
        result_props=None,
        cfg=None,
        checkout_gitsha=None):
    """Internal helper to perform cloning without sanity checks (assumed done)

    This helper does not handle any saving of subdataset modification or adding
    in a superdataset.

    Parameters
    ----------
    srcs : list
      Any suitable clone source specifications (paths, URLs)
    destds : Dataset
      Dataset instance for the clone destination
    reckless : {None, 'auto', 'ephemeral', 'shared-...'}, optional
      Mode switch to put cloned dataset into unsafe/throw-away configurations, i.e.
      sacrifice data safety for performance or resource footprint. When None
      and `cfg` is specified, use the value of `datalad.clone.reckless`.
    description : str, optional
      Location description for the annex of the dataset clone (if there is any).
    result_props : dict, optional
      Default properties for any yielded result, passed on to get_status_dict().
    cfg : ConfigManager, optional
      Configuration for parent dataset. This will be queried instead
      of the global DataLad configuration.
    checkout_gitsha : str, optional
      If given, a specific commit, identified by shasum, will be checked out after
      cloning. A dedicated follow-up fetch will be performed, if the initial clone
      did not obtain the commit object. Should the checkout of the target commit
      cause a detached HEAD, the previously active branch will be reset to the
      target commit.

    Yields
    ------
    dict
      DataLad result records
    """
    if not result_props:
        # in case the caller had no specific idea on how results should look
        # like, provide sensible defaults
        result_props = dict(
            action='install',
            logger=lgr,
            ds=destds,
        )

    if reckless is None and cfg:
        # if reckless is not explicitly given, but we operate on a
        # superdataset, query whether it has been instructed to operate
        # in a reckless mode, and inherit it for the coming clone
        reckless = cfg.get('datalad.clone.reckless', None)

    dest_path = destds.pathobj

    # check for configured URL mappings, either in the given config manager
    # or in the one of the destination dataset, which is typically not existent
    # yet and the process config manager is then used effectively
    srcs = _map_urls(cfg or destds.config, srcs)

    # decode all source candidate specifications
    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]

    # now expand the candidate sources with additional variants of the decoded
    # giturl, while duplicating the other properties in the additional records
    # for simplicity. The hope is to overcome a few corner cases and be more
    # robust than git clone
    candidate_sources = [
        dict(props, giturl=s) for props in candidate_sources
        for s in _get_flexible_source_candidates(props['giturl'])
    ]

    # important test! based on this `rmtree` will happen below after failed clone
    dest_path_existed = dest_path.exists()
    if dest_path_existed and any(dest_path.iterdir()):
        if destds.is_installed():
            # check if dest was cloned from the given source before
            # this is where we would have installed this from
            # this is where it was actually installed from
            track_name, track_url = _get_tracking_source(destds)
            try:
                # this will get us track_url in system native path conventions,
                # whenever it is a path (and not a URL)
                # this is needed to match it to any potentially incoming local
                # source path in the 'notneeded' test below
                track_path = str(Path(track_url))
            except Exception:
                # this should never happen, because Path() will let any non-path stringification
                # pass through unmodified, but we do not want any potential crash due to
                # pathlib behavior changes
                lgr.debug("Unexpected behavior of pathlib!")
                track_path = None
            for cand in candidate_sources:
                src = cand['giturl']
                if track_url == src \
                        or (not is_url(track_url)
                            and get_local_file_url(track_url, compatibility='git') == src) \
                        or track_path == expanduser(src):
                    yield get_status_dict(
                        status='notneeded',
                        message=("dataset %s was already cloned from '%s'",
                                 destds,
                                 src),
                        **result_props)
                    return
        # anything else is an error
        yield get_status_dict(
            status='error',
            message='target path already exists and not empty, refuse to clone into target path',
            **result_props)
        return

    log_progress(
        lgr.info,
        'cloneds',
        'Cloning dataset to %s', destds,
        total=len(candidate_sources),
        label='Clone attempt',
        unit=' Candidate locations',
    )
    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url
    for cand in candidate_sources:
        log_progress(
            lgr.info,
            'cloneds',
            'Attempting to clone from %s to %s', cand['giturl'], dest_path,
            update=1,
            increment=True)

        clone_opts = {}

        if cand.get('version', None):
            clone_opts['branch'] = cand['version']
        try:
            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35
            # doesn't make it happen seamlessly
            GitRepo.clone(
                path=str(dest_path),
                url=cand['giturl'],
                clone_options=clone_opts,
                create=True)

        except CommandError as e:
            e_stderr = e.stderr

            error_msgs[cand['giturl']] = e
            lgr.debug("Failed to clone from URL: %s (%s)",
                      cand['giturl'], exc_str(e))
            if dest_path.exists():
                lgr.debug("Wiping out unsuccessful clone attempt at: %s",
                          dest_path)
                # We must not just rmtree since it might be curdir etc
                # we should remove all files/directories under it
                # TODO stringification can be removed once patlib compatible
                # or if PY35 is no longer supported
                rmtree(str(dest_path), children_only=dest_path_existed)

            if e_stderr and 'could not create work tree' in e_stderr.lower():
                # this cannot be fixed by trying another URL
                re_match = re.match(r".*fatal: (.*)$", e_stderr,
                                    flags=re.MULTILINE | re.DOTALL)
                # cancel progress bar
                log_progress(
                    lgr.info,
                    'cloneds',
                    'Completed clone attempts for %s', destds
                )
                yield get_status_dict(
                    status='error',
                    message=re_match.group(1).strip()
                    if re_match else "stderr: " + e_stderr,
                    **result_props)
                return
            # next candidate
            continue

        result_props['source'] = cand
        # do not bother with other sources if succeeded
        break

    log_progress(
        lgr.info,
        'cloneds',
        'Completed clone attempts for %s', destds
    )

    if not destds.is_installed():
        if len(error_msgs):
            if all(not e.stdout and not e.stderr for e in error_msgs.values()):
                # there is nothing we can learn from the actual exception,
                # the exit code is uninformative, the command is predictable
                error_msg = "Failed to clone from all attempted sources: %s"
                error_args = list(error_msgs.keys())
            else:
                error_msg = "Failed to clone from any candidate source URL. " \
                            "Encountered errors per each url were:\n- %s"
                error_args = '\n- '.join(
                    '{}\n  {}'.format(url, exc_str(exc))
                    for url, exc in error_msgs.items()
                )
        else:
            # yoh: Not sure if we ever get here but I felt that there could
            #      be a case when this might happen and original error would
            #      not be sufficient to troubleshoot what is going on.
            error_msg = "Awkward error -- we failed to clone properly. " \
                        "Although no errors were encountered, target " \
                        "dataset at %s seems to be not fully installed. " \
                        "The 'succesful' source was: %s"
            error_args = (destds.path, cand['giturl'])
        yield get_status_dict(
            status='error',
            message=(error_msg, error_args),
            **result_props)
        return

    dest_repo = destds.repo

    remotes = dest_repo.get_remotes(with_urls_only=True)
    nremotes = len(remotes)
    if nremotes == 1:
        remote = remotes[0]
        lgr.debug("Determined %s to be remote of %s", remote, destds)
    elif remotes > 1:
        lgr.warning(
            "Fresh clone %s unexpected has multiple remotes: %s. Using %s",
            destds.path, remotes, remotes[0])
        remote = remotes[0]
    else:
        raise RuntimeError("bug: fresh clone has zero remotes")

    if not cand.get("version"):
        postclone_check_head(destds, remote=remote)

    # act on --reckless=shared-...
    # must happen prior git-annex-init, where we can cheaply alter the repo
    # setup through safe re-init'ing
    if reckless and reckless.startswith('shared-'):
        lgr.debug('Reinit %s to enable shared access permissions', destds)
        destds.repo.call_git(['init', '--shared={}'.format(reckless[7:])])

    # In case of RIA stores we need to prepare *before* annex is called at all
    if result_props['source']['type'] == 'ria':
        postclone_preannex_cfg_ria(destds, remote=remote)

    yield from postclonecfg_annexdataset(
        destds,
        reckless,
        description,
        remote=remote)

    if checkout_gitsha and \
       dest_repo.get_hexsha(dest_repo.get_corresponding_branch()) != checkout_gitsha:
        try:
            postclone_checkout_commit(dest_repo, checkout_gitsha,
                                      remote=remote)
        except Exception as e:
            yield get_status_dict(
                status='error',
                message=str(e),
                **result_props,
            )

            # We were supposed to clone a particular version but failed to.
            # This is particularly pointless in case of subdatasets and
            # potentially fatal with current implementation of recursion.
            # see gh-5387
            lgr.debug("Failed to checkout %s, removing this clone attempt at %s", checkout_gitsha, dest_path)
            # TODO stringification can be removed once pathlib compatible
            # or if PY35 is no longer supported
            rmtree(str(dest_path), children_only=dest_path_existed)
            return

    # perform any post-processing that needs to know details of the clone
    # source
    if result_props['source']['type'] == 'ria':
        yield from postclonecfg_ria(destds, result_props['source'],
                                    remote=remote)

    if reckless:
        # store the reckless setting in the dataset to make it
        # known to later clones of subdatasets via get()
        destds.config.set(
            'datalad.clone.reckless', reckless,
            where='local',
            reload=True)
    else:
        # We would still want to reload configuration to ensure that any of the
        # above git invocations could have potentially changed the config
        # TODO: might no longer be necessary if 0.14.0 adds reloading upon
        # non-readonly commands invocation
        destds.config.reload()

    # yield successful clone of the base dataset now, as any possible
    # subdataset clone down below will not alter the Git-state of the
    # parent
    yield get_status_dict(status='ok', **result_props)


def postclone_checkout_commit(repo, target_commit, remote="origin"):
    """Helper to check out a specific target commit in a fresh clone.

    Will not check (again) if current commit and target commit are already
    the same!
    """
    # record what branch we were on right after the clone
    active_branch = repo.get_active_branch()
    corr_branch = repo.get_corresponding_branch(branch=active_branch)
    was_adjusted = bool(corr_branch)
    repo_orig_branch = corr_branch or active_branch
    # if we are on a branch this hexsha will be the tip of that branch
    repo_orig_hexsha = repo.get_hexsha(repo_orig_branch)
    # make sure we have the desired commit locally
    # expensive and possibly error-prone fetch conditional on cheap
    # local check
    if not repo.commit_exists(target_commit):
        try:
            repo.fetch(remote=remote, refspec=target_commit)
        except CommandError:
            pass
        # instead of inspecting the fetch results for possible ways
        # with which it could failed to produced the desired result
        # let's verify the presence of the commit directly, we are in
        # expensive-land already anyways
        if not repo.commit_exists(target_commit):
            # there is nothing we can do about this
            # MIH thinks that removing the clone is not needed, as a likely
            # next step will have to be a manual recovery intervention
            # and not another blind attempt
            raise ValueError(
                'Target commit %s does not exist in the clone, and '
                'a fetch that commit from remote failed'
                % target_commit[:8])
    # checkout the desired commit
    repo.call_git(['checkout', target_commit])
    # did we detach?
    if repo_orig_branch and not repo.get_active_branch():
        # trace if current state is a predecessor of the branch_hexsha
        lgr.debug(
            "Detached HEAD after resetting worktree of %s "
            "(original branch: %s)", repo, repo_orig_branch)
        if repo.get_merge_base(
                [repo_orig_hexsha, target_commit]) == target_commit:
            # we assume the target_commit to be from the same branch,
            # because it is an ancestor -- update that original branch
            # to point to the target_commit, and update HEAD to point to
            # that location
            lgr.info(
                "Reset branch '%s' to %s (from %s) to "
                "avoid a detached HEAD",
                repo_orig_branch, target_commit[:8], repo_orig_hexsha[:8])
            branch_ref = 'refs/heads/%s' % repo_orig_branch
            repo.update_ref(branch_ref, target_commit)
            repo.update_ref('HEAD', branch_ref, symbolic=True)
            if was_adjusted:
                # Note: The --force is needed because the adjust branch already
                # exists.
                repo.adjust(options=["--unlock", "--force"])
        else:
            lgr.warning(
                "%s has a detached HEAD, because the target commit "
                "%s has no unique ancestor with branch '%s'",
                repo, target_commit[:8], repo_orig_branch)


def postclone_check_head(ds, remote="origin"):
    repo = ds.repo
    if not repo.commit_exists("HEAD"):
        # HEAD points to an unborn branch. A likely cause of this is that the
        # remote's main branch is something other than master but HEAD wasn't
        # adjusted accordingly.
        #
        # Let's choose the most recently updated remote ref (according to
        # commit date). In the case of a submodule, switching to a ref with
        # commits prevents .update_submodule() from failing. It is likely that
        # the ref includes the registered commit, but we don't have the
        # information here to know for sure. If it doesn't, .update_submodule()
        # will check out a detached HEAD.
        remote_branches = (
            b["refname:strip=2"] for b in repo.for_each_ref_(
                fields="refname:strip=2", sort="-committerdate",
                pattern="refs/remotes/" + remote))
        for rbranch in remote_branches:
            if rbranch in [remote + "/git-annex", "HEAD"]:
                continue
            if rbranch.startswith(remote + "/adjusted/"):
                # If necessary for this file system, a downstream
                # git-annex-init call will handle moving into an
                # adjusted state.
                continue
            repo.call_git(["checkout", "-b",
                           rbranch[len(remote) + 1:],  # drop "<remote>/"
                           "--track", rbranch])
            lgr.debug("Checked out local branch from %s", rbranch)
            return
        lgr.warning("Cloned %s but could not find a branch "
                    "with commits", ds.path)


def postclone_preannex_cfg_ria(ds, remote="origin"):

    # We need to annex-ignore the remote before annex-init is called on the clone,
    # due to issues 5186 and 5253 (and we would have done it afterwards anyway).
    # annex/objects in RIA stores is special for several reasons.
    # 1. the remote doesn't know about it (no actual local annex for the remote)
    # 2. RIA may use hashdir mixed, copying data to it via git-annex (if cloned
    #    via ssh or local) would make it see a bare repo and establish a
    #    hashdir lower annex object tree.
    # 3. We want the ORA remote to receive all data for the store, so its
    #    objects could be moved into archives (the main point of a RIA store).

    # Note, that this function might need an enhancement as theoretically a RIA
    # store could also hold simple standard annexes w/o an intended ORA remote.
    # This needs the introduction of a new version label in RIA datasets, making
    # the following call conditional.
    ds.config.set(f'remote.{remote}.annex-ignore', 'true', where='local')


def postclonecfg_ria(ds, props, remote="origin"):
    """Configure a dataset freshly cloned from a RIA store"""
    repo = ds.repo

    def get_uuid_from_store(store_url):
        # First figure whether we cloned via SSH, HTTP or local path and then
        # get that config file the same way:
        config_content = None
        scheme = store_url.split(':', 1)[0]
        if scheme in ['http', 'https']:
            try:
                config_content = download_url(
                    "{}{}config".format(
                        store_url,
                        '/' if not store_url.endswith('/') else ''))
            except DownloadError as e:
                lgr.debug("Failed to get config file from source:\n%s",
                          exc_str(e))
        elif scheme == 'ssh':
            # TODO: switch the following to proper command abstraction:
            # SSHRemoteIO ignores the path part ATM. No remote CWD! (To be
            # changed with command abstractions). So we need to get that part to
            # have a valid path to the remote's config file:
            cfg_path = PurePosixPath(URL(store_url).path) / 'config'
            op = SSHRemoteIO(store_url)
            try:
                config_content = op.read_file(cfg_path)
            except RIARemoteError as e:
                lgr.debug("Failed to get config file from source: %s",
                          exc_str(e))

        elif scheme == 'file':
            # TODO: switch the following to proper command abstraction:
            op = LocalIO()
            cfg_path = Path(URL(store_url).localpath) / 'config'
            try:
                config_content = op.read_file(cfg_path)
            except (RIARemoteError, OSError) as e:
                lgr.debug("Failed to get config file from source: %s",
                          exc_str(e))
        else:
            lgr.debug("Unknown URL-Scheme %s in %s. Can handle SSH, HTTP or "
                      "FILE scheme URLs.", scheme, props['source'])

        # And read it
        uuid = None
        if config_content:
            # TODO: We might be able to spare the saving to a file.
            #       "git config -f -" is not explicitly documented but happens
            #       to work and would read from stdin. Make sure we know this
            #       works for required git versions and on all platforms.
            with make_tempfile(content=config_content) as cfg_file:
                runner = GitWitlessRunner()
                try:
                    result = runner.run(
                        ['git', 'config', '-f', cfg_file,
                         'datalad.ora-remote.uuid'],
                        protocol=StdOutCapture
                    )
                    uuid = result['stdout'].strip()
                except CommandError as e:
                    # doesn't contain what we are looking for
                    lgr.debug("Found no UUID for ORA special remote at "
                              "'%s' (%s)", remote, exc_str(e))

        return uuid




    # chances are that if this dataset came from a RIA store, its subdatasets
    # may live there too. Place a subdataset source candidate config that makes
    # get probe this RIA store when obtaining subdatasets
    ria_store_url = props['source'].split('#', maxsplit=1)[0]
    ds.config.set(
        # we use the label 'origin' for this candidate in order to not have to
        # generate a complicated name from the actual source specification.
        # we pick a cost of 200 to sort it before datalad's default candidates
        # for non-RIA URLs, because they prioritize hierarchical layouts that
        # cannot be found in a RIA store
        'datalad.get.subdataset-source-candidate-200origin',
        # use the entire original URL, up to the fragment + plus dataset ID
        # placeholder, this should make things work with any store setup we
        # support (paths, ports, ...)
        ria_store_url + '#{id}',
        where='local')

    # setup publication dependency, if a corresponding special remote exists
    # and was enabled (there could be RIA stores that actually only have repos)
    # make this function be a generator
    ora_remotes = [s for s in ds.siblings('query', result_renderer='disabled')
                   if s.get('annex-externaltype') == 'ora']
    # get full special remotes' config for access to stored URL
    srs = repo.get_special_remotes() \
        if hasattr(repo, 'get_special_remotes') else dict()

    has_only_disabled_ora = \
        not ora_remotes and \
        any(r.get('externaltype') == 'ora' for r in srs.values())

    def match_in_urls(special_remote_cfg, url_to_match):
        # Figure whether either `url` or `push-url` in an ORA remote's config
        # match a given URL (to a RIA store).
        return special_remote_cfg['url'].startswith(url_to_match) or \
               (special_remote_cfg['push-url'].startswith(url_to_match)
                if 'push-url' in special_remote_cfg else False)

    no_enabled_ora_matches_url = \
        all(not match_in_urls(srs[r['annex-uuid']], ria_store_url)
            for r in ora_remotes)

    if has_only_disabled_ora or no_enabled_ora_matches_url:

        # No ORA remote autoenabled, but configuration known about at least one,
        # or enabled ORA remotes seem to not match clone URL.
        # Let's check the remote's config for datalad.ora-remote.uuid as stored
        # by create-sibling-ria and try enabling that one.
        lgr.debug("Found no autoenabled ORA special remote. Trying to look it "
                  "up in source config ...")

        org_uuid = get_uuid_from_store(props['giturl'])

        # Now, enable it. If annex-init didn't fail to enable it as stored, we
        # wouldn't end up here, so enable with store URL as suggested by the URL
        # we cloned from.
        if org_uuid:
            if org_uuid in srs.keys():
                # TODO: - Double-check autoenable value and only do this when
                #         true?
                #       - What if still fails? -> Annex shouldn't change config
                #         in that case

                # we only need the store:
                new_url = props['source'].split('#')[0]
                try:
                    repo.enable_remote(srs[org_uuid]['name'],
                                       options=['url={}'.format(new_url)]
                                       )
                    lgr.info("Reconfigured %s for %s",
                             srs[org_uuid]['name'], new_url)
                    # update ora_remotes for considering publication dependency
                    # below
                    ora_remotes = [s for s in
                                   ds.siblings('query',
                                               result_renderer='disabled')
                                   if s.get('annex-externaltype', None) ==
                                   'ora']
                except CommandError as e:
                    lgr.debug("Failed to reconfigure ORA special remote: %s",
                              exc_str(e))
            else:
                lgr.debug("Unknown ORA special remote uuid at '%s': %s",
                          remote, org_uuid)

    # Set publication dependency for `remote` on the respective ORA remote:
    if ora_remotes:
        url_matching_remotes = [r for r in ora_remotes
                                if srs[r['annex-uuid']]['url'] == ria_store_url]

        if len(url_matching_remotes) == 1:
            # We have exactly one ORA remote with the same store URL we used for
            # cloning (includes previously reconfigured remote).
            # Set publication dependency:
            yield from ds.siblings('configure',
                                   name=remote,
                                   publish_depends=url_matching_remotes[0]['name'],
                                   result_filter=None,
                                   result_renderer='disabled')

        elif not url_matching_remotes:
            # No matches but we have successfully autoenabled ORA remotes. Could
            # be the same store accessed by different method (cloning via HTTP
            # but special remote access via SSH). We can confidently set
            # publication dependency if the store knows the UUID.
            org_uuid = get_uuid_from_store(props['giturl'])
            uuid_matching_remotes = [r for r in ora_remotes
                                     if r['annex-uuid'] == org_uuid]
            if uuid_matching_remotes:
                # Multiple uuid matches are actually possible via same-as.
                # However, in that case we can't decide which one is supposed to
                # be used with publishing to `remote`.
                if len(uuid_matching_remotes) == 1:
                    yield from ds.siblings(
                        'configure',
                        name=remote,
                        publish_depends=uuid_matching_remotes[0]['name'],
                        result_filter=None,
                        result_renderer='disabled')
                else:
                    lgr.warning(
                        "Found multiple matching ORA remotes. Couldn't decide "
                        "which one publishing to '%s' should depend on: %s."
                        " Consider running 'datalad siblings configure -s "
                        "%s --publish-depends ORAREMOTENAME' to set "
                        "publication dependency manually.",
                        remote,
                        [r['name'] for r in uuid_matching_remotes],
                        remote)

        else:
            # We have multiple ORA remotes with the same store URL we cloned
            # from.
            lgr.warning("Found multiple matching ORA remotes. Couldn't decide "
                        "which one publishing to '%s' should depend on: %s."
                        " Consider running 'datalad siblings configure -s "
                        "%s --publish-depends ORAREMOTENAME' to set "
                        "publication dependency manually.",
                        remote,
                        [r['name'] for r in url_matching_remotes],
                        remote)


def postclonecfg_annexdataset(ds, reckless, description=None, remote="origin"):
    """If ds "knows annex" -- annex init it, set into reckless etc

    Provides additional tune up to a possibly an annex repo, e.g.
    "enables" reckless mode, sets up description
    """
    # in any case check whether we need to annex-init the installed thing:
    if not knows_annex(ds.path):
        # not for us
        return

    # init annex when traces of a remote annex can be detected
    if reckless == 'auto':
        lgr.debug(
            "Instruct annex to hardlink content in %s from local "
            "sources, if possible (reckless)", ds.path)
        ds.config.set(
            'annex.hardlink', 'true', where='local', reload=True)

    lgr.debug("Initializing annex repo at %s", ds.path)
    # Note, that we cannot enforce annex-init via AnnexRepo().
    # If such an instance already exists, its __init__ will not be executed.
    # Therefore do quick test once we have an object and decide whether to call
    # its _init().
    #
    # Additionally, call init if we need to add a description (see #1403),
    # since AnnexRepo.__init__ can only do it with create=True
    repo = AnnexRepo(ds.path, init=True)
    if not repo.is_initialized() or description:
        repo._init(description=description)
    if reckless == 'auto' or (reckless and reckless.startswith('shared-')):
        repo.call_annex(['untrust', 'here'])

    elif reckless == 'ephemeral':
        # with ephemeral we declare 'here' as 'dead' right away, whenever
        # we symlink the remote's annex, since availability from 'here' should
        # not be propagated for an ephemeral clone when we publish back to
        # the remote.
        # This will cause stuff like this for a locally present annexed file:
        # % git annex whereis d1
        # whereis d1 (0 copies) failed
        # BUT this works:
        # % git annex find . --not --in here
        # % git annex find . --in here
        # d1

        # we don't want annex copy-to <remote>
        ds.config.set(
            f'remote.{remote}.annex-ignore', 'true',
            where='local')

        ds.repo.set_remote_dead('here')

        if check_symlink_capability(ds.repo.dot_git / 'dl_link_test',
                                    ds.repo.dot_git / 'dl_target_test'):
            # symlink the annex to avoid needless copies in an ephemeral clone
            annex_dir = ds.repo.dot_git / 'annex'
            origin_annex_url = ds.config.get(f"remote.{remote}.url", None)
            origin_git_path = None
            if origin_annex_url:
                try:
                    # Deal with file:// scheme URLs as well as plain paths.
                    # If origin isn't local, we have nothing to do.
                    origin_git_path = Path(RI(origin_annex_url).localpath)

                    # we are local; check for a bare repo first to not mess w/
                    # the path
                    if GitRepo(origin_git_path, create=False).bare:
                        # origin is a bare repo -> use path as is
                        pass
                    elif origin_git_path.name != '.git':
                        origin_git_path /= '.git'
                except ValueError:
                    # Note, that accessing localpath on a non-local RI throws
                    # ValueError rather than resulting in an AttributeError.
                    # TODO: Warning level okay or is info level sufficient?
                    # Note, that setting annex-dead is independent of
                    # symlinking .git/annex. It might still make sense to
                    # have an ephemeral clone that doesn't propagate its avail.
                    # info. Therefore don't fail altogether.
                    lgr.warning("reckless=ephemeral mode: %s doesn't seem "
                                "local: %s\nno symlinks being used",
                                remote, origin_annex_url)
            if origin_git_path:
                # TODO make sure that we do not delete any unique data
                rmtree(str(annex_dir)) \
                    if not annex_dir.is_symlink() else annex_dir.unlink()
                annex_dir.symlink_to(origin_git_path / 'annex',
                                     target_is_directory=True)
        else:
            # TODO: What level? + note, that annex-dead is independent
            lgr.warning("reckless=ephemeral mode: Unable to create symlinks on "
                        "this file system.")

    srs = {True: [], False: []}  # special remotes by "autoenable" key
    remote_uuids = None  # might be necessary to discover known UUIDs

    repo_config = repo.config
    # Note: The purpose of this function is to inform the user. So if something
    # looks misconfigured, we'll warn and move on to the next item.
    for uuid, config in repo.get_special_remotes().items():
        sr_name = config.get('name', None)
        if sr_name is None:
            lgr.warning(
                'Ignoring special remote %s because it does not have a name. '
                'Known information: %s',
                uuid, config)
            continue
        sr_autoenable = config.get('autoenable', False)
        try:
            sr_autoenable = ensure_bool(sr_autoenable)
        except ValueError:
            lgr.warning(
                'Failed to process "autoenable" value %r for sibling %s in '
                'dataset %s as bool.'
                'You might need to enable it later manually and/or fix it up to'
                ' avoid this message in the future.',
                sr_autoenable, sr_name, ds.path)
            continue

        # If it looks like a type=git special remote, make sure we have up to
        # date information. See gh-2897.
        if sr_autoenable and repo_config.get("remote.{}.fetch".format(sr_name)):
            try:
                repo.fetch(remote=sr_name)
            except CommandError as exc:
                lgr.warning("Failed to fetch type=git special remote %s: %s",
                            sr_name, exc_str(exc))

        # determine whether there is a registered remote with matching UUID
        if uuid:
            if remote_uuids is None:
                remote_uuids = {
                    # Check annex-config-uuid first. For sameas annex remotes,
                    # this will point to the UUID for the configuration (i.e.
                    # the key returned by get_special_remotes) rather than the
                    # shared UUID.
                    (repo_config.get('remote.%s.annex-config-uuid' % r) or
                     repo_config.get('remote.%s.annex-uuid' % r))
                    for r in repo.get_remotes()
                }
            if uuid not in remote_uuids:
                srs[sr_autoenable].append(sr_name)

    if srs[True]:
        lgr.debug(
            "configuration for %s %s added because of autoenable,"
            " but no UUIDs for them yet known for dataset %s",
            # since we are only at debug level, we could call things their
            # proper names
            single_or_plural("special remote",
                             "special remotes", len(srs[True]), True),
            ", ".join(srs[True]),
            ds.path
        )

    if srs[False]:
        # if has no auto-enable special remotes
        lgr.info(
            'access to %s %s not auto-enabled, enable with:\n'
            '\t\tdatalad siblings -d "%s" enable -s %s',
            # but since humans might read it, we better confuse them with our
            # own terms!
            single_or_plural("dataset sibling",
                             "dataset siblings", len(srs[False]), True),
            ", ".join(srs[False]),
            ds.path,
            srs[False][0] if len(srs[False]) == 1 else "SIBLING",
        )

    # we have just cloned the repo, so it has a remote `remote`, configure any
    # reachable origin of origins
    yield from configure_origins(ds, ds, remote=remote)


_handle_possible_annex_dataset = postclonecfg_annexdataset


def configure_origins(cfgds, probeds, label=None, remote="origin"):
    """Configure any discoverable local dataset sibling as a remote

    Parameters
    ----------
    cfgds : Dataset
      Dataset to receive the remote configurations
    probeds : Dataset
      Dataset to start looking for `remote` remotes. May be identical with
      `cfgds`.
    label : int, optional
      Each discovered remote will be configured as a remote under the name
      '<remote>-<label>'. If no label is given, '2' will be used by default,
      given that there is typically a remote named `remote` already.
    remote : str, optional
      Name of the default remote on clone.
    """
    if label is None:
        label = 1
    # let's look at the URL for that remote and see if it is a local
    # dataset
    origin_url = probeds.config.get(f'remote.{remote}.url')
    if not origin_url:
        # no remote with default name, nothing to do
        return
    if not cfgds.config.obtain(
            'datalad.install.inherit-local-origin',
            default=True):
        # no inheritance wanted
        return
    if not isinstance(RI(origin_url), PathRI):
        # not local path
        return

    # no need to reconfigure original/direct remote again
    if cfgds != probeds:
        # prevent duplicates
        known_remote_urls = set(
            cfgds.config.get(r + '.url', None)
            for r in cfgds.config.sections()
            if r.startswith('remote.')
        )
        if origin_url not in known_remote_urls:
            yield from cfgds.siblings(
                'configure',
                # no chance for conflict, can only be the second configured
                # remote
                name='{}-{}'.format(remote, label),
                url=origin_url,
                # fetch to get all annex info
                fetch=True,
                result_renderer='disabled',
                on_failure='ignore',
            )
    # and dive deeper
    # given the clone source is a local dataset, we can have a
    # cheap look at it, and configure its own `remote` as a remote
    # (if there is any), and benefit from additional annex availability
    yield from configure_origins(
        cfgds,
        Dataset(probeds.pathobj / origin_url),
        label=label + 1,
        remote=remote)


def _get_tracking_source(ds):
    """Returns name and url of a potential configured source
    tracking remote"""
    vcs = ds.repo
    # if we have a remote, let's check the location of that remote
    # for the presence of the desired submodule

    remote_name, tracking_branch = vcs.get_tracking_branch()
    if not remote_name and isinstance(vcs, AnnexRepo):
        # maybe cloned from a source repo that was in adjusted mode
        # https://github.com/datalad/datalad/issues/3969
        remote_name, tracking_branch = vcs.get_tracking_branch(
            corresponding=False)
    # TODO: better default `None`? Check where we might rely on '':
    remote_url = ''
    if remote_name:
        remote_url = vcs.get_remote_url(remote_name, push=False)

    return remote_name, remote_url


def _get_installationpath_from_url(url):
    """Returns a relative path derived from the trailing end of a URL

    This can be used to determine an installation path of a Dataset
    from a URL, analog to what `git clone` does.
    """
    ri = RI(url)
    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL
        path = ri.path.rstrip('/')
        path = urlunquote(path) if path else ri.hostname
        if '/' in path:
            path = path.split('/')
            if path[-1] == '.git':
                path = path[-2]
            else:
                path = path[-1]
    else:
        path = Path(url).parts[-1]
    if path.endswith('.git'):
        path = path[:-4]
    return path


def decode_source_spec(spec, cfg=None):
    """Decode information from a clone source specification

    Parameters
    ----------
    spec : str
      Any supported clone source specification
    cfg : ConfigManager, optional
      Configuration will be queried from the instance (i.e. from a particular
      dataset). If None is given, the global DataLad configuration will be
      queried.

    Returns
    -------
    dict
      The value of each decoded property is stored under its own key in this
      dict. By default the following keys are return: 'type', a specification
      type label {'giturl', 'dataladri', 'ria'}; 'source' the original
      source specification; 'giturl' a URL for the source that is a suitable
      source argument for git-clone; 'version' a version-identifer, if present
      (None else); 'default_destpath' a relative path that that can be used as
      a clone destination.
    """
    if cfg is None:
        from datalad import cfg
    # standard property dict composition
    props = dict(
        source=spec,
        version=None,
    )

    # Git never gets to see these URLs, so let's manually apply any
    # rewrite configuration Git might know about.
    # Note: We need to rewrite before parsing, otherwise parsing might go wrong.
    # This is particularly true for insteadOf labels replacing even the URL
    # scheme.
    spec = cfg.rewrite_url(spec)
    # common starting point is a RI instance, support for accepting an RI
    # instance is kept for backward-compatibility reasons
    source_ri = RI(spec) if not isinstance(spec, RI) else spec

    # scenario switch, each case must set 'giturl' at the very minimum
    if isinstance(source_ri, DataLadRI):
        # we have got our DataLadRI as the source, so expand it
        props['type'] = 'dataladri'
        props['giturl'] = source_ri.as_git_url()
    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):
        # parse a RIA URI
        dsid, version = source_ri.fragment.split('@', maxsplit=1) \
            if '@' in source_ri.fragment else (source_ri.fragment, None)
        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'
        if re.match(uuid_regex, dsid):
            trace = '{}/{}'.format(dsid[:3], dsid[3:])
            default_destpath = dsid
        elif dsid.startswith('~'):
            trace = 'alias/{}'.format(dsid[1:])
            default_destpath = dsid[1:]
        else:
            raise ValueError(
                'RIA URI not recognized, no valid dataset ID or other supported '
                'scheme: {}'.format(spec))
        # now we cancel the fragment in the original URL, but keep everything else
        # in order to be able to support the various combinations of ports, paths,
        # and everything else
        source_ri.fragment = ''
        # strip the custom protocol and go with standard one
        source_ri.scheme = source_ri.scheme[4:]
        # take any existing path, and add trace to dataset within the store
        source_ri.path = '{urlpath}{urldelim}{trace}'.format(
            urlpath=source_ri.path if source_ri.path else '',
            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',
            trace=trace,
        )
        props.update(
            type='ria',
            giturl=str(source_ri),
            version=version,
            default_destpath=default_destpath,
        )
    else:
        # let's assume that anything else is a URI that Git can handle
        props['type'] = 'giturl'
        # use original input verbatim
        props['giturl'] = spec

    if 'default_destpath' not in props:
        # if we still have no good idea on where a dataset could be cloned to if no
        # path was given, do something similar to git clone and derive the path from
        # the source
        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])

    return props

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user