{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Plumbing command for dataset installation\"\"\"\n\n\nimport logging\nimport os\nimport re\nfrom os.path import expanduser\nfrom collections import OrderedDict\nfrom urllib.parse import unquote as urlunquote\n\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import eval_results\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.common_opts import (\n    location_description,\n    reckless_opt,\n)\nfrom datalad.log import log_progress\nfrom datalad.support.gitrepo import (\n    GitRepo,\n    GitCommandError,\n)\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.constraints import (\n    EnsureNone,\n    EnsureStr,\n    EnsureKeyChoice,\n)\nfrom datalad.support.param import Parameter\nfrom datalad.support.network import (\n    get_local_file_url,\n    URL,\n    RI,\n    DataLadRI,\n    PathRI,\n)\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\nfrom datalad.utils import (\n    rmtree,\n    assure_bool,\n    knows_annex,\n    Path,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    resolve_path,\n    require_dataset,\n    EnsureDataset,\n)\nfrom datalad.distribution.utils import (\n    _get_flexible_source_candidates,\n)\n\n__docformat__ = 'restructuredtext'\n\nlgr = logging.getLogger('datalad.core.distributed.clone')\n\n\n@build_doc\nclass Clone(Interface):\n    \"\"\"Obtain a dataset (copy) from a URL or local directory\n\n    The purpose of this command is to obtain a new clone (copy) of a dataset\n    and place it into a not-yet-existing or empty directory. As such `clone`\n    provides a strict subset of the functionality offered by `install`. Only a\n    single dataset can be obtained, and immediate recursive installation of\n    subdatasets is not supported. However, once a (super)dataset is installed\n    via `clone`, any content, including subdatasets can be obtained by a\n    subsequent `get` command.\n\n    Primary differences over a direct `git clone` call are 1) the automatic\n    initialization of a dataset annex (pure Git repositories are equally\n    supported); 2) automatic registration of the newly obtained dataset\n    as a subdataset (submodule), if a parent dataset is specified; and\n    3) support for datalad's resource identifiers and automatic configurable\n    generation of alternative access URL for common cases (such as appending\n    '.git' to the URL in case the accessing the base URL failed).\n\n    || PYTHON >>By default, the command returns a single Dataset instance for\n    an installed dataset, regardless of whether it was newly installed ('ok'\n    result), or found already installed from the specified source ('notneeded'\n    result).<< PYTHON ||\n    \"\"\"\n    # by default ignore everything but install results\n    # i.e. no \"add to super dataset\"\n    result_filter = EnsureKeyChoice('action', ('install',))\n    # very frequently this command will yield exactly one installed dataset\n    # spare people the pain of going through a list by default\n    return_type = 'item-or-list'\n    # as discussed in #1409 and #1470, we want to return dataset instances\n    # matching what is actually available after command completion (and\n    # None for any failed dataset installation)\n    result_xfm = 'successdatasets-or-none'\n\n    _examples_ = [\n        dict(text=\"Install a dataset from Github into the current directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install a dataset into a specific directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git', path='myfavpodcasts')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git \"\n             \"myfavpodcasts\"),\n        dict(text=\"Install a dataset as a subdataset into the current dataset\",\n             code_py=\"clone(dataset='.', \"\n             \"source='https://github.com/datalad-datasets/longnow-podcasts.git')\",\n             code_cmd=\"datalad clone -d . \"\n             \"--source='https://github.com/datalad-datasets/longnow-podcasts.git'\"),\n    ]\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"(parent) dataset to clone into. If given, the newly cloned\n            dataset is registered as a subdataset of the parent. Also, if given,\n            relative paths are interpreted as being relative to the parent\n            dataset, and not relative to the working directory.\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        source=Parameter(\n            args=(\"source\",),\n            metavar='SOURCE',\n            doc=\"\"\"URL, DataLad resource identifier, local path or instance of\n            dataset to be cloned\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            nargs=\"?\",\n            doc=\"\"\"path to clone into.  If no `path` is provided a\n            destination path will be derived from a source URL\n            similar to :command:`git clone`\"\"\"),\n        description=location_description,\n        reckless=reckless_opt,\n    )\n\n    @staticmethod\n    @datasetmethod(name='clone')\n    @eval_results\n    def __call__(\n            source,\n            path=None,\n            dataset=None,\n            description=None,\n            reckless=None):\n        # legacy compatibility\n        if reckless is True:\n            # so that we can forget about how things used to be\n            reckless = 'auto'\n\n        # did we explicitly get a dataset to install into?\n        # if we got a dataset, path will be resolved against it.\n        # Otherwise path will be resolved first.\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='cloning') \\\n            if dataset is not None else dataset\n        refds_path = ds.path if ds else None\n\n        if isinstance(source, Dataset):\n            source = source.path\n\n        if source == path:\n            # even if they turn out to be identical after resolving symlinks\n            # and more sophisticated witchcraft, it would still happily say\n            # \"it appears to be already installed\", so we just catch an\n            # obviously pointless input combination\n            raise ValueError(\n                \"clone `source` and destination `path` are identical [{}]. \"\n                \"If you are trying to add a subdataset simply use `save`\".format(\n                    path))\n\n        if path is not None:\n            path = resolve_path(path, dataset)\n\n        # derive target from source:\n        if path is None:\n            # we got nothing but a source. do something similar to git clone\n            # and derive the path from the source and continue\n            # since this is a relative `path`, resolve it:\n            # we are not going to reuse the decoded URL, as this is done for\n            # all source candidates in clone_dataset(), we just use to determine\n            # a destination path here in order to perform a bunch of additional\n            # checks that shall not pollute the helper function\n            source_ = decode_source_spec(\n                source, cfg=None if ds is None else ds.config)\n            path = resolve_path(source_['default_destpath'], dataset)\n            lgr.debug(\"Determined clone target path from source\")\n        lgr.debug(\"Resolved clone target path to: '%s'\", path)\n\n        # there is no other way -- my intoxicated brain tells me\n        assert(path is not None)\n\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            refds=refds_path,\n            source_url=source)\n\n        try:\n            # this will implicitly cause pathlib to run a bunch of checks\n            # whether the present path makes any sense on the platform\n            # we are running on -- we don't care if the path actually\n            # exists at this point, but we want to abort early if the path\n            # spec is determined to be useless\n            path.exists()\n        except OSError as e:\n            yield get_status_dict(\n                status='error',\n                path=path,\n                message=('cannot handle target path: %s', exc_str(e)),\n                **result_props)\n            return\n\n        destination_dataset = Dataset(path)\n        result_props['ds'] = destination_dataset\n\n        if ds is not None and ds.pathobj not in path.parents:\n            yield get_status_dict(\n                status='error',\n                message=(\"clone target path '%s' not in specified target dataset '%s'\",\n                         path, ds),\n                **result_props)\n            return\n\n        # perform the actual cloning operation\n        yield from clone_dataset(\n            [source],\n            destination_dataset,\n            reckless,\n            description,\n            result_props,\n            cfg=None if ds is None else ds.config,\n        )\n\n        # TODO handle any 'version' property handling and verification using a dedicated\n        # public helper\n\n        if ds is not None:\n            # we created a dataset in another dataset\n            # -> make submodule\n            for r in ds.save(\n                    path,\n                    return_type='generator',\n                    result_filter=None,\n                    result_xfm=None,\n                    on_failure='ignore'):\n                yield r\n\n\ndef clone_dataset(\n        srcs,\n        destds,\n        reckless=None,\n        description=None,\n        result_props=None,\n        cfg=None):\n    \"\"\"Internal helper to perform cloning without sanity checks (assumed done)\n\n    This helper does not handle any saving of subdataset modification or adding\n    in a superdataset.\n\n    Parameters\n    ----------\n    srcs : list\n      Any suitable clone source specifications (paths, URLs)\n    destds : Dataset\n      Dataset instance for the clone destination\n    reckless : {None, 'auto'}, optional\n      Mode switch to put cloned dataset into throw-away configurations, i.e.\n      sacrifice data safety for performance or resource footprint.\n    description : str, optional\n      Location description for the annex of the dataset clone (if there is any).\n    result_props : dict, optional\n      Default properties for any yielded result, passed on to get_status_dict().\n    cfg : ConfigManager, optional\n      Configuration will be queried from this instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Yields\n    ------\n    dict\n      DataLad result records\n    \"\"\"\n    if not result_props:\n        # in case the caller had no specific idea on how results should look\n        # like, provide sensible defaults\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            ds=destds,\n        )\n\n    dest_path = destds.pathobj\n\n    # decode all source candidate specifications\n    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]\n\n    # now expand the candidate sources with additional variants of the decoded\n    # giturl, while duplicating the other properties in the additional records\n    # for simplicity. The hope is to overcome a few corner cases and be more\n    # robust than git clone\n    candidate_sources = [\n        dict(props, giturl=s) for props in candidate_sources\n        for s in _get_flexible_source_candidates(props['giturl'])\n    ]\n\n    # important test! based on this `rmtree` will happen below after failed clone\n    dest_path_existed = dest_path.exists()\n    if dest_path_existed and any(dest_path.iterdir()):\n        if destds.is_installed():\n            # check if dest was cloned from the given source before\n            # this is where we would have installed this from\n            # this is where it was actually installed from\n            track_name, track_url = _get_tracking_source(destds)\n            try:\n                # this will get us track_url in system native path conventions,\n                # whenever it is a path (and not a URL)\n                # this is needed to match it to any potentially incoming local\n                # source path in the 'notneeded' test below\n                track_path = str(Path(track_url))\n            except Exception:\n                # this should never happen, because Path() will let any non-path stringification\n                # pass through unmodified, but we do not want any potential crash due to\n                # pathlib behavior changes\n                lgr.debug(\"Unexpected behavior of pathlib!\")\n                track_path = None\n            for cand in candidate_sources:\n                src = cand['giturl']\n                if track_url == src \\\n                        or get_local_file_url(track_url, compatibility='git') == src \\\n                        or track_path == expanduser(src):\n                    yield get_status_dict(\n                        status='notneeded',\n                        message=(\"dataset %s was already cloned from '%s'\",\n                                 destds,\n                                 src),\n                        **result_props)\n                    return\n        # anything else is an error\n        yield get_status_dict(\n            status='error',\n            message='target path already exists and not empty, refuse to clone into target path',\n            **result_props)\n        return\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Cloning dataset to %s', destds,\n        total=len(candidate_sources),\n        label='Clone attempt',\n        unit=' Candidate locations',\n    )\n    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url\n    for cand in candidate_sources:\n        try:\n            log_progress(\n                lgr.info,\n                'cloneds',\n                'Attempting to clone from %s to %s', cand['giturl'], dest_path,\n                update=1,\n                increment=True)\n            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35\n            # doesn't make it happen seemlessly\n            GitRepo.clone(path=str(dest_path), url=cand['giturl'], create=True)\n\n        except GitCommandError as e:\n            error_msgs[cand['giturl']] = e\n            lgr.debug(\"Failed to clone from URL: %s (%s)\",\n                      cand['giturl'], exc_str(e))\n            if dest_path.exists():\n                lgr.debug(\"Wiping out unsuccessful clone attempt at: %s\",\n                          dest_path)\n                # We must not just rmtree since it might be curdir etc\n                # we should remove all files/directories under it\n                # TODO stringification can be removed once patlib compatible\n                # or if PY35 is no longer supported\n                rmtree(str(dest_path), children_only=dest_path_existed)\n            # Whenever progress reporting is enabled, as it is now,\n            # we end up without e.stderr since it is \"processed\" out by\n            # GitPython/our progress handler.\n            e_stderr = e.stderr\n            from datalad.support.gitrepo import GitPythonProgressBar\n            if not e_stderr and GitPythonProgressBar._last_error_lines:\n                e_stderr = os.linesep.join(GitPythonProgressBar._last_error_lines)\n            if 'could not create work tree' in e_stderr.lower():\n                # this cannot be fixed by trying another URL\n                re_match = re.match(r\".*fatal: (.*)$\", e_stderr,\n                                    flags=re.MULTILINE | re.DOTALL)\n                # cancel progress bar\n                log_progress(\n                    lgr.info,\n                    'cloneds',\n                    'Completed clone attempts for %s', destds\n                )\n                yield get_status_dict(\n                    status='error',\n                    message=re_match.group(1) if re_match else \"stderr: \" + e_stderr,\n                    **result_props)\n                return\n            # next candidate\n            continue\n\n        # TODO perform any post-processing that needs to know details of the clone\n        # source\n\n        # do not bother with other sources if succeeded\n        break\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Completed clone attempts for %s', destds\n    )\n\n    if not destds.is_installed():\n        if len(error_msgs):\n            if all(not e.stdout and not e.stderr for e in error_msgs.values()):\n                # there is nothing we can learn from the actual exception,\n                # the exit code is uninformative, the command is predictable\n                error_msg = \"Failed to clone from all attempted sources: %s\"\n                error_args = list(error_msgs.keys())\n            else:\n                error_msg = \"Failed to clone from any candidate source URL. \" \\\n                            \"Encountered errors per each url were:\\n- %s\"\n                error_args = '\\n- '.join(\n                    '{}\\n  {}'.format(url, exc_str(exc))\n                    for url, exc in error_msgs.items()\n                )\n        else:\n            # yoh: Not sure if we ever get here but I felt that there could\n            #      be a case when this might happen and original error would\n            #      not be sufficient to troubleshoot what is going on.\n            error_msg = \"Awkward error -- we failed to clone properly. \" \\\n                        \"Although no errors were encountered, target \" \\\n                        \"dataset at %s seems to be not fully installed. \" \\\n                        \"The 'succesful' source was: %s\"\n            error_args = (destds.path, cand['giturl'])\n        yield get_status_dict(\n            status='error',\n            message=(error_msg, error_args),\n            **result_props)\n        return\n\n    yield from postclonecfg_annexdataset(\n        destds,\n        reckless,\n        description)\n\n    # yield successful clone of the base dataset now, as any possible\n    # subdataset clone down below will not alter the Git-state of the\n    # parent\n    yield get_status_dict(status='ok', **result_props)\n\n\ndef postclonecfg_annexdataset(ds, reckless, description=None):\n    \"\"\"If ds \"knows annex\" -- annex init it, set into reckless etc\n\n    Provides additional tune up to a possibly an annex repo, e.g.\n    \"enables\" reckless mode, sets up description\n    \"\"\"\n    # in any case check whether we need to annex-init the installed thing:\n    if not knows_annex(ds.path):\n        # not for us\n        return\n\n    # init annex when traces of a remote annex can be detected\n    if reckless:\n        lgr.debug(\n            \"Instruct annex to hardlink content in %s from local \"\n            \"sources, if possible (reckless)\", ds.path)\n        ds.config.add(\n            'annex.hardlink', 'true', where='local', reload=True)\n\n    # we have just cloned the repo, so it has 'origin', configure any\n    # reachable origin of origins\n    yield from configure_origins(ds, ds)\n\n    lgr.debug(\"Initializing annex repo at %s\", ds.path)\n    # Note, that we cannot enforce annex-init via AnnexRepo().\n    # If such an instance already exists, its __init__ will not be executed.\n    # Therefore do quick test once we have an object and decide whether to call its _init().\n    #\n    # Additionally, call init if we need to add a description (see #1403),\n    # since AnnexRepo.__init__ can only do it with create=True\n    repo = AnnexRepo(ds.path, init=True)\n    if not repo.is_initialized() or description:\n        repo._init(description=description)\n    if reckless:\n        repo._run_annex_command('untrust', annex_options=['here'])\n\n    srs = {True: [], False: []}  # special remotes by \"autoenable\" key\n    remote_uuids = None  # might be necessary to discover known UUIDs\n\n    # Note: The purpose of this function is to inform the user. So if something\n    # looks misconfigured, we'll warn and move on to the next item.\n    for uuid, config in repo.get_special_remotes().items():\n        sr_name = config.get('name', None)\n        if sr_name is None:\n            lgr.warning(\n                'Ignoring special remote %s because it does not have a name. '\n                'Known information: %s',\n                uuid, config)\n            continue\n        sr_autoenable = config.get('autoenable', False)\n        try:\n            sr_autoenable = assure_bool(sr_autoenable)\n        except ValueError:\n            lgr.warning(\n                'Failed to process \"autoenable\" value %r for sibling %s in '\n                'dataset %s as bool.  You might need to enable it later '\n                'manually and/or fix it up to avoid this message in the future.',\n                sr_autoenable, sr_name, ds.path)\n            continue\n\n        # determine whether there is a registered remote with matching UUID\n        if uuid:\n            if remote_uuids is None:\n                remote_uuids = {\n                    # Check annex-config-uuid first. For sameas annex remotes,\n                    # this will point to the UUID for the configuration (i.e.\n                    # the key returned by get_special_remotes) rather than the\n                    # shared UUID.\n                    (repo.config.get('remote.%s.annex-config-uuid' % r) or\n                     repo.config.get('remote.%s.annex-uuid' % r))\n                    for r in repo.get_remotes()\n                }\n            if uuid not in remote_uuids:\n                srs[sr_autoenable].append(sr_name)\n\n    if srs[True]:\n        lgr.debug(\n            \"configuration for %s %s added because of autoenable,\"\n            \" but no UUIDs for them yet known for dataset %s\",\n            # since we are only at debug level, we could call things their\n            # proper names\n            single_or_plural(\"special remote\", \"special remotes\", len(srs[True]), True),\n            \", \".join(srs[True]),\n            ds.path\n        )\n\n    if srs[False]:\n        # if has no auto-enable special remotes\n        lgr.info(\n            'access to %s %s not auto-enabled, enable with:\\n\\t\\tdatalad siblings -d \"%s\" enable -s %s',\n            # but since humans might read it, we better confuse them with our\n            # own terms!\n            single_or_plural(\"dataset sibling\", \"dataset siblings\", len(srs[False]), True),\n            \", \".join(srs[False]),\n            ds.path,\n            srs[False][0] if len(srs[False]) == 1 else \"SIBLING\",\n        )\n\n_handle_possible_annex_dataset = postclonecfg_annexdataset\n\n\ndef configure_origins(cfgds, probeds, label=None):\n    \"\"\"Configure any discoverable local dataset 'origin' sibling as a remote\n\n    Parameters\n    ----------\n    cfgds : Dataset\n      Dataset to receive the remote configurations\n    probeds : Dataset\n      Dataset to start looking for 'origin' remotes. May be identical with `cfgds`.\n    label : int, optional\n      Each discovered 'origin' will be configured as a remote under the name\n      'origin-<label>'. If no label is given, '2' will be used by default, given that\n      there is typically a 'origin' remote already.\n    \"\"\"\n    if label is None:\n        label = 2\n    # let's look at the URL for that remote and see if it is a local\n    # dataset\n    origin_url = probeds.config.get('remote.origin.url')\n    if origin_url and cfgds.config.obtain(\n            'datalad.install.inherit-local-origin',\n            default=True) and isinstance(RI(origin_url), PathRI):\n        # given the clone source is a local dataset, we can have a\n        # cheap look at it, and configure its own 'origin' as a remote\n        # (if there is any), and benefit from additional annex availability\n        originorigin_ds = Dataset(origin_url)\n        originorigin_url = originorigin_ds.config.get('remote.origin.url')\n        if originorigin_url:\n            yield from cfgds.siblings(\n                'configure',\n                # no chance for config, can only be the second configured remote\n                name='origin-{}'.format(label),\n                url=originorigin_url,\n                # fetch to get all annex info\n                fetch=True,\n                result_renderer='disabled',\n                on_failure='ignore',\n            )\n        # and dive deeper\n        yield from configure_origins(cfgds, originorigin_ds, label=label + 1)\n\n\ndef _get_tracking_source(ds):\n    \"\"\"Returns name and url of a potential configured source\n    tracking remote\"\"\"\n    vcs = ds.repo\n    # if we have a remote, let's check the location of that remote\n    # for the presence of the desired submodule\n\n    remote_name, tracking_branch = vcs.get_tracking_branch()\n    if not remote_name and isinstance(vcs, AnnexRepo):\n        # maybe cloned from a source repo that was in adjusted mode\n        # https://github.com/datalad/datalad/issues/3969\n        remote_name, tracking_branch = vcs.get_tracking_branch(\n            corresponding=False)\n    # TODO: better default `None`? Check where we might rely on '':\n    remote_url = ''\n    if remote_name:\n        remote_url = vcs.get_remote_url(remote_name, push=False)\n\n    return remote_name, remote_url\n\n\ndef _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path\n\n\ndef decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        # check for store related configuration for SSH access\n        if source_ri.scheme == 'ria+ssh':\n            hostname = cfg.get(\n                'annex.ria-remote.{}.ssh-host'.format(source_ri.hostname),\n                source_ri.hostname)\n            basepath = cfg.get(\n                'annex.ria-remote.{}.base-path'.format(source_ri.hostname),\n                '')\n        else:\n            hostname = source_ri.hostname\n            basepath = ''\n\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if not re.match(uuid_regex, dsid):\n            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))\n        props.update(\n            type='ria',\n            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(\n                proto=source_ri.scheme[4:],\n                host=hostname,\n                delim=':' if source_ri.scheme == 'ria+ssh' else '/',\n                basepath=basepath,\n                pathdelim='/' if basepath else '',\n                id1=dsid[:3],\n                id2=dsid[3:]),\n            version=version,\n            default_destpath=dsid,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        # TODO are there any advantages of going through a URL-reconversion\n        # by the RI instance, instead of using the original input right away?\n        # it feels like we could only accumulate bugs. Keeping how it was,\n        # for now...\n        props['type'] = 'giturl'\n        props['giturl'] = str(source_ri)\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Plumbing command for dataset installation\"\"\"\n\n\nimport logging\nimport os\nimport re\nfrom os.path import expanduser\nfrom collections import OrderedDict\nfrom urllib.parse import unquote as urlunquote\n\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import eval_results\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.common_opts import (\n    location_description,\n    reckless_opt,\n)\nfrom datalad.log import log_progress\nfrom datalad.support.gitrepo import (\n    GitRepo,\n    GitCommandError,\n)\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.constraints import (\n    EnsureNone,\n    EnsureStr,\n    EnsureKeyChoice,\n)\nfrom datalad.support.param import Parameter\nfrom datalad.support.network import (\n    get_local_file_url,\n    URL,\n    RI,\n    DataLadRI,\n    PathRI,\n)\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\nfrom datalad.utils import (\n    rmtree,\n    assure_bool,\n    knows_annex,\n    Path,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    resolve_path,\n    require_dataset,\n    EnsureDataset,\n)\nfrom datalad.distribution.utils import (\n    _get_flexible_source_candidates,\n)\n\n__docformat__ = 'restructuredtext'\n\nlgr = logging.getLogger('datalad.core.distributed.clone')\n\n\n@build_doc\nclass Clone(Interface):\n    \"\"\"Obtain a dataset (copy) from a URL or local directory\n\n    The purpose of this command is to obtain a new clone (copy) of a dataset\n    and place it into a not-yet-existing or empty directory. As such `clone`\n    provides a strict subset of the functionality offered by `install`. Only a\n    single dataset can be obtained, and immediate recursive installation of\n    subdatasets is not supported. However, once a (super)dataset is installed\n    via `clone`, any content, including subdatasets can be obtained by a\n    subsequent `get` command.\n\n    Primary differences over a direct `git clone` call are 1) the automatic\n    initialization of a dataset annex (pure Git repositories are equally\n    supported); 2) automatic registration of the newly obtained dataset\n    as a subdataset (submodule), if a parent dataset is specified; and\n    3) support for datalad's resource identifiers and automatic configurable\n    generation of alternative access URL for common cases (such as appending\n    '.git' to the URL in case the accessing the base URL failed).\n\n    || PYTHON >>By default, the command returns a single Dataset instance for\n    an installed dataset, regardless of whether it was newly installed ('ok'\n    result), or found already installed from the specified source ('notneeded'\n    result).<< PYTHON ||\n    \"\"\"\n    # by default ignore everything but install results\n    # i.e. no \"add to super dataset\"\n    result_filter = EnsureKeyChoice('action', ('install',))\n    # very frequently this command will yield exactly one installed dataset\n    # spare people the pain of going through a list by default\n    return_type = 'item-or-list'\n    # as discussed in #1409 and #1470, we want to return dataset instances\n    # matching what is actually available after command completion (and\n    # None for any failed dataset installation)\n    result_xfm = 'successdatasets-or-none'\n\n    _examples_ = [\n        dict(text=\"Install a dataset from Github into the current directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install a dataset into a specific directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git', path='myfavpodcasts')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git \"\n             \"myfavpodcasts\"),\n        dict(text=\"Install a dataset as a subdataset into the current dataset\",\n             code_py=\"clone(dataset='.', \"\n             \"source='https://github.com/datalad-datasets/longnow-podcasts.git')\",\n             code_cmd=\"datalad clone -d . \"\n             \"--source='https://github.com/datalad-datasets/longnow-podcasts.git'\"),\n    ]\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"(parent) dataset to clone into. If given, the newly cloned\n            dataset is registered as a subdataset of the parent. Also, if given,\n            relative paths are interpreted as being relative to the parent\n            dataset, and not relative to the working directory.\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        source=Parameter(\n            args=(\"source\",),\n            metavar='SOURCE',\n            doc=\"\"\"URL, DataLad resource identifier, local path or instance of\n            dataset to be cloned\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            nargs=\"?\",\n            doc=\"\"\"path to clone into.  If no `path` is provided a\n            destination path will be derived from a source URL\n            similar to :command:`git clone`\"\"\"),\n        description=location_description,\n        reckless=reckless_opt,\n    )\n\n    @staticmethod\n    @datasetmethod(name='clone')\n    @eval_results\n    def __call__(\n            source,\n            path=None,\n            dataset=None,\n            description=None,\n            reckless=None):\n        # legacy compatibility\n        if reckless is True:\n            # so that we can forget about how things used to be\n            reckless = 'auto'\n\n        # did we explicitly get a dataset to install into?\n        # if we got a dataset, path will be resolved against it.\n        # Otherwise path will be resolved first.\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='cloning') \\\n            if dataset is not None else dataset\n        refds_path = ds.path if ds else None\n\n        if isinstance(source, Dataset):\n            source = source.path\n\n        if source == path:\n            # even if they turn out to be identical after resolving symlinks\n            # and more sophisticated witchcraft, it would still happily say\n            # \"it appears to be already installed\", so we just catch an\n            # obviously pointless input combination\n            raise ValueError(\n                \"clone `source` and destination `path` are identical [{}]. \"\n                \"If you are trying to add a subdataset simply use `save`\".format(\n                    path))\n\n        if path is not None:\n            path = resolve_path(path, dataset)\n\n        # derive target from source:\n        if path is None:\n            # we got nothing but a source. do something similar to git clone\n            # and derive the path from the source and continue\n            # since this is a relative `path`, resolve it:\n            # we are not going to reuse the decoded URL, as this is done for\n            # all source candidates in clone_dataset(), we just use to determine\n            # a destination path here in order to perform a bunch of additional\n            # checks that shall not pollute the helper function\n            source_ = decode_source_spec(\n                source, cfg=None if ds is None else ds.config)\n            path = resolve_path(source_['default_destpath'], dataset)\n            lgr.debug(\"Determined clone target path from source\")\n        lgr.debug(\"Resolved clone target path to: '%s'\", path)\n\n        # there is no other way -- my intoxicated brain tells me\n        assert(path is not None)\n\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            refds=refds_path,\n            source_url=source)\n\n        try:\n            # this will implicitly cause pathlib to run a bunch of checks\n            # whether the present path makes any sense on the platform\n            # we are running on -- we don't care if the path actually\n            # exists at this point, but we want to abort early if the path\n            # spec is determined to be useless\n            path.exists()\n        except OSError as e:\n            yield get_status_dict(\n                status='error',\n                path=path,\n                message=('cannot handle target path: %s', exc_str(e)),\n                **result_props)\n            return\n\n        destination_dataset = Dataset(path)\n        result_props['ds'] = destination_dataset\n\n        if ds is not None and ds.pathobj not in path.parents:\n            yield get_status_dict(\n                status='error',\n                message=(\"clone target path '%s' not in specified target dataset '%s'\",\n                         path, ds),\n                **result_props)\n            return\n\n        # perform the actual cloning operation\n        yield from clone_dataset(\n            [source],\n            destination_dataset,\n            reckless,\n            description,\n            result_props,\n            cfg=None if ds is None else ds.config,\n        )\n\n        # TODO handle any 'version' property handling and verification using a dedicated\n        # public helper\n\n        if ds is not None:\n            # we created a dataset in another dataset\n            # -> make submodule\n            for r in ds.save(\n                    path,\n                    return_type='generator',\n                    result_filter=None,\n                    result_xfm=None,\n                    on_failure='ignore'):\n                yield r\n\n\ndef clone_dataset(\n        srcs,\n        destds,\n        reckless=None,\n        description=None,\n        result_props=None,\n        cfg=None):\n    \"\"\"Internal helper to perform cloning without sanity checks (assumed done)\n\n    This helper does not handle any saving of subdataset modification or adding\n    in a superdataset.\n\n    Parameters\n    ----------\n    srcs : list\n      Any suitable clone source specifications (paths, URLs)\n    destds : Dataset\n      Dataset instance for the clone destination\n    reckless : {None, 'auto'}, optional\n      Mode switch to put cloned dataset into throw-away configurations, i.e.\n      sacrifice data safety for performance or resource footprint.\n    description : str, optional\n      Location description for the annex of the dataset clone (if there is any).\n    result_props : dict, optional\n      Default properties for any yielded result, passed on to get_status_dict().\n    cfg : ConfigManager, optional\n      Configuration will be queried from this instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Yields\n    ------\n    dict\n      DataLad result records\n    \"\"\"\n    if not result_props:\n        # in case the caller had no specific idea on how results should look\n        # like, provide sensible defaults\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            ds=destds,\n        )\n\n    dest_path = destds.pathobj\n\n    # decode all source candidate specifications\n    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]\n\n    # now expand the candidate sources with additional variants of the decoded\n    # giturl, while duplicating the other properties in the additional records\n    # for simplicity. The hope is to overcome a few corner cases and be more\n    # robust than git clone\n    candidate_sources = [\n        dict(props, giturl=s) for props in candidate_sources\n        for s in _get_flexible_source_candidates(props['giturl'])\n    ]\n\n    # important test! based on this `rmtree` will happen below after failed clone\n    dest_path_existed = dest_path.exists()\n    if dest_path_existed and any(dest_path.iterdir()):\n        if destds.is_installed():\n            # check if dest was cloned from the given source before\n            # this is where we would have installed this from\n            # this is where it was actually installed from\n            track_name, track_url = _get_tracking_source(destds)\n            try:\n                # this will get us track_url in system native path conventions,\n                # whenever it is a path (and not a URL)\n                # this is needed to match it to any potentially incoming local\n                # source path in the 'notneeded' test below\n                track_path = str(Path(track_url))\n            except Exception:\n                # this should never happen, because Path() will let any non-path stringification\n                # pass through unmodified, but we do not want any potential crash due to\n                # pathlib behavior changes\n                lgr.debug(\"Unexpected behavior of pathlib!\")\n                track_path = None\n            for cand in candidate_sources:\n                src = cand['giturl']\n                if track_url == src \\\n                        or get_local_file_url(track_url, compatibility='git') == src \\\n                        or track_path == expanduser(src):\n                    yield get_status_dict(\n                        status='notneeded',\n                        message=(\"dataset %s was already cloned from '%s'\",\n                                 destds,\n                                 src),\n                        **result_props)\n                    return\n        # anything else is an error\n        yield get_status_dict(\n            status='error',\n            message='target path already exists and not empty, refuse to clone into target path',\n            **result_props)\n        return\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Cloning dataset to %s', destds,\n        total=len(candidate_sources),\n        label='Clone attempt',\n        unit=' Candidate locations',\n    )\n    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url\n    for cand in candidate_sources:\n        try:\n            log_progress(\n                lgr.info,\n                'cloneds',\n                'Attempting to clone from %s to %s', cand['giturl'], dest_path,\n                update=1,\n                increment=True)\n            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35\n            # doesn't make it happen seemlessly\n            GitRepo.clone(path=str(dest_path), url=cand['giturl'], create=True)\n\n        except GitCommandError as e:\n            error_msgs[cand['giturl']] = e\n            lgr.debug(\"Failed to clone from URL: %s (%s)\",\n                      cand['giturl'], exc_str(e))\n            if dest_path.exists():\n                lgr.debug(\"Wiping out unsuccessful clone attempt at: %s\",\n                          dest_path)\n                # We must not just rmtree since it might be curdir etc\n                # we should remove all files/directories under it\n                # TODO stringification can be removed once patlib compatible\n                # or if PY35 is no longer supported\n                rmtree(str(dest_path), children_only=dest_path_existed)\n            # Whenever progress reporting is enabled, as it is now,\n            # we end up without e.stderr since it is \"processed\" out by\n            # GitPython/our progress handler.\n            e_stderr = e.stderr\n            from datalad.support.gitrepo import GitPythonProgressBar\n            if not e_stderr and GitPythonProgressBar._last_error_lines:\n                e_stderr = os.linesep.join(GitPythonProgressBar._last_error_lines)\n            if 'could not create work tree' in e_stderr.lower():\n                # this cannot be fixed by trying another URL\n                re_match = re.match(r\".*fatal: (.*)$\", e_stderr,\n                                    flags=re.MULTILINE | re.DOTALL)\n                # cancel progress bar\n                log_progress(\n                    lgr.info,\n                    'cloneds',\n                    'Completed clone attempts for %s', destds\n                )\n                yield get_status_dict(\n                    status='error',\n                    message=re_match.group(1) if re_match else \"stderr: \" + e_stderr,\n                    **result_props)\n                return\n            # next candidate\n            continue\n\n        # TODO perform any post-processing that needs to know details of the clone\n        # source\n\n        # do not bother with other sources if succeeded\n        break\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Completed clone attempts for %s', destds\n    )\n\n    if not destds.is_installed():\n        if len(error_msgs):\n            if all(not e.stdout and not e.stderr for e in error_msgs.values()):\n                # there is nothing we can learn from the actual exception,\n                # the exit code is uninformative, the command is predictable\n                error_msg = \"Failed to clone from all attempted sources: %s\"\n                error_args = list(error_msgs.keys())\n            else:\n                error_msg = \"Failed to clone from any candidate source URL. \" \\\n                            \"Encountered errors per each url were:\\n- %s\"\n                error_args = '\\n- '.join(\n                    '{}\\n  {}'.format(url, exc_str(exc))\n                    for url, exc in error_msgs.items()\n                )\n        else:\n            # yoh: Not sure if we ever get here but I felt that there could\n            #      be a case when this might happen and original error would\n            #      not be sufficient to troubleshoot what is going on.\n            error_msg = \"Awkward error -- we failed to clone properly. \" \\\n                        \"Although no errors were encountered, target \" \\\n                        \"dataset at %s seems to be not fully installed. \" \\\n                        \"The 'succesful' source was: %s\"\n            error_args = (destds.path, cand['giturl'])\n        yield get_status_dict(\n            status='error',\n            message=(error_msg, error_args),\n            **result_props)\n        return\n\n    yield from postclonecfg_annexdataset(\n        destds,\n        reckless,\n        description)\n\n    # yield successful clone of the base dataset now, as any possible\n    # subdataset clone down below will not alter the Git-state of the\n    # parent\n    yield get_status_dict(status='ok', **result_props)\n\n\ndef postclonecfg_annexdataset(ds, reckless, description=None):\n    \"\"\"If ds \"knows annex\" -- annex init it, set into reckless etc\n\n    Provides additional tune up to a possibly an annex repo, e.g.\n    \"enables\" reckless mode, sets up description\n    \"\"\"\n    # in any case check whether we need to annex-init the installed thing:\n    if not knows_annex(ds.path):\n        # not for us\n        return\n\n    # init annex when traces of a remote annex can be detected\n    if reckless:\n        lgr.debug(\n            \"Instruct annex to hardlink content in %s from local \"\n            \"sources, if possible (reckless)\", ds.path)\n        ds.config.add(\n            'annex.hardlink', 'true', where='local', reload=True)\n\n    # we have just cloned the repo, so it has 'origin', configure any\n    # reachable origin of origins\n    yield from configure_origins(ds, ds)\n\n    lgr.debug(\"Initializing annex repo at %s\", ds.path)\n    # Note, that we cannot enforce annex-init via AnnexRepo().\n    # If such an instance already exists, its __init__ will not be executed.\n    # Therefore do quick test once we have an object and decide whether to call its _init().\n    #\n    # Additionally, call init if we need to add a description (see #1403),\n    # since AnnexRepo.__init__ can only do it with create=True\n    repo = AnnexRepo(ds.path, init=True)\n    if not repo.is_initialized() or description:\n        repo._init(description=description)\n    if reckless:\n        repo._run_annex_command('untrust', annex_options=['here'])\n\n    srs = {True: [], False: []}  # special remotes by \"autoenable\" key\n    remote_uuids = None  # might be necessary to discover known UUIDs\n\n    # Note: The purpose of this function is to inform the user. So if something\n    # looks misconfigured, we'll warn and move on to the next item.\n    for uuid, config in repo.get_special_remotes().items():\n        sr_name = config.get('name', None)\n        if sr_name is None:\n            lgr.warning(\n                'Ignoring special remote %s because it does not have a name. '\n                'Known information: %s',\n                uuid, config)\n            continue\n        sr_autoenable = config.get('autoenable', False)\n        try:\n            sr_autoenable = assure_bool(sr_autoenable)\n        except ValueError:\n            lgr.warning(\n                'Failed to process \"autoenable\" value %r for sibling %s in '\n                'dataset %s as bool.  You might need to enable it later '\n                'manually and/or fix it up to avoid this message in the future.',\n                sr_autoenable, sr_name, ds.path)\n            continue\n\n        # determine whether there is a registered remote with matching UUID\n        if uuid:\n            if remote_uuids is None:\n                remote_uuids = {\n                    # Check annex-config-uuid first. For sameas annex remotes,\n                    # this will point to the UUID for the configuration (i.e.\n                    # the key returned by get_special_remotes) rather than the\n                    # shared UUID.\n                    (repo.config.get('remote.%s.annex-config-uuid' % r) or\n                     repo.config.get('remote.%s.annex-uuid' % r))\n                    for r in repo.get_remotes()\n                }\n            if uuid not in remote_uuids:\n                srs[sr_autoenable].append(sr_name)\n\n    if srs[True]:\n        lgr.debug(\n            \"configuration for %s %s added because of autoenable,\"\n            \" but no UUIDs for them yet known for dataset %s\",\n            # since we are only at debug level, we could call things their\n            # proper names\n            single_or_plural(\"special remote\", \"special remotes\", len(srs[True]), True),\n            \", \".join(srs[True]),\n            ds.path\n        )\n\n    if srs[False]:\n        # if has no auto-enable special remotes\n        lgr.info(\n            'access to %s %s not auto-enabled, enable with:\\n\\t\\tdatalad siblings -d \"%s\" enable -s %s',\n            # but since humans might read it, we better confuse them with our\n            # own terms!\n            single_or_plural(\"dataset sibling\", \"dataset siblings\", len(srs[False]), True),\n            \", \".join(srs[False]),\n            ds.path,\n            srs[False][0] if len(srs[False]) == 1 else \"SIBLING\",\n        )\n\n_handle_possible_annex_dataset = postclonecfg_annexdataset\n\n\ndef configure_origins(cfgds, probeds, label=None):\n    \"\"\"Configure any discoverable local dataset 'origin' sibling as a remote\n\n    Parameters\n    ----------\n    cfgds : Dataset\n      Dataset to receive the remote configurations\n    probeds : Dataset\n      Dataset to start looking for 'origin' remotes. May be identical with `cfgds`.\n    label : int, optional\n      Each discovered 'origin' will be configured as a remote under the name\n      'origin-<label>'. If no label is given, '2' will be used by default, given that\n      there is typically a 'origin' remote already.\n    \"\"\"\n    if label is None:\n        label = 2\n    # let's look at the URL for that remote and see if it is a local\n    # dataset\n    origin_url = probeds.config.get('remote.origin.url')\n    if origin_url and cfgds.config.obtain(\n            'datalad.install.inherit-local-origin',\n            default=True) and isinstance(RI(origin_url), PathRI):\n        # given the clone source is a local dataset, we can have a\n        # cheap look at it, and configure its own 'origin' as a remote\n        # (if there is any), and benefit from additional annex availability\n        originorigin_ds = Dataset(origin_url)\n        originorigin_url = originorigin_ds.config.get('remote.origin.url')\n        if originorigin_url:\n            yield from cfgds.siblings(\n                'configure',\n                # no chance for config, can only be the second configured remote\n                name='origin-{}'.format(label),\n                url=originorigin_url,\n                # fetch to get all annex info\n                fetch=True,\n                result_renderer='disabled',\n                on_failure='ignore',\n            )\n        # and dive deeper\n        yield from configure_origins(cfgds, originorigin_ds, label=label + 1)\n\n\ndef _get_tracking_source(ds):\n    \"\"\"Returns name and url of a potential configured source\n    tracking remote\"\"\"\n    vcs = ds.repo\n    # if we have a remote, let's check the location of that remote\n    # for the presence of the desired submodule\n\n    remote_name, tracking_branch = vcs.get_tracking_branch()\n    if not remote_name and isinstance(vcs, AnnexRepo):\n        # maybe cloned from a source repo that was in adjusted mode\n        # https://github.com/datalad/datalad/issues/3969\n        remote_name, tracking_branch = vcs.get_tracking_branch(\n            corresponding=False)\n    # TODO: better default `None`? Check where we might rely on '':\n    remote_url = ''\n    if remote_name:\n        remote_url = vcs.get_remote_url(remote_name, push=False)\n\n    return remote_name, remote_url\n\n\ndef _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path\n\n\ndef decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        # check for store related configuration for SSH access\n        if source_ri.scheme == 'ria+ssh':\n            hostname = cfg.get(\n                'annex.ria-remote.{}.ssh-host'.format(source_ri.hostname),\n                source_ri.hostname)\n            basepath = cfg.get(\n                'annex.ria-remote.{}.base-path'.format(source_ri.hostname),\n                '')\n        else:\n            hostname = source_ri.hostname\n            basepath = ''\n\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if not re.match(uuid_regex, dsid):\n            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))\n        # now we cancel the fragment in the original URL, but keep everthing else\n        # in order to be able to support the various combinations of ports, paths,\n        # and everything else\n        source_ri.fragment = ''\n        # strip the custom protocol and go with standard one\n        source_ri.scheme = source_ri.scheme[4:]\n        # take any existing path, and add trace to dataset within the store\n        source_ri.path = '{urlpath}{urldelim}{basepath}{basedelim}{trace}'.format(\n            urlpath=source_ri.path if source_ri.path else '',\n            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',\n            basepath=basepath,\n            basedelim='' if not basepath or basepath.endswith('/') else '/',\n            trace='{}/{}'.format(dsid[:3], dsid[3:]),\n        )\n        props.update(\n            type='ria',\n            giturl=str(source_ri),\n            version=version,\n            default_destpath=dsid,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        # TODO are there any advantages of going through a URL-reconversion\n        # by the RI instance, instead of using the original input right away?\n        # it feels like we could only accumulate bugs. Keeping how it was,\n        # for now...\n        props['type'] = 'giturl'\n        props['giturl'] = str(source_ri)\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props\n",
            "header": {
                "file": {
                    "path": "datalad/core/distributed/clone.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "84df56880",
                "new_commit_hash": "2ceeb6bd8",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/core/distributed/clone.py"
            },
            "plus_file": {
                "path": "b/datalad/core/distributed/clone.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 717,
                            "length": 16
                        },
                        "new_range": {
                            "start": 717,
                            "length": 23
                        },
                        "section": "def decode_source_spec(spec, cfg=None):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'",
                                "type": "context"
                            },
                            {
                                "content": "        if not re.match(uuid_regex, dsid):",
                                "type": "context"
                            },
                            {
                                "content": "            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))",
                                "type": "context"
                            },
                            {
                                "content": "        # now we cancel the fragment in the original URL, but keep everthing else",
                                "type": "added"
                            },
                            {
                                "content": "        # in order to be able to support the various combinations of ports, paths,",
                                "type": "added"
                            },
                            {
                                "content": "        # and everything else",
                                "type": "added"
                            },
                            {
                                "content": "        source_ri.fragment = ''",
                                "type": "added"
                            },
                            {
                                "content": "        # strip the custom protocol and go with standard one",
                                "type": "added"
                            },
                            {
                                "content": "        source_ri.scheme = source_ri.scheme[4:]",
                                "type": "added"
                            },
                            {
                                "content": "        # take any existing path, and add trace to dataset within the store",
                                "type": "added"
                            },
                            {
                                "content": "        source_ri.path = '{urlpath}{urldelim}{basepath}{basedelim}{trace}'.format(",
                                "type": "added"
                            },
                            {
                                "content": "            urlpath=source_ri.path if source_ri.path else '',",
                                "type": "added"
                            },
                            {
                                "content": "            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',",
                                "type": "added"
                            },
                            {
                                "content": "            basepath=basepath,",
                                "type": "added"
                            },
                            {
                                "content": "            basedelim='' if not basepath or basepath.endswith('/') else '/',",
                                "type": "added"
                            },
                            {
                                "content": "            trace='{}/{}'.format(dsid[:3], dsid[3:]),",
                                "type": "added"
                            },
                            {
                                "content": "        )",
                                "type": "added"
                            },
                            {
                                "content": "        props.update(",
                                "type": "context"
                            },
                            {
                                "content": "            type='ria',",
                                "type": "context"
                            },
                            {
                                "content": "            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(",
                                "type": "deleted"
                            },
                            {
                                "content": "                proto=source_ri.scheme[4:],",
                                "type": "deleted"
                            },
                            {
                                "content": "                host=hostname,",
                                "type": "deleted"
                            },
                            {
                                "content": "                delim=':' if source_ri.scheme == 'ria+ssh' else '/',",
                                "type": "deleted"
                            },
                            {
                                "content": "                basepath=basepath,",
                                "type": "deleted"
                            },
                            {
                                "content": "                pathdelim='/' if basepath else '',",
                                "type": "deleted"
                            },
                            {
                                "content": "                id1=dsid[:3],",
                                "type": "deleted"
                            },
                            {
                                "content": "                id2=dsid[3:]),",
                                "type": "deleted"
                            },
                            {
                                "content": "            giturl=str(source_ri),",
                                "type": "added"
                            },
                            {
                                "content": "            version=version,",
                                "type": "context"
                            },
                            {
                                "content": "            default_destpath=dsid,",
                                "type": "context"
                            },
                            {
                                "content": "        )",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/core/distributed/clone.py",
                            "type": "function",
                            "name": "decode_source_spec",
                            "content": "def decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        # check for store related configuration for SSH access\n        if source_ri.scheme == 'ria+ssh':\n            hostname = cfg.get(\n                'annex.ria-remote.{}.ssh-host'.format(source_ri.hostname),\n                source_ri.hostname)\n            basepath = cfg.get(\n                'annex.ria-remote.{}.base-path'.format(source_ri.hostname),\n                '')\n        else:\n            hostname = source_ri.hostname\n            basepath = ''\n\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if not re.match(uuid_regex, dsid):\n            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))\n        props.update(\n            type='ria',\n            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(\n                proto=source_ri.scheme[4:],\n                host=hostname,\n                delim=':' if source_ri.scheme == 'ria+ssh' else '/',\n                basepath=basepath,\n                pathdelim='/' if basepath else '',\n                id1=dsid[:3],\n                id2=dsid[3:]),\n            version=version,\n            default_destpath=dsid,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        # TODO are there any advantages of going through a URL-reconversion\n        # by the RI instance, instead of using the original input right away?\n        # it feels like we could only accumulate bugs. Keeping how it was,\n        # for now...\n        props['type'] = 'giturl'\n        props['giturl'] = str(source_ri)\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 662,
                            "end_lineno": 748,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test clone action\n\n\"\"\"\n\n\nfrom datalad.tests.utils import (\n    get_datasets_topdir,\n    integration,\n    slow\n)\n\nimport logging\nimport os\nimport os.path as op\n\nfrom unittest.mock import patch\n\nfrom datalad import consts\nfrom datalad.api import (\n    create,\n    clone,\n    remove,\n)\nfrom datalad.utils import (\n    chpwd,\n    Path,\n    on_windows,\n)\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.cmd import Runner\nfrom datalad.tests.utils import (\n    create_tree,\n    with_tempfile,\n    assert_in,\n    with_tree,\n    with_testrepos,\n    eq_,\n    ok_,\n    assert_false,\n    ok_file_has_content,\n    assert_not_in,\n    assert_raises,\n    assert_status,\n    assert_message,\n    assert_result_count,\n    assert_result_values_equal,\n    ok_startswith,\n    assert_repo_status,\n    serve_path_via_http,\n    swallow_logs,\n    use_cassette,\n    skip_if_no_network,\n    skip_if,\n    with_sameas_remote,\n    known_failure,\n    known_failure_appveyor,\n)\nfrom datalad.core.distributed.clone import (\n    decode_source_spec,\n    _get_installationpath_from_url,\n)\nfrom datalad.distribution.dataset import Dataset\n\n# this is the dataset ID of our test dataset in the main datalad RIA store\ndatalad_store_testds_id = '76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561'\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_args(path, otherpath, alienpath):\n    # source == path\n    assert_raises(ValueError, clone, 'Zoidberg', path='Zoidberg')\n    assert_raises(ValueError, clone, 'ssh://mars/Zoidberg', path='ssh://mars/Zoidberg')\n\n    # \"invalid URL\" is a valid filepath... and since no clone to remote\n    # is possible - we can just assume that it is the (legit) file path\n    # which is provided, not a URL.  So both below should fail as any\n    # other clone from a non-existing source and not for the reason of\n    # \"invalid something\".  Behavior is similar to how Git performs - can\n    # clone into a URL-like path.\n\n    # install to an \"invalid URL\" path\n    res = clone('Zoidberg', path='ssh://mars:Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # install to a \"remote location\" path\n    res = clone('Zoidberg', path='ssh://mars/Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # make fake dataset\n    ds = create(path)\n    assert_raises(IncompleteResultsError, ds.clone, '/higherup.', 'Zoidberg')\n    # make real dataset, try to install outside\n    ds_target = create(Path(otherpath) / 'target')\n    assert_raises(ValueError, ds_target.clone, ds.path, path=ds.path)\n    assert_status('error', ds_target.clone(ds.path, path=alienpath, on_failure='ignore'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_crcns(tdir, ds_path):\n    with chpwd(tdir):\n        res = clone('///', path=\"all-nonrecursive\", on_failure='ignore',\n                    result_xfm=None, return_type='list')\n        assert_status('ok', res)\n\n    # again, but into existing dataset:\n    ds = create(ds_path)\n    crcns = ds.clone(\"///crcns\", result_xfm='datasets', return_type='item-or-list')\n    ok_(crcns.is_installed())\n    eq_(crcns.pathobj, ds.pathobj / \"crcns\")\n    assert_in(crcns.path, ds.subdatasets(result_xfm='paths'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tree(tree={'sub': {}})\ndef test_clone_datasets_root(tdir):\n    tdir = Path(tdir)\n    with chpwd(tdir):\n        ds = clone(\"///\")\n        ok_(ds.is_installed())\n        eq_(ds.pathobj, tdir / get_datasets_topdir())\n\n        # do it a second time:\n        res = clone(\"///\", on_failure='ignore', result_xfm=None, return_type='list')\n        assert_message(\n            \"dataset %s was already cloned from '%s'\",\n            res)\n        assert_status('notneeded', res)\n\n        # and a third time into an existing something, that is not a dataset:\n        (tdir / 'sub' / 'a_file.txt').write_text(\"something\")\n\n        res = clone('///', path=\"sub\", on_failure='ignore')\n        assert_message(\n            'target path already exists and not empty, refuse to clone into target path',\n            res)\n        assert_status('error', res)\n\n\n@with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n@with_tempfile(mkdir=True)\ndef test_clone_simple_local(src, path):\n    origin = Dataset(path)\n\n    # now install it somewhere else\n    ds = clone(src, path, description='mydummy',\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    ok_(ds.is_installed())\n    if not isinstance(origin.repo, AnnexRepo):\n        # this means it is a GitRepo\n        ok_(isinstance(origin.repo, GitRepo))\n        # stays plain Git repo\n        ok_(isinstance(ds.repo, GitRepo))\n        ok_(not isinstance(ds.repo, AnnexRepo))\n        ok_(GitRepo.is_valid_repo(ds.path))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt'})\n        assert_repo_status(path, annex=False)\n    else:\n        # must be an annex\n        ok_(isinstance(ds.repo, AnnexRepo))\n        ok_(AnnexRepo.is_valid_repo(ds.path, allow_noninitialized=False))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt', 'test-annex.dat'})\n        assert_repo_status(path, annex=True)\n        # no content was installed:\n        ok_(not ds.repo.file_has_content('test-annex.dat'))\n        uuid_before = ds.repo.uuid\n        eq_(ds.repo.get_description(), 'mydummy')\n\n    # installing it again, shouldn't matter:\n    res = clone(src, path, result_xfm=None, return_type='list')\n    assert_result_values_equal(res, 'source_url', [src])\n    assert_status('notneeded', res)\n    assert_message(\"dataset %s was already cloned from '%s'\", res)\n    ok_(ds.is_installed())\n    if isinstance(origin.repo, AnnexRepo):\n        eq_(uuid_before, ds.repo.uuid)\n\n\n@with_testrepos(flavors=['local-url', 'network', 'local'])\n@with_tempfile\ndef test_clone_dataset_from_just_source(url, path):\n    with chpwd(path, mkdir=True):\n        ds = clone(url, result_xfm='datasets', return_type='item-or-list')\n\n    ok_startswith(ds.path, path)\n    ok_(ds.is_installed())\n    ok_(GitRepo.is_valid_repo(ds.path))\n    assert_repo_status(ds.path, annex=None)\n    assert_in('INFO.txt', ds.repo.get_indexed_files())\n\n\n# test fails randomly, likely a bug in one of the employed test helpers\n# https://github.com/datalad/datalad/pull/3966#issuecomment-571267932\n@known_failure\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_clone_dataladri(src, topurl, path):\n    # make plain git repo\n    ds_path = Path(src) / 'ds'\n    gr = GitRepo(ds_path, create=True)\n    gr.add('test.txt')\n    gr.commit('demo')\n    Runner(cwd=gr.path)(['git', 'update-server-info'])\n    # now install it somewhere else\n    with patch('datalad.consts.DATASETS_TOPURL', topurl):\n        ds = clone('///ds', path, result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    assert_repo_status(path, annex=False)\n    ok_file_has_content(ds.pathobj / 'test.txt', 'some')\n\n\n@with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_isnot_recursive(src, path_nr, path_r):\n    ds = clone(src, path_nr, result_xfm='datasets', return_type='item-or-list')\n    ok_(ds.is_installed())\n    # check nothin is unintentionally installed\n    subdss = ds.subdatasets(recursive=True)\n    assert_result_count(subdss, len(subdss), state='absent')\n    # this also means, subdatasets to be listed as not fulfilled:\n    eq_(set(ds.subdatasets(recursive=True, fulfilled=False, result_xfm='relpaths')),\n        {'subm 1', '2'})\n\n\n@slow  # 23.1478s\n@with_testrepos(flavors=['local'])\n# 'local-url', 'network'\n# TODO: Somehow annex gets confused while initializing installed ds, whose\n# .git/config show a submodule url \"file:///aaa/bbb%20b/...\"\n# this is delivered by with_testrepos as the url to clone\n@with_tempfile\ndef test_clone_into_dataset(source, top_path):\n\n    ds = create(top_path)\n    assert_repo_status(ds.path)\n\n    subds = ds.clone(source, \"sub\",\n                     result_xfm='datasets', return_type='item-or-list')\n    ok_((subds.pathobj / '.git').is_dir())\n    ok_(subds.is_installed())\n    assert_in('sub', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # sub is clean:\n    assert_repo_status(subds.path, annex=None)\n    # top is clean:\n    assert_repo_status(ds.path, annex=None)\n\n    # but we could also save while installing and there should be no side-effect\n    # of saving any other changes if we state to not auto-save changes\n    # Create a dummy change\n    create_tree(ds.path, {'dummy.txt': 'buga'})\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n    subds_ = ds.clone(source, \"sub2\",\n                      result_xfm='datasets', return_type='item-or-list')\n    eq_(subds_.pathobj, ds.pathobj / \"sub2\")  # for paranoid yoh ;)\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n\n\n@with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n@with_tempfile(mkdir=True)\ndef test_notclone_known_subdataset(src, path):\n    # get the superdataset:\n    ds = clone(src, path,\n               result_xfm='datasets', return_type='item-or-list')\n\n    # subdataset not installed:\n    subds = Dataset(ds.pathobj / 'subm 1')\n    assert_false(subds.is_installed())\n    assert_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # clone is not meaningful\n    res = ds.clone('subm 1', on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n    # get does the job\n    res = ds.get(path='subm 1', get_data=False)\n    assert_status('ok', res)\n    ok_(subds.is_installed())\n    ok_(AnnexRepo.is_valid_repo(subds.path, allow_noninitialized=False))\n    # Verify that it is the correct submodule installed and not\n    # new repository initiated\n    eq_(set(subds.repo.get_indexed_files()),\n        {'test.dat', 'INFO.txt', 'test-annex.dat'})\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n\n\n@with_tempfile(mkdir=True)\ndef test_failed_clone(dspath):\n    ds = create(dspath)\n    res = ds.clone(\"http://nonexistingreallyanything.datalad.org/bla\", \"sub\",\n                   on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n\n\n@with_testrepos('submodule_annex', flavors=['local'])\n@with_tempfile(mkdir=True)\ndef test_reckless(path, top_path):\n    ds = clone(path, top_path, reckless=True,\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.config.get('annex.hardlink', None), 'true')\n    eq_(ds.repo.repo_info()['untrusted repositories'][0]['here'], True)\n\n\n@with_tempfile\n@with_tempfile\ndef test_install_source_relpath(src, dest):\n    src = Path(src)\n    create(src)\n    src_ = src.name\n    with chpwd(src.parent):\n        clone(src_, dest)\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_isnt_a_smartass(origin_path, path):\n    origin = create(origin_path)\n    cloned = clone(origin, path,\n                   result_xfm='datasets', return_type='item-or-list')\n    with chpwd(path):\n        # no were are inside a dataset clone, and we make another one\n        # we do not want automatic subdatasetification without given a dataset\n        # explicitely\n        clonedsub = clone(origin, 'testsub',\n                          result_xfm='datasets', return_type='item-or-list')\n    # correct destination\n    assert clonedsub.path.startswith(path)\n    # no subdataset relation\n    eq_(cloned.subdatasets(), [])\n\n\n@skip_if(on_windows or not os.geteuid(), \"Will fail under super-user\")\n@with_tempfile(mkdir=True)\ndef test_clone_report_permission_issue(tdir):\n    pdir = Path(tdir) / 'protected'\n    pdir.mkdir()\n    # make it read-only\n    pdir.chmod(0o555)\n    with chpwd(pdir):\n        res = clone('///', result_xfm=None, return_type='list', on_failure='ignore')\n        assert_status('error', res)\n        assert_result_count(\n            res, 1, status='error',\n            message=\"could not create work tree dir '%s/%s': Permission denied\"\n                    % (pdir, get_datasets_topdir())\n        )\n\n\n# Started to hang on appveyor.\n@known_failure_appveyor  #FIXME - hangs\n@skip_if_no_network\n@with_tempfile\ndef test_autoenabled_remote_msg(path):\n    # Verify that no message about a remote not been enabled is displayed\n    # whenever the remote we clone is the  type=git special remote, so the name\n    # of the remote might not match\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone('///repronim/containers', path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_not_in(\"not auto-enabled\", cml.out)\n\n\n@with_sameas_remote(autoenabled=True)\n@with_tempfile(mkdir=True)\ndef test_clone_autoenable_msg_handles_sameas(repo, clone_path):\n    ds = Dataset(repo.path)\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone(ds, clone_path)\n        assert_status('ok', res)\n        assert_in(\"r_dir\", cml.out)\n        assert_in(\"not auto-enabled\", cml.out)\n        # The rsyncurl remote was enabled.\n        assert_not_in(\"r_rsync\", cml.out)\n    ds_cloned = Dataset(clone_path)\n    remotes = ds_cloned.repo.get_remotes()\n    assert_in(\"r_rsync\", remotes)\n    assert_not_in(\"r_dir\", remotes)\n\n\ndef test_installationpath_from_url():\n    cases = (\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n    ) + (\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ) if on_windows else (\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    )\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')\n\n\n# https://github.com/datalad/datalad/issues/3958\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_expanduser(srcpath, destpath):\n    src = Dataset(Path(srcpath) / 'src').create()\n    dest = Dataset(Path(destpath) / 'dest').create()\n\n    with chpwd(destpath), patch.dict('os.environ', {'HOME': srcpath}):\n        res = clone(op.join('~', 'src'), 'dest', result_xfm=None, return_type='list',\n                    on_failure='ignore')\n        assert_result_count(res, 1)\n        assert_result_count(\n            res, 1, action='install', status='error', path=dest.path,\n            message='target path already exists and not empty, refuse to '\n            'clone into target path')\n        # wipe out destination, and try again\n        assert_status('ok', remove(dataset=dest, check=False))\n        # now it should do it, and clone the right one\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n        eq_(src.id, cloneds.id)\n        # and it shouldn't fail when doing it again, because it detects\n        # the re-clone\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n\n\n@with_tempfile(mkdir=True)\ndef test_cfg_originorigin(path):\n    path = Path(path)\n    origin = Dataset(path / 'origin').create()\n    (origin.pathobj / 'file1.txt').write_text('content')\n    origin.save()\n    clone_direct = clone(origin, path / 'clone_direct')\n    clone_clone = clone(clone_direct, path / 'clone_clone')\n    # the goal is to be able to get file content from origin without\n    # the need to configure it manually\n    assert_result_count(\n        clone_clone.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_clone.pathobj / 'file1.txt'),\n    )\n    eq_((clone_clone.pathobj / 'file1.txt').read_text(), 'content')\n    eq_(\n        Path(clone_clone.siblings(\n            'query',\n            name='origin-2',\n            return_type='item-or-list')['url']),\n        origin.pathobj\n    )\n\n\n# test fix for gh-2601/gh-3538\n@with_tempfile()\ndef test_relative_submodule_url(path):\n    Dataset(op.join(path, 'origin')).create()\n    ds = Dataset(op.join(path, 'ds')).create()\n    with chpwd(ds.path):\n        ds.clone(\n            source=op.join(op.pardir, 'origin'),\n            path='sources')\n    subinfo = ds.subdatasets(return_type='item-or-list')\n    eq_(subinfo['gitmodule_url'],\n        # must be a relative URL, not platform-specific relpath!\n        '{}/{}'.format(op.pardir, 'origin'))\n\n\ndef test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}{}{}/{}'.format(\n                    proto,\n                    loc,\n                    ':' if proto == 'ssh' else '/',\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')\n\n\n@skip_if_no_network\n@with_tempfile()\ndef test_ria_http(path):\n    # can we clone from the store w/o any dedicated config\n    ds = clone('ria+http://store.datalad.org#{}'.format(datalad_store_testds_id), path)\n    ok_(ds.is_installed())\n    eq_(ds.id, datalad_store_testds_id)\n",
            "new_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test clone action\n\n\"\"\"\n\n\nfrom datalad.tests.utils import (\n    get_datasets_topdir,\n    integration,\n    slow\n)\n\nimport logging\nimport os\nimport os.path as op\n\nfrom unittest.mock import patch\n\nfrom datalad import consts\nfrom datalad.api import (\n    create,\n    clone,\n    remove,\n)\nfrom datalad.utils import (\n    chpwd,\n    Path,\n    on_windows,\n)\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.cmd import Runner\nfrom datalad.tests.utils import (\n    create_tree,\n    with_tempfile,\n    assert_in,\n    with_tree,\n    with_testrepos,\n    eq_,\n    ok_,\n    assert_false,\n    ok_file_has_content,\n    assert_not_in,\n    assert_raises,\n    assert_status,\n    assert_message,\n    assert_result_count,\n    assert_result_values_equal,\n    ok_startswith,\n    assert_repo_status,\n    serve_path_via_http,\n    swallow_logs,\n    use_cassette,\n    skip_if_no_network,\n    skip_if,\n    with_sameas_remote,\n    known_failure,\n    known_failure_appveyor,\n)\nfrom datalad.core.distributed.clone import (\n    decode_source_spec,\n    _get_installationpath_from_url,\n)\nfrom datalad.distribution.dataset import Dataset\n\n# this is the dataset ID of our test dataset in the main datalad RIA store\ndatalad_store_testds_id = '76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561'\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_args(path, otherpath, alienpath):\n    # source == path\n    assert_raises(ValueError, clone, 'Zoidberg', path='Zoidberg')\n    assert_raises(ValueError, clone, 'ssh://mars/Zoidberg', path='ssh://mars/Zoidberg')\n\n    # \"invalid URL\" is a valid filepath... and since no clone to remote\n    # is possible - we can just assume that it is the (legit) file path\n    # which is provided, not a URL.  So both below should fail as any\n    # other clone from a non-existing source and not for the reason of\n    # \"invalid something\".  Behavior is similar to how Git performs - can\n    # clone into a URL-like path.\n\n    # install to an \"invalid URL\" path\n    res = clone('Zoidberg', path='ssh://mars:Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # install to a \"remote location\" path\n    res = clone('Zoidberg', path='ssh://mars/Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # make fake dataset\n    ds = create(path)\n    assert_raises(IncompleteResultsError, ds.clone, '/higherup.', 'Zoidberg')\n    # make real dataset, try to install outside\n    ds_target = create(Path(otherpath) / 'target')\n    assert_raises(ValueError, ds_target.clone, ds.path, path=ds.path)\n    assert_status('error', ds_target.clone(ds.path, path=alienpath, on_failure='ignore'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_crcns(tdir, ds_path):\n    with chpwd(tdir):\n        res = clone('///', path=\"all-nonrecursive\", on_failure='ignore',\n                    result_xfm=None, return_type='list')\n        assert_status('ok', res)\n\n    # again, but into existing dataset:\n    ds = create(ds_path)\n    crcns = ds.clone(\"///crcns\", result_xfm='datasets', return_type='item-or-list')\n    ok_(crcns.is_installed())\n    eq_(crcns.pathobj, ds.pathobj / \"crcns\")\n    assert_in(crcns.path, ds.subdatasets(result_xfm='paths'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tree(tree={'sub': {}})\ndef test_clone_datasets_root(tdir):\n    tdir = Path(tdir)\n    with chpwd(tdir):\n        ds = clone(\"///\")\n        ok_(ds.is_installed())\n        eq_(ds.pathobj, tdir / get_datasets_topdir())\n\n        # do it a second time:\n        res = clone(\"///\", on_failure='ignore', result_xfm=None, return_type='list')\n        assert_message(\n            \"dataset %s was already cloned from '%s'\",\n            res)\n        assert_status('notneeded', res)\n\n        # and a third time into an existing something, that is not a dataset:\n        (tdir / 'sub' / 'a_file.txt').write_text(\"something\")\n\n        res = clone('///', path=\"sub\", on_failure='ignore')\n        assert_message(\n            'target path already exists and not empty, refuse to clone into target path',\n            res)\n        assert_status('error', res)\n\n\n@with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n@with_tempfile(mkdir=True)\ndef test_clone_simple_local(src, path):\n    origin = Dataset(path)\n\n    # now install it somewhere else\n    ds = clone(src, path, description='mydummy',\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    ok_(ds.is_installed())\n    if not isinstance(origin.repo, AnnexRepo):\n        # this means it is a GitRepo\n        ok_(isinstance(origin.repo, GitRepo))\n        # stays plain Git repo\n        ok_(isinstance(ds.repo, GitRepo))\n        ok_(not isinstance(ds.repo, AnnexRepo))\n        ok_(GitRepo.is_valid_repo(ds.path))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt'})\n        assert_repo_status(path, annex=False)\n    else:\n        # must be an annex\n        ok_(isinstance(ds.repo, AnnexRepo))\n        ok_(AnnexRepo.is_valid_repo(ds.path, allow_noninitialized=False))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt', 'test-annex.dat'})\n        assert_repo_status(path, annex=True)\n        # no content was installed:\n        ok_(not ds.repo.file_has_content('test-annex.dat'))\n        uuid_before = ds.repo.uuid\n        eq_(ds.repo.get_description(), 'mydummy')\n\n    # installing it again, shouldn't matter:\n    res = clone(src, path, result_xfm=None, return_type='list')\n    assert_result_values_equal(res, 'source_url', [src])\n    assert_status('notneeded', res)\n    assert_message(\"dataset %s was already cloned from '%s'\", res)\n    ok_(ds.is_installed())\n    if isinstance(origin.repo, AnnexRepo):\n        eq_(uuid_before, ds.repo.uuid)\n\n\n@with_testrepos(flavors=['local-url', 'network', 'local'])\n@with_tempfile\ndef test_clone_dataset_from_just_source(url, path):\n    with chpwd(path, mkdir=True):\n        ds = clone(url, result_xfm='datasets', return_type='item-or-list')\n\n    ok_startswith(ds.path, path)\n    ok_(ds.is_installed())\n    ok_(GitRepo.is_valid_repo(ds.path))\n    assert_repo_status(ds.path, annex=None)\n    assert_in('INFO.txt', ds.repo.get_indexed_files())\n\n\n# test fails randomly, likely a bug in one of the employed test helpers\n# https://github.com/datalad/datalad/pull/3966#issuecomment-571267932\n@known_failure\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_clone_dataladri(src, topurl, path):\n    # make plain git repo\n    ds_path = Path(src) / 'ds'\n    gr = GitRepo(ds_path, create=True)\n    gr.add('test.txt')\n    gr.commit('demo')\n    Runner(cwd=gr.path)(['git', 'update-server-info'])\n    # now install it somewhere else\n    with patch('datalad.consts.DATASETS_TOPURL', topurl):\n        ds = clone('///ds', path, result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    assert_repo_status(path, annex=False)\n    ok_file_has_content(ds.pathobj / 'test.txt', 'some')\n\n\n@with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_isnot_recursive(src, path_nr, path_r):\n    ds = clone(src, path_nr, result_xfm='datasets', return_type='item-or-list')\n    ok_(ds.is_installed())\n    # check nothin is unintentionally installed\n    subdss = ds.subdatasets(recursive=True)\n    assert_result_count(subdss, len(subdss), state='absent')\n    # this also means, subdatasets to be listed as not fulfilled:\n    eq_(set(ds.subdatasets(recursive=True, fulfilled=False, result_xfm='relpaths')),\n        {'subm 1', '2'})\n\n\n@slow  # 23.1478s\n@with_testrepos(flavors=['local'])\n# 'local-url', 'network'\n# TODO: Somehow annex gets confused while initializing installed ds, whose\n# .git/config show a submodule url \"file:///aaa/bbb%20b/...\"\n# this is delivered by with_testrepos as the url to clone\n@with_tempfile\ndef test_clone_into_dataset(source, top_path):\n\n    ds = create(top_path)\n    assert_repo_status(ds.path)\n\n    subds = ds.clone(source, \"sub\",\n                     result_xfm='datasets', return_type='item-or-list')\n    ok_((subds.pathobj / '.git').is_dir())\n    ok_(subds.is_installed())\n    assert_in('sub', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # sub is clean:\n    assert_repo_status(subds.path, annex=None)\n    # top is clean:\n    assert_repo_status(ds.path, annex=None)\n\n    # but we could also save while installing and there should be no side-effect\n    # of saving any other changes if we state to not auto-save changes\n    # Create a dummy change\n    create_tree(ds.path, {'dummy.txt': 'buga'})\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n    subds_ = ds.clone(source, \"sub2\",\n                      result_xfm='datasets', return_type='item-or-list')\n    eq_(subds_.pathobj, ds.pathobj / \"sub2\")  # for paranoid yoh ;)\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n\n\n@with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n@with_tempfile(mkdir=True)\ndef test_notclone_known_subdataset(src, path):\n    # get the superdataset:\n    ds = clone(src, path,\n               result_xfm='datasets', return_type='item-or-list')\n\n    # subdataset not installed:\n    subds = Dataset(ds.pathobj / 'subm 1')\n    assert_false(subds.is_installed())\n    assert_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # clone is not meaningful\n    res = ds.clone('subm 1', on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n    # get does the job\n    res = ds.get(path='subm 1', get_data=False)\n    assert_status('ok', res)\n    ok_(subds.is_installed())\n    ok_(AnnexRepo.is_valid_repo(subds.path, allow_noninitialized=False))\n    # Verify that it is the correct submodule installed and not\n    # new repository initiated\n    eq_(set(subds.repo.get_indexed_files()),\n        {'test.dat', 'INFO.txt', 'test-annex.dat'})\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n\n\n@with_tempfile(mkdir=True)\ndef test_failed_clone(dspath):\n    ds = create(dspath)\n    res = ds.clone(\"http://nonexistingreallyanything.datalad.org/bla\", \"sub\",\n                   on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n\n\n@with_testrepos('submodule_annex', flavors=['local'])\n@with_tempfile(mkdir=True)\ndef test_reckless(path, top_path):\n    ds = clone(path, top_path, reckless=True,\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.config.get('annex.hardlink', None), 'true')\n    eq_(ds.repo.repo_info()['untrusted repositories'][0]['here'], True)\n\n\n@with_tempfile\n@with_tempfile\ndef test_install_source_relpath(src, dest):\n    src = Path(src)\n    create(src)\n    src_ = src.name\n    with chpwd(src.parent):\n        clone(src_, dest)\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_isnt_a_smartass(origin_path, path):\n    origin = create(origin_path)\n    cloned = clone(origin, path,\n                   result_xfm='datasets', return_type='item-or-list')\n    with chpwd(path):\n        # no were are inside a dataset clone, and we make another one\n        # we do not want automatic subdatasetification without given a dataset\n        # explicitely\n        clonedsub = clone(origin, 'testsub',\n                          result_xfm='datasets', return_type='item-or-list')\n    # correct destination\n    assert clonedsub.path.startswith(path)\n    # no subdataset relation\n    eq_(cloned.subdatasets(), [])\n\n\n@skip_if(on_windows or not os.geteuid(), \"Will fail under super-user\")\n@with_tempfile(mkdir=True)\ndef test_clone_report_permission_issue(tdir):\n    pdir = Path(tdir) / 'protected'\n    pdir.mkdir()\n    # make it read-only\n    pdir.chmod(0o555)\n    with chpwd(pdir):\n        res = clone('///', result_xfm=None, return_type='list', on_failure='ignore')\n        assert_status('error', res)\n        assert_result_count(\n            res, 1, status='error',\n            message=\"could not create work tree dir '%s/%s': Permission denied\"\n                    % (pdir, get_datasets_topdir())\n        )\n\n\n# Started to hang on appveyor.\n@known_failure_appveyor  #FIXME - hangs\n@skip_if_no_network\n@with_tempfile\ndef test_autoenabled_remote_msg(path):\n    # Verify that no message about a remote not been enabled is displayed\n    # whenever the remote we clone is the  type=git special remote, so the name\n    # of the remote might not match\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone('///repronim/containers', path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_not_in(\"not auto-enabled\", cml.out)\n\n\n@with_sameas_remote(autoenabled=True)\n@with_tempfile(mkdir=True)\ndef test_clone_autoenable_msg_handles_sameas(repo, clone_path):\n    ds = Dataset(repo.path)\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone(ds, clone_path)\n        assert_status('ok', res)\n        assert_in(\"r_dir\", cml.out)\n        assert_in(\"not auto-enabled\", cml.out)\n        # The rsyncurl remote was enabled.\n        assert_not_in(\"r_rsync\", cml.out)\n    ds_cloned = Dataset(clone_path)\n    remotes = ds_cloned.repo.get_remotes()\n    assert_in(\"r_rsync\", remotes)\n    assert_not_in(\"r_dir\", remotes)\n\n\ndef test_installationpath_from_url():\n    cases = (\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n    ) + (\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ) if on_windows else (\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    )\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')\n\n\n# https://github.com/datalad/datalad/issues/3958\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_expanduser(srcpath, destpath):\n    src = Dataset(Path(srcpath) / 'src').create()\n    dest = Dataset(Path(destpath) / 'dest').create()\n\n    with chpwd(destpath), patch.dict('os.environ', {'HOME': srcpath}):\n        res = clone(op.join('~', 'src'), 'dest', result_xfm=None, return_type='list',\n                    on_failure='ignore')\n        assert_result_count(res, 1)\n        assert_result_count(\n            res, 1, action='install', status='error', path=dest.path,\n            message='target path already exists and not empty, refuse to '\n            'clone into target path')\n        # wipe out destination, and try again\n        assert_status('ok', remove(dataset=dest, check=False))\n        # now it should do it, and clone the right one\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n        eq_(src.id, cloneds.id)\n        # and it shouldn't fail when doing it again, because it detects\n        # the re-clone\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n\n\n@with_tempfile(mkdir=True)\ndef test_cfg_originorigin(path):\n    path = Path(path)\n    origin = Dataset(path / 'origin').create()\n    (origin.pathobj / 'file1.txt').write_text('content')\n    origin.save()\n    clone_direct = clone(origin, path / 'clone_direct')\n    clone_clone = clone(clone_direct, path / 'clone_clone')\n    # the goal is to be able to get file content from origin without\n    # the need to configure it manually\n    assert_result_count(\n        clone_clone.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_clone.pathobj / 'file1.txt'),\n    )\n    eq_((clone_clone.pathobj / 'file1.txt').read_text(), 'content')\n    eq_(\n        Path(clone_clone.siblings(\n            'query',\n            name='origin-2',\n            return_type='item-or-list')['url']),\n        origin.pathobj\n    )\n\n\n# test fix for gh-2601/gh-3538\n@with_tempfile()\ndef test_relative_submodule_url(path):\n    Dataset(op.join(path, 'origin')).create()\n    ds = Dataset(op.join(path, 'ds')).create()\n    with chpwd(ds.path):\n        ds.clone(\n            source=op.join(op.pardir, 'origin'),\n            path='sources')\n    subinfo = ds.subdatasets(return_type='item-or-list')\n    eq_(subinfo['gitmodule_url'],\n        # must be a relative URL, not platform-specific relpath!\n        '{}/{}'.format(op.pardir, 'origin'))\n\n\ndef test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}/{}/{}'.format(\n                    proto,\n                    loc,\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')\n\n\n@skip_if_no_network\n@with_tempfile()\ndef test_ria_http(path):\n    # can we clone from the store w/o any dedicated config\n    ds = clone('ria+http://store.datalad.org#{}'.format(datalad_store_testds_id), path)\n    ok_(ds.is_installed())\n    eq_(ds.id, datalad_store_testds_id)\n",
            "header": {
                "file": {
                    "path": "datalad/core/distributed/tests/test_clone.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "7dc3ec4bb",
                "new_commit_hash": "13e7e9dcd",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/core/distributed/tests/test_clone.py"
            },
            "plus_file": {
                "path": "b/datalad/core/distributed/tests/test_clone.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 546,
                            "length": 10
                        },
                        "new_range": {
                            "start": 546,
                            "length": 9
                        },
                        "section": "def test_decode_source_spec():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        eq_(decode_source_spec(spec),",
                                "type": "context"
                            },
                            {
                                "content": "            dict(",
                                "type": "context"
                            },
                            {
                                "content": "                source=spec,",
                                "type": "context"
                            },
                            {
                                "content": "                giturl='{}://{}{}{}/{}'.format(",
                                "type": "deleted"
                            },
                            {
                                "content": "                giturl='{}://{}/{}/{}'.format(",
                                "type": "added"
                            },
                            {
                                "content": "                    proto,",
                                "type": "context"
                            },
                            {
                                "content": "                    loc,",
                                "type": "context"
                            },
                            {
                                "content": "                    ':' if proto == 'ssh' else '/',",
                                "type": "deleted"
                            },
                            {
                                "content": "                    dsid[:3],",
                                "type": "context"
                            },
                            {
                                "content": "                    dsid[3:]),",
                                "type": "context"
                            },
                            {
                                "content": "                version=version,",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/core/distributed/tests/test_clone.py",
                            "type": "function",
                            "name": "test_decode_source_spec",
                            "content": "def test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}{}{}/{}'.format(\n                    proto,\n                    loc,\n                    ':' if proto == 'ssh' else '/',\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 511,
                            "end_lineno": 560,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "fbb05051f6c01ff8c3257ce4458cfe47644f8ddb^",
    "new_commit_hash": "fbb05051f6c01ff8c3257ce4458cfe47644f8ddb",
    "commit_message": "BF: Robustify ria URL transformation\n\nShould now be able to deal with ports and paths.\n\nThis also fixes a bug where we generated SSH URLs with ':' host-path\ndelimiters, which should not be there.",
    "commit_date": "2020-01-15T10:34:57+01:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "fbb05051f6c01ff8c3257ce4458cfe47644f8ddb",
    "test_file_codes": [
        "# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Test clone action\n\n\"\"\"\n\n\nfrom datalad.tests.utils import (\n    get_datasets_topdir,\n    integration,\n    slow\n)\n\nimport logging\nimport os\nimport os.path as op\n\nfrom unittest.mock import patch\n\nfrom datalad import consts\nfrom datalad.api import (\n    create,\n    clone,\n    remove,\n)\nfrom datalad.utils import (\n    chpwd,\n    Path,\n    on_windows,\n)\nfrom datalad.support.exceptions import IncompleteResultsError\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.cmd import Runner\nfrom datalad.tests.utils import (\n    create_tree,\n    with_tempfile,\n    assert_in,\n    with_tree,\n    with_testrepos,\n    eq_,\n    ok_,\n    assert_false,\n    ok_file_has_content,\n    assert_not_in,\n    assert_raises,\n    assert_status,\n    assert_message,\n    assert_result_count,\n    assert_result_values_equal,\n    ok_startswith,\n    assert_repo_status,\n    serve_path_via_http,\n    swallow_logs,\n    use_cassette,\n    skip_if_no_network,\n    skip_if,\n    with_sameas_remote,\n    known_failure,\n    known_failure_appveyor,\n)\nfrom datalad.core.distributed.clone import (\n    decode_source_spec,\n    _get_installationpath_from_url,\n)\nfrom datalad.distribution.dataset import Dataset\n\n# this is the dataset ID of our test dataset in the main datalad RIA store\ndatalad_store_testds_id = '76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561'\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_invalid_args(path, otherpath, alienpath):\n    # source == path\n    assert_raises(ValueError, clone, 'Zoidberg', path='Zoidberg')\n    assert_raises(ValueError, clone, 'ssh://mars/Zoidberg', path='ssh://mars/Zoidberg')\n\n    # \"invalid URL\" is a valid filepath... and since no clone to remote\n    # is possible - we can just assume that it is the (legit) file path\n    # which is provided, not a URL.  So both below should fail as any\n    # other clone from a non-existing source and not for the reason of\n    # \"invalid something\".  Behavior is similar to how Git performs - can\n    # clone into a URL-like path.\n\n    # install to an \"invalid URL\" path\n    res = clone('Zoidberg', path='ssh://mars:Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # install to a \"remote location\" path\n    res = clone('Zoidberg', path='ssh://mars/Zoidberg', on_failure='ignore')\n    assert_status('error', res)\n\n    # make fake dataset\n    ds = create(path)\n    assert_raises(IncompleteResultsError, ds.clone, '/higherup.', 'Zoidberg')\n    # make real dataset, try to install outside\n    ds_target = create(Path(otherpath) / 'target')\n    assert_raises(ValueError, ds_target.clone, ds.path, path=ds.path)\n    assert_status('error', ds_target.clone(ds.path, path=alienpath, on_failure='ignore'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_crcns(tdir, ds_path):\n    with chpwd(tdir):\n        res = clone('///', path=\"all-nonrecursive\", on_failure='ignore',\n                    result_xfm=None, return_type='list')\n        assert_status('ok', res)\n\n    # again, but into existing dataset:\n    ds = create(ds_path)\n    crcns = ds.clone(\"///crcns\", result_xfm='datasets', return_type='item-or-list')\n    ok_(crcns.is_installed())\n    eq_(crcns.pathobj, ds.pathobj / \"crcns\")\n    assert_in(crcns.path, ds.subdatasets(result_xfm='paths'))\n\n\n@integration\n@skip_if_no_network\n@use_cassette('test_install_crcns')\n@with_tree(tree={'sub': {}})\ndef test_clone_datasets_root(tdir):\n    tdir = Path(tdir)\n    with chpwd(tdir):\n        ds = clone(\"///\")\n        ok_(ds.is_installed())\n        eq_(ds.pathobj, tdir / get_datasets_topdir())\n\n        # do it a second time:\n        res = clone(\"///\", on_failure='ignore', result_xfm=None, return_type='list')\n        assert_message(\n            \"dataset %s was already cloned from '%s'\",\n            res)\n        assert_status('notneeded', res)\n\n        # and a third time into an existing something, that is not a dataset:\n        (tdir / 'sub' / 'a_file.txt').write_text(\"something\")\n\n        res = clone('///', path=\"sub\", on_failure='ignore')\n        assert_message(\n            'target path already exists and not empty, refuse to clone into target path',\n            res)\n        assert_status('error', res)\n\n\n@with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n@with_tempfile(mkdir=True)\ndef test_clone_simple_local(src, path):\n    origin = Dataset(path)\n\n    # now install it somewhere else\n    ds = clone(src, path, description='mydummy',\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    ok_(ds.is_installed())\n    if not isinstance(origin.repo, AnnexRepo):\n        # this means it is a GitRepo\n        ok_(isinstance(origin.repo, GitRepo))\n        # stays plain Git repo\n        ok_(isinstance(ds.repo, GitRepo))\n        ok_(not isinstance(ds.repo, AnnexRepo))\n        ok_(GitRepo.is_valid_repo(ds.path))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt'})\n        assert_repo_status(path, annex=False)\n    else:\n        # must be an annex\n        ok_(isinstance(ds.repo, AnnexRepo))\n        ok_(AnnexRepo.is_valid_repo(ds.path, allow_noninitialized=False))\n        eq_(set(ds.repo.get_indexed_files()),\n            {'test.dat', 'INFO.txt', 'test-annex.dat'})\n        assert_repo_status(path, annex=True)\n        # no content was installed:\n        ok_(not ds.repo.file_has_content('test-annex.dat'))\n        uuid_before = ds.repo.uuid\n        eq_(ds.repo.get_description(), 'mydummy')\n\n    # installing it again, shouldn't matter:\n    res = clone(src, path, result_xfm=None, return_type='list')\n    assert_result_values_equal(res, 'source_url', [src])\n    assert_status('notneeded', res)\n    assert_message(\"dataset %s was already cloned from '%s'\", res)\n    ok_(ds.is_installed())\n    if isinstance(origin.repo, AnnexRepo):\n        eq_(uuid_before, ds.repo.uuid)\n\n\n@with_testrepos(flavors=['local-url', 'network', 'local'])\n@with_tempfile\ndef test_clone_dataset_from_just_source(url, path):\n    with chpwd(path, mkdir=True):\n        ds = clone(url, result_xfm='datasets', return_type='item-or-list')\n\n    ok_startswith(ds.path, path)\n    ok_(ds.is_installed())\n    ok_(GitRepo.is_valid_repo(ds.path))\n    assert_repo_status(ds.path, annex=None)\n    assert_in('INFO.txt', ds.repo.get_indexed_files())\n\n\n# test fails randomly, likely a bug in one of the employed test helpers\n# https://github.com/datalad/datalad/pull/3966#issuecomment-571267932\n@known_failure\n@with_tree(tree={\n    'ds': {'test.txt': 'some'},\n    })\n@serve_path_via_http\n@with_tempfile(mkdir=True)\ndef test_clone_dataladri(src, topurl, path):\n    # make plain git repo\n    ds_path = Path(src) / 'ds'\n    gr = GitRepo(ds_path, create=True)\n    gr.add('test.txt')\n    gr.commit('demo')\n    Runner(cwd=gr.path)(['git', 'update-server-info'])\n    # now install it somewhere else\n    with patch('datalad.consts.DATASETS_TOPURL', topurl):\n        ds = clone('///ds', path, result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.path, path)\n    assert_repo_status(path, annex=False)\n    ok_file_has_content(ds.pathobj / 'test.txt', 'some')\n\n\n@with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_clone_isnot_recursive(src, path_nr, path_r):\n    ds = clone(src, path_nr, result_xfm='datasets', return_type='item-or-list')\n    ok_(ds.is_installed())\n    # check nothin is unintentionally installed\n    subdss = ds.subdatasets(recursive=True)\n    assert_result_count(subdss, len(subdss), state='absent')\n    # this also means, subdatasets to be listed as not fulfilled:\n    eq_(set(ds.subdatasets(recursive=True, fulfilled=False, result_xfm='relpaths')),\n        {'subm 1', '2'})\n\n\n@slow  # 23.1478s\n@with_testrepos(flavors=['local'])\n# 'local-url', 'network'\n# TODO: Somehow annex gets confused while initializing installed ds, whose\n# .git/config show a submodule url \"file:///aaa/bbb%20b/...\"\n# this is delivered by with_testrepos as the url to clone\n@with_tempfile\ndef test_clone_into_dataset(source, top_path):\n\n    ds = create(top_path)\n    assert_repo_status(ds.path)\n\n    subds = ds.clone(source, \"sub\",\n                     result_xfm='datasets', return_type='item-or-list')\n    ok_((subds.pathobj / '.git').is_dir())\n    ok_(subds.is_installed())\n    assert_in('sub', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # sub is clean:\n    assert_repo_status(subds.path, annex=None)\n    # top is clean:\n    assert_repo_status(ds.path, annex=None)\n\n    # but we could also save while installing and there should be no side-effect\n    # of saving any other changes if we state to not auto-save changes\n    # Create a dummy change\n    create_tree(ds.path, {'dummy.txt': 'buga'})\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n    subds_ = ds.clone(source, \"sub2\",\n                      result_xfm='datasets', return_type='item-or-list')\n    eq_(subds_.pathobj, ds.pathobj / \"sub2\")  # for paranoid yoh ;)\n    assert_repo_status(ds.path, untracked=['dummy.txt'])\n\n\n@with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n@with_tempfile(mkdir=True)\ndef test_notclone_known_subdataset(src, path):\n    # get the superdataset:\n    ds = clone(src, path,\n               result_xfm='datasets', return_type='item-or-list')\n\n    # subdataset not installed:\n    subds = Dataset(ds.pathobj / 'subm 1')\n    assert_false(subds.is_installed())\n    assert_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n    # clone is not meaningful\n    res = ds.clone('subm 1', on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n    # get does the job\n    res = ds.get(path='subm 1', get_data=False)\n    assert_status('ok', res)\n    ok_(subds.is_installed())\n    ok_(AnnexRepo.is_valid_repo(subds.path, allow_noninitialized=False))\n    # Verify that it is the correct submodule installed and not\n    # new repository initiated\n    eq_(set(subds.repo.get_indexed_files()),\n        {'test.dat', 'INFO.txt', 'test-annex.dat'})\n    assert_not_in('subm 1', ds.subdatasets(fulfilled=False, result_xfm='relpaths'))\n    assert_in('subm 1', ds.subdatasets(fulfilled=True, result_xfm='relpaths'))\n\n\n@with_tempfile(mkdir=True)\ndef test_failed_clone(dspath):\n    ds = create(dspath)\n    res = ds.clone(\"http://nonexistingreallyanything.datalad.org/bla\", \"sub\",\n                   on_failure='ignore')\n    assert_status('error', res)\n    assert_message('Failed to clone from all attempted sources: %s',\n                   res)\n\n\n@with_testrepos('submodule_annex', flavors=['local'])\n@with_tempfile(mkdir=True)\ndef test_reckless(path, top_path):\n    ds = clone(path, top_path, reckless=True,\n               result_xfm='datasets', return_type='item-or-list')\n    eq_(ds.config.get('annex.hardlink', None), 'true')\n    eq_(ds.repo.repo_info()['untrusted repositories'][0]['here'], True)\n\n\n@with_tempfile\n@with_tempfile\ndef test_install_source_relpath(src, dest):\n    src = Path(src)\n    create(src)\n    src_ = src.name\n    with chpwd(src.parent):\n        clone(src_, dest)\n\n\n@with_tempfile\n@with_tempfile\ndef test_clone_isnt_a_smartass(origin_path, path):\n    origin = create(origin_path)\n    cloned = clone(origin, path,\n                   result_xfm='datasets', return_type='item-or-list')\n    with chpwd(path):\n        # no were are inside a dataset clone, and we make another one\n        # we do not want automatic subdatasetification without given a dataset\n        # explicitely\n        clonedsub = clone(origin, 'testsub',\n                          result_xfm='datasets', return_type='item-or-list')\n    # correct destination\n    assert clonedsub.path.startswith(path)\n    # no subdataset relation\n    eq_(cloned.subdatasets(), [])\n\n\n@skip_if(on_windows or not os.geteuid(), \"Will fail under super-user\")\n@with_tempfile(mkdir=True)\ndef test_clone_report_permission_issue(tdir):\n    pdir = Path(tdir) / 'protected'\n    pdir.mkdir()\n    # make it read-only\n    pdir.chmod(0o555)\n    with chpwd(pdir):\n        res = clone('///', result_xfm=None, return_type='list', on_failure='ignore')\n        assert_status('error', res)\n        assert_result_count(\n            res, 1, status='error',\n            message=\"could not create work tree dir '%s/%s': Permission denied\"\n                    % (pdir, get_datasets_topdir())\n        )\n\n\n# Started to hang on appveyor.\n@known_failure_appveyor  #FIXME - hangs\n@skip_if_no_network\n@with_tempfile\ndef test_autoenabled_remote_msg(path):\n    # Verify that no message about a remote not been enabled is displayed\n    # whenever the remote we clone is the  type=git special remote, so the name\n    # of the remote might not match\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone('///repronim/containers', path, result_xfm=None, return_type='list')\n        assert_status('ok', res)\n        assert_not_in(\"not auto-enabled\", cml.out)\n\n\n@with_sameas_remote(autoenabled=True)\n@with_tempfile(mkdir=True)\ndef test_clone_autoenable_msg_handles_sameas(repo, clone_path):\n    ds = Dataset(repo.path)\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = clone(ds, clone_path)\n        assert_status('ok', res)\n        assert_in(\"r_dir\", cml.out)\n        assert_in(\"not auto-enabled\", cml.out)\n        # The rsyncurl remote was enabled.\n        assert_not_in(\"r_rsync\", cml.out)\n    ds_cloned = Dataset(clone_path)\n    remotes = ds_cloned.repo.get_remotes()\n    assert_in(\"r_rsync\", remotes)\n    assert_not_in(\"r_dir\", remotes)\n\n\ndef test_installationpath_from_url():\n    cases = (\n        'http://example.com/lastbit',\n        'http://example.com/lastbit.git',\n        'http://lastbit:8000',\n    ) + (\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit',\n        'C:\\\\Users\\\\mih\\\\AppData\\\\Local\\\\Temp\\\\lastbit\\\\',\n        'Temp\\\\lastbit',\n        'Temp\\\\lastbit\\\\',\n        'lastbit.git',\n        'lastbit.git\\\\',\n    ) if on_windows else (\n        'lastbit',\n        'lastbit/',\n        '/lastbit',\n        'lastbit.git',\n        'lastbit.git/',\n    )\n    for p in cases:\n        eq_(_get_installationpath_from_url(p), 'lastbit')\n    # we need to deal with quoted urls\n    for url in (\n        # although some docs say that space could've been replaced with +\n        'http://localhost:8000/+last%20bit',\n        'http://localhost:8000/%2Blast%20bit',\n        '///%2Blast%20bit',\n        '///d1/%2Blast%20bit',\n        '///d1/+last bit',\n    ):\n        eq_(_get_installationpath_from_url(url), '+last bit')\n    # and the hostname alone\n    eq_(_get_installationpath_from_url(\"http://hostname\"), 'hostname')\n    eq_(_get_installationpath_from_url(\"http://hostname/\"), 'hostname')\n\n\n# https://github.com/datalad/datalad/issues/3958\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_expanduser(srcpath, destpath):\n    src = Dataset(Path(srcpath) / 'src').create()\n    dest = Dataset(Path(destpath) / 'dest').create()\n\n    with chpwd(destpath), patch.dict('os.environ', {'HOME': srcpath}):\n        res = clone(op.join('~', 'src'), 'dest', result_xfm=None, return_type='list',\n                    on_failure='ignore')\n        assert_result_count(res, 1)\n        assert_result_count(\n            res, 1, action='install', status='error', path=dest.path,\n            message='target path already exists and not empty, refuse to '\n            'clone into target path')\n        # wipe out destination, and try again\n        assert_status('ok', remove(dataset=dest, check=False))\n        # now it should do it, and clone the right one\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n        eq_(src.id, cloneds.id)\n        # and it shouldn't fail when doing it again, because it detects\n        # the re-clone\n        cloneds = clone(op.join('~', 'src'), 'dest')\n        eq_(cloneds.pathobj, Path(destpath) / 'dest')\n\n\n@with_tempfile(mkdir=True)\ndef test_cfg_originorigin(path):\n    path = Path(path)\n    origin = Dataset(path / 'origin').create()\n    (origin.pathobj / 'file1.txt').write_text('content')\n    origin.save()\n    clone_direct = clone(origin, path / 'clone_direct')\n    clone_clone = clone(clone_direct, path / 'clone_clone')\n    # the goal is to be able to get file content from origin without\n    # the need to configure it manually\n    assert_result_count(\n        clone_clone.get('file1.txt', on_failure='ignore'),\n        1,\n        action='get',\n        status='ok',\n        path=str(clone_clone.pathobj / 'file1.txt'),\n    )\n    eq_((clone_clone.pathobj / 'file1.txt').read_text(), 'content')\n    eq_(\n        Path(clone_clone.siblings(\n            'query',\n            name='origin-2',\n            return_type='item-or-list')['url']),\n        origin.pathobj\n    )\n\n\n# test fix for gh-2601/gh-3538\n@with_tempfile()\ndef test_relative_submodule_url(path):\n    Dataset(op.join(path, 'origin')).create()\n    ds = Dataset(op.join(path, 'ds')).create()\n    with chpwd(ds.path):\n        ds.clone(\n            source=op.join(op.pardir, 'origin'),\n            path='sources')\n    subinfo = ds.subdatasets(return_type='item-or-list')\n    eq_(subinfo['gitmodule_url'],\n        # must be a relative URL, not platform-specific relpath!\n        '{}/{}'.format(op.pardir, 'origin'))\n\n\ndef test_decode_source_spec():\n    # resolves datalad RIs:\n    eq_(decode_source_spec('///subds'),\n        dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n             type='dataladri', default_destpath='subds'))\n    assert_raises(NotImplementedError, decode_source_spec,\n                  '//custom/subds')\n\n    # doesn't harm others:\n    for url in (\n            'http://example.com',\n            '/absolute/path',\n            'file://localhost/some',\n            'localhost/another/path',\n            'user@someho.st/mydir',\n            'ssh://somewhe.re/else',\n            'git://github.com/datalad/testrepo--basic--r1',\n    ):\n        props = decode_source_spec(url)\n        dest = props.pop('default_destpath')\n        eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n\n    # RIA URIs with and without version specification\n    dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n    for proto, loc, version in (\n            ('http', 'example.com', None),\n            ('http', 'example.com', 'v1.0'),\n            ('http', 'example.com', 'some_with@in_it'),\n            ('ssh', 'example.com', 'some_with@in_it'),\n    ):\n        spec = 'ria+{}://{}{}{}'.format(\n            proto,\n            loc,\n            '#{}'.format(dsid),\n            '@{}'.format(version) if version else '')\n        eq_(decode_source_spec(spec),\n            dict(\n                source=spec,\n                giturl='{}://{}/{}/{}'.format(\n                    proto,\n                    loc,\n                    dsid[:3],\n                    dsid[3:]),\n                version=version,\n                default_destpath=dsid,\n                type='ria')\n        )\n    # not a dataset UUID\n    assert_raises(ValueError, decode_source_spec, 'ria+http://example.com#123')\n\n\n@skip_if_no_network\n@with_tempfile()\ndef test_ria_http(path):\n    # can we clone from the store w/o any dedicated config\n    ds = clone('ria+http://store.datalad.org#{}'.format(datalad_store_testds_id), path)\n    ok_(ds.is_installed())\n    eq_(ds.id, datalad_store_testds_id)\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.12.0.dev17\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 149ms\nInstalled 25 packages in 5.15s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 65 packages in 4.50s\nPrepared 1 package in 3.26s\nInstalled 59 packages in 362ms\n + appdirs==1.4.4\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.12.0 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + gitdb==4.0.12\n + gitpython==3.1.44\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + jsmin==3.0.1\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.1.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + smmap==5.0.2\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 99 packages in 208ms\nInstalled 32 packages in 665ms\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb\nplugins: cov-6.0.0\ncollected 22 items\n\nr2e_tests/test_1.py EEEEEEEEEEEEEEFE.EFF.F                               [100%]\n\n==================================== ERRORS ====================================\n_____________________ ERROR at setup of test_invalid_args ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 77\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_invalid_args(path, otherpath, alienpath):\nE       fixture 'otherpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:77\n______________________ ERROR at setup of test_clone_crcns ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 109\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_crcns(tdir, ds_path):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:109\n__________________ ERROR at setup of test_clone_datasets_root __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 128\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tree(tree={'sub': {}})\n  def test_clone_datasets_root(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:128\n__________________ ERROR at setup of test_clone_simple_local ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 156\n  @with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n  @with_tempfile(mkdir=True)\n  def test_clone_simple_local(src, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:156\n____________ ERROR at setup of test_clone_dataset_from_just_source _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 198\n  @with_testrepos(flavors=['local-url', 'network', 'local'])\n  @with_tempfile\n  def test_clone_dataset_from_just_source(url, path):\nE       fixture 'url' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:198\n____________________ ERROR at setup of test_clone_dataladri ____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 213\n  @known_failure\n  @with_tree(tree={\n      'ds': {'test.txt': 'some'},\n      })\n  @serve_path_via_http\n  @with_tempfile(mkdir=True)\n  def test_clone_dataladri(src, topurl, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:213\n_________________ ERROR at setup of test_clone_isnot_recursive _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 234\n  @with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_isnot_recursive(src, path_nr, path_r):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:234\n__________________ ERROR at setup of test_clone_into_dataset ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 248\n  @slow  # 23.1478s\n  @with_testrepos(flavors=['local'])\n  # 'local-url', 'network'\n  # TODO: Somehow annex gets confused while initializing installed ds, whose\n  # .git/config show a submodule url \"file:///aaa/bbb%20b/...\"\n  # this is delivered by with_testrepos as the url to clone\n  @with_tempfile\n  def test_clone_into_dataset(source, top_path):\nE       fixture 'source' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:248\n_______________ ERROR at setup of test_notclone_known_subdataset _______________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 281\n  @with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n  @with_tempfile(mkdir=True)\n  def test_notclone_known_subdataset(src, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:281\n_____________________ ERROR at setup of test_failed_clone ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 311\n  @with_tempfile(mkdir=True)\n  def test_failed_clone(dspath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:311\n_______________________ ERROR at setup of test_reckless ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 321\n  @with_testrepos('submodule_annex', flavors=['local'])\n  @with_tempfile(mkdir=True)\n  def test_reckless(path, top_path):\nE       fixture 'top_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:321\n________________ ERROR at setup of test_install_source_relpath _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 330\n  @with_tempfile\n  @with_tempfile\n  def test_install_source_relpath(src, dest):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:330\n_________________ ERROR at setup of test_clone_isnt_a_smartass _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 340\n  @with_tempfile\n  @with_tempfile\n  def test_clone_isnt_a_smartass(origin_path, path):\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:340\n_____________ ERROR at setup of test_clone_report_permission_issue _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 358\n  @skip_if(on_windows or not os.geteuid(), \"Will fail under super-user\")\n  @with_tempfile(mkdir=True)\n  def test_clone_report_permission_issue(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:358\n__________ ERROR at setup of test_clone_autoenable_msg_handles_sameas __________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 389\n  @with_sameas_remote(autoenabled=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_autoenable_msg_handles_sameas(repo, clone_path):\nE       fixture 'repo' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:389\n______________________ ERROR at setup of test_expanduser _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 443\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_expanduser(srcpath, destpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:443\n=================================== FAILURES ===================================\n_________________________ test_autoenabled_remote_msg __________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexvxz8votm'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def newfunc(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:141: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexvxz8votm'}\nfilename = '/tmp/datalad_temp_test_autoenabled_remote_msgfd1frn87'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_autoenabled_remote_msg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n____________________________ test_cfg_originorigin _____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexayhbybvr'}\nfilename = '/tmp/datalad_temp_test_cfg_originoriginct2q6vl3'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_cfg_originorigin() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n_________________________ test_relative_submodule_url __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex34uf08s9'}\nfilename = '/tmp/datalad_temp_test_relative_submodule_urluewvcbee'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_relative_submodule_url() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n________________________________ test_ria_http _________________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annex7oxtb7sr'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def newfunc(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:141: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annex7oxtb7sr'}\nfilename = '/tmp/datalad_temp_test_ria_http8wclbydp'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_ria_http() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_installationpath_from_url\nPASSED r2e_tests/test_1.py::test_decode_source_spec\nERROR r2e_tests/test_1.py::test_invalid_args\nERROR r2e_tests/test_1.py::test_clone_crcns\nERROR r2e_tests/test_1.py::test_clone_datasets_root\nERROR r2e_tests/test_1.py::test_clone_simple_local\nERROR r2e_tests/test_1.py::test_clone_dataset_from_just_source\nERROR r2e_tests/test_1.py::test_clone_dataladri\nERROR r2e_tests/test_1.py::test_clone_isnot_recursive\nERROR r2e_tests/test_1.py::test_clone_into_dataset\nERROR r2e_tests/test_1.py::test_notclone_known_subdataset\nERROR r2e_tests/test_1.py::test_failed_clone\nERROR r2e_tests/test_1.py::test_reckless\nERROR r2e_tests/test_1.py::test_install_source_relpath\nERROR r2e_tests/test_1.py::test_clone_isnt_a_smartass\nERROR r2e_tests/test_1.py::test_clone_report_permission_issue\nERROR r2e_tests/test_1.py::test_clone_autoenable_msg_handles_sameas\nERROR r2e_tests/test_1.py::test_expanduser\nFAILED r2e_tests/test_1.py::test_autoenabled_remote_msg - TypeError: test_aut...\nFAILED r2e_tests/test_1.py::test_cfg_originorigin - TypeError: test_cfg_origi...\nFAILED r2e_tests/test_1.py::test_relative_submodule_url - TypeError: test_rel...\nFAILED r2e_tests/test_1.py::test_ria_http - TypeError: test_ria_http() got mu...\n==================== 4 failed, 2 passed, 16 errors in 2.27s ====================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb\nplugins: cov-6.0.0\ncollected 22 items\n\nr2e_tests/test_1.py EEEEEEEEEEEEEEFE.EFFFF                               [100%]\n\n==================================== ERRORS ====================================\n_____________________ ERROR at setup of test_invalid_args ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 77\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_invalid_args(path, otherpath, alienpath):\nE       fixture 'otherpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:77\n______________________ ERROR at setup of test_clone_crcns ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 109\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_crcns(tdir, ds_path):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:109\n__________________ ERROR at setup of test_clone_datasets_root __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 128\n  @integration\n  @skip_if_no_network\n  @use_cassette('test_install_crcns')\n  @with_tree(tree={'sub': {}})\n  def test_clone_datasets_root(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:128\n__________________ ERROR at setup of test_clone_simple_local ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 156\n  @with_testrepos('.*basic.*', flavors=['local-url', 'network', 'local'])\n  @with_tempfile(mkdir=True)\n  def test_clone_simple_local(src, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:156\n____________ ERROR at setup of test_clone_dataset_from_just_source _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 198\n  @with_testrepos(flavors=['local-url', 'network', 'local'])\n  @with_tempfile\n  def test_clone_dataset_from_just_source(url, path):\nE       fixture 'url' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:198\n____________________ ERROR at setup of test_clone_dataladri ____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 213\n  @known_failure\n  @with_tree(tree={\n      'ds': {'test.txt': 'some'},\n      })\n  @serve_path_via_http\n  @with_tempfile(mkdir=True)\n  def test_clone_dataladri(src, topurl, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:213\n_________________ ERROR at setup of test_clone_isnot_recursive _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 234\n  @with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_isnot_recursive(src, path_nr, path_r):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:234\n__________________ ERROR at setup of test_clone_into_dataset ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 248\n  @slow  # 23.1478s\n  @with_testrepos(flavors=['local'])\n  # 'local-url', 'network'\n  # TODO: Somehow annex gets confused while initializing installed ds, whose\n  # .git/config show a submodule url \"file:///aaa/bbb%20b/...\"\n  # this is delivered by with_testrepos as the url to clone\n  @with_tempfile\n  def test_clone_into_dataset(source, top_path):\nE       fixture 'source' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:248\n_______________ ERROR at setup of test_notclone_known_subdataset _______________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 281\n  @with_testrepos('submodule_annex', flavors=['local', 'local-url', 'network'])\n  @with_tempfile(mkdir=True)\n  def test_notclone_known_subdataset(src, path):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:281\n_____________________ ERROR at setup of test_failed_clone ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 311\n  @with_tempfile(mkdir=True)\n  def test_failed_clone(dspath):\nE       fixture 'dspath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:311\n_______________________ ERROR at setup of test_reckless ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 321\n  @with_testrepos('submodule_annex', flavors=['local'])\n  @with_tempfile(mkdir=True)\n  def test_reckless(path, top_path):\nE       fixture 'top_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:321\n________________ ERROR at setup of test_install_source_relpath _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 330\n  @with_tempfile\n  @with_tempfile\n  def test_install_source_relpath(src, dest):\nE       fixture 'src' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:330\n_________________ ERROR at setup of test_clone_isnt_a_smartass _________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 340\n  @with_tempfile\n  @with_tempfile\n  def test_clone_isnt_a_smartass(origin_path, path):\nE       fixture 'origin_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:340\n_____________ ERROR at setup of test_clone_report_permission_issue _____________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 358\n  @skip_if(on_windows or not os.geteuid(), \"Will fail under super-user\")\n  @with_tempfile(mkdir=True)\n  def test_clone_report_permission_issue(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:358\n__________ ERROR at setup of test_clone_autoenable_msg_handles_sameas __________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 389\n  @with_sameas_remote(autoenabled=True)\n  @with_tempfile(mkdir=True)\n  def test_clone_autoenable_msg_handles_sameas(repo, clone_path):\nE       fixture 'repo' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:389\n______________________ ERROR at setup of test_expanduser _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py, line 443\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_expanduser(srcpath, destpath):\nE       fixture 'srcpath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_fbb05051f6c01ff8c3257ce4458cfe47644f8ddb/r2e_tests/test_1.py:443\n=================================== FAILURES ===================================\n_________________________ test_autoenabled_remote_msg __________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexzq0ozd2m'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def newfunc(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:141: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexzq0ozd2m'}\nfilename = '/tmp/datalad_temp_test_autoenabled_remote_msg6lm7q18o'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_autoenabled_remote_msg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n____________________________ test_cfg_originorigin _____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex2sm1eiaf'}\nfilename = '/tmp/datalad_temp_test_cfg_originorigincdnsrs_y'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_cfg_originorigin() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n_________________________ test_relative_submodule_url __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexcclrmzt8'}\nfilename = '/tmp/datalad_temp_test_relative_submodule_url9f1p_l3b'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_relative_submodule_url() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n___________________________ test_decode_source_spec ____________________________\n\n    def test_decode_source_spec():\n        # resolves datalad RIs:\n        eq_(decode_source_spec('///subds'),\n            dict(source='///subds', giturl=consts.DATASETS_TOPURL + 'subds', version=None,\n                 type='dataladri', default_destpath='subds'))\n        assert_raises(NotImplementedError, decode_source_spec,\n                      '//custom/subds')\n    \n        # doesn't harm others:\n        for url in (\n                'http://example.com',\n                '/absolute/path',\n                'file://localhost/some',\n                'localhost/another/path',\n                'user@someho.st/mydir',\n                'ssh://somewhe.re/else',\n                'git://github.com/datalad/testrepo--basic--r1',\n        ):\n            props = decode_source_spec(url)\n            dest = props.pop('default_destpath')\n            eq_(props, dict(source=url, version=None, giturl=url, type='giturl'))\n    \n        # RIA URIs with and without version specification\n        dsid = '6d69ca68-7e85-11e6-904c-002590f97d84'\n        for proto, loc, version in (\n                ('http', 'example.com', None),\n                ('http', 'example.com', 'v1.0'),\n                ('http', 'example.com', 'some_with@in_it'),\n                ('ssh', 'example.com', 'some_with@in_it'),\n        ):\n            spec = 'ria+{}://{}{}{}'.format(\n                proto,\n                loc,\n                '#{}'.format(dsid),\n                '@{}'.format(version) if version else '')\n>           eq_(decode_source_spec(spec),\n                dict(\n                    source=spec,\n                    giturl='{}://{}/{}/{}'.format(\n                        proto,\n                        loc,\n                        dsid[:3],\n                        dsid[3:]),\n                    version=version,\n                    default_destpath=dsid,\n                    type='ria')\n            )\n\nr2e_tests/test_1.py:546: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\na = {'default_destpath': '6d69ca68-7e85-11e6-904c-002590f97d84', 'giturl': 'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84', 'source': 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it', 'type': 'ria', ...}\nb = {'default_destpath': '6d69ca68-7e85-11e6-904c-002590f97d84', 'giturl': 'ssh://example.com/6d6/9ca68-7e85-11e6-904c-002590f97d84', 'source': 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it', 'type': 'ria', ...}\nmsg = None\n\n    def eq_(a, b, msg=None):\n        \"\"\"Shorthand for 'assert a == b, \"%r != %r\" % (a, b)\n        \"\"\"\n        if not a == b:\n>           raise AssertionError(msg or \"%r != %r\" % (a, b))\nE           AssertionError: {'source': 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it', 'version': 'some_with@in_it', 'type': 'ria', 'giturl': 'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84', 'default_destpath': '6d69ca68-7e85-11e6-904c-002590f97d84'} != {'source': 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it', 'giturl': 'ssh://example.com/6d6/9ca68-7e85-11e6-904c-002590f97d84', 'version': 'some_with@in_it', 'default_destpath': '6d69ca68-7e85-11e6-904c-002590f97d84', 'type': 'ria'}\n\n.venv/lib/python3.9/site-packages/nose/tools/trivial.py:29: AssertionError\n________________________________ test_ria_http _________________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexzq56nn_z'}\n\n    @wraps(func)\n    @attr('network')\n    @attr('skip_if_no_network')\n    def newfunc(*args, **kwargs):\n        check_and_raise()\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:141: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexzq56nn_z'}\nfilename = '/tmp/datalad_temp_test_ria_http5hnihj2g'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_ria_http() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:732: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_installationpath_from_url\nERROR r2e_tests/test_1.py::test_invalid_args\nERROR r2e_tests/test_1.py::test_clone_crcns\nERROR r2e_tests/test_1.py::test_clone_datasets_root\nERROR r2e_tests/test_1.py::test_clone_simple_local\nERROR r2e_tests/test_1.py::test_clone_dataset_from_just_source\nERROR r2e_tests/test_1.py::test_clone_dataladri\nERROR r2e_tests/test_1.py::test_clone_isnot_recursive\nERROR r2e_tests/test_1.py::test_clone_into_dataset\nERROR r2e_tests/test_1.py::test_notclone_known_subdataset\nERROR r2e_tests/test_1.py::test_failed_clone\nERROR r2e_tests/test_1.py::test_reckless\nERROR r2e_tests/test_1.py::test_install_source_relpath\nERROR r2e_tests/test_1.py::test_clone_isnt_a_smartass\nERROR r2e_tests/test_1.py::test_clone_report_permission_issue\nERROR r2e_tests/test_1.py::test_clone_autoenable_msg_handles_sameas\nERROR r2e_tests/test_1.py::test_expanduser\nFAILED r2e_tests/test_1.py::test_autoenabled_remote_msg - TypeError: test_aut...\nFAILED r2e_tests/test_1.py::test_cfg_originorigin - TypeError: test_cfg_origi...\nFAILED r2e_tests/test_1.py::test_relative_submodule_url - TypeError: test_rel...\nFAILED r2e_tests/test_1.py::test_decode_source_spec - AssertionError: {'sourc...\nFAILED r2e_tests/test_1.py::test_ria_http - TypeError: test_ria_http() got mu...\n==================== 5 failed, 1 passed, 16 errors in 1.01s ====================\n",
    "old_commit_res_stderr": ""
}
datalad/core/distributed/clone.py datalad/core/distributed/tests/test_clone.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title:** SSH URLs Generated with Incorrect Host-Path Delimiter

**Description:**

When generating SSH URLs for RIA repositories, the URLs incorrectly use a colon (`:`) to separate the host and path. This results in malformed URLs that do not conform to the expected format.

**Buggy Code Example:**

```python
spec = 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it'
props = decode_source_spec(spec)
print(props['giturl'])  # Outputs: 'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

**Expected Behavior:**

The `giturl` should use a slash (`/`) to separate the host and path, resulting in:
```
'ssh://example.com/6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

**Actual Behavior:**

The `giturl` incorrectly uses a colon (`:`) as the delimiter between the host and path:
```
'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

This malformed URL leads to failures in operations that rely on correctly formatted SSH URLs.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/core/distributed/clone.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Plumbing command for dataset installation"""


import logging
import os
import re
from os.path import expanduser
from collections import OrderedDict
from urllib.parse import unquote as urlunquote

from datalad.interface.base import Interface
from datalad.interface.utils import eval_results
from datalad.interface.base import build_doc
from datalad.interface.results import get_status_dict
from datalad.interface.common_opts import (
    location_description,
    reckless_opt,
)
from datalad.log import log_progress
from datalad.support.gitrepo import (
    GitRepo,
    GitCommandError,
)
from datalad.support.annexrepo import AnnexRepo
from datalad.support.constraints import (
    EnsureNone,
    EnsureStr,
    EnsureKeyChoice,
)
from datalad.support.param import Parameter
from datalad.support.network import (
    get_local_file_url,
    URL,
    RI,
    DataLadRI,
    PathRI,
)
from datalad.dochelpers import (
    exc_str,
    single_or_plural,
)
from datalad.utils import (
    rmtree,
    assure_bool,
    knows_annex,
    Path,
)

from datalad.distribution.dataset import (
    Dataset,
    datasetmethod,
    resolve_path,
    require_dataset,
    EnsureDataset,
)
from datalad.distribution.utils import (
    _get_flexible_source_candidates,
)

__docformat__ = 'restructuredtext'

lgr = logging.getLogger('datalad.core.distributed.clone')


@build_doc
class Clone(Interface):
    """Obtain a dataset (copy) from a URL or local directory

    The purpose of this command is to obtain a new clone (copy) of a dataset
    and place it into a not-yet-existing or empty directory. As such `clone`
    provides a strict subset of the functionality offered by `install`. Only a
    single dataset can be obtained, and immediate recursive installation of
    subdatasets is not supported. However, once a (super)dataset is installed
    via `clone`, any content, including subdatasets can be obtained by a
    subsequent `get` command.

    Primary differences over a direct `git clone` call are 1) the automatic
    initialization of a dataset annex (pure Git repositories are equally
    supported); 2) automatic registration of the newly obtained dataset
    as a subdataset (submodule), if a parent dataset is specified; and
    3) support for datalad's resource identifiers and automatic configurable
    generation of alternative access URL for common cases (such as appending
    '.git' to the URL in case the accessing the base URL failed).

    || PYTHON >>By default, the command returns a single Dataset instance for
    an installed dataset, regardless of whether it was newly installed ('ok'
    result), or found already installed from the specified source ('notneeded'
    result).<< PYTHON ||
    """
    # by default ignore everything but install results
    # i.e. no "add to super dataset"
    result_filter = EnsureKeyChoice('action', ('install',))
    # very frequently this command will yield exactly one installed dataset
    # spare people the pain of going through a list by default
    return_type = 'item-or-list'
    # as discussed in #1409 and #1470, we want to return dataset instances
    # matching what is actually available after command completion (and
    # None for any failed dataset installation)
    result_xfm = 'successdatasets-or-none'

    _examples_ = [
        dict(text="Install a dataset from Github into the current directory",
             code_py="clone("
             "source='https://github.com/datalad-datasets/longnow"
             "-podcasts.git')",
             code_cmd="datalad clone "
             "https://github.com/datalad-datasets/longnow-podcasts.git"),
        dict(text="Install a dataset into a specific directory",
             code_py="clone("
             "source='https://github.com/datalad-datasets/longnow"
             "-podcasts.git', path='myfavpodcasts')",
             code_cmd="datalad clone "
             "https://github.com/datalad-datasets/longnow-podcasts.git "
             "myfavpodcasts"),
        dict(text="Install a dataset as a subdataset into the current dataset",
             code_py="clone(dataset='.', "
             "source='https://github.com/datalad-datasets/longnow-podcasts.git')",
             code_cmd="datalad clone -d . "
             "--source='https://github.com/datalad-datasets/longnow-podcasts.git'"),
    ]

    _params_ = dict(
        dataset=Parameter(
            args=("-d", "--dataset"),
            doc="""(parent) dataset to clone into. If given, the newly cloned
            dataset is registered as a subdataset of the parent. Also, if given,
            relative paths are interpreted as being relative to the parent
            dataset, and not relative to the working directory.""",
            constraints=EnsureDataset() | EnsureNone()),
        source=Parameter(
            args=("source",),
            metavar='SOURCE',
            doc="""URL, DataLad resource identifier, local path or instance of
            dataset to be cloned""",
            constraints=EnsureStr() | EnsureNone()),
        path=Parameter(
            args=("path",),
            metavar='PATH',
            nargs="?",
            doc="""path to clone into.  If no `path` is provided a
            destination path will be derived from a source URL
            similar to :command:`git clone`"""),
        description=location_description,
        reckless=reckless_opt,
    )

    @staticmethod
    @datasetmethod(name='clone')
    @eval_results
    def __call__(
            source,
            path=None,
            dataset=None,
            description=None,
            reckless=None):
        # legacy compatibility
        if reckless is True:
            # so that we can forget about how things used to be
            reckless = 'auto'

        # did we explicitly get a dataset to install into?
        # if we got a dataset, path will be resolved against it.
        # Otherwise path will be resolved first.
        ds = require_dataset(
            dataset, check_installed=True, purpose='cloning') \
            if dataset is not None else dataset
        refds_path = ds.path if ds else None

        if isinstance(source, Dataset):
            source = source.path

        if source == path:
            # even if they turn out to be identical after resolving symlinks
            # and more sophisticated witchcraft, it would still happily say
            # "it appears to be already installed", so we just catch an
            # obviously pointless input combination
            raise ValueError(
                "clone `source` and destination `path` are identical [{}]. "
                "If you are trying to add a subdataset simply use `save`".format(
                    path))

        if path is not None:
            path = resolve_path(path, dataset)

        # derive target from source:
        if path is None:
            # we got nothing but a source. do something similar to git clone
            # and derive the path from the source and continue
            # since this is a relative `path`, resolve it:
            # we are not going to reuse the decoded URL, as this is done for
            # all source candidates in clone_dataset(), we just use to determine
            # a destination path here in order to perform a bunch of additional
            # checks that shall not pollute the helper function
            source_ = decode_source_spec(
                source, cfg=None if ds is None else ds.config)
            path = resolve_path(source_['default_destpath'], dataset)
            lgr.debug("Determined clone target path from source")
        lgr.debug("Resolved clone target path to: '%s'", path)

        # there is no other way -- my intoxicated brain tells me
        assert(path is not None)

        result_props = dict(
            action='install',
            logger=lgr,
            refds=refds_path,
            source_url=source)

        try:
            # this will implicitly cause pathlib to run a bunch of checks
            # whether the present path makes any sense on the platform
            # we are running on -- we don't care if the path actually
            # exists at this point, but we want to abort early if the path
            # spec is determined to be useless
            path.exists()
        except OSError as e:
            yield get_status_dict(
                status='error',
                path=path,
                message=('cannot handle target path: %s', exc_str(e)),
                **result_props)
            return

        destination_dataset = Dataset(path)
        result_props['ds'] = destination_dataset

        if ds is not None and ds.pathobj not in path.parents:
            yield get_status_dict(
                status='error',
                message=("clone target path '%s' not in specified target dataset '%s'",
                         path, ds),
                **result_props)
            return

        # perform the actual cloning operation
        yield from clone_dataset(
            [source],
            destination_dataset,
            reckless,
            description,
            result_props,
            cfg=None if ds is None else ds.config,
        )

        # TODO handle any 'version' property handling and verification using a dedicated
        # public helper

        if ds is not None:
            # we created a dataset in another dataset
            # -> make submodule
            for r in ds.save(
                    path,
                    return_type='generator',
                    result_filter=None,
                    result_xfm=None,
                    on_failure='ignore'):
                yield r


def clone_dataset(
        srcs,
        destds,
        reckless=None,
        description=None,
        result_props=None,
        cfg=None):
    """Internal helper to perform cloning without sanity checks (assumed done)

    This helper does not handle any saving of subdataset modification or adding
    in a superdataset.

    Parameters
    ----------
    srcs : list
      Any suitable clone source specifications (paths, URLs)
    destds : Dataset
      Dataset instance for the clone destination
    reckless : {None, 'auto'}, optional
      Mode switch to put cloned dataset into throw-away configurations, i.e.
      sacrifice data safety for performance or resource footprint.
    description : str, optional
      Location description for the annex of the dataset clone (if there is any).
    result_props : dict, optional
      Default properties for any yielded result, passed on to get_status_dict().
    cfg : ConfigManager, optional
      Configuration will be queried from this instance (i.e. from a particular
      dataset). If None is given, the global DataLad configuration will be
      queried.

    Yields
    ------
    dict
      DataLad result records
    """
    if not result_props:
        # in case the caller had no specific idea on how results should look
        # like, provide sensible defaults
        result_props = dict(
            action='install',
            logger=lgr,
            ds=destds,
        )

    dest_path = destds.pathobj

    # decode all source candidate specifications
    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]

    # now expand the candidate sources with additional variants of the decoded
    # giturl, while duplicating the other properties in the additional records
    # for simplicity. The hope is to overcome a few corner cases and be more
    # robust than git clone
    candidate_sources = [
        dict(props, giturl=s) for props in candidate_sources
        for s in _get_flexible_source_candidates(props['giturl'])
    ]

    # important test! based on this `rmtree` will happen below after failed clone
    dest_path_existed = dest_path.exists()
    if dest_path_existed and any(dest_path.iterdir()):
        if destds.is_installed():
            # check if dest was cloned from the given source before
            # this is where we would have installed this from
            # this is where it was actually installed from
            track_name, track_url = _get_tracking_source(destds)
            try:
                # this will get us track_url in system native path conventions,
                # whenever it is a path (and not a URL)
                # this is needed to match it to any potentially incoming local
                # source path in the 'notneeded' test below
                track_path = str(Path(track_url))
            except Exception:
                # this should never happen, because Path() will let any non-path stringification
                # pass through unmodified, but we do not want any potential crash due to
                # pathlib behavior changes
                lgr.debug("Unexpected behavior of pathlib!")
                track_path = None
            for cand in candidate_sources:
                src = cand['giturl']
                if track_url == src \
                        or get_local_file_url(track_url, compatibility='git') == src \
                        or track_path == expanduser(src):
                    yield get_status_dict(
                        status='notneeded',
                        message=("dataset %s was already cloned from '%s'",
                                 destds,
                                 src),
                        **result_props)
                    return
        # anything else is an error
        yield get_status_dict(
            status='error',
            message='target path already exists and not empty, refuse to clone into target path',
            **result_props)
        return

    log_progress(
        lgr.info,
        'cloneds',
        'Cloning dataset to %s', destds,
        total=len(candidate_sources),
        label='Clone attempt',
        unit=' Candidate locations',
    )
    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url
    for cand in candidate_sources:
        try:
            log_progress(
                lgr.info,
                'cloneds',
                'Attempting to clone from %s to %s', cand['giturl'], dest_path,
                update=1,
                increment=True)
            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35
            # doesn't make it happen seemlessly
            GitRepo.clone(path=str(dest_path), url=cand['giturl'], create=True)

        except GitCommandError as e:
            error_msgs[cand['giturl']] = e
            lgr.debug("Failed to clone from URL: %s (%s)",
                      cand['giturl'], exc_str(e))
            if dest_path.exists():
                lgr.debug("Wiping out unsuccessful clone attempt at: %s",
                          dest_path)
                # We must not just rmtree since it might be curdir etc
                # we should remove all files/directories under it
                # TODO stringification can be removed once patlib compatible
                # or if PY35 is no longer supported
                rmtree(str(dest_path), children_only=dest_path_existed)
            # Whenever progress reporting is enabled, as it is now,
            # we end up without e.stderr since it is "processed" out by
            # GitPython/our progress handler.
            e_stderr = e.stderr
            from datalad.support.gitrepo import GitPythonProgressBar
            if not e_stderr and GitPythonProgressBar._last_error_lines:
                e_stderr = os.linesep.join(GitPythonProgressBar._last_error_lines)
            if 'could not create work tree' in e_stderr.lower():
                # this cannot be fixed by trying another URL
                re_match = re.match(r".*fatal: (.*)$", e_stderr,
                                    flags=re.MULTILINE | re.DOTALL)
                # cancel progress bar
                log_progress(
                    lgr.info,
                    'cloneds',
                    'Completed clone attempts for %s', destds
                )
                yield get_status_dict(
                    status='error',
                    message=re_match.group(1) if re_match else "stderr: " + e_stderr,
                    **result_props)
                return
            # next candidate
            continue

        # TODO perform any post-processing that needs to know details of the clone
        # source

        # do not bother with other sources if succeeded
        break

    log_progress(
        lgr.info,
        'cloneds',
        'Completed clone attempts for %s', destds
    )

    if not destds.is_installed():
        if len(error_msgs):
            if all(not e.stdout and not e.stderr for e in error_msgs.values()):
                # there is nothing we can learn from the actual exception,
                # the exit code is uninformative, the command is predictable
                error_msg = "Failed to clone from all attempted sources: %s"
                error_args = list(error_msgs.keys())
            else:
                error_msg = "Failed to clone from any candidate source URL. " \
                            "Encountered errors per each url were:\n- %s"
                error_args = '\n- '.join(
                    '{}\n  {}'.format(url, exc_str(exc))
                    for url, exc in error_msgs.items()
                )
        else:
            # yoh: Not sure if we ever get here but I felt that there could
            #      be a case when this might happen and original error would
            #      not be sufficient to troubleshoot what is going on.
            error_msg = "Awkward error -- we failed to clone properly. " \
                        "Although no errors were encountered, target " \
                        "dataset at %s seems to be not fully installed. " \
                        "The 'succesful' source was: %s"
            error_args = (destds.path, cand['giturl'])
        yield get_status_dict(
            status='error',
            message=(error_msg, error_args),
            **result_props)
        return

    yield from postclonecfg_annexdataset(
        destds,
        reckless,
        description)

    # yield successful clone of the base dataset now, as any possible
    # subdataset clone down below will not alter the Git-state of the
    # parent
    yield get_status_dict(status='ok', **result_props)


def postclonecfg_annexdataset(ds, reckless, description=None):
    """If ds "knows annex" -- annex init it, set into reckless etc

    Provides additional tune up to a possibly an annex repo, e.g.
    "enables" reckless mode, sets up description
    """
    # in any case check whether we need to annex-init the installed thing:
    if not knows_annex(ds.path):
        # not for us
        return

    # init annex when traces of a remote annex can be detected
    if reckless:
        lgr.debug(
            "Instruct annex to hardlink content in %s from local "
            "sources, if possible (reckless)", ds.path)
        ds.config.add(
            'annex.hardlink', 'true', where='local', reload=True)

    # we have just cloned the repo, so it has 'origin', configure any
    # reachable origin of origins
    yield from configure_origins(ds, ds)

    lgr.debug("Initializing annex repo at %s", ds.path)
    # Note, that we cannot enforce annex-init via AnnexRepo().
    # If such an instance already exists, its __init__ will not be executed.
    # Therefore do quick test once we have an object and decide whether to call its _init().
    #
    # Additionally, call init if we need to add a description (see #1403),
    # since AnnexRepo.__init__ can only do it with create=True
    repo = AnnexRepo(ds.path, init=True)
    if not repo.is_initialized() or description:
        repo._init(description=description)
    if reckless:
        repo._run_annex_command('untrust', annex_options=['here'])

    srs = {True: [], False: []}  # special remotes by "autoenable" key
    remote_uuids = None  # might be necessary to discover known UUIDs

    # Note: The purpose of this function is to inform the user. So if something
    # looks misconfigured, we'll warn and move on to the next item.
    for uuid, config in repo.get_special_remotes().items():
        sr_name = config.get('name', None)
        if sr_name is None:
            lgr.warning(
                'Ignoring special remote %s because it does not have a name. '
                'Known information: %s',
                uuid, config)
            continue
        sr_autoenable = config.get('autoenable', False)
        try:
            sr_autoenable = assure_bool(sr_autoenable)
        except ValueError:
            lgr.warning(
                'Failed to process "autoenable" value %r for sibling %s in '
                'dataset %s as bool.  You might need to enable it later '
                'manually and/or fix it up to avoid this message in the future.',
                sr_autoenable, sr_name, ds.path)
            continue

        # determine whether there is a registered remote with matching UUID
        if uuid:
            if remote_uuids is None:
                remote_uuids = {
                    # Check annex-config-uuid first. For sameas annex remotes,
                    # this will point to the UUID for the configuration (i.e.
                    # the key returned by get_special_remotes) rather than the
                    # shared UUID.
                    (repo.config.get('remote.%s.annex-config-uuid' % r) or
                     repo.config.get('remote.%s.annex-uuid' % r))
                    for r in repo.get_remotes()
                }
            if uuid not in remote_uuids:
                srs[sr_autoenable].append(sr_name)

    if srs[True]:
        lgr.debug(
            "configuration for %s %s added because of autoenable,"
            " but no UUIDs for them yet known for dataset %s",
            # since we are only at debug level, we could call things their
            # proper names
            single_or_plural("special remote", "special remotes", len(srs[True]), True),
            ", ".join(srs[True]),
            ds.path
        )

    if srs[False]:
        # if has no auto-enable special remotes
        lgr.info(
            'access to %s %s not auto-enabled, enable with:\n\t\tdatalad siblings -d "%s" enable -s %s',
            # but since humans might read it, we better confuse them with our
            # own terms!
            single_or_plural("dataset sibling", "dataset siblings", len(srs[False]), True),
            ", ".join(srs[False]),
            ds.path,
            srs[False][0] if len(srs[False]) == 1 else "SIBLING",
        )

_handle_possible_annex_dataset = postclonecfg_annexdataset


def configure_origins(cfgds, probeds, label=None):
    """Configure any discoverable local dataset 'origin' sibling as a remote

    Parameters
    ----------
    cfgds : Dataset
      Dataset to receive the remote configurations
    probeds : Dataset
      Dataset to start looking for 'origin' remotes. May be identical with `cfgds`.
    label : int, optional
      Each discovered 'origin' will be configured as a remote under the name
      'origin-<label>'. If no label is given, '2' will be used by default, given that
      there is typically a 'origin' remote already.
    """
    if label is None:
        label = 2
    # let's look at the URL for that remote and see if it is a local
    # dataset
    origin_url = probeds.config.get('remote.origin.url')
    if origin_url and cfgds.config.obtain(
            'datalad.install.inherit-local-origin',
            default=True) and isinstance(RI(origin_url), PathRI):
        # given the clone source is a local dataset, we can have a
        # cheap look at it, and configure its own 'origin' as a remote
        # (if there is any), and benefit from additional annex availability
        originorigin_ds = Dataset(origin_url)
        originorigin_url = originorigin_ds.config.get('remote.origin.url')
        if originorigin_url:
            yield from cfgds.siblings(
                'configure',
                # no chance for config, can only be the second configured remote
                name='origin-{}'.format(label),
                url=originorigin_url,
                # fetch to get all annex info
                fetch=True,
                result_renderer='disabled',
                on_failure='ignore',
            )
        # and dive deeper
        yield from configure_origins(cfgds, originorigin_ds, label=label + 1)


def _get_tracking_source(ds):
    """Returns name and url of a potential configured source
    tracking remote"""
    vcs = ds.repo
    # if we have a remote, let's check the location of that remote
    # for the presence of the desired submodule

    remote_name, tracking_branch = vcs.get_tracking_branch()
    if not remote_name and isinstance(vcs, AnnexRepo):
        # maybe cloned from a source repo that was in adjusted mode
        # https://github.com/datalad/datalad/issues/3969
        remote_name, tracking_branch = vcs.get_tracking_branch(
            corresponding=False)
    # TODO: better default `None`? Check where we might rely on '':
    remote_url = ''
    if remote_name:
        remote_url = vcs.get_remote_url(remote_name, push=False)

    return remote_name, remote_url


def _get_installationpath_from_url(url):
    """Returns a relative path derived from the trailing end of a URL

    This can be used to determine an installation path of a Dataset
    from a URL, analog to what `git clone` does.
    """
    ri = RI(url)
    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL
        path = ri.path.rstrip('/')
        path = urlunquote(path) if path else ri.hostname
        if '/' in path:
            path = path.split('/')
            if path[-1] == '.git':
                path = path[-2]
            else:
                path = path[-1]
    else:
        path = Path(url).parts[-1]
    if path.endswith('.git'):
        path = path[:-4]
    return path


def decode_source_spec(spec, cfg=None):
    """Decode information from a clone source specification

    Parameters
    ----------
    spec : str
      Any supported clone source specification
    cfg : ConfigManager, optional
      Configuration will be queried from the instance (i.e. from a particular
      dataset). If None is given, the global DataLad configuration will be
      queried.

    Returns
    -------
    dict
      The value of each decoded property is stored under its own key in this
      dict. By default the following keys are return: 'type', a specification
      type label {'giturl', 'dataladri', 'ria'}; 'source' the original
      source specification; 'giturl' a URL for the source that is a suitable
      source argument for git-clone; 'version' a version-identifer, if present
      (None else); 'default_destpath' a relative path that that can be used as
      a clone destination.
    """
    if cfg is None:
        from datalad import cfg
    # standard property dict composition
    props = dict(
        source=spec,
        version=None,
    )
    # common starting point is a RI instance, support for accepting an RI
    # instance is kept for backward-compatibility reasons
    source_ri = RI(spec) if not isinstance(spec, RI) else spec

    # scenario switch, each case must set 'giturl' at the very minimum
    if isinstance(source_ri, DataLadRI):
        # we have got our DataLadRI as the source, so expand it
        props['type'] = 'dataladri'
        props['giturl'] = source_ri.as_git_url()
    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):
        # parse a RIA URI
        # check for store related configuration for SSH access
        if source_ri.scheme == 'ria+ssh':
            hostname = cfg.get(
                'annex.ria-remote.{}.ssh-host'.format(source_ri.hostname),
                source_ri.hostname)
            basepath = cfg.get(
                'annex.ria-remote.{}.base-path'.format(source_ri.hostname),
                '')
        else:
            hostname = source_ri.hostname
            basepath = ''

        dsid, version = source_ri.fragment.split('@', maxsplit=1) \
            if '@' in source_ri.fragment else (source_ri.fragment, None)
        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'
        if not re.match(uuid_regex, dsid):
            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))
        props.update(
            type='ria',
            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(
                proto=source_ri.scheme[4:],
                host=hostname,
                delim=':' if source_ri.scheme == 'ria+ssh' else '/',
                basepath=basepath,
                pathdelim='/' if basepath else '',
                id1=dsid[:3],
                id2=dsid[3:]),
            version=version,
            default_destpath=dsid,
        )
    else:
        # let's assume that anything else is a URI that Git can handle
        # TODO are there any advantages of going through a URL-reconversion
        # by the RI instance, instead of using the original input right away?
        # it feels like we could only accumulate bugs. Keeping how it was,
        # for now...
        props['type'] = 'giturl'
        props['giturl'] = str(source_ri)

    if 'default_destpath' not in props:
        # if we still have no good idea on where a dataset could be cloned to if no
        # path was given, do something similar to git clone and derive the path from
        # the source
        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])

    return props

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]

**Title:** SSH URLs Generated with Incorrect Host-Path Delimiter

**Description:**

When generating SSH URLs for RIA repositories, the URLs incorrectly use a colon (`:`) to separate the host and path. This results in malformed URLs that do not conform to the expected format.

**Buggy Code Example:**

```python
spec = 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it'
props = decode_source_spec(spec)
print(props['giturl'])  # Outputs: 'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

**Expected Behavior:**

The `giturl` should use a slash (`/`) to separate the host and path, resulting in:
```
'ssh://example.com/6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

**Actual Behavior:**

The `giturl` incorrectly uses a colon (`:`) as the delimiter between the host and path:
```
'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

This malformed URL leads to failures in operations that rely on correctly formatted SSH URLs.

[/ISSUE]
{
    "test_installationpath_from_url": "PASSED",
    "test_decode_source_spec": "PASSED",
    "test_invalid_args": "ERROR",
    "test_clone_crcns": "ERROR",
    "test_clone_datasets_root": "ERROR",
    "test_clone_simple_local": "ERROR",
    "test_clone_dataset_from_just_source": "ERROR",
    "test_clone_dataladri": "ERROR",
    "test_clone_isnot_recursive": "ERROR",
    "test_clone_into_dataset": "ERROR",
    "test_notclone_known_subdataset": "ERROR",
    "test_failed_clone": "ERROR",
    "test_reckless": "ERROR",
    "test_install_source_relpath": "ERROR",
    "test_clone_isnt_a_smartass": "ERROR",
    "test_clone_report_permission_issue": "ERROR",
    "test_clone_autoenable_msg_handles_sameas": "ERROR",
    "test_expanduser": "ERROR",
    "test_autoenabled_remote_msg": "FAILED",
    "test_cfg_originorigin": "FAILED",
    "test_relative_submodule_url": "FAILED",
    "test_ria_http": "FAILED"
}
diff --git a/datalad/core/distributed/clone.py b/datalad/core/distributed/clone.py
index 84df56880..2ceeb6bd8 100644
--- a/datalad/core/distributed/clone.py
+++ b/datalad/core/distributed/clone.py
@@ -717,16 +717,23 @@ def decode_source_spec(spec, cfg=None):
         uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'
         if not re.match(uuid_regex, dsid):
             raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))
+        # now we cancel the fragment in the original URL, but keep everthing else
+        # in order to be able to support the various combinations of ports, paths,
+        # and everything else
+        source_ri.fragment = ''
+        # strip the custom protocol and go with standard one
+        source_ri.scheme = source_ri.scheme[4:]
+        # take any existing path, and add trace to dataset within the store
+        source_ri.path = '{urlpath}{urldelim}{basepath}{basedelim}{trace}'.format(
+            urlpath=source_ri.path if source_ri.path else '',
+            urldelim='' if not source_ri.path or source_ri.path.endswith('/') else '/',
+            basepath=basepath,
+            basedelim='' if not basepath or basepath.endswith('/') else '/',
+            trace='{}/{}'.format(dsid[:3], dsid[3:]),
+        )
         props.update(
             type='ria',
-            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(
-                proto=source_ri.scheme[4:],
-                host=hostname,
-                delim=':' if source_ri.scheme == 'ria+ssh' else '/',
-                basepath=basepath,
-                pathdelim='/' if basepath else '',
-                id1=dsid[:3],
-                id2=dsid[3:]),
+            giturl=str(source_ri),
             version=version,
             default_destpath=dsid,
         )

{"datalad/core/distributed/clone.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Plumbing command for dataset installation\"\"\"\n\n\nimport logging\nimport os\nimport re\nfrom os.path import expanduser\nfrom collections import OrderedDict\nfrom urllib.parse import unquote as urlunquote\n\nfrom datalad.interface.base import Interface\nfrom datalad.interface.utils import eval_results\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.results import get_status_dict\nfrom datalad.interface.common_opts import (\n    location_description,\n    reckless_opt,\n)\nfrom datalad.log import log_progress\nfrom datalad.support.gitrepo import (\n    GitRepo,\n    GitCommandError,\n)\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.constraints import (\n    EnsureNone,\n    EnsureStr,\n    EnsureKeyChoice,\n)\nfrom datalad.support.param import Parameter\nfrom datalad.support.network import (\n    get_local_file_url,\n    URL,\n    RI,\n    DataLadRI,\n    PathRI,\n)\nfrom datalad.dochelpers import (\n    exc_str,\n    single_or_plural,\n)\nfrom datalad.utils import (\n    rmtree,\n    assure_bool,\n    knows_annex,\n    Path,\n)\n\nfrom datalad.distribution.dataset import (\n    Dataset,\n    datasetmethod,\n    resolve_path,\n    require_dataset,\n    EnsureDataset,\n)\nfrom datalad.distribution.utils import (\n    _get_flexible_source_candidates,\n)\n\n__docformat__ = 'restructuredtext'\n\nlgr = logging.getLogger('datalad.core.distributed.clone')\n\n\n@build_doc\nclass Clone(Interface):\n    \"\"\"Obtain a dataset (copy) from a URL or local directory\n\n    The purpose of this command is to obtain a new clone (copy) of a dataset\n    and place it into a not-yet-existing or empty directory. As such `clone`\n    provides a strict subset of the functionality offered by `install`. Only a\n    single dataset can be obtained, and immediate recursive installation of\n    subdatasets is not supported. However, once a (super)dataset is installed\n    via `clone`, any content, including subdatasets can be obtained by a\n    subsequent `get` command.\n\n    Primary differences over a direct `git clone` call are 1) the automatic\n    initialization of a dataset annex (pure Git repositories are equally\n    supported); 2) automatic registration of the newly obtained dataset\n    as a subdataset (submodule), if a parent dataset is specified; and\n    3) support for datalad's resource identifiers and automatic configurable\n    generation of alternative access URL for common cases (such as appending\n    '.git' to the URL in case the accessing the base URL failed).\n\n    || PYTHON >>By default, the command returns a single Dataset instance for\n    an installed dataset, regardless of whether it was newly installed ('ok'\n    result), or found already installed from the specified source ('notneeded'\n    result).<< PYTHON ||\n    \"\"\"\n    # by default ignore everything but install results\n    # i.e. no \"add to super dataset\"\n    result_filter = EnsureKeyChoice('action', ('install',))\n    # very frequently this command will yield exactly one installed dataset\n    # spare people the pain of going through a list by default\n    return_type = 'item-or-list'\n    # as discussed in #1409 and #1470, we want to return dataset instances\n    # matching what is actually available after command completion (and\n    # None for any failed dataset installation)\n    result_xfm = 'successdatasets-or-none'\n\n    _examples_ = [\n        dict(text=\"Install a dataset from Github into the current directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git\"),\n        dict(text=\"Install a dataset into a specific directory\",\n             code_py=\"clone(\"\n             \"source='https://github.com/datalad-datasets/longnow\"\n             \"-podcasts.git', path='myfavpodcasts')\",\n             code_cmd=\"datalad clone \"\n             \"https://github.com/datalad-datasets/longnow-podcasts.git \"\n             \"myfavpodcasts\"),\n        dict(text=\"Install a dataset as a subdataset into the current dataset\",\n             code_py=\"clone(dataset='.', \"\n             \"source='https://github.com/datalad-datasets/longnow-podcasts.git')\",\n             code_cmd=\"datalad clone -d . \"\n             \"--source='https://github.com/datalad-datasets/longnow-podcasts.git'\"),\n    ]\n\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"(parent) dataset to clone into. If given, the newly cloned\n            dataset is registered as a subdataset of the parent. Also, if given,\n            relative paths are interpreted as being relative to the parent\n            dataset, and not relative to the working directory.\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        source=Parameter(\n            args=(\"source\",),\n            metavar='SOURCE',\n            doc=\"\"\"URL, DataLad resource identifier, local path or instance of\n            dataset to be cloned\"\"\",\n            constraints=EnsureStr() | EnsureNone()),\n        path=Parameter(\n            args=(\"path\",),\n            metavar='PATH',\n            nargs=\"?\",\n            doc=\"\"\"path to clone into.  If no `path` is provided a\n            destination path will be derived from a source URL\n            similar to :command:`git clone`\"\"\"),\n        description=location_description,\n        reckless=reckless_opt,\n    )\n\n    @staticmethod\n    @datasetmethod(name='clone')\n    @eval_results\n    def __call__(\n            source,\n            path=None,\n            dataset=None,\n            description=None,\n            reckless=None):\n        # legacy compatibility\n        if reckless is True:\n            # so that we can forget about how things used to be\n            reckless = 'auto'\n\n        # did we explicitly get a dataset to install into?\n        # if we got a dataset, path will be resolved against it.\n        # Otherwise path will be resolved first.\n        ds = require_dataset(\n            dataset, check_installed=True, purpose='cloning') \\\n            if dataset is not None else dataset\n        refds_path = ds.path if ds else None\n\n        if isinstance(source, Dataset):\n            source = source.path\n\n        if source == path:\n            # even if they turn out to be identical after resolving symlinks\n            # and more sophisticated witchcraft, it would still happily say\n            # \"it appears to be already installed\", so we just catch an\n            # obviously pointless input combination\n            raise ValueError(\n                \"clone `source` and destination `path` are identical [{}]. \"\n                \"If you are trying to add a subdataset simply use `save`\".format(\n                    path))\n\n        if path is not None:\n            path = resolve_path(path, dataset)\n\n        # derive target from source:\n        if path is None:\n            # we got nothing but a source. do something similar to git clone\n            # and derive the path from the source and continue\n            # since this is a relative `path`, resolve it:\n            # we are not going to reuse the decoded URL, as this is done for\n            # all source candidates in clone_dataset(), we just use to determine\n            # a destination path here in order to perform a bunch of additional\n            # checks that shall not pollute the helper function\n            source_ = decode_source_spec(\n                source, cfg=None if ds is None else ds.config)\n            path = resolve_path(source_['default_destpath'], dataset)\n            lgr.debug(\"Determined clone target path from source\")\n        lgr.debug(\"Resolved clone target path to: '%s'\", path)\n\n        # there is no other way -- my intoxicated brain tells me\n        assert(path is not None)\n\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            refds=refds_path,\n            source_url=source)\n\n        try:\n            # this will implicitly cause pathlib to run a bunch of checks\n            # whether the present path makes any sense on the platform\n            # we are running on -- we don't care if the path actually\n            # exists at this point, but we want to abort early if the path\n            # spec is determined to be useless\n            path.exists()\n        except OSError as e:\n            yield get_status_dict(\n                status='error',\n                path=path,\n                message=('cannot handle target path: %s', exc_str(e)),\n                **result_props)\n            return\n\n        destination_dataset = Dataset(path)\n        result_props['ds'] = destination_dataset\n\n        if ds is not None and ds.pathobj not in path.parents:\n            yield get_status_dict(\n                status='error',\n                message=(\"clone target path '%s' not in specified target dataset '%s'\",\n                         path, ds),\n                **result_props)\n            return\n\n        # perform the actual cloning operation\n        yield from clone_dataset(\n            [source],\n            destination_dataset,\n            reckless,\n            description,\n            result_props,\n            cfg=None if ds is None else ds.config,\n        )\n\n        # TODO handle any 'version' property handling and verification using a dedicated\n        # public helper\n\n        if ds is not None:\n            # we created a dataset in another dataset\n            # -> make submodule\n            for r in ds.save(\n                    path,\n                    return_type='generator',\n                    result_filter=None,\n                    result_xfm=None,\n                    on_failure='ignore'):\n                yield r\n\n\ndef clone_dataset(\n        srcs,\n        destds,\n        reckless=None,\n        description=None,\n        result_props=None,\n        cfg=None):\n    \"\"\"Internal helper to perform cloning without sanity checks (assumed done)\n\n    This helper does not handle any saving of subdataset modification or adding\n    in a superdataset.\n\n    Parameters\n    ----------\n    srcs : list\n      Any suitable clone source specifications (paths, URLs)\n    destds : Dataset\n      Dataset instance for the clone destination\n    reckless : {None, 'auto'}, optional\n      Mode switch to put cloned dataset into throw-away configurations, i.e.\n      sacrifice data safety for performance or resource footprint.\n    description : str, optional\n      Location description for the annex of the dataset clone (if there is any).\n    result_props : dict, optional\n      Default properties for any yielded result, passed on to get_status_dict().\n    cfg : ConfigManager, optional\n      Configuration will be queried from this instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Yields\n    ------\n    dict\n      DataLad result records\n    \"\"\"\n    if not result_props:\n        # in case the caller had no specific idea on how results should look\n        # like, provide sensible defaults\n        result_props = dict(\n            action='install',\n            logger=lgr,\n            ds=destds,\n        )\n\n    dest_path = destds.pathobj\n\n    # decode all source candidate specifications\n    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]\n\n    # now expand the candidate sources with additional variants of the decoded\n    # giturl, while duplicating the other properties in the additional records\n    # for simplicity. The hope is to overcome a few corner cases and be more\n    # robust than git clone\n    candidate_sources = [\n        dict(props, giturl=s) for props in candidate_sources\n        for s in _get_flexible_source_candidates(props['giturl'])\n    ]\n\n    # important test! based on this `rmtree` will happen below after failed clone\n    dest_path_existed = dest_path.exists()\n    if dest_path_existed and any(dest_path.iterdir()):\n        if destds.is_installed():\n            # check if dest was cloned from the given source before\n            # this is where we would have installed this from\n            # this is where it was actually installed from\n            track_name, track_url = _get_tracking_source(destds)\n            try:\n                # this will get us track_url in system native path conventions,\n                # whenever it is a path (and not a URL)\n                # this is needed to match it to any potentially incoming local\n                # source path in the 'notneeded' test below\n                track_path = str(Path(track_url))\n            except Exception:\n                # this should never happen, because Path() will let any non-path stringification\n                # pass through unmodified, but we do not want any potential crash due to\n                # pathlib behavior changes\n                lgr.debug(\"Unexpected behavior of pathlib!\")\n                track_path = None\n            for cand in candidate_sources:\n                src = cand['giturl']\n                if track_url == src \\\n                        or get_local_file_url(track_url, compatibility='git') == src \\\n                        or track_path == expanduser(src):\n                    yield get_status_dict(\n                        status='notneeded',\n                        message=(\"dataset %s was already cloned from '%s'\",\n                                 destds,\n                                 src),\n                        **result_props)\n                    return\n        # anything else is an error\n        yield get_status_dict(\n            status='error',\n            message='target path already exists and not empty, refuse to clone into target path',\n            **result_props)\n        return\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Cloning dataset to %s', destds,\n        total=len(candidate_sources),\n        label='Clone attempt',\n        unit=' Candidate locations',\n    )\n    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url\n    for cand in candidate_sources:\n        try:\n            log_progress(\n                lgr.info,\n                'cloneds',\n                'Attempting to clone from %s to %s', cand['giturl'], dest_path,\n                update=1,\n                increment=True)\n            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35\n            # doesn't make it happen seemlessly\n            GitRepo.clone(path=str(dest_path), url=cand['giturl'], create=True)\n\n        except GitCommandError as e:\n            error_msgs[cand['giturl']] = e\n            lgr.debug(\"Failed to clone from URL: %s (%s)\",\n                      cand['giturl'], exc_str(e))\n            if dest_path.exists():\n                lgr.debug(\"Wiping out unsuccessful clone attempt at: %s\",\n                          dest_path)\n                # We must not just rmtree since it might be curdir etc\n                # we should remove all files/directories under it\n                # TODO stringification can be removed once patlib compatible\n                # or if PY35 is no longer supported\n                rmtree(str(dest_path), children_only=dest_path_existed)\n            # Whenever progress reporting is enabled, as it is now,\n            # we end up without e.stderr since it is \"processed\" out by\n            # GitPython/our progress handler.\n            e_stderr = e.stderr\n            from datalad.support.gitrepo import GitPythonProgressBar\n            if not e_stderr and GitPythonProgressBar._last_error_lines:\n                e_stderr = os.linesep.join(GitPythonProgressBar._last_error_lines)\n            if 'could not create work tree' in e_stderr.lower():\n                # this cannot be fixed by trying another URL\n                re_match = re.match(r\".*fatal: (.*)$\", e_stderr,\n                                    flags=re.MULTILINE | re.DOTALL)\n                # cancel progress bar\n                log_progress(\n                    lgr.info,\n                    'cloneds',\n                    'Completed clone attempts for %s', destds\n                )\n                yield get_status_dict(\n                    status='error',\n                    message=re_match.group(1) if re_match else \"stderr: \" + e_stderr,\n                    **result_props)\n                return\n            # next candidate\n            continue\n\n        # TODO perform any post-processing that needs to know details of the clone\n        # source\n\n        # do not bother with other sources if succeeded\n        break\n\n    log_progress(\n        lgr.info,\n        'cloneds',\n        'Completed clone attempts for %s', destds\n    )\n\n    if not destds.is_installed():\n        if len(error_msgs):\n            if all(not e.stdout and not e.stderr for e in error_msgs.values()):\n                # there is nothing we can learn from the actual exception,\n                # the exit code is uninformative, the command is predictable\n                error_msg = \"Failed to clone from all attempted sources: %s\"\n                error_args = list(error_msgs.keys())\n            else:\n                error_msg = \"Failed to clone from any candidate source URL. \" \\\n                            \"Encountered errors per each url were:\\n- %s\"\n                error_args = '\\n- '.join(\n                    '{}\\n  {}'.format(url, exc_str(exc))\n                    for url, exc in error_msgs.items()\n                )\n        else:\n            # yoh: Not sure if we ever get here but I felt that there could\n            #      be a case when this might happen and original error would\n            #      not be sufficient to troubleshoot what is going on.\n            error_msg = \"Awkward error -- we failed to clone properly. \" \\\n                        \"Although no errors were encountered, target \" \\\n                        \"dataset at %s seems to be not fully installed. \" \\\n                        \"The 'succesful' source was: %s\"\n            error_args = (destds.path, cand['giturl'])\n        yield get_status_dict(\n            status='error',\n            message=(error_msg, error_args),\n            **result_props)\n        return\n\n    yield from postclonecfg_annexdataset(\n        destds,\n        reckless,\n        description)\n\n    # yield successful clone of the base dataset now, as any possible\n    # subdataset clone down below will not alter the Git-state of the\n    # parent\n    yield get_status_dict(status='ok', **result_props)\n\n\ndef postclonecfg_annexdataset(ds, reckless, description=None):\n    \"\"\"If ds \"knows annex\" -- annex init it, set into reckless etc\n\n    Provides additional tune up to a possibly an annex repo, e.g.\n    \"enables\" reckless mode, sets up description\n    \"\"\"\n    # in any case check whether we need to annex-init the installed thing:\n    if not knows_annex(ds.path):\n        # not for us\n        return\n\n    # init annex when traces of a remote annex can be detected\n    if reckless:\n        lgr.debug(\n            \"Instruct annex to hardlink content in %s from local \"\n            \"sources, if possible (reckless)\", ds.path)\n        ds.config.add(\n            'annex.hardlink', 'true', where='local', reload=True)\n\n    # we have just cloned the repo, so it has 'origin', configure any\n    # reachable origin of origins\n    yield from configure_origins(ds, ds)\n\n    lgr.debug(\"Initializing annex repo at %s\", ds.path)\n    # Note, that we cannot enforce annex-init via AnnexRepo().\n    # If such an instance already exists, its __init__ will not be executed.\n    # Therefore do quick test once we have an object and decide whether to call its _init().\n    #\n    # Additionally, call init if we need to add a description (see #1403),\n    # since AnnexRepo.__init__ can only do it with create=True\n    repo = AnnexRepo(ds.path, init=True)\n    if not repo.is_initialized() or description:\n        repo._init(description=description)\n    if reckless:\n        repo._run_annex_command('untrust', annex_options=['here'])\n\n    srs = {True: [], False: []}  # special remotes by \"autoenable\" key\n    remote_uuids = None  # might be necessary to discover known UUIDs\n\n    # Note: The purpose of this function is to inform the user. So if something\n    # looks misconfigured, we'll warn and move on to the next item.\n    for uuid, config in repo.get_special_remotes().items():\n        sr_name = config.get('name', None)\n        if sr_name is None:\n            lgr.warning(\n                'Ignoring special remote %s because it does not have a name. '\n                'Known information: %s',\n                uuid, config)\n            continue\n        sr_autoenable = config.get('autoenable', False)\n        try:\n            sr_autoenable = assure_bool(sr_autoenable)\n        except ValueError:\n            lgr.warning(\n                'Failed to process \"autoenable\" value %r for sibling %s in '\n                'dataset %s as bool.  You might need to enable it later '\n                'manually and/or fix it up to avoid this message in the future.',\n                sr_autoenable, sr_name, ds.path)\n            continue\n\n        # determine whether there is a registered remote with matching UUID\n        if uuid:\n            if remote_uuids is None:\n                remote_uuids = {\n                    # Check annex-config-uuid first. For sameas annex remotes,\n                    # this will point to the UUID for the configuration (i.e.\n                    # the key returned by get_special_remotes) rather than the\n                    # shared UUID.\n                    (repo.config.get('remote.%s.annex-config-uuid' % r) or\n                     repo.config.get('remote.%s.annex-uuid' % r))\n                    for r in repo.get_remotes()\n                }\n            if uuid not in remote_uuids:\n                srs[sr_autoenable].append(sr_name)\n\n    if srs[True]:\n        lgr.debug(\n            \"configuration for %s %s added because of autoenable,\"\n            \" but no UUIDs for them yet known for dataset %s\",\n            # since we are only at debug level, we could call things their\n            # proper names\n            single_or_plural(\"special remote\", \"special remotes\", len(srs[True]), True),\n            \", \".join(srs[True]),\n            ds.path\n        )\n\n    if srs[False]:\n        # if has no auto-enable special remotes\n        lgr.info(\n            'access to %s %s not auto-enabled, enable with:\\n\\t\\tdatalad siblings -d \"%s\" enable -s %s',\n            # but since humans might read it, we better confuse them with our\n            # own terms!\n            single_or_plural(\"dataset sibling\", \"dataset siblings\", len(srs[False]), True),\n            \", \".join(srs[False]),\n            ds.path,\n            srs[False][0] if len(srs[False]) == 1 else \"SIBLING\",\n        )\n\n_handle_possible_annex_dataset = postclonecfg_annexdataset\n\n\ndef configure_origins(cfgds, probeds, label=None):\n    \"\"\"Configure any discoverable local dataset 'origin' sibling as a remote\n\n    Parameters\n    ----------\n    cfgds : Dataset\n      Dataset to receive the remote configurations\n    probeds : Dataset\n      Dataset to start looking for 'origin' remotes. May be identical with `cfgds`.\n    label : int, optional\n      Each discovered 'origin' will be configured as a remote under the name\n      'origin-<label>'. If no label is given, '2' will be used by default, given that\n      there is typically a 'origin' remote already.\n    \"\"\"\n    if label is None:\n        label = 2\n    # let's look at the URL for that remote and see if it is a local\n    # dataset\n    origin_url = probeds.config.get('remote.origin.url')\n    if origin_url and cfgds.config.obtain(\n            'datalad.install.inherit-local-origin',\n            default=True) and isinstance(RI(origin_url), PathRI):\n        # given the clone source is a local dataset, we can have a\n        # cheap look at it, and configure its own 'origin' as a remote\n        # (if there is any), and benefit from additional annex availability\n        originorigin_ds = Dataset(origin_url)\n        originorigin_url = originorigin_ds.config.get('remote.origin.url')\n        if originorigin_url:\n            yield from cfgds.siblings(\n                'configure',\n                # no chance for config, can only be the second configured remote\n                name='origin-{}'.format(label),\n                url=originorigin_url,\n                # fetch to get all annex info\n                fetch=True,\n                result_renderer='disabled',\n                on_failure='ignore',\n            )\n        # and dive deeper\n        yield from configure_origins(cfgds, originorigin_ds, label=label + 1)\n\n\ndef _get_tracking_source(ds):\n    \"\"\"Returns name and url of a potential configured source\n    tracking remote\"\"\"\n    vcs = ds.repo\n    # if we have a remote, let's check the location of that remote\n    # for the presence of the desired submodule\n\n    remote_name, tracking_branch = vcs.get_tracking_branch()\n    if not remote_name and isinstance(vcs, AnnexRepo):\n        # maybe cloned from a source repo that was in adjusted mode\n        # https://github.com/datalad/datalad/issues/3969\n        remote_name, tracking_branch = vcs.get_tracking_branch(\n            corresponding=False)\n    # TODO: better default `None`? Check where we might rely on '':\n    remote_url = ''\n    if remote_name:\n        remote_url = vcs.get_remote_url(remote_name, push=False)\n\n    return remote_name, remote_url\n\n\ndef _get_installationpath_from_url(url):\n    \"\"\"Returns a relative path derived from the trailing end of a URL\n\n    This can be used to determine an installation path of a Dataset\n    from a URL, analog to what `git clone` does.\n    \"\"\"\n    ri = RI(url)\n    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL\n        path = ri.path.rstrip('/')\n        path = urlunquote(path) if path else ri.hostname\n        if '/' in path:\n            path = path.split('/')\n            if path[-1] == '.git':\n                path = path[-2]\n            else:\n                path = path[-1]\n    else:\n        path = Path(url).parts[-1]\n    if path.endswith('.git'):\n        path = path[:-4]\n    return path\n\n\ndef decode_source_spec(spec, cfg=None):\n    \"\"\"Decode information from a clone source specification\n\n    Parameters\n    ----------\n    spec : str\n      Any supported clone source specification\n    cfg : ConfigManager, optional\n      Configuration will be queried from the instance (i.e. from a particular\n      dataset). If None is given, the global DataLad configuration will be\n      queried.\n\n    Returns\n    -------\n    dict\n      The value of each decoded property is stored under its own key in this\n      dict. By default the following keys are return: 'type', a specification\n      type label {'giturl', 'dataladri', 'ria'}; 'source' the original\n      source specification; 'giturl' a URL for the source that is a suitable\n      source argument for git-clone; 'version' a version-identifer, if present\n      (None else); 'default_destpath' a relative path that that can be used as\n      a clone destination.\n    \"\"\"\n    if cfg is None:\n        from datalad import cfg\n    # standard property dict composition\n    props = dict(\n        source=spec,\n        version=None,\n    )\n    # common starting point is a RI instance, support for accepting an RI\n    # instance is kept for backward-compatibility reasons\n    source_ri = RI(spec) if not isinstance(spec, RI) else spec\n\n    # scenario switch, each case must set 'giturl' at the very minimum\n    if isinstance(source_ri, DataLadRI):\n        # we have got our DataLadRI as the source, so expand it\n        props['type'] = 'dataladri'\n        props['giturl'] = source_ri.as_git_url()\n    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):\n        # parse a RIA URI\n        # check for store related configuration for SSH access\n        if source_ri.scheme == 'ria+ssh':\n            hostname = cfg.get(\n                'annex.ria-remote.{}.ssh-host'.format(source_ri.hostname),\n                source_ri.hostname)\n            basepath = cfg.get(\n                'annex.ria-remote.{}.base-path'.format(source_ri.hostname),\n                '')\n        else:\n            hostname = source_ri.hostname\n            basepath = ''\n\n        dsid, version = source_ri.fragment.split('@', maxsplit=1) \\\n            if '@' in source_ri.fragment else (source_ri.fragment, None)\n        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'\n        if not re.match(uuid_regex, dsid):\n            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))\n        props.update(\n            type='ria',\n            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(\n                proto=source_ri.scheme[4:],\n                host=hostname,\n                delim=':' if source_ri.scheme == 'ria+ssh' else '/',\n                basepath=basepath,\n                pathdelim='/' if basepath else '',\n                id1=dsid[:3],\n                id2=dsid[3:]),\n            version=version,\n            default_destpath=dsid,\n        )\n    else:\n        # let's assume that anything else is a URI that Git can handle\n        # TODO are there any advantages of going through a URL-reconversion\n        # by the RI instance, instead of using the original input right away?\n        # it feels like we could only accumulate bugs. Keeping how it was,\n        # for now...\n        props['type'] = 'giturl'\n        props['giturl'] = str(source_ri)\n\n    if 'default_destpath' not in props:\n        # if we still have no good idea on where a dataset could be cloned to if no\n        # path was given, do something similar to git clone and derive the path from\n        # the source\n        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])\n\n    return props\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title:** SSH URLs Generated with Incorrect Host-Path Delimiter

**Description:**

When generating SSH URLs for RIA repositories, the URLs incorrectly use a colon (`:`) to separate the host and path. This results in malformed URLs that do not conform to the expected format.

**Buggy Code Example:**

```python
spec = 'ria+ssh://example.com#6d69ca68-7e85-11e6-904c-002590f97d84@some_with@in_it'
props = decode_source_spec(spec)
print(props['giturl'])  # Outputs: 'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

**Expected Behavior:**

The `giturl` should use a slash (`/`) to separate the host and path, resulting in:
```
'ssh://example.com/6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

**Actual Behavior:**

The `giturl` incorrectly uses a colon (`:`) as the delimiter between the host and path:
```
'ssh://example.com:6d6/9ca68-7e85-11e6-904c-002590f97d84'
```

This malformed URL leads to failures in operations that rely on correctly formatted SSH URLs.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/core/distributed/clone.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Plumbing command for dataset installation"""


import logging
import os
import re
from os.path import expanduser
from collections import OrderedDict
from urllib.parse import unquote as urlunquote

from datalad.interface.base import Interface
from datalad.interface.utils import eval_results
from datalad.interface.base import build_doc
from datalad.interface.results import get_status_dict
from datalad.interface.common_opts import (
    location_description,
    reckless_opt,
)
from datalad.log import log_progress
from datalad.support.gitrepo import (
    GitRepo,
    GitCommandError,
)
from datalad.support.annexrepo import AnnexRepo
from datalad.support.constraints import (
    EnsureNone,
    EnsureStr,
    EnsureKeyChoice,
)
from datalad.support.param import Parameter
from datalad.support.network import (
    get_local_file_url,
    URL,
    RI,
    DataLadRI,
    PathRI,
)
from datalad.dochelpers import (
    exc_str,
    single_or_plural,
)
from datalad.utils import (
    rmtree,
    assure_bool,
    knows_annex,
    Path,
)

from datalad.distribution.dataset import (
    Dataset,
    datasetmethod,
    resolve_path,
    require_dataset,
    EnsureDataset,
)
from datalad.distribution.utils import (
    _get_flexible_source_candidates,
)

__docformat__ = 'restructuredtext'

lgr = logging.getLogger('datalad.core.distributed.clone')


@build_doc
class Clone(Interface):
    """Obtain a dataset (copy) from a URL or local directory

    The purpose of this command is to obtain a new clone (copy) of a dataset
    and place it into a not-yet-existing or empty directory. As such `clone`
    provides a strict subset of the functionality offered by `install`. Only a
    single dataset can be obtained, and immediate recursive installation of
    subdatasets is not supported. However, once a (super)dataset is installed
    via `clone`, any content, including subdatasets can be obtained by a
    subsequent `get` command.

    Primary differences over a direct `git clone` call are 1) the automatic
    initialization of a dataset annex (pure Git repositories are equally
    supported); 2) automatic registration of the newly obtained dataset
    as a subdataset (submodule), if a parent dataset is specified; and
    3) support for datalad's resource identifiers and automatic configurable
    generation of alternative access URL for common cases (such as appending
    '.git' to the URL in case the accessing the base URL failed).

    || PYTHON >>By default, the command returns a single Dataset instance for
    an installed dataset, regardless of whether it was newly installed ('ok'
    result), or found already installed from the specified source ('notneeded'
    result).<< PYTHON ||
    """
    # by default ignore everything but install results
    # i.e. no "add to super dataset"
    result_filter = EnsureKeyChoice('action', ('install',))
    # very frequently this command will yield exactly one installed dataset
    # spare people the pain of going through a list by default
    return_type = 'item-or-list'
    # as discussed in #1409 and #1470, we want to return dataset instances
    # matching what is actually available after command completion (and
    # None for any failed dataset installation)
    result_xfm = 'successdatasets-or-none'

    _examples_ = [
        dict(text="Install a dataset from Github into the current directory",
             code_py="clone("
             "source='https://github.com/datalad-datasets/longnow"
             "-podcasts.git')",
             code_cmd="datalad clone "
             "https://github.com/datalad-datasets/longnow-podcasts.git"),
        dict(text="Install a dataset into a specific directory",
             code_py="clone("
             "source='https://github.com/datalad-datasets/longnow"
             "-podcasts.git', path='myfavpodcasts')",
             code_cmd="datalad clone "
             "https://github.com/datalad-datasets/longnow-podcasts.git "
             "myfavpodcasts"),
        dict(text="Install a dataset as a subdataset into the current dataset",
             code_py="clone(dataset='.', "
             "source='https://github.com/datalad-datasets/longnow-podcasts.git')",
             code_cmd="datalad clone -d . "
             "--source='https://github.com/datalad-datasets/longnow-podcasts.git'"),
    ]

    _params_ = dict(
        dataset=Parameter(
            args=("-d", "--dataset"),
            doc="""(parent) dataset to clone into. If given, the newly cloned
            dataset is registered as a subdataset of the parent. Also, if given,
            relative paths are interpreted as being relative to the parent
            dataset, and not relative to the working directory.""",
            constraints=EnsureDataset() | EnsureNone()),
        source=Parameter(
            args=("source",),
            metavar='SOURCE',
            doc="""URL, DataLad resource identifier, local path or instance of
            dataset to be cloned""",
            constraints=EnsureStr() | EnsureNone()),
        path=Parameter(
            args=("path",),
            metavar='PATH',
            nargs="?",
            doc="""path to clone into.  If no `path` is provided a
            destination path will be derived from a source URL
            similar to :command:`git clone`"""),
        description=location_description,
        reckless=reckless_opt,
    )

    @staticmethod
    @datasetmethod(name='clone')
    @eval_results
    def __call__(
            source,
            path=None,
            dataset=None,
            description=None,
            reckless=None):
        # legacy compatibility
        if reckless is True:
            # so that we can forget about how things used to be
            reckless = 'auto'

        # did we explicitly get a dataset to install into?
        # if we got a dataset, path will be resolved against it.
        # Otherwise path will be resolved first.
        ds = require_dataset(
            dataset, check_installed=True, purpose='cloning') \
            if dataset is not None else dataset
        refds_path = ds.path if ds else None

        if isinstance(source, Dataset):
            source = source.path

        if source == path:
            # even if they turn out to be identical after resolving symlinks
            # and more sophisticated witchcraft, it would still happily say
            # "it appears to be already installed", so we just catch an
            # obviously pointless input combination
            raise ValueError(
                "clone `source` and destination `path` are identical [{}]. "
                "If you are trying to add a subdataset simply use `save`".format(
                    path))

        if path is not None:
            path = resolve_path(path, dataset)

        # derive target from source:
        if path is None:
            # we got nothing but a source. do something similar to git clone
            # and derive the path from the source and continue
            # since this is a relative `path`, resolve it:
            # we are not going to reuse the decoded URL, as this is done for
            # all source candidates in clone_dataset(), we just use to determine
            # a destination path here in order to perform a bunch of additional
            # checks that shall not pollute the helper function
            source_ = decode_source_spec(
                source, cfg=None if ds is None else ds.config)
            path = resolve_path(source_['default_destpath'], dataset)
            lgr.debug("Determined clone target path from source")
        lgr.debug("Resolved clone target path to: '%s'", path)

        # there is no other way -- my intoxicated brain tells me
        assert(path is not None)

        result_props = dict(
            action='install',
            logger=lgr,
            refds=refds_path,
            source_url=source)

        try:
            # this will implicitly cause pathlib to run a bunch of checks
            # whether the present path makes any sense on the platform
            # we are running on -- we don't care if the path actually
            # exists at this point, but we want to abort early if the path
            # spec is determined to be useless
            path.exists()
        except OSError as e:
            yield get_status_dict(
                status='error',
                path=path,
                message=('cannot handle target path: %s', exc_str(e)),
                **result_props)
            return

        destination_dataset = Dataset(path)
        result_props['ds'] = destination_dataset

        if ds is not None and ds.pathobj not in path.parents:
            yield get_status_dict(
                status='error',
                message=("clone target path '%s' not in specified target dataset '%s'",
                         path, ds),
                **result_props)
            return

        # perform the actual cloning operation
        yield from clone_dataset(
            [source],
            destination_dataset,
            reckless,
            description,
            result_props,
            cfg=None if ds is None else ds.config,
        )

        # TODO handle any 'version' property handling and verification using a dedicated
        # public helper

        if ds is not None:
            # we created a dataset in another dataset
            # -> make submodule
            for r in ds.save(
                    path,
                    return_type='generator',
                    result_filter=None,
                    result_xfm=None,
                    on_failure='ignore'):
                yield r


def clone_dataset(
        srcs,
        destds,
        reckless=None,
        description=None,
        result_props=None,
        cfg=None):
    """Internal helper to perform cloning without sanity checks (assumed done)

    This helper does not handle any saving of subdataset modification or adding
    in a superdataset.

    Parameters
    ----------
    srcs : list
      Any suitable clone source specifications (paths, URLs)
    destds : Dataset
      Dataset instance for the clone destination
    reckless : {None, 'auto'}, optional
      Mode switch to put cloned dataset into throw-away configurations, i.e.
      sacrifice data safety for performance or resource footprint.
    description : str, optional
      Location description for the annex of the dataset clone (if there is any).
    result_props : dict, optional
      Default properties for any yielded result, passed on to get_status_dict().
    cfg : ConfigManager, optional
      Configuration will be queried from this instance (i.e. from a particular
      dataset). If None is given, the global DataLad configuration will be
      queried.

    Yields
    ------
    dict
      DataLad result records
    """
    if not result_props:
        # in case the caller had no specific idea on how results should look
        # like, provide sensible defaults
        result_props = dict(
            action='install',
            logger=lgr,
            ds=destds,
        )

    dest_path = destds.pathobj

    # decode all source candidate specifications
    candidate_sources = [decode_source_spec(s, cfg=cfg) for s in srcs]

    # now expand the candidate sources with additional variants of the decoded
    # giturl, while duplicating the other properties in the additional records
    # for simplicity. The hope is to overcome a few corner cases and be more
    # robust than git clone
    candidate_sources = [
        dict(props, giturl=s) for props in candidate_sources
        for s in _get_flexible_source_candidates(props['giturl'])
    ]

    # important test! based on this `rmtree` will happen below after failed clone
    dest_path_existed = dest_path.exists()
    if dest_path_existed and any(dest_path.iterdir()):
        if destds.is_installed():
            # check if dest was cloned from the given source before
            # this is where we would have installed this from
            # this is where it was actually installed from
            track_name, track_url = _get_tracking_source(destds)
            try:
                # this will get us track_url in system native path conventions,
                # whenever it is a path (and not a URL)
                # this is needed to match it to any potentially incoming local
                # source path in the 'notneeded' test below
                track_path = str(Path(track_url))
            except Exception:
                # this should never happen, because Path() will let any non-path stringification
                # pass through unmodified, but we do not want any potential crash due to
                # pathlib behavior changes
                lgr.debug("Unexpected behavior of pathlib!")
                track_path = None
            for cand in candidate_sources:
                src = cand['giturl']
                if track_url == src \
                        or get_local_file_url(track_url, compatibility='git') == src \
                        or track_path == expanduser(src):
                    yield get_status_dict(
                        status='notneeded',
                        message=("dataset %s was already cloned from '%s'",
                                 destds,
                                 src),
                        **result_props)
                    return
        # anything else is an error
        yield get_status_dict(
            status='error',
            message='target path already exists and not empty, refuse to clone into target path',
            **result_props)
        return

    log_progress(
        lgr.info,
        'cloneds',
        'Cloning dataset to %s', destds,
        total=len(candidate_sources),
        label='Clone attempt',
        unit=' Candidate locations',
    )
    error_msgs = OrderedDict()  # accumulate all error messages formatted per each url
    for cand in candidate_sources:
        try:
            log_progress(
                lgr.info,
                'cloneds',
                'Attempting to clone from %s to %s', cand['giturl'], dest_path,
                update=1,
                increment=True)
            # TODO for now GitRepo.clone() cannot handle Path instances, and PY35
            # doesn't make it happen seemlessly
            GitRepo.clone(path=str(dest_path), url=cand['giturl'], create=True)

        except GitCommandError as e:
            error_msgs[cand['giturl']] = e
            lgr.debug("Failed to clone from URL: %s (%s)",
                      cand['giturl'], exc_str(e))
            if dest_path.exists():
                lgr.debug("Wiping out unsuccessful clone attempt at: %s",
                          dest_path)
                # We must not just rmtree since it might be curdir etc
                # we should remove all files/directories under it
                # TODO stringification can be removed once patlib compatible
                # or if PY35 is no longer supported
                rmtree(str(dest_path), children_only=dest_path_existed)
            # Whenever progress reporting is enabled, as it is now,
            # we end up without e.stderr since it is "processed" out by
            # GitPython/our progress handler.
            e_stderr = e.stderr
            from datalad.support.gitrepo import GitPythonProgressBar
            if not e_stderr and GitPythonProgressBar._last_error_lines:
                e_stderr = os.linesep.join(GitPythonProgressBar._last_error_lines)
            if 'could not create work tree' in e_stderr.lower():
                # this cannot be fixed by trying another URL
                re_match = re.match(r".*fatal: (.*)$", e_stderr,
                                    flags=re.MULTILINE | re.DOTALL)
                # cancel progress bar
                log_progress(
                    lgr.info,
                    'cloneds',
                    'Completed clone attempts for %s', destds
                )
                yield get_status_dict(
                    status='error',
                    message=re_match.group(1) if re_match else "stderr: " + e_stderr,
                    **result_props)
                return
            # next candidate
            continue

        # TODO perform any post-processing that needs to know details of the clone
        # source

        # do not bother with other sources if succeeded
        break

    log_progress(
        lgr.info,
        'cloneds',
        'Completed clone attempts for %s', destds
    )

    if not destds.is_installed():
        if len(error_msgs):
            if all(not e.stdout and not e.stderr for e in error_msgs.values()):
                # there is nothing we can learn from the actual exception,
                # the exit code is uninformative, the command is predictable
                error_msg = "Failed to clone from all attempted sources: %s"
                error_args = list(error_msgs.keys())
            else:
                error_msg = "Failed to clone from any candidate source URL. " \
                            "Encountered errors per each url were:\n- %s"
                error_args = '\n- '.join(
                    '{}\n  {}'.format(url, exc_str(exc))
                    for url, exc in error_msgs.items()
                )
        else:
            # yoh: Not sure if we ever get here but I felt that there could
            #      be a case when this might happen and original error would
            #      not be sufficient to troubleshoot what is going on.
            error_msg = "Awkward error -- we failed to clone properly. " \
                        "Although no errors were encountered, target " \
                        "dataset at %s seems to be not fully installed. " \
                        "The 'succesful' source was: %s"
            error_args = (destds.path, cand['giturl'])
        yield get_status_dict(
            status='error',
            message=(error_msg, error_args),
            **result_props)
        return

    yield from postclonecfg_annexdataset(
        destds,
        reckless,
        description)

    # yield successful clone of the base dataset now, as any possible
    # subdataset clone down below will not alter the Git-state of the
    # parent
    yield get_status_dict(status='ok', **result_props)


def postclonecfg_annexdataset(ds, reckless, description=None):
    """If ds "knows annex" -- annex init it, set into reckless etc

    Provides additional tune up to a possibly an annex repo, e.g.
    "enables" reckless mode, sets up description
    """
    # in any case check whether we need to annex-init the installed thing:
    if not knows_annex(ds.path):
        # not for us
        return

    # init annex when traces of a remote annex can be detected
    if reckless:
        lgr.debug(
            "Instruct annex to hardlink content in %s from local "
            "sources, if possible (reckless)", ds.path)
        ds.config.add(
            'annex.hardlink', 'true', where='local', reload=True)

    # we have just cloned the repo, so it has 'origin', configure any
    # reachable origin of origins
    yield from configure_origins(ds, ds)

    lgr.debug("Initializing annex repo at %s", ds.path)
    # Note, that we cannot enforce annex-init via AnnexRepo().
    # If such an instance already exists, its __init__ will not be executed.
    # Therefore do quick test once we have an object and decide whether to call its _init().
    #
    # Additionally, call init if we need to add a description (see #1403),
    # since AnnexRepo.__init__ can only do it with create=True
    repo = AnnexRepo(ds.path, init=True)
    if not repo.is_initialized() or description:
        repo._init(description=description)
    if reckless:
        repo._run_annex_command('untrust', annex_options=['here'])

    srs = {True: [], False: []}  # special remotes by "autoenable" key
    remote_uuids = None  # might be necessary to discover known UUIDs

    # Note: The purpose of this function is to inform the user. So if something
    # looks misconfigured, we'll warn and move on to the next item.
    for uuid, config in repo.get_special_remotes().items():
        sr_name = config.get('name', None)
        if sr_name is None:
            lgr.warning(
                'Ignoring special remote %s because it does not have a name. '
                'Known information: %s',
                uuid, config)
            continue
        sr_autoenable = config.get('autoenable', False)
        try:
            sr_autoenable = assure_bool(sr_autoenable)
        except ValueError:
            lgr.warning(
                'Failed to process "autoenable" value %r for sibling %s in '
                'dataset %s as bool.  You might need to enable it later '
                'manually and/or fix it up to avoid this message in the future.',
                sr_autoenable, sr_name, ds.path)
            continue

        # determine whether there is a registered remote with matching UUID
        if uuid:
            if remote_uuids is None:
                remote_uuids = {
                    # Check annex-config-uuid first. For sameas annex remotes,
                    # this will point to the UUID for the configuration (i.e.
                    # the key returned by get_special_remotes) rather than the
                    # shared UUID.
                    (repo.config.get('remote.%s.annex-config-uuid' % r) or
                     repo.config.get('remote.%s.annex-uuid' % r))
                    for r in repo.get_remotes()
                }
            if uuid not in remote_uuids:
                srs[sr_autoenable].append(sr_name)

    if srs[True]:
        lgr.debug(
            "configuration for %s %s added because of autoenable,"
            " but no UUIDs for them yet known for dataset %s",
            # since we are only at debug level, we could call things their
            # proper names
            single_or_plural("special remote", "special remotes", len(srs[True]), True),
            ", ".join(srs[True]),
            ds.path
        )

    if srs[False]:
        # if has no auto-enable special remotes
        lgr.info(
            'access to %s %s not auto-enabled, enable with:\n\t\tdatalad siblings -d "%s" enable -s %s',
            # but since humans might read it, we better confuse them with our
            # own terms!
            single_or_plural("dataset sibling", "dataset siblings", len(srs[False]), True),
            ", ".join(srs[False]),
            ds.path,
            srs[False][0] if len(srs[False]) == 1 else "SIBLING",
        )

_handle_possible_annex_dataset = postclonecfg_annexdataset


def configure_origins(cfgds, probeds, label=None):
    """Configure any discoverable local dataset 'origin' sibling as a remote

    Parameters
    ----------
    cfgds : Dataset
      Dataset to receive the remote configurations
    probeds : Dataset
      Dataset to start looking for 'origin' remotes. May be identical with `cfgds`.
    label : int, optional
      Each discovered 'origin' will be configured as a remote under the name
      'origin-<label>'. If no label is given, '2' will be used by default, given that
      there is typically a 'origin' remote already.
    """
    if label is None:
        label = 2
    # let's look at the URL for that remote and see if it is a local
    # dataset
    origin_url = probeds.config.get('remote.origin.url')
    if origin_url and cfgds.config.obtain(
            'datalad.install.inherit-local-origin',
            default=True) and isinstance(RI(origin_url), PathRI):
        # given the clone source is a local dataset, we can have a
        # cheap look at it, and configure its own 'origin' as a remote
        # (if there is any), and benefit from additional annex availability
        originorigin_ds = Dataset(origin_url)
        originorigin_url = originorigin_ds.config.get('remote.origin.url')
        if originorigin_url:
            yield from cfgds.siblings(
                'configure',
                # no chance for config, can only be the second configured remote
                name='origin-{}'.format(label),
                url=originorigin_url,
                # fetch to get all annex info
                fetch=True,
                result_renderer='disabled',
                on_failure='ignore',
            )
        # and dive deeper
        yield from configure_origins(cfgds, originorigin_ds, label=label + 1)


def _get_tracking_source(ds):
    """Returns name and url of a potential configured source
    tracking remote"""
    vcs = ds.repo
    # if we have a remote, let's check the location of that remote
    # for the presence of the desired submodule

    remote_name, tracking_branch = vcs.get_tracking_branch()
    if not remote_name and isinstance(vcs, AnnexRepo):
        # maybe cloned from a source repo that was in adjusted mode
        # https://github.com/datalad/datalad/issues/3969
        remote_name, tracking_branch = vcs.get_tracking_branch(
            corresponding=False)
    # TODO: better default `None`? Check where we might rely on '':
    remote_url = ''
    if remote_name:
        remote_url = vcs.get_remote_url(remote_name, push=False)

    return remote_name, remote_url


def _get_installationpath_from_url(url):
    """Returns a relative path derived from the trailing end of a URL

    This can be used to determine an installation path of a Dataset
    from a URL, analog to what `git clone` does.
    """
    ri = RI(url)
    if isinstance(ri, (URL, DataLadRI)):  # decode only if URL
        path = ri.path.rstrip('/')
        path = urlunquote(path) if path else ri.hostname
        if '/' in path:
            path = path.split('/')
            if path[-1] == '.git':
                path = path[-2]
            else:
                path = path[-1]
    else:
        path = Path(url).parts[-1]
    if path.endswith('.git'):
        path = path[:-4]
    return path


def decode_source_spec(spec, cfg=None):
    """Decode information from a clone source specification

    Parameters
    ----------
    spec : str
      Any supported clone source specification
    cfg : ConfigManager, optional
      Configuration will be queried from the instance (i.e. from a particular
      dataset). If None is given, the global DataLad configuration will be
      queried.

    Returns
    -------
    dict
      The value of each decoded property is stored under its own key in this
      dict. By default the following keys are return: 'type', a specification
      type label {'giturl', 'dataladri', 'ria'}; 'source' the original
      source specification; 'giturl' a URL for the source that is a suitable
      source argument for git-clone; 'version' a version-identifer, if present
      (None else); 'default_destpath' a relative path that that can be used as
      a clone destination.
    """
    if cfg is None:
        from datalad import cfg
    # standard property dict composition
    props = dict(
        source=spec,
        version=None,
    )
    # common starting point is a RI instance, support for accepting an RI
    # instance is kept for backward-compatibility reasons
    source_ri = RI(spec) if not isinstance(spec, RI) else spec

    # scenario switch, each case must set 'giturl' at the very minimum
    if isinstance(source_ri, DataLadRI):
        # we have got our DataLadRI as the source, so expand it
        props['type'] = 'dataladri'
        props['giturl'] = source_ri.as_git_url()
    elif isinstance(source_ri, URL) and source_ri.scheme.startswith('ria+'):
        # parse a RIA URI
        # check for store related configuration for SSH access
        if source_ri.scheme == 'ria+ssh':
            hostname = cfg.get(
                'annex.ria-remote.{}.ssh-host'.format(source_ri.hostname),
                source_ri.hostname)
            basepath = cfg.get(
                'annex.ria-remote.{}.base-path'.format(source_ri.hostname),
                '')
        else:
            hostname = source_ri.hostname
            basepath = ''

        dsid, version = source_ri.fragment.split('@', maxsplit=1) \
            if '@' in source_ri.fragment else (source_ri.fragment, None)
        uuid_regex = r'[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'
        if not re.match(uuid_regex, dsid):
            raise ValueError('RIA URI does not contain a valid dataset ID: {}'.format(spec))
        props.update(
            type='ria',
            giturl='{proto}://{host}{delim}{basepath}{pathdelim}{id1}/{id2}'.format(
                proto=source_ri.scheme[4:],
                host=hostname,
                delim=':' if source_ri.scheme == 'ria+ssh' else '/',
                basepath=basepath,
                pathdelim='/' if basepath else '',
                id1=dsid[:3],
                id2=dsid[3:]),
            version=version,
            default_destpath=dsid,
        )
    else:
        # let's assume that anything else is a URI that Git can handle
        # TODO are there any advantages of going through a URL-reconversion
        # by the RI instance, instead of using the original input right away?
        # it feels like we could only accumulate bugs. Keeping how it was,
        # for now...
        props['type'] = 'giturl'
        props['giturl'] = str(source_ri)

    if 'default_destpath' not in props:
        # if we still have no good idea on where a dataset could be cloned to if no
        # path was given, do something similar to git clone and derive the path from
        # the source
        props['default_destpath'] = _get_installationpath_from_url(props['giturl'])

    return props

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user